~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
Python academiaanthropicarxivbravedeep-researchencryptionhome-automationhomeserverlocallocal-deep-researchlocal-llmmistralollamaopenaipubmedresearchresearch-toolretrieval-augmented-generationsearxngself-hosted
ci(release): build-once-promote refactor for Docker pipeline (#3977)
LearningCircuit1f0b0a4a95ef
diff --git a/.github/workflows/docker-publish.yml b/.github/workflows/docker-publish.ymlindex 99e0705330..f4eb081269 100644--- a/.github/workflows/docker-publish.yml+++ b/.github/workflows/docker-publish.yml@@ -1,187 +1,69 @@name: Publish Docker image-# SECURITY: Only triggered via repository_dispatch from release.yml-# (after security gate passes). We intentionally do NOT support-# workflow_dispatch because that would bypass security checks.+# SECURITY: Only invoked as a reusable workflow (`workflow_call`) from+# release.yml after the release security gate AND the `release` environment+# approval pass. We intentionally do NOT support workflow_dispatch — that+# would bypass gates. We also intentionally do NOT support+# repository_dispatch any more: with the atomicity refactor, this workflow+# runs as a job inside release.yml's run so its result is visible to+# downstream jobs (create-release, cleanup-on-rejection) — a property that+# repository_dispatch fanout broke.+#+# This workflow is a thin RETAG step in the build-once-promote pipeline.+# The actual build happens in prerelease-docker.yml; the multi-arch manifest+# is signed and attested there. Here we only:+# - Verify the source manifest digest matches what prerelease produced+# - Retag the prerelease manifest to release tags (:1.6.9, :1.6, :latest)+# - Verify the digest is preserved (defends against imagetools re-encoding)+# - Re-run Trivy against the digest (catches CVE-database updates between+# prerelease build and release promote)+# - Verify cosign signature transitivity from the original digest+# - Clean up the prerelease tags+## To re-publish, trigger a new release through release.yml.on:- repository_dispatch:- types: [publish-docker]+ workflow_call:+ inputs:+ tag:+ description: "Release tag, e.g. 'v1.6.9' (with leading 'v')"+ type: string+ required: true+ source_tag:+ description: "Prerelease manifest tag to retag, e.g. 'prerelease-v1.6.9-abc1234'"+ type: string+ required: true+ expected_digest:+ description: "sha256:... digest of the prerelease manifest, captured by prerelease-docker.yml. Used to verify retag preserves the digest end-to-end."+ type: string+ required: true+ secrets:+ DOCKER_USERNAME:+ required: true+ description: "Docker Hub username (env-scoped to `release`)"+ DOCKER_PASSWORD:+ required: true+ description: "Docker Hub PAT with Read+Write+Delete scopes (env-scoped to `release`)"permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions-jobs:- build-amd64:- name: Build AMD64 Image- runs-on: ubuntu-latest- environment: release- permissions:- contents: read- outputs:- digest: ${{ steps.build.outputs.digest }}-- steps:- - name: Harden Runner- uses: step-security/harden-runner@ab7a9404c0f3da075243ca237b5fac12c98deaa5 # v2.19.3- with:- egress-policy: audit-- - name: Check out the repo- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2- with:- persist-credentials: false- fetch-depth: 0-- - name: Set up Docker Buildx- uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4.0.0-- - name: Log in to Docker Hub- uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4.1.0- with:- username: ${{ secrets.DOCKER_USERNAME }}- password: ${{ secrets.DOCKER_PASSWORD }}-- - name: Build and push AMD64 image- id: build- uses: docker/build-push-action@bcafcacb16a39f128d818304e6c9c0c18556b85f # v7.1.0- with:- context: .- platforms: linux/amd64- push: true- tags: ${{ secrets.DOCKER_USERNAME }}/local-deep-research:amd64-${{ github.sha }}- cache-from: type=gha,scope=linux-amd64- cache-to: type=gha,mode=max,scope=linux-amd64- build-args: |- DEPS_HASH=${{ hashFiles('pdm.lock') }}-- build-arm64:- name: Build ARM64 Image- runs-on: ubuntu-24.04-arm- environment: release- permissions:- contents: read- outputs:- digest: ${{ steps.build.outputs.digest }}-- steps:- - name: Harden Runner- uses: step-security/harden-runner@ab7a9404c0f3da075243ca237b5fac12c98deaa5 # v2.19.3- with:- egress-policy: audit-- - name: Check out the repo- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2- with:- persist-credentials: false- fetch-depth: 0-- - name: Set up Docker Buildx- uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4.0.0-- - name: Log in to Docker Hub- uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4.1.0- with:- username: ${{ secrets.DOCKER_USERNAME }}- password: ${{ secrets.DOCKER_PASSWORD }}-- - name: Build and push ARM64 image- id: build- uses: docker/build-push-action@bcafcacb16a39f128d818304e6c9c0c18556b85f # v7.1.0- with:- context: .- platforms: linux/arm64- push: true- tags: ${{ secrets.DOCKER_USERNAME }}/local-deep-research:arm64-${{ github.sha }}- cache-from: type=gha,scope=linux-arm64- cache-to: type=gha,mode=max,scope=linux-arm64- build-args: |- DEPS_HASH=${{ hashFiles('pdm.lock') }}-- security-scan:- name: Security Scan- runs-on: ubuntu-latest- environment: release- permissions:- contents: read-- steps:- - name: Harden the runner (Audit all outbound calls)- uses: step-security/harden-runner@ab7a9404c0f3da075243ca237b5fac12c98deaa5 # v2.19.3- with:- egress-policy: audit-- - name: Check out the repo- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2- with:- persist-credentials: false- fetch-depth: 0+# NOTE: no workflow-level `concurrency:` block. As a reusable workflow+# called from release.yml, this workflow runs as part of the caller's run,+# and release.yml's caller-level concurrency (keyed on github.workflow ++# github.ref) already serialises release runs for the same tag. Adding a+# callee-level block would be a no-op for the documented threat model+# (two simultaneous releases for the same ref) and would not be reachable+# anyway because there is no longer any standalone invocation path.- - name: Free disk space- run: |- # Remove unnecessary packages to free up space for Docker build + Trivy scan- # GitHub runners have limited space; Trivy needs to export the full image to /tmp- sudo rm -rf /usr/share/dotnet || true- sudo rm -rf /usr/local/lib/android || true- sudo rm -rf /opt/ghc || true- sudo rm -rf /opt/hostedtoolcache/CodeQL || true- sudo docker image prune --all --force || true- df -h-- - name: Set up Docker Buildx- uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4.0.0-- - name: Build Docker image for security scan- uses: docker/build-push-action@bcafcacb16a39f128d818304e6c9c0c18556b85f # v7.1.0- with:- context: .- platforms: linux/amd64- push: false- load: true- tags: local-deep-research:security-scan- cache-from: type=gha,scope=linux-amd64- cache-to: type=gha,mode=max,scope=linux-amd64- build-args: |- DEPS_HASH=${{ hashFiles('pdm.lock') }}-- # Generate SARIF report for GitHub Security tab (all severities, doesn't fail)- - name: Generate Trivy SARIF report- uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25 # v0.36.0- with:- image-ref: local-deep-research:security-scan- format: 'sarif'- output: 'trivy-release-scan.sarif'- ignore-unfixed: true- exit-code: '0'- version: 'v0.69.2'-- # Separate scan that fails build only on fixable HIGH/CRITICAL vulnerabilities- - name: Check for fixable HIGH/CRITICAL vulnerabilities- uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25 # v0.36.0- with:- image-ref: local-deep-research:security-scan- severity: 'CRITICAL,HIGH'- ignore-unfixed: true # Only fail on vulnerabilities with available fixes- trivyignores: '.trivyignore' # Ignore bundled library CVEs that can't be fixed- exit-code: '1'- version: 'v0.69.2'-- - name: Upload Trivy scan results from release- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1- if: always()- with:- name: trivy-release-scan- path: trivy-release-scan.sarif- retention-days: 7 # Reduced for security-- create-manifest:- name: Create Multi-Platform Manifest- needs: [build-amd64, build-arm64, security-scan]+jobs:+ promote:+ name: Retag prerelease manifest as releaseruns-on: ubuntu-latestenvironment: releasepermissions:+ # cosign verify is read-only against public Rekor/Fulcio — does not+ # mint a GitHub OIDC token, so id-token: write is not required here.+ # Signing (which does need id-token: write) happens in prerelease-docker.yml.contents: read- id-token: write # Required for Sigstore keyless signing- packages: writesteps:- name: Harden Runner@@ -189,9 +71,19 @@ jobs:with:egress-policy: audit- - name: Check out the repo+ - name: Check out the repo at the triggering commit+ # Pin the checkout to the EXACT commit that the prerelease was+ # built/scanned from, so .trivyignore (and any other repo-state-+ # dependent file the promote step reads) matches that commit.+ # We use github.sha, NOT inputs.tag — the v* git tag is created+ # by create-release LATER in this run (after publish-docker+ # completes), so it does not exist yet when this checkout runs+ # on a push-to-main trigger. github.sha is the triggering commit+ # for every event type (push to main, tag push, workflow_dispatch)+ # and is the same SHA the build/prerelease-docker jobs used.uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2with:+ ref: ${{ github.sha }}persist-credentials: falsefetch-depth: 0@@ -206,192 +98,158 @@ jobs:- name: Install Cosignuses: sigstore/cosign-installer@6f9f17788090df1f26f669e9d70d6ae9567deba6 # v4.1.2+ with:+ # Pin to cosign v2.x to match the version that signed the artifact+ # in prerelease-docker.yml. Mismatched versions across sign/verify+ # work today but new-bundle-format (cosign v3 default) would only+ # produce/consume on v3.+ cosign-release: 'v2.6.3'- - name: Install Syft- uses: anchore/sbom-action/download-syft@e22c389904149dbc22b58101806040fa8d37a610 # v0.24.0-- - name: Determine version tag+ - name: Determine release tagsid: versionenv:- EVENT_NAME: ${{ github.event_name }}- DISPATCH_TAG: ${{ github.event.client_payload.tag }}- RELEASE_TAG: ${{ github.event.release.tag_name }}+ DISPATCH_TAG: ${{ inputs.tag }}+ SOURCE_TAG: ${{ inputs.source_tag }}+ EXPECTED_DIGEST: ${{ inputs.expected_digest }}run: |set -euo pipefail- if [ "$EVENT_NAME" = "repository_dispatch" ]; then- TAG="$DISPATCH_TAG"- else- TAG="$RELEASE_TAG"+ if [[ -z "$DISPATCH_TAG" || -z "$SOURCE_TAG" || -z "$EXPECTED_DIGEST" ]]; then+ echo "::error::Missing required workflow_call input. Got tag='${DISPATCH_TAG}' source_tag='${SOURCE_TAG}' expected_digest='${EXPECTED_DIGEST}'"+ exit 1+ fi+ if [[ "$EXPECTED_DIGEST" != sha256:* ]]; then+ echo "::error::expected_digest must be of the form sha256:... — got '${EXPECTED_DIGEST}'"+ exit 1fi- echo "tag=$TAG" >> "$GITHUB_OUTPUT"- # Extract version without 'v' prefix- VERSION="${TAG#v}"- echo "version=$VERSION" >> "$GITHUB_OUTPUT"- # Extract major.minor+ VERSION="${DISPATCH_TAG#v}"MAJOR_MINOR=$(echo "$VERSION" | cut -d. -f1,2)- echo "major_minor=$MAJOR_MINOR" >> "$GITHUB_OUTPUT"-- - name: Extract metadata for Docker- id: meta- uses: docker/metadata-action@030e881283bb7a6894de51c315a6bfe6a94e05cf # v6.0.0- with:- images: ${{ secrets.DOCKER_USERNAME }}/local-deep-research- tags: |- type=raw,value=${{ steps.version.outputs.version }}- type=raw,value=${{ steps.version.outputs.major_minor }}- type=raw,value=latest-- - name: Create and push multi-platform manifest- id: manifest- env:- DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}- META_TAGS: ${{ steps.meta.outputs.tags }}- run: |- set -euo pipefail- # Get the tags from metadata (newline separated)- TAGS="$META_TAGS"-- # Store first tag for attestation- FIRST_TAG=$(echo "$TAGS" | head -n 1)- echo "primary_tag=$FIRST_TAG" >> "$GITHUB_OUTPUT"-- # Create manifest for each tag- while IFS= read -r tag; do- if [ -n "$tag" ]; then- echo "Creating manifest for: $tag"- docker buildx imagetools create -t "$tag" \- "${DOCKER_USERNAME}/local-deep-research:amd64-${{ github.sha }}" \- "${DOCKER_USERNAME}/local-deep-research:arm64-${{ github.sha }}"- fi- done <<< "$TAGS"-- # Get the manifest digest for the primary tag (used for signing)- DIGEST=$(docker buildx imagetools inspect "$FIRST_TAG" --format '{{json .Manifest.Digest}}' | tr -d '"')- echo "digest=$DIGEST" >> "$GITHUB_OUTPUT"- echo "Manifest digest: $DIGEST"-- - name: Sign Docker images with Cosign+ # Group writes into one redirect (shellcheck SC2129).+ {+ echo "tag=${DISPATCH_TAG}"+ echo "version=${VERSION}"+ echo "major_minor=${MAJOR_MINOR}"+ echo "source_tag=${SOURCE_TAG}"+ echo "expected_digest=${EXPECTED_DIGEST}"+ } >> "$GITHUB_OUTPUT"++ - name: Verify source digest matches expectedenv:- DIGEST: ${{ steps.manifest.outputs.digest }}DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}+ SOURCE_TAG: ${{ steps.version.outputs.source_tag }}+ EXPECTED_DIGEST: ${{ steps.version.outputs.expected_digest }}run: |set -euo pipefail- # Sign by digest for reliability with manifest lists- # All tags point to the same manifest, so we sign once by digest- IMAGE_REF="${DOCKER_USERNAME}/local-deep-research@${DIGEST}"- echo "Signing image by digest: $IMAGE_REF"- cosign sign --yes "$IMAGE_REF"-- # Brief sleep to allow registry to propagate signature- echo "Waiting for signature propagation..."- sleep 5+ # Defends against the prerelease tag being swapped between+ # prerelease-docker's signing and this promote step.+ SOURCE="${DOCKER_USERNAME}/local-deep-research:${SOURCE_TAG}"+ ACTUAL=$(docker buildx imagetools inspect "$SOURCE" --format '{{json .Manifest.Digest}}' | tr -d '"')+ if [[ "$ACTUAL" != "$EXPECTED_DIGEST" ]]; then+ echo "::error::Source digest mismatch — possible tag tampering between prerelease and promote"+ echo " expected: $EXPECTED_DIGEST"+ echo " actual: $ACTUAL"+ exit 1+ fi+ echo "Source digest verified: $ACTUAL"- - name: Generate SLSA provenance attestation+ - name: Promote (retag) prerelease manifest to release tagsenv:- DIGEST: ${{ steps.manifest.outputs.digest }}DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}+ SOURCE_TAG: ${{ steps.version.outputs.source_tag }}+ VERSION: ${{ steps.version.outputs.version }}+ MAJOR_MINOR: ${{ steps.version.outputs.major_minor }}run: |set -euo pipefail- IMAGE_REF="${DOCKER_USERNAME}/local-deep-research@${DIGEST}"-- # Generate SLSA provenance predicate- # Note: SLSA spec requires "sha1" field name for git commit digest- cat > provenance.json <<EOF- {- "buildType": "https://github.com/${{ github.repository }}/docker-build@v1",- "builder": {- "id": "https://github.com/actions/runner"- },- "invocation": {- "configSource": {- "uri": "https://github.com/${{ github.repository }}",- "digest": {- "sha1": "${{ github.sha }}"- },- "entryPoint": ".github/workflows/docker-publish.yml"- }- },- "metadata": {- "buildInvocationId": "${{ github.run_id }}",- "buildStartedOn": "$(date -u +%Y-%m-%dT%H:%M:%SZ)",- "completeness": {- "parameters": true,- "environment": true,- "materials": true- },- "reproducible": false- },- "materials": [- {- "uri": "https://github.com/${{ github.repository }}",- "digest": {- "sha1": "${{ github.sha }}"- }- }- ]- }- EOF-- # Attach provenance to image by digest- cosign attest --yes --predicate provenance.json --type slsaprovenance "$IMAGE_REF"-- - name: Verify image signature+ SOURCE="${DOCKER_USERNAME}/local-deep-research:${SOURCE_TAG}"+ # Single imagetools create with multiple -t — registry-side+ # metadata-only operation, takes seconds, preserves digest.+ docker buildx imagetools create \+ -t "${DOCKER_USERNAME}/local-deep-research:${VERSION}" \+ -t "${DOCKER_USERNAME}/local-deep-research:${MAJOR_MINOR}" \+ -t "${DOCKER_USERNAME}/local-deep-research:latest" \+ "$SOURCE"+ echo "Promoted ${SOURCE} to :${VERSION}, :${MAJOR_MINOR}, :latest"++ - name: Verify promoted tags share the source digestenv:- DIGEST: ${{ steps.manifest.outputs.digest }}DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}+ VERSION: ${{ steps.version.outputs.version }}+ MAJOR_MINOR: ${{ steps.version.outputs.major_minor }}+ EXPECTED_DIGEST: ${{ steps.version.outputs.expected_digest }}run: |set -euo pipefail- IMAGE_REF="${DOCKER_USERNAME}/local-deep-research@${DIGEST}"- echo "Verifying signature for: $IMAGE_REF"-- # Retry logic to handle registry propagation delay after signing- MAX_RETRIES=5- RETRY_DELAY=10- for i in $(seq 1 "$MAX_RETRIES"); do- echo "Verification attempt $i of $MAX_RETRIES..."- if cosign verify \- --certificate-identity-regexp="https://github.com/${{ github.repository }}" \- --certificate-oidc-issuer=https://token.actions.githubusercontent.com \- "$IMAGE_REF"; then- echo "Signature verification successful!"- exit 0- fi- if [ "$i" -lt "$MAX_RETRIES" ]; then- echo "Verification failed, waiting ${RETRY_DELAY}s before retry..."- sleep "$RETRY_DELAY"+ # Defends against `imagetools create` re-encoding the manifest.+ # If digests diverge, signatures and attestations (keyed by the+ # original digest) won't be discoverable from the new tags.+ for TAG in "${VERSION}" "${MAJOR_MINOR}" "latest"; do+ REF="${DOCKER_USERNAME}/local-deep-research:${TAG}"+ ACTUAL=$(docker buildx imagetools inspect "$REF" --format '{{json .Manifest.Digest}}' | tr -d '"')+ if [[ "$ACTUAL" != "$EXPECTED_DIGEST" ]]; then+ echo "::error::Digest mismatch on ${TAG} — imagetools create may have re-encoded the manifest"+ echo " expected: $EXPECTED_DIGEST"+ echo " actual: $ACTUAL"+ exit 1fi+ echo "${TAG} -> ${ACTUAL} ✓"done- echo "Signature verification failed after $MAX_RETRIES attempts"- exit 1- - name: Attach SBOM to image+ # Catches CVE-database updates that landed between the prerelease+ # build and this promote step. Use the SHA-pinned action wrapper+ # (same pin as prerelease-docker.yml's security-scan) with an+ # explicit binary version pin — the prior `apt-get install -y trivy`+ # approach was unpinned and exposed the release path to the Trivy+ # apt-repo supply chain. The pinned action downloads the v0.69.2+ # binary from GitHub releases by exact tag, which is the same+ # binary the prerelease scan validated, keeping the two scans+ # consistent.+ #+ # Unlike prerelease-docker.yml's security-scan (which scans a+ # locally-loaded image, no registry pull), this step scans by+ # registry digest — Trivy must pull the manifest + layers from+ # Docker Hub. TRIVY_USERNAME/TRIVY_PASSWORD is the action's+ # documented auth path; the `docker/login-action` above also+ # writes ~/.docker/config.json which Trivy reads as a fallback,+ # but the explicit env vars are more reliable (Trivy has+ # documented docker.io credential-helper quirks — aquasecurity/+ # trivy#432, aquasecurity/trivy#8385) and the image we scan IS on+ # Docker Hub so this is the path most likely to keep working.+ - name: Re-scan release digest with Trivy+ uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25 # v0.36.0env:- DIGEST: ${{ steps.manifest.outputs.digest }}- DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}- run: |- set -euo pipefail- IMAGE_REF="${DOCKER_USERNAME}/local-deep-research@${DIGEST}"-- # Generate SBOM using syft- docker pull "$IMAGE_REF"- syft "$IMAGE_REF" -o spdx-json > sbom.spdx.json-- # Attach SBOM to image by digest- cosign attach sbom --sbom sbom.spdx.json "$IMAGE_REF"-- # Sign the SBOM- cosign sign --yes --attachment sbom "$IMAGE_REF"-- - name: Upload SBOM artifact- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1+ TRIVY_USERNAME: ${{ secrets.DOCKER_USERNAME }}+ TRIVY_PASSWORD: ${{ secrets.DOCKER_PASSWORD }}with:- name: sbom- path: sbom.spdx.json- retention-days: 90+ image-ref: ${{ secrets.DOCKER_USERNAME }}/local-deep-research@${{ steps.version.outputs.expected_digest }}+ severity: 'CRITICAL,HIGH'+ ignore-unfixed: true+ trivyignores: '.trivyignore'+ exit-code: '1'+ version: 'v0.69.2'- - name: Clean up temporary tags+ - name: Verify cosign signature on promoted digest+ env:+ DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}+ EXPECTED_DIGEST: ${{ steps.version.outputs.expected_digest }}+ REPO: ${{ github.repository }}run: |- echo "Manifest creation complete. Temporary platform-specific tags can be removed manually if desired."+ set -euo pipefail+ # The cert was issued to the prerelease-docker.yml workflow when+ # signing happened there, so the identity regex must match that+ # workflow's path. Fulcio's SAN is built from job_workflow_ref,+ # which for reusable workflows is the CALLEE.+ #+ # Verify by IMMUTABLE digest (not the :VERSION tag) so this step+ # is invariant under any retag race between the verify-promoted-+ # tags step above and this one. Trivy's re-scan above also uses+ # @${EXPECTED_DIGEST}; keeping cosign on the same reference is+ # consistent and avoids a tag-resolution TOCTOU window.+ IMAGE_REF="${DOCKER_USERNAME}/local-deep-research@${EXPECTED_DIGEST}"+ echo "Verifying signature for: $IMAGE_REF"+ cosign verify \+ --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \+ --certificate-identity-regexp "^https://github.com/${REPO}/\.github/workflows/prerelease-docker\.yml@refs/(heads|tags)/" \+ --certificate-github-workflow-repository "${REPO}" \+ "$IMAGE_REF"+ echo "Signature transitivity verified ✓"- name: Clean up prerelease tagscontinue-on-error: true@@ -454,3 +312,23 @@ jobs:doneecho "Prerelease tag cleanup complete. Deleted ${DELETED} tag(s) matching ${PREFIX}*."++ - name: Summary+ env:+ DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}+ VERSION: ${{ steps.version.outputs.version }}+ MAJOR_MINOR: ${{ steps.version.outputs.major_minor }}+ EXPECTED_DIGEST: ${{ steps.version.outputs.expected_digest }}+ run: |+ {+ echo "## Docker Release Promoted"+ echo ""+ echo "**Digest:** \`${EXPECTED_DIGEST}\`"+ echo ""+ echo "**Tags:** \`${VERSION}\`, \`${MAJOR_MINOR}\`, \`latest\` — all share the same digest as the prerelease manifest, so cosign signatures, SBOM, and SLSA provenance from the prerelease step are transitively valid."+ echo ""+ echo '```'+ echo "docker pull ${DOCKER_USERNAME}/local-deep-research:${VERSION}"+ echo "docker pull ${DOCKER_USERNAME}/local-deep-research:latest"+ echo '```'+ } >> "$GITHUB_STEP_SUMMARY"diff --git a/.github/workflows/prerelease-docker.yml b/.github/workflows/prerelease-docker.ymlindex 7569efab70..0c648ed993 100644--- a/.github/workflows/prerelease-docker.yml+++ b/.github/workflows/prerelease-docker.yml@@ -1,17 +1,56 @@name: Prerelease Docker Image-# Build a prerelease Docker image for local testing before the official-# release is published. Triggered exclusively via repository_dispatch from-# release.yml (after all gates pass and approval is granted).+# Build the canonical Docker image for a release. In the build-once-promote+# pipeline, this workflow IS the build — docker-publish.yml only retags the+# manifest produced here. Cosign signing, SBOM attestation, and SLSA+# provenance are attached here once, keyed by manifest digest, so they're+# discoverable from any tag (including the release tags later created by+# imagetools create).#-# NO workflow_dispatch — security is enforced upstream in release.yml.+# Triggered exclusively via workflow_call from release.yml (after security+# gates pass). No workflow_dispatch — security and gate semantics are+# enforced by the caller. The build runs automatically; the only human+# approval in the release flow is the `release` env on this workflow's+# jobs + publish-docker + trigger-pypi + create-release in release.yml+# (gates the actual publish, not the canonical build).on:- repository_dispatch:- types: [publish-prerelease-docker]+ workflow_call:+ inputs:+ version:+ description: "Bare semver, e.g. '1.6.9' (no leading 'v')"+ type: string+ required: true+ short_sha:+ description: "First 7 chars of commit SHA (used in the prerelease tag)"+ type: string+ required: true+ secrets:+ # Explicit secrets contract instead of `secrets: inherit` on the+ # caller side — narrower blast radius if a future caller misuses+ # this reusable workflow.+ DOCKER_USERNAME:+ required: true+ description: "Docker Hub username for image push"+ DOCKER_PASSWORD:+ required: true+ description: "Docker Hub PAT (Read+Write+Delete scopes)"+ outputs:+ manifest_digest:+ description: "sha256:... digest of the multi-arch prerelease manifest. Used by docker-publish.yml to verify retag preserves the digest."+ value: ${{ jobs.create-manifest.outputs.digest }}permissions: {} # Minimal top-level for OSSF Scorecard Token-Permissions+# No approval gate at the build step — the build runs automatically once+# security gates and CI gates in release.yml pass. The only meaningful+# human decision in the release flow is "should this signed, attested,+# tested image become the official release?" — gated by the `release`+# environment on this workflow's jobs + `publish-docker` + `trigger-pypi`+# + `create-release` in release.yml. The maintainer can pull+# `:prerelease-v<ver>-<sha>` and smoke-test between build completion+# and approving the release env.+jobs:build-amd64:name: Build AMD64 Prerelease Image@@ -48,11 +87,9 @@ jobs:context: .platforms: linux/amd64push: true- tags: ${{ secrets.DOCKER_USERNAME }}/local-deep-research:prerelease-v${{ github.event.client_payload.version }}-${{ github.event.client_payload.short_sha }}-amd64+ tags: ${{ secrets.DOCKER_USERNAME }}/local-deep-research:prerelease-v${{ inputs.version }}-${{ inputs.short_sha }}-amd64cache-from: type=gha,scope=linux-amd64cache-to: type=gha,mode=max,scope=linux-amd64- build-args: |- DEPS_HASH=${{ hashFiles('pdm.lock') }}build-arm64:name: Build ARM64 Prerelease Image@@ -89,11 +126,9 @@ jobs:context: .platforms: linux/arm64push: true- tags: ${{ secrets.DOCKER_USERNAME }}/local-deep-research:prerelease-v${{ github.event.client_payload.version }}-${{ github.event.client_payload.short_sha }}-arm64+ tags: ${{ secrets.DOCKER_USERNAME }}/local-deep-research:prerelease-v${{ inputs.version }}-${{ inputs.short_sha }}-arm64cache-from: type=gha,scope=linux-arm64cache-to: type=gha,mode=max,scope=linux-arm64- build-args: |- DEPS_HASH=${{ hashFiles('pdm.lock') }}security-scan:name: Security Scan@@ -136,8 +171,6 @@ jobs:tags: local-deep-research:security-scancache-from: type=gha,scope=linux-amd64cache-to: type=gha,mode=max,scope=linux-amd64- build-args: |- DEPS_HASH=${{ hashFiles('pdm.lock') }}# Generate Trivy SARIF for archival as a workflow artifact (all severities, never fails).# Severity-gating happens in the next step.@@ -177,6 +210,11 @@ jobs:environment: releasepermissions:contents: read+ id-token: write # Required for cosign keyless OIDC signing+ # No `packages: write` — Docker Hub auth uses DOCKER_PASSWORD secret,+ # not GITHUB_TOKEN. `packages: write` only matters for ghcr.io pushes.+ outputs:+ digest: ${{ steps.capture-digest.outputs.digest }}steps:- name: Harden Runner@@ -199,11 +237,22 @@ jobs:username: ${{ secrets.DOCKER_USERNAME }}password: ${{ secrets.DOCKER_PASSWORD }}+ - name: Install Cosign+ uses: sigstore/cosign-installer@6f9f17788090df1f26f669e9d70d6ae9567deba6 # v4.1.2+ with:+ # Pin to cosign v2.x — see release.yml for rationale (v3 enables+ # --new-bundle-format by default which changes the on-wire format+ # and breaks downstream verifiers still on v2).+ cosign-release: 'v2.6.3'++ - name: Install Syft+ uses: anchore/sbom-action/download-syft@e22c389904149dbc22b58101806040fa8d37a610 # v0.24.0+- name: Create and push multi-platform manifestenv:DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}- VERSION: ${{ github.event.client_payload.version }}- SHORT_SHA: ${{ github.event.client_payload.short_sha }}+ VERSION: ${{ inputs.version }}+ SHORT_SHA: ${{ inputs.short_sha }}run: |set -euo pipefailTAG="prerelease-v${VERSION}-${SHORT_SHA}"@@ -216,17 +265,220 @@ jobs:# Floating tag: re-point :prerelease at the manifest just created so# testers can pin compose to `:prerelease` and pull the latest RC via# `docker compose pull` without editing the tag each cycle. The- # versioned tag above remains for reproducibility.+ # versioned tag above remains for reproducibility (and is what+ # docker-publish.yml retags by digest into :1.6.9 / :1.6 / :latest).echo "Updating floating tag: ${DOCKER_USERNAME}/local-deep-research:prerelease"docker buildx imagetools create -t "${DOCKER_USERNAME}/local-deep-research:prerelease" \"${DOCKER_USERNAME}/local-deep-research:${TAG}"echo "Floating :prerelease tag updated"+ - name: Capture manifest digest+ id: capture-digest+ env:+ DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}+ VERSION: ${{ inputs.version }}+ SHORT_SHA: ${{ inputs.short_sha }}+ run: |+ set -euo pipefail+ IMAGE_REF="${DOCKER_USERNAME}/local-deep-research:prerelease-v${VERSION}-${SHORT_SHA}"+ # Same form as the existing docker-publish.yml inspector — avoids jq.+ DIGEST=$(docker buildx imagetools inspect "$IMAGE_REF" --format '{{json .Manifest.Digest}}' | tr -d '"')+ if [[ -z "$DIGEST" || "$DIGEST" != sha256:* ]]; then+ echo "::error::Failed to capture manifest digest (got '${DIGEST}')"+ exit 1+ fi+ echo "digest=${DIGEST}" >> "$GITHUB_OUTPUT"+ echo "Manifest digest: ${DIGEST}"++ - name: Sign manifest with Cosign+ env:+ DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}+ DIGEST: ${{ steps.capture-digest.outputs.digest }}+ run: |+ set -euo pipefail+ # Sign by digest — signature artifact lands at sha256-<digest>.sig+ # in the same repo, discoverable from ANY tag pointing at the same+ # digest (including release tags created later by docker-publish.yml's+ # imagetools-create retag).+ IMAGE_REF="${DOCKER_USERNAME}/local-deep-research@${DIGEST}"+ echo "Signing image by digest: $IMAGE_REF"+ cosign sign --yes "$IMAGE_REF"+ # Brief sleep to allow registry to propagate signature+ sleep 5++ - name: Generate SLSA provenance attestation+ env:+ DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}+ DIGEST: ${{ steps.capture-digest.outputs.digest }}+ run: |+ set -euo pipefail+ IMAGE_REF="${DOCKER_USERNAME}/local-deep-research@${DIGEST}"++ # entryPoint is the TOP-LEVEL caller (release.yml), not this+ # reusable workflow. Per SLSA GHA buildtype v1 and the canonical+ # slsa-github-generator, reusable workflows are explicitly NOT+ # entryPoints. github.run_id / github.repository / github.sha all+ # resolve to the caller's run context inside a reusable workflow.+ # builder.id pins the workflow that actually defines the build+ # steps — the trust root a verifier policy can pin against. We+ # compose it from `github.repository` and a hardcoded path to+ # THIS workflow file, with `github.ref` for the ref portion.+ # Rationale: inside a workflow_call callee, the `github` context+ # is scoped to the CALLER, so `github.workflow_ref` would point+ # at release.yml (the wrong builder). The `job` context has no+ # `workflow_ref` property either (only check_run_id, container,+ # services, status — actionlint confirms). For a local-path+ # reusable workflow (`uses: ./.github/workflows/...`), the+ # callee's ref equals the caller's `github.ref`, so composing+ # the path manually gives the correct+ # `<owner>/<repo>/.github/workflows/prerelease-docker.yml@<ref>`+ # format that matches the Fulcio cert SAN. Cosign and+ # slsa-verifier both anchor on the cert anyway, so this fix is+ # about correctness for raw-JSON policy engines / audit tools+ # that read builder.id directly.+ #+ # completeness.* are FALSE because we don't capture invocation+ # parameters or environment, and the build does network I/O for+ # apt/pip/npm. Honest emptiness > false claims of completeness.+ #+ # buildInvocationId includes run_attempt so re-runs are+ # distinguishable in audit logs.+ cat > provenance.json <<EOF+ {+ "buildType": "https://github.com/${{ github.repository }}/docker-build@v1",+ "builder": {+ "id": "https://github.com/${{ github.repository }}/.github/workflows/prerelease-docker.yml@${{ github.ref }}"+ },+ "invocation": {+ "configSource": {+ "uri": "https://github.com/${{ github.repository }}",+ "digest": {+ "sha1": "${{ github.sha }}"+ },+ "entryPoint": ".github/workflows/release.yml"+ }+ },+ "metadata": {+ "buildInvocationId": "${{ github.run_id }}-${{ github.run_attempt }}",+ "buildStartedOn": "$(date -u +%Y-%m-%dT%H:%M:%SZ)",+ "completeness": {+ "parameters": false,+ "environment": false,+ "materials": false+ },+ "reproducible": false+ },+ "materials": [+ {+ "uri": "https://github.com/${{ github.repository }}",+ "digest": {+ "sha1": "${{ github.sha }}"+ }+ }+ ]+ }+ EOF++ # --replace prevents duplicate SLSA attestations on re-run. Cosign's+ # Replace logic is keyed by predicate-type URI, so it leaves the+ # SBOM SPDX attestation (different predicateType) untouched.+ cosign attest --yes --replace --predicate provenance.json --type slsaprovenance "$IMAGE_REF"++ - name: Verify image signature+ env:+ DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}+ DIGEST: ${{ steps.capture-digest.outputs.digest }}+ run: |+ set -euo pipefail+ IMAGE_REF="${DOCKER_USERNAME}/local-deep-research@${DIGEST}"+ echo "Verifying signature for: $IMAGE_REF"+ # Retry to handle registry propagation delay after signing+ MAX_RETRIES=5+ RETRY_DELAY=10+ for i in $(seq 1 "$MAX_RETRIES"); do+ echo "Verification attempt $i of $MAX_RETRIES..."+ if cosign verify \+ --certificate-identity-regexp="https://github.com/${{ github.repository }}" \+ --certificate-oidc-issuer=https://token.actions.githubusercontent.com \+ "$IMAGE_REF"; then+ echo "Signature verification successful!"+ exit 0+ fi+ if [ "$i" -lt "$MAX_RETRIES" ]; then+ echo "Verification failed, waiting ${RETRY_DELAY}s before retry..."+ sleep "$RETRY_DELAY"+ fi+ done+ echo "::error::Signature verification failed after $MAX_RETRIES attempts"+ exit 1++ - name: Generate per-platform SBOMs and attest each+ env:+ DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}+ DIGEST: ${{ steps.capture-digest.outputs.digest }}+ run: |+ set -euo pipefail+ REPO="${DOCKER_USERNAME}/local-deep-research"+ MANIFEST_REF="${REPO}@${DIGEST}"++ # Multi-arch SBOM correctness: syft against a manifest list digest+ # only scans the host platform's layers (per anchore/syft#1708),+ # which would lie to ARM64 consumers. We attest each per-arch+ # digest with its OWN SBOM so end-user verification is honest.+ # We deliberately do NOT also produce a "manifest-level SBOM" —+ # that would be amd64-only (host arch) and re-introduce the lie+ # for any arm64 consumer running the README verifier recipe.+ # The README documents the per-arch verification flow instead.+ MANIFEST_JSON=$(docker buildx imagetools inspect "${MANIFEST_REF}" --raw)++ # Defense against future buildx output changes: assert at least+ # one per-arch entry exists. Without this, an empty/malformed+ # manifest list would silently produce zero SBOMs and pass CI green.+ PER_ARCH_COUNT=$(echo "${MANIFEST_JSON}" \+ | jq '[.manifests[] | select(.platform.architecture != "unknown")] | length')+ if [[ "${PER_ARCH_COUNT}" -lt 1 ]]; then+ echo "::error::No per-arch manifest entries found in ${MANIFEST_REF} — SBOM generation cannot proceed"+ echo "Raw manifest: ${MANIFEST_JSON}"+ exit 1+ fi+ echo "Found ${PER_ARCH_COUNT} per-arch manifest(s) to scan"++ echo "$MANIFEST_JSON" \+ | jq -r '.manifests[] | select(.platform.architecture != "unknown") | "\(.platform.os)/\(.platform.architecture)\t\(.digest)"' \+ | while IFS=$'\t' read -r PLAT PER_ARCH_DIGEST; do+ ARCH="${PLAT##*/}"+ PER_ARCH_REF="${REPO}@${PER_ARCH_DIGEST}"+ SBOM_FILE="sbom-${ARCH}.spdx.json"+ echo "=== Scanning ${PLAT} (${PER_ARCH_DIGEST}) ==="+ # --platform tells syft which arch to scan — matters when+ # the host runner can't natively execute the image.+ syft --platform "${PLAT}" "${PER_ARCH_REF}" -o spdx-json > "${SBOM_FILE}"+ # --replace prevents accumulation when a re-run lands on+ # the same digest (e.g. "Re-run failed jobs" after a flake).+ # Per cosign source pkg/cosign/remote/remote.go, --replace+ # is per-predicate-type, so it doesn't disturb the SLSA+ # attestation already on the manifest list digest.+ cosign attest --yes --replace \+ --predicate "${SBOM_FILE}" --type spdxjson "${PER_ARCH_REF}"+ done++ - name: Upload SBOMs artifact+ uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1+ with:+ name: sbom+ # Per-arch SBOMs only — `sbom-amd64.spdx.json`, `sbom-arm64.spdx.json`,+ # one per platform in the manifest list. No manifest-level SBOM is+ # produced (would be host-arch-only and misleading for non-amd64+ # consumers).+ path: sbom-*.spdx.json+ retention-days: 90+- name: Summaryenv:DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}- VERSION: ${{ github.event.client_payload.version }}- SHORT_SHA: ${{ github.event.client_payload.short_sha }}+ VERSION: ${{ inputs.version }}+ SHORT_SHA: ${{ inputs.short_sha }}+ DIGEST: ${{ steps.capture-digest.outputs.digest }}run: |TAG="prerelease-v${VERSION}-${SHORT_SHA}"{@@ -235,8 +487,14 @@ jobs:echo "**Versioned tag:** \`${TAG}\`"echo "**Floating tag:** \`prerelease\` (now points at this build)"echo ""+ echo "**Digest:** \`${DIGEST}\`"+ echo ""echo '```'echo "docker pull ${DOCKER_USERNAME}/local-deep-research:${TAG}"echo "docker pull ${DOCKER_USERNAME}/local-deep-research:prerelease"echo '```'+ echo ""+ echo "Signed and attested. After release approval, docker-publish.yml"+ echo "will retag this exact digest as \`:${VERSION}\`, \`:major.minor\`,"+ echo "and \`:latest\`."} >> "$GITHUB_STEP_SUMMARY"diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.ymlindex 24f5aff571..67c160519b 100644--- a/.github/workflows/publish.yml+++ b/.github/workflows/publish.yml@@ -17,7 +17,11 @@ jobs:outputs:has-frontend: ${{ steps.check.outputs.has-frontend }}container:- image: node:20-alpine@sha256:bcd88137d802e2482c9df3cdec71e0431857ebbbdba6973776b5593214056d86 # node:20-alpine+ # Node 24 to match `package.json`'s `engines: { node: ">=24.0.0" }`.+ # Was previously node:20 — npm could resolve dependencies that target+ # APIs missing on 20 and the wheel-building publish path could ship+ # frontend assets that break at runtime on the Node-24 Docker image.+ image: node:24-alpine@sha256:d1b3b4da11eefd5941e7f0b9cf17783fc99d9c6fc34884a665f40a06dbdfc94f # node:24-alpine# Note: Network is needed for npm ci to work, but no secrets are availableoptions: --user 1001permissions:diff --git a/.github/workflows/release-gate.yml b/.github/workflows/release-gate.ymlindex bb876fab83..4af3be4537 100644--- a/.github/workflows/release-gate.yml+++ b/.github/workflows/release-gate.yml@@ -343,7 +343,12 @@ jobs:- name: Install Node.js for Vite builduses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0with:- node-version: '20'+ # Match package.json's `engines: { node: ">=24.0.0" }` and+ # publish.yml's node:24-alpine. Building the wheel-tested+ # frontend on Node 20 here while the real publish runs on+ # Node 24 means the gate validates a different runtime than+ # ships, defeating the gate's stated purpose.+ node-version: '24'- name: Set up PDM (build only)uses: pdm-project/setup-pdm@973541a5febeafcfdadf8a51211435be6ecfd90f # v4.5diff --git a/.github/workflows/release.yml b/.github/workflows/release.ymlindex fbaa257b3d..7073a3f4af 100644--- a/.github/workflows/release.yml+++ b/.github/workflows/release.yml@@ -86,9 +86,13 @@ jobs:# The build job depends on this gate, ensuring no release can be created# without passing all security checks.#- # NOTE: Docker/PyPI publishing only happens via repository_dispatch from this- # workflow. Creating a release via GitHub UI will NOT trigger publishing- # (this is by design to prevent security gate bypass).+ # NOTE: Docker publishing now runs inline (reusable workflow_call to+ # docker-publish.yml), and PyPI publishing is still dispatched via+ # repository_dispatch to publish.yml (PyPI Trusted Publishing does not+ # support reusable workflows — see pypa/gh-action-pypi-publish#166 and+ # pypi/warehouse#11096). Neither publish path is triggered by creating a+ # release via the GitHub UI — only this workflow can dispatch them — so+ # the security-gate flow is preserved.# ============================================================================release-gate:needs: [version-check]@@ -282,41 +286,65 @@ jobs:fiecho "Version verified: $EXPECTED_VERSION"- - name: Generate SBOM (Software Bill of Materials)+ # SBOM generation uses the SHA-pinned trivy-action with an explicit+ # binary version pin. Previously this step shelled out+ # `apt-get install -y trivy` from the (unpinned) Aqua apt repo, which+ # in a job carrying `id-token: write` (line 229) meant a compromised+ # apt mirror could exfiltrate the OIDC request token AND tamper with+ # the SBOM bytes that the next step (Sign release artifacts) signs+ # under this repo's Sigstore identity — i.e., a fraudulent SBOM+ # carrying a legitimate cosign cert. Pin the action by SHA + the+ # binary by version tag so the toolchain is reproducible and matches+ # docker-publish.yml / prerelease-docker.yml.+ - name: Generate SBOM (SPDX JSON)if: steps.check_release.outputs.exists == 'false'- run: |- echo "=== Generating Software Bill of Materials ==="-- # Install Trivy for SBOM generation (using signed-by for secure key management)- curl -fsSL https://aquasecurity.github.io/trivy-repo/deb/public.key | gpg --dearmor | sudo tee /usr/share/keyrings/trivy.gpg > /dev/null- echo "deb [signed-by=/usr/share/keyrings/trivy.gpg] https://aquasecurity.github.io/trivy-repo/deb generic main" | sudo tee /etc/apt/sources.list.d/trivy.list- sudo apt-get update- sudo apt-get install -y trivy-- # Generate SBOM in SPDX format (JSON)- trivy fs --format spdx-json --output sbom-spdx.json .- echo "Generated SBOM in SPDX-JSON format"+ uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25 # v0.36.0+ with:+ scan-type: 'fs'+ scan-ref: '.'+ format: 'spdx-json'+ output: 'sbom-spdx.json'+ version: 'v0.69.2'- # Generate SBOM in CycloneDX format (JSON)- trivy fs --format cyclonedx --output sbom-cyclonedx.json .- echo "Generated SBOM in CycloneDX format"+ - name: Generate SBOM (CycloneDX)+ if: steps.check_release.outputs.exists == 'false'+ uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25 # v0.36.0+ with:+ scan-type: 'fs'+ scan-ref: '.'+ format: 'cyclonedx'+ output: 'sbom-cyclonedx.json'+ version: 'v0.69.2'- # Display summary- echo ""+ - name: List SBOMs+ if: steps.check_release.outputs.exists == 'false'+ run: |echo "SBOM files generated:"ls -lh sbom-*.json- name: Install Cosignif: steps.check_release.outputs.exists == 'false'uses: sigstore/cosign-installer@6f9f17788090df1f26f669e9d70d6ae9567deba6 # v4.1.2+ with:+ # Pin to cosign v2.x — v4.1.2 of the installer ships cosign v3.0.6+ # by default, but v3 enables --new-bundle-format ON BY DEFAULT.+ # That changes the signature/attestation on-wire format and would+ # require all downstream verifiers (including end-users running+ # the README cosign-verify recipe) to also be on v3+. Pin to+ # v2.6.3 (latest v2.x, includes the GHSA-w6c6-c85g-mmv6 patch)+ # for the legacy tag-based sigstore format until we explicitly+ # migrate to v3.+ cosign-release: 'v2.6.3'- name: Sign release artifacts with Sigstoreif: steps.check_release.outputs.exists == 'false'run: |echo "=== Signing release artifacts with Sigstore ==="- # Sign SBOM files using keyless signing (OIDC)- # cosign v3.0.2+ uses --bundle which contains signature, certificate, and metadata in one file+ # Sign SBOM files using keyless signing (OIDC).+ # `--bundle` writes a Sigstore protobuf bundle containing the+ # signature, certificate, and Rekor inclusion proof in one file.+ # Supported in cosign v2.4.0+ (we're pinned to v2.6.3 above).cosign sign-blob --yes --bundle sbom-spdx.json.bundle sbom-spdx.jsoncosign sign-blob --yes --bundle sbom-cyclonedx.json.bundle sbom-cyclonedx.json@@ -364,9 +392,87 @@ jobs:provenance-name: "provenance.intoto.jsonl"compile-generator: true # Build from source to bypass TUF key validation issues- create-release:+ # Build, sign, and push the canonical Docker image. This is the single+ # build for the release — publish-docker (the reusable workflow_call+ # to docker-publish.yml) later retags this manifest by digest. The jobs+ # inside this reusable workflow declare `environment: release` so they+ # can read the env-scoped DOCKER_USERNAME / DOCKER_PASSWORD; the FIRST+ # (and only) `release` env approval click in this run unlocks ALL+ # release-env jobs together: prerelease-docker, publish-docker,+ # trigger-pypi, monitor-pypi, and create-release. After the+ # atomicity refactor, docker-publish is inline (reusable workflow_call)+ # so it does not require a separate second approval — its result is+ # visible to downstream jobs (create-release, cleanup-on-rejection) in+ # the same run. (Earlier iterations of the build-once-promote refactor+ # tried to run the canonical build pre-approval to give a real+ # test-then-approve window; that required repo-level Docker Hub+ # secrets, which we deliberately don't have.) The manifest_digest+ # output flows through to the promote step so digest preservation can+ # be verified end-to-end.+ prerelease-docker:needs: [build, provenance]if: ${{ !cancelled() && needs.build.result == 'success' && needs.provenance.result == 'success' && needs.build.outputs.release_exists == 'false' }}+ uses: ./.github/workflows/prerelease-docker.yml+ with:+ version: ${{ needs.build.outputs.version }}+ short_sha: ${{ needs.build.outputs.short_sha }}+ secrets:+ # Explicit list (not `secrets: inherit`) — defense-in-depth so a+ # future edit to prerelease-docker.yml that references an unrelated+ # secret would fail loudly rather than silently accessing inherited+ # values like PAT_TOKEN, OPENROUTER_API_KEY, or SERPER_API_KEY.+ DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}+ DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }}+ permissions:+ # Caller permissions cap what the called workflow can request.+ # Must include id-token: write so cosign keyless OIDC works on the+ # called workflow's create-manifest job.+ # NOTE: no `packages: write` — Docker Hub auth uses DOCKER_PASSWORD+ # secret, not GITHUB_TOKEN. `packages: write` only matters for ghcr.io.+ contents: read+ id-token: write++ # Retag the prerelease manifest as the release tags (:VERSION, :MAJOR_MINOR,+ # :latest). Runs as a reusable workflow_call so its result is visible to+ # downstream jobs in this run — specifically: create-release blocks on+ # success here, and cleanup-on-rejection keys its cosign-deletion logic on+ # `needs.publish-docker.result`. Before this refactor, docker-publish.yml+ # was triggered via repository_dispatch and its outcome was invisible to+ # the parent release.yml run, which made cosign artifact cleanup unsafe+ # after a partial retag failure (release tags could exist sharing the+ # prerelease manifest digest, and deleting `sha256-<digest>.{sig,att}`+ # would invalidate those release-tag signatures).+ publish-docker:+ needs: [build, prerelease-docker]+ if: ${{ !cancelled() && needs.prerelease-docker.result == 'success' && needs.build.outputs.release_exists == 'false' }}+ uses: ./.github/workflows/docker-publish.yml+ with:+ tag: ${{ needs.build.outputs.tag }}+ source_tag: prerelease-v${{ needs.build.outputs.version }}-${{ needs.build.outputs.short_sha }}+ expected_digest: ${{ needs.prerelease-docker.outputs.manifest_digest }}+ secrets:+ # Explicit secret pass — env-scoped (`release`) DOCKER_USERNAME and+ # DOCKER_PASSWORD become available to the callee's `promote` job+ # because that job declares `environment: release`.+ DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}+ DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }}+ permissions:+ # cosign VERIFY (the only sigstore operation in the callee) is a+ # read-only check against the public Rekor/Fulcio infrastructure —+ # no GitHub OIDC token is minted, so id-token: write is unnecessary.+ # OIDC is only needed for cosign SIGN, which happens upstream in+ # prerelease-docker.yml.+ contents: read++ create-release:+ # create-release waits for prerelease-docker (canonical image exists,+ # signed and attested), publish-docker (release tags retagged with+ # digest preserved, cosign transitivity verified), and monitor-pypi+ # (PyPI publish has completed). Only then is the GitHub Release+ # published — so the Release page never points at non-existent+ # artifacts. Gated by the `release` environment.+ needs: [build, provenance, prerelease-docker, publish-docker, monitor-pypi]+ if: ${{ !cancelled() && needs.build.result == 'success' && needs.provenance.result == 'success' && needs.prerelease-docker.result == 'success' && needs.publish-docker.result == 'success' && needs.monitor-pypi.result == 'success' && needs.build.outputs.release_exists == 'false' }}runs-on: ubuntu-latestenvironment: releasepermissions:@@ -436,6 +542,35 @@ jobs:with:name: provenance.intoto.jsonl+ - name: Download container-image SBOMs+ # Produced by prerelease-docker.yml — one SBOM per architecture+ # (sbom-amd64.spdx.json, sbom-arm64.spdx.json) from Syft. The+ # cosign attestations on each per-arch digest are the+ # cryptographically authoritative copies; attaching the raw JSON+ # to the GitHub Release makes them discoverable to humans+ # browsing the release page.+ uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1+ with:+ name: sbom+ path: container-sbom++ - name: Stage container-image SBOMs for release attachment+ # Rename per-arch SBOMs to avoid collision with the filesystem SBOM+ # `sbom-spdx.json` produced by the build job (Trivy scan of source+ # tree). Container SBOMs are the Syft scan of built image layers+ # per platform — different content.+ run: |+ set -euo pipefail+ # Per-arch SBOMs: rename sbom-amd64.spdx.json → sbom-container-amd64.spdx.json+ # (and similarly for arm64). prerelease-docker.yml no longer+ # produces a manifest-level sbom.spdx.json.+ for f in container-sbom/sbom-*.spdx.json; do+ [ -f "$f" ] || continue+ base=$(basename "$f")+ mv "$f" "sbom-container-${base#sbom-}"+ done+ ls -la sbom-container*.spdx.json || echo "No container SBOMs found"+- name: List artifactsrun: |echo "Release artifacts:"@@ -780,7 +915,18 @@ jobs:print(f"Release body truncated to {len(content)} chars", file=sys.stderr)PYEOF- # Includes SBOM files, Sigstore signatures, and SLSA provenance+ # Includes filesystem SBOM (Trivy scan of source tree) AND+ # container-image SBOM (Syft scan of built image layers, per-arch),+ # Sigstore signatures, and SLSA provenance. The two SBOM kinds+ # describe different surfaces — both are useful for downstream+ # supply-chain consumers.+ # `find ... -print0 | xargs -0 ...` handles the variable number+ # of per-arch container SBOMs gracefully (zero or more).+ set -euo pipefail+ CONTAINER_SBOMS=()+ while IFS= read -r -d '' f; do+ CONTAINER_SBOMS+=("$f")+ done < <(find . -maxdepth 1 -name 'sbom-container*.spdx.json' -print0)gh release create "$RELEASE_TAG" \--repo "$GITHUB_REPOSITORY" \--title "Release $RELEASE_VERSION" \@@ -789,11 +935,17 @@ jobs:sbom-spdx.json.bundle \sbom-cyclonedx.json \sbom-cyclonedx.json.bundle \- provenance.intoto.jsonl+ provenance.intoto.jsonl \+ "${CONTAINER_SBOMS[@]}"env:- # PAT_TOKEN required here because GITHUB_TOKEN cannot trigger downstream- # workflows (repository_dispatch). Minimum scopes: repo (for release- # creation + dispatch triggers), workflow (for triggering workflows).+ # PAT_TOKEN is required here because GITHUB_TOKEN cannot trigger+ # workflows that listen on `release:` (backwards-compatibility.yml,+ # sbom.yml) — that's a documented GITHUB_TOKEN limitation.+ # Minimum scope: `repo` (full scope; `public_repo` is NOT sufficient+ # for the release-creation API on private repos and `repo` works+ # uniformly). The `workflow` scope is NOT needed — it only governs+ # editing files under .github/workflows/ via the API, which this+ # step does not do.GITHUB_TOKEN: ${{ secrets.PAT_TOKEN }}RELEASE_TAG: ${{ needs.build.outputs.tag }}RELEASE_VERSION: ${{ needs.build.outputs.version }}@@ -979,47 +1131,23 @@ jobs:automationmaintenance- trigger-prerelease-docker:- needs: [build, provenance]- if: ${{ !cancelled() && needs.build.result == 'success' && needs.provenance.result == 'success' && needs.build.outputs.release_exists == 'false' }}- runs-on: ubuntu-latest- # Separate environment from `release` so the GitHub "Review deployments"- # modal shows two independent checkboxes — letting maintainers approve or- # reject the prerelease Docker test independently of the actual release- # publish (create-release + trigger-workflows still gate on `release`).- environment: prerelease- permissions:- contents: read-- steps:- - name: Harden the runner (Audit all outbound calls)- uses: step-security/harden-runner@ab7a9404c0f3da075243ca237b5fac12c98deaa5 # v2.19.3- with:- egress-policy: audit-- - name: Trigger prerelease Docker build- uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0- env:- RELEASE_VERSION: ${{ needs.build.outputs.version }}- SHORT_SHA: ${{ needs.build.outputs.short_sha }}- with:- # PAT_TOKEN: repository_dispatch requires a PAT (GITHUB_TOKEN cannot trigger workflows)- github-token: ${{ secrets.PAT_TOKEN }}- script: |- await github.rest.repos.createDispatchEvent({- owner: context.repo.owner,- repo: context.repo.repo,- event_type: 'publish-prerelease-docker',- client_payload: {- version: process.env.RELEASE_VERSION,- short_sha: process.env.SHORT_SHA- }- });- console.log('Triggered prerelease Docker build');-- trigger-workflows:- needs: [build, create-release]- if: ${{ !cancelled() && needs.build.result == 'success' && needs.create-release.result == 'success' && needs.build.outputs.release_exists == 'false' }}+ # NOTE: trigger-prerelease-docker and the old combined trigger-workflows+ # (which used to dispatch BOTH publish-docker AND publish-pypi) were+ # removed in the build-once-promote + atomicity refactors.+ # prerelease-docker.yml and docker-publish.yml are now invoked as+ # reusable workflows earlier in this file. Only PyPI publishing remains+ # on repository_dispatch — PyPI Trusted Publishing requires the publish+ # step to run in a top-level (non-reusable) workflow.++ # Dispatch PyPI publish via repository_dispatch. Kept as a dispatch+ # (rather than a reusable workflow_call) because PyPI's Trusted+ # Publisher matches the OIDC `workflow_ref` claim, which points to the+ # CALLER when a workflow is invoked via workflow_call — so a reusable+ # publish.yml would fail with `invalid-publisher`. Tracked in+ # pypa/gh-action-pypi-publish#166 and pypi/warehouse#11096.+ trigger-pypi:+ needs: [build, prerelease-docker, publish-docker]+ if: ${{ !cancelled() && needs.prerelease-docker.result == 'success' && needs.publish-docker.result == 'success' && needs.build.outputs.release_exists == 'false' }}runs-on: ubuntu-latestenvironment: releasepermissions:@@ -1033,63 +1161,50 @@ jobs:with:egress-policy: audit- # Recorded BEFORE the dispatches so monitor-publish can filter on createdAt- # without missing runs created during this job.+ # Recorded BEFORE the dispatch so monitor-pypi can filter on createdAt+ # without missing the run created during this job.- name: Record dispatch timeid: dispatch-timerun: echo "timestamp=$(date -u +%Y-%m-%dT%H:%M:%SZ)" >> "$GITHUB_OUTPUT"- - name: Trigger Docker publish workflow- uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0- env:- RELEASE_TAG: ${{ needs.build.outputs.tag }}- with:- # PAT_TOKEN: repository_dispatch requires a PAT (GITHUB_TOKEN cannot trigger workflows)- github-token: ${{ secrets.PAT_TOKEN }}- script: |- await github.rest.repos.createDispatchEvent({- owner: context.repo.owner,- repo: context.repo.repo,- event_type: 'publish-docker',- client_payload: {- tag: process.env.RELEASE_TAG- }- });- console.log('Triggered Docker publish workflow');-- name: Trigger PyPI publish workflowuses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0env:RELEASE_TAG: ${{ needs.build.outputs.tag }}with:- # PAT_TOKEN: repository_dispatch requires a PAT (GITHUB_TOKEN cannot trigger workflows)+ # PAT_TOKEN: repository_dispatch from a workflow run cannot be+ # fired with GITHUB_TOKEN — the API rejects it to prevent+ # workflow-trigger loops. Minimum scope: `repo` (full scope —+ # `public_repo` is rejected by createDispatchEvent regardless+ # of repo visibility).github-token: ${{ secrets.PAT_TOKEN }}script: |+ // Forward `prerelease: false` explicitly. publish.yml gates+ // Test PyPI vs prod PyPI on `client_payload.prerelease == true`;+ // if absent, the expression evaluates to '' and falls through to+ // prod PyPI. Setting false here makes the choice explicit. The+ // current pipeline only releases stable versions through this+ // dispatch — true prereleases (if added later) would need a+ // separate trigger path that flips this flag.await github.rest.repos.createDispatchEvent({owner: context.repo.owner,repo: context.repo.repo,event_type: 'publish-pypi',client_payload: {- tag: process.env.RELEASE_TAG+ tag: process.env.RELEASE_TAG,+ prerelease: false}});console.log('Triggered PyPI publish workflow');- - name: Summary- env:- RELEASE_VERSION: ${{ needs.build.outputs.version }}- run: |- echo "Release $RELEASE_VERSION created successfully!"- echo "Triggered PyPI and Docker publishing workflows via repository_dispatch"- echo "SBOM files (SPDX, CycloneDX) attached to release"- echo "Sigstore bundles (.bundle) attached for verification (contain signature + certificate)"- echo "SLSA provenance (provenance.intoto.jsonl) attached for supply chain security"- echo "Check the releases page: https://github.com/LearningCircuit/local-deep-research/releases"-- # Monitor publish workflows and create issue on partial failure- monitor-publish:- needs: [build, trigger-workflows]- if: ${{ !cancelled() && needs.trigger-workflows.result == 'success' }}+ # Block on the dispatched publish.yml run so create-release downstream+ # only fires once PyPI has actually shipped. If PyPI fails, this job+ # fails and create-release is skipped — preventing the GH Release from+ # publishing with a missing PyPI artifact. (Docker promote already+ # blocked synchronously above via publish-docker reusable call.)+ monitor-pypi:+ needs: [build, trigger-pypi]+ if: ${{ !cancelled() && needs.trigger-pypi.result == 'success' }}runs-on: ubuntu-latesttimeout-minutes: 90permissions:@@ -1103,120 +1218,106 @@ jobs:with:egress-policy: audit- - name: Wait for publish workflows to complete+ - name: Wait for PyPI publish workflow to complete+ id: waitenv:GH_TOKEN: ${{ github.token }}# gh CLI cannot infer the repo here (this job has no checkout step),# and it does NOT fall back to GITHUB_REPOSITORY. Without GH_REPO,# `gh run list` fails with "failed to determine base repo", the- # error is swallowed, and every poll sees an empty result — which- # made monitor-publish always time out and falsely open a- # "Partial publish failure" issue even when both publishes succeeded.+ # error is swallowed, and every poll sees an empty result.GH_REPO: ${{ github.repository }}RELEASE_TAG: ${{ needs.build.outputs.tag }}- DISPATCH_TIME: ${{ needs.trigger-workflows.outputs.dispatch_time }}+ DISPATCH_TIME: ${{ needs.trigger-pypi.outputs.dispatch_time }}run: |- echo "Monitoring publish workflows for tag $RELEASE_TAG (dispatched at $DISPATCH_TIME)..."+ # pipefail is essential here: the poll loop pipes `gh run list`+ # into `jq`, and without pipefail a transient `gh` failure+ # (network, auth, rate limit) is swallowed by `jq` returning+ # empty input, causing the loop to spin silently for the full+ # 40-minute budget rather than surfacing the error immediately.+ set -euo pipefail+ echo "Monitoring publish.yml for tag $RELEASE_TAG (dispatched at $DISPATCH_TIME)..."- # Wait for workflows to start (repository_dispatch is async)+ # Wait for the dispatched run to start (repository_dispatch is async)sleep 30- check_workflow() {- local workflow_name="$1"- local max_wait=2400 # 40 minutes- local elapsed=0- local run status conclusion-- while [ "$elapsed" -lt "$max_wait" ]; do- # Compare timestamps numerically — lexicographic comparison breaks- # when GitHub returns sub-second precision (e.g. "...:56.500Z"- # sorts before "...:56Z"). fromdateiso8601 doesn't accept- # fractional seconds, so strip them first. The 60s buffer- # tolerates minor clock skew between the runner and GitHub's API.- # Stderr is intentionally NOT redirected so gh failures are- # visible in the runner log (silent failures previously caused- # the loop to time out without explanation).- run=$(gh run list --workflow="$workflow_name" --limit=20 --json status,conclusion,createdAt \- | jq -r --arg since "$DISPATCH_TIME" '- def to_epoch: sub("\\.[0-9]+Z$"; "Z") | fromdateiso8601;- (($since | to_epoch) - 60) as $s- | [.[] | select((.createdAt | to_epoch) >= $s)] | .[0]- ')-- if [ "$run" = "null" ] || [ -z "$run" ]; then- echo "$workflow_name: waiting for run to appear..." >&2- sleep 30- elapsed=$((elapsed + 30))- continue- fi-- status=$(echo "$run" | jq -r '.status')- conclusion=$(echo "$run" | jq -r '.conclusion')-- if [ "$status" = "completed" ]; then- echo "$workflow_name: $conclusion" >&2- echo "$conclusion"- return- fi-+ max_wait=2400 # 40 minutes+ elapsed=0+ conclusion=""++ while [ "$elapsed" -lt "$max_wait" ]; do+ # Compare timestamps numerically — lexicographic comparison+ # breaks when GitHub returns sub-second precision (e.g.+ # "...:56.500Z" sorts before "...:56Z"). fromdateiso8601+ # doesn't accept fractional seconds, so strip them first. The+ # 60s buffer tolerates minor clock skew between the runner and+ # GitHub's API. Stderr is intentionally NOT redirected so gh+ # failures stay visible in the runner log.+ run=$(gh run list --workflow=publish.yml --limit=20 --json status,conclusion,createdAt \+ | jq -r --arg since "$DISPATCH_TIME" '+ def to_epoch: sub("\\.[0-9]+Z$"; "Z") | fromdateiso8601;+ (($since | to_epoch) - 60) as $s+ | [.[] | select((.createdAt | to_epoch) >= $s)] | .[0]+ ')++ if [ "$run" = "null" ] || [ -z "$run" ]; then+ echo "publish.yml: waiting for run to appear..." >&2sleep 30elapsed=$((elapsed + 30))- done+ continue+ fi- echo "$workflow_name: timed out after ${max_wait}s" >&2- echo "timed_out"- }+ status=$(echo "$run" | jq -r '.status')+ conclusion=$(echo "$run" | jq -r '.conclusion')- DOCKER_RESULT=$(check_workflow "docker-publish.yml")- PYPI_RESULT=$(check_workflow "publish.yml")+ if [ "$status" = "completed" ]; then+ echo "publish.yml: $conclusion" >&2+ break+ fi- # Mark that monitoring completed (distinguishes from infra failure)- {- echo "monitor_completed=true"- echo "docker_result=$DOCKER_RESULT"- echo "pypi_result=$PYPI_RESULT"- } >> "$GITHUB_ENV"+ sleep 30+ elapsed=$((elapsed + 30))+ done- if [ "$DOCKER_RESULT" = "success" ] && [ "$PYPI_RESULT" = "success" ]; then- echo "Both publish workflows completed successfully!"+ if [ -z "$conclusion" ]; then+ echo "::error::publish.yml run did not start or complete within ${max_wait}s"+ echo "pypi_result=timed_out" >> "$GITHUB_OUTPUT"+ exit 1+ fi++ echo "pypi_result=$conclusion" >> "$GITHUB_OUTPUT"+ if [ "$conclusion" = "success" ]; then+ echo "PyPI publish completed successfully"+ exit 0else- echo "::warning::Partial publish failure detected"+ echo "::error::PyPI publish completed with conclusion: $conclusion"+ exit 1fi- - name: Create issue on partial failure- if: env.monitor_completed == 'true' && (env.docker_result != 'success' || env.pypi_result != 'success')+ - name: Create issue on PyPI publish failure+ # Always run on failure so maintainers see a clear, persistent+ # record that captures the workflow run ID. The Actions UI shows+ # the failure too but issues are easier to search and triage.+ if: failure()uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0env:RELEASE_TAG: ${{ needs.build.outputs.tag }}+ PYPI_RESULT: ${{ steps.wait.outputs.pypi_result }}with:script: |const { owner, repo } = context.repo;const tag = process.env.RELEASE_TAG;- const docker = process.env.docker_result || 'unknown';- const pypi = process.env.pypi_result || 'unknown';- const title = `Partial publish failure for ${tag}`;-- const failed = [- { name: 'Docker', result: docker },- { name: 'PyPI', result: pypi },- ].filter(t => t.result !== 'success');- const hasTimeout = failed.some(t => t.result === 'timed_out');- const hasFailure = failed.some(t => t.result === 'failure');-- let action;- if (hasTimeout && !hasFailure) {- action = '**Suggested action:** One or more workflows did not complete within 40 minutes. Check the [Actions tab](https://github.com/' + owner + '/' + repo + '/actions) — if a run never appeared, this likely indicates an infrastructure or dispatch issue and the publish can be re-triggered. If still running, no action may be needed.';- } else if (hasFailure && !hasTimeout) {- action = '**Suggested action:** Inspect the failed workflow logs in the [Actions tab](https://github.com/' + owner + '/' + repo + '/actions), fix the underlying cause, then re-trigger the publish via `repository_dispatch`.';- } else {- action = '**Suggested action:** Mixed results — investigate each non-success target individually in the [Actions tab](https://github.com/' + owner + '/' + repo + '/actions).';- }+ const pypi = process.env.PYPI_RESULT || 'unknown';+ const title = `PyPI publish failure for ${tag}`;- const body = `## Publish Status for ${tag}\n\n| Target | Result |\n|--------|--------|\n| Docker | ${docker} |\n| PyPI | ${pypi} |\n\n${action}`;+ const action = pypi === 'timed_out'+ ? '**Suggested action:** publish.yml did not complete within 40 minutes. Check the [Actions tab](https://github.com/' + owner + '/' + repo + '/actions) — if the run never appeared, this likely indicates a dispatch issue; re-trigger via `repository_dispatch` event_type `publish-pypi`. If still running, no action may be needed.'+ : '**Suggested action:** Inspect publish.yml logs in the [Actions tab](https://github.com/' + owner + '/' + repo + '/actions), fix the underlying cause, then either (a) re-dispatch publish.yml with the same tag via `repository_dispatch` and re-run release.yml to complete create-release, or (b) manually publish the GitHub Release if Docker promote also succeeded.';- // De-dup: if an open issue with the same title already exists- // (e.g. from a re-run of this workflow on the same tag), comment- // on it instead of opening a duplicate.+ const body = `## PyPI publish failed for ${tag}\n\nResult: \`${pypi}\`\n\nNote: At this point in the atomicity flow, Docker promote (publish-docker job) has already succeeded — the Docker release tags exist and are signed. Only PyPI and the GitHub Release are missing.\n\n${action}`;++ // De-dup: if an open issue with the same title already exists,+ // comment on it instead of opening a duplicate.const existing = await github.paginate(github.rest.issues.listForRepo, {owner, repo, state: 'open', labels: 'ci-cd', per_page: 100,});@@ -1232,3 +1333,218 @@ jobs:owner, repo, title, body, labels: ['ci-cd'],});}++ # ============================================================================+ # CLEANUP ON REJECTION+ # ============================================================================+ # In the build-once-promote model, prerelease-docker signs the manifest+ # BEFORE the publish step runs. If publish-docker (the retag step) fails+ # or the maintainer rejects the `release` env approval, the prerelease+ # tag and its cosign artifacts (`sha256-<digest>.{sig,att}`) are left+ # orphaned on Docker Hub forever — the existing cleanup loop inside+ # docker-publish.yml only runs on its success path.+ #+ # SAFETY: cosign signature/attestation artifacts are stored at a tag named+ # `sha256-<manifest-digest>.{sig,att}` and discovered by manifest digest,+ # NOT by image tag. After publish-docker SUCCEEDS, the release tags+ # (`:VERSION`, `:MAJOR_MINOR`, `:latest`) share the prerelease manifest+ # digest (imagetools retag preserves the digest), so the cosign artifacts+ # anchor BOTH the deleted prerelease tag AND the live release tags.+ # Deleting them after a successful retag would invalidate release-tag+ # cosign verification.+ #+ # Therefore: this job ONLY fires when publish-docker did not succeed.+ # Beyond that point, docker-publish.yml's success-path cleanup handles+ # prerelease-tag deletion, and cosign artifacts must stay.+ #+ # Edge case: partial retag failure (e.g., `:1.6.9` lands but `:latest`+ # fails) — publish-docker exits failure with some release tags already+ # created. The cleanup script enumerates the three possible release tags+ # against Docker Hub and rolls back any that exist BEFORE deleting+ # cosign artifacts. This is the only case where we delete release tags+ # rather than leave them.+ # ============================================================================+ cleanup-on-rejection:+ name: Clean up orphan prerelease tags and signatures+ needs: [build, prerelease-docker, publish-docker]+ if: >-+ ${{ always()+ && needs.prerelease-docker.result != 'skipped'+ && needs.build.outputs.release_exists == 'false'+ && (needs.prerelease-docker.result == 'failure'+ || needs.prerelease-docker.result == 'cancelled'+ || needs.publish-docker.result == 'failure'+ || needs.publish-docker.result == 'cancelled') }}+ runs-on: ubuntu-latest+ # DOCKER_USERNAME / DOCKER_PASSWORD are env-scoped to `release`+ # (deliberately not repo-level — see comment on the build job above).+ # Without `environment: release` here, those secrets resolve to empty+ # strings and the Docker Hub login at the bottom of this job exits 1,+ # leaving the orphan tags + cosign artifacts the cleanup is meant to+ # remove. The `release` env approval was already granted upstream in+ # this run, so this does not add a new prompt.+ environment: release+ permissions:+ contents: read++ steps:+ - name: Harden the runner (Audit all outbound calls)+ uses: step-security/harden-runner@ab7a9404c0f3da075243ca237b5fac12c98deaa5 # v2.19.3+ with:+ egress-policy: audit++ - name: Roll back partial release tags then delete prerelease + cosign artifacts+ env:+ DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}+ DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }}+ VERSION: ${{ needs.build.outputs.version }}+ SHORT_SHA: ${{ needs.build.outputs.short_sha }}+ DIGEST: ${{ needs.prerelease-docker.outputs.manifest_digest }}+ run: |+ set -euo pipefail+ # Authenticate with Docker Hub API (same JWT flow as docker-publish.yml cleanup)+ TOKEN=$(curl -sS -X POST -H "Content-Type: application/json" \+ -d "{\"username\":\"${DOCKER_USERNAME}\",\"password\":\"${DOCKER_PASSWORD}\"}" \+ https://hub.docker.com/v2/users/login/ | jq -r .token)++ if [ -z "$TOKEN" ] || [ "$TOKEN" = "null" ]; then+ echo "::error::Failed to authenticate with Docker Hub API — cannot clean up orphans"+ exit 1+ fi++ REPO="${DOCKER_USERNAME}/local-deep-research"+ MANIFEST_TAG="prerelease-v${VERSION}-${SHORT_SHA}"+ MAJOR_MINOR=$(echo "$VERSION" | cut -d. -f1,2)++ delete_tag() {+ local TAG="$1"+ local STATUS+ STATUS=$(curl -sS -o /dev/null -w "%{http_code}" -X DELETE \+ -H "Authorization: JWT ${TOKEN}" \+ "https://hub.docker.com/v2/repositories/${REPO}/tags/${TAG}/" || echo "ERR")+ case "$STATUS" in+ 200|204) echo "Deleted ${TAG}";;+ 404) echo "Skip ${TAG} (already absent — expected)";;+ 401|403)+ echo "::error::Auth failure deleting ${TAG} — DOCKER_PASSWORD may be missing Delete scope on the Docker Hub PAT"+ exit 1+ ;;+ *) echo "::warning::Unexpected HTTP ${STATUS} for ${TAG}";;+ esac+ }++ # =====================================================================+ # STEP 1 — Partial retag rollback.+ # =====================================================================+ # If publish-docker failed mid-way through `docker buildx imagetools+ # create -t :VERSION -t :MAJOR_MINOR -t :latest`, one or two of the+ # three release tags may have landed. Those tags would point at the+ # prerelease manifest digest, which we're about to delete cosign+ # artifacts for. We MUST roll back any landed release tags BEFORE+ # touching cosign — otherwise the rollback leaves them broken.+ #+ # If publish-docker was skipped (prerelease failed before reaching+ # retag), the release tags can't exist, so this loop is a cheap no-op+ # (three 404 responses).+ # =====================================================================+ echo "STEP 1 — Roll back any release tags that landed during a partial retag..."+ for RELEASE_TAG in "${VERSION}" "${MAJOR_MINOR}" "latest"; do+ # HEAD-check before DELETE so the "expected absent" case is silent.+ # Docker Hub's tag endpoint returns 200 on HEAD if tag exists, 404 if not.+ CHECK=$(curl -sS -o /dev/null -w "%{http_code}" \+ -H "Authorization: JWT ${TOKEN}" \+ "https://hub.docker.com/v2/repositories/${REPO}/tags/${RELEASE_TAG}/" || echo "ERR")+ if [ "$CHECK" = "200" ]; then+ echo "::warning::Release tag ${RELEASE_TAG} exists from a partial retag — rolling back"+ delete_tag "${RELEASE_TAG}"+ else+ echo "Release tag ${RELEASE_TAG}: not present (HTTP ${CHECK}) — nothing to roll back"+ fi+ done++ # =====================================================================+ # STEP 2 — Prerelease tag cleanup.+ # =====================================================================+ # Always-safe targets: the prerelease manifest list + its per-arch+ # children + the floating `:prerelease` tag. Cleanup is best-effort+ # here (404s are normal — e.g., build failed before pushing arm64).+ #+ # The floating `:prerelease` tag (re-pointed by prerelease-docker.yml+ # to the current run's manifest) is included so a rejected release+ # does not leave `:prerelease` pointing at a manifest whose cosign+ # artifacts step 4 below is about to delete — pulling `:prerelease`+ # in that window would yield an image whose signature is gone, and+ # the README cosign-verify recipe would fail. Returning 404 on+ # `:prerelease` is unambiguously safer than serving an unverifiable+ # image; the next successful prerelease-docker run re-creates it.+ # =====================================================================+ echo "STEP 2 — Delete prerelease tags..."+ for TAG in "${MANIFEST_TAG}" "${MANIFEST_TAG}-amd64" "${MANIFEST_TAG}-arm64" "prerelease"; do+ delete_tag "${TAG}"+ done++ # =====================================================================+ # STEP 3 — Discover per-arch digests from the manifest list BEFORE+ # we delete cosign artifacts.+ # =====================================================================+ # prerelease-docker.yml's per-arch SBOM step attests against each+ # per-arch digest, producing artifacts at+ # `sha256-<per-arch-digest>.{sig,att,sbom}` in addition to the+ # manifest-list-digest artifacts. Need to enumerate from the manifest+ # while it still exists (we just deleted the TAG, but the manifest+ # body persists until Docker Hub GC, and we can inspect by digest).+ # Use the captured DIGEST to inspect the manifest list directly.+ # =====================================================================+ PER_ARCH_TAGS=()+ if [[ -n "${DIGEST:-}" && "$DIGEST" == sha256:* ]]; then+ echo "STEP 3 — Discovering per-arch digests from manifest list ${DIGEST}..."+ if docker buildx imagetools inspect "${REPO}@${DIGEST}" --raw > /tmp/manifest.json 2>/dev/null; then+ while IFS= read -r PER_ARCH_DIGEST; do+ if [[ -n "${PER_ARCH_DIGEST}" && "${PER_ARCH_DIGEST}" == sha256:* ]]; then+ PER_ARCH_TAG_PREFIX="${PER_ARCH_DIGEST/:/-}"+ PER_ARCH_TAGS+=(+ "${PER_ARCH_TAG_PREFIX}.sig"+ "${PER_ARCH_TAG_PREFIX}.att"+ "${PER_ARCH_TAG_PREFIX}.sbom"+ )+ echo " queued cleanup for per-arch artifacts at ${PER_ARCH_DIGEST}"+ fi+ done < <(jq -r '.manifests[] | select(.platform.architecture != "unknown") | .digest' /tmp/manifest.json)+ else+ echo "::warning::Could not inspect manifest at ${DIGEST} — skipping per-arch attestation cleanup"+ fi+ else+ echo "STEP 3 — No valid manifest digest captured (got '${DIGEST:-<empty>}'); skipping per-arch discovery"+ fi++ # =====================================================================+ # STEP 4 — Cosign signature/attestation artifact cleanup.+ # =====================================================================+ # Safe to delete only because STEP 1 already rolled back any release+ # tags that might be sharing these digests. The .sbom entries are+ # legacy from `cosign attach sbom` (current code uses `cosign attest+ # --type spdxjson` which writes to .att) — kept as belt-and-suspenders+ # cleanup for any leftovers from older releases.+ # =====================================================================+ COSIGN_TAGS=()+ if [[ -n "${DIGEST:-}" && "$DIGEST" == sha256:* ]]; then+ DIGEST_TAG_PREFIX="${DIGEST/:/-}"+ COSIGN_TAGS+=(+ "${DIGEST_TAG_PREFIX}.sig"+ "${DIGEST_TAG_PREFIX}.att"+ "${DIGEST_TAG_PREFIX}.sbom"+ )+ fi+ COSIGN_TAGS+=("${PER_ARCH_TAGS[@]}")++ if [ "${#COSIGN_TAGS[@]}" -gt 0 ]; then+ echo "STEP 4 — Delete cosign signature/attestation artifacts..."+ for TAG in "${COSIGN_TAGS[@]}"; do+ delete_tag "${TAG}"+ done+ else+ echo "STEP 4 — No cosign artifact tags to clean up (no captured digest)"+ fi+ # NOTE: NO `continue-on-error: true` at job level — auth failures+ # (401/403) are loud so missing token scopes can't silently let+ # orphans accumulate forever.diff --git a/Dockerfile b/Dockerfileindex fc255c773d..1ffab37f93 100644--- a/Dockerfile+++ b/Dockerfile@@ -8,6 +8,14 @@ SHELL ["/bin/bash", "-o", "pipefail", "-c"]ARG DEBIAN_FRONTEND=noninteractive+# `apt-get upgrade -y` is INTENTIONAL — we want every build to pull the+# latest patched Debian packages so security fixes flow into the image.+# This trades bit-for-bit reproducibility (two rebuilds of the same source+# can produce different layer digests across a Debian patch window) for+# always-fresh-on-CVE behavior. The build-once-promote pipeline mitigates+# the reproducibility loss: prerelease-docker.yml builds once per release+# and the resulting digest is what gets retagged to :1.6.9 / :1.6 / :latest,+# so the released image is bit-identical to the one tested.# Install system dependencies for SQLCipher and Node.js for frontend build# Using Acquire::Retries to handle transient Debian mirror errors during CIRUN apt-get update -o Acquire::Retries=3 && apt-get upgrade -y -o Acquire::Retries=3 \@@ -63,9 +71,11 @@ ENV PDM_CHECK_UPDATE=false# This helps prevent httpcore.ReadTimeout errors during CI network congestionENV PDM_REQUEST_TIMEOUT=120-# Build argument to invalidate cache when dependencies change-ARG DEPS_HASH-+# NOTE: `DEPS_HASH` was previously declared as a cache-invalidation arg but+# never referenced in a RUN/COPY, so it had no effect — Docker only honors+# ARG values for cache when they're actually used downstream. Cache+# invalidation on dependency changes happens naturally via `COPY pdm.lock`+# below, since the file's content hash changes when deps change.WORKDIR /install# Copy dependency files first (changes rarely)@@ -120,6 +130,8 @@ ARG DEBIAN_FRONTEND=noninteractive# Install additional runtime dependencies for testing tools# Note: Node.js is already installed from builder-base# Using Acquire::Retries to handle transient Debian mirror errors during CI+# `apt-get upgrade -y` is INTENTIONAL — see the rationale comment on the+# corresponding upgrade in the builder-base stage (top of file).RUN apt-get update -o Acquire::Retries=3 && apt-get upgrade -y -o Acquire::Retries=3 \&& apt-get install -y --no-install-recommends -o Acquire::Retries=3 \xauth \@@ -238,7 +250,9 @@ ARG DEBIAN_FRONTEND=noninteractive# — Scorecard alert #7742 dismissed as won't-fix on the same basis.RUN pip3 install --no-cache-dir pip==26.1-# Install runtime dependencies for SQLCipher and WeasyPrint+# Install runtime dependencies for SQLCipher and WeasyPrint.+# `apt-get upgrade -y` is INTENTIONAL — see rationale on the builder-base+# upgrade (top of file). Trade reproducibility for always-fresh CVE patches.RUN apt-get update && apt-get upgrade -y \&& apt-get install -y --no-install-recommends \sqlcipher \@@ -294,13 +308,30 @@ RUN HOME=/home/ldruser setpriv --reuid=ldruser --regid=ldruser --init-groups --sqlcipher = get_sqlcipher_module(); \print(f'✓ SQLCipher module loaded successfully: {sqlcipher}')"-# Create volume for persistent configuration-# Use /app for configuration to support non-root user+# Persistent state. Without VOLUME directives the user loses all research+# data + DBs on `docker rm`. Recommend bind-mounting these in production.+# - /app/.config/local_deep_research: legacy config path (kept for backcompat)+# - /data: where the entrypoint creates logs/, cache/, encrypted_databases/ —+# the actual user state, see scripts/ldr_entrypoint.sh.+#+# LDR_DATA_DIR pins the application to /data. Without this, the Python+# code falls back to platformdirs.user_data_dir() which resolves to+# /home/ldruser/.local/share/local-deep-research — NOT under any+# declared VOLUME, so a `docker run -v vol:/data ...` user (without+# also setting -e LDR_DATA_DIR=/data) would silently lose all data on+# `docker rm`. Documented run paths (docker-compose.yml, README docker+# run examples) already pass this env var explicitly; setting it here+# makes the VOLUME actually load-bearing for bare `docker run -v ...`+# invocations too.+ENV LDR_DATA_DIR=/dataVOLUME /app/.config/local_deep_research+VOLUME /data-# Create volume for Ollama start script-VOLUME /scripts/-# Copy the Ollama entrypoint script+# NOTE: /scripts/ is image content (ollama entrypoint baked in below), NOT+# user state. Previously declared as VOLUME, but a VOLUME on a directory+# that the image populates causes anonymous-volume creation on every+# `docker run` and silently shadows the script if a user bind-mounts it.+# Removed for correctness.COPY --chown=ldruser:ldruser scripts/ollama_entrypoint.sh /scripts/ollama_entrypoint.sh# Copy LDR entrypoint script to handle volume permissionsdiff --git a/README.md b/README.mdindex d8780efec2..7bea484d5f 100644--- a/README.md+++ b/README.md@@ -158,11 +158,38 @@ Your data stays yours. Each user gets their own isolated SQLCipher database encr**In-memory credentials**: Like all applications that use secrets at runtime — including [password managers](https://www.ise.io/casestudies/password-manager-hacking/), browsers, and API clients — credentials are held in plain text in process memory during active sessions. This is an [industry-wide accepted reality](https://cheatsheetseries.owasp.org/cheatsheets/Secrets_Management_Cheat_Sheet.html), not specific to LDR: if an attacker can read process memory, they can also read any in-process decryption key. We mitigate this with session-scoped credential lifetimes and core dump exclusion. Ideas for further improvements are always welcome via [GitHub Issues](https://github.com/LearningCircuit/local-deep-research/issues). See our [Security Policy](SECURITY.md) for details.-**Supply Chain Security**: Docker images are signed with [Cosign](https://github.com/sigstore/cosign), include SLSA provenance attestations, and attach SBOMs. Verify with:+**Supply Chain Security**: Docker images are signed with [Cosign](https://github.com/sigstore/cosign) using GitHub's keyless OIDC flow, include SLSA provenance attestations, and ship with attested SPDX SBOMs. Verify the image and its SBOM before running:+```bash-cosign verify localdeepresearch/local-deep-research:latest+# 1. Verify image signature+cosign verify \+ --certificate-identity-regexp "^https://github\.com/LearningCircuit/local-deep-research/\.github/workflows/prerelease-docker\.yml@.*$" \+ --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \+ --certificate-github-workflow-repository "LearningCircuit/local-deep-research" \+ localdeepresearch/local-deep-research:latest++# 2. Verify SBOM attestation (SPDX JSON) for YOUR platform+# SBOM attestations are stored per-architecture (amd64, arm64) on the+# per-arch image digest, not on the multi-arch manifest list. Resolve to+# your platform's digest first.+ARCH=$(uname -m | sed -e 's/^x86_64$/amd64/' -e 's/^aarch64$/arm64/')+PLATFORM_DIGEST=$(docker buildx imagetools inspect localdeepresearch/local-deep-research:latest --raw \+ | jq -r --arg arch "$ARCH" '.manifests[] | select(.platform.architecture==$arch) | .digest')+if [ -z "$PLATFORM_DIGEST" ]; then+ echo "No per-arch digest found for $ARCH — image may be single-arch or" \+ "from a pre-build-once-promote release. Skip step 2 in that case."+ exit 1+fi+cosign verify-attestation \+ --type spdxjson \+ --certificate-identity-regexp "^https://github\.com/LearningCircuit/local-deep-research/\.github/workflows/prerelease-docker\.yml@.*$" \+ --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \+ --certificate-github-workflow-repository "LearningCircuit/local-deep-research" \+ "localdeepresearch/local-deep-research@${PLATFORM_DIGEST}"```+The image-signature check confirms the image was built by the official `prerelease-docker.yml` workflow in `LearningCircuit/local-deep-research` — not by a forked repo or a leaked credential. The per-platform SBOM verification ensures you're inspecting the actual package set you're going to run, not the SBOM of a different architecture. Requires [cosign v2.0+](https://docs.sigstore.dev/cosign/installation/), [`jq`](https://jqlang.github.io/jq/), and `docker buildx` (bundled with Docker Desktop and Docker Engine ≥ 23.0; install the standalone plugin on older installs). Releases before the build-once-promote refactor were signed by `docker-publish.yml` and carried a single manifest-level SBOM rather than per-arch ones; for those, substitute `docker-publish.yml` for `prerelease-docker.yml` in the regex on both steps and skip the per-platform digest lookup (use the manifest list tag directly).+**Security Transparency**: Scanner suppressions are documented with justifications in [Security Alerts Assessment](.github/SECURITY_ALERTS.md), [Scorecard Compliance](.github/SECURITY_SCORECARD.md), [Container CVE Suppressions](.trivyignore), and [SAST Rule Rationale](bearer.yml). Some alerts (Dependabot, code scanning) can only be dismissed or are very difficult to suppress outside the [GitHub Security tab](https://docs.github.com/en/code-security/dependabot/dependabot-alerts/viewing-and-updating-dependabot-alerts), so the files above do not cover every dismissed finding.[Detailed Architecture →](docs/architecture.md) | [Security Policy →](SECURITY.md) | [Security Review Process →](docs/processes/security-review-process/)diff --git a/docs/CI_CD_INFRASTRUCTURE.md b/docs/CI_CD_INFRASTRUCTURE.mdindex 6decc7e2dc..ff0a4227f2 100644--- a/docs/CI_CD_INFRASTRUCTURE.md+++ b/docs/CI_CD_INFRASTRUCTURE.md@@ -129,10 +129,11 @@ pre-commit install-hooks| Workflow | Trigger | Purpose ||----------|---------|---------|-| `docker-publish.yml` | Release, push | Build and publish Docker images |+| `prerelease-docker.yml` | `workflow_call` from release.yml | Canonical multi-arch Docker build, cosign sign, SBOM/SLSA attestations. Jobs declare `environment: release` so the first `release` env approval gates the build (env-scoped Docker Hub secrets). |+| `docker-publish.yml` | `workflow_call` from release.yml | Retag prerelease manifest as `:1.6.9` / `:1.6` / `:latest` (gated by `release` env). No rebuild — registry-side metadata only. Inlined as a reusable workflow so its result is visible to downstream jobs in release.yml (lets create-release block on Docker success, lets cleanup-on-rejection safely scope cosign artifact deletion). || `docker-multiarch-test.yml` | PR, push | Multi-architecture build test |-| `publish.yml` | Release | Publish to PyPI |-| `release.yml` | Manual | Create releases |+| `publish.yml` | `repository_dispatch` from release.yml | Publish to PyPI. Stays on `repository_dispatch` (not `workflow_call`) because PyPI Trusted Publishing rejects OIDC claims from reusable workflows — `pypa/gh-action-pypi-publish#166`, `pypi/warehouse#11096`. |+| `release.yml` | Push to `main`, tag `v*.*.*`, manual | Orchestrate release: gates → build → provenance → prerelease-docker → publish-docker → trigger-pypi → monitor-pypi → create-release (last) |### Code Qualitydiff --git a/docs/RELEASE_GUIDE.md b/docs/RELEASE_GUIDE.mdindex 5a7b1ec79f..7cfd0a2087 100644--- a/docs/RELEASE_GUIDE.md+++ b/docs/RELEASE_GUIDE.md@@ -43,10 +43,82 @@ and short-circuits everything downstream).`version-check` job sets `should_release=false` and every downstreamjob (security gate, CI gate, build, publish) is skipped.-### 2. **Automatic Publishing** (with approval)-- **GitHub Release** → triggers:- - **PyPI publishing** (requires `release` environment approval)- - **Docker publishing** (requires `release` environment approval)+### 2. **Approval and Publishing**++The release pipeline uses the `release` GitHub environment to gate the+publish steps. `DOCKER_USERNAME` / `DOCKER_PASSWORD` are scoped to that+environment, so any job that pushes to Docker Hub must declare+`environment: release` and therefore goes through the approval gate.++When you merge to `main` (or push a tag), the pipeline runs in this+order:++1. Security gates + CI gates run automatically.+2. `build` job runs (version pin, SBOM, Sigstore bundles), then+ `provenance` job generates SLSA provenance for those artifacts.+3. **One `release` env approval prompt** in `release.yml`. Approving+ unlocks all release-env jobs in the same run, which then execute+ sequentially:+ 1. `prerelease-docker` — canonical multi-arch Docker build, cosign+ sign, SBOM/SLSA attestations, push as `prerelease-v<ver>-<sha>`+ and re-point the floating `:prerelease` tag.+ 2. `publish-docker` — retags the prerelease manifest as `:1.6.9`,+ `:1.6`, `:latest` (no rebuild, digest-preserving), then re-verifies+ digest + cosign + Trivy on the promoted tag.+ 3. `trigger-pypi` — dispatches `publish.yml` via `repository_dispatch`+ (PyPI Trusted Publishing requires the publish step to run in a+ top-level workflow, so this can't be a reusable workflow_call).+ 4. `monitor-pypi` — polls `publish.yml` for completion. The inner+ polling loop times out at 40 minutes (after which the job fails);+ the surrounding GH Actions `timeout-minutes` is 90 to leave a+ safety margin around the poll budget.+ 5. `create-release` — publishes the GitHub Release with+ SBOM/sig/provenance assets. Runs **last**, gated on all of the+ above succeeding, so the public Release never points at missing+ Docker tags or a missing PyPI version.++If any of `prerelease-docker`, `publish-docker`, or `monitor-pypi` fails,+`create-release` is skipped and no public GitHub Release is created. The+`cleanup-on-rejection` job then handles failure-mode cleanup:++- If `publish-docker` failed mid-retag (e.g., `:1.6.9` landed but+ `:latest` failed), it rolls back any landed release tags BEFORE+ deleting prerelease tags and cosign artifacts (deleting cosign+ artifacts while release tags share the manifest digest would invalidate+ release-tag signatures).+- If `publish-docker` succeeded but a later step (PyPI or+ create-release) failed, `cleanup-on-rejection` does NOT fire — Docker+ release tags exist and their cosign artifacts must stay. See+ "Recovery from PyPI failure" below.++### Recovery from PyPI failure (atomicity hole)++The one orphan state the pipeline cannot fully clean up: `publish-docker`+succeeded, PyPI failed. At this point Docker `:1.6.9` / `:1.6` /+`:latest` exist and are signed; PyPI has nothing; no GitHub Release.+`monitor-pypi` opens a tracking issue labeled `ci-cd`. To recover:++1. Inspect the `publish.yml` workflow run, fix the underlying cause.+2. Manually re-dispatch PyPI publish:+ ```bash+ gh api repos/LearningCircuit/local-deep-research/dispatches \+ -f event_type=publish-pypi \+ -F 'client_payload[tag]=v<X.Y.Z>'+ ```+3. Once PyPI publishes successfully, manually create the GitHub Release+ from the existing tag (the SBOM/sig/provenance artifacts are still+ uploaded as workflow artifacts on the failed `release.yml` run; you+ can download them and attach manually, or re-run `create-release`+ manually if the run is still re-runnable in the Actions UI).++> Earlier iterations of this refactor described a single approval gate+> with a pre-approval testing window. That design required+> `DOCKER_USERNAME` / `DOCKER_PASSWORD` to be repo-level secrets so the+> canonical build could run without env approval. They are env-scoped to+> `release` instead, so the gate sits in front of the build. The+> atomicity refactor preserves this single-approval model — one click+> unlocks the whole chain, and create-release runs last so the+> "published Release with broken artifacts" failure mode is closed.## 👥 Who Can Release@@ -95,11 +167,14 @@ with auto-notes plus AI summary only).### Option A: Manual Trigger- Go to Actions → "Create Release" → "Run workflow"-- Specify version and prerelease flag+- No inputs are required: the workflow reads the version from+ `src/local_deep_research/__version__.py` at HEAD. To release an+ older or different version, use Option B (push a version tag).### Option B: Version Tags- `git tag v0.4.3 && git push origin v0.4.3`-- Automatically creates release+- Automatically creates release; the workflow uses the tag's commit+ SHA (not `main` HEAD), so this is the correct path for backporting.## 🛡️ Branch Protection@@ -117,10 +192,27 @@ Follow [Semantic Versioning](https://semver.org/):## 🚨 Emergency Procedures-If automation fails:-1. **Manual GitHub release** still triggers PyPI/Docker-2. **Contact code owners** for assistance-3. **Check workflow logs** in GitHub Actions+If automation fails, do NOT create a GitHub release through the UI as+the first recovery step — under the atomicity refactor, a manually+created GitHub release does NOT trigger `publish.yml` (it listens only+on `repository_dispatch`) and does NOT trigger `docker-publish.yml`+(workflow_call only). The downstream `release:` listeners that DO fire+(`backwards-compatibility.yml`, `sbom.yml`) are observability-only.++Recovery, in order of preference:+1. **Check workflow logs** in GitHub Actions to identify which job+ failed, and use the targeted recovery for that failure mode:+ - PyPI failure with Docker already promoted: see+ [Recovery from PyPI failure](#recovery-from-pypi-failure-atomicity-hole) above.+ - Any other failure: re-run the failed job via the Actions UI if+ it's still re-runnable (typically within 30 days).+2. **Re-trigger the full pipeline** via `workflow_dispatch` if+ re-running individual jobs isn't possible. Safe for digest-keyed+ cosign verification — old digests remain valid because their cosign+ artifacts persist; the new run produces a new digest with its own+ signatures.+3. **Contact code owners** if recovery requires manual Docker Hub or+ PyPI intervention.## 📝 Release-notes flow (towncrier news fragments)