Skip to content

armgit / LEANN

Public Imported from GitHub
0

[MLsys2026 Best Paper]: https://arxiv.org/abs/2506.08276. RAG on Everything with LEANN. Enjoy 97% storage savings while running a fast, accurate, and 100% private RAG application on your personal device.

feat: implement comprehensive CI/CD pipeline with two-stage release

Andy Lee0a17d2c9d891
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
new file mode 100644
index 0000000..7426ca7
--- /dev/null
+++ b/.github/workflows/ci.yml
@@ -0,0 +1,105 @@
+name: CI - Build and Test
+
+on:
+ push:
+ branches: [ main ]
+ pull_request:
+ branches: [ main ]
+
+jobs:
+ build-test:
+ strategy:
+ matrix:
+ os: [ubuntu-latest, macos-latest]
+ python-version: ['3.9', '3.10', '3.11']
+
+ runs-on: ${{ matrix.os }}
+
+ steps:
+ - uses: actions/checkout@v4
+ with:
+ submodules: recursive
+
+ - name: Set up Python ${{ matrix.python-version }}
+ uses: actions/setup-python@v4
+ with:
+ python-version: ${{ matrix.python-version }}
+
+ - name: Install uv
+ run: |
+ curl -LsSf https://astral.sh/uv/install.sh | sh
+ echo "$HOME/.cargo/bin" >> $GITHUB_PATH
+
+ - name: Install system dependencies (Ubuntu)
+ if: runner.os == 'Linux'
+ run: |
+ sudo apt-get update
+ sudo apt-get install -y libomp-dev libboost-all-dev libzmq3-dev \
+ pkg-config libopenblas-dev patchelf \
+ libaio-dev protobuf-compiler libprotobuf-dev libabsl-dev
+
+ # Install Intel MKL for DiskANN
+ wget https://registrationcenter-download.intel.com/akdlm/IRC_NAS/79153e0f-74d7-45af-b8c2-258941adf58a/intel-onemkl-2025.0.0.940.sh
+ sudo sh intel-onemkl-2025.0.0.940.sh -a --components intel.oneapi.lin.mkl.devel --action install --eula accept -s
+ source /opt/intel/oneapi/setvars.sh
+ echo "MKLROOT=/opt/intel/oneapi/mkl/latest" >> $GITHUB_ENV
+ echo "LD_LIBRARY_PATH=/opt/intel/oneapi/mkl/latest/lib/intel64:$LD_LIBRARY_PATH" >> $GITHUB_ENV
+
+ - name: Install system dependencies (macOS)
+ if: runner.os == 'macOS'
+ run: |
+ brew install libomp boost zeromq protobuf
+
+ - name: Build all packages
+ run: |
+ echo "🔨 Building on ${{ matrix.os }} with Python ${{ matrix.python-version }}..."
+ export UV_SYSTEM_PYTHON=1
+
+ # Build each package
+ for pkg in leann-core leann-backend-hnsw leann-backend-diskann leann; do
+ echo "Building $pkg..."
+ cd packages/$pkg
+ rm -rf dist/ build/ _skbuild/
+ uv build --wheel
+ if [ ! -f dist/*.whl ]; then
+ echo "❌ Failed to build $pkg!"
+ exit 1
+ fi
+ echo "✅ $pkg built successfully"
+ cd ../..
+ done
+
+ - name: Install and test packages
+ run: |
+ # Create clean test environment
+ python -m venv test_env
+ if [[ "$OSTYPE" == "msys" || "$OSTYPE" == "win32" ]]; then
+ source test_env/Scripts/activate
+ else
+ source test_env/bin/activate
+ fi
+
+ # Install built packages
+ pip install packages/*/dist/*.whl
+
+ # Basic import test
+ python -c "import leann; print('✅ LEANN imported successfully')"
+ python -c "import leann_backend_hnsw; print('✅ HNSW backend imported')"
+ python -c "import leann_backend_diskann; print('✅ DiskANN backend imported')"
+
+ - name: Upload artifacts
+ uses: actions/upload-artifact@v3
+ with:
+ name: wheels-${{ matrix.os }}-py${{ matrix.python-version }}
+ path: packages/*/dist/*.whl
+ retention-days: 7
+
+ # Summary job to ensure all builds pass
+ ci-success:
+ needs: build-test
+ runs-on: ubuntu-latest
+ steps:
+ - name: CI Success
+ run: |
+ echo "✅ All CI builds passed!"
+ echo "Ready for manual release when needed."
\ No newline at end of file
diff --git a/.github/workflows/release-manual.yml b/.github/workflows/release-manual.yml
new file mode 100644
index 0000000..96139eb
--- /dev/null
+++ b/.github/workflows/release-manual.yml
@@ -0,0 +1,124 @@
+name: Manual Release
+
+on:
+ workflow_dispatch:
+ inputs:
+ version:
+ description: 'Version to release (e.g., 0.1.1)'
+ required: true
+ type: string
+ test_pypi:
+ description: 'Test on TestPyPI first'
+ required: false
+ type: boolean
+ default: true
+
+jobs:
+ validate-and-release:
+ runs-on: ubuntu-latest
+ permissions:
+ contents: write
+
+ steps:
+ - uses: actions/checkout@v4
+ with:
+ token: ${{ secrets.GITHUB_TOKEN }}
+
+ - name: Check CI status
+ run: |
+ echo "ℹ️ This workflow assumes CI has already passed on the current commit."
+ echo " If not, please wait for CI to complete before releasing."
+ echo ""
+
+ - name: Validate version format
+ run: |
+ if ! [[ "${{ inputs.version }}" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
+ echo "❌ Invalid version format. Use semantic versioning (e.g., 0.1.1)"
+ exit 1
+ fi
+ echo "✅ Version format valid: ${{ inputs.version }}"
+
+ - name: Check if version already exists
+ run: |
+ if git tag | grep -q "^v${{ inputs.version }}$"; then
+ echo "❌ Version v${{ inputs.version }} already exists!"
+ exit 1
+ fi
+ echo "✅ Version is new"
+
+ - name: Set up Python
+ uses: actions/setup-python@v4
+ with:
+ python-version: '3.11'
+
+ - name: Install uv
+ run: |
+ curl -LsSf https://astral.sh/uv/install.sh | sh
+ echo "$HOME/.cargo/bin" >> $GITHUB_PATH
+
+ - name: Update versions
+ run: |
+ ./scripts/bump_version.sh ${{ inputs.version }}
+ git config user.name "GitHub Actions"
+ git config user.email "[email protected]"
+ git add packages/*/pyproject.toml
+ git commit -m "chore: release v${{ inputs.version }}"
+
+ - name: Build packages for TestPyPI
+ run: |
+ echo "🔨 Building packages for testing..."
+ ./scripts/build_and_test.sh all
+ echo "✅ Build successful! (Already tested on multiple platforms)"
+
+ - name: Test on TestPyPI (optional)
+ if: inputs.test_pypi
+ env:
+ TWINE_USERNAME: __token__
+ TWINE_PASSWORD: ${{ secrets.TEST_PYPI_API_TOKEN }}
+ run: |
+ pip install twine
+ echo "📦 Uploading to TestPyPI..."
+ twine upload --repository testpypi packages/*/dist/* --verbose
+ echo "✅ Test upload successful!"
+ echo "📋 Check packages at: https://test.pypi.org/user/your-username/"
+ echo ""
+ echo "To test installation:"
+ echo "pip install -i https://test.pypi.org/simple/ leann"
+
+ - name: Create and push tag
+ run: |
+ git tag "v${{ inputs.version }}"
+ git push origin main
+ git push origin "v${{ inputs.version }}"
+ echo "✅ Tag v${{ inputs.version }} created and pushed"
+
+ - name: Create GitHub Release
+ uses: softprops/action-gh-release@v1
+ with:
+ tag_name: v${{ inputs.version }}
+ name: Release v${{ inputs.version }}
+ body: |
+ ## 🚀 Release v${{ inputs.version }}
+
+ ### What's Changed
+ See the [full changelog](https://github.com/${{ github.repository }}/compare/...v${{ inputs.version }})
+
+ ### Installation
+ ```bash
+ pip install leann==${{ inputs.version }}
+ ```
+
+ ### Test Installation (if using TestPyPI)
+ ```bash
+ pip install -i https://test.pypi.org/simple/ leann==${{ inputs.version }}
+ ```
+ draft: false
+ prerelease: false
+ env:
+ GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+
+ - name: Trigger PyPI publish
+ run: |
+ echo "🚀 Triggering PyPI publish workflow..."
+ # The existing build-and-publish.yml will be triggered by the tag push
+ echo "✅ Release process completed! The publish workflow will run automatically."
\ No newline at end of file
diff --git a/docs/RELEASE.md b/docs/RELEASE.md
index 72bb251..6ccf6b5 100644
--- a/docs/RELEASE.md
+++ b/docs/RELEASE.md
@@ -1,17 +1,52 @@
# Release Guide
-## One-line Release 🚀
+## 📋 Prerequisites
+
+Before releasing, ensure:
+1. ✅ All code changes are committed and pushed
+2. ✅ CI has passed on the latest commit (check [Actions](https://github.com/yichuan-w/LEANN/actions/workflows/ci.yml))
+3. ✅ You have determined the new version number
+
+## 🚀 Recommended: Manual Release Workflow
+
+### Via GitHub UI (Most Reliable)
+
+1. **Verify CI Status**: Check that the latest commit has a green checkmark ✅
+2. Go to [Actions → Manual Release](https://github.com/yichuan-w/LEANN/actions/workflows/release-manual.yml)
+3. Click "Run workflow"
+4. Enter version (e.g., `0.1.1`)
+5. Toggle "Test on TestPyPI first" if desired
+6. Click "Run workflow"
+
+**What happens:**
+- ✅ Validates version format
+- ✅ Updates all package versions
+- ✅ Optionally tests on TestPyPI
+- ✅ Creates tag and GitHub release
+- ✅ Automatically triggers PyPI publish
+
+### Via Command Line
+
+```bash
+gh workflow run release-manual.yml -f version=0.1.1 -f test_pypi=true
+```
+
+## ⚡ Quick Release (One-Line)
+
+For experienced users who want the fastest path:
```bash
./scripts/release.sh 0.1.1
```
-That's it! This script will:
+This script will:
1. Update all package versions
2. Commit and push changes
3. Create GitHub release
4. CI automatically builds and publishes to PyPI
+⚠️ **Note**: If CI fails, you'll need to manually fix and re-tag
+
## Manual Testing Before Release
For testing specific packages locally (especially DiskANN on macOS):
@@ -32,13 +67,6 @@ pip install packages/*/dist/*.whl
./scripts/upload_to_pypi.sh prod
```
-### Why Manual Build for DiskANN?
-
-DiskANN's complex dependencies (protobuf, abseil, etc.) sometimes require local testing before release. The build script will:
-- Compile the C++ extension
-- Use `delocate` (macOS) or `auditwheel` (Linux) to bundle system libraries
-- Create a self-contained wheel with no external dependencies
-
## First-time setup
1. Install GitHub CLI: