[MLsys2026 Best Paper]: https://arxiv.org/abs/2506.08276. RAG on Everything with LEANN. Enjoy 97% storage savings while running a fast, accurate, and 100% private RAG application on your personal device.
Python aifaissgpt-osslangchainllama-indexllmlocalstorageoffline-firstollamaprivacypythonragretrieval-augmented-generationvector-databasevector-searchvectors
feat: implement comprehensive CI/CD pipeline with two-stage release
Andy Lee0a17d2c9d891
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlnew file mode 100644index 0000000..7426ca7--- /dev/null+++ b/.github/workflows/ci.yml@@ -0,0 +1,105 @@+name: CI - Build and Test++on:+ push:+ branches: [ main ]+ pull_request:+ branches: [ main ]++jobs:+ build-test:+ strategy:+ matrix:+ os: [ubuntu-latest, macos-latest]+ python-version: ['3.9', '3.10', '3.11']++ runs-on: ${{ matrix.os }}++ steps:+ - uses: actions/checkout@v4+ with:+ submodules: recursive++ - name: Set up Python ${{ matrix.python-version }}+ uses: actions/setup-python@v4+ with:+ python-version: ${{ matrix.python-version }}++ - name: Install uv+ run: |+ curl -LsSf https://astral.sh/uv/install.sh | sh+ echo "$HOME/.cargo/bin" >> $GITHUB_PATH++ - name: Install system dependencies (Ubuntu)+ if: runner.os == 'Linux'+ run: |+ sudo apt-get update+ sudo apt-get install -y libomp-dev libboost-all-dev libzmq3-dev \+ pkg-config libopenblas-dev patchelf \+ libaio-dev protobuf-compiler libprotobuf-dev libabsl-dev++ # Install Intel MKL for DiskANN+ wget https://registrationcenter-download.intel.com/akdlm/IRC_NAS/79153e0f-74d7-45af-b8c2-258941adf58a/intel-onemkl-2025.0.0.940.sh+ sudo sh intel-onemkl-2025.0.0.940.sh -a --components intel.oneapi.lin.mkl.devel --action install --eula accept -s+ source /opt/intel/oneapi/setvars.sh+ echo "MKLROOT=/opt/intel/oneapi/mkl/latest" >> $GITHUB_ENV+ echo "LD_LIBRARY_PATH=/opt/intel/oneapi/mkl/latest/lib/intel64:$LD_LIBRARY_PATH" >> $GITHUB_ENV++ - name: Install system dependencies (macOS)+ if: runner.os == 'macOS'+ run: |+ brew install libomp boost zeromq protobuf++ - name: Build all packages+ run: |+ echo "🔨 Building on ${{ matrix.os }} with Python ${{ matrix.python-version }}..."+ export UV_SYSTEM_PYTHON=1++ # Build each package+ for pkg in leann-core leann-backend-hnsw leann-backend-diskann leann; do+ echo "Building $pkg..."+ cd packages/$pkg+ rm -rf dist/ build/ _skbuild/+ uv build --wheel+ if [ ! -f dist/*.whl ]; then+ echo "❌ Failed to build $pkg!"+ exit 1+ fi+ echo "✅ $pkg built successfully"+ cd ../..+ done++ - name: Install and test packages+ run: |+ # Create clean test environment+ python -m venv test_env+ if [[ "$OSTYPE" == "msys" || "$OSTYPE" == "win32" ]]; then+ source test_env/Scripts/activate+ else+ source test_env/bin/activate+ fi++ # Install built packages+ pip install packages/*/dist/*.whl++ # Basic import test+ python -c "import leann; print('✅ LEANN imported successfully')"+ python -c "import leann_backend_hnsw; print('✅ HNSW backend imported')"+ python -c "import leann_backend_diskann; print('✅ DiskANN backend imported')"++ - name: Upload artifacts+ uses: actions/upload-artifact@v3+ with:+ name: wheels-${{ matrix.os }}-py${{ matrix.python-version }}+ path: packages/*/dist/*.whl+ retention-days: 7++ # Summary job to ensure all builds pass+ ci-success:+ needs: build-test+ runs-on: ubuntu-latest+ steps:+ - name: CI Success+ run: |+ echo "✅ All CI builds passed!"+ echo "Ready for manual release when needed."\ No newline at end of filediff --git a/.github/workflows/release-manual.yml b/.github/workflows/release-manual.ymlnew file mode 100644index 0000000..96139eb--- /dev/null+++ b/.github/workflows/release-manual.yml@@ -0,0 +1,124 @@+name: Manual Release++on:+ workflow_dispatch:+ inputs:+ version:+ description: 'Version to release (e.g., 0.1.1)'+ required: true+ type: string+ test_pypi:+ description: 'Test on TestPyPI first'+ required: false+ type: boolean+ default: true++jobs:+ validate-and-release:+ runs-on: ubuntu-latest+ permissions:+ contents: write++ steps:+ - uses: actions/checkout@v4+ with:+ token: ${{ secrets.GITHUB_TOKEN }}++ - name: Check CI status+ run: |+ echo "ℹ️ This workflow assumes CI has already passed on the current commit."+ echo " If not, please wait for CI to complete before releasing."+ echo ""++ - name: Validate version format+ run: |+ if ! [[ "${{ inputs.version }}" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]; then+ echo "❌ Invalid version format. Use semantic versioning (e.g., 0.1.1)"+ exit 1+ fi+ echo "✅ Version format valid: ${{ inputs.version }}"++ - name: Check if version already exists+ run: |+ if git tag | grep -q "^v${{ inputs.version }}$"; then+ echo "❌ Version v${{ inputs.version }} already exists!"+ exit 1+ fi+ echo "✅ Version is new"++ - name: Set up Python+ uses: actions/setup-python@v4+ with:+ python-version: '3.11'++ - name: Install uv+ run: |+ curl -LsSf https://astral.sh/uv/install.sh | sh+ echo "$HOME/.cargo/bin" >> $GITHUB_PATH++ - name: Update versions+ run: |+ ./scripts/bump_version.sh ${{ inputs.version }}+ git config user.name "GitHub Actions"+ git config user.email "[email protected]"+ git add packages/*/pyproject.toml+ git commit -m "chore: release v${{ inputs.version }}"++ - name: Build packages for TestPyPI+ run: |+ echo "🔨 Building packages for testing..."+ ./scripts/build_and_test.sh all+ echo "✅ Build successful! (Already tested on multiple platforms)"++ - name: Test on TestPyPI (optional)+ if: inputs.test_pypi+ env:+ TWINE_USERNAME: __token__+ TWINE_PASSWORD: ${{ secrets.TEST_PYPI_API_TOKEN }}+ run: |+ pip install twine+ echo "📦 Uploading to TestPyPI..."+ twine upload --repository testpypi packages/*/dist/* --verbose+ echo "✅ Test upload successful!"+ echo "📋 Check packages at: https://test.pypi.org/user/your-username/"+ echo ""+ echo "To test installation:"+ echo "pip install -i https://test.pypi.org/simple/ leann"++ - name: Create and push tag+ run: |+ git tag "v${{ inputs.version }}"+ git push origin main+ git push origin "v${{ inputs.version }}"+ echo "✅ Tag v${{ inputs.version }} created and pushed"++ - name: Create GitHub Release+ uses: softprops/action-gh-release@v1+ with:+ tag_name: v${{ inputs.version }}+ name: Release v${{ inputs.version }}+ body: |+ ## 🚀 Release v${{ inputs.version }}++ ### What's Changed+ See the [full changelog](https://github.com/${{ github.repository }}/compare/...v${{ inputs.version }})++ ### Installation+ ```bash+ pip install leann==${{ inputs.version }}+ ```++ ### Test Installation (if using TestPyPI)+ ```bash+ pip install -i https://test.pypi.org/simple/ leann==${{ inputs.version }}+ ```+ draft: false+ prerelease: false+ env:+ GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}++ - name: Trigger PyPI publish+ run: |+ echo "🚀 Triggering PyPI publish workflow..."+ # The existing build-and-publish.yml will be triggered by the tag push+ echo "✅ Release process completed! The publish workflow will run automatically."\ No newline at end of filediff --git a/docs/RELEASE.md b/docs/RELEASE.mdindex 72bb251..6ccf6b5 100644--- a/docs/RELEASE.md+++ b/docs/RELEASE.md@@ -1,17 +1,52 @@# Release Guide-## One-line Release 🚀+## 📋 Prerequisites++Before releasing, ensure:+1. ✅ All code changes are committed and pushed+2. ✅ CI has passed on the latest commit (check [Actions](https://github.com/yichuan-w/LEANN/actions/workflows/ci.yml))+3. ✅ You have determined the new version number++## 🚀 Recommended: Manual Release Workflow++### Via GitHub UI (Most Reliable)++1. **Verify CI Status**: Check that the latest commit has a green checkmark ✅+2. Go to [Actions → Manual Release](https://github.com/yichuan-w/LEANN/actions/workflows/release-manual.yml)+3. Click "Run workflow"+4. Enter version (e.g., `0.1.1`)+5. Toggle "Test on TestPyPI first" if desired+6. Click "Run workflow"++**What happens:**+- ✅ Validates version format+- ✅ Updates all package versions+- ✅ Optionally tests on TestPyPI+- ✅ Creates tag and GitHub release+- ✅ Automatically triggers PyPI publish++### Via Command Line++```bash+gh workflow run release-manual.yml -f version=0.1.1 -f test_pypi=true+```++## ⚡ Quick Release (One-Line)++For experienced users who want the fastest path:```bash./scripts/release.sh 0.1.1```-That's it! This script will:+This script will:1. Update all package versions2. Commit and push changes3. Create GitHub release4. CI automatically builds and publishes to PyPI+⚠️ **Note**: If CI fails, you'll need to manually fix and re-tag+## Manual Testing Before ReleaseFor testing specific packages locally (especially DiskANN on macOS):@@ -32,13 +67,6 @@ pip install packages/*/dist/*.whl./scripts/upload_to_pypi.sh prod```-### Why Manual Build for DiskANN?--DiskANN's complex dependencies (protobuf, abseil, etc.) sometimes require local testing before release. The build script will:-- Compile the C++ extension-- Use `delocate` (macOS) or `auditwheel` (Linux) to bundle system libraries-- Create a self-contained wheel with no external dependencies-## First-time setup1. Install GitHub CLI: