Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

9.8 KB
Raw Blame

AI Code Reviewer Setup Guide

This guide explains how to set up the automated AI PR review system using OpenRouter to analyze pull requests with your choice of AI model.

Overview

The AI Code Reviewer provides automated, comprehensive code reviews covering:

  • Security 🔒 - Hardcoded secrets, SQL injection, XSS, authentication issues, input validation
  • Performance ⚡ - Inefficient algorithms, N+1 queries, memory issues, blocking operations
  • Code Quality 🎨 - Readability, maintainability, error handling, naming conventions
  • Best Practices 📋 - Coding standards, proper patterns, type safety, dead code

The review is posted as a single comprehensive comment on your pull request.

Setup Instructions

1. Get OpenRouter API Key

  1. Go to OpenRouter.ai
  2. Sign up or log in
  3. Navigate to API Keys section
  4. Create a new API key
  5. Copy the key (it starts with sk-or-v1-...)

2. Add API Key to GitHub Secrets

  1. Go to your GitHub repository
  2. Navigate to Settings → Secrets and variables → Actions
  3. Click New repository secret
  4. Name it: OPENROUTER_API_KEY
  5. Paste your OpenRouter API key
  6. Click Add secret

3. Configure Workflow (Optional)

The workflow is pre-configured with sensible defaults. You can override them without editing the workflow by adding GitHub repository variables (not secrets):

  1. Go to your repository → Settings → Secrets and variables → Actions
  2. Open the Variables tab → New repository variable
  3. Add any of:
  • AI_REVIEW_MODELS: Comma-separated list of models. Each model runs the reviewer independently, so you get one review per model in a single combined comment (see Multiple reviewers). When unset, falls back to AI_MODEL, then to the built-in default.
  • AI_MODEL: A single model (see OpenRouter models). Used by the reviewer when AI_REVIEW_MODELS is unset, and also by the release-notes workflow. Default: moonshotai/kimi-k2-thinking.
  • AI_TEMPERATURE: Adjust randomness (default: 0.1 for consistent reviews)
  • AI_MAX_TOKENS: Maximum response length (default: 64000)
  • MAX_DIFF_SIZE: Maximum diff size in bytes (default: 800000 / 800KB)
  • MAX_SUMMARY_COMMITS: How many of the PR's most recent commits feed the commit overview the reviewer sees (commit counts per author, added/removed line totals); 0 shows the count-only header (default: 15)
  • MAX_COMMIT_MESSAGES: How many commit messages are fully quoted in the prompt — the token-expensive part (default: 3)
  • MAX_HUMAN_COMMENTS / MAX_HUMAN_COMMENT_LENGTH / MAX_HUMAN_COMMENTS_TOTAL: Caps for the human-comment context block (newest-first selection; defaults: 100 comments, 4000 chars each, 20000 bytes overall; 0 disables)
  • CUSTOM_PROMPT / CUSTOM_PROMPT_FILE: Additional review instructions applied on top of the standard contract (inline text and/or a file path such as .github/ai-review-instructions.md; the file is read from the workflow's checkout, which in our pull_request_target setup is the trusted base branch). Capped at 8000 bytes, marked when truncated.

Multiple reviewers

Set AI_REVIEW_MODELS to a comma-separated list to have several models review the same diff. For example:

minimax/minimax-m3, z-ai/glm-5.2

Notes:

  • Separate models with commas (spaces around them are fine and trimmed). A space-separated value is treated as a single model name and will fail.
  • Each model is shown anonymously as Reviewer 1, Reviewer 2, … (the model→reviewer mapping appears only in the workflow logs), so readers judge the feedback rather than the model.
  • All reviews are collected into one sticky PR comment. Approved label suggestions are unioned across reviewers; unknown and control labels are discarded. If FAIL_ON_REQUESTED_CHANGES is enabled, the workflow fails when any reviewer requests changes. This is deliberately "any" rather than a majority vote — a single model catching a real blocker shouldn't be outvoted — and it only takes effect when you opt into FAIL_ON_REQUESTED_CHANGES (off by default). A reviewer that errors out (rather than returning a verdict) never counts as a "fail"; it just shows a note in its section.
  • Models run in parallel, so the job's wall-clock is the slowest single review rather than the sum. Each model still makes its own set of GitHub API calls for context, so a very long list means many concurrent calls; two or three models are well within limits.

Usage

Triggering AI Reviews

To trigger an AI review on a PR:

  1. Go to the PR page
  2. Click Labels
  3. Add the label: ai_code_review

The review will automatically start and post results as a comment when complete.

Label suggestions and opt-in CI

The reviewer may apply semantic labels from the repository-owned allowlist in .github/ai-review-label-policy.json. It cannot create new labels or apply human-only, lifecycle, or workflow-control labels.

Resource-intensive test and research labels are handled as recommendations. They appear under Suggested CI in the review comment, and a maintainer then applies or re-adds the appropriate label to start the workflow:

  • test:puppeteer — paid Puppeteer browser E2E
  • test:notes — focused Notes Playwright E2E
  • test:ui-full-shards — complete sharded UI matrix plus strict Docker CI
  • ldr_research / ldr_research_static — PR-diff or fixed-query research

The handoff exists because each of these workflows spends paid API quota and CI wall-clock, so the decision to start one stays with a maintainer rather than with model output steered by the PR diff. It also preserves GitHub's normal read-only-token and secret restrictions for fork PRs.

Do not reach for GITHUB_TOKEN label application as a shortcut. GitHub does not start a second workflow from a label event created with the first workflow's GITHUB_TOKEN, so test:puppeteer, test:ui-full-shards, and the ldr_research selectors (all labeled-only triggers) would simply be stranded — but test:notes also triggers on synchronize, so applying it here would run the suite on the next push. The cost handoff, not the token behavior, is what makes this the right split.

test:notes does not require repository secrets. Puppeteer, LDR research, and the strict UI wrapper may lack credentials or write permissions on fork PRs; do not replace those protections with a privileged checkout of fork code.

Re-running Reviews

To re-run the AI review after making changes:

  1. Remove the ai_code_review label
  2. Add the ai_code_review label again

This will generate a fresh review of the current PR state.

Review Results

The AI posts a comment whose findings are split into two sections: New problems (introduced by the PR) as bullets tagged must fix, should fix, or nit, in that order, each with its file/line location, a concrete failure scenario, the impact, and a suggested fix; and Pre-existing problems (predating the PR) as one-liners for documentation and issue extraction — they are not to be fixed in this PR and never influence the verdict. Inferences are highlighted with an explicit "Inference (not verified):" label so assumptions never read as verified facts, and anything that cannot be verified from the diff but is worth a human look is collected in a final "Should be checked" section. Praise, change summaries, and empty filler sections (such as a boilerplate "no security concerns" line) are deliberately omitted; a clean diff yields "No actionable findings." followed by the verdict. The review is meant to assist human reviewers, not replace them.

Cost Estimation

Costs vary by model, but most code-focused models on OpenRouter are very affordable:

  • Typical small PR (< 1000 lines): $0.001 - $0.01
  • Large PR (1000-5000 lines): $0.01 - $0.05

Check OpenRouter pricing for specific model costs.

Customization

Changing the Review Focus

The review prompt lives in ai-reviewer.sh, which the workflow downloads at run time from the Friendly-AI-Reviewer repository (see the "Download AI reviewer script" step in .github/workflows/ai-code-reviewer.yml) — it is not stored in this repo. The current focus areas are:

  • Security (secrets, injection attacks, authentication)
  • Performance (algorithms, queries, memory)
  • Code Quality (readability, maintainability, error handling)
  • Best Practices (standards, patterns, type safety)

To adjust them, edit the prompt in that repository (or fork it and point the workflow's download step at your fork).

Troubleshooting

Reviews Not Running

  • Ensure the ai_code_review label is added (not just present)
  • Check that OPENROUTER_API_KEY secret is correctly configured
  • Verify GitHub Actions permissions are properly set

API Errors

  • Check OpenRouter API key validity
  • Verify OpenRouter account has sufficient credits
  • Review GitHub Actions logs for specific error messages

Diff Too Large Error

If you get a "Diff is too large" error:

  • Split your PR into smaller, focused changes
  • Or increase MAX_DIFF_SIZE in the workflow file
  • Default limit is 800KB (~200K tokens)

Security Considerations

  • API keys are stored securely in GitHub Secrets and passed via environment variables
  • Reviews only run when the ai_code_review label is manually added
  • All API calls are made through secure HTTPS connections
  • Code diffs are sent to OpenRouter/AI provider - review their data policies
  • The workflow has minimal permissions (read contents, write PR comments)

Support

For issues with: