OpenRAG is a comprehensive, single package Retrieval-Augmented Generation platform built on Langflow, Docling, and Opensearch.
Retrieval evaluation harness
This harness measures how well OpenRAG search finds the right documents. It
ingests a fixed set of documents, sends a fixed set of labeled questions
through the public search API (/v1/search), and scores where the expected
documents land in the results. Run it before and after a change to search,
chunking or embeddings to see which questions got better and which got worse.
It measures retrieval only. It does not judge chat answers.
FRAMEWORK_COMPARISON.md explains why this harness is homegrown rather than built on RAGAS, DeepEval or LangSmith.
Quick start
Run these from this directory against a running stack. You need an API key from Settings → API Keys.
cd scripts/eval/retrieval
export OPENRAG_URL=http://localhost:3000
export OPENRAG_API_KEY=orag_your_api_key
# Check the corpus and questions. Needs no stack.
uv run run_eval.py run --validate-only
# Run every question and keep the results as a baseline.
uv run run_eval.py run
cp results/results.json results/baseline.json
# After a change to search, chunking or embeddings, run again and compare.
uv run run_eval.py run --baseline results/baseline.json
# Read the summary of the latest run.
cat results/summary.md
The same steps are available as make targets from the repository root.
Paths are relative to this directory.
make target |
Same as |
|---|---|
make eval-validate |
uv run run_eval.py run --validate-only |
make eval-check |
uv run run_eval.py check |
make eval-cleanup |
uv run run_eval.py cleanup (add CONFIRM=1 for --yes) |
make eval-run |
uv run run_eval.py run |
make eval-run BASELINE=results/baseline.json |
uv run run_eval.py run --baseline results/baseline.json |
make eval-baseline |
uv run run_eval.py run, then copy the results to results/baseline.json |
make eval-compare |
uv run run_eval.py compare results/baseline.json results/results.json |
make eval-compare BASELINE=a.json CURRENT=b.json |
uv run run_eval.py compare a.json b.json |
make eval-baseline OUTPUT_DIR=results/demo |
uv run run_eval.py run --output-dir results/demo, then copy the results to results/demo/baseline.json |
Pass any other option with EVAL_ARGS, for example
make eval-run EVAL_ARGS="--keep-documents --fetch-limit 100". To write results
somewhere else, set OUTPUT_DIR rather than passing --output-dir in
EVAL_ARGS, so eval-baseline copies the file the run actually wrote.
make eval-baseline stops with an error if EVAL_ARGS contains --output-dir.
The rest of this file explains each step.
What is here
| Path | Purpose |
|---|---|
corpus/ |
21 Markdown documents: the handbook of a fictional product, "Relaywise" |
queries.jsonl |
83 hand-labeled questions, one JSON object per line |
run_eval.py |
The command-line tool, with run, check, cleanup, compare and promote commands |
dataset.py |
Loads and validates the corpus and questions |
metrics.py |
Turns ranked chunks into ranked files and scores them |
report.py |
Writes results, the summary and the baseline diff |
FRAMEWORK_COMPARISON.md |
How this harness compares with RAGAS, DeepEval, LangSmith and others |
results/ |
Output of each run (not committed) |
The corpus is invented so that no model has seen it and no license applies. Its documents overlap on purpose. For example, API keys and signing secrets both have a "rotation" section, so the ranking has to tell them apart.
Prerequisites
A running OpenRAG stack that has finished onboarding, with working ingestion and an embedding model.
Python 3.11 or later and uv.
An OpenRAG API key. When RBAC is enforced (
OPENRAG_RBAC_ENFORCE=true), the key needs these permissions:Permission Used for knowledge:uploadIngesting the corpus knowledge:read:ownReading ingestion task status search:useRunning the questions knowledge:delete:ownDeleting the corpus when the run ends providers:readRecording the embedding and chunk settings (optional)
The harness depends only on openrag-sdk, which it takes from sdks/python in
this checkout, so a run tests the SDK and API of the code you have checked out.
Run it
cd scripts/eval/retrieval
export OPENRAG_URL=http://localhost:3000
export OPENRAG_API_KEY=orag_your_api_key
uv run run_eval.py run
A run does the following:
- Validates the corpus and the question file. A bad label stops the run before anything is uploaded.
- Uploads each corpus file under a name unique to the run, such as
eval-3f9a12cd-api-keys.md, and waits until every file is searchable. - Sends each question to
/v1/search, limited to the files this run uploaded, so other documents in the stack are never returned. They can still shift the scores slightly; see Repeatability. - Prints a summary and writes it, with the full results, to
results/. - Deletes the files it uploaded.
Documents that were already in the stack are never searched, changed or
deleted. If a run is interrupted before cleanup, delete the leftover
eval-<run id>-* files with run_eval.py cleanup (see
Clean up after an interrupted run).
Useful options:
| Option | Default | Purpose |
|---|---|---|
--baseline FILE |
none | Compare this run with an earlier results file |
--output-dir DIR |
results/ |
Where to write results |
--k 1,3,5,10 |
1,3,5,10 |
Cutoffs to report |
--fetch-limit N |
50 |
Chunks requested per question; must be at least the largest --k |
--score-threshold X |
0 |
Minimum score passed to search |
--concurrency N |
4 |
Parallel uploads |
--keep-documents |
off | Leave the uploaded files in place for debugging |
--validate-only |
off | Check the dataset and exit; needs no stack |
Run uv run run_eval.py run --help for the full list, including timeouts.
Run against a hosted environment
The harness runs from your machine against any stack it can reach, so a QA or staging environment needs no GitHub Actions. Point it at the environment and check it first:
cd scripts/eval/retrieval
export OPENRAG_URL=https://openrag-qa.example.com
export OPENRAG_API_KEY=orag_key_for_that_environment
uv run run_eval.py check
check uploads nothing. It confirms that the stack answers and accepts the
key, that it can read the embedding and chunk settings, and whether earlier
runs left documents behind:
Checking https://openrag-qa.example.com
ok dataset: 21 documents, 83 questions
ok search: the stack answered and accepted the API key
ok settings: watsonx ibm/granite-embedding-278m-multilingual, chunk size 1000, overlap 200
ok documents: no earlier run left documents behind
Not checked: uploading and deleting documents (knowledge:upload, knowledge:delete:own) need a real run.
Ready for a run.
It exits with status 1 when a run would fail or its results could not be compared, and says what to fix. A run needs the settings: without them its results cannot be compared with a baseline or become one.
Keep a separate baseline for each environment, because scores from different embedding models or settings are not comparable. Write each environment's results to its own directory:
# Once, to record where the environment stands:
make eval-baseline OUTPUT_DIR=results/qa
# After each deploy to that environment:
make eval-run OUTPUT_DIR=results/qa BASELINE=results/qa/baseline.json
The tracked baseline.json belongs to the GitHub workflow's stack. Do not
compare a hosted environment with it.
If the environment uses a certificate from a private CA, set SSL_CERT_FILE to
the CA bundle before running.
Clean up after an interrupted run
A run deletes its documents when it ends, even when it fails. If it is killed first, they stay in the stack. On a stack other people use, list them with:
uv run run_eval.py cleanup # or: make eval-cleanup
uv run run_eval.py cleanup --yes # or: make eval-cleanup CONFIRM=1
cleanup only matches files named like an eval upload (eval-<run id>- plus a
corpus filename), and only those the API key can see. It keeps any document
indexed in the last 60 minutes, because it may belong to a run still in
progress, possibly someone else's. Use --min-age to change that window.
Compare before and after a change
# 1. On main, or before your change:
uv run run_eval.py run
cp results/results.json results/baseline.json
# 2. Apply the change, restart the stack, then:
uv run run_eval.py run --baseline results/baseline.json
The summary then ends with a comparison that lists the questions that regressed first, then the ones that improved, then the change in each average. For every changed question it shows the rank of each expected file before and after.
To make a run exit with status 1 when average recall falls too far, add
--max-recall-drop 0.02 (two points) next to --baseline. It is off by default.
To compare two results files without running anything:
uv run run_eval.py compare results/baseline.json results/results.json
The comparison warns when the two runs differ in something other than the change under test: the corpus, the question file, the k values, the fetch limit, the score threshold, or the embedding and chunk settings of the stack. Scores from different embedding models or chunk sizes are not comparable unless that setting is what you are testing.
Repeatability
Two runs against the same stack do not give exactly the same scores. Search combines vector similarity with keyword matching, and the keyword part weighs each word by how common it is across the whole index. That count includes documents outside the eval, and chunks that earlier runs deleted but OpenSearch has not yet purged. Each run therefore changes the index a little for the next.
On a local stack with watsonx embeddings, three unchanged runs in a row gave the same recall@5 (0.904) and an MRR within 0.002, but up to 13 of the 83 questions moved by one rank and were listed as regressed or improved. Halving the chunk size moved recall@5 by 0.036 and MRR by 0.070.
To tell a real change from this noise:
- Run the baseline twice and compare the two runs. Whatever that comparison lists is the noise level of your stack.
- Trust the averages and changes in recall@5 over a one-rank move on a single question.
- For the steadiest numbers, use a stack that holds no other documents.
Scores depend on the embedding model and settings of the stack that produced them, so create the baseline on the same stack you test against. The GitHub Actions workflow keeps a tracked baseline for its own stack; see Run it from GitHub Actions.
Run it from GitHub Actions
The Retrieval Eval workflow (eval-retrieval.yml)
runs the harness on a fresh stack built from the branch you pick and compares
it with the tracked baseline.json in this directory, which exists once the
first run has created it. It runs only when someone
starts it. It never runs on a push or a pull request, and it is not a required
check.
To start it, open the Actions tab, choose Retrieval Eval, click Run workflow, pick a branch and fill in the inputs. Or from the command line:
gh workflow run eval-retrieval.yml --ref my-branch
GitHub lists a manually started workflow only once its file is on the default branch, so this works after the workflow has been merged.
| Input | Meaning |
|---|---|
max_recall_drop |
Fail the run if average recall at the compared cutoff falls by more than this below the baseline, for example 0.02 for two points. Leave it empty to report without failing. |
update_baseline |
If the run improves on the baseline, commit the new baseline.json to the selected branch. Refused on the default branch. |
min_gain |
Average recall must rise by more than this to count as an improvement. Default 0.025. One of the 83 questions flipping moves average recall by about 0.012, so a lower value lets noise raise the baseline. |
Each run writes the summary to the run's page and uploads results.json and
summary.md as the retrieval-eval-results artifact, kept for 90 days.
The tracked baseline
baseline.json is committed to the repository and is the reference for every
run. It changes only through update_baseline:
- Run the workflow on your branch with
update_baselineticked. - If average recall rose by more than
min_gain, the workflow commits the newbaseline.jsonto your branch. Otherwise it leaves the file alone and says why in the run summary. - Merge the branch in a pull request. The reviewer sees the new scores in the diff.
The workflow never commits to the default branch, so a baseline change is
always reviewed. A run is not promoted when it did not record the stack settings, or
when it is not like for like with the baseline (a different corpus, question file, cutoffs or stack settings),
because a higher number would not mean better search. To start over after such
a change, for example a new embedding model, delete baseline.json and run
with update_baseline. With no baseline, the first run becomes the baseline.
The same logic is available locally:
uv run run_eval.py promote results/results.json baseline.json --min-gain 0.025
It exits with 0 when it wrote the baseline and 3 when it left it alone.
The stack uses the embedding provider and model set by the repository's
EMBEDDING_PROVIDER and EMBEDDING_MODEL secrets, the same as the end-to-end
tests. Scores from different models are not comparable, and the comparison
warns when the settings of the two runs differ.
The harness gives the same stack slightly different scores from run to run (see
Repeatability). Set max_recall_drop and min_gain above that
noise, or runs will fail, or the baseline will creep up, on unchanged code.
How questions are scored
Search returns chunks, but questions are labeled with files, because chunk boundaries move whenever chunking changes. For each question the harness:
- Requests
--fetch-limitchunks. - Keeps the first chunk of each file, which gives a ranked list of unique files.
- Treats an expected file as relevant if it is in that list. If the question
has an
expected_text, at least one returned chunk of that file must also contain the text. The match ignores case and differences in whitespace.
| Metric | Meaning |
|---|---|
| recall@k | Share of the expected files that appear in the top k files |
| MRR | 1 divided by the rank of the first relevant file, averaged over questions; 0 when none is returned |
| nDCG@k | Like recall@k, but a relevant file counts for more the nearer it is to the top |
All three run from 0 to 1, and higher is better. Runs are compared on recall@5. If either run did not evaluate k=5, the comparison uses a cutoff both runs have and says so. Two runs with no cutoff in common cannot be compared.
Passage scores
A file can rank first while the chunk that answers the question sits much
lower. For every question with an expected_text, the harness also records
the rank of the first returned chunk that comes from an expected file and
contains that text. It counts every chunk this run uploaded, not just the first
chunk of each file.
| Metric | Meaning |
|---|---|
| passage hit@k | 1 when the answer chunk is in the top k chunks, otherwise 0 |
| Passage MRR | 1 divided by the rank of the answer chunk, averaged over questions; 0 when no chunk holds the text |
Chunks are matched on their text, not on chunk IDs, so passage scores can still
be compared when the chunk size changes. Questions without an expected_text
have no passage scores and are left out of these averages. The summary says how
many questions the passage scores cover.
Passage scores are reported for information. Questions are still classified as regressed or improved on the file scores above. A baseline written before passage scores existed still compares; the comparison notes that only file scores were compared.
Results files
Each run writes four files to the output directory: results.json and
summary.md for the latest run, and a timestamped copy of each.
results.json has these parts:
| Field | Contents |
|---|---|
schema_version |
Format version of the file |
run |
Run ID, time, target URL, SDK version, git commit, hashes of the corpus and question file, k values, fetch limit and score threshold |
settings |
Embedding provider and model, chunk size and overlap, as reported by the stack; null if they could not be read |
metrics.overall |
Mean of each metric over all questions |
metrics.by_tag |
The same means for each tag, with the number of questions |
queries |
One record per question |
Each record in queries holds the question, its labels, the ranked files with
the score of their first chunk, the rank of each expected file
(expected_ranks, null when a file was not returned), whether the expected
text was found, the rank of the answer chunk (passage_rank, only for questions
with an expected_text, null when no chunk held it), and the metrics for that
question.
Add or change a question
Add a line to queries.jsonl:
{"id": "key-rotate", "query": "How do I swap out an API key without causing downtime?", "expected_files": ["api-keys.md"], "expected_text": "overlap window", "tags": ["semantic"]}
| Field | Required | Meaning |
|---|---|---|
query |
Yes | The question, as a user would type it |
expected_files |
Yes | Corpus filenames that answer it; list more than one only if each file answers it |
id |
No | Stable name used to match the question between runs; defaults to the query text |
expected_text |
No | A short phrase that appears in every expected file |
tags |
No | Labels used to break the scores down |
Guidelines:
- Write the question in your own words. A question copied from the document mostly tests keyword matching.
- Give every question an
id. Without one, rewording the question makes it look like one question was removed and another added. - Keep
expected_textto a few words from a single line of body text, not a heading. A long phrase can be split across two chunks and then never match. - Use the existing tags where they fit:
semanticfor a paraphrased question,keywordfor one that uses the document's own terms,exact-tokenfor an identifier such as an error code or a setting name, andmulti-docwhen more than one file is expected.
To add a document, put a Markdown file in corpus/ and add at least one
question that expects it.
Then check the dataset. This needs no stack:
uv run run_eval.py run --validate-only
Changing the corpus or the questions changes the scores, so create a new baseline afterwards. The comparison warns if you forget.
Tests
The scoring, validation and comparison logic is covered by
tests/unit/test_retrieval_eval.py, which needs neither the SDK nor a stack.
From the repository root:
uv run pytest tests/unit/test_retrieval_eval.py
Answer quality
Answer evaluation is opt-in and informational. The CI gate still checks file recall@5; answer scores do not affect gating or baseline promotion.
uv run run_eval.py check --answers
uv run run_eval.py run --answers --output-dir results/answers
# From the repository root:
make eval-answers OUTPUT_DIR=results/answers BASELINE=results/answers/baseline.json
make eval-answers also accepts EVAL_ARGS. --answer-limit controls how many
chunks chat may retrieve (default 10); it is independent of --fetch-limit.
Chat uses the run's exact uploaded-file scope and the same score threshold as
search. Every question starts a fresh conversation; conversations are deleted
in the run's cleanup, including when a later request fails. --keep-documents
keeps documents only, not conversations. Cleanup failures are printed as warnings.
| Score | Meaning |
|---|---|
| Cites expected file | 1 when chat's returned sources include an expected file; 0 otherwise. This checks source metadata, not inline citation syntax. |
| Contains answer phrase | 1 when the answer contains expected_text, ignoring case and whitespace; 0 otherwise. Only questions with that label contribute. A correct paraphrase can score 0. |
| Answer correctness | Judge's 0–2 score divided by 2: wrong or absent = 0, partly correct or missing a key detail = 0.5, matches the reference = 1. |
| Answer groundedness | Whether substantive claims are supported by the source text supplied to the judge: yes = 1, partial = 0.5, no = 0. |
Add expected_answer as one or two non-empty sentences drawn from the corpus.
Answer the question directly and include the essential conditions, units,
plan limits or exceptions. It is a semantic reference, not a required verbatim
phrase; keep expected_text for phrase matching. The field is optional; a
question without it still gets deterministic scores but no judgement. The
83 shipped references and their review status are described in
answer-label-review.md.
By default the judge calls the stack's OpenAI-compatible completions gateway
with its default chat model, using the same API key. This can bias it toward
its own answers. Choose another configured model with --judge-model MODEL
to reduce that bias. --no-judge runs only the deterministic checks and avoids
the judge calls. Unusable judge replies and request failures are recorded in
answer.judge_error, counted in the summary and excluded from judged averages;
they are never silently scored as wrong. A failed chat request is handled the
same way: it is recorded in answer.chat_error, listed under "Chat errors" in
the summary and left out of the answer averages, and the run continues. After
three chat failures in a row the remaining questions are skipped and marked,
so a dead chat endpoint cannot stall the run; the retrieval scores are still
written. Chat and judge requests use --chat-timeout (default 180 seconds)
instead of --request-timeout, since an agent answer with tool calls can take
longer than a search.
A full 83-question pass adds 83 chat calls and up to 83 judge calls. Budget
roughly 10–20 minutes, depending on the provider; tokens and charges depend
on the configured model, retrieved text and answer lengths. check --answers
also costs one chat call and one judge call, unless --no-judge is passed;
it uploads nothing and deletes its test conversation. Besides the existing
permissions, the key needs chat:use and conversations:delete:own.
Answers and judgements can vary across identical runs. Trust the averages more than a single question flipping, and compare two unchanged runs to establish the noise level for your stack. No measured noise level is claimed until those live runs have been completed. Review roughly ten answers against the corpus, including every answer judged 0, before relying on a judge.
Results record agent provider/model, a SHA-256 of the system prompt (never the
prompt text), judge model, judge enablement and chat retrieval limit in the
separate answer_settings object. Differences produce comparison notes,
not retrieval warnings. The full question-file hash is retained for provenance;
retrieval_queries_hash excludes additive answer references so they do not
invalidate the phase-1 retrieval baseline. Other retrieval-label edits still
produce comparison warnings. The JSON schema remains version 1.
The summary includes overall and by-tag answer scores and answers judged wrong. Comparing two answer runs lists correctness changes of at least one point on the judge's raw 0–2 scale. Comparing against a run without answers adds a note. Save the first run before rerunning so it is not overwritten:
uv run run_eval.py run --answers --output-dir results/answers-first
uv run run_eval.py run --answers --output-dir results/answers-second
uv run run_eval.py compare results/answers-first/results.json results/answers-second/results.json