Skip to content

OpenRAG is a comprehensive, single package Retrieval-Augmented Generation platform built on Langflow, Docling, and Opensearch.

Python
7.9 KB
Raw Blame

OpenRAG Python SDK — QA Test Checklist

Live integration tests against a running OpenRAG instance (http://localhost:3000 by default).

Run all SDK tests:

make test-sdk

Authentication (test_auth.py)

# Test Expected
1 Construct client with no API key Raises AuthenticationError immediately
2 Construct client with extra_headers only (IBM auth mode) and make a real API call Headers reach the server end-to-end; succeeds, or raises a clean AuthenticationError (401) if IBM auth mode isn't enabled on the target instance
3 Construct client with only OPENRAG_API_KEY env var set Client constructs without error
4 Explicit api_key argument with OPENRAG_API_KEY env var also set Explicit value takes precedence over the env var
5 Send request with invalid API key Raises AuthenticationError with status 401 or 403
6 Send request with well-formed but non-existent key Raises AuthenticationError

Chat (test_chat.py)

# Test Expected
7 Non-streaming chat Returns non-empty response string; any sources are valid Sources with score in [0, 1]
8 Chat with filters, limit, and score_threshold together len(response.sources) <= limit
9 Streaming chat (create(stream=True)) Yields content events with text deltas; the last event is done with a non-null chat_id
10 Streaming via context manager (stream()) Accumulated stream.text is non-empty
11 text_stream async iterator Yields plain text chunks
12 final_text() Returns full accumulated response
13 Conversation continuation (pass chat_id) Second reply uses same conversation; followup.chat_id == chat_id
14 List conversations Returns list of conversations; title, created_at, last_activity, message_count are correctly typed
15 Get conversation by ID Returns conversation with message history; message role/content are correctly typed
16 Delete existing conversation Returns True
17 Chat with ingested document (RAG) Response sources include the ingested file
18 Stream continuation with chat_id Follow-up stream uses existing conversation
19 Every response includes chat_id chat_id is a non-empty string
20 chat_id available after stream consumed stream.chat_id is populated
21 sources field on response Always a list (may be empty)

Documents (test_documents.py)

# Test Expected
22 Ingest file (async, wait=False) Returns a non-empty task_id; polling reaches terminal state
23 Ingest file (blocking, wait=True) Returns terminal status with total_files, processed_files, successful_files, failed_files all populated
24 Delete ingested document (deterministic ingest) success=True, deleted_chunks > 0
25 Delete never-ingested filename success=False, deleted_chunks=0, error message present
26 get_task_status() with a nonexistent task id Raises NotFoundError
27 Delete by a genuinely nonexistent filter_id Raises NotFoundError (distinct from the wildcard-data_sources rejection case, which raises a generic OpenRAGError)
28 wait_for_task() with a very small timeout Raises TimeoutError before the task completes
29 Ingest via file object (io.BytesIO) Accepted and processed without error
30 Re-ingest same filename twice Does not raise; second call returns a status
31 Ingest .md file Accepted and processed without error
32 Poll task status manually get_task_status() returns a status; wait_for_task() returns completed or failed
33 Delete documents by filter_id Removes only the filenames in the filter's data_sources
34 Delete by filter_id with wildcard data_sources Rejected with OpenRAGError
35 Delete with both filename and filter_id Rejected with ValueError
36 Delete with neither filename nor filter_id Rejected with ValueError

Search (test_search.py)

# Test Expected
37 Basic search query Returns a results list; each result's score is in [0, 1]
38 Search with limit=1 Returns at most 1 result
39 Search with score_threshold=0.99 Returns a list (may be empty) without error
40 Search with score_threshold=0.5 All returned results have score >= 0.5
41 Nonsense/obscure query Returns empty list, no error
42 Unicode and emoji in query Returns list, no error
43 Result fields (limit=5) At most 5 results; text is a non-empty string; page/mimetype are None or correctly typed
44 Whitespace-only query (" ") Raises ValidationError
44a Search with SearchFilters(data_sources=[filename]) Wildcard query returns only chunks from that file; a second ingested file is excluded

Settings (test_settings.py)

# Test Expected
45 Get settings Response includes agent and knowledge sections, including chunk_overlap, table_structure, ocr, picture_descriptions, and agent.system_prompt (each None or correctly typed)
46 Update chunk_size setting Update succeeds; value readable back unchanged

Models (test_models.py)

# Test Expected
47 List models for a provider (openai) Returns language_models and embedding_models as lists
48 List models, parametrized per provider (openai, anthropic, ollama, watsonx) openai is required to return typed ModelOption entries (value: str) with at most one default; other providers are marked SKIPPED (not failed) if unconfigured (ValidationError)
49 List models for an invalid provider Raises ValidationError

Knowledge Filters (test_filters.py)

# Test Expected
50 Create filter success=True, id returned, error is None
51 Search filters by name Returns list containing the created filter; a guaranteed-no-match query returns []; limit=1 returns at most 1
52 Get filter by ID Returns filter with correct id, name, query_data, owner, created_at, updated_at
53 Update filter description Update returns True; description readable back
54 Delete filter (wrapped in try/finally so it can't leak) Returns True
55 Get deleted filter Returns None
56 Pass filter_id to chat.create() No error; response returned
57 Pass filter_id to search.query() No error; results returned

Error Handling (test_errors.py)

# Test Expected
58 Connect to dead port Raises a network exception within timeout
59 Get conversation with random UUID Raises NotFoundError with status_code == 404
60 Delete conversation with random UUID Returns False
61 Update settings with invalid value (chunk_size=-999999) Raises OpenRAGError subclass
62 Call ingest() with no arguments Raises ValueError
63 Call ingest() with BytesIO but no filename Raises ValueError
64 Iterate a fully-consumed ChatStream a second time Raises RuntimeError
65 client.close() with a caller-supplied http_client Does not close the external httpx.AsyncClient
66 API call after an async with OpenRAGClient(...) block exits Raises (the SDK-owned client was closed on __aexit__)

End-to-End (test_e2e.py)

# Test Expected
67 Full RAG pipeline: ingest → search → chat Chat sources include the ingested document
68 Multi-turn conversation with RAG Second turn uses same chat_id; context carried over
69 Knowledge filter scopes search and chat Search and chat succeed with filter_id; filter cleaned up

Total: 70 tests across 9 domains.