The agent engineering platform.
type: "Testing & QA" title: "Integration Testing: Standard Test Suites & Providers" description: "How LangChain integration tests work: VCR cassettes for API recording/playback, standard test suite inheritance for providers, CI/CD integration, and test markers for live vs. recorded testing." tags: [integration-tests, vcr, cassettes, api-testing, pytest, ci-cd, model-testing, standard-tests, test-markers] verified:
- by: openwiki/0.5.0 at: 2026-10-20T08:25:57.412Z sources:
- id: openwiki-source-a12f520c2572fe032a111e3b resource: repo://.github/workflows/_compile_integration_test.yml
- id: openwiki-source-d8a8900818f4abab719bd1b7 resource: repo://.github/workflows/_test_vcr.yml
- id: openwiki-source-6e3a52c89729b5704dbd7eec resource: repo://.github/workflows/check_diffs.yml
- id: openwiki-source-9069a5dd5fbb579fbd5470ce resource: repo://.github/workflows/integration_tests.yml
- id: openwiki-source-ff76574b014ac8b5c67560a6 resource: repo://libs/langchain_v1/tests/integration_tests/chat_models/test_base.py
- id: openwiki-source-bcf7be66f36f862f639f3c7a resource: repo://libs/langchain_v1/tests/integration_tests/conftest.py
- id: openwiki-source-bae620f3bb8d2668c69ac079 resource: repo://libs/langchain/tests/integration_tests/.env.example
- id: openwiki-source-9a9f29414e5bd045f28c0081 resource: repo://libs/partners/openai/Makefile
- id: openwiki-source-df762860acfcc6abf0ce804b resource: repo://libs/partners/openai/pyproject.toml
- id: openwiki-source-ed8ac34fd0216d557729038e resource: repo://libs/partners/openai/tests/conftest.py
- id: openwiki-source-ce605960b642234837131282 resource: repo://libs/partners/openai/tests/integration_tests/chat_models/conftest.py
- id: openwiki-source-711811fb8df6b32329091d23 resource: repo://libs/partners/openai/tests/integration_tests/chat_models/test_base_standard.py
- id: openwiki-source-d619d2c7cfe653fb9b712740 resource: repo://libs/partners/openai/tests/integration_tests/chat_models/test_base.py
- id: openwiki-source-e758f7cc743f83c081224f08 resource: repo://libs/partners/openai/tests/integration_tests/embeddings/test_base.py
- id: openwiki-source-db02c1dda8563ab005cd9d62 resource: repo://libs/standard-tests/langchain_tests/conftest.py
- id: openwiki-source-13cd70e2dd4252d8b05ac9c1 resource: repo://libs/standard-tests/langchain_tests/integration_tests/init.py
- id: openwiki-source-f7c26a6473517c7b426ef023 resource: repo://libs/standard-tests/langchain_tests/integration_tests/chat_models.py generated: { by: "openwiki/0.5.0", at: "2026-10-20T08:25:57.412Z" }
Overview
Integration tests in LangChain differ fundamentally from unit tests: they call real model APIs (OpenAI, Anthropic, Groq, etc.) and require network access and valid credentials. To make these tests reproducible and CI-friendly without exposing API keys or relying on external services, LangChain uses VCR cassettes—recorded HTTP interactions that are replayed during test execution.
All LangChain partner packages inherit a standard suite of integration tests via ChatModelIntegrationTests and other base classes. This ensures consistent test coverage across providers while allowing provider-specific customization via property overrides and method extension.
This page covers:
- VCR cassette recording, playback, and security (scrubbing secrets)
- Test markers (
@pytest.mark.vcr,@pytest.mark.scheduled) for classification - Standard test suite inheritance and customization points
- CI/CD integration: compilation checks, VCR playback in PR checks, scheduled live testing
- Environment setup, cassette management, and best practices
Test Markers: Classification and Execution
LangChain uses pytest markers to classify integration tests into three categories, each running in different CI contexts:
@pytest.mark.vcr — Cassette-Backed Tests
Tests marked with @pytest.mark.vcr use VCR cassettes for playback. No API key is needed in CI; VCR intercepts HTTP calls and replays pre-recorded responses.
@pytest.mark.vcr
def test_chat_invoke():
"""Test chat invoke using a cassette."""
chat = ChatOpenAI(model="gpt-4o-mini")
response = chat.invoke("Hello")
assert response.content
When cassettes are used:
- In CI: Always. VCR runs with
--record-mode=none(playback-only). - Locally: On first run, if a cassette doesn't exist,
--record-mode=oncewill record it (if API key is present). On subsequent runs, VCR replays the cassette.
Advantages:
- Tests are fast (no network latency)
- Tests are deterministic (same response every time)
- No credentials needed in CI
- Cassettes are committed to git so all developers can run tests
@pytest.mark.scheduled — Live API Tests
Tests marked with @pytest.mark.scheduled run live API calls without cassettes. These tests are skipped in regular CI but run in scheduled daily workflows with real credentials.
@pytest.mark.scheduled
def test_chat_streaming_live():
"""Test streaming behavior with a live API call."""
chat = ChatOpenAI(model="gpt-4o-mini")
response = chat.invoke("Tell me a joke")
assert response.content
When these run:
- In PR checks: Skipped
- In scheduled daily workflows (
.github/workflows/integration_tests.yml): Run with real API credentials - On-demand: Can be manually triggered from GitHub Actions UI
Purpose: Validates that the model integrations work against live APIs. Catches issues not covered by cassettes (API changes, model behavior changes, breaking SDK changes).
No Marker — Unit-Like Tests
Tests without @pytest.mark.vcr or @pytest.mark.scheduled run as offline unit tests. They don't require API keys and don't make network calls.
def test_model_name_set():
"""Test model initialization without API calls."""
chat = ChatOpenAI(model="gpt-4o-mini")
assert chat.model_name == "gpt-4o-mini"
VCR Cassettes: Recording, Playback, and Security
The VCR Pattern
VCR (Video Cassette Recorder, implemented by the vcrpy library) records HTTP interactions—requests and responses—the first time a test runs with a live API. On subsequent runs, VCR replays the recorded cassette instead of making live network calls.
Recording Phase (Once, by Developers)
When a new integration test is written or an existing test's behavior changes:
- Run the test with a valid API key in
.envand--record-mode=once(default) or--record-mode=new_episodes - VCR intercepts all HTTP calls your test makes
- Before recording, sensitive headers and tokens in request/response bodies are scrubbed (redacted)
- The cassette is saved as a YAML file, optionally compressed as
.yaml.gz - The cassette is committed to the repository
cd libs/partners/openai
uv run --group test --group test_integration pytest \
--record-mode=new_episodes \
tests/integration_tests/chat_models/test_base.py::test_chat_openai_invoke
Playback Phase (Always, in CI and Local Development)
When running tests:
- VCR loads the cassette file
- Instead of making real API calls, VCR intercepts your HTTP client and returns the pre-recorded response
- No API key is needed; no credentials are exposed
- Tests are fast and deterministic
Cassette Location and Naming
Cassettes are stored relative to test modules in a cassettes/ subdirectory:
libs/partners/openai/tests/
integration_tests/
chat_models/
test_base.py
test_base_standard.py
cassettes/
test_base/
test_invoke.yaml.gz
test_chat_openai.yaml.gz
test_base_standard/
TestOpenAIStandard.test_invoke.yaml.gz
TestOpenAIStandard.test_stream[model0].yaml.gz
For parameterized tests, cassette names include parameter identifiers:
test_invoke[gpt-4o-mini].yaml.gz
test_invoke[gpt-4o].yaml.gz
The vcr_cassette_dir fixture (from conftest.py) computes the correct directory per test module:
@pytest.fixture(scope="module")
def vcr_cassette_dir(request: pytest.FixtureRequest) -> str:
module = Path(request.module.__file__)
return str(module.parent / "cassettes" / module.stem)
Security: Scrubbing Sensitive Data
VCR cassettes are committed to git and visible to the world (in public repositories). To prevent accidental exposure of API keys, JWTs, OAuth tokens, and other secrets, LangChain uses a multi-layer scrubbing pipeline.
What Gets Redacted
Request and Response Headers (configured in conftest.py):
_EXTRA_HEADERS = [
("authorization", "PLACEHOLDER"),
("openai-organization", "PLACEHOLDER"),
("cookie", "PLACEHOLDER"),
("set-cookie", "PLACEHOLDER"),
("chatgpt-account-id", "PLACEHOLDER"),
]
All headers are replaced with **REDACTED** during recording.
Request and Response Bodies (OAuth secret fields):
_OAUTH_SECRET_FIELDS = frozenset({
"access_token",
"refresh_token",
"id_token",
"code",
"device_code",
"client_secret",
"code_verifier",
})
Redaction is applied with specialized handlers:
- JSON bodies: Recursively walks the parsed JSON tree and redacts any field matching
_OAUTH_SECRET_FIELDS - Form-encoded bodies: Splits on
&and redacts matching keys - JWT patterns: Uses regex to detect and redact JWT-shaped strings anywhere in the body
Binary payloads (PNG, JPEG, PDF, audio, etc.) are skipped—their magic bytes are detected and the scrubbing stack is bypassed for performance. JWTs and OAuth secrets are ASCII, so binary bodies cannot carry them.
Scrubbing Configuration in conftest.py
def remove_request_headers(request: Any) -> Any:
"""Remove sensitive headers and OAuth secrets from the request."""
for k in request.headers:
request.headers[k] = "**REDACTED**"
request.uri = "**REDACTED**"
request.body = _scrub_oauth_secrets(request.body)
return request
@pytest.fixture(scope="session")
def vcr_config() -> dict:
config = base_vcr_config()
config["before_record_request"] = remove_request_headers
config["before_record_response"] = remove_response_headers
config["serializer"] = "yaml.gz"
return config
Environment Setup: API Keys and .env
Integration tests require valid API credentials to record cassettes (once) and for running live integration tests in scheduled or on-demand scenarios. Store credentials in a .env file at the package's test directory root.
.env File Location and Format
For partner packages (e.g., OpenAI), the structure is:
libs/partners/openai/tests/.env
Example .env.example (visible in repo):
# openai
# your api key from https://platform.openai.com/account/api-keys
OPENAI_API_KEY=your_openai_api_key_here
# searchapi
SEARCHAPI_API_KEY=your_searchapi_api_key_here
Important: .env files are .gitignored and never committed. Copy .env.example to .env locally and populate with valid keys.
Environment Loading
The test conftest automatically loads .env on import:
from pathlib import Path
from dotenv import load_dotenv
PROJECT_DIR = Path(__file__).resolve().parent.parent
def _load_env() -> None:
dotenv_path = PROJECT_DIR / "tests" / ".env"
if dotenv_path.exists():
load_dotenv(dotenv_path)
_load_env()
Credentials are then accessed via os.environ["OPENAI_API_KEY"] or similar in tests.
Writing Integration Tests
Basic Structure with VCR
Integration tests live in tests/integration_tests/ (mirroring the layout of unit tests):
libs/partners/openai/tests/
unit_tests/
integration_tests/
chat_models/
__init__.py
conftest.py
test_base.py
test_base_standard.py
cassettes/
A minimal integration test using @pytest.mark.vcr:
import pytest
from langchain_openai import ChatOpenAI
from langchain_core.messages import HumanMessage
@pytest.mark.vcr
def test_chat_openai_invoke():
chat = ChatOpenAI(model="gpt-4o-mini")
response = chat.invoke([HumanMessage(content="Hello")])
assert response.content
The @pytest.mark.vcr marker tells pytest-recording to automatically:
- Look for a cassette file named
test_chat_openai_invoke.yaml.gzin the module'scassettes/directory - Use VCR to intercept HTTP calls
- Replay the cassette if it exists; record a new one if it doesn't (or if
--record-mode=new_episodesis passed)
Parameterized Integration Tests
When testing the same scenario against multiple models or configurations, use @pytest.mark.parametrize:
import pytest
from langchain_openai import ChatOpenAI
from langchain_core.messages import HumanMessage
MODELS = ["gpt-4o-mini", "gpt-4o"]
@pytest.mark.vcr
@pytest.mark.parametrize("model_name", MODELS)
def test_chat_models_invoke(model_name: str):
"""Test invoke on different models."""
chat = ChatOpenAI(model=model_name)
response = chat.invoke([HumanMessage(content="Hello")])
assert response.content
VCR automatically generates separate cassette files for each parameter combination:
cassettes/
test_chat_models_invoke[gpt-4o-mini].yaml.gz
test_chat_models_invoke[gpt-4o].yaml.gz
When the test runs with model_name="gpt-4o-mini", VCR loads the cassette matching [gpt-4o-mini]. This allows testing multiple models with independent recorded interactions.
Testing Streaming
Stream tests benefit from cassettes because they capture the full stream sequence:
@pytest.mark.vcr
def test_chat_openai_streaming():
"""Test streaming behavior."""
chat = ChatOpenAI(model="gpt-4o-mini")
chunks = []
for chunk in chat.stream("Hello"):
chunks.append(chunk)
assert len(chunks) > 0
assert chunks[-1].content
The cassette records all the streaming HTTP chunks, and VCR replays them in order during playback. Async tests work identically—VCR intercepts the underlying httpx/requests library for both sync and async.
Testing Error Paths
To test error handling (rate limits, malformed inputs, API errors), you can either:
- Record once with a real error (e.g., using an invalid API key or endpoint), then replay the error response
- Create a cassette manually with a canned error response for testing
Example: testing a RateLimitError:
@pytest.mark.vcr
def test_chat_openai_handles_rate_limit():
"""Test that rate limit errors are handled gracefully."""
chat = ChatOpenAI(model="gpt-4o-mini")
try:
response = chat.invoke("Hello")
except Exception as e:
assert "rate" in str(e).lower()
The cassette for this test should contain a recorded 429 (Too Many Requests) response from the API.
Cassette Management
Recording New Cassettes
To record a new cassette or re-record an existing one:
cd libs/partners/openai
# Record new cassettes (default --record-mode=once)
uv run --group test --group test_integration pytest \
tests/integration_tests/chat_models/test_base.py::test_chat_openai_invoke
# Or explicitly with --record-mode=new_episodes to overwrite
uv run --group test --group test_integration pytest \
--record-mode=new_episodes \
tests/integration_tests/chat_models/test_base.py::test_chat_openai_invoke
Prerequisites:
- A valid
OPENAI_API_KEYintests/.env - Network access
What happens:
- pytest-recording starts VCR in record mode
- Your test runs and makes real HTTP calls to the API
- VCR intercepts all requests and responses
- Scrubbing functions redact sensitive headers and secrets
- The cassette is serialized to YAML, compressed to
.yaml.gz, and saved
Commit the cassette:
git add libs/partners/openai/tests/cassettes/test_chat_openai_invoke.yaml.gz
git commit -m "Add cassette for chat_openai_invoke integration test"
Refreshing Stale Cassettes
When test code changes (e.g., you change the prompt, the model, or the parameters), the cassette may become stale. VCR will attempt to match the new request body against the old cassette; if there's a mismatch, it will fail with a CannotOverwriteExistingCassetteException error (in playback mode).
To refresh:
cd libs/partners/openai
uv run --group test --group test_integration pytest \
--record-mode=new_episodes \
tests/integration_tests/chat_models/test_base.py::test_chat_openai_invoke
This overwrites the cassette with the new interaction. Re-verify the cassette was scrubbed correctly, then commit.
Inspecting Cassettes
To inspect a cassette, decompress and read:
gunzip -c libs/partners/openai/tests/cassettes/test_invoke.yaml.gz | head -50
Or, for a permanent view:
gunzip libs/partners/openai/tests/cassettes/test_invoke.yaml.gz
# Now test_invoke.yaml is readable
(Remember to re-compress or re-record before committing.)
Cassette Format
Cassettes are YAML files (optionally gzip-compressed). A typical cassette structure:
interactions:
- request:
body: null
headers: {}
method: POST
uri: https://api.openai.com/v1/chat/completions
response:
body:
string: '{"choices": [{"message": {"content": "Hello!", "role": "assistant"}}]}'
headers:
content-type:
- application/json
status:
code: 200
message: OK
version: 1
Headers and secrets are replaced with PLACEHOLDER or **REDACTED**:
request:
headers:
authorization:
- "**REDACTED**"
openai-organization:
- "PLACEHOLDER"
Running Integration Tests
Full Integration Test Suite (Live)
Run all integration tests against live APIs (requires credentials):
cd libs/partners/openai
make integration_tests # Runs full suite with live API calls
This invokes:
uv run --group test --group test_integration pytest -v --tb=short tests/integration_tests/
It also smoke-tests a single live request on each supported SDK major version (e.g., OpenAI SDK 2.x and 3.x).
VCR Cassette Tests (CI Mode)
Run integration tests using only cassettes (no live API calls, no credentials):
cd libs/partners/openai
make test_vcr
This invokes:
uv run --group test pytest --record-mode=none -m vcr tests/integration_tests/
The --record-mode=none flag tells VCR to only replay cassettes and fail if a cassette is missing (never attempt a live call or record a new one).
This is what CI runs; it ensures cassettes are up-to-date and tests are repeatable without exposing secrets.
Scheduled Integration Tests
Scheduled CI workflows (daily or on-demand) run live integration tests with real credentials:
cd libs/partners/openai
make integration_tests
Only tests marked @pytest.mark.scheduled run in these workflows. See CI/CD Workflows for details.
Standard Test Suite Inheritance
LangChain partner packages inherit a standard suite of integration tests via ChatModelIntegrationTests from langchain_tests.integration_tests. This allows all chat model providers to run a common set of tests without duplicating test code.
How It Works
Each partner package defines a test class that inherits from ChatModelIntegrationTests:
# libs/partners/openai/tests/integration_tests/chat_models/test_base_standard.py
from langchain_tests.integration_tests import ChatModelIntegrationTests
from langchain_openai import ChatOpenAI
class TestOpenAIStandard(ChatModelIntegrationTests):
@property
def chat_model_class(self) -> type[BaseChatModel]:
return ChatOpenAI
@property
def chat_model_params(self) -> dict:
return {"model": "gpt-4o-mini"}
@property
def supports_image_inputs(self) -> bool:
return True
@property
def has_tool_calling(self) -> bool:
return True
The base class (ChatModelIntegrationTests) implements dozens of test methods that exercise standard chat model capabilities: invoke, ainvoke, stream, astream, tool calling, structured output, image inputs, and more. Each test method is automatically marked with @pytest.mark.vcr so cassettes are recorded and replayed.
Required Properties
Partner implementations must override two properties:
chat_model_class: The model class to instantiate (e.g.,ChatOpenAI)chat_model_params: Initialization parameters for the model (e.g.,{"model": "gpt-4o-mini"})
Feature Declaration Properties
Partner implementations can declare model capabilities via properties. These control which tests run and how they are executed:
has_tool_calling: Whether the model supports tool/function callinghas_tool_choice: Whether the model supports forcing a tool call viatool_choiceparameterhas_structured_output: Whether the model supports structured outputsupports_image_inputs: Whether the model accepts image inputs (base64)supports_image_urls: Whether the model accepts image URLssupports_json_mode: Whether the model supportsmethod='json_mode'in structured outputsupports_pdf_inputs: Whether the model accepts PDF file inputsstructured_output_kwargs: Dict of additional kwargs forwith_structured_output()(e.g.,{"method": "json_schema"})
Customization: Overriding Test Methods
Partners can override individual test methods to provide provider-specific behavior:
class TestOpenAIStandard(ChatModelIntegrationTests):
# ... properties ...
def invoke_with_cache_read_input(self, *, stream: bool = False) -> AIMessage:
"""Override to test cache read with specific model features."""
llm = ChatOpenAI(model="gpt-4o-mini", stream_usage=True)
return llm.invoke("Cached prompt", ...)
VCR Integration with Standard Tests
Standard test methods inherit VCR integration automatically:
- Each test method is pre-marked with
@pytest.mark.vcr - Cassettes are generated automatically with names like
TestOpenAIStandard.test_invoke.yaml.gz,TestOpenAIStandard.test_invoke[model0].yaml.gz(for parameterized variants) - Tests run in CI via
make test_vcr --record-mode=noneto validate cassettes are up-to-date - When a partner updates a test (e.g., adds a new model to test), new cassettes are recorded with
--record-mode=new_episodes
This approach ensures that all LangChain partners maintain consistent test coverage and that changes to the standard test suite (new test methods, new capabilities) are automatically picked up by all partners.
VCR Configuration Details
conftest.py Setup
Each package's conftest.py configures VCR globally:
from langchain_tests.conftest import CustomPersister, CustomSerializer, base_vcr_config
@pytest.fixture(scope="session")
def vcr_config() -> dict:
"""Extend the default configuration coming from langchain_tests."""
config = base_vcr_config()
# Base config includes:
# - record_mode: "once"
# - filter_headers: ["authorization", "x-api-key", "api-key"]
# - match_on: ["method", "uri", "body"]
# - cassette_library_dir: "tests/cassettes"
# Extend for OpenAI:
config["match_on"] = ["json_body"] # Custom JSON matching
config["serializer"] = "yaml.gz" # Compress cassettes
config["before_record_request"] = remove_request_headers
config["before_record_response"] = remove_response_headers
return config
def pytest_recording_configure(config: dict, vcr: VCR) -> None:
"""Register custom serializer, persister, and matchers."""
vcr.register_persister(CustomPersister())
vcr.register_serializer("yaml.gz", CustomSerializer())
vcr.register_matcher("json_body", _json_body_matcher)
Record Modes
VCR supports several record modes:
once(default): Record new cassettes; replay existing ones. Use for initial development.new_episodes: Record/overwrite cassettes even if they exist. Use when refreshing stale cassettes.none: Never record; replay only. Use in CI. Fails if cassette is missing.all: Always record, even for existing cassettes. Use sparingly.
Request Matching
By default, VCR matches requests by:
match_on: ["method", "uri", "body"]
For APIs with non-deterministic bodies (e.g., timestamps in the request), you can customize matchers or use allow_playback_repeats to reuse the same response for multiple request variants.
OpenAI tests use a custom json_body matcher:
config["match_on"] = ["json_body"] # Only match on JSON-parsed body (ignores whitespace/key order)
This allows cassettes to match even if JSON key order differs. The matcher is registered in pytest_recording_configure:
def _json_body_matcher(r1: Any, r2: Any) -> None:
"""Match request bodies as parsed JSON, ignoring key order and tool description whitespace."""
# Parses both bodies as JSON and compares parsed trees
# Normalizes tool descriptions to handle Python version differences
# Fails gracefully for non-JSON bodies
CI/CD Integration
Workflow Structure
The LangChain CI system runs integration tests at multiple points:
- Pull request checks (
check_diffs.yml): Compiles integration tests without running them, then runsmake test_vcrfor cassette playback tests - Scheduled integration tests (
integration_tests.yml): Daily remote API testing with live credentials against partner libraries
Integration Test Compilation (_compile_integration_test.yml)
Validates that integration tests compile without syntax or import errors:
uv run pytest -m compile tests/integration_tests
This is a fast check that catches import errors and syntax issues before attempting to run tests. It runs in check_diffs.yml for every package with changed files.
VCR Cassette Tests (_test_vcr.yml)
Runs cassette-backed integration tests in playback-only mode:
uv run pytest --record-mode=none -m vcr tests/integration_tests/
Key features:
- No API keys needed (fake key like
sk-fakeis set) - Cassettes are never recorded or overwritten in CI
- If a cassette is missing, the test fails immediately
- If request bodies don't match, the test fails immediately
- Tests run in parallel for speed (
-n auto)
This catches stale cassettes caused by test code changes without attempting live calls.
Scheduled Integration Tests (integration_tests.yml)
Runs with make integration_tests daily or on-demand:
uv run --group test --group test_integration pytest -v --tb=short tests/integration_tests/
Key features:
- Runs with real API credentials from secrets
- Runs only tests marked
@pytest.mark.scheduled - Tests marked
@pytest.mark.vcrare skipped (use cassettes in PR checks instead) - Smoke-tests each supported SDK major version
Cassette Validation in CI
The test_vcr job runs with --record-mode=none, which means:
- Cassettes are never recorded or overwritten in CI
- If a cassette is missing, the test fails immediately
- If request bodies don't match, the test fails immediately
This catches stale cassettes caused by test code changes without attempting live calls.
Advanced Configuration
Custom VCR Matchers
For advanced matching logic (e.g., ignoring certain request fields), register a custom matcher in conftest.py:
def _custom_matcher(r1: Any, r2: Any) -> None:
"""Match requests with custom logic."""
# Example: match method and URI, ignore timestamps in body
assert r1.method == r2.method
assert r1.uri == r2.uri
# Don't check body
@pytest.fixture(scope="session")
def vcr_config() -> dict:
config = base_vcr_config()
config["match_on"] = ["custom"]
return config
def pytest_recording_configure(config: dict, vcr: VCR) -> None:
vcr.register_matcher("custom", _custom_matcher)
Custom Scrubbers
To add additional scrubbing for proprietary secrets:
def remove_proprietary_header(request: Any) -> Any:
"""Remove custom header."""
request.headers.pop("X-Proprietary-Token", None)
return request
@pytest.fixture(scope="session")
def vcr_config() -> dict:
config = base_vcr_config()
config["before_record_request"] = remove_proprietary_header
return config
Disabling Specific Tests in CI
Some providers (e.g., Codex, audio models) require special handling in CI. Use pytest_collection_modifyitems hooks to skip tests at collection time:
def pytest_collection_modifyitems(
config: pytest.Config, items: list[pytest.Item]
) -> None:
"""Skip Codex tests in CI unless they are replaying VCR cassettes."""
if not os.getenv("CI") or _vcr_record_mode(config) == "none":
return
# Skip Codex tests unless in playback mode
And use autouse fixtures to stub dependencies:
@pytest.fixture(autouse=True)
def _fake_codex_oauth_token(
request: pytest.FixtureRequest, monkeypatch: pytest.MonkeyPatch
) -> None:
"""Stub OAuth token reads for Codex VCR tests."""
if "codex" not in request.module.__name__:
return
# Monkeypatch OAuth token provider with in-memory fake token
Best Practices
Always scrub cassettes: Use
before_record_requestandbefore_record_responseto redact all sensitive data. Inspect cassettes before committing.Commit cassettes: Cassettes are part of your test suite. Commit them to git so others can run tests without live API access.
Keep cassettes small: Avoid recording huge responses (e.g., embeddings of long documents). If needed, use manual cassette creation with canned data.
Use markers correctly:
@pytest.mark.vcrfor cassette-backed tests (run in PR checks)@pytest.mark.scheduledfor live-only tests (run in scheduled workflows)- No marker for offline tests
Test against multiple models/versions: Use
@pytest.mark.parametrizeto test multiple configurations and generate separate cassettes for each.Document cassette dependencies: If a cassette depends on specific model behavior (e.g., a reasoning model), document it in the test.
Refresh cassettes when code changes: If you modify the test (new prompt, new parameters), refresh the cassette with
--record-mode=new_episodes.Review cassette diffs: When committing cassette updates, review the diff in your PR to ensure no secrets leaked.
Inherit standard tests: Use
ChatModelIntegrationTestsand similar base classes fromlangchain_tests.integration_teststo ensure consistent coverage across providers.Override properties, not methods: Prefer declaring capabilities via properties (
supports_*,has_*) rather than overriding test methods, unless provider-specific logic is needed.