:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
LLMA Acceptance Test Implementation Plan
Overview
Python-based acceptance tests for the AI observability capture pipeline, testing against an existing PostHog instance.
Test Execution
Local Development
- User starts PostHog stack manually (using docker-compose or other method)
- Set POSTHOG_TEST_BASE_URL environment variable to PostHog instance URL
- Run acceptance tests:
python run_tests.py
GitHub Actions
- Triggered on PRs affecting rust/capture/ or nodejs/
- Sets up PostHog stack in CI
- Runs acceptance tests against the stack
- Collects logs on failure
Implementation Steps
Commit 1: Basic acceptance test infrastructure ✓
- Create common/ingestion/acceptance_tests/ directory structure
- Add utils.py for service URL management
- Add requirements.txt with test dependencies
- Add run_tests.py test runner script
Commit 2: PostHog API client for test setup
- Add api_client.py with PostHogTestClient class
- Implement organization creation via API
- Implement project creation with API key retrieval
- Add project deletion for cleanup
- Add event querying through PostHog Query API
- Implement polling mechanism for event arrival
Commit 3: Basic event capture test
- Add test_basic_capture.py
- Test sending a regular PostHog event to /capture
- Poll Query API until event appears
- Verify event properties match what was sent
- Confirm end-to-end pipeline works
Commit 4: Test orchestration
- Add conftest.py with pytest fixtures
- Add per-test project isolation
- Add environment variable handling
- Handle cleanup of test data
Commit 5: GitHub Actions workflow
- Add .github/workflows/llma-acceptance-tests.yml
- Configure triggers for relevant paths
- Add PostHog stack startup steps
- Add test execution with pytest
- Implement log collection on failure
- Add cleanup steps
Commit 6: Documentation
- Add README.md with setup instructions
- Document local running procedures
- Document environment variables
- Add troubleshooting guide
- Create .env.example file
- Document how to extend the test suite
Usage
# Start PostHog manually (e.g., with docker-compose)
docker-compose -f docker-compose.dev-full.yml up -d
# Set environment variable
export POSTHOG_TEST_BASE_URL=http://localhost:8010
# Run tests
cd common/ingestion/acceptance_tests
python run_tests.py