Skip to content

Self-hosted AI workspace.

Python
12.1 KB
Raw Blame

Odysseus Testing Standard & Taxonomy

Purpose

This document defines how we write and refactor tests in Odysseus. It is the standard that the incremental test-suite refactor (issue #2523) works toward, and it applies to both human contributors and coding agents.

It is intentionally split from tests/README.md:

  • README.md - the concrete, current helper reference: what each helper in tests/helpers/ does and how to call it.
  • TESTING_STANDARD.md (this file) - the rules and taxonomy: what a good test looks like, where it belongs, and the policy refactor PRs must follow.

When the two ever disagree, this file states the intent and README.md states the current mechanics; fix whichever is stale.

This document changes no test behavior. It is guidance only.

What the test suite is for

The goal is not only to reorganize tests/. The goal is for the suite to be a reliable foundation for future development: deterministic, modular, informative, behavior-focused, and complete enough to replace manual QA wherever practical.

Run tests with the project virtualenv interpreter (./venv/bin/python -m pytest). The system python3 may be missing pinned dependencies (e.g. nh3), which shows up as import/collection errors that are environmental, not real failures.

What "done" means for a single test

Every new or refactored test should be:

  • Deterministic - same result every run, no reliance on wall-clock, network, RNG seeds, or collection order.
  • Behavior-first - asserts on observable behavior, not on the source text or AST of the code under test (see Behavioral-first policy).
  • Explicit - setup and expected result are visible in the test, not hidden in broad fixtures.
  • Isolated from global process state - no leaked sys.modules, os.environ, CWD, or package parent-attribute mutation (see Determinism & isolation).
  • Order-independent - passes regardless of which tests ran before it.
  • Environment-independent - does not assume a venv layout, a developer's home directory, an existing ./data dir, or optional packages that may be absent.
  • Informative on failure - the assertion message or structure makes the cause obvious without a debugger.
  • Small - understandable quickly; one behavior per test where practical.
  • Backed by shared helpers only when duplication is proven - not abstracted preemptively.

Test taxonomy

Tests are classified by the categories below. Today the suite is mostly flat under tests/ (the current area_cli set has moved to tests/cli/); the Target dir column is the phased layout from #2523 that we move toward after helpers and determinism are stable. Until a category is moved, new tests in that category stay in flat tests/ but should still follow this standard.

Category What it covers Examples today Target dir
Route / API integration Real ASGI request/response, auth gates, admin gates, owner isolation through the app files using TestClient tests/routes/
CLI / script scripts/ entry points and dev tooling tests.helpers.cli_loader.load_script users, test_pr_blocker_audit.py tests/cli/
Frontend / JS Browser-coupled JS run via Node subprocess; streaming-render invariants *_js.py wrappers, tests/streaming/*.test.mjs tests/js/
Tool execution / parsing Tool-call parsing, malformed/nonstring args, tool policy test_unknown_tool_calls.py, test_tool_policy.py, *_nonstring.py tests/unit/ or tests/services/
LLM / provider Provider response parsing, streaming, sanitize, reasoning fallback test_llm_core_*, test_anthropic_response_parse.py tests/services/
Session / history / DB Session lifecycle, history, schema, ownership at the data layer test_session_*, test_sqlite_foreign_keys.py tests/services/ or tests/unit/
Security / owner-scope / regression Owner isolation, auth, SSRF, path confinement, XSS, prompt injection, pinned regressions *_owner_scope.py, test_security_regressions.py, test_*ssrf*, test_*confinement* tests/security/
Cookbook / bootstrap Model serve lifecycle, dependency completion test_cookbook_* tests/services/
Scheduler / background Cron computation, background jobs, delivery test_compute_next_run_*, test_bg_*, test_task_scheduler_* tests/services/
Import / module isolation The isolation helpers themselves and their guarantees test_helpers_import_state.py tests/unit/

A test that genuinely spans categories (e.g. a route test that also pins a security invariant) is classified by its primary assertion target and may be split if it grows.

Fast lane policy

The fast lane is not slow: tests/run_focus.py --fast selects every test that is not marked slow. The slow marker is opt-in, and slow marks must be evidence-driven from --durations output - mark a test slow only when its measured duration shows it is genuinely expensive, never by guessing. The fast lane exists for quick local and reviewer feedback; it is not a replacement for broader focused or full-suite validation before merge, and a test must never be marked slow to hide a failure or skip coverage.

Determinism & isolation rules

Do not mutate shared process state without a controlled helper and guaranteed cleanup. Specifically:

  • sys.modules / parent-package attributes - never assign at module scope. Use tests.helpers.import_state.preserve_import_state, clear_module, or monkeypatch.setitem(sys.modules, ...). Restoring sys.modules alone is not enough; the parent-package attribute must be restored too (the import-state helpers handle both).
  • os.environ - use monkeypatch.setenv / monkeypatch.delenv, never raw os.environ[...] = ... that outlives the test.
  • Current working directory - never chdir without restoring; never assert against cwd-relative paths like ./data. Use a temp workspace helper instead.
  • Database - the root conftest.py defaults DATABASE_URL to an in-memory SQLite for collection safety. A test that needs a real file-backed DB must opt in explicitly via tests.helpers.sqlite_db.make_temp_sqlite and bind its SessionLocal onto the module under test. Do not rely on a persistent on-disk DB existing.
  • Optional dependencies - do not require packages that may be absent in a clean environment (e.g. python-multipart). Guard or stub them locally.
  • Node-subprocess JS tests - skip cleanly when node is absent (shutil.which("node")), matching the existing wrappers. Treat a skip as a coverage gap to be aware of, not a pass.
  • Order independence - a test must not depend on a sibling having imported, cached, or stubbed something first. Order-sensitivity is a bug to fix, not a constraint to encode.

Behavioral-first policy

Prefer tests that exercise real behavior over tests that inspect source code.

  • Avoid read_text() + substring assertions, ast.parse, and inspect.getsource checks when the behavior can be driven directly. Source-text assertions break on benign refactors (renames, reformatting) and can pass even when behavior regresses, because the asserted string still appears somewhere.
  • Prefer calling the function/route and asserting the outcome. Example: to pin owner-scoping of get_upcoming_events, seed a temp DB with two owners and assert one owner cannot see the other's events - rather than asserting the source contains q.filter(CalendarCal.owner == owner).
  • Narrow exception - a source-text/AST assertion is acceptable only when the invariant cannot be practically exercised at runtime (e.g. pinning that a required constant or guard literally exists in a module that is hard to drive). When used, say why in the test docstring so it is a deliberate choice, not a shortcut.
  • Do not convert source-text assertions to behavioral ones in the same PR that moves files or changes unrelated setup.

Helper & factory extraction rules

  • Extract a shared helper only when the duplicated shape is proven - the same setup repeated (ideally byte-identical) across multiple files.
  • Prefer plain functions in tests/helpers/ over fixtures. Reach for a fixture only when it is clearly scoped to one directory/category, and put it in that directory's conftest.py, not the root.
  • Keep the root conftest.py minimal - sys.path, the DB-URL default, and not-installed heavy-dependency stubs only. It is not a place for feature-specific fixtures.
  • Each helper documents its intended use and its limits ("do not stretch this to cover X"), as the existing helpers in README.md do.
  • Do not build a generic abstraction layer (factory framework, broad base fixtures) before the repeated semantics are clear. Small and boring beats clever and general.
  • Candidate factories, to add only after the duplication audit confirms the shapes: fake users, fake sessions, fake requests, fake DB rows, fake LLM responses, fake tool calls.

PR discipline for #2523 refactor slices

  • Keep each PR small, reviewable, and behavior-preserving - unless the PR's stated purpose is to add new coverage.
  • One kind of change per PR. Do not mix:
    • file moves with assertion changes;
    • helper extraction with logic changes;
    • import-state cleanup with DB-fixture changes.
  • Do not weaken assertions, add skip/xfail, or delete coverage just to make CI green. A red test is a signal to investigate, not to silence.
  • Prefer 3-6 files per refactor batch, and only when they share the same pattern.
  • Distinguish a stale test expectation from a real production-policy change before "fixing" a failing test - never edit a test to match a regression.

Validation expectations

Run locally before opening or approving a refactor PR:

  • git diff --check - whitespace and conflict-marker errors.
  • ./venv/bin/python -m py_compile <changed .py files> - changed files compile.
  • Focused ./venv/bin/python -m pytest on the changed files.
  • ./venv/bin/python -m pytest on neighboring / order-sensitive groups that share import state with the changed files.
  • When replacing boilerplate, grep for the old pattern to confirm no stragglers.
  • When changing a helper itself, validate in a fresh worktree so stale __pycache__ or import state cannot mask a regression.
  • For order-sensitivity, a randomized run (once pytest-randomly is available in the dev environment) is the strongest check; record the seed on failures.

Target directory structure (phased)

Move toward this layout gradually, only after helper conventions and determinism are stable. Low-risk categories move first; oversized catch-all files are split last.

tests/
  conftest.py        # stays minimal
  README.md          # helper reference
  TESTING_STANDARD.md
  helpers/           # plain helper functions (exists)
  unit/              # pure helper/module tests
  cli/               # scripts/ + CLI tests
  js/                # node-subprocess + streaming tests
  security/          # owner-scope, auth, SSRF, confinement, regressions
  routes/            # TestClient integration (per-dir conftest for the client)
  services/          # service-layer tests
  integration/       # only if a cross-cutting flow needs it, later

Suggested move order: js / cli first → security / routes / services → split oversized catch-all files last. Each move is mechanical (no assertion changes in the same PR), with an identical pass set before and after.

Related: CI-hardening track (tracked separately)

Making the suite an enforced gate is broader than #2523's organization scope and should be tracked as its own effort. The intended sequence:

  1. Add non-blocking randomized pytest reporting (pytest-randomly) so hidden order-dependence becomes visible without changing any test.
  2. Fix surfaced order-dependence in small same-pattern batches.
  3. Add coverage reporting with no threshold gate.
  4. Only then make the pytest job a blocking CI gate.
  5. Consider pytest-xdist / parallel isolation after deterministic single-process randomized runs are stable.