Make your computer a home for your AI partner. Avibe is the local-first Agent OS: Vibey, its built-in AI partner, works out of the box, and Claude Code, Codex & OpenCode run unchanged beside it — reach them from your browser, phone, or any chat app.
Model Hub E2E Test Plan
Status: routing-mode acceptance revision, contract_version 10, 2026-09-06.
Contract: model-hub-routing-modes.md (original 2db273891, owner scope/synchronization
decision c1d398d5f, Empty Route Inheritance 352486374). The older survey
below remains background for unrelated capabilities; routing expectations follow this revision.
Companion inventories (kept in session, summarized here): capability list C1–C81 and issue list B1–B17 produced during the 2026-09-01 survey.
1. Goals and ground rules
- Exercise the shipped surface end to end over real HTTP: browser-visible API (
/api/models/*) and the per-turn gateway, against a mock upstream provider, never real vendor credentials. - Assert current behavior as baseline where the owner has ruled (cross-vendor fidelity ruling 2026-08-08; "user owns the order" contract in
docs/plans/model-hub.md§4.2), and mark scenarios fix-first or expected-fail where the survey found unintended drift (see §5 decision ledger). - Every scenario names the feature ids (C#) it covers and the evidence layer it lives in:
e2e-auto(this suite),unit/contract(existing),manual(Incus regression).
Out of scope (documented, not dropped silently):
- Real-vendor OAuth completion (Anthropic/OpenAI login dance) — needs real accounts; covered by
manualin Incus regression. Flow lifecycle (start/status/cancel/expire/nonce replay) is in scope via the hub OAuth endpoints up to the auth URL. - OpenCode provenance (by design never written;
model-hub-l3-correlation.md). - Protocol-conversion fidelity beyond the functional floor (owner ruling: converted supply is functionally usable while its reasoning chain may degrade — expected, not a defect). E2E asserts: turn completes, tool call ids correlate on the final turn, stream terminates with a well-formed terminal frame. It does not assert reasoning/system-prompt fidelity.
2. Environment and fixtures
- Runtime: hermetic
AVIBE_HOMEunder the test dir;VIBE_MODEL_HUB_ENABLED=1. Dockertests/e2eservice withcommand: ["full"](controller + UI in one process pair) or an equivalent local hermetic run; whichever the implementing lane proves stable first. - Engine binary: no GitHub egress in CI. Use
VIBE_MODEL_HUB_ENGINE_MANIFEST_PATH/VIBE_MODEL_HUB_ENGINE_OFFLINE(vibe/model_hub_runtime/installer.py:105-109) with the pinned cliproxyapi v7.2.95 assets vendored per platform (linux-amd64/arm64, darwin-arm64/x64). - Offline archive seeding: when the selected manifest asset uses
file://, the harness verifies its declared size and SHA-256, then copies it into the hermetic$AVIBE_HOME/runtime/model-hub/engine/downloads/cache before spawning child processes.VIBE_MODEL_HUB_ENGINE_OFFLINE=1remains set, and non-file assets are never fetched. - Mock upstream provider (new test fixture,
tests/e2e/drivers/mock_llm_upstream.py): implements, per protocol —POST /v1/messages,POST /v1/responses,POST /v1/chat/completions,GET /v1/models— with:- configurable model inventory (ids only, plus Anthropic-style
display_nameand relay-stylecontext_lengthextensions to prove they are dropped, B-list); - auth modes: accept / 401 / 403(banned-pattern) / 402 / 429 / quota-message / 5xx;
- stream modes: healthy SSE per protocol, and interrupt-after-first-model-output;
- request capture (last N bodies) so tests assert what the upstream received (effort stripping, model prefix, header survival).
- configurable model inventory (ids only, plus Anthropic-style
- Auth/roles: two API tokens (owner, member) for the permission matrix.
3. Suites
A. Gating and runtime lifecycle (C1,C2,C5,C9–C15)
| ID | Steps | Expect | Status |
|---|---|---|---|
| A1 | flag off | /api/models/* → 404 feature_disabled; UI redirects to /settings/backends |
assert |
| A2 | lifecycle API install → start → stop; UI recovery action | install/start recovery stays available without a user-facing on/off switch; API hardening and lifecycle checks remain unchanged | assert |
| A3 | agent in hub mode → lifecycle API stop | API retains 409 runtime_in_use with data.backends; the Models page exposes no stop action |
assert |
| A4 | agents read fails while runtime on | keep runtime status readable without introducing an on/off control | assert |
| A5 | controller restart mid OAuth poll | poll reports engine_down copy, never a fake materialization failure (B10) |
assert |
B. API-key sources (C16–C29)
| ID | Steps | Expect | Status |
|---|---|---|---|
| B1 | save custom/catalog providers under their selected/pinned protocol | save-first UI invokes no model and does not require authentication evidence; inventory discovery is best effort and pending verification remains (2026-09-08 owner decision) | assert |
| B2 | custom declaration differs from the upstream interface | save persists exactly the selected protocol, without an inferred protocol or verified claim; legacy observation API behavior is separate | assert |
| B3 | save, inspect automatically fetched inventory, choose a model and run Test | preserve upstream inventory order and metadata ownership; preferred exact ID wins, otherwise first model; invoke only the chosen Source/model with no route fallback and clear only its matching credential marker | assert |
| B4 | discovery fails | save succeeds with pending verification; preserve manual inventory and allow manual addition without refetching | assert |
| B5 | replay create with same client_nonce |
idempotent, no duplicate source | assert |
| B6 | replace key happy + rollback; rename; patch base URL | C22/C23 contracts | assert |
| B7 | sole owned default source + exact one-hop manual route, verified by reads → delete source → guard 409 → echo plan → force | actual inherited supply gap; the one delete confirm is the whole decision, so the 409 is resent forced with its plan echoed without reopening the dialog, and the commit announces itself in one toast; malformed echo rejected; canonical route and exact captured default order restored independently (only the owned deleted ID may be omitted) | assert |
| B8 | force transport asymmetry (?force= vs body) |
document current split (B6 issue); decide normalization | baseline |
| B9 | refetch after upstream inventory change | added/removed diff; discovered_at preserved for pre-existing models (currently overwritten — fix-first, see B-list) |
fix-first |
| B10 | inspect provider inventory and add/remove a manual model | no reasoning column, provenance badge, tier editor or empty advanced toggle; manual creation needs only a model ID. Compatibility API tests retain existing metadata ownership coverage | assert |
| B11 | trigger each of: mapping_target_unavailable, runtime_in_use, source_nonce_conflict, reauth_confirmation_required, turn_not_found |
UI renders human copy, never the raw modelHub.errors.* string (B1 — fix-first; baseline expected-fail) |
fix-first |
C. OAuth lifecycle (C30–C40, partial by §1)
| ID | Steps | Expect | Status |
|---|---|---|---|
| C1 | start hub OAuth (anthropic/openai) | auth-url presentation per vendor; status polling; cancel | assert |
| C2 | flow expiry + nonce replay | flow_expired; replayed nonce resumes, no duplicate flow |
assert |
| C3 | native channel start-failure copy | native_subscription_exists, native_login_in_progress |
assert |
| C4 | full login → materialize → adoption note | manual (Incus, real accounts) | manual |
D. Per-turn routing, fallback, capability behavior (C41–C50, C74–C81)
| ID | Steps | Expect | Status |
|---|---|---|---|
| D1 | add api_key source first, subscription-class source second | the subscription is placed ahead of the key (decision D-1, #2156); a contract test owns it, since adding a subscription hermetically is impossible (a takeover validates its grant with the provider) | contract |
| D2 | turn with hop0 healthy | served; provenance records hop; usage metered | assert |
| D3 | hop0 → 429 before first output | same-turn failover to hop1; switch event; hop0 cools for the later of its initial local backoff (60–72s) and a valid Retry-After that survives the managed engine (D-4); chat copy unchanged (silent) |
assert |
| D4 | hop0 → quota (insufficient_quota) |
fallback, cooldown 300s | assert |
| D5 | hop0 → 401 (api_key, non-refreshable) | credential_revoked needs_action; UI shows repair action; no mid-turn refresh attempt |
assert |
| D6 | hop0 → 5xx | source backoff with positive bounded jitter; later valid Retry-After survives the managed engine and takes precedence | assert |
| D7 | stream interrupt after first output | terminal frame injected per protocol; no replay/failover; source still settles; next turn resolves hop1 (takeover pill visible) | assert |
| D8 | cooldown retry_at elapses |
hop0 becomes eligible; only recognized output or valid empty completion clears its state and emits recover |
unit/loopback covered; local Incus outstanding |
| D9 | all temporary hops keep failing | one closed recovery-exhausted result (Codex 400, Claude/OpenCode 424); localized final copy and original upstream provenance | gateway/scenario covered; local Incus outstanding |
| D10 | chat selects effort high; fallback hop declares tiers [] |
each exact adapter attempt retains the requested effort; turn provenance identifies the serving hop without inventing stripping. This fixture covers the Avibe gateway/adapter boundary, not the proxy engine translator | assert |
| D11 | POST /{backend}/v1/messages/count_tokens |
currently 404 not_found_error (D-2 decision: fix or document impact on Claude Code auto-compact) |
baseline |
| D12 | env/catalog injection for claude/codex/opencode | ANTHROPIC_BASE_URL/TOKEN only for claude; codex catalog neutralized 4 keys; opencode overlay model projection shape |
assert |
| D13 | member role matrix | member can PUT chains/PATCH mode; cannot create/delete source, start runtime, read usage (B15) — assert the dead-ends are at least visible errors, not silent UI | assert |
E. Usage and events (C54–C60)
| ID | Steps | Expect | Status |
|---|---|---|---|
| E1 | run metered turns (mock emits usage incl. cache tokens) | totals/by-source/by-model/by-day consistent; token_reports shortfall honest |
assert |
| E2 | ?days= boundary: 1, 62, garbage, 100000 |
1/62 ok; garbage clamps; 100000 must not pass unbounded (B14) | fix-first |
| E3 | event feed pagination + redaction | no credential substring anywhere in events payload | assert |
F. Migration (C61–C67)
| ID | Steps | Expect | Status |
|---|---|---|---|
| F1 | seed hermetic HOME with native claude/codex/opencode configs; scan | complete Claude API-key custody is importable; access-only Codex OAuth and unsupported OpenCode credentials remain visible blockers; no credential material is returned | assert |
| F2 | apply the selected complete backend group | Claude API-key custody is committed and its direct native auth is cleaned; blocked Codex/OpenCode material and unrelated Claude settings remain unchanged | assert |
| F3 | first open /settings/models after upgrade with importable items |
banner visible (B2 — currently only in wizard: expected-fail until mounted) | fix-first |
| F4 | already-Hub agents with populated runtime auth; import saved CLI/shell files, then repair damaged private custody | runtime values neither supply nor block import; dynamic writers refuse without cleanup; static Bearer survives migration and replacement through actual CPA even when the old private file is missing/corrupt; recorded/unknown OAuth history survives later batches and refuses speculative reprovision; public/wrong-key proof preserves files and unrelated bytes/modes | assert |
G. Guards and contract hygiene (C51–C53, B3,B7,B13,B16)
| ID | Steps | Expect | Status |
|---|---|---|---|
| G1 | guard echo with a gap missing agents |
no confirm-loop (B7) — fix-first; baseline documents | fix-first |
| G2 | Sources PUT and compatibility chains/reorder change defaults | same effective guard plan and canonical projection; manual arrays unchanged; omitted reorder order is read-only. Default-routing UI Save automatically echoes one exact refusal plan without a second confirmation; changed-plan refusal keeps the draft, and lost responses reconcile before any new write | assert |
| G3 | malformed JSON on mode/chain routes | error code distinguishable from business refusals (B13) | fix-first |
H. Cross-protocol functional floor (owner ruling baseline)
| ID | Steps | Expect | Status |
|---|---|---|---|
| H1 | claude-protocol turn served by openai_chat source (mock) | completes; tool call↔result tuple correlates on final turn | assert |
| H2 | same with parallel tools | passes (M0 matrix failed 5/8 — if still failing, recorded as known relay behavior per ruling, surfaced in release notes, not a gate) | baseline |
| H3 | image block passthrough | behavior documented (currently untested anywhere); at minimum: no crash, terminal frame well-formed | probe |
I. Routing modes and compatibility
| ID | Property | Evidence |
|---|---|---|
| RM-1 | Nonempty manual intent survives unrelated config/catalog/health/process transitions; equal-to-automatic nonempty saves remain manual, while absent and valid empty values inherit across all backends | Seed every supported saved shape, mutate defaults/inventory/catalog, reload and compare exact nonempty/stale/dormant arrays, catalog identities and unrelated config; canonical output maps omit empty values and manual_override is nonempty or null |
| RM-2 | Automatic matching precedes passthrough, and live health/transport restrictions never choose another tier | All backends with API-key unknown passthrough only, unmatched subscription-only defaults Unconfigured, known-model subscription admission and stale-hop retention, native Claude aliases/date/version rules, complete non-retired evidence, literal API matching, native-only matching with Hub-only invocation, tombstones and empty inventories |
| RM-3 | One planning input produces identical effective hops/origin in preview, read/list, summaries, adoption, guards, probe, launch and execution | Real API/controller boundaries, assert canonical model ids and Source ids; unknown inventory carries no invented reasoning tiers |
| RM-4 | Restore/Undo/Cancel/Save preserve normalized draft and persistence truth, including final-hop removal | Browser GET -> edit -> remove final hop or Restore -> preview null -> Undo to prior unsaved draft or Cancel -> reload; inherited Save uses DELETE, nonempty manual Save uses PUT even when equal to automatic. Cover actual inherited origin/targets or Unconfigured, no empty Manual save, failed saves, stale preview, confirmation, Done and duplicate submission |
| RM-5 | Effective removal wire confirmation is exact and atomic for defaults/Restore/inventory/catalog/Source writes | Refuse, alter state, retry stale plan, confirm current plan; no partial writes. Default-routing Save performs one automatic exact-plan confirmation with no second user action, stops on a changed-plan refusal and reconciles lost responses before replay; Restore/inventory/catalog/Source UI confirmations remain. Empty PUT and DELETE have identical effective-removal/supply guards, idempotency, admission/lease and rollback behavior; only nonempty manual PUT retains the visible-edit interruption-only guard. A cascade removing the final hop drops its override before recomputing inherited impact; pure reorders preserve nonempty manual arrays and need no confirmation |
| RM-6 | Pinned CPA reaches the selected upstream with the exact canonical unknown model, without fake inventory or cross-Source fallback | Real backend launch -> correlation/gateway -> adapter -> pinned CPA -> local upstream; Hub API-key credentials on all API protocols and admitted non-ASCII ids/payloads; no subscription expansion |
| RM-7 | Transport registration remains separate from inventory and capabilities | Compare truthful SourceBinding.model_ids/Source.models with deterministic route_model_ids and compiler union; load absent field as empty; unchanged target sets do not restart; registration changes drain active transport leases before supervised restart, block new invocations behind the barrier and release leases on completion/close/cancellation independently of service settlement; prove sync cancellation/rollback and concurrent Save without deadlock; Hub OAuth registration remains unchanged |
| RM-8 | Request-only unknown-model errors preserve Source health and existing failure/stream boundaries | Typed upstream model_not_found, exact Source/model attribution, no fallback or global health transition |
| RM-9 | Released config and turn records remain readable; normalization is not a validation bypass | Seed supported valid empty/populated/stale/dormant routes on every backend and direct-constructed config: validate original shapes first, normalize valid empty values without losing nonempty routes, catalog identities or unrelated sections, and persist the normalized map only on normal save. Reject malformed wrappers/hops/identifiers and dangling references before normalization. Output manual arrays require minItems 1, but accepted empty PUT/preview and legacy configuration do not. Read TurnProvenance versions 5 through 10 without requiring new optional metadata on old records |
| RM-10 | Approved help, labels and layout work on desktop/mobile in both themes | Screenshots against approved routing frames; hover/focus/touch help, Escape/outside dismissal, no row-open from badge, no overlapping labels/buttons |
| RM-11 | Read and preview are operationally pure, including normalization of valid legacy empties | With runtime stopped assert unchanged config bytes/events/engine lifecycle, no credential access or upstream egress, for nonempty manual, accepted empty and null restored drafts. Empty and null previews return the same inherited chain/origin and manual_override null without changing stored intent |
| RM-12 | Recorded error detail uses the latest retained exactly attributed turn, never a synthetic last-request history | Exercise backend/model provenance GET against the existing bounded store: exact backend/model isolation, latest persisted ordering independent of outcome, later success/cancel clearing, empty/evicted/ambiguous history, old records without optional metadata and deleted historical Sources. Capture real model_not_found with observed HTTP status; reject unsafe upstream text and invented codes. Prove retrieval never starts/syncs the engine or writes config/history/events. Browser evidence covers the Latest recorded turn label, same-record structured details, generic old-record copy, retry, pending-read invalidation on close/model change, unchanged route origin and no history writes from Cancel/Restore/Save |
| RM-13 | Every actual invocation atomically admits a current effective candidate before transport, without holding the global mutation lock during network | Deterministically pause resolution across target removal/change and Source configuration changes; exercise ordinary/fallback, bounded credential-refresh retry and probe through the same admission path. Preserve failed-Source exclusions, reject stale exact refresh pairs without fabricated attempts or synthetic upstream errors, and verify service-mutation then adapter-routing lock order. Pre-admission failure/cancellation releases exclusion; callback failure releases the lease before network; post-admission completion/cancel releases it. Concurrent independent HTTP requests progress during buffered/first-token/stream waits, with no per-request registration or persistent generation store |
All fixtures use test-owned HOME/XDG/config/credentials and local upstreams. Never use personal state. Development acceptance uses coordinated local Incus only, preserving master state. Mocked peer suites do not replace the pinned-engine boundary proof.
Implemented unit/contract evidence for the empty-inheritance boundary supplements the end-to-end properties above; these are actual consuming tests, not proposed cases:
| Property | Existing consuming test |
|---|---|
| G2/RM-5: one-click defaults, bounded exact-plan handshake and lost-response reconciliation | ui/src/components/settings/models/SourceOrderDrawer.test.tsx |
| RM-1/RM-3/RM-11: absent/empty parity across consumers | tests/test_model_hub_routing_modes.py::test_empty_override_equals_absence_at_all_route_consumers |
| RM-4/RM-5: empty PUT equals guarded Restore; final-hop deletion inherits | tests/test_model_hub_routing_modes.py::test_empty_put_has_exact_restore_guard_and_final_hop_deletion_inherits |
| RM-9: original invalid input never becomes automatic | tests/test_model_hub_routing_modes.py::test_invalid_empty_intent_is_validated_before_normalization |
| RM-5: inventory final-hop changes use inherited effective impact | tests/test_model_hub_routing_modes.py::test_final_inventory_hop_removal_uses_inherited_guard_plan |
| RM-5/RM-7: failed sync rolls back canonical intent and permits retry | tests/test_model_hub_routing_modes.py::test_failed_route_sync_rolls_back_canonical_intent_and_retry |
| RM-1/RM-9: dormant/direct empty keys do not fabricate routes or guards | tests/test_model_hub_routing_modes.py::test_direct_dormant_empty_is_absent_from_route_and_guard_enumeration |
| RM-9/RM-11: load is read-only; normal save persists normalization | tests/test_model_hub_config.py::test_v2_empty_route_normalizes_on_load_without_writing_until_save |
The Python MH-ROUTING scenarios in
tests/scenarios/model_hub/test_model_hub_configured_route_scenarios.py remain
canonical catalog evidence. ui/e2e/model-routing-origin.spec.ts adds Playwright
evidence for RM-4: Restore, Undo and Cancel send no mutation; Save uses DELETE,
handles confirmation and retains automatic intent after reload. Its test label
does not create a catalog citation: the current browser citation resolver is
Vitest-only. No new catalog engine or browser INDEX row is required. Run this
supplement separately against the coordinated hermetic gateway fixture.
4. Mapping to existing coverage
Unit/contract suites already cover: config validation, guard structure, classification tables, oauth registry, usage ledger, runtime injection. E2E must not duplicate them; it owns the seams those can't see: double-observe cost, cooldown-vs-next-turn timing, UI copy on live errors, guard echo round-trip, engine config conservation, member-role dead-ends.
5. Owner decision ledger
- D-1 subscription-first placement (decided, #2156). A subscription is placed before API keys at placement time. Affects D1.
- D-2
count_tokens404. Serve it (proxy to upstream tokenizers or estimate) or document Claude Code context-estimation degradation in hub mode? Affects D11. - D-3 missing
modelHub.*browser i18n keys + absent Python-bundle codes. Fix now (makes B11 pass) or baseline as expected-fail for this round? - D-4 Retry-After honoring (approved 2026-09-09). Use the later of reason-specific backoff and valid upstream advice measured at header receipt. D6 asserts the real managed-engine forwarding boundary. See bounded quiet recovery.
- D-5 tier provenance ladder (approved 2026-09-03). Apply the first available
upstream > catalog > user > nullrung; managed tiers are read-only, a refresh override is observable, and silent stripping is attributed to the exact attempt.
5a. Cross-lane contracts (frozen 2026-09-01)
Two implementation lanes build the suites in parallel. These contracts are the interface between them; deviations route through the orchestrator, never lane-to-lane.
Mock upstream provider (owned by the pytest lane)
- Path:
tests/e2e/drivers/mock_llm_upstream.py. Stdlib-only (http.server + threading or asyncio): it must run standalone on a bare Python 3.11+ with no repo install. - Runs standalone:
python3 tests/e2e/drivers/mock_llm_upstream.py --port N [--host 127.0.0.1]; also importable as a pytest fixture intests/e2e/. - Serving surface (per configured protocol correctness):
GET /v1/models— configurable inventory; supports Anthropic-style (data[].id,display_name,type) and OpenAI-style (data[].id,created,owned_by) plus relay extension fields (context_length,pricing) so drop-behavior can be asserted.POST /v1/messages|/v1/responses|/v1/chat/completions— protocol- correct non-stream and SSE-stream responses, including the shape evidence auto-detect needs (type: "message"/object: "response"/object: "chat.completion*", and family-distinctiveerror.parambodies for invalid probes).
- Control plane (all under
/__control/, never proxied):POST /__control/config— set behavior JSON:{auth: ok|401|403_banned|402| 429|quota_message|5xx, stream: healthy|interrupt_after_first_output, models: [...], protocol: anthropic|openai_responses|openai_chat, models_endpoint: ok|http_404|http_500|timeout|malformed_json}.models_endpoint(defaultok) applies ONLY toGET /v1/models; inference endpoints are unaffected. Evaluation order per request:authapplies first to every endpoint (including/v1/models); only whenauth: okdoesmodels_endpointapply. An emptymodelsarray withauth: okandmodels_endpoint: okis a SUCCESSFUL empty inventory, distinct from discovery failure. Scenario B4 (protocol proven, discovery failed → 422inventory_unavailable→accept_unavailable_ inventorycommit path) is produced byauth: ok+models_endpoint: http_500(orhttp_404/timeout/malformed_json). (Contract amendment 2026-09-02, orchestrator ruling on the pytest lane's B4 blocker.)GET /__control/requests— returns an OBJECT{"requests": [...]}(not a bare array; extensible envelope), each record carrying method, path, headers subset, parsed body;DELETE /__control/requestsresets. (Shape frozen 2026-09-02.)
- Credential-witness fixture amendment (2026-09-20):
required_api_key: <synthetic non-empty string>|nullis opt-in and defaults tonull, preserving the public catalogue behavior. When set, the mock requires that exactx-api-keyfor Anthropic orAuthorization: Bearer ...for OpenAI on all data endpoints, before applying the configuredauth, protocol, inventory, or stream behavior. Missing/wrong credentials return 401;/__control/remains independently accessible. Correct credentials do not turn model-less schema errors into successful inference responses. F2 migration success explicitly opts into this gate, proving both the credentialed catalogue success and unauthenticated refusal over HTTP/IPC. Public-catalogue and wrong-key controls must refuse migration and preserve every seeded native file and the empty Source list. This is a fixture-only contract: production authentication evidence and other suites' defaults are unchanged.required_auth_scheme: "protocol"|"bearer"defaults to"protocol"and leaves that behavior unchanged. Opting into"bearer"models a custom Anthropic relay accepting only the exact Authorization bearer value, not the same value sent asx-api-key. F2 uses a savedANTHROPIC_AUTH_TOKENand this gate, keeping its positive and six-file-preserving negative controls consistent with the pinned engine's custom-endpoint transport. The persisted-migration suite uses this to check that a savedANTHROPIC_AUTH_TOKENkeeps its authentication semantics through proof and actual CPA requests; missing/wrong credentials still return 401, and authenticated schema probes still return 400. Custom API-key-header settings/shell controls must instead show the explicit transport blocker before any upstream request, preserving files and Sources. - The Playwright lane consumes this ONLY over HTTP via env
VIBE_E2E_MOCK_UPSTREAM_URL; it never imports the Python module. Mock- dependent specs skip with a clear message when the env is absent.
Environment variables (shared vocabulary)
VIBE_E2E_BASE_URL— target Avibe UI origin (defaulthttp://127.0.0.1:5123).VIBE_E2E_MOCK_UPSTREAM_URL— running mock upstream origin.VIBE_MODEL_HUB_ENABLED— optional; Model Hub is enabled when absent. Harnesses may explicitly set1for isolation or0to exercise the disabled surface.VIBE_MODEL_HUB_ENGINE_MANIFEST_PATH— offline engine manifest override for hermetic engine install; engine-dependent pytest scenarios must skip (not fail) with a clear reason when neither this nor GitHub egress is available.
Suite conventions
- pytest: files
tests/e2e/test_model_hub_*.py, markere2e_model_hub, not part of pull-request CI..github/workflows/model-hub-e2e.ymlruns it nightly and on demand, and a stable release needs a green run at its SHA (docs/release.md). Scenario IDs from §3 (A1…H3) appear in test names or docstrings. - Playwright: lives in
ui/e2e/, configui/playwright.config.ts, dependency added explicitly toui/package.jsondevDependencies, scriptsnpm run e2e/npm run e2e:headed. Chromium only. Not wired into CI in this PR. - Baseline discipline: scenarios marked
baselinein §3 assert CURRENT behavior with a comment naming the open decision (D-1…D-4);fix-firstscenarios land asxfail(pytest) /test.fixme(Playwright) with the issue reference, never as red tests. - Discipline tightening (orchestrator ruling 2026-09-02, post circuit
breaker on PR #1812):
- Skip policy: only ENVIRONMENTAL preconditions may
skip(engine binary unavailable BEFORE an install attempt, mock upstream absent, credentials required but not provisioned). Once a precondition has passed, any subsequent product-side failure (installer error after manifest accepted, OAuth-start HTTP error after engine confirmed up, etc.) is a test FAILURE. Every skip site states the precondition it guards. - xfail granularity: an
xfail/fixmewraps ONLY the assertion known to be broken (e.g. the human-readable error copy). All currently enforceable assertions (structured API guards, state transitions, status codes) live in separate passing tests. A broad xfail over a mixed test is forbidden. - Skip vs gap (round 5):
skip= this layer CANNOT execute the scenario (state the blocking precondition and the owning layer);gap= executable but not yet implemented (actionable; must surface in coverage next_priority). Never useskipfor "not written yet." - Self-audit before push (round 5): before every push, the lane audits EACH spec invariant (skip policy, xfail granularity, layer honesty, ownership, catalog single resolver) across ALL call sites in the diff — invariants bind at their extensions, not at the site a review last named. The report lists the checklist outcome.
- Environment ownership invariant (orchestrator ruling 2026-09-02, round 4):
the harness owns every process it starts, by invariant, not by leak-list.
Cleanup must PROVE ABSENCE: after teardown, scan the process table for any
process whose argv references this test's unique hermetic marker (its
AVIBE_HOMEpath) and terminate it, including detached / separately- sessioned engine children whose controller leader already exited. Never trust the recorded leader alone. Scoped strictly to the unique marker — no broad host-process killing — on the SUPPORTED MATRIX (macOS + Linux); on other platforms the opt-in subprocess harness skips with a platform precondition. Regression tests must cover the detached-child escape. SOUND ABSENCE PROOF (round 7): "absence" means no RUNNING process (state != zombie) references the marker. Direct children are reaped via waitpid; detached PIDs are polled until gone or zombie (use the process-status API, e.g. psutil which the product already depends on — never argv-scan alone, and never kill(pid,0) as an existence proof). A zombie holds no resources and cannot be reaped by a non-parent: a persistent zombie after the poll window is a logged warning, not a failure. STOP CONDITION (refined 2026-09-02, round 7): its substance is a RUNNING product process (controller, UI, managed engine) holding resources after teardown on a supported platform, surviving a reviewed head. If that happens, DO NOT patch again — reduce the owned surface: keep the controller+UI pair under one setsid-owned process group with pgid kill + waitpid, move engine-lifecycle scenarios (A2/A5/D2–D6) to skip rows owned by a dedicated future suite. Evidence-soundness refinements (like the zombie corner) and test-owned mock resources are fixed normally and do not trigger it. - Catalog single resolver (round 4): scenario references (top-level tests
AND
partial_evidence.test, AND skip-row references) resolve through ONE resolver shared by the checker and the metadata; the checker may not hold a second notion of "what a scenario references." Skip rows without a top-level test remain valid but their refs still resolve. - Layer honesty: a scenario this layer cannot execute is a catalog
skiprow with a reason and a pointer to the layer that owns it (unit suite or the Playwright lane) — never anxfailand never an unconditional-fail sentinel (sentinels are forbidden: they neither detect a regression nor a fix). Partial evidence is fine when labeled partial: a passing test that covers a subset of a scenario must say so in its docstring, and the scenario's catalog row names the covered subset. - B6 rollback evidence boundary: the pending-revocation journal at
$AVIBE_HOME/state/model_hub_pending_revocations.jsoninside the test-owned hermetic home is the public evidence surface for rejected credential material. Tests may read it; they must NOT read engine-private credential blobs. If the product writes neither journal entry nor observable removal on failed replacement, B6 narrows to an xfail with a product-gap finding — never private-storage inspection.
- Skip policy: only ENVIRONMENTAL preconditions may
Playwright scope (this round)
Core flows against a live instance: capability gate redirect, runtime recovery entry point and the absence of a runtime off switch, add-API-key dialog full branches (auto-detect / manual protocol / pull models / empty-inventory add-anyway / replace key) against the mock upstream, guard refusal → confirm → committed toast (incl. B7 echo probe as fixme), source detail rename/refetch/tier edit, route chain dialog add/remove/keyboard reorder, global priority drawer, usage & logs tabs render with real metered data, error-copy assertions (fixme where copy is known-missing per B1/D-3), member-role dead-ends (second auth context).
6. Execution plan
- Lane 1 (fixtures): mock upstream driver + offline engine manifest packaging + full-mode e2e bootstrap. No product code changes.
- Lane 2 (scenarios A/B/E/F/G): API-level e2e against fixtures.
- Lane 3 (scenarios D/H): turn-level e2e incl. stream capture and capture-based upstream assertions.
- Manual C4 + native-channel D-variants folded into the next Incus regression run.
- Fix-first items ship as separate small PRs before their scenario is flipped from baseline to assert.
Each lane follows pr-delivery-loop; suites land under tests/e2e/test_model_hub_*.py reusing tests/e2e/conftest.py and drivers/.