Make your computer a home for your AI partner. Avibe is the local-first Agent OS: Vibey, its built-in AI partner, works out of the box, and Claude Code, Codex & OpenCode run unchanged beside it — reach them from your browser, phone, or any chat app.
Standard
1. Testing Model
The testing model is capability-first.
The development question is not:
- which function changed
- which module changed
- which backend changed
The development question is:
- which user-visible capability changed
- what scenarios define correctness for that capability
- which layers of evidence now need to be updated
2. Layers Of Evidence
Every capability should be validated through four distinct layers.
Unit
Purpose:
- local logic
- parsing
- heuristics
- invariant enforcement
Typical traits:
- fast
- deterministic
- narrow
- implementation-proximate
Contract
Purpose:
- validate the boundary between the system and a dependency or adapter
Typical examples:
- CLI wrapper request shape
- SDK control-channel payload shape
- HTTP auth endpoint contract
- platform adapter message schema
Scenario
Purpose:
- prove that a capability closes as a user journey inside the service boundary
Typical scope:
- start flow
- accept follow-up input
- simulate external completion
- verify terminal state
- assert user-visible outputs
Scenario tests should be:
- capability-shaped
- deterministic
- built on reusable harness primitives
Smoke / Manual
Purpose:
- real external systems
- real accounts
- real transports
- final integration confidence
These are still required when the project depends on OAuth pages, IM platforms, or vendor CLIs that cannot be fully simulated with high confidence.
3. Core Objects
Capability
A user-visible behavior the system owns.
Examples:
- IM-driven backend auth recovery
- upload and attach a local file to an IM conversation
- schedule a delayed follow-up and deliver it into the correct thread
Scenario
A stable, named flow that demonstrates one meaningful behavior of a capability.
Scenarios should have:
- stable ID
- short name
- actor
- trigger
- external conditions
- expected outputs
Harness
The reusable test apparatus that drives scenarios.
Harness primitives should be shared, not rewritten per scenario.
Examples:
- fake transport
- fake process runner
- fake OAuth callback source
- fake persistence store
- probes for sent messages, state transitions, refresh hooks
Treat fakes as assets with explicit maturity:
ideal fake- bootstraps expected behavior quickly
contract fake- models documented or wire-level dependency guarantees
observed fake- models verified runtime behavior discovered through reproductions, incidents, or manual regression
Recommended layering:
- generic core primitives
- event probe
- fake transport/client
- fake process / completion handle
- base scenario harness
- step runner
- shared expectation helpers
- capability-specific harness
- auth setup
- attachment delivery
- scheduled follow-up delivery
4. Scenario Catalog Rules
Each capability must maintain a scenario catalog.
Projects should also maintain a project-level capability index.
Recommended canonical files:
tests/scenarios/INDEX.yamltests/scenarios/<capability>/catalog.yamltests/scenarios/<capability>/observations.yamltests/scenarios/<capability>/test_*.py
Scenario IDs should be stable and human-usable:
AUTH-SETUP-001AUTH-SETUP-101ATTACHMENT-002
Recommended bands:
001-099happy paths100-199negative or validation paths200-299recovery or retry paths900+known historical regressions
Each scenario entry should describe:
GivenWhenThen- automation layer
- current test owner
Every scenario-backed automated test should expose the scenario ID in a searchable form, preferably:
- test docstring
- test metadata/decorator
- or a nearby scenario mapping constant
Every citation a row makes must name a case, and that name must resolve to
exactly one case the suite collects in the cited file — in whichever key the row
states its evidence, test and any layer-specific key alike. When several cases
prove a row's contract together, test lists each of them, and each entry
resolves to exactly one case on its own; a class or a file standing in for the
group is not a citation of any of them. Only test takes a list:
partial_evidence.test names one case. A citation that
names only a file is not evidence for the row that makes it: the file stays
collected for whatever else cites it, so deleting the one case that answered
this row leaves the row reading covered with nothing executable tied to it.
That is not hypothetical — it is how four dead citations survived a test-file
rewrite unnoticed for five days. A pointer key such as related_tests may name
bare files precisely because it claims no coverage; anything that does claim
coverage names its case. A pointer that does name a case must still name one
that exists, or a rename leaves it leading nowhere. A pointer may also name a
group: every case whose name contains the term it names, such as a test class
or, for a vitest file, an ID; or, for a pytest file, a scenario ID, which names
every case whose docstring carries it.
A citation resolves against the suite's own collection of the cited file —
pytest's for Python, vitest's for UI — so a case the runner would not collect,
such as one in a suppressed or deselected class, is not evidence. A pytest
evidence case must also name the citing row's ID in its docstring; a case that
proves several rows names each of them. Then a case written for something else
cannot be attached to a row unnoticed. Finally, the evidence of a covered row
must run: every case it cites passes in some CI job's run, or a workflow a
release requires runs it. A case skipped wherever it ran — by a condition that
always holds in CI, an imperative skip, or a deselecting hook — does not make
its row covered.
5. Harness Design Rules
Preferred Design
Use a lightweight scenario harness made of reusable ports and fakes.
Good harnesses:
- drive real public service APIs where possible
- simulate only the external boundary
- capture transcripts and terminal outputs
- are cheap to extend
High-risk capabilities should evolve away from purely ideal fakes over time.
Avoid
- giant one-off mocks per test
- hardcoded assertions copied across scenarios
- harnesses that mirror the production implementation too closely
- full E2E infrastructure when a service-boundary scenario would do
6. Reality Feedback Rule
Scenario harnesses must absorb verified reality, not only idealized assumptions.
When runtime behavior disproves the current fake or scenario model:
- record the observation
- classify it as product rule, dependency contract, code bug, or fake gap
- update the harness, contract fixtures, or scenario catalog as needed
- preserve it as regression coverage where the capability risk justifies it
This is how scenario tests become more trustworthy over time instead of drifting away from real behavior.
Reality feedback should live next to the capability it changes.
Preferred location:
tests/scenarios/<capability>/observations.yaml
7. Delivery Standard
Every feature or bug fix that changes a capability must update the relevant testing evidence.
Minimum rule:
- update the capability/scenario definition if behavior changed
- add or update unit tests for local logic changes
- add or update scenario coverage when the capability flow changed
- document remaining manual validation explicitly
This standard is only complete when PR review and CI both treat scenario coverage as a first-class requirement.