Skip to content

armgit / avibe

Public Imported from GitHub
0

Make your computer a home for your AI partner. Avibe is the local-first Agent OS: Vibey, its built-in AI partner, works out of the box, and Claude Code, Codex & OpenCode run unchanged beside it — reach them from your browser, phone, or any chat app.

8.1 KB
Raw Blame

Offline evaluations

LLM-in-the-loop evaluations of agent behavior that unit tests cannot pin: they make real model calls, so CI never runs them, and pytest never collects them (no test_ prefix). Each records a baseline per prompt version, so a prompt change is judged against the last one.

A12: compaction quality

Plan: docs/plans/avibe-agent-core.md §6, A12. Script: a12_compaction_quality.py. Baselines: a12_baselines.md.

What it does.

  • It drives a real Vibey Session through the instance's own vibe agent run. Every model call goes through that instance's Model Hub, and every checkpoint through the shipped C-9 code.

  • The scripted session runs in a temporary project with a temporary Agent. Its first Turns plant every scored fact. The user:

    • states three ground rules;
    • has the agent change two files, each for a stated reason;
    • has it run a check that fails twice;
    • agrees on a next step.
  • Only then do log reads, sized from the model's window and one per Turn, push the context over the threshold. They continue, up to seven, until two checkpoints have summarized the context past the last planted fact.

  • A probe Turn then asks, from memory, for the ground rules, the files changed and why, the last error verbatim, and the next step, as one JSON object. Each of the ten planted facts is judged on its own field:

    • a rule only with its polarity and its action: each rule is {"action", "kind": "must" | "must_not"}, and the action must name the constrained act (modify, rename public functions, record);
    • a changed value only inside its file's entry and as the new value ("to 5", "= 5"), and a reason only in that entry's why (the REFILL_RATE reason only about refilling);
    • the changelog edits as an entry for CHANGES.md among the changed files;
    • the last error as the whole line;
    • the next step with its action and boundary (add a test in tests/test_burst.py for a burst of exactly 5), not only the file.
  • An eleventh fact checks that the checkpoint request stays out of the user's record. The request comes from the harness, not the user (context.md §11, checkpoint-v4). The fact reads the text of every checkpoint written before the probe, as stored in its row, and the probe's rules and next step. It fails on any of these:

    • a no-tools instruction anywhere;
    • the checkpoint itself (not an earlier one) credited to the user;
    • the checkpoint listed under the goals, the user requirements, or the next steps.

    The JSON lists every such line under leaks. A checkpoint the probe triggers is not read: it may rightly record the probe's own "do not read any file or run any command".

Valid runs only. A run is scored only if ground truth (the database rows and the fixture on disk) shows that it happened as scripted:

  • every Agent run succeeded within its wait;
  • the fixture ends with BURST = 5, REFILL_RATE = 1, exactly the two CHANGES.md lines for those changes, allow() still defined, and vendor/legacy_api.py untouched;
  • both check outputs are in the Session's tool results, in order;
  • at least two checkpoints before the probe summarized the context past the last planted fact: their summarized_to_seq reaches the Session's newest context position after the planting Turns, so every scored fact went through both;
  • the probe ran no tool (judged by committed tool results, not the display traces);
  • every model-written checkpoint used the checkout's prompt version.

Otherwise the JSON says "valid": false with the reasons, no score is given, and the script exits with status 3. A failed or detached Agent run stops it at once. A non-empty --project-dir is refused. Everything reported comes from the rows: a checkpoint is model-written only when its row names a summarizer, and the prompt version is the summarizers'.

Where it runs. It writes real Sessions and spends real tokens, so it runs only inside a regression instance (VIBE_DEPLOYMENT_ENV=regression); elsewhere it exits with status 2.

How to run it.

  1. Give Vibey a model whose Model Hub definition has a window of at most 32,000 tokens (the size the fixture is built and tested for). For example, add claude-haiku-4-5 in Settings → Models → Vibey and edit its window to 32,000 and its max output to 4,000.

  2. In the instance (python3 scripts/incus_regression.py shell --target ...), run:

    cd /opt/avibe/source && /opt/avibe/venv/bin/python tests/evals/a12_compaction_quality.py --model claude-haiku-4-5
    

    It drives the vibe next to that Python (/opt/avibe/venv/bin/vibe); --vibe names another one.

  3. Read the JSON it prints:

    • whether the run is valid, and why not;
    • the prompt version, the model, and the output budget Vibey used on it;
    • the last planted fact's context position, and each compaction (mode, how far it summarized, sizes, and whether the model wrote it);
    • each checkpoint turn's outcome and repair;
    • the per-fact score and the probe answer.
  4. Add a row to a12_baselines.md, then restore the model definition.

The run makes about ten Turns, plus their checkpoint turns, on the small window.

Known limits. The threat model is an honest model behaving imperfectly, not an adversary gaming the scorer.

  • The scorer matches keywords inside the right field, so a correct reason in unexpected words can be missed. Read the answer before trusting a drop.
  • A run is checked against the scripted end state, not against every way an agent could misbehave. Checks that only matter against deliberate gaming, or that the recorded runs already satisfy, are not added.
  • If the agent deleted or renamed a fixture file, the script stops with an error instead of the JSON report. The run is not scored either way.
  • The two check outputs are looked for in every tool result. A read of check.py, whose source contains the last error's text, could therefore stand in for the second check run. The recorded runs ran both checks with bash; confirm that in the Session when a run looks suspicious.
  • A checkpoint turn may read files: C-9 allows read while dreaming, and its results stay in the fork_turn audit row instead of the Session's tool results. A checkpoint that read the fixture could recover a changed value from disk instead of from the conversation. The recorded runs' checkpoint turns made no tool round (rounds is 0 in every fork_turn row); check that in the Session before recording a row.
  • The two checkpoints the gate counts may include a dropped row, which carries the earlier checkpoint without a model call. The JSON marks each compaction's model_written; the recorded run's were all model-written.
  • A reason passes on its keyword, so a reversed one ("the upstream API accepts bursts above 5") would still pass.
  • A changed file is matched by the end of its path, so tests/rate_limit.py would count for src/rate_limit.py. The recorded answer names the exact paths.
  • The fixture check takes the first BURST = 5 or REFILL_RATE = 1 line, not the effective last assignment. The recorded run's edits replaced the old lines.
  • The next step is checked for its verb, file and boundary, not its polarity: "do not write tests/test_burst.py for a burst of exactly 5" would pass. The recorded answer states the plan.
  • The last error is compared ignoring case and wrapping quotes or backticks. The recorded answer is exact.
  • --vibe is split on whitespace, so a path with spaces needs a link without them. The documented run uses the default.
  • The leak fact finds the request's marks by keyword. Before the probe, the fixture's user mentions neither checkpoints nor tools, so here such a line is the request's. A checkpoint that writes "this checkpoint" under its next steps fails it.
  • Heading lines only set the section and are not read for leaks, so "## Next steps: write the checkpoint" would pass. The recorded runs' headings contain neither word.
  • The model is recorded as requested, not as served. If Model Hub fails over to another hop during a run, the row would name the requested model. The recorded runs were served by it throughout (every checkpoint's and response's origin); check the origins in the Session before recording a row.