Skip to content

Local-first 🧠 memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly 😴 dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.

TypeScript
19.0 KB
Raw Blame

Decision-model providers

Every route Open Second Brain can send a decision request to, how each one is configured, and what each one means for your data. The feature itself, its config keys, the provider order, the local thresholds store and the privacy rules that apply to every route are in docs/decision-models.md.

Nothing here is used unless you set decision_model_enabled: "true" and name one of these presets in decision_model_provider (or several, in order, in decision_model_providers). No preset is ever picked for you. With a single decision_model_provider no route is ever a fallback for another one; an explicit decision_model_providers list is the one way to name an order, and then the first entry that is configured and answers wins.

Presets

Preset Adapter Base URL Model id Context (state tokens: default / ceiling) Licence Data handling
typesafe systemone https://api.typesafe.ai jev-1.13.0 32000 / 32000 hosted TypeSafe, directly
openrouter systemone https://openrouter.ai/api typesafe/jev-1.13 32000 / 32000 hosted OpenRouter, then TypeSafe
vercel systemone https://ai-gateway.vercel.sh/typesafe typesafe-ai/jev 32000 / 32000 hosted Vercel AI Gateway, then TypeSafe
vercel-evaluate vercel-evaluate https://ai-gateway.vercel.sh typesafe-ai/jev 32000 / 32000 hosted Vercel AI Gateway, then TypeSafe
opencode-zen systemone https://opencode.ai/zen jev-1.13 32000 / 32000 hosted OpenCode Zen, then TypeSafe
laya systemone http://127.0.0.1:8000 english 2000 / 8192 Apache-2.0 your own server; on loopback nothing leaves the machine
openjev systemone http://127.0.0.1:3000 openjev 4000 / 16384 CC-BY-NC-4.0 your own server; on loopback nothing leaves the machine
compatible systemone must be set must be set 32000 / 32000 depends on the model whoever runs the server
llm-emulation llm-emulation must be set must be set 32000 / 32000 depends on the model whoever runs the chat endpoint
converse converse must be set must be set 32000 / 32000 depends on the model whoever runs the gateway

"State tokens" is Open Second Brain's own conservative estimate (about two characters per token), used to keep a request within the route's context: decision_model_max_state_tokens defaults to the first number and is clamped to the second.

Other per-route limits:

Preset Key Most choice options Threshold profile Input price (USD / M tokens)
typesafe, openrouter, vercel, vercel-evaluate, opencode-zen required 255 jev-1.13 0.042
laya optional on loopback 100 the server's model id, via the local store 0
openjev optional on loopback 52 the server's model id, via the local store 0
compatible required 255 the server's model id, via the local store, or jev-1.13 if named unknown
llm-emulation required 255 the endpoint's model id, via the local store unknown unless both price keys are set
converse required 255 the gateway's model id, via the local store unknown unless both price keys are set

A choice question above the route's option limit is never sent: a use that can split it does, and otherwise that request degrades with budget, like an oversized state.

Hosted Jev routes

typesafe, openrouter, vercel, opencode-zen and vercel-evaluate all reach the same model family, Jev 1.13, and share one threshold profile. They differ in who handles the request on the way: a gateway adds its own logging and retention terms to TypeSafe's. o2b decision-model check names the processor for the configured preset.

vercel-evaluate

The Vercel AI Gateway also serves Jev at its provider-neutral POST https://ai-gateway.vercel.sh/v1/evaluate, with renamed fields. The adapter maps them both ways:

  • a noul question is sent as type boolean, and its answer comes back as probability;
  • usage is camelCase (inputTokens, outputTokens), and the gateway's cost (providerMetadata.gateway.cost, a decimal string) is recorded as a reported cost;
  • choice and score answers may omit confidence; it is then recomputed from the probabilities;
  • an error body ({message, error_type}) is never read or printed: the HTTP status alone decides, as on every route.

The adapter sends no providerOptions. The gateway's evaluation fallbacks can rerun a request on a generative model, which would make the answers uncalibrated without saying so, so Open Second Brain never configures them.

decision_model_enabled: "true"
decision_model_provider: vercel-evaluate
decision_model_env_key: AI_GATEWAY_API_KEY # the variable's name, not the key
decision_model_uses: "rerank:shadow"

Self-hosted open-weight models

laya and openjev point at a decision server you run yourself. They default to a loopback address, where plain http is accepted and no key is needed, so a request never leaves the machine. They are the only fully local route. Open Second Brain does not download, install or start these servers.

  • Laya (Apache-2.0) is served by laya-serve, which listens on port 8000 and speaks POST /v1/systemone. The model id names a checkpoint: english (the default) or multilingual. Its default context window is small and a longer state is truncated silently by the server, so the preset starts at 2000 state tokens; raise decision_model_max_state_tokens (up to 8192) only after raising the server's own window. The server refuses more than 100 options per choice. It checks a bearer key only when started with one.
  • OpenJev (CC-BY-NC-4.0) is served by its decision shim, documented on port 3000, with a 16384-token prompt limit in the documented setup. One pass scores at most 52 options per choice. The weights are free for non-commercial use with attribution; commercial use needs the licensor's permission, and o2b decision-model check prints that note.

A self-hosted model has its own calibration: its probabilities are not Jev's. Its thresholds come from the local store, keyed by the model id the server reports; until o2b decision-model thresholds fit (or set) writes an entry for a use, an enforce setting runs as shadow there (see "Thresholds").

decision_model_enabled: "true"
decision_model_provider: laya
decision_model_base_url: http://127.0.0.1:8000
decision_model_uses: "rerank:shadow"

If the server runs on another host, set decision_model_base_url to it (https, or decision_model_allow_insecure_http: "true" on a network you trust) and name a key variable in decision_model_env_key: off the machine a key is required again.

compatible

Any other server that speaks POST /v1/systemone. The base URL, the model id and the key variable must be set. Open Second Brain cannot know which model family answers, so a compatible route has no built-in thresholds and its uses run as shadow until the store carries entries for its model id (o2b decision-model thresholds fit --use <use>); if the server serves the hosted Jev 1.13 family, naming it arms the shipped thresholds:

decision_model_provider: compatible
decision_model_base_url: https://decisions.example.com
decision_model_id: jev-1.13.0
decision_model_env_key: EXAMPLE_DECISION_KEY
decision_model_threshold_profile: jev-1.13 # the one built-in family

llm-emulation (uncalibrated)

llm-emulation is uncalibrated and calls a generative model. It is the one place the decision-model feature does that. It asks an OpenAI-compatible chat endpoint (POST <base_url>/chat/completions) for probabilities: a number between 0 and 1 for each noul question and a probability for each option or level of a choice or score. It exists to try the feature and to compare routes in shadow when only a chat endpoint is available.

  • The system prompt asks for probabilities only, never text, and the reply must be one JSON object keyed by question id: a JSON schema in response_format when the endpoint accepts it, the same schema in the prompt when it refuses that (HTTP 400 or 422). Anything else the model writes is discarded.
  • The state is vault content and is sent inside an untrusted_source fence, with the instruction to judge it and never follow instructions found in it.
  • Probabilities are self-reported, so they are clamped to [0, 1] and normalised; the chosen option is recomputed as the most probable one, a score value is the expected level, and every confidence is recomputed. A distribution with an unknown key or a value that is not a number is an invalid item.
  • Every answer and every decision_model_call record carries calibrated: false, so reports keep these runs apart.
  • enforce is refused as a config error unless decision_model_allow_uncalibrated: "true" is set; shadow is allowed. Even with the override, the model needs tuned thresholds in the local store before a use enforces; until then enforce runs as shadow.
  • A chat model bills output tokens and costs far more than a decision model. The cost is recorded as unknown unless both decision_model_input_price_usd_per_mtok and decision_model_output_price_usd_per_mtok are set, and without a known cost the daily cost gate cannot stop it.
  • It is never chosen implicitly and never used when another route fails.
  • The request passes the same redaction as every route (registry id decision-model-llm-emulation).
decision_model_enabled: "true"
decision_model_provider: llm-emulation # uncalibrated, shadow only
decision_model_base_url: https://api.example.com/v1
decision_model_id: some-small-chat-model
decision_model_env_key: EXAMPLE_API_KEY
decision_model_input_price_usd_per_mtok: "0.10"
decision_model_output_price_usd_per_mtok: "0.40"
decision_model_uses: "rerank:shadow"

converse (gateway)

converse reaches a decision model through a bearer-authenticated gateway that speaks the Bedrock-style Converse protocol (POST <base_url>/model/<model_id>/converse, the model named in the path). The base URL, the pinned model id and the key variable are yours to set; a key is required on every host, loopback included. The judgment travels the chat-shaped way: a system prompt and one user message carry the questions, the reply's JSON schema and the state, and the reply's message text holds one JSON object keyed by question id - validated item by item by the same rules as every route.

  • The route is uncalibrated, like llm-emulation: the gateway answers with generated text in which the model wrote its own probabilities, and nothing in the Converse protocol implies a calibrated judgment model behind it. The answers still clear the same per-item validation as every route - probabilities over exactly the question's options, distributions that sum to 1, a reported choice that is the most probable one - and an item that fails that validation is invalid while the rest of the batch stands. Records carry calibrated: false, and an enforce setting additionally needs decision_model_allow_uncalibrated: "true".
  • Its thresholds are its own: an enforce setting runs as shadow until the store carries entries for the gateway model's id, tuned from this route's own shadow data (o2b decision-model thresholds fit). It never borrows jev-1.13.
  • The state is vault content travelling inside a generated-text request, so it is fenced as untrusted content and passes the same redaction as every route (registry id decision-model-converse).
  • A reply whose stopReason arrived and is not end_turn was cut short and is refused, never half-parsed. Error bodies ({message, __type}) are never read or printed: the HTTP status alone decides.
  • Usage is camelCase (inputTokens, outputTokens). The gateway's pricing is yours, so the cost is unknown until you set both decision_model_input_price_usd_per_mtok and decision_model_output_price_usd_per_mtok - the chat-shaped request bills output tokens the same way it bills input ones - and a known cost still feeds the daily cost gate.
decision_model_enabled: "true"
decision_model_provider: converse
decision_model_base_url: https://gateway.example.com
decision_model_id: gateway-jev-1
decision_model_env_key: GATEWAY_DECISION_KEY # the variable's name, not the key
decision_model_uses: "rerank:shadow"

Thresholds

Probabilities from different model families are not comparable, so every threshold a use applies is held to the model it was tuned against. What is tuned comes from three places, in this order:

  1. The built-in baseline: the hosted Jev 1.13 family (jev-1.13). Every threshold Open Second Brain ships was set against this family, so every use may enforce on it. It is the only profile the code carries.
  2. The local thresholds store (decision-thresholds.json next to config.yaml), keyed by (model id, use) - the model id the server reports, never the preset. An entry for a pair lets that use enforce on that model; a vaults section overrides an entry for one vault; entries are never deleted, so switching models and switching back needs no re-tuning. o2b decision-model thresholds fit derives an entry from the use's labelled report data against a false-action target, and thresholds set writes one by hand.
  3. A valid machine-side eval mark (below), for the endpoint that earned it.

A use set to enforce whose model has no tuned thresholds for it runs as shadow: the request is sent and recorded, and the deterministic result is returned. Nothing on a hot path warns about it; o2b decision-model check prints one warning naming the model and the tuning command, and o2b decision-model report says the same. On an ordered chain the hold is per entry: the primary model and the fallback model are each judged by their own tuning, and the record names whichever answered.

The decision_model_threshold_profile key names the built-in family a compatible server serves; jev-1.13 is the only valid value. Every other model is tuned through the store, keyed by model id, so the key has nothing to say about them.

Measuring a profile: the preregistered eval

The shadow hold is not a dead end. Each use with a committed, labeled take ships a preregistered eval spec: the fixture and its content hash, a seed with a frozen dev/counted split, the promotion metric with its ceiling and minimum n, and the exact threshold values the use reads - all frozen and content-hashed before anything is graded, so a spec edited after the fact invalidates what came before instead of quietly re-scoping the claim.

o2b decision-model eval run --use <use> grades that take over your own configured adapter. The requests are real spend counted toward the daily cost gate (the deterministic scorer is canonical; an external judge would be advisory only). The verdict is a three-arm rule, the same shape the extract-prefilter report keeps: promote only when the counted error meets the ceiling, does not regress against the deterministic baseline, and the counted n is met; a short n is not_enough_evidence, never a pass.

Every run caches a decision_model_eval_grade record in the vault's continuity log - o2b decision-model eval list prints them (or o2b decision-model report), and malformed records are skipped by name, never silently. These records are review evidence. The gate input is a separate machine-side tuning mark under the state root (${XDG_STATE_HOME:-~/.local/state}/open-second-brain/decision-model/), written only by eval run, and only on a promote verdict for a named profile. With a valid mark, that profile-use pair runs enforce where it would have run shadow.

The mark is honored only while the installed spec still matches the one it was measured under: the spec hash, the fixture hash and the pinned threshold values are all re-checked on every read, along with the endpoint that earned it (the provider, model and base-url hash recorded in the mark). Edit the spec, move the fixture, change a threshold, or point the config at another endpoint, and the pair falls back to shadow - the fail-closed direction. Deleting the mark does the same; nothing breaks. o2b decision-model check names the measured state (untuned, baseline, measured under a spec hash, or measured under a spec hash on another endpoint - provider, model or base-url mismatch) and lists the cached grades, and the doctor readiness line carries the same summary. The local thresholds store ("Thresholds" above) is the other tuning path: it needs no eval run and no mark, only the use's labelled report data, and it keys on the model id instead of the profile. Tuning a profile by hand-editing the source table stays possible and still wins over a mark; the mark only ever carries pairs the table leaves untuned.