Local-first 🧠memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly 😴 dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.
TypeScript
/**
* Announced decommissions of embedding models (t_e7a226ce).
*
* Changing `embedding_model` invalidates every stored vector: the index
* stamps the model into its ABI tokens and the only remedy is a full
* re-embed of the corpus (`EMBEDDING_ABI_FIX_COMMAND`). That machinery
* fires only AFTER the operator has already changed the model. This module
* is the strictly earlier signal - it fires while the config is still
* correct and the index still valid, so a migration can be scheduled
* rather than performed under an index that has just stopped working.
*
* ## Where the dates come from, and why
*
* Four sources were available and three are refused:
*
* - **A network registry.** Refused. This project takes no unrequested
* outbound call for a fact like this, and says so at every relevant
* site: the presets catalog is "No server, no network"; runtime
* notices are computed "deterministically - no network, no LLM, no DB
* open"; `search check`'s provider probe is the ONE outbound call that
* verb makes and has an opt-out for the air-gapped caller. A cached
* registry is also the same staleness as a static table with more
* machinery, and an unreachable one is a third state to design for.
* - **A field on `EmbeddingModelPreset`.** Refused. That catalog is a
* list of RECOMMENDATIONS - its first entry is
* `RECOMMENDED_EMBEDDING_MODEL` unconditionally and is defaulted into
* every new provider profile - and a curated list of recommendations
* that also carries obituaries is two catalogs in one file. Worse, the
* populations barely overlap: five of its six entries are open-weight
* checkpoints with no vendor able to announce anything, while the
* hosted, versioned models that actually get decommissioned live in
* the disjoint `EMBEDDING_PRICING` table.
* - **An operator-declared date** (a flat config scalar, or a field on a
* registry profile). Refused as the SOLE source, because neither can
* tell "the operator declared nothing" from "no shutdown exists" - one
* optional key carries both meanings, which is the exact defect this
* check exists to remove. A profile-borne date additionally cannot
* reach the configuration this tool's own onboarding recommends, which
* is a bare `embedding_base_url` + `embedding_model` pair with no
* profile at all.
*
* What ships is a SURVEY: a static table of models somebody actually
* looked up, each entry carrying either a date or an explicit negative,
* plus the date the survey as a whole was reviewed. That is the mechanism
* this repository already uses for third-party facts - `inputWindowTokens`
* is hardcoded from published specs with a per-entry source comment - and
* it costs no network call.
*
* ## How the survey handles its own staleness
*
* The unfixable weakness of a static table is that it goes stale between
* releases, and the honest response is asymmetric:
*
* - A **negative** is a claim about the world and it EXPIRES. Past
* {@link EMBEDDING_SUNSET_SURVEY_HORIZON_DAYS} from `reviewedAt`,
* `none_announced` degrades to `undetermined` with `survey_stale`
* rather than remaining a confident negative nobody re-checked.
* - A **positive** survives a stale survey. A vendor can move a date,
* but a shutdown that was announced was announced, and an operator who
* is told about it can verify it. The verdict carries `surveyed_at` so
* the message can say how old the claim is.
* - **Absence** from the survey is `unsurveyed` and is never
* `none_announced`. `embedding_model` is a free string with zero
* validation and this tool's own onboarding recommends a preview model
* that no catalog here carries, so off-survey is the NORMAL case. This
* is the same polarity `declaredInputWindowTokens` chose for an
* unlisted window and `resolveEmbeddingPrice` chose for an unlisted
* price: unknown is reported as unknown, never as a passing value.
*
* ## What an entry may honestly claim
*
* A negative here is a claim about the MODEL, not about any endpoint
* serving it. An open-weight checkpoint has no operator who could
* decommission it - that is the fact the entry records - while the
* third-party endpoint an operator points `embedding_base_url` at can
* disappear tomorrow. That is a liveness question and `o2b search check`
* already owns it with a live probe.
*
* Nothing in this module names a vendor. The resolved `provider` is one of
* four TRANSPORT kinds (`openai-compat`, `zeroentropy`, `local`,
* `disabled`); vendor identity exists only inside a free-form
* `embedding_base_url` that nothing in the tree maps, so a message that
* named one would be inferring it from a URL.
*
* Pure and injectable: the clock is a parameter and so is the survey, so
* every state below is reachable from a test without moving a wall clock
* or editing the shipped table.
*/
import { isValidIsoInstant, parseIsoUtc } from "../../brain/health/iso-time.ts";
import { EMBEDDING_MODEL_PRESETS } from "./presets.ts";
import { LOCAL_EMBEDDING_MODEL } from "./signature.ts";
// ----- Vocabularies ---------------------------------------------------------
/**
* What this build can say about one model's decommission.
*
* `unsurveyed` and `none_announced` are separate members and that
* separation is the unit: one says the survey looked and found no
* announcement, the other says the survey never looked at this model.
*/
export const EMBEDDING_SUNSET = Object.freeze({
/** The survey declares a decommission date for this model. */
announced: "announced",
/** The survey covers this model and records no announcement. */
noneAnnounced: "none_announced",
/** The model is outside the survey; no statement either way. */
unsurveyed: "unsurveyed",
/** No verdict was reached; {@link EmbeddingSunsetVerdict.reason} says why. */
undetermined: "undetermined",
} as const);
/** Closed union over {@link EMBEDDING_SUNSET}. */
export type EmbeddingSunsetState = (typeof EMBEDDING_SUNSET)[keyof typeof EMBEDDING_SUNSET];
/** Membership list, ordered from most to least that can be said. */
export const EMBEDDING_SUNSET_STATES: ReadonlyArray<EmbeddingSunsetState> = Object.freeze([
EMBEDDING_SUNSET.announced,
EMBEDDING_SUNSET.noneAnnounced,
EMBEDDING_SUNSET.unsurveyed,
EMBEDDING_SUNSET.undetermined,
]);
/** Narrow a string read back off disk or across a tool boundary. */
export function isEmbeddingSunsetState(value: unknown): value is EmbeddingSunsetState {
return (
typeof value === "string" && (EMBEDDING_SUNSET_STATES as ReadonlyArray<string>).includes(value)
);
}
/**
* Why no verdict was reached. A separate vocabulary so a guard cannot let
* `survey_stale` be read back where a state belongs.
*/
export const EMBEDDING_SUNSET_UNDETERMINED_REASON = Object.freeze({
/** No embedding model resolved from the configuration. */
modelUnresolved: "model_unresolved",
/** The survey's own review date is older than the horizon. */
surveyStale: "survey_stale",
/**
* A survey entry carries a date this build cannot parse.
*
* Produced by the SHIPPED table going wrong, which is why it survives
* while its declaration-side twin did not: the survey is data compiled
* into the binary and injectable at the check's own factory
* (`makeEmbeddingSunsetCheck`), so a release that mistypes a date has
* to say so rather than report the model as un-announced. The
* operator's declaration has no such gap - see
* {@link classifyEmbeddingSunset}.
*/
surveyEntryMalformed: "survey_entry_malformed",
} as const);
/** Closed union over {@link EMBEDDING_SUNSET_UNDETERMINED_REASON}. */
export type EmbeddingSunsetUndeterminedReason =
(typeof EMBEDDING_SUNSET_UNDETERMINED_REASON)[keyof typeof EMBEDDING_SUNSET_UNDETERMINED_REASON];
/** Membership list for {@link EMBEDDING_SUNSET_UNDETERMINED_REASON}. */
export const EMBEDDING_SUNSET_UNDETERMINED_REASONS: ReadonlyArray<EmbeddingSunsetUndeterminedReason> =
Object.freeze([
EMBEDDING_SUNSET_UNDETERMINED_REASON.modelUnresolved,
EMBEDDING_SUNSET_UNDETERMINED_REASON.surveyStale,
EMBEDDING_SUNSET_UNDETERMINED_REASON.surveyEntryMalformed,
]);
/** Narrow a string read back off disk or across a tool boundary. */
export function isEmbeddingSunsetUndeterminedReason(
value: unknown,
): value is EmbeddingSunsetUndeterminedReason {
return (
typeof value === "string" &&
(EMBEDDING_SUNSET_UNDETERMINED_REASONS as ReadonlyArray<string>).includes(value)
);
}
/**
* Which layer answered.
*
* A closed vocabulary rather than a boolean because there are three
* answers, not two: the operator's declaration, this build's table, and
* NEITHER - which is what `unsurveyed` and `undetermined` rest on. A
* boolean would have made "came from the survey" and "came from nowhere"
* the same value.
*/
export const EMBEDDING_SUNSET_SOURCE = Object.freeze({
/** The operator declared this date in `Brain/_brain.yaml`. */
declaration: "declaration",
/** This build's shipped survey answered. */
survey: "survey",
/** Neither layer had anything to say about the model. */
none: "none",
} as const);
/** Closed union over {@link EMBEDDING_SUNSET_SOURCE}. */
export type EmbeddingSunsetSource =
(typeof EMBEDDING_SUNSET_SOURCE)[keyof typeof EMBEDDING_SUNSET_SOURCE];
/** Membership list, in precedence order. */
export const EMBEDDING_SUNSET_SOURCES: ReadonlyArray<EmbeddingSunsetSource> = Object.freeze([
EMBEDDING_SUNSET_SOURCE.declaration,
EMBEDDING_SUNSET_SOURCE.survey,
EMBEDDING_SUNSET_SOURCE.none,
]);
/** Narrow a string read back off disk or across a tool boundary. */
export function isEmbeddingSunsetSource(value: unknown): value is EmbeddingSunsetSource {
return (
typeof value === "string" && (EMBEDDING_SUNSET_SOURCES as ReadonlyArray<string>).includes(value)
);
}
// ----- The survey -----------------------------------------------------------
/** One model somebody looked up, and what they found. */
export interface EmbeddingSunsetEntry {
/** Exact model string, matched the way `findEmbeddingPreset` matches. */
readonly model: string;
/**
* Announced decommission date (`YYYY-MM-DD` or a full ISO instant), or
* `null` for an explicit surveyed NEGATIVE: this model was looked up and
* no decommission had been announced as of the survey's review date.
*/
readonly sunsetAt: string | null;
/** Where the answer came from. Never empty - a claim owes its origin. */
readonly source: string;
/** Anything a reader needs beyond the date itself. */
readonly note: string;
}
export interface EmbeddingSunsetSurvey {
/** `YYYY-MM-DD` the whole table was last checked. Bounds every negative. */
readonly reviewedAt: string;
readonly entries: ReadonlyArray<EmbeddingSunsetEntry>;
}
/**
* How long a NEGATIVE stays believable.
*
* A year, because that is the period over which "nobody has announced
* anything" stops being a fact somebody established and becomes a fact
* nobody re-checked. It bounds only the negatives: an announced date is
* still reported from an older survey, with its age named.
*/
export const EMBEDDING_SUNSET_SURVEY_HORIZON_DAYS = 365;
const MS_PER_DAY = 24 * 60 * 60 * 1000;
/**
* Justification shared by every open-weight row below. Stated once because
* it is one argument, not six: nobody operates these checkpoints, so there
* is no party with the authority to announce a decommission. The endpoint
* an operator points at CAN disappear, which is a liveness question
* `o2b search check` probes live and this table does not answer.
*/
const OPEN_CHECKPOINT_SOURCE =
"open-weight checkpoint distributed under a public licence: there is no operator of a hosted " +
"endpoint with the authority to decommission the model itself";
function openCheckpoint(model: string): EmbeddingSunsetEntry {
return {
model,
sunsetAt: null,
source: OPEN_CHECKPOINT_SOURCE,
note: "the endpoint serving it can still be withdrawn; that is liveness, not a sunset",
};
}
/** Where the OpenAI rows below were read from. Cited, not remembered. */
const OPENAI_DEPRECATIONS_URL = "https://developers.openai.com/api/docs/deprecations";
/**
* The one embedding deprecation that page carries, and the only positive
* this build can source.
*
* Announced 2023-07-06, shut down 2024-01-04, replacement
* `text-embedding-3-small`. The date is in the PAST, which is not a
* weakness of the entry - it is the arm that matters most. An operator
* restoring an old configuration, or copying one out of a years-old
* tutorial, is told the model is already gone instead of watching the
* first index build fail against an endpoint that no longer answers.
*/
const OPENAI_FIRST_GENERATION_SHUTDOWN = "2024-01-04";
const OPENAI_FIRST_GENERATION_SOURCE =
`OpenAI deprecations, announced 2023-07-06, shutdown ${OPENAI_FIRST_GENERATION_SHUTDOWN}, ` +
`replacement text-embedding-3-small: ${OPENAI_DEPRECATIONS_URL}`;
/**
* Every model named in that entry, verbatim. Listed rather than pattern
* matched: `text-search-ada-doc-001` is shut down and
* `text-embedding-ada-002` is not, so a prefix rule over "ada" would
* report a live model as decommissioned.
*/
const OPENAI_FIRST_GENERATION_EMBEDDINGS: ReadonlyArray<string> = Object.freeze([
"text-similarity-ada-001",
"text-search-ada-doc-001",
"text-search-ada-query-001",
"code-search-ada-code-001",
"code-search-ada-text-001",
"text-similarity-babbage-001",
"text-search-babbage-doc-001",
"text-search-babbage-query-001",
"code-search-babbage-code-001",
"code-search-babbage-text-001",
"text-similarity-curie-001",
"text-search-curie-doc-001",
"text-search-curie-query-001",
"text-similarity-davinci-001",
"text-search-davinci-doc-001",
"text-search-davinci-query-001",
]);
/**
* The hosted models this repository already names in `EMBEDDING_PRICING`.
*
* These are SOURCED negatives, not assumed ones: the deprecations page
* carries exactly one embedding entry and none of these three appears in
* it. That is a real reading of a real page, which is what separates a
* negative worth recording from the silence this check exists to remove.
*/
const OPENAI_CURRENT_EMBEDDINGS: ReadonlyArray<string> = Object.freeze([
"text-embedding-3-small",
"text-embedding-3-large",
"text-embedding-ada-002",
]);
/**
* The shipped survey.
*
* Adding an entry is a one-line edit plus a bump of `reviewedAt`; every
* row owes a `source` and a test enforces it.
*/
export const EMBEDDING_SUNSET_SURVEY: EmbeddingSunsetSurvey = Object.freeze({
reviewedAt: "2026-08-15",
entries: Object.freeze([
...OPENAI_FIRST_GENERATION_EMBEDDINGS.map((model) => ({
model,
sunsetAt: OPENAI_FIRST_GENERATION_SHUTDOWN,
source: OPENAI_FIRST_GENERATION_SOURCE,
note: "first-generation OpenAI embedding model; the endpoint stopped answering on the date",
})),
...OPENAI_CURRENT_EMBEDDINGS.map((model) => ({
model,
sunsetAt: null,
source: `not listed in any deprecation entry on ${OPENAI_DEPRECATIONS_URL} as of the review date`,
note: "a hosted model, so this negative is the one on this table with a real expiry",
})),
{
model: LOCAL_EMBEDDING_MODEL,
sunsetAt: null,
source: "this build ships the local embedder, so this project is the only party that could",
note: "a hashing embedder with no service behind it; retiring it would be a release note",
},
...EMBEDDING_MODEL_PRESETS.map((preset) => openCheckpoint(preset.model)),
]),
});
// ----- The classifier -------------------------------------------------------
/**
* An operator's own record of an announcement they have read.
*
* It names its MODEL explicitly rather than meaning "whatever is
* configured". A declaration that meant the latter would silently
* re-target itself the next time `embedding_model` changed, asserting one
* vendor's shutdown date about a different vendor's model - which is the
* confidently-wrong output this whole unit exists to prevent.
*/
export interface EmbeddingSunsetDeclaration {
readonly model: string;
readonly sunsetAt: string;
}
export interface EmbeddingSunsetVerdict {
readonly state: EmbeddingSunsetState;
/** The model the verdict is about, or `null` when none resolved. */
readonly model: string | null;
/** The declared date, or `null` for every other state. */
readonly sunset_at: string | null;
/** Whole days until the date; negative once it has passed. */
readonly days_remaining: number | null;
/** Why no verdict was reached, or `null` when one was. */
readonly reason: EmbeddingSunsetUndeterminedReason | null;
/** The survey review date this verdict rests on. */
readonly surveyed_at: string;
/** Which layer answered. */
readonly source: EmbeddingSunsetSource;
/**
* True when a declaration answered AND the survey held a different
* answer for the same model. Distinguishes "the survey said nothing
* about this model" from "the survey disagreed" - one of the two
* records is then wrong, and hiding either is how a wrong date
* survives.
*/
readonly overrode_survey: boolean;
/**
* What the survey held for this model when a declaration overrode it: a
* date, or `null` meaning the survey recorded no announcement. Only
* meaningful alongside {@link overrode_survey}.
*/
readonly survey_sunset_at: string | null;
}
function undetermined(
model: string | null,
reason: EmbeddingSunsetUndeterminedReason,
surveyedAt: string,
): EmbeddingSunsetVerdict {
return {
state: EMBEDDING_SUNSET.undetermined,
model,
sunset_at: null,
days_remaining: null,
reason,
surveyed_at: surveyedAt,
source: EMBEDDING_SUNSET_SOURCE.none,
overrode_survey: false,
survey_sunset_at: null,
};
}
/**
* What the survey says about `model` at `nowMs`.
*
* Lookup is by exact model string with no family heuristic, for the reason
* `declaredInputWindowTokens` gives: a lifecycle differs between
* checkpoints of one family, so inferring a date from a shared prefix
* would invent the value.
*
* Nothing here reads a global clock, and nothing here refuses a FUTURE
* instant. The future-instant refusals elsewhere in this codebase
* (`usableAuthoredAtSeconds`, the stale-dependency filter) discard a
* declared time later than the reading clock; a sunset date is supposed to
* be later than the reading clock, so only the injectable-clock half of
* that idiom applies.
*/
export function classifyEmbeddingSunset(
model: string | null,
nowMs: number,
survey: EmbeddingSunsetSurvey = EMBEDDING_SUNSET_SURVEY,
declaration?: EmbeddingSunsetDeclaration,
): EmbeddingSunsetVerdict {
const reviewedAt = survey.reviewedAt;
if (model === null || model.trim() === "") {
return undetermined(null, EMBEDDING_SUNSET_UNDETERMINED_REASON.modelUnresolved, reviewedAt);
}
const entry = survey.entries.find((e) => e.model === model) ?? null;
// The operator's layer first. It exists BECAUSE a table baked into the
// binary goes stale between releases, and whoever wrote it has just
// read the announcement; the survey was frozen at release. A
// declaration about some OTHER model is not about this one and does not
// participate - see EmbeddingSunsetDeclaration for why it names its
// model rather than meaning "whatever is configured".
if (declaration !== undefined && declaration.model === model) {
if (!isValidIsoInstant(declaration.sunsetAt)) {
// Not a verdict, because no vault can reach this. `sunset_at` is
// validated with this same predicate by `parseEmbeddingsBlock`,
// which is the ONLY producer of `BrainConfig.embeddings` and throws
// a `BrainConfigError` naming the key - so an operator who mistypes
// the date is told at load time, long before any check runs. An
// `undetermined/declaration_malformed` verdict here would be a
// reported state with a four-sentence operator message and no input
// that could produce it, which is the defect this release removes.
// A caller that reaches this has skipped the parser, and that is a
// programming error: it gets said rather than turned into a
// fabricated instant.
throw new TypeError(
`classifyEmbeddingSunset: declared sunsetAt for ${model} is not an ISO-8601 instant ` +
`(${declaration.sunsetAt}); declarations come from parseEmbeddingsBlock, which ` +
"validates them",
);
}
// A disagreement is carried, not erased: one of the two records is
// wrong, and a silent override is how the wrong one survives.
const disagrees = entry !== null && entry.sunsetAt !== declaration.sunsetAt;
return {
state: EMBEDDING_SUNSET.announced,
model,
sunset_at: declaration.sunsetAt,
days_remaining: Math.floor((parseIsoUtc(declaration.sunsetAt) - nowMs) / MS_PER_DAY),
reason: null,
surveyed_at: reviewedAt,
source: EMBEDDING_SUNSET_SOURCE.declaration,
overrode_survey: disagrees,
survey_sunset_at: disagrees ? entry.sunsetAt : null,
};
}
if (entry === null) {
return {
state: EMBEDDING_SUNSET.unsurveyed,
model,
sunset_at: null,
days_remaining: null,
reason: null,
surveyed_at: reviewedAt,
source: EMBEDDING_SUNSET_SOURCE.none,
overrode_survey: false,
survey_sunset_at: null,
};
}
if (entry.sunsetAt !== null) {
if (!isValidIsoInstant(entry.sunsetAt)) {
// Refused loudly rather than folded into "absent": a date this
// build cannot read is a defect in the table, and reporting it as
// "no announcement" would hide the one thing worth fixing.
return undetermined(
model,
EMBEDDING_SUNSET_UNDETERMINED_REASON.surveyEntryMalformed,
reviewedAt,
);
}
return {
state: EMBEDDING_SUNSET.announced,
model,
sunset_at: entry.sunsetAt,
days_remaining: Math.floor((parseIsoUtc(entry.sunsetAt) - nowMs) / MS_PER_DAY),
reason: null,
surveyed_at: reviewedAt,
source: EMBEDDING_SUNSET_SOURCE.survey,
overrode_survey: false,
survey_sunset_at: null,
};
}
// A surveyed NEGATIVE, and negatives expire.
const reviewedMs = parseIsoUtc(reviewedAt);
const staleAfter = reviewedMs + EMBEDDING_SUNSET_SURVEY_HORIZON_DAYS * MS_PER_DAY;
if (!Number.isFinite(reviewedMs) || nowMs > staleAfter) {
return undetermined(model, EMBEDDING_SUNSET_UNDETERMINED_REASON.surveyStale, reviewedAt);
}
return {
state: EMBEDDING_SUNSET.noneAnnounced,
model,
sunset_at: null,
days_remaining: null,
reason: null,
surveyed_at: reviewedAt,
source: EMBEDDING_SUNSET_SOURCE.survey,
overrode_survey: false,
survey_sunset_at: null,
};
}