Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

10.3 KB
Raw Blame
/**
 * The training corpus a project-specific jev head is fit on.
 *
 * Schema is daseinlabs/open-jev's **Route A**: one JSON object per line,
 * `{"context": str, "options": [str, ...], "label": int}`, where `label`
 * indexes `options`. Adopted deliberately rather than invented — it keeps
 * `openjev eval` usable as an independent cross-check on a corpus PortOS
 * built, which is the only outside opinion available on whether a head that
 * beat our baselines actually learned anything.
 *
 * ## What the labels are worth
 *
 * They are WEAK. A merged pull request is evidence that the maintainer wanted
 * it, not a hand-annotation that it advances the clause the retriever paired it
 * with; a closed-unmerged one is evidence they did not. The whole apparatus
 * downstream — a held-out gold split, two baselines, a strictly-better
 * adoption gate — exists because these labels are weak enough that "the head
 * trained fine" proves nothing on its own.
 *
 * ## The split is the safety property
 *
 * `splitCorpus` assigns each example to train or gold by hashing its own
 * content, so the split is deterministic across machines and rebuilds, and an
 * example cannot migrate between splits when the corpus grows around it.
 * `assertSplitDisjoint` then REFUSES a split whose two halves share an example
 * key. A gold set contaminated by its training split reports a score that is
 * partly memorization, and that score is the single number the adoption gate
 * reads — so the refusal is a hard stop, never a warning.
 *
 * Pure: no I/O, no forge access. `services/jevCorpusBuilder.js` runs the
 * queries and writes the file.
 */

import { createHash } from 'crypto';
import { z } from 'zod';

export const JEV_CORPUS_SCHEMA_VERSION = 1;

/**
 * Bounds on one example. `context` is a composed premise, so it inherits the
 * scorer's premise budget; the option list inherits the decision's.
 */
export const JEV_CORPUS_MAX_CONTEXT_CHARS = 32_000;
export const JEV_CORPUS_MAX_OPTIONS = 32;

/**
 * The share of examples held out for the gold set.
 *
 * A third, not the customary tenth: these corpora are small (a few hundred
 * examples on a busy repository, far fewer on a young one), and the gold set
 * has to separate three numbers that may differ by a few points. A 10% split of
 * 200 examples is 20 rows, where one flipped label moves accuracy 5 points.
 */
export const JEV_CORPUS_GOLD_FRACTION = 1 / 3;

/**
 * The fewest examples a corpus may have and still be trained on.
 *
 * Below this the gold set cannot support a comparison against two baselines
 * at all, and the honest answer is "this install has no history to learn
 * from", not a head with a score.
 */
export const JEV_CORPUS_MIN_EXAMPLES = 60;

/** The weak-evidence sources a corpus may draw from. */
export const JEV_CORPUS_SOURCES = Object.freeze([
  'merged-pr',
  'closed-unmerged-pr',
  'closed-not-planned-issue',
  'parked-issue',
]);

export const jevCorpusExampleSchema = z.object({
  context: z.string().min(1).max(JEV_CORPUS_MAX_CONTEXT_CHARS),
  options: z.array(z.string().min(1)).min(2).max(JEV_CORPUS_MAX_OPTIONS),
  label: z.number().int().min(0),
  // PortOS's own provenance, ignored by `openjev eval`. Kept so a corpus can be
  // audited for source balance without re-running the forge queries.
  source: z.enum(JEV_CORPUS_SOURCES),
}).strict().refine((row) => row.label < row.options.length, { path: ['label'] });

/**
 * A stable identity for one example.
 *
 * Content-addressed, and covering the OPTIONS as well as the context: the same
 * change scored against the same clause under a different option wording is a
 * different question, and collapsing the two would let a re-worded decision
 * silently inherit the old split assignment.
 */
export function corpusExampleKey(example) {
  return createHash('sha256')
    .update(String(example?.context || ''))
    .update('\u0000')
    .update((example?.options || []).join('\u0000'))
    .digest('hex')
    .slice(0, 32);
}

/**
 * Hash of the whole corpus, order-independent.
 *
 * Recorded on both the manifest and every head fit from it. Order-independent
 * because the builder's forge queries are paginated and a page boundary moving
 * must not re-key a corpus that contains exactly the same examples.
 */
export function corpusHash(examples) {
  const keys = (Array.isArray(examples) ? examples : [])
    .map((example) => `${corpusExampleKey(example)}:${example?.label}`)
    .sort();
  return createHash('sha256').update(keys.join('\n')).digest('hex').slice(0, 32);
}

/**
 * Turn a weakly-labelled observation into a Route A example.
 *
 * `options` is the decision's own hypothesis list, in its own order, so a
 * corpus row asks exactly the question the scorer is asked at inference time.
 * Returns null rather than throwing on anything malformed — the builder pulls
 * from a forge, and one unusable row must not fail a build of four hundred.
 */
export function buildCorpusExample({ context, options, chosen, source } = {}) {
  if (!Array.isArray(options)) return null;
  const label = options.indexOf(chosen);
  if (label < 0) return null;
  const candidate = { context: String(context || '').slice(0, JEV_CORPUS_MAX_CONTEXT_CHARS), options, label, source };
  return jevCorpusExampleSchema.safeParse(candidate).success ? candidate : null;
}

/**
 * Drop examples that ask the same question twice.
 *
 * A repository re-states a goal across `PRD.md` and `GOALS.md` often enough
 * that the retriever pairs one change with two byte-identical clauses. Left in,
 * those duplicates land in different splits by definition of the hash — the
 * same question appearing in both train and gold, which is precisely the
 * contamination `assertSplitDisjoint` exists to catch, arriving through the
 * front door. The FIRST label wins; a duplicate that disagrees is dropped with
 * it, because there is no principled way to pick between two weak labels.
 */
export function dedupeCorpus(examples) {
  return [...keyedCorpus(examples).values()];
}

/**
 * Deduplicated examples by key, so a caller that needs both does not hash twice.
 *
 * A context can be up to 32 KB and a corpus a thousand rows, so the SHA-256 is
 * not free: `dedupeCorpus` followed by a bucketing loop that re-derived the
 * same key digested the whole corpus twice to learn nothing new.
 */
function keyedCorpus(examples) {
  const seen = new Map();
  for (const example of Array.isArray(examples) ? examples : []) {
    const key = corpusExampleKey(example);
    if (!seen.has(key)) seen.set(key, example);
  }
  return seen;
}

/**
 * Deterministic train/gold split.
 *
 * The bucket comes from the example's OWN content hash, not from a shuffle and
 * not from its position: two builds of the same corpus on two machines must
 * produce the same gold set, or a head's reported score cannot be reproduced by
 * the person reading it.
 */
export function splitCorpus(examples, { goldFraction = JEV_CORPUS_GOLD_FRACTION } = {}) {
  const train = [];
  const gold = [];
  // 16 bits of the key, compared against the same fraction of 65536. Enough
  // resolution that a corpus of a few hundred lands within a point of the
  // requested fraction, and cheap to recompute anywhere.
  const cutoff = Math.round(goldFraction * 0x10000);
  // Dedupe and bucket in one pass over the keys the dedupe already computed.
  for (const [key, example] of keyedCorpus(examples)) {
    (parseInt(key.slice(0, 4), 16) < cutoff ? gold : train).push(example);
  }
  return { train, gold };
}

/**
 * REFUSE a split whose halves overlap.
 *
 * The gold score is the only evidence the adoption gate reads. An example that
 * appears in both halves makes that score partly a memorization check, and
 * nothing downstream could tell the difference — the head would simply look
 * better than it is and get adopted for it.
 *
 * Also refuses a gold set too small to separate three numbers: with fewer than
 * `JEV_CORPUS_MIN_GOLD` rows, one label flip moves accuracy by more than the
 * margin the gate is deciding on.
 */
export const JEV_CORPUS_MIN_GOLD = 20;

export function assertSplitDisjoint({ train, gold } = {}) {
  if (!Array.isArray(train) || !Array.isArray(gold)) return { ok: false, code: 'jev-corpus-split-invalid' };
  const trainKeys = new Set(train.map(corpusExampleKey));
  const overlap = gold.filter((example) => trainKeys.has(corpusExampleKey(example)));
  if (overlap.length) return { ok: false, code: 'jev-corpus-split-overlap', overlap: overlap.length };
  if (gold.length < JEV_CORPUS_MIN_GOLD) return { ok: false, code: 'jev-corpus-gold-too-small' };
  if (!train.length) return { ok: false, code: 'jev-corpus-split-invalid' };
  return { ok: true, train: train.length, gold: gold.length };
}

/**
 * Accuracy of always predicting the most common gold label.
 *
 * One of the two baselines the adoption gate compares against, computed here
 * rather than in the trainer so the Node side can state it without a Python
 * round trip — and so the number an operator is shown comes from the same code
 * path in a test as it does in the panel.
 */
export function majorityClassAccuracy(gold) {
  const rows = Array.isArray(gold) ? gold : [];
  if (!rows.length) return null;
  const counts = new Map();
  for (const row of rows) counts.set(row.label, (counts.get(row.label) || 0) + 1);
  return Math.max(...counts.values()) / rows.length;
}

/** Per-source example counts, for the manifest and the panel's build summary. */
export function summarizeCorpusSources(examples) {
  const counts = Object.fromEntries(JEV_CORPUS_SOURCES.map((source) => [source, 0]));
  for (const example of Array.isArray(examples) ? examples : []) {
    if (Object.hasOwn(counts, example?.source)) counts[example.source] += 1;
  }
  return counts;
}

/** Serialize to JSONL. `source` is kept: it is ours, and `openjev eval` ignores it. */
export const toCorpusJsonl = (examples) => (Array.isArray(examples) ? examples : [])
  .map((example) => JSON.stringify(example))
  .join('\n');

/**
 * Validate one already-parsed JSONL row, or null.
 *
 * Takes an OBJECT, not a line of text: `JSON.parse` throws, and the sanctioned
 * non-throwing wrapper lives in `fileUtils.js`, whose closure reaches `fs` and
 * `child_process`. Keeping the decode in `services/jevCorpusBuilder.js` is what
 * lets this file stay the pure leaf it advertises itself as
 * (`server/lib/importScoping.test.js`).
 */
export function validateCorpusRow(parsed) {
  const result = jevCorpusExampleSchema.safeParse(parsed);
  return result.success ? result.data : null;
}