Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

18.6 KB
Raw Blame
/**
 * Operator-action ledger (#5594, epic #5593) — the durable, filterable record of
 * what the HUMAN did in PortOS.
 *
 * PortOS already logs a great deal, and none of it answers that question:
 * `history.jsonl` is PM2 lifecycle, `human_activity_events` is ingested external
 * activity (messages, calendar, media), the CoS JSONL ledgers are per-agent
 * transcripts, and `record_audit` is row-level before/after diffs. This store is
 * the one place that says "the user queued THIS task with THIS provider", "the
 * user rated THAT run negative", "the user changed these settings keys".
 *
 * Contract:
 * - **Closed vocabulary.** `type` must be in `USER_ACTION_TYPES`
 *   (`lib/userActionTypes.js`); an unknown type THROWS. A ledger whose type
 *   column is free text is a second `history.jsonl`, not a queryable log.
 * - **Idempotent.** Every event carries a stable `dedupeKey`; the unique
 *   `(type, dedupe_key)` index + `ON CONFLICT DO NOTHING` make a retried request
 *   a no-op. Same contract as `human_activity_events` / `tribe_touchpoints`.
 * - **Redacted.** Payload keys that look like a credential are dropped (value
 *   and all) and their paths listed under `payload.redactedKeys`, and long
 *   strings are truncated. A prompt or a settings patch can carry a token; the
 *   ledger must not become the place it leaks from.
 * - **Machine-local.** Never federated (PII must not ride the federation layer —
 *   ADR docs/decisions/2026-08-08-privacy-records-machine-local.md); guarded in
 *   sharing/peerSync.test.js. This log names what one operator did on one
 *   machine, which is exactly the kind of record that must not cross the wire.
 * - **Bounded.** Retention is enforced inline after an insert (row cap AND age
 *   cap) rather than by a new cron — one more scheduled job for a table that
 *   only grows when the user acts is not worth the moving part.
 *
 * PostgreSQL is authoritative. The file backend exists ONLY for the dev/test
 * escape hatch (`NODE_ENV=test` / `MEMORY_BACKEND=file`, see AGENTS.md's Storage
 * backend policy), which is what lets route suites assert a persisted row
 * without a database. It is deliberately built on `createPgFileFacade` rather
 * than copying `humanActivity.js`'s DB-only `ensureReady()` shape, which would
 * force every instrumented route's tests onto Postgres.
 *
 * Callers await writes; this recorder rejects on failure. Hooks that run after
 * a task mutation has committed catch and log that rejection so a successful
 * task response does not imply that its diagnostic ledger row also landed.
 * No fire-and-forget layer.
 */

import { join } from 'path';
import { v4 as uuidv4 } from '../lib/uuid.js';
import { atomicWrite, ensureDir, PATHS, readJSONFile } from '../lib/fileUtils.js';
import { createFileWriteQueue } from '../lib/fileWriteQueue.js';
import { isPlainObject } from '../lib/objects.js';
import { isSecretKey } from '../lib/secretKeys.js';
import { createPgFileFacade, resolvePgBackend } from '../lib/pgFileFacade.js';
import { isTestRunner, isVitestRunner } from '../lib/runtimeEnv.js';
import { isUserActionActor, isUserActionType } from '../lib/userActionTypes.js';
import { insertUserActionEvent, listUserActionEvents, pruneUserActionEvents } from './userActionsDb.js';

// Resolved per call, not at module load: PATHS.data is re-rooted by the shared
// test proxy (lib/mockPathsDataRoot.js), and a load-time join would both bind the
// pre-mock value and crash any suite whose fileUtils stub omits PATHS.data.
const eventsFile = () => join(PATHS.data, 'user-action-events.json');

// `isInsideRealDataRoot` (and its pathContainment.js closure) is loaded ONLY
// under the test runner, the same shape `fileCore.js`'s `loadGuard` uses for
// #6176's write guard — this file is reached by enough suites that a static
// import pushed importScoping.test.js's tree-wide budget over its ceiling.
// The module promise is memoized, so the dynamic import resolves once per worker.
let realDataRootGuard = null;
const loadRealDataRootGuard = () => (realDataRootGuard ??= import('../lib/testDataIsolation.js'));

/**
 * Structural guard against the bug class in #3683/#3687/#5605: a suite that
 * exercises a route wired to `recordUserAction` without redirecting
 * PATHS.data to a temp root would otherwise silently write
 * `user-action-events.json` into the developer's live `data/` tree the next
 * time such a route gets exercised — #5594 patched three known offenders
 * one at a time, which is a per-suite fix, not a guard against the next one.
 * Fires only under the test runner, and only at the moment the file backend
 * actually touches the ledger, so a suite that never reaches `recordUserAction`
 * / `listUserActions` is unaffected either way.
 *
 * Reads are guarded as well as writes: the live ledger holds machine-local
 * operator records (ADR docs/decisions/2026-08-08-privacy-records-machine-local.md),
 * so an untethered suite must not pull them into the test process either. That
 * read half is why this stays a local guard rather than folding into #6176's
 * write backstop; the real-root derivation itself is shared with it, so the two
 * can't disagree about which tree is live.
 *
 * @param {string} attempted what the file backend was about to do, e.g.
 *   `'recordUserAction attempted a write of'`
 */
async function assertTestDataRootRedirected(attempted) {
  if (!isVitestRunner()) return;
  const { isInsideRealDataRoot } = await loadRealDataRootGuard();
  if (!isInsideRealDataRoot(eventsFile())) return;
  throw new Error(
    `${attempted} user-action-events.json in the repo's real data/ tree. ` +
      'This suite exercises the user-action ledger but never redirected ' +
      'PATHS.data to a temp root - mock `../lib/fileUtils.js` with ' +
      "lib/mockPathsDataRoot.js's makePathsProxy/createTempDataRoot (the same " +
      'fix #5594 applied to cos.test.js / cosTaskRoutes.test.js / ' +
      'cosAgentFeedback.test.js) rather than letting the file backend touch the ' +
      'real tree.',
  );
}

/** LLM-readable one-liner cap — the ledger is read by a model, not paged through. */
export const SUMMARY_MAX_CHARS = 240;
/** Per-string payload cap; the containing object gets `truncated: true`. */
export const PAYLOAD_STRING_MAX_CHARS = 4096;
/** Retention bounds. BOTH hold: no more rows than this, nothing older than this. */
export const DEFAULT_MAX_ROWS = 20_000;
export const DEFAULT_MAX_AGE_DAYS = 90;

/** List clamps — an unbounded read of an operator log is never what a caller wants. */
export const MAX_LIST_LIMIT = 500;
export const DEFAULT_LIST_LIMIT = 100;

let retention = { maxRows: DEFAULT_MAX_ROWS, maxAgeDays: DEFAULT_MAX_AGE_DAYS };

/**
 * Test seam — lower the retention bounds so a suite can prove pruning with a
 * handful of rows instead of 20,000. Production code never calls this; pass no
 * argument to restore the defaults.
 */
export function __setUserActionRetention({ maxRows, maxAgeDays } = {}) {
  retention = {
    maxRows: Number.isFinite(maxRows) && maxRows > 0 ? Math.floor(maxRows) : DEFAULT_MAX_ROWS,
    maxAgeDays: Number.isFinite(maxAgeDays) && maxAgeDays > 0 ? maxAgeDays : DEFAULT_MAX_AGE_DAYS,
  };
}

// ---------------------------------------------------------------------------
// Pure helpers (exported for focused tests — no I/O, no side effects).
// ---------------------------------------------------------------------------

/** Collapse whitespace and clamp to one LLM-readable line. */
export function clampSummary(text, max = SUMMARY_MAX_CHARS) {
  const line = String(text ?? '').replace(/\s+/g, ' ').trim();
  if (line.length <= max) return line;
  return `${line.slice(0, max - 1).trimEnd()}…`;
}

// Recursive redact+truncate. Returns `{ value, truncated }` where `truncated`
// reports a truncation the CALLER must flag — an object flags its own children,
// so it always reports false upward; an array can't carry a flag, so it hands
// the responsibility to the nearest enclosing object.
function redactNode(value, path, redactedKeys) {
  if (typeof value === 'string') {
    return value.length > PAYLOAD_STRING_MAX_CHARS
      ? { value: `${value.slice(0, PAYLOAD_STRING_MAX_CHARS)}…`, truncated: true }
      : { value, truncated: false };
  }
  if (Array.isArray(value)) {
    let truncated = false;
    const out = value.map((item, index) => {
      const child = redactNode(item, path ? `${path}.${index}` : String(index), redactedKeys);
      truncated = truncated || child.truncated;
      return child.value;
    });
    return { value: out, truncated };
  }
  if (isPlainObject(value)) {
    const out = {};
    let truncated = false;
    for (const [key, child] of Object.entries(value)) {
      const childPath = path ? `${path}.${key}` : key;
      if (isSecretKey(key)) {
        redactedKeys.push(childPath);
        continue;
      }
      const result = redactNode(child, childPath, redactedKeys);
      if (result.truncated) truncated = true;
      out[key] = result.value;
    }
    if (truncated) out.truncated = true;
    return { value: out, truncated: false };
  }
  return { value, truncated: false };
}

/**
 * Drop credential-shaped keys and truncate long strings.
 * @returns {{ payload: object, redactedKeys: string[] }} dotted paths of what was dropped.
 */
export function redactPayload(payload) {
  const redactedKeys = [];
  const { value } = redactNode(isPlainObject(payload) ? payload : {}, '', redactedKeys);
  return { payload: value, redactedKeys };
}

/**
 * Validate + normalize one action into the stored row shape. Throws on an
 * unknown type/actor or a missing dedupe key — all three are programmer errors
 * at a fixed hook site, and a silently-dropped row is worse than a red test.
 */
export function normalizeUserAction(input = {}) {
  const { type, actor = 'user', dedupeKey, summary } = input;
  if (!isUserActionType(type)) {
    throw new Error(`Unknown user action type '${String(type)}' — add it to USER_ACTION_TYPES first`);
  }
  if (!isUserActionActor(actor)) {
    throw new Error(`Unknown user action actor '${String(actor)}' for ${type}`);
  }
  const key = String(dedupeKey ?? '').trim();
  if (!key) throw new Error(`User action ${type} is missing a dedupeKey`);

  const when = input.happenedAt ? new Date(input.happenedAt) : new Date();
  if (Number.isNaN(when.getTime())) throw new Error(`User action ${type} has an invalid happenedAt`);

  // `redactedKeys` is written by the recorder alone — producers hand over raw
  // values and this is the single denylist that decides what is withheld.
  const { payload, redactedKeys } = redactPayload(input.payload);
  if (redactedKeys.length > 0) payload.redactedKeys = [...redactedKeys].sort();

  return {
    id: input.id || uuidv4(),
    type,
    actor,
    happenedAt: when.toISOString(),
    target: input.target ?? null,
    targetName: input.targetName ? clampSummary(input.targetName, SUMMARY_MAX_CHARS) : null,
    success: input.success !== false,
    summary: clampSummary(summary || type),
    payload,
    source: isPlainObject(input.source) ? input.source : {},
    dedupeKey: key,
  };
}

/** Clamp list options to the documented bounds. Exported for the route test. */
export function normalizeListOptions(options = {}) {
  const types = Array.isArray(options.types)
    ? options.types
    : (options.types ? [options.types] : []);
  const rawTypes = [
    ...(options.type ? [options.type] : []),
    ...types,
  ].filter((value) => typeof value === 'string' && value);
  const limit = Number.isFinite(Number(options.limit))
    ? Math.min(MAX_LIST_LIMIT, Math.max(1, Math.floor(Number(options.limit))))
    : DEFAULT_LIST_LIMIT;
  const offset = Number.isFinite(Number(options.offset))
    ? Math.max(0, Math.floor(Number(options.offset)))
    : 0;
  return {
    types: [...new Set(rawTypes)],
    actor: options.actor || null,
    target: options.target || null,
    // Absent (undefined/null) means "either"; only an explicit boolean filters.
    success: typeof options.success === 'boolean' ? options.success : null,
    from: options.from ? new Date(options.from).toISOString() : null,
    to: options.to ? new Date(options.to).toISOString() : null,
    limit,
    offset,
  };
}

// Id breaks the tie so the file backend orders identically to the PG backend's
// `ORDER BY happened_at DESC, id DESC` — otherwise two events stamped in the same
// millisecond would page differently depending on which backend answered.
const newestFirst = (a, b) => (
  String(b.happenedAt).localeCompare(String(a.happenedAt))
  || String(b.id).localeCompare(String(a.id))
);

// Shared by both backends so "what retention keeps" has ONE definition.
function retentionCutoff(now = Date.now()) {
  return new Date(now - retention.maxAgeDays * 24 * 60 * 60 * 1000).toISOString();
}

// ---------------------------------------------------------------------------
// File backend — dev/test escape hatch only.
// ---------------------------------------------------------------------------

// Every file write goes through one tail: the read-modify-write below would
// otherwise let two concurrent records clobber each other's row.
const queueWrite = createFileWriteQueue();

async function loadFileEvents() {
  const raw = await readJSONFile(eventsFile(), { events: [] }, { allowArray: false, strict: true });
  return Array.isArray(raw?.events) ? raw.events : [];
}

function pruneEvents(events) {
  const cutoff = retentionCutoff();
  return events
    .filter((event) => String(event.happenedAt) >= cutoff)
    .sort(newestFirst)
    .slice(0, retention.maxRows);
}

function matchesFilters(event, filters) {
  if (filters.types.length > 0 && !filters.types.includes(event.type)) return false;
  if (filters.actor && event.actor !== filters.actor) return false;
  if (filters.target && event.target !== filters.target) return false;
  if (filters.success !== null && event.success !== filters.success) return false;
  if (filters.from && String(event.happenedAt) < filters.from) return false;
  if (filters.to && String(event.happenedAt) > filters.to) return false;
  return true;
}

function makeFileBackend() {
  return {
    name: 'file',
    record: (event) => queueWrite(async () => {
      // Hoisted above loadFileEvents()/the dedupe check on purpose (#5627
      // review): an un-redirected suite replaying an existing
      // (type, dedupeKey) used to hit the dedupe short-circuit's `return null`
      // BEFORE this guard ever ran, silently no-op'ing past it with no throw —
      // and by then loadFileEvents() had already read the real ledger into the
      // test process regardless. Running the guard first closes both holes.
      await assertTestDataRootRedirected('recordUserAction attempted a file-backend write of');
      const events = await loadFileEvents();
      if (events.some((row) => row.type === event.type && row.dedupeKey === event.dedupeKey)) return null;
      await ensureDir(PATHS.data);
      await atomicWrite(eventsFile(), { events: pruneEvents([...events, event]) });
      return event;
    }),
    list: async (filters) => {
      // Same guard as the write path: an un-redirected suite must not get to READ
      // the developer's live ledger either — those rows are machine-local operator
      // records (docs/decisions/2026-08-08-privacy-records-machine-local.md).
      await assertTestDataRootRedirected('listUserActions attempted a file-backend read of');
      const events = await loadFileEvents();
      return events
        .filter((event) => matchesFilters(event, filters))
        .sort(newestFirst)
        .slice(filters.offset, filters.offset + filters.limit);
    },
  };
}

// ---------------------------------------------------------------------------
// PostgreSQL backend — authoritative. The SQL itself lives in userActionsDb.js
// (taking `db` as an argument) so `npm run test:db` can exercise it: this facade
// selects the FILE backend under NODE_ENV=test, which every DB-backed suite runs
// under. Same split as postRunStore.js / postRunDb.js.
// ---------------------------------------------------------------------------

function makePgBackend(db) {
  return {
    name: 'postgres',
    record: async (event) => {
      const inserted = await insertUserActionEvent(db, event);
      if (!inserted) return null;
      // Retention runs only after a row actually landed — a deduped retry must
      // not pay for two DELETE scans.
      await pruneUserActionEvents(db, { maxRows: retention.maxRows, cutoffIso: retentionCutoff() });
      return event;
    },
    list: (filters) => listUserActionEvents(db, filters),
  };
}

const facade = createPgFileFacade({
  // Vitest's VITEST marker is the resilient test signal when a wrapper drops
  // NODE_ENV; never let a focused route suite fall through to the live Postgres.
  isFile: () => process.env.MEMORY_BACKEND === 'file' || isTestRunner(),
  makeFile: makeFileBackend,
  makePg: () => resolvePgBackend({
    requirement: 'The operator-action ledger requires PostgreSQL — run `npm run setup:db` (dev/test only: set MEMORY_BACKEND=file)',
    loadDb: () => import('../lib/db.js'),
    makePg: makePgBackend,
  }),
});

// ---------------------------------------------------------------------------
// Public surface.
// ---------------------------------------------------------------------------

/**
 * Persist one operator action. Returns the stored event, or `null` when an event
 * with the same `(type, dedupeKey)` already exists (a retried request).
 *
 * @param {object} action
 * @param {string} action.type        One of USER_ACTION_TYPES; unknown values throw.
 * @param {string} [action.actor]     'user' (default) | 'mind' | 'schedule' | 'system'.
 * @param {string} [action.target]    Stable id of what was acted on (task id, agent id).
 * @param {string} [action.targetName] Human label for that target.
 * @param {boolean} [action.success]  Defaults true; pass false only when the handler
 *                                    already holds a structured failure result.
 * @param {string} action.summary     One LLM-readable line, clamped to 240 chars.
 * @param {object} [action.payload]   Structured detail; redacted + truncated here.
 * @param {object} [action.source]    `{ route, method }` or `{ service, fn }`.
 * @param {string} action.dedupeKey   Stable idempotency key; required.
 * @param {string|Date} [action.happenedAt] Defaults to now.
 */
export async function recordUserAction(action) {
  const event = normalizeUserAction(action);
  const backend = await facade.getBackend();
  return backend.record(event);
}

/** Read the ledger newest-first. See `normalizeListOptions` for the clamps. */
export async function listUserActions(options = {}) {
  const filters = normalizeListOptions(options);
  const backend = await facade.getBackend();
  return backend.list(filters);
}

/** Test seam — drop the memoized backend selection so a suite can re-select. */
export function _resetUserActionsBackend() {
  facade.reset();
}