Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS
JavaScript ai-agentsdashboarddeveloper-toolsdigital-twinquantified-selfsecond-brainself-hostedtailscale
/**
* Beeper outbound outbox (fork issue #36, decided on #8, confirmation
* transport decided on #12).
*
* PortOS's send posture is upstream's tier 2 — draft → human approve → human
* send, through ONE service whose only non-test caller is a human HTTP route
* (`server/routes/beeper.js`). `messageSender.sendDraft` is the shape this
* reuses: the `status === 'approved'` gate, the mismatch gate, and the
* one-human-caller rule. What it deliberately does NOT reuse is that service's
* FILE-BACKED draft store — the `beeper_*` tables are `db-primary`, and #8
* rejected straddling two storage classifications. Hence a Beeper-native
* `beeper_outbox` row.
*
* Four rules this module exists to enforce:
*
* 1. **The row is written before the POST.** Intent survives a crash between
* the click and the send, and the row is the one serialization point per
* entry — the `approved → sending` transition is a conditional UPDATE, so
* a double-click cannot double-post.
* 2. **A send is NEVER retried automatically.** Beeper has no idempotency key
* on `POST /v1/chats/{chatID}/messages`, so a retry delivers a second real
* message to a real person. A response-less transport failure leaves the
* row in `awaiting-confirmation`, because the POST may have landed. Only a
* definitive rejection becomes `failed` and offers a new human send.
* 3. **Confirmation is a resolve, never a re-send.** The send is asynchronous
* and answers only `{ chatID, pendingMessageID }`. The row confirms on the
* `message.upserted` invalidation relayed by #33, with a 30-second fallback
* to `GET /v1/chats/{chatID}/messages/{messageID}` — WebSocket delivery is
* at-most-once with no replay, so the socket cannot be the only path.
* Neither path sends anything.
* 4. **A runaway breaker, not a rate quota.** Every send is one human click,
* so a quota paces a human who is already paced. The breaker catches the
* failure this design cannot otherwise see — a software loop — and only a
* human clears it.
*
* No AI drafting and no AI review (#8 decision 8), and no agent reach: no
* `cosToolRegistry` entry, no voice tool, no scheduler. `beeperOutboxHumanGate.test.js`
* asserts that structurally rather than by convention.
*/
import { query, withTransaction } from '../lib/db.js';
import { BeeperApiError, getMessage, listMessagesPage, sendMessage } from './beeperClient.js';
import { normalizeMessageRow, upsertMirroredMessage } from './beeperSync.js';
import { beeperSocketEvents } from './beeperSocketEvents.js';
const LOG_PREFIX = '🫧 Beeper outbox';
/**
* The socket is the fast path; this is the floor under it. 30 s is #12's
* decision: long enough that a healthy `message.upserted` almost always wins
* the race (so the fallback GET is rare rather than routine), short enough
* that a dropped frame does not leave a send unresolved on screen.
*/
export const CONFIRMATION_TIMEOUT_MS = 30_000;
/**
* The code and the recorded reason for a send this process found stranded in
* `sending` at boot — a POST that was in flight when the previous process died.
*
* The outcome is UNKNOWABLE: the request may have been delivered, refused, or
* never have left. Rule 2 above therefore still holds — nothing re-POSTs it.
* The row lands in the existing terminal `failed` state so it cannot spin
* forever, while this error code tells the client to show uncertainty with no
* Retry control. The copy says what actually happened rather than claiming a
* delivery verdict PortOS does not have. The client renders this sentence
* verbatim off the code, so the same string is repeated in
* `client/src/components/messages/beeper/BeeperThread.jsx` — the two bundles
* cannot share a module, so they share a test instead.
*/
export const SEND_INTERRUPTED_CODE = 'SEND_INTERRUPTED';
export const SEND_INTERRUPTED_MESSAGE = 'Delivery unconfirmed: PortOS restarted mid-send. Check the chat before retrying.';
/** A send whose POST may have landed, but whose response never reached PortOS. */
export const DELIVERY_UNCONFIRMED_CODE = 'DELIVERY_UNCONFIRMED';
export const DELIVERY_UNCONFIRMED_MESSAGE = 'Delivery unconfirmed; check the chat before sending again.';
/**
* PostgreSQL's `undefined_table`. The boot reconcile runs before the DB phase
* has called `ensureSchema()`, so a brand-new install's very first start can
* reach it with no `beeper_outbox` table — which is not a fault to narrate: an
* absent table holds no interrupted sends. Every other error still logs.
*/
const UNDEFINED_TABLE = '42P01';
/**
* Runaway-breaker thresholds. Deliberately far above anything a human at a
* composer produces and far below what an unattended loop produces in the same
* minute: eleven sends inside a minute is one every 5.5 seconds, sustained,
* which is a fault rather than a conversation. The consecutive-failure arm
* catches the other loop shape — a caller that keeps firing at a Beeper that
* keeps refusing.
*/
export const BREAKER_WINDOW_MS = 60_000;
export const BREAKER_MAX_SENDS_IN_WINDOW = 10;
export const BREAKER_MAX_CONSECUTIVE_FAILURES = 3;
const DEFAULT_RUNTIME = Object.freeze({
now: () => Date.now(),
setTimeout: (fn, ms) => setTimeout(fn, ms),
clearTimeout: (handle) => clearTimeout(handle),
});
let runtime = { ...DEFAULT_RUNTIME };
// Breaker + confirmation state is in-memory ON PURPOSE. The failure it defends
// against is a live in-process loop, so process-lifetime scope is the right
// scope: a restart clears it, and a loop that survived the restart re-trips it
// within the same window. Persisting it would instead let one bad minute wedge
// the composer across restarts with no way back except the same human click.
let sendTimestamps = [];
let consecutiveFailures = 0;
let breaker = { tripped: false, reason: null, trippedAt: null };
// outboxId → { chatId, body, pendingMessageId, requestedAt,
// deliveryOutcomeUnknown, timer, resolving }
const pendingConfirmations = new Map();
const localTransitions = new Map();
const activeSends = new Set();
const reconciliationOperations = new Map();
export const PERSISTENCE_RETRY_DELAYS_MS = Object.freeze([1000, 2000, 4000]);
const PERSISTENCE_RECOVERY_MESSAGE = 'Delivery unconfirmed: saving the send outcome failed. Check delivery to reconcile without sending again.';
function invalidateOutbox(id) {
beeperSocketEvents.emit('invalidate', { kind: 'outbox.updated', chatID: null, ids: [id], seq: null, ts: new Date(runtime.now()).toISOString() });
}
function recoveryView(entry) {
if (!entry) return entry;
if (!localTransitions.has(entry.id)) return entry;
return { ...entry, state: 'awaiting-confirmation', errorCode: DELIVERY_UNCONFIRMED_CODE, errorMessage: PERSISTENCE_RECOVERY_MESSAGE };
}
// Own only local persistence here: the closure must NEVER contain a send POST.
// Retain it after the bounded timer budget, so a human can resume it later.
async function runLocalTransition(id, transition) {
if (transition.running) return transition.running;
runtime.clearTimeout(transition.timer);
transition.running = transition.write().then((value) => {
if (localTransitions.get(id) === transition) localTransitions.delete(id);
transition.committed?.(value);
invalidateOutbox(id);
return value;
}).catch(() => {
if (transition.attempt < PERSISTENCE_RETRY_DELAYS_MS.length) {
const delay = PERSISTENCE_RETRY_DELAYS_MS[transition.attempt++];
transition.timer = runtime.setTimeout(() => {
runLocalTransition(id, transition).catch((err) => console.error(`${LOG_PREFIX}: local recovery failed (${err.code || 'unknown'})`));
}, delay);
}
invalidateOutbox(id);
return null;
}).finally(() => { transition.running = null; });
return transition.running;
}
function persistTransition(id, write, committed) {
const transition = { write, committed, attempt: 0, running: null, timer: null };
localTransitions.set(id, transition);
return runLocalTransition(id, transition);
}
let invalidateListenerAttached = false;
/** Test seam for injected time (mirrors `beeperSocket.js`'s runtime object). */
export function configureOutboxRuntime(overrides = {}) {
runtime = { ...DEFAULT_RUNTIME, ...overrides };
}
export function resetOutboxRuntime() {
runtime = { ...DEFAULT_RUNTIME };
}
// ---------------------------------------------------------------------------
// Runaway breaker
// ---------------------------------------------------------------------------
function trip(reason) {
breaker = { tripped: true, reason, trippedAt: new Date(runtime.now()).toISOString() };
console.error(`${LOG_PREFIX}: runaway breaker tripped (${reason}) — sends blocked until a human clears it`);
}
/**
* The breaker's read model, surfaced on `GET /api/beeper/status`. `tripped` is
* the gate; the two counters ride along so the card can say WHY rather than
* just that something is blocked.
*/
export function getOutboxBreakerState() {
const cutoff = runtime.now() - BREAKER_WINDOW_MS;
return {
tripped: breaker.tripped,
reason: breaker.reason,
trippedAt: breaker.trippedAt,
sendsInWindow: sendTimestamps.filter((ts) => ts > cutoff).length,
consecutiveFailures,
windowMs: BREAKER_WINDOW_MS,
maxSendsInWindow: BREAKER_MAX_SENDS_IN_WINDOW,
maxConsecutiveFailures: BREAKER_MAX_CONSECUTIVE_FAILURES,
};
}
/**
* The ONLY way back. Reachable exclusively from `POST /api/beeper/outbox/breaker/clear`
* — no timer, no backoff, no automatic recovery: a breaker that resets itself
* is a delay, not a breaker.
*/
export function clearOutboxBreaker() {
breaker = { tripped: false, reason: null, trippedAt: null };
sendTimestamps = [];
consecutiveFailures = 0;
console.log(`${LOG_PREFIX}: runaway breaker cleared by a human action`);
return getOutboxBreakerState();
}
function assertBreakerClosed() {
if (!breaker.tripped) return;
throw new BeeperApiError(
`Beeper sending is blocked by the runaway breaker (${breaker.reason}). Clear it to send again.`,
{ status: 429, code: 'OUTBOX_BREAKER_OPEN', retryable: false },
);
}
/**
* Count this attempt against the window BEFORE the POST, and refuse the one
* that crosses the line. Counting after the POST would let the loop's own
* first burst through before the breaker ever saw it.
*/
function registerSendAttempt() {
const now = runtime.now();
sendTimestamps = sendTimestamps.filter((ts) => ts > now - BREAKER_WINDOW_MS);
sendTimestamps.push(now);
if (sendTimestamps.length > BREAKER_MAX_SENDS_IN_WINDOW) {
trip(`more than ${BREAKER_MAX_SENDS_IN_WINDOW} sends in ${BREAKER_WINDOW_MS / 1000}s`);
assertBreakerClosed();
}
}
function registerSendFailure() {
consecutiveFailures += 1;
if (consecutiveFailures >= BREAKER_MAX_CONSECUTIVE_FAILURES) {
trip(`${consecutiveFailures} consecutive send failures`);
}
}
function registerSendSuccess() {
consecutiveFailures = 0;
}
// ---------------------------------------------------------------------------
// Rows
// ---------------------------------------------------------------------------
const ENTRY_COLUMNS = `id, conversation_id AS "conversationId", chat_id AS "chatId", body, state,
pending_message_id AS "pendingMessageId", message_id AS "messageId",
error_code AS "errorCode", error_message AS "errorMessage",
send_requested_at AS "requestedAt", created_at AS "createdAt", updated_at AS "updatedAt", approved_at AS "approvedAt", sent_at AS "sentAt"`;
async function readEntry(id) {
const result = await query(`SELECT ${ENTRY_COLUMNS} FROM beeper_outbox WHERE id = $1`, [id]);
return result?.rows?.[0] ?? null;
}
/** Every outbox row for one conversation, newest first — including failed ones. */
export async function listOutboxEntries({ conversationId, limit = 50 } = {}) {
const result = await query(
`SELECT ${ENTRY_COLUMNS} FROM beeper_outbox
WHERE conversation_id = $1 ORDER BY created_at DESC LIMIT $2`,
[conversationId, Math.min(200, Math.max(1, Math.trunc(limit) || 50))],
);
return Array.isArray(result?.rows) ? result.rows.map(recoveryView) : [];
}
/**
* Step one of the two-step send: a human composed this text and submitted it.
* The row lands `approved` (with `approved_at` stamped) because the create call
* itself IS the approval — it is reachable only from a human HTTP route, and
* the send is a second, separate human action against the row this returns.
* `draft` exists in the state CHECK for a later persisted/AI-assisted draft and
* has no writer here; the send gate keys on `approved`, so nothing can send
* from it.
*/
export async function createOutboxEntry({ conversationId, body }) {
const text = typeof body === 'string' ? body.trim() : '';
if (!text) {
throw new BeeperApiError('An outbox entry needs a non-empty message body', {
status: 400, code: 'OUTBOX_EMPTY_BODY', retryable: false,
});
}
const conversation = await query(
'SELECT id, source_chat_id AS "sourceChatId" FROM beeper_conversations WHERE id = $1',
[conversationId],
);
const chatId = conversation?.rows?.[0]?.sourceChatId;
if (!chatId) {
throw new BeeperApiError('Conversation not found in the Beeper mirror', {
status: 404, code: 'CONVERSATION_NOT_FOUND', retryable: false,
});
}
const inserted = await query(
`INSERT INTO beeper_outbox (conversation_id, chat_id, body, state, approved_at)
VALUES ($1, $2, $3, 'approved', NOW())
RETURNING ${ENTRY_COLUMNS}`,
[conversationId, chatId, text],
);
console.log(`${LOG_PREFIX}: entry approved for chat (${text.length} chars)`);
return inserted.rows[0];
}
/**
* States that mean PortOS has already ADDRESSED this conversation — the POST
* reached Beeper, whatever the mirror has confirmed since. `sent` alone is the
* wrong test: an `awaiting-confirmation` row is a send that left this machine
* and was very likely delivered (the unresolved case is deliberately not
* `failed` for exactly that reason), and `sending` is one in flight right now.
* Counting only `sent` re-asks the first-contact question on the next message
* to someone PortOS has already messaged — which is precisely the prompt the
* design says must fire once and never again, or the user learns to click
* through it.
*/
const CONTACTED_STATES = ['sent', 'awaiting-confirmation', 'sending'];
/**
* Whether PortOS has ever ADDRESSED this conversation. First contact is the
* case #8 decision 5 puts a confirmation on — a first message to a possibly
* mis-resolved handle on a possibly wrong network — and it is the only one,
* because confirming every reply trains the user to click through.
*
* Keyed on EITHER source of outbound history, not the outbox alone (fork
* issue #82): a mirrored `beeper_messages` row with `is_sender` true means the
* user has messaged this chat from some client, which is exactly the "have I
* addressed this recipient and network before" question the confirmation
* exists to answer — restricting it to sends this particular PortOS install
* made re-asked the question on every chat with older phone/desktop history,
* which is precisely the "fires when it should not" case #82 reports. The
* outbox arm stays for the gap the mirror cannot see: a send whose POST left
* this machine but has not (yet, or ever) come back through the mirror as an
* `isSender` message — `awaiting-confirmation` and `sending` are exactly that
* gap, per `CONTACTED_STATES` above. Either source alone is sufficient; the
* confirmation is reserved for a conversation neither source has any
* outbound history for.
*/
export async function isFirstContact(conversationId) {
const outboxSend = await query(
`SELECT 1 FROM beeper_outbox
WHERE conversation_id = $1 AND state = ANY($2::text[]) LIMIT 1`,
[conversationId, CONTACTED_STATES],
);
if ((outboxSend?.rows?.length ?? 0) > 0) return false;
const mirroredSend = await query(
`SELECT 1 FROM beeper_messages
WHERE conversation_id = $1 AND is_sender = TRUE LIMIT 1`,
[conversationId],
);
return (mirroredSend?.rows?.length ?? 0) === 0;
}
async function markFailed(id, code, message) {
await query(
`UPDATE beeper_outbox SET state = 'failed', error_code = $2, error_message = $3, updated_at = NOW()
WHERE id = $1`,
[id, code || 'SEND_FAILED', String(message || '').slice(0, 2000)],
);
}
async function markAwaitingConfirmation(id, {
pendingMessageId = null, errorCode = null, errorMessage = null,
} = {}) {
const updated = await query(
`UPDATE beeper_outbox SET state = 'awaiting-confirmation', pending_message_id = $2,
error_code = $3, error_message = $4, updated_at = NOW()
WHERE id = $1 RETURNING ${ENTRY_COLUMNS}`,
[id, pendingMessageId, errorCode, errorMessage],
);
return updated.rows[0];
}
/** No HTTP response means the server cannot prove whether the POST landed. */
function isDeliveryOutcomeUnknown(err) {
return err?.status === 0 || err?.code === 'NETWORK_ERROR';
}
/**
* Discard a row the human declined to send — the first-contact confirmation's
* "Cancel" (#53's fix; the original design deliberately left the row
* `approved` and visible, which the client had no way to render or dismiss,
* so it surfaced as a permanent phantom "Sending…" bubble). Reachable only
* while the row is `approved`: nothing has POSTed to Beeper for it yet, so
* removing it is a pure local record change, never an unsend. Anything past
* `approved` is a record of a real send attempt and stays — the client
* composes a new entry rather than discarding one that already touched the
* wire.
*/
export async function discardOutboxEntry(id) {
const entry = await readEntry(id);
if (!entry) {
throw new BeeperApiError('Outbox entry not found', {
status: 404, code: 'OUTBOX_ENTRY_NOT_FOUND', retryable: false,
});
}
if (entry.state !== 'approved') {
throw new BeeperApiError(
`Outbox entry is "${entry.state}", must be "approved" to discard — it has already been sent or attempted.`,
{ status: 409, code: 'OUTBOX_INVALID_STATE', retryable: false },
);
}
await query("DELETE FROM beeper_outbox WHERE id = $1 AND state = 'approved'", [id]);
}
// ---------------------------------------------------------------------------
// Send
// ---------------------------------------------------------------------------
/**
* Step two: perform the send. Every gate below runs BEFORE the row is claimed,
* so a refused send leaves the row exactly as it was and re-sendable by the
* same human action.
*
* `confirmFirstContact` must be explicitly `true` on the first outbound message
* to a conversation. The client obtains it from an inline confirmation row, not
* a `window.confirm` (client conventions forbid the latter).
*/
export async function sendOutboxEntry(id, { confirmFirstContact = false } = {}) {
const entry = await readEntry(id);
if (!entry) {
throw new BeeperApiError('Outbox entry not found', {
status: 404, code: 'OUTBOX_ENTRY_NOT_FOUND', retryable: false,
});
}
if (entry.state !== 'approved') {
throw new BeeperApiError(
`Outbox entry is "${entry.state}", must be "approved" to send. A failed or already-sent entry is never re-sent in place — compose a new message.`,
{ status: 409, code: 'OUTBOX_INVALID_STATE', retryable: false },
);
}
assertBreakerClosed();
if (!confirmFirstContact && await isFirstContact(entry.conversationId)) {
throw new BeeperApiError(
'This is the first message PortOS has ever sent to this conversation — confirm the recipient and network before sending.',
{ status: 409, code: 'FIRST_CONTACT_CONFIRMATION_REQUIRED', retryable: false },
);
}
registerSendAttempt();
// The serialization point. A second concurrent send of the same row loses
// this conditional UPDATE and never reaches the POST.
const requestedAt = runtime.now();
const claimed = await query(
"UPDATE beeper_outbox SET state = 'sending', send_requested_at = $2::timestamptz, updated_at = NOW() WHERE id = $1 AND state = 'approved' RETURNING id",
[id, new Date(requestedAt).toISOString()],
);
if ((claimed?.rowCount ?? 0) !== 1) {
throw new BeeperApiError('Outbox entry is already being sent', {
status: 409, code: 'OUTBOX_INVALID_STATE', retryable: false,
});
}
// Retry is OFF (the client's send-safe default): no idempotency key means a
// retried POST is a second real message. One attempt, one row, no second POST.
activeSends.add(id);
const result = await sendMessage(entry.chatId, { text: entry.body })
.then((value) => ({ ok: true, value }))
.catch((err) => ({ ok: false, err }));
activeSends.delete(id);
const confirmation = { ...entry, pendingMessageId: null, requestedAt };
if (!result.ok) {
const err = result.err;
if (isDeliveryOutcomeUnknown(err)) {
registerSendFailure();
const updated = await persistTransition(id,
() => markAwaitingConfirmation(id, { errorCode: DELIVERY_UNCONFIRMED_CODE, errorMessage: DELIVERY_UNCONFIRMED_MESSAGE }),
() => armConfirmation({ ...confirmation, deliveryOutcomeUnknown: true }));
return updated ?? recoveryView(entry);
}
await persistTransition(id, () => markFailed(id, err?.code, err?.message));
registerSendFailure();
console.error(`${LOG_PREFIX}: send failed (${err?.code || 'SEND_FAILED'})`);
throw new BeeperApiError(err?.message || 'Beeper send failed', {
status: err?.status === 0 || !err?.status ? 0 : err.status,
code: err?.code || 'SEND_FAILED',
retryable: false,
details: { outboxId: id },
});
}
registerSendSuccess();
const pendingMessageId = typeof result.value?.pendingMessageID === 'string' ? result.value.pendingMessageID : null;
const updated = await persistTransition(id,
() => markAwaitingConfirmation(id, { pendingMessageId }),
() => armConfirmation({ ...confirmation, pendingMessageId }));
console.log(`${LOG_PREFIX}: sent, awaiting confirmation${pendingMessageId ? '' : ' (no pending id returned)'}`);
return updated ?? recoveryView(entry);
}
// ---------------------------------------------------------------------------
// Confirmation (socket first, 30s GET fallback)
// ---------------------------------------------------------------------------
function detachInvalidateListener() {
if (!invalidateListenerAttached || pendingConfirmations.size > 0) return;
beeperSocketEvents.off('invalidate', handleInvalidation);
invalidateListenerAttached = false;
}
/**
* Relayed invalidation frames carry ids and kinds only — never bodies — so a
* frame is a PROMPT to resolve, not the resolution. Any `message.upserted` on a
* chat with a send in flight triggers the same lookup the timer would have run,
* just sooner.
*
* Runs outside the Express lifecycle (an EventEmitter callback), so the
* rejection is caught and logged here rather than crashing the process.
*/
function handleInvalidation(frame) {
if (frame?.kind !== 'message.upserted') return;
for (const [id, pending] of pendingConfirmations) {
if (frame.chatID && frame.chatID !== pending.chatId) continue;
resolveConfirmation(id, 'socket').catch((err) => {
console.error(`${LOG_PREFIX}: socket-triggered confirmation failed: ${err.message}`);
});
}
}
/**
* `requestedAt` defaults to now because the ordinary caller is the send that
* just happened. The boot reconcile passes the row's persisted send moment
* instead: it is the floor `lookupSentMessage` matches timestamps against, so
* dating a week-old re-armed row to boot time would reject the very message it
* is looking for.
*/
function armConfirmation({
id, chatId, conversationId, body, pendingMessageId,
requestedAt = runtime.now(), deliveryOutcomeUnknown = false,
}) {
const timer = runtime.setTimeout(() => {
resolveConfirmation(id, 'fallback-timeout').catch((err) => {
console.error(`${LOG_PREFIX}: fallback confirmation failed: ${err.message}`);
});
}, CONFIRMATION_TIMEOUT_MS);
pendingConfirmations.set(id, {
chatId, conversationId, body, pendingMessageId, requestedAt,
deliveryOutcomeUnknown, timer, resolving: false,
});
if (!invalidateListenerAttached) {
beeperSocketEvents.on('invalidate', handleInvalidation);
invalidateListenerAttached = true;
}
}
/**
* Find the message this send became. Two lookups, in order, and NEITHER sends:
*
* 1. `GET /v1/chats/{chatID}/messages/{messageID}` with the `pendingMessageID`
* — the spec states that endpoint accepts a pending id, a final id, or a
* Matrix event id, which is exactly what makes the resolve step work.
* 2. Failing that (no pending id came back, or the id 404s because the network
* assigned a final one), the chat's newest page, matched on
* `isSender` + the exact body + a timestamp at or after the send. The body
* match is deliberately exact: a near-match is a different message.
*/
async function lookupSentMessage(pending) {
if (pending.pendingMessageId) {
const direct = await getMessage(pending.chatId, pending.pendingMessageId)
.then((value) => value)
.catch((err) => {
if (err?.code === 'NOT_FOUND') return null;
throw err;
});
if (direct?.id) return direct;
}
const page = await listMessagesPage(pending.chatId);
const items = Array.isArray(page?.items) ? page.items : [];
// A normal send response proves the POST landed, so tolerate clock skew (and
// old payloads without a timestamp) while finding its final id. When the
// response itself was lost, body alone is not evidence: an earlier identical
// message could otherwise turn an unknowable outcome into a false `sent`.
// Be conservative there and require a valid network timestamp at or after
// this attempt. Clock skew may leave a real send unconfirmed, which is safer
// than attaching a prior message and inviting the wrong delivery verdict.
const floor = pending.deliveryOutcomeUnknown
? pending.requestedAt
: pending.requestedAt - 60_000;
return items.find((message) => {
if (message?.isSender !== true || message?.text !== pending.body) return false;
const messageTime = Date.parse(message?.timestamp ?? '');
return pending.deliveryOutcomeUnknown
? Number.isFinite(messageTime) && messageTime >= floor
: !message?.timestamp || (Number.isFinite(messageTime) && messageTime >= floor);
}) ?? null;
}
/** Mirror confirmed sends through the same version/retention gate as ingestion.
* PortOS knows this is outbound even when the source omits isSender.
*/
async function mirrorSentMessage(client, conversationId, message, observedAt) {
const row = normalizeMessageRow(message, observedAt);
if (!row.id) return;
await upsertMirroredMessage(client, conversationId, { ...row, isSender: true });
}
function releasePending(id) {
const pending = pendingConfirmations.get(id);
if (pending?.timer !== undefined) runtime.clearTimeout(pending.timer);
pendingConfirmations.delete(id);
detachInvalidateListener();
}
/**
* Resolve one in-flight send. Idempotent by construction: the `resolving` flag
* stops the socket path and the timer path overlapping, and the terminal UPDATE
* is conditional on `awaiting-confirmation`, so whichever path lands first is
* the one that writes.
*
* The unresolved case is deliberately NOT `failed`. A send whose message cannot
* be found may well have been delivered, and marking it failed invites a
* re-send — the one mistake that cannot be taken back, since Beeper's own
* `DELETE` unsends for everyone. It stays `awaiting-confirmation` with a
* recorded reason, which reads as "sent, unconfirmed" and offers no re-send.
*/
async function resolveConfirmation(id, reason, { terminal = false } = {}) {
const pending = pendingConfirmations.get(id);
if (!pending || localTransitions.has(id)) return null;
// A `manual` probe is terminal ONLY when the caller armed the row itself —
// a row stranded across a restart, whose real deadline elapsed long ago. When
// a live timer is already counting down, the send is seconds old and the user
// simply pressed "Check delivery" early: a null lookup there means "not yet",
// not "never". Treating it as the deadline would record a false 30s verdict
// AND release the pending entry, disarming the timer and the socket listener
// that were about to confirm it.
if (reason === 'fallback-timeout' || (reason === 'manual' && terminal)) pending.deadlineReached = true;
// The one-shot timer can fire during a socket lookup. Let that lookup own
// the deadline outcome rather than dropping the only fallback signal.
if (pending.resolving) return null;
pending.resolving = true;
const observedAt = new Date(runtime.now()).toISOString();
const found = await lookupSentMessage(pending)
.then((value) => ({ ok: true, value }))
.catch((err) => ({ ok: false, err }));
pending.resolving = false;
if (pendingConfirmations.get(id) !== pending) return null;
if (pending.deadlineReached) reason = 'fallback-timeout';
if (!found.ok) {
// A lookup failure on the socket path costs nothing — the timer is still
// armed. On the fallback path it is the end of the road.
console.error(`${LOG_PREFIX}: confirmation lookup failed (${found.err?.code || 'unknown'})`);
if (reason === 'fallback-timeout') {
await noteUnresolved(
id,
`Confirmation lookup failed: ${found.err?.code || 'unknown error'}`,
pending.deliveryOutcomeUnknown,
);
}
return null;
}
const message = found.value;
if (!message) {
if (reason === 'fallback-timeout') {
await noteUnresolved(
id,
'Beeper reported no matching message within 30s — it may still have been delivered, so it was not re-sent.',
pending.deliveryOutcomeUnknown,
);
}
return null;
}
const status = typeof message?.sendStatus?.status === 'string' ? message.sendStatus.status : null;
if (status?.startsWith('FAIL')) {
await persistTransition(id,
() => markFailed(id, `SEND_${status}`, message?.sendStatus?.message || message?.sendStatus?.reason || 'Beeper reported a failed send'),
() => releasePending(id));
return null;
}
return persistTransition(id, () => withTransaction(async (client) => {
const settled = await client.query(
`UPDATE beeper_outbox SET state = 'sent', message_id = $2, sent_at = COALESCE($3::timestamptz, NOW()),
error_code = NULL, error_message = NULL, updated_at = NOW()
WHERE id = $1 AND state = 'awaiting-confirmation' RETURNING ${ENTRY_COLUMNS}`,
[id, String(message.id), message?.timestamp ?? null],
);
if ((settled?.rowCount ?? 0) !== 1) return null;
await mirrorSentMessage(client, pending.conversationId, message, observedAt);
return settled.rows[0];
}), () => {
releasePending(id);
beeperSocketEvents.emit('invalidate', {
kind: 'message.upserted', chatID: pending.chatId, ids: [String(message.id)],
seq: null, ts: new Date(runtime.now()).toISOString(),
});
console.log(`${LOG_PREFIX}: send confirmed via ${reason}`);
});
}
/** Record why a send is unconfirmed without moving it out of flight. */
async function noteUnresolved(id, message, deliveryOutcomeUnknown = false) {
await persistTransition(id, () => query(
`UPDATE beeper_outbox SET error_code = $2, error_message = $3, updated_at = NOW()
WHERE id = $1 AND state = 'awaiting-confirmation'`,
[id, deliveryOutcomeUnknown ? DELIVERY_UNCONFIRMED_CODE : 'CONFIRMATION_UNRESOLVED', message],
), () => releasePending(id));
console.warn(`${LOG_PREFIX}: send unconfirmed — never re-sent`);
}
/**
* Drop every armed confirmation timer. Used by the suite between cases; there
* is no production caller, because a process that is exiting takes its timers
* with it and the durable state is the row, not the timer.
*/
export function cancelPendingConfirmations() {
for (const id of [...pendingConfirmations.keys()]) releasePending(id);
for (const transition of localTransitions.values()) runtime.clearTimeout(transition.timer);
localTransitions.clear();
activeSends.clear();
reconciliationOperations.clear();
}
/** Human-triggered reconciliation. Remote operations are GETs only. */
export function reconcileOutboxEntry(id) {
if (reconciliationOperations.has(id)) return reconciliationOperations.get(id);
const operation = reconcileEntry(id).finally(() => reconciliationOperations.delete(id));
reconciliationOperations.set(id, operation);
return operation;
}
async function reconcileEntry(id) {
let entry = await readEntry(id);
if (!entry) throw new BeeperApiError('Outbox entry not found', { status: 404, code: 'OUTBOX_ENTRY_NOT_FOUND' });
if (activeSends.has(id)) return entry;
const transition = localTransitions.get(id);
if (transition) {
await runLocalTransition(id, transition);
if (localTransitions.has(id)) return recoveryView(entry);
entry = await readEntry(id);
}
const uncertainFailure = entry.state === 'failed' && [SEND_INTERRUPTED_CODE, DELIVERY_UNCONFIRMED_CODE, 'NETWORK_ERROR'].includes(entry.errorCode);
if (!['sending', 'awaiting-confirmation'].includes(entry.state) && !uncertainFailure) return entry;
// Armed here means the row had no live in-process timer — it was stranded by
// a restart, so its real deadline is long past and this lookup is its last
// word. A row that already had one is still inside its 30s window.
const stranded = !pendingConfirmations.has(id);
if (stranded) {
const unknown = entry.state === 'sending' || uncertainFailure || entry.errorCode === DELIVERY_UNCONFIRMED_CODE;
const requestedAt = persistedSendMoment(entry);
if (entry.state !== 'awaiting-confirmation') {
entry = await markAwaitingConfirmation(id, { errorCode: DELIVERY_UNCONFIRMED_CODE, errorMessage: DELIVERY_UNCONFIRMED_MESSAGE });
}
armConfirmation({ ...entry, requestedAt, deliveryOutcomeUnknown: unknown });
}
await resolveConfirmation(id, 'manual', { terminal: stranded });
return recoveryView(await readEntry(id));
}
// ---------------------------------------------------------------------------
// Boot reconcile
// ---------------------------------------------------------------------------
/** The send moment a re-armed row is matched against, from what the row kept. */
function persistedSendMoment(row) {
const stamped = Date.parse(row?.requestedAt ?? row?.updatedAt ?? row?.createdAt ?? '');
return Number.isFinite(stamped) ? stamped : runtime.now();
}
/**
* Reconcile the two NON-TERMINAL states against a process that no longer
* exists. Called once per server start, from `bootstrap.js`, regardless of
* whether the Beeper feature is armed — a stranded row is stranded whether or
* not ingestion is switched on.
*
* `sending` and `awaiting-confirmation` are exit-only through in-memory state:
* `pendingConfirmations`, its timer, and the socket listener. A restart mid-
* flight takes all three with it. Boot reconciles automatically; a human can
* also invoke the lookup-only reconciliation route without restarting.
*
* Two different faults, two different answers:
*
* - **`sending` → `failed` / `SEND_INTERRUPTED`.** The POST was in flight; its
* outcome is unknowable. Never auto-resent (rule 2 has no exception for a
* crash — Beeper has no idempotency key, so a "recovery" resend is a second
* real message). `failed` terminates the spinner, while the error code keeps
* the client from presenting the ordinary definitive-failure Retry control.
* - **`awaiting-confirmation` → re-armed.** The send DID leave; only the
* resolve was lost. Re-arming replays the lookup from the row's own
* `chat_id` / `pending_message_id` / `body`, which is a read on both paths
* (`getMessage`, then the newest page) and sends nothing. A row already
* recorded `CONFIRMATION_UNRESOLVED` is re-armed too: the message may have
* landed in the mirror since, and one more GET can only upgrade the record.
*
* Terminal rows (`sent`, `failed`) are not touched. Idempotent: the second call
* finds no `sending` rows left, and skips every `awaiting-confirmation` row the
* first call already armed rather than stacking a second timer on it.
*
* @returns {Promise<{interrupted: number, rearmed: number}>}
*/
export async function reconcileOutboxOnBoot() {
const interrupted = await query(
`UPDATE beeper_outbox SET state = 'failed', error_code = $1, error_message = $2
WHERE state = 'sending' RETURNING id`,
[SEND_INTERRUPTED_CODE, SEND_INTERRUPTED_MESSAGE],
).catch((err) => {
if (err?.code === UNDEFINED_TABLE) return null;
throw err;
});
// The table does not exist yet, so neither do the rows this would reconcile.
if (!interrupted) return { interrupted: 0, rearmed: 0 };
// JSON access tolerates the first upgrade boot, which can precede ensureSchema.
// Old rows have no immutable timestamp: keep their prior conservative floor.
const inFlight = await query(
`SELECT id, conversation_id AS "conversationId", chat_id AS "chatId", body,
pending_message_id AS "pendingMessageId", error_code AS "errorCode",
created_at AS "createdAt", updated_at AS "updatedAt",
to_jsonb(beeper_outbox)->>'send_requested_at' AS "requestedAt"
FROM beeper_outbox WHERE state = 'awaiting-confirmation'`,
);
let rearmed = 0;
for (const row of inFlight?.rows ?? []) {
if (pendingConfirmations.has(row.id)) continue;
armConfirmation({
id: row.id,
chatId: row.chatId,
conversationId: row.conversationId,
body: row.body,
pendingMessageId: row.pendingMessageId,
requestedAt: persistedSendMoment(row),
deliveryOutcomeUnknown: row.errorCode === DELIVERY_UNCONFIRMED_CODE,
});
rearmed += 1;
}
const counts = { interrupted: interrupted.rows?.length ?? 0, rearmed };
if (counts.interrupted || counts.rearmed) {
console.log(`${LOG_PREFIX}: boot reconcile — ${counts.interrupted} interrupted send(s) failed, ${counts.rearmed} awaiting confirmation re-armed`);
}
return counts;
}
/** The outbox's slice of `GET /api/beeper/status`. */
export function getOutboxStatus() {
return {
breaker: getOutboxBreakerState(),
awaitingConfirmation: pendingConfirmations.size,
};
}