Skip to content

Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. πŸ”₯

4.8 KB
Raw Blame
import { Counter } from "prom-client";

/**
 * What firebill said. `refused` is an explicit `success: false` β€” firebill does
 * not own the event and nobody else does, so usage is being dropped.
 * `ambiguous` is a transport failure: firebill may have accepted it before the
 * answer was lost. Both mean the caller was told false; only `refused` is proof
 * the usage is gone. Alert on the pair.
 */
export const firebillTrackTotal = new Counter({
  name: "firecrawl_firebill_track_total",
  help: "Outcomes of usage events sent to firebill",
  // operation: track|refund   outcome: accepted|refused|ambiguous
  labelNames: ["operation", "outcome"] as const,
});

/**
 * **Who failed, when a firebill call did not produce a usable answer.**
 *
 * A separate series rather than a `cause` label on the counters above, and that
 * is the whole point. `increase()` evaluates per series, so
 * `increase(firecrawl_firebill_check_total{outcome="unavailable"}[15m]) > 5` β€”
 * the live `FirebillCheckUnavailable` rule β€” would silently start comparing 5
 * against each cause on its own: two causes sitting at 4 apiece would be 8
 * unanswered credit checks that nobody is paged about. The counters those rules
 * read stay byte-for-byte what they were; this one adds the detail beside them.
 *
 * - `timeout` / `connection` β€” the request never completed. **A client-side or
 *   transport failure, not firebill**: it never answered, and its own
 *   server-side p99 may be fine (73ms, while ~1,800 of these were logged per
 *   30h against a 5s client deadline).
 * - `non_ok` β€” firebill answered, with a status we cannot use.
 * - `unusable` β€” firebill answered, and the answer is not one we can read.
 *   Never proof that anything was lost.
 * - `refused` β€” firebill answered `success: false`: on a track it did not take
 *   the event, on a check it declined to answer.
 * - `ambiguous` β€” firebill answered "I do not know" (a 504, or
 *   `ambiguous: true`).
 *
 * Only incremented on failure, so a healthy service produces no series at all.
 */
export const firebillFailureCauseTotal = new Counter({
  name: "firecrawl_firebill_failure_cause_total",
  help: "Why a firebill call did not produce a usable answer",
  // operation: track|refund|check|lock|finalize
  // cause: timeout|connection|non_ok|unusable|refused|ambiguous
  labelNames: ["operation", "cause"] as const,
});

/**
 * The durable retry behind a track or refund that firebill never confirmed.
 * Counted where each step happens: the `enqueue*` outcomes by the
 * caller that gave up, `retried`, `settled` and `exhausted` by the
 * retry worker.
 *
 * - `enqueued` β€” handed to the retry queue instead of being dropped.
 * - `enqueue_failed` β€” the queue rejected the hand-off; the event is
 *   unbilled unless someone replays it from the log line.
 * - `enqueue_timeout` β€” the hand-off did not confirm in time and may still
 *   land; check the queue for the key before treating it as lost.
 * - `retried` β€” one re-send under the same idempotency key.
 * - `settled` β€” firebill accepted a re-send.
 * - `exhausted` β€” every attempt failed; see the exhausted value counter
 *   and the "firebill durable retry exhausted" log for the key and amount.
 */
export const firebillTrackRetryTotal = new Counter({
  name: "firecrawl_firebill_track_retry_total",
  help: "Durable retries of usage events firebill did not confirm",
  // operation: track|refund
  // outcome: enqueued|enqueue_failed|enqueue_timeout|retried|settled|exhausted
  labelNames: ["operation", "outcome"] as const,
});

/** The amount, made positive, behind each exhausted durable retry. */
export const firebillTrackRetryExhaustedValueTotal = new Counter({
  name: "firecrawl_firebill_track_retry_exhausted_value_total",
  help: "Usage value whose durable firebill retry was exhausted",
  labelNames: ["operation", "feature"] as const,
});

/** Retries of a firebill call that answered `false` or threw. */
export const firebillRetryTotal = new Counter({
  name: "firecrawl_firebill_retry_total",
  help: "Retried firebill calls, by why the previous attempt failed",
  labelNames: ["reason"] as const, // not_ok | not_success | exception
});

/**
 * What firebill's credit check answered. `denied` is a real "cannot afford it"
 * and becomes a 402; `unavailable` means firebill could not answer, so the
 * caller **failed open** and the request proceeded unauthorized.
 *
 * Watch `unavailable`: unlike a refused charge it is invisible to the customer
 * and to the balance, so a firebill wobble here shows up nowhere else. And for
 * a gateway org `denied` means a partner's pool ran dry, which is a commercial
 * event rather than a fault.
 */
export const firebillCheckTotal = new Counter({
  name: "firecrawl_firebill_check_total",
  help: "Outcomes of credit checks sent to firebill",
  labelNames: ["outcome"] as const, // allowed | denied | unavailable
});