Skip to content

Turbocharge Claude Code, Cursor, Codex, Gemini & every coding agent: faster, cheaper, with contextual understanding specific to your codebase.

feat(app): run each review in its own process so the server stays responsive (#281)

anirudhkumar-nanonetsf2ca4a7bab92
diff --git a/src/app/review-process.ts b/src/app/review-process.ts
new file mode 100644
index 00000000..1be03f03
--- /dev/null
+++ b/src/app/review-process.ts
@@ -0,0 +1,268 @@
+/**
+ * Running a review somewhere it cannot stop the App answering.
+ *
+ * A review is synchronous from end to end, and not in one place we could fix: the
+ * pull request is fetched with `spawnSync` (`./checkout.ts`), the whole repository
+ * is parsed in one `files.forEach` loop of native tree-sitter calls
+ * (`graph/build.ts`), the graph is written with a blocking `JSON.stringify`, and
+ * the viewer page is exported synchronously too. In-process that holds the event
+ * loop for the entire review.
+ *
+ * That is not a theory. On the deployed App, probing `GET /p/<id>` from outside
+ * every ten seconds while one review ran: three consecutive timeouts (no response
+ * at all, not even with `--max-time 45`), then 404 in 0.9s the moment the queue
+ * emptied. Meanwhile the box's load average was 0.96 on eight vCPUs — one thread
+ * pegged, seven cores idle — and the App's own logs put a single review at 92s to
+ * 274s ("3 changed → 0 affected in 180134ms"). So a *single* in-flight review took
+ * the whole HTTP surface down for minutes: the container's 5s `HEALTHCHECK` failed,
+ * Docker marked the container unhealthy, and pages the App had already produced
+ * were unreachable. On a busy repository that is most of the working day.
+ *
+ * Neither bounding the queue nor lowering `GRAFT_CONCURRENCY` touches this —
+ * concurrency was already 2 and one job is enough. Cooperative yielding inside the
+ * parse loop does not either: a `git fetch` of a large repository is one
+ * uninterruptible `spawnSync` that can legitimately take a minute, and it is not
+ * even using a core while it blocks. The work has to leave the process.
+ *
+ * A child process rather than a `worker_threads` worker, deliberately:
+ *
+ * - **The native grammars keep loading the way they already do.** `graph/extract.ts`
+ * imports nine tree-sitter addons at module scope and constructs a `Parser` there
+ * too, and `graph/generic.ts` loads more through `createRequire`. In a worker all
+ * of that is re-instantiated per thread, which puts the N-API-in-worker behaviour
+ * of nine third-party native modules between the App and every review. A fork runs
+ * byte-identical code to what production runs today.
+ * - **A review can be killed.** A wedged parse or a 10-minute clone is a process
+ * to SIGKILL, not a thread with no safe way to stop it.
+ * - **A crash stops being an outage.** An OOM kill or a grammar segfault on a
+ * stranger's source now costs one review and is reported as a failure, instead
+ * of taking the HTTP server with it. This process clones code written by
+ * people who opened a pull request; that isolation is worth having on its own.
+ * - **The heap goes away.** A repository's graph is freed by the child exiting
+ * rather than left for the server process to fragment around.
+ *
+ * The parent keeps exactly the two things a review cannot mint for itself, and
+ * hands each across the IPC channel rather than argv or the environment (`ps` and
+ * `/proc/<pid>/environ` are readable; an installation token belongs in neither):
+ * the token, so the App's private key never crosses the boundary, and page
+ * publishing, because the store and the secret it signs URLs with live with the
+ * server.
+ */
+import { fork, type ChildProcess } from "node:child_process";
+import { existsSync } from "node:fs";
+import { fileURLToPath } from "node:url";
+import { jobKey, type ReviewJob } from "./events.js";
+import type { reviewPullRequest, ReviewDeps, ReviewResult } from "./review.js";
+
+/**
+ * How long one review may take before the process running it is killed.
+ *
+ * Generously above the worst honest review observed in production (274s) and above
+ * what `checkout.ts`'s own 120s-per-git-call budget can add up to, because the cost
+ * of being wrong here is a review that never happens. It is a ceiling on a *lost*
+ * queue slot, not on responsiveness — the server no longer cares how long a review
+ * runs, only that the slot comes back.
+ */
+const DEFAULT_TIMEOUT_MS = 15 * 60 * 1000;
+
+/** Parent → child, once, immediately after the fork. */
+export interface StartMessage {
+ t: "start";
+ job: ReviewJob;
+ /** Minted by the parent; the child never sees the App's private key. */
+ token: string;
+ api?: string;
+}
+
+/** Parent → child: the URL a `publish` request ended up at, or null. */
+export interface PublishedMessage {
+ t: "published";
+ seq: number;
+ url: string | null;
+}
+
+export type ToChild = StartMessage | PublishedMessage;
+
+/** Child → parent: a line for the App's log, so a review is still traceable. */
+export interface LogMessage {
+ t: "log";
+ msg: string;
+}
+
+/** Child → parent: store this page and tell me its URL. */
+export interface PublishMessage {
+ t: "publish";
+ seq: number;
+ html: string;
+}
+
+export interface DoneMessage {
+ t: "done";
+ result: ReviewResult;
+}
+
+export interface FailedMessage {
+ t: "failed";
+ message: string;
+}
+
+export type FromChild = LogMessage | PublishMessage | DoneMessage | FailedMessage;
+
+export interface ChildReviewerOptions {
+ /** The module the child runs. Only tests pass this. */
+ entry?: string;
+ timeoutMs?: number;
+}
+
+/**
+ * The module a review runs in.
+ *
+ * `dist/app/review-worker.js` in an installed or containerised App. Run from a
+ * checkout there is no `dist`, so fall back to the TypeScript source next door:
+ * `fork` inherits `process.execArgv`, so a parent started through tsx hands the
+ * child the same loader and a `.ts` entry runs. Resolved once, at import.
+ *
+ * Exported so a test can assert the path a deployed App will fork actually exists —
+ * getting this wrong is the one failure here that no unit test of the protocol
+ * would catch and every review would hit.
+ */
+export function reviewWorkerEntry(): string {
+ const js = fileURLToPath(new URL("./review-worker.js", import.meta.url));
+ if (existsSync(js)) return js;
+ const ts = js.replace(/\.js$/, ".ts");
+ return existsSync(ts) ? ts : js;
+}
+
+/**
+ * Live review processes.
+ *
+ * `main.ts` force-exits 25 seconds after SIGTERM if a review has not finished
+ * draining. That used to kill the review along with the process; now the review is
+ * a separate process that would outlive us, still holding a clone made with an
+ * installation token. So the last thing this process does is take them with it.
+ *
+ * A container stop is unaffected either way: Docker signals PID 1, not the process
+ * group, so a review keeps going through the grace period and the drain in `main.ts`
+ * means what it says. Ctrl-C in a terminal signals the whole group, so the children
+ * die with the default disposition and their reviews are reported as killed — which
+ * is what an interrupted review was before this too.
+ */
+const live = new Set<ChildProcess>();
+let hooked = false;
+
+function track(child: ChildProcess): void {
+ live.add(child);
+ if (hooked) return;
+ hooked = true;
+ process.on("exit", () => {
+ for (const c of live) c.kill("SIGKILL");
+ });
+}
+
+/** A dead channel is not worth an exception: the exit handler already covers it. */
+function post(child: ChildProcess, msg: ToChild): void {
+ if (!child.connected) return;
+ try {
+ child.send(msg);
+ } catch {
+ /* channel closed under us — `exit` will settle the review */
+ }
+}
+
+/**
+ * A reviewer with `reviewPullRequest`'s signature that runs it in a child process.
+ *
+ * Same shape on purpose: `server.ts` swaps one for the other in a single line, and
+ * every existing test that injects its own reviewer through `AppSeams.review` keeps
+ * running in-process, which is what a test wants.
+ */
+export function childReviewer(opts: ChildReviewerOptions = {}): typeof reviewPullRequest {
+ const entry = opts.entry ?? reviewWorkerEntry();
+ const timeoutMs = opts.timeoutMs ?? DEFAULT_TIMEOUT_MS;
+ return (job, deps) => runInChild(job, deps, entry, timeoutMs);
+}
+
+async function runInChild(
+ job: ReviewJob,
+ deps: ReviewDeps,
+ entry: string,
+ timeoutMs: number,
+): Promise<ReviewResult> {
+ const log = deps.log ?? ((): void => {});
+ // Minted here rather than in the child: this is a network round trip, never a
+ // blocking one, and the cache that owns installation tokens — including the
+ // forget-a-rejected-token behaviour — belongs to the one process that sees every
+ // delivery.
+ const token = await deps.token(job.installationId);
+
+ // The job key as argv makes `ps` in a container say which pull request each
+ // review is; the token stays out of it.
+ const child = fork(entry, [jobKey(job)], { stdio: ["ignore", "inherit", "inherit", "ipc"] });
+ track(child);
+
+ try {
+ return await new Promise<ReviewResult>((resolve, reject) => {
+ let settled = false;
+ const settle = (fn: () => void): void => {
+ if (settled) return;
+ settled = true;
+ clearTimeout(timer);
+ fn();
+ };
+
+ const timer = setTimeout(() => {
+ settle(() => reject(new Error(`review exceeded ${timeoutMs}ms and was killed`)));
+ }, timeoutMs);
+ timer.unref();
+
+ child.on("message", (raw) => {
+ const msg = raw as FromChild;
+ if (msg.t === "log") {
+ log(msg.msg);
+ return;
+ }
+ if (msg.t === "publish") {
+ // Publishing happens here because the store and its signing secret do.
+ // A page that cannot be stored costs the link, never the review — the
+ // same contract `review.ts` already has with a null `publish`.
+ const stored = deps.publish ? deps.publish(job, msg.html) : Promise.resolve(null);
+ void stored
+ .catch(() => null)
+ .then((url) => post(child, { t: "published", seq: msg.seq, url }));
+ return;
+ }
+ if (msg.t === "done") {
+ settle(() => resolve(msg.result));
+ return;
+ }
+ settle(() => reject(new Error(msg.message)));
+ });
+
+ child.on("error", (err) => settle(() => reject(err)));
+
+ // The case that used to be an App-wide outage: a child that dies without
+ // reporting. An OOM kill, a segfault in a grammar on hostile source. Now it
+ // is one failed review, named, with the signal that did it.
+ child.on("exit", (code, signal) =>
+ settle(() =>
+ reject(
+ new Error(
+ `review process exited ${signal ? `on ${signal}` : `with code ${code}`} before reporting a result`,
+ ),
+ ),
+ ),
+ );
+
+ post(child, { t: "start", job, token, api: deps.api });
+ });
+ } finally {
+ live.delete(child);
+ // Whatever happened — done, failed, timed out, or a delivery superseded by the
+ // queue — nothing forked here is allowed to outlive the review. `review.ts`
+ // deletes the checkout before it reports, so this leaves nothing behind.
+ if (child.exitCode === null && child.signalCode === null) child.kill("SIGKILL");
+ }
+}
+
+/** The App's default reviewer. */
+export const reviewInChildProcess = childReviewer();
diff --git a/src/app/review-worker.ts b/src/app/review-worker.ts
new file mode 100644
index 00000000..241eee68
--- /dev/null
+++ b/src/app/review-worker.ts
@@ -0,0 +1,109 @@
+/**
+ * One review, in its own process. Forked by `./review-process.ts`, never imported.
+ *
+ * The whole file is glue: it waits for a `start`, calls the same
+ * `reviewPullRequest` the server used to call directly, and reports back. Nothing
+ * about a review changes by running here — that is the point. The two dependencies
+ * it cannot satisfy locally are wired to the parent instead:
+ *
+ * - `token` is handed over in the start message. This process never holds the
+ * App's private key, so a hostile repository that somehow got code running here
+ * finds one repository's short-lived token and not the App's identity.
+ * - `publish` is a round trip. The page store and the webhook secret its signed
+ * URLs are keyed on live with the HTTP server, and a page is only useful to the
+ * process that will serve it.
+ *
+ * Deliberately no `graft` state of its own: no lock, no cache directory, no
+ * telemetry. A review works on a throwaway clone in /tmp and takes its findings
+ * with it when it exits.
+ */
+import { reviewPullRequest } from "./review.js";
+import type { Fetch } from "./identity.js";
+import type { FromChild, StartMessage, ToChild } from "./review-process.js";
+
+/** Publish requests waiting on the parent, by sequence number. */
+const awaiting = new Map<number, (url: string | null) => void>();
+let seq = 0;
+
+/** Fire and forget. `process.send` throws on a closed channel, and a log line is
+ * not worth crashing a review over — the `disconnect` handler below has already
+ * decided what a closed channel means. */
+const send = (msg: FromChild): void => {
+ if (!process.connected) return;
+ try {
+ process.send?.(msg);
+ } catch {
+ /* parent went away mid-review */
+ }
+};
+
+/** Close the channel, or leave if it is already gone. Either way this is the end of
+ * the process — see the `disconnect` handler below. */
+function bye(): void {
+ try {
+ if (process.connected) process.disconnect?.();
+ else process.exit(0);
+ } catch {
+ process.exit(0);
+ }
+}
+
+/**
+ * The last word, then close the channel.
+ *
+ * Sent with a callback because that callback is the flush: `done` has to be on the
+ * wire before the channel closes, or the parent sees a process that exited without
+ * reporting and turns a finished review into a failure.
+ */
+function finish(msg: FromChild): void {
+ if (!process.connected) return bye();
+ try {
+ process.send?.(msg, () => bye());
+ } catch {
+ bye();
+ }
+}
+
+async function run(start: StartMessage): Promise<void> {
+ try {
+ const result = await reviewPullRequest(start.job, {
+ token: async () => start.token,
+ fetch: globalThis.fetch as unknown as Fetch,
+ api: start.api,
+ publish: (_job, html) =>
+ new Promise<string | null>((resolve) => {
+ const id = (seq += 1);
+ awaiting.set(id, resolve);
+ send({ t: "publish", seq: id, html });
+ }),
+ log: (msg) => send({ t: "log", msg }),
+ });
+ finish({ t: "done", result });
+ } catch (err) {
+ // `reviewPullRequest` has already run the token through `redact`, so this
+ // message is safe to put in the parent's log — which is where it is going.
+ finish({ t: "failed", message: err instanceof Error ? err.message : String(err) });
+ }
+}
+
+/**
+ * No parent, no point.
+ *
+ * A review whose channel has closed has nowhere to publish a page and nobody to
+ * report to. If the server process died, nothing will SIGKILL this one either, and
+ * without this it would sit in a clone for the rest of the parent's 15-minute
+ * ceiling. Going now leaves the same throwaway tree behind that a SIGKILL would —
+ * what it does not do is spend a quarter of an hour producing a review no one can
+ * receive.
+ */
+process.on("disconnect", () => process.exit(0));
+
+process.on("message", (raw) => {
+ const msg = raw as ToChild;
+ if (msg.t === "start") {
+ void run(msg);
+ return;
+ }
+ awaiting.get(msg.seq)?.(msg.url);
+ awaiting.delete(msg.seq);
+});
diff --git a/src/app/server.ts b/src/app/server.ts
index dcce6a40..3ddec8ed 100644
--- a/src/app/server.ts
+++ b/src/app/server.ts
@@ -9,13 +9,20 @@
* GitHub retries a delivery it considers failed and gives up after ten seconds,
* while a review takes tens of them — so the handler validates, queues, and
* answers 202. Everything real happens on the queue.
+ *
+ * "On the queue" used to mean "on this event loop", and that was the whole of a
+ * multi-minute outage: a review is synchronous end to end, so one in flight left
+ * this server unable to answer /healthz or serve a page it had already produced.
+ * The queue now runs each review in its own process (`./review-process.ts`) and
+ * everything in here is I/O again.
*/
import { createServer, type IncomingMessage, type Server, type ServerResponse } from "node:http";
import { jobKey, reviewJobFor, type ReviewJob } from "./events.js";
import { InstallationTokens, verifySignature, type AppCredentials, type Fetch } from "./identity.js";
import { PageStore } from "./pages.js";
import { WorkQueue } from "./queue.js";
-import { reviewPullRequest } from "./review.js";
+import { reviewInChildProcess } from "./review-process.js";
+import type { reviewPullRequest } from "./review.js";
/** A webhook body larger than this is not a pull_request event we can use. */
const MAX_BODY_BYTES = 2 * 1024 * 1024;
@@ -23,7 +30,8 @@ const MAX_BODY_BYTES = 2 * 1024 * 1024;
/** Seams for tests: nothing here has a default that touches the network. */
export interface AppSeams {
fetch?: Fetch;
- /** Swapped out to assert what the queue was handed, without a clone. */
+ /** Swapped out to assert what the queue was handed, without a clone — and, being
+ * called in-process, without the fork the default reviewer does. */
review?: typeof reviewPullRequest;
now?: () => number;
}
@@ -46,7 +54,7 @@ export function createApp(
): { server: Server; queue: WorkQueue<ReviewJob>; pages: PageStore } {
const log = config.log ?? ((msg: string) => console.log(msg));
const fetchImpl = seams.fetch ?? (globalThis.fetch as unknown as Fetch);
- const review = seams.review ?? reviewPullRequest;
+ const review = seams.review ?? reviewInChildProcess;
const tokens = new InstallationTokens(config, fetchImpl, seams.now ?? Date.now, config.api);
const pages = new PageStore({ secret: config.webhookSecret, dir: config.pageDir });
const origin = config.publicUrl.replace(/\/$/, "");
diff --git a/test/app-review-process-probe.ts b/test/app-review-process-probe.ts
new file mode 100644
index 00000000..81779421
--- /dev/null
+++ b/test/app-review-process-probe.ts
@@ -0,0 +1,85 @@
+/**
+ * A stand-in for `src/app/review-worker.ts`, forked by `app-review-process.test.ts`.
+ *
+ * Speaks the same IPC protocol, and instead of cloning a repository it does the one
+ * thing the real review does that matters to the test: it blocks. `job.baseRef`
+ * carries the mode, because a fixture may as well use a field it has no other use
+ * for than invent a channel for it:
+ *
+ * block:<ms> hold this process's event loop, using no CPU at all — the
+ * `spawnSync` git fetch, which is what most of a review's wall clock
+ * actually is (load average 0.96 on eight vCPUs, in production).
+ * burn:<ms> hold it with a busy loop, one core pegged — the tree-sitter parse.
+ * fail:<msg> report a failed review.
+ * crash exit without reporting: an OOM kill, or a grammar segfault.
+ * hang never report, and stay alive.
+ * ok publish and report immediately.
+ *
+ * Every reporting mode publishes a page through the parent BEFORE it blocks, which
+ * exercises the round trip that puts a page in the server's store and gives the
+ * test a sync point: once the parent has answered the publish, this process is
+ * about to go unresponsive, and the parent had better not be.
+ */
+import type { FromChild, ToChild } from "../src/app/review-process.js";
+
+const awaiting = new Map<number, (url: string | null) => void>();
+let seq = 0;
+
+const send = (msg: FromChild): void => {
+ process.send?.(msg);
+};
+
+function finish(msg: FromChild): void {
+ process.send?.(msg, () => process.disconnect?.());
+}
+
+/** Block without spending CPU, the way a synchronous `git fetch` does. */
+function block(ms: number): void {
+ Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, ms);
+}
+
+function burn(ms: number): void {
+ const until = Date.now() + ms;
+ while (Date.now() < until) {
+ /* peg one core, the way the parse loop does */
+ }
+}
+
+const publish = (html: string): Promise<string | null> =>
+ new Promise((resolve) => {
+ const id = (seq += 1);
+ awaiting.set(id, resolve);
+ send({ t: "publish", seq: id, html });
+ });
+
+async function run(mode: string): Promise<void> {
+ const [kind, arg] = mode.split(":");
+
+ if (kind === "crash") process.exit(3);
+ if (kind === "hang") {
+ setInterval(() => {}, 1000);
+ return;
+ }
+ if (kind === "fail") {
+ finish({ t: "failed", message: arg ?? "probe failure" });
+ return;
+ }
+
+ send({ t: "log", msg: `probe: ${mode}` });
+ const url = await publish(`<h1>${mode}</h1>`);
+
+ if (kind === "block") block(Number(arg));
+ if (kind === "burn") burn(Number(arg));
+
+ finish({ t: "done", result: { commentUrl: null, areas: 1, affected: 2, viewerUrl: url } });
+}
+
+process.on("message", (raw) => {
+ const msg = raw as ToChild;
+ if (msg.t === "start") {
+ void run(msg.job.baseRef);
+ return;
+ }
+ awaiting.get(msg.seq)?.(msg.url);
+ awaiting.delete(msg.seq);
+});
diff --git a/test/app-review-process.test.ts b/test/app-review-process.test.ts
new file mode 100644
index 00000000..069e1ca2
--- /dev/null
+++ b/test/app-review-process.test.ts
@@ -0,0 +1,343 @@
+/**
+ * The App answering while a review runs.
+ *
+ * This is the regression that mattered most in production, so it is asserted with
+ * numbers rather than shape. A review is synchronous end to end — `spawnSync` git,
+ * a tree-sitter parse loop over the whole repository, a blocking graph write — and
+ * running it on the server's own event loop meant one in-flight review made the
+ * whole HTTP surface disappear for its entire duration: 92s to 274s per review in
+ * the logs, `GET /p/<id>` timing out from outside for minutes at a stretch, and the
+ * container's own 5s HEALTHCHECK failing until the queue emptied.
+ *
+ * So the property under test is not "the code runs". It is: with the queue full of
+ * reviews that have deliberately wedged their own event loops, this server still
+ * answers /healthz and still serves a page it already produced, comfortably inside
+ * the healthcheck's budget. The second test is the control — the same blocking work
+ * back on the server's loop, failing the same assertion — so a future change that
+ * quietly moves reviews back in-process is caught here rather than in an incident.
+ *
+ * The reviews are `app-review-process-probe.ts`, forked through the real
+ * `childReviewer`: the IPC protocol, the token hand-off, the publish round trip and
+ * the process teardown are all the production ones. Only the graph build is faked,
+ * because cloning a repository is not what is being measured.
+ */
+import { test } from "node:test";
+import assert from "node:assert/strict";
+import { fork } from "node:child_process";
+import { createHmac, generateKeyPairSync } from "node:crypto";
+import { existsSync } from "node:fs";
+import type { AddressInfo } from "node:net";
+import { dirname, join } from "node:path";
+import { monitorEventLoopDelay } from "node:perf_hooks";
+import { fileURLToPath } from "node:url";
+import { PageStore } from "../src/app/pages.js";
+import { childReviewer, reviewWorkerEntry } from "../src/app/review-process.js";
+import { createApp, type AppSeams } from "../src/app/server.js";
+import type { ReviewJob } from "../src/app/events.js";
+import type { ReviewDeps, ReviewResult } from "../src/app/review.js";
+
+const PROBE = join(dirname(fileURLToPath(import.meta.url)), "app-review-process-probe.ts");
+
+/** The container HEALTHCHECK's own timeout — the budget everything here beats. */
+const HEALTHCHECK_BUDGET_MS = 5000;
+/**
+ * What the assertions actually demand: a fifth of the budget, so a CI box can be
+ * five times slower than a laptop and the test still means what it says.
+ */
+const BUDGET_MS = 1000;
+
+/** Production's default `GRAFT_CONCURRENCY` — and one was already too many. */
+const CONCURRENCY = 2;
+const BURN_MS = 2000;
+
+const secret = "webhook-secret";
+const { privateKey } = generateKeyPairSync("rsa", { modulusLength: 2048 });
+
+/** `baseRef` carries the probe's mode; see `app-review-process-probe.ts`. */
+const job = (number: number, mode: string): ReviewJob => ({
+ installationId: 5,
+ owner: "NanoNets",
+ repo: "Graft",
+ number,
+ baseRef: mode,
+ headSha: `sha${number}`,
+ fromFork: false,
+});
+
+const noNetwork: ReviewDeps["fetch"] = async () => ({
+ ok: false,
+ status: 500,
+ text: async () => "no network in tests",
+});
+
+const deps = (): ReviewDeps => ({ token: async () => "t0ken", fetch: noNetwork });
+
+/** What `PageStore` will have signed a given page's link with. */
+const linkFor = (url: string, number: number): string => {
+ const id = PageStore.idFor("NanoNets", "Graft", number);
+ return `${url}/p/${id}?t=${createHmac("sha256", secret).update(id).digest("hex").slice(0, 32)}`;
+};
+
+interface Harness {
+ url: string;
+ app: ReturnType<typeof createApp>;
+ logs: string[];
+ results: ReviewResult[];
+ close: () => Promise<void>;
+}
+
+/** The real App, on a real socket, with the reviewer under test behind the queue. */
+function start(review: AppSeams["review"], concurrency = CONCURRENCY): Harness {
+ const logs: string[] = [];
+ const results: ReviewResult[] = [];
+ const app = createApp(
+ {
+ appId: "1",
+ privateKey: privateKey.export({ type: "pkcs1", format: "pem" }).toString(),
+ webhookSecret: secret,
+ publicUrl: "http://localhost",
+ concurrency,
+ log: (msg) => logs.push(msg),
+ },
+ {
+ fetch: noNetwork,
+ // The token is stubbed rather than minted: `InstallationTokens` wants the
+ // network and has its own tests. Everything else the queue hands a reviewer —
+ // the publish callback, the log — is the App's own.
+ review: async (j, d) => {
+ const res = await review!(j, { ...d, token: async () => "t0ken" });
+ results.push(res);
+ return res;
+ },
+ },
+ );
+ app.server.listen(0);
+ const port = (app.server.address() as AddressInfo).port;
+ return {
+ url: `http://127.0.0.1:${port}`,
+ app,
+ logs,
+ results,
+ close: () => new Promise((r) => app.server.close(() => r())),
+ };
+}
+
+const sleep = (ms: number): Promise<void> => new Promise((r) => setTimeout(r, ms));
+
+/** Wait for a condition, or fail — never hang the suite on a fork that didn't. */
+async function until(what: string, pred: () => boolean, timeoutMs = 20_000): Promise<void> {
+ const deadline = Date.now() + timeoutMs;
+ while (!pred()) {
+ assert.ok(Date.now() < deadline, `timed out waiting for ${what}`);
+ await sleep(20);
+ }
+}
+
+/** One request, on the same 5s budget the container's healthcheck gives itself. */
+async function timed(url: string): Promise<{ ms: number; status: number | "timeout" }> {
+ const t0 = Date.now();
+ const ac = new AbortController();
+ const kill = setTimeout(() => ac.abort(), HEALTHCHECK_BUDGET_MS);
+ try {
+ const res = await fetch(url, { signal: ac.signal });
+ await res.text();
+ return { ms: Date.now() - t0, status: res.status };
+ } catch {
+ return { ms: Date.now() - t0, status: "timeout" };
+ } finally {
+ clearTimeout(kill);
+ }
+}
+
+const completed = (logs: string[]): string[] => logs.filter((l) => / changed → /.test(l));
+
+test("reviews: every worker wedged and the server still answers /healthz and /p/<id> in milliseconds", async () => {
+ const app = start(childReviewer({ entry: PROBE }));
+ const loop = monitorEventLoopDelay({ resolution: 20 });
+ try {
+ // One review per worker — the exact shape production was running when a single
+ // one of them was enough to take the published pages offline — and both halves
+ // of what a review does to whatever thread it is on: `burn` pegs a core the way
+ // the parse loop does, `block` holds the loop while using none, the way the
+ // `spawnSync` git fetch does.
+ app.app.queue.push("NanoNets/Graft#1", job(1, `burn:${BURN_MS}`));
+ app.app.queue.push("NanoNets/Graft#2", job(2, `block:${BURN_MS}`));
+
+ // Each probe publishes its page and only then wedges itself, so a page in the
+ // store is the signal that the process which produced it has stopped
+ // answering — and that this one had better not have.
+ await until("both reviews to publish and go unresponsive", () => app.app.pages.size === CONCURRENCY);
+
+ loop.enable();
+ const seen: Array<{ ms: number; status: number | "timeout" }> = [];
+ const stop = Date.now() + 1200;
+ while (Date.now() < stop) {
+ seen.push(await timed(`${app.url}/healthz`));
+ // The symptom users actually reported: a page the App had already produced,
+ // unreachable. Serve it from under the same load.
+ seen.push(await timed(linkFor(app.url, 1)));
+ }
+ loop.disable();
+
+ assert.ok(seen.length >= 10, `expected a steady stream of requests, got ${seen.length}`);
+ assert.equal(
+ seen.filter((r) => r.status === "timeout").length,
+ 0,
+ `no request may exceed the container's ${HEALTHCHECK_BUDGET_MS}ms budget`,
+ );
+ assert.deepEqual([...new Set(seen.map((r) => r.status))].sort(), [200], "and every one of them is an answer");
+
+ const worst = Math.max(...seen.map((r) => r.ms));
+ assert.ok(
+ worst < BUDGET_MS,
+ `worst response ${worst}ms — must stay under ${BUDGET_MS}ms (the container allows ${HEALTHCHECK_BUDGET_MS}ms)`,
+ );
+
+ // The direct measurement of the bug: the longest the server's loop was ever
+ // held. In-process that was the whole review; it must now be a hiccup.
+ const held = loop.max / 1e6;
+ assert.ok(held < BUDGET_MS, `event loop was held for ${held.toFixed(0)}ms — a review is still running on it`);
+
+ // Responsiveness bought by dropping work would not be a fix: the reviews that
+ // were accepted still finish, and still publish.
+ await app.app.queue.drain();
+ assert.equal(completed(app.logs).length, CONCURRENCY);
+ assert.equal(app.results.filter((r) => r.viewerUrl !== null).length, CONCURRENCY);
+ } finally {
+ loop.disable();
+ await app.close();
+ await app.app.queue.drain();
+ }
+});
+
+test("reviews: the same blocking work on the server's own loop is what took the pages offline", async () => {
+ // The control. Deliberately the old shape — a reviewer called in-process — so the
+ // numbers in the test above are a measured difference and not a hope.
+ const app = start(async () => {
+ const stop = Date.now() + BURN_MS;
+ while (Date.now() < stop) {
+ /* exactly what the parse loop does to whichever thread it is on */
+ }
+ return { commentUrl: null, areas: 1, affected: 2, viewerUrl: null };
+ }, 1);
+ const loop = monitorEventLoopDelay({ resolution: 20 });
+ try {
+ // The meter has to be armed by a turn of the loop before the loop stops
+ // turning, or it measures a block that started before it was watching — and
+ // reports a serene zero.
+ loop.enable();
+ await sleep(50);
+
+ // Scheduled before the block starts and measured against when it was DUE: an
+ // outside healthcheck's clock keeps running while this process's does not, and
+ // that gap is what Docker saw.
+ const due = Date.now() + 100;
+ const probe = new Promise<number>((resolve) => {
+ setTimeout(() => void timed(`${app.url}/healthz`).then(() => resolve(Date.now() - due)), 100);
+ });
+ app.app.queue.push("NanoNets/Graft#1", job(1, "in-process"));
+ const late = await probe;
+ loop.disable();
+
+ const held = loop.max / 1e6;
+ assert.ok(
+ held > BURN_MS * 0.75,
+ `expected the review to hold the loop for ~${BURN_MS}ms, saw ${held.toFixed(0)}ms — the control is not reproducing the bug`,
+ );
+ assert.ok(late > BUDGET_MS, `expected /healthz to answer ${BUDGET_MS}ms+ after it was due, saw ${late}ms`);
+ } finally {
+ loop.disable();
+ await app.close();
+ await app.app.queue.drain();
+ }
+});
+
+test("reviews: a page published from the review process is served by the server that holds the store", async () => {
+ const app = start(childReviewer({ entry: PROBE }));
+ try {
+ app.app.queue.push("NanoNets/Graft#9", job(9, "ok"));
+ await app.app.queue.drain();
+
+ assert.equal(completed(app.logs).length, 1, `expected a completed review, got ${JSON.stringify(app.logs)}`);
+ // The child has no page store and no webhook secret to sign a URL with, so the
+ // link in its result can only have come back across the IPC channel.
+ assert.equal(app.results[0].viewerUrl, linkFor("http://localhost", 9));
+
+ const res = await fetch(linkFor(app.url, 9));
+ assert.equal(res.status, 200);
+ assert.equal(await res.text(), "<h1>ok</h1>");
+ } finally {
+ await app.close();
+ }
+});
+
+test("reviews: a review process that dies without reporting is one failed review, not a dead App", async () => {
+ // Before the split this was an outage: an OOM kill, or a segfault in a grammar on
+ // a stranger's source, took the HTTP server down with the review. Now it is a
+ // rejected promise the queue turns into a log line.
+ const reviewer = childReviewer({ entry: PROBE });
+ await assert.rejects(() => reviewer(job(1, "crash"), deps()), /exited with code 3 before reporting a result/);
+
+ // A review that fails honestly still says why, in the parent's log.
+ await assert.rejects(() => reviewer(job(2, "fail:no diff against main"), deps()), /no diff against main/);
+
+ // And the process is still able to run a review afterwards.
+ assert.equal((await reviewer(job(3, "ok"), deps())).areas, 1);
+});
+
+test("reviews: a wedged review is killed rather than holding a worker forever", async () => {
+ const t0 = Date.now();
+ const reviewer = childReviewer({ entry: PROBE, timeoutMs: 400 });
+ await assert.rejects(() => reviewer(job(1, "hang"), deps()), /exceeded 400ms and was killed/);
+ // The ceiling exists so the queue slot comes back: the next delivery, from some
+ // other installation, must not queue behind a review that never ends.
+ assert.ok(Date.now() - t0 < 20_000, "the timeout has to actually settle the promise");
+});
+
+test("reviews: the real review worker loads the whole graph stack in a forked process", async () => {
+ // Everything above forks a probe, which proves the protocol and proves nothing
+ // about the module a deployed App actually forks. Two things can only break here:
+ // the entry path (`dist/app/review-worker.js`, or the source next door when the
+ // App runs from a checkout), and loading the review stack in a child — nine
+ // tree-sitter native addons imported at `graph/extract.ts`'s module scope, which
+ // also constructs a `Parser` there, plus the WASM grammars behind it. Getting this
+ // far is the assertion; no review runs, because a review wants a clone.
+ const entry = reviewWorkerEntry();
+ assert.ok(existsSync(entry), `${entry} must exist — it is what a deployed App forks`);
+
+ const child = fork(entry, ["load-probe"], { stdio: ["ignore", "pipe", "pipe", "ipc"] });
+ let stderr = "";
+ child.stderr?.on("data", (c) => {
+ stderr += String(c);
+ });
+ const ended = new Promise<string>((resolve) => {
+ child.on("exit", (code, signal) => resolve(signal ? `signal ${signal}` : `code ${code}`));
+ });
+ try {
+ // A grammar that cannot load in a forked child dies during import, in well under
+ // this. Still waiting for work at the end of it means the stack came up.
+ const outcome = await Promise.race([ended, sleep(2500).then(() => "still waiting for work")]);
+ assert.equal(outcome, "still waiting for work", `the review worker did not come up: ${stderr}`);
+ assert.equal(stderr, "", "loading the review stack must be silent");
+ } finally {
+ child.kill("SIGKILL");
+ }
+});
+
+test("reviews: repeat pushes for one pull request still collapse across the process boundary", async () => {
+ // Superseding is why the queue exists and moving the work out of process must not
+ // cost it: five pushes to one PR in a minute is normal, and four of those reviews
+ // would only be overwritten. One running plus the newest, never five.
+ const app = start(childReviewer({ entry: PROBE }), 1);
+ try {
+ for (let n = 0; n < 5; n++) app.app.queue.push("NanoNets/Graft#4", job(4, "ok"));
+ await app.app.queue.drain();
+ assert.equal(
+ completed(app.logs).length,
+ 2,
+ `expected the running review plus the newest, got ${JSON.stringify(app.logs)}`,
+ );
+ } finally {
+ await app.close();
+ }
+});