Turbocharge Claude Code, Cursor, Codex, Gemini & every coding agent: faster, cheaper, with contextual understanding specific to your codebase.
TypeScript ai-agentsanthropicclaude-codeclicode-graphcodexcontext-engineeringcursordeveloper-toolsgeminiknowledge-graphllmmcpmcp-serveropen-sourceopenaitree-sitter
feat(app): run each review in its own process so the server stays responsive (#281)
anirudhkumar-nanonetsf2ca4a7bab92
diff --git a/src/app/review-process.ts b/src/app/review-process.tsnew file mode 100644index 00000000..1be03f03--- /dev/null+++ b/src/app/review-process.ts@@ -0,0 +1,268 @@+/**+ * Running a review somewhere it cannot stop the App answering.+ *+ * A review is synchronous from end to end, and not in one place we could fix: the+ * pull request is fetched with `spawnSync` (`./checkout.ts`), the whole repository+ * is parsed in one `files.forEach` loop of native tree-sitter calls+ * (`graph/build.ts`), the graph is written with a blocking `JSON.stringify`, and+ * the viewer page is exported synchronously too. In-process that holds the event+ * loop for the entire review.+ *+ * That is not a theory. On the deployed App, probing `GET /p/<id>` from outside+ * every ten seconds while one review ran: three consecutive timeouts (no response+ * at all, not even with `--max-time 45`), then 404 in 0.9s the moment the queue+ * emptied. Meanwhile the box's load average was 0.96 on eight vCPUs — one thread+ * pegged, seven cores idle — and the App's own logs put a single review at 92s to+ * 274s ("3 changed → 0 affected in 180134ms"). So a *single* in-flight review took+ * the whole HTTP surface down for minutes: the container's 5s `HEALTHCHECK` failed,+ * Docker marked the container unhealthy, and pages the App had already produced+ * were unreachable. On a busy repository that is most of the working day.+ *+ * Neither bounding the queue nor lowering `GRAFT_CONCURRENCY` touches this —+ * concurrency was already 2 and one job is enough. Cooperative yielding inside the+ * parse loop does not either: a `git fetch` of a large repository is one+ * uninterruptible `spawnSync` that can legitimately take a minute, and it is not+ * even using a core while it blocks. The work has to leave the process.+ *+ * A child process rather than a `worker_threads` worker, deliberately:+ *+ * - **The native grammars keep loading the way they already do.** `graph/extract.ts`+ * imports nine tree-sitter addons at module scope and constructs a `Parser` there+ * too, and `graph/generic.ts` loads more through `createRequire`. In a worker all+ * of that is re-instantiated per thread, which puts the N-API-in-worker behaviour+ * of nine third-party native modules between the App and every review. A fork runs+ * byte-identical code to what production runs today.+ * - **A review can be killed.** A wedged parse or a 10-minute clone is a process+ * to SIGKILL, not a thread with no safe way to stop it.+ * - **A crash stops being an outage.** An OOM kill or a grammar segfault on a+ * stranger's source now costs one review and is reported as a failure, instead+ * of taking the HTTP server with it. This process clones code written by+ * people who opened a pull request; that isolation is worth having on its own.+ * - **The heap goes away.** A repository's graph is freed by the child exiting+ * rather than left for the server process to fragment around.+ *+ * The parent keeps exactly the two things a review cannot mint for itself, and+ * hands each across the IPC channel rather than argv or the environment (`ps` and+ * `/proc/<pid>/environ` are readable; an installation token belongs in neither):+ * the token, so the App's private key never crosses the boundary, and page+ * publishing, because the store and the secret it signs URLs with live with the+ * server.+ */+import { fork, type ChildProcess } from "node:child_process";+import { existsSync } from "node:fs";+import { fileURLToPath } from "node:url";+import { jobKey, type ReviewJob } from "./events.js";+import type { reviewPullRequest, ReviewDeps, ReviewResult } from "./review.js";++/**+ * How long one review may take before the process running it is killed.+ *+ * Generously above the worst honest review observed in production (274s) and above+ * what `checkout.ts`'s own 120s-per-git-call budget can add up to, because the cost+ * of being wrong here is a review that never happens. It is a ceiling on a *lost*+ * queue slot, not on responsiveness — the server no longer cares how long a review+ * runs, only that the slot comes back.+ */+const DEFAULT_TIMEOUT_MS = 15 * 60 * 1000;++/** Parent → child, once, immediately after the fork. */+export interface StartMessage {+ t: "start";+ job: ReviewJob;+ /** Minted by the parent; the child never sees the App's private key. */+ token: string;+ api?: string;+}++/** Parent → child: the URL a `publish` request ended up at, or null. */+export interface PublishedMessage {+ t: "published";+ seq: number;+ url: string | null;+}++export type ToChild = StartMessage | PublishedMessage;++/** Child → parent: a line for the App's log, so a review is still traceable. */+export interface LogMessage {+ t: "log";+ msg: string;+}++/** Child → parent: store this page and tell me its URL. */+export interface PublishMessage {+ t: "publish";+ seq: number;+ html: string;+}++export interface DoneMessage {+ t: "done";+ result: ReviewResult;+}++export interface FailedMessage {+ t: "failed";+ message: string;+}++export type FromChild = LogMessage | PublishMessage | DoneMessage | FailedMessage;++export interface ChildReviewerOptions {+ /** The module the child runs. Only tests pass this. */+ entry?: string;+ timeoutMs?: number;+}++/**+ * The module a review runs in.+ *+ * `dist/app/review-worker.js` in an installed or containerised App. Run from a+ * checkout there is no `dist`, so fall back to the TypeScript source next door:+ * `fork` inherits `process.execArgv`, so a parent started through tsx hands the+ * child the same loader and a `.ts` entry runs. Resolved once, at import.+ *+ * Exported so a test can assert the path a deployed App will fork actually exists —+ * getting this wrong is the one failure here that no unit test of the protocol+ * would catch and every review would hit.+ */+export function reviewWorkerEntry(): string {+ const js = fileURLToPath(new URL("./review-worker.js", import.meta.url));+ if (existsSync(js)) return js;+ const ts = js.replace(/\.js$/, ".ts");+ return existsSync(ts) ? ts : js;+}++/**+ * Live review processes.+ *+ * `main.ts` force-exits 25 seconds after SIGTERM if a review has not finished+ * draining. That used to kill the review along with the process; now the review is+ * a separate process that would outlive us, still holding a clone made with an+ * installation token. So the last thing this process does is take them with it.+ *+ * A container stop is unaffected either way: Docker signals PID 1, not the process+ * group, so a review keeps going through the grace period and the drain in `main.ts`+ * means what it says. Ctrl-C in a terminal signals the whole group, so the children+ * die with the default disposition and their reviews are reported as killed — which+ * is what an interrupted review was before this too.+ */+const live = new Set<ChildProcess>();+let hooked = false;++function track(child: ChildProcess): void {+ live.add(child);+ if (hooked) return;+ hooked = true;+ process.on("exit", () => {+ for (const c of live) c.kill("SIGKILL");+ });+}++/** A dead channel is not worth an exception: the exit handler already covers it. */+function post(child: ChildProcess, msg: ToChild): void {+ if (!child.connected) return;+ try {+ child.send(msg);+ } catch {+ /* channel closed under us — `exit` will settle the review */+ }+}++/**+ * A reviewer with `reviewPullRequest`'s signature that runs it in a child process.+ *+ * Same shape on purpose: `server.ts` swaps one for the other in a single line, and+ * every existing test that injects its own reviewer through `AppSeams.review` keeps+ * running in-process, which is what a test wants.+ */+export function childReviewer(opts: ChildReviewerOptions = {}): typeof reviewPullRequest {+ const entry = opts.entry ?? reviewWorkerEntry();+ const timeoutMs = opts.timeoutMs ?? DEFAULT_TIMEOUT_MS;+ return (job, deps) => runInChild(job, deps, entry, timeoutMs);+}++async function runInChild(+ job: ReviewJob,+ deps: ReviewDeps,+ entry: string,+ timeoutMs: number,+): Promise<ReviewResult> {+ const log = deps.log ?? ((): void => {});+ // Minted here rather than in the child: this is a network round trip, never a+ // blocking one, and the cache that owns installation tokens — including the+ // forget-a-rejected-token behaviour — belongs to the one process that sees every+ // delivery.+ const token = await deps.token(job.installationId);++ // The job key as argv makes `ps` in a container say which pull request each+ // review is; the token stays out of it.+ const child = fork(entry, [jobKey(job)], { stdio: ["ignore", "inherit", "inherit", "ipc"] });+ track(child);++ try {+ return await new Promise<ReviewResult>((resolve, reject) => {+ let settled = false;+ const settle = (fn: () => void): void => {+ if (settled) return;+ settled = true;+ clearTimeout(timer);+ fn();+ };++ const timer = setTimeout(() => {+ settle(() => reject(new Error(`review exceeded ${timeoutMs}ms and was killed`)));+ }, timeoutMs);+ timer.unref();++ child.on("message", (raw) => {+ const msg = raw as FromChild;+ if (msg.t === "log") {+ log(msg.msg);+ return;+ }+ if (msg.t === "publish") {+ // Publishing happens here because the store and its signing secret do.+ // A page that cannot be stored costs the link, never the review — the+ // same contract `review.ts` already has with a null `publish`.+ const stored = deps.publish ? deps.publish(job, msg.html) : Promise.resolve(null);+ void stored+ .catch(() => null)+ .then((url) => post(child, { t: "published", seq: msg.seq, url }));+ return;+ }+ if (msg.t === "done") {+ settle(() => resolve(msg.result));+ return;+ }+ settle(() => reject(new Error(msg.message)));+ });++ child.on("error", (err) => settle(() => reject(err)));++ // The case that used to be an App-wide outage: a child that dies without+ // reporting. An OOM kill, a segfault in a grammar on hostile source. Now it+ // is one failed review, named, with the signal that did it.+ child.on("exit", (code, signal) =>+ settle(() =>+ reject(+ new Error(+ `review process exited ${signal ? `on ${signal}` : `with code ${code}`} before reporting a result`,+ ),+ ),+ ),+ );++ post(child, { t: "start", job, token, api: deps.api });+ });+ } finally {+ live.delete(child);+ // Whatever happened — done, failed, timed out, or a delivery superseded by the+ // queue — nothing forked here is allowed to outlive the review. `review.ts`+ // deletes the checkout before it reports, so this leaves nothing behind.+ if (child.exitCode === null && child.signalCode === null) child.kill("SIGKILL");+ }+}++/** The App's default reviewer. */+export const reviewInChildProcess = childReviewer();diff --git a/src/app/review-worker.ts b/src/app/review-worker.tsnew file mode 100644index 00000000..241eee68--- /dev/null+++ b/src/app/review-worker.ts@@ -0,0 +1,109 @@+/**+ * One review, in its own process. Forked by `./review-process.ts`, never imported.+ *+ * The whole file is glue: it waits for a `start`, calls the same+ * `reviewPullRequest` the server used to call directly, and reports back. Nothing+ * about a review changes by running here — that is the point. The two dependencies+ * it cannot satisfy locally are wired to the parent instead:+ *+ * - `token` is handed over in the start message. This process never holds the+ * App's private key, so a hostile repository that somehow got code running here+ * finds one repository's short-lived token and not the App's identity.+ * - `publish` is a round trip. The page store and the webhook secret its signed+ * URLs are keyed on live with the HTTP server, and a page is only useful to the+ * process that will serve it.+ *+ * Deliberately no `graft` state of its own: no lock, no cache directory, no+ * telemetry. A review works on a throwaway clone in /tmp and takes its findings+ * with it when it exits.+ */+import { reviewPullRequest } from "./review.js";+import type { Fetch } from "./identity.js";+import type { FromChild, StartMessage, ToChild } from "./review-process.js";++/** Publish requests waiting on the parent, by sequence number. */+const awaiting = new Map<number, (url: string | null) => void>();+let seq = 0;++/** Fire and forget. `process.send` throws on a closed channel, and a log line is+ * not worth crashing a review over — the `disconnect` handler below has already+ * decided what a closed channel means. */+const send = (msg: FromChild): void => {+ if (!process.connected) return;+ try {+ process.send?.(msg);+ } catch {+ /* parent went away mid-review */+ }+};++/** Close the channel, or leave if it is already gone. Either way this is the end of+ * the process — see the `disconnect` handler below. */+function bye(): void {+ try {+ if (process.connected) process.disconnect?.();+ else process.exit(0);+ } catch {+ process.exit(0);+ }+}++/**+ * The last word, then close the channel.+ *+ * Sent with a callback because that callback is the flush: `done` has to be on the+ * wire before the channel closes, or the parent sees a process that exited without+ * reporting and turns a finished review into a failure.+ */+function finish(msg: FromChild): void {+ if (!process.connected) return bye();+ try {+ process.send?.(msg, () => bye());+ } catch {+ bye();+ }+}++async function run(start: StartMessage): Promise<void> {+ try {+ const result = await reviewPullRequest(start.job, {+ token: async () => start.token,+ fetch: globalThis.fetch as unknown as Fetch,+ api: start.api,+ publish: (_job, html) =>+ new Promise<string | null>((resolve) => {+ const id = (seq += 1);+ awaiting.set(id, resolve);+ send({ t: "publish", seq: id, html });+ }),+ log: (msg) => send({ t: "log", msg }),+ });+ finish({ t: "done", result });+ } catch (err) {+ // `reviewPullRequest` has already run the token through `redact`, so this+ // message is safe to put in the parent's log — which is where it is going.+ finish({ t: "failed", message: err instanceof Error ? err.message : String(err) });+ }+}++/**+ * No parent, no point.+ *+ * A review whose channel has closed has nowhere to publish a page and nobody to+ * report to. If the server process died, nothing will SIGKILL this one either, and+ * without this it would sit in a clone for the rest of the parent's 15-minute+ * ceiling. Going now leaves the same throwaway tree behind that a SIGKILL would —+ * what it does not do is spend a quarter of an hour producing a review no one can+ * receive.+ */+process.on("disconnect", () => process.exit(0));++process.on("message", (raw) => {+ const msg = raw as ToChild;+ if (msg.t === "start") {+ void run(msg);+ return;+ }+ awaiting.get(msg.seq)?.(msg.url);+ awaiting.delete(msg.seq);+});diff --git a/src/app/server.ts b/src/app/server.tsindex dcce6a40..3ddec8ed 100644--- a/src/app/server.ts+++ b/src/app/server.ts@@ -9,13 +9,20 @@* GitHub retries a delivery it considers failed and gives up after ten seconds,* while a review takes tens of them — so the handler validates, queues, and* answers 202. Everything real happens on the queue.+ *+ * "On the queue" used to mean "on this event loop", and that was the whole of a+ * multi-minute outage: a review is synchronous end to end, so one in flight left+ * this server unable to answer /healthz or serve a page it had already produced.+ * The queue now runs each review in its own process (`./review-process.ts`) and+ * everything in here is I/O again.*/import { createServer, type IncomingMessage, type Server, type ServerResponse } from "node:http";import { jobKey, reviewJobFor, type ReviewJob } from "./events.js";import { InstallationTokens, verifySignature, type AppCredentials, type Fetch } from "./identity.js";import { PageStore } from "./pages.js";import { WorkQueue } from "./queue.js";-import { reviewPullRequest } from "./review.js";+import { reviewInChildProcess } from "./review-process.js";+import type { reviewPullRequest } from "./review.js";/** A webhook body larger than this is not a pull_request event we can use. */const MAX_BODY_BYTES = 2 * 1024 * 1024;@@ -23,7 +30,8 @@ const MAX_BODY_BYTES = 2 * 1024 * 1024;/** Seams for tests: nothing here has a default that touches the network. */export interface AppSeams {fetch?: Fetch;- /** Swapped out to assert what the queue was handed, without a clone. */+ /** Swapped out to assert what the queue was handed, without a clone — and, being+ * called in-process, without the fork the default reviewer does. */review?: typeof reviewPullRequest;now?: () => number;}@@ -46,7 +54,7 @@ export function createApp(): { server: Server; queue: WorkQueue<ReviewJob>; pages: PageStore } {const log = config.log ?? ((msg: string) => console.log(msg));const fetchImpl = seams.fetch ?? (globalThis.fetch as unknown as Fetch);- const review = seams.review ?? reviewPullRequest;+ const review = seams.review ?? reviewInChildProcess;const tokens = new InstallationTokens(config, fetchImpl, seams.now ?? Date.now, config.api);const pages = new PageStore({ secret: config.webhookSecret, dir: config.pageDir });const origin = config.publicUrl.replace(/\/$/, "");diff --git a/test/app-review-process-probe.ts b/test/app-review-process-probe.tsnew file mode 100644index 00000000..81779421--- /dev/null+++ b/test/app-review-process-probe.ts@@ -0,0 +1,85 @@+/**+ * A stand-in for `src/app/review-worker.ts`, forked by `app-review-process.test.ts`.+ *+ * Speaks the same IPC protocol, and instead of cloning a repository it does the one+ * thing the real review does that matters to the test: it blocks. `job.baseRef`+ * carries the mode, because a fixture may as well use a field it has no other use+ * for than invent a channel for it:+ *+ * block:<ms> hold this process's event loop, using no CPU at all — the+ * `spawnSync` git fetch, which is what most of a review's wall clock+ * actually is (load average 0.96 on eight vCPUs, in production).+ * burn:<ms> hold it with a busy loop, one core pegged — the tree-sitter parse.+ * fail:<msg> report a failed review.+ * crash exit without reporting: an OOM kill, or a grammar segfault.+ * hang never report, and stay alive.+ * ok publish and report immediately.+ *+ * Every reporting mode publishes a page through the parent BEFORE it blocks, which+ * exercises the round trip that puts a page in the server's store and gives the+ * test a sync point: once the parent has answered the publish, this process is+ * about to go unresponsive, and the parent had better not be.+ */+import type { FromChild, ToChild } from "../src/app/review-process.js";++const awaiting = new Map<number, (url: string | null) => void>();+let seq = 0;++const send = (msg: FromChild): void => {+ process.send?.(msg);+};++function finish(msg: FromChild): void {+ process.send?.(msg, () => process.disconnect?.());+}++/** Block without spending CPU, the way a synchronous `git fetch` does. */+function block(ms: number): void {+ Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, ms);+}++function burn(ms: number): void {+ const until = Date.now() + ms;+ while (Date.now() < until) {+ /* peg one core, the way the parse loop does */+ }+}++const publish = (html: string): Promise<string | null> =>+ new Promise((resolve) => {+ const id = (seq += 1);+ awaiting.set(id, resolve);+ send({ t: "publish", seq: id, html });+ });++async function run(mode: string): Promise<void> {+ const [kind, arg] = mode.split(":");++ if (kind === "crash") process.exit(3);+ if (kind === "hang") {+ setInterval(() => {}, 1000);+ return;+ }+ if (kind === "fail") {+ finish({ t: "failed", message: arg ?? "probe failure" });+ return;+ }++ send({ t: "log", msg: `probe: ${mode}` });+ const url = await publish(`<h1>${mode}</h1>`);++ if (kind === "block") block(Number(arg));+ if (kind === "burn") burn(Number(arg));++ finish({ t: "done", result: { commentUrl: null, areas: 1, affected: 2, viewerUrl: url } });+}++process.on("message", (raw) => {+ const msg = raw as ToChild;+ if (msg.t === "start") {+ void run(msg.job.baseRef);+ return;+ }+ awaiting.get(msg.seq)?.(msg.url);+ awaiting.delete(msg.seq);+});diff --git a/test/app-review-process.test.ts b/test/app-review-process.test.tsnew file mode 100644index 00000000..069e1ca2--- /dev/null+++ b/test/app-review-process.test.ts@@ -0,0 +1,343 @@+/**+ * The App answering while a review runs.+ *+ * This is the regression that mattered most in production, so it is asserted with+ * numbers rather than shape. A review is synchronous end to end — `spawnSync` git,+ * a tree-sitter parse loop over the whole repository, a blocking graph write — and+ * running it on the server's own event loop meant one in-flight review made the+ * whole HTTP surface disappear for its entire duration: 92s to 274s per review in+ * the logs, `GET /p/<id>` timing out from outside for minutes at a stretch, and the+ * container's own 5s HEALTHCHECK failing until the queue emptied.+ *+ * So the property under test is not "the code runs". It is: with the queue full of+ * reviews that have deliberately wedged their own event loops, this server still+ * answers /healthz and still serves a page it already produced, comfortably inside+ * the healthcheck's budget. The second test is the control — the same blocking work+ * back on the server's loop, failing the same assertion — so a future change that+ * quietly moves reviews back in-process is caught here rather than in an incident.+ *+ * The reviews are `app-review-process-probe.ts`, forked through the real+ * `childReviewer`: the IPC protocol, the token hand-off, the publish round trip and+ * the process teardown are all the production ones. Only the graph build is faked,+ * because cloning a repository is not what is being measured.+ */+import { test } from "node:test";+import assert from "node:assert/strict";+import { fork } from "node:child_process";+import { createHmac, generateKeyPairSync } from "node:crypto";+import { existsSync } from "node:fs";+import type { AddressInfo } from "node:net";+import { dirname, join } from "node:path";+import { monitorEventLoopDelay } from "node:perf_hooks";+import { fileURLToPath } from "node:url";+import { PageStore } from "../src/app/pages.js";+import { childReviewer, reviewWorkerEntry } from "../src/app/review-process.js";+import { createApp, type AppSeams } from "../src/app/server.js";+import type { ReviewJob } from "../src/app/events.js";+import type { ReviewDeps, ReviewResult } from "../src/app/review.js";++const PROBE = join(dirname(fileURLToPath(import.meta.url)), "app-review-process-probe.ts");++/** The container HEALTHCHECK's own timeout — the budget everything here beats. */+const HEALTHCHECK_BUDGET_MS = 5000;+/**+ * What the assertions actually demand: a fifth of the budget, so a CI box can be+ * five times slower than a laptop and the test still means what it says.+ */+const BUDGET_MS = 1000;++/** Production's default `GRAFT_CONCURRENCY` — and one was already too many. */+const CONCURRENCY = 2;+const BURN_MS = 2000;++const secret = "webhook-secret";+const { privateKey } = generateKeyPairSync("rsa", { modulusLength: 2048 });++/** `baseRef` carries the probe's mode; see `app-review-process-probe.ts`. */+const job = (number: number, mode: string): ReviewJob => ({+ installationId: 5,+ owner: "NanoNets",+ repo: "Graft",+ number,+ baseRef: mode,+ headSha: `sha${number}`,+ fromFork: false,+});++const noNetwork: ReviewDeps["fetch"] = async () => ({+ ok: false,+ status: 500,+ text: async () => "no network in tests",+});++const deps = (): ReviewDeps => ({ token: async () => "t0ken", fetch: noNetwork });++/** What `PageStore` will have signed a given page's link with. */+const linkFor = (url: string, number: number): string => {+ const id = PageStore.idFor("NanoNets", "Graft", number);+ return `${url}/p/${id}?t=${createHmac("sha256", secret).update(id).digest("hex").slice(0, 32)}`;+};++interface Harness {+ url: string;+ app: ReturnType<typeof createApp>;+ logs: string[];+ results: ReviewResult[];+ close: () => Promise<void>;+}++/** The real App, on a real socket, with the reviewer under test behind the queue. */+function start(review: AppSeams["review"], concurrency = CONCURRENCY): Harness {+ const logs: string[] = [];+ const results: ReviewResult[] = [];+ const app = createApp(+ {+ appId: "1",+ privateKey: privateKey.export({ type: "pkcs1", format: "pem" }).toString(),+ webhookSecret: secret,+ publicUrl: "http://localhost",+ concurrency,+ log: (msg) => logs.push(msg),+ },+ {+ fetch: noNetwork,+ // The token is stubbed rather than minted: `InstallationTokens` wants the+ // network and has its own tests. Everything else the queue hands a reviewer —+ // the publish callback, the log — is the App's own.+ review: async (j, d) => {+ const res = await review!(j, { ...d, token: async () => "t0ken" });+ results.push(res);+ return res;+ },+ },+ );+ app.server.listen(0);+ const port = (app.server.address() as AddressInfo).port;+ return {+ url: `http://127.0.0.1:${port}`,+ app,+ logs,+ results,+ close: () => new Promise((r) => app.server.close(() => r())),+ };+}++const sleep = (ms: number): Promise<void> => new Promise((r) => setTimeout(r, ms));++/** Wait for a condition, or fail — never hang the suite on a fork that didn't. */+async function until(what: string, pred: () => boolean, timeoutMs = 20_000): Promise<void> {+ const deadline = Date.now() + timeoutMs;+ while (!pred()) {+ assert.ok(Date.now() < deadline, `timed out waiting for ${what}`);+ await sleep(20);+ }+}++/** One request, on the same 5s budget the container's healthcheck gives itself. */+async function timed(url: string): Promise<{ ms: number; status: number | "timeout" }> {+ const t0 = Date.now();+ const ac = new AbortController();+ const kill = setTimeout(() => ac.abort(), HEALTHCHECK_BUDGET_MS);+ try {+ const res = await fetch(url, { signal: ac.signal });+ await res.text();+ return { ms: Date.now() - t0, status: res.status };+ } catch {+ return { ms: Date.now() - t0, status: "timeout" };+ } finally {+ clearTimeout(kill);+ }+}++const completed = (logs: string[]): string[] => logs.filter((l) => / changed → /.test(l));++test("reviews: every worker wedged and the server still answers /healthz and /p/<id> in milliseconds", async () => {+ const app = start(childReviewer({ entry: PROBE }));+ const loop = monitorEventLoopDelay({ resolution: 20 });+ try {+ // One review per worker — the exact shape production was running when a single+ // one of them was enough to take the published pages offline — and both halves+ // of what a review does to whatever thread it is on: `burn` pegs a core the way+ // the parse loop does, `block` holds the loop while using none, the way the+ // `spawnSync` git fetch does.+ app.app.queue.push("NanoNets/Graft#1", job(1, `burn:${BURN_MS}`));+ app.app.queue.push("NanoNets/Graft#2", job(2, `block:${BURN_MS}`));++ // Each probe publishes its page and only then wedges itself, so a page in the+ // store is the signal that the process which produced it has stopped+ // answering — and that this one had better not have.+ await until("both reviews to publish and go unresponsive", () => app.app.pages.size === CONCURRENCY);++ loop.enable();+ const seen: Array<{ ms: number; status: number | "timeout" }> = [];+ const stop = Date.now() + 1200;+ while (Date.now() < stop) {+ seen.push(await timed(`${app.url}/healthz`));+ // The symptom users actually reported: a page the App had already produced,+ // unreachable. Serve it from under the same load.+ seen.push(await timed(linkFor(app.url, 1)));+ }+ loop.disable();++ assert.ok(seen.length >= 10, `expected a steady stream of requests, got ${seen.length}`);+ assert.equal(+ seen.filter((r) => r.status === "timeout").length,+ 0,+ `no request may exceed the container's ${HEALTHCHECK_BUDGET_MS}ms budget`,+ );+ assert.deepEqual([...new Set(seen.map((r) => r.status))].sort(), [200], "and every one of them is an answer");++ const worst = Math.max(...seen.map((r) => r.ms));+ assert.ok(+ worst < BUDGET_MS,+ `worst response ${worst}ms — must stay under ${BUDGET_MS}ms (the container allows ${HEALTHCHECK_BUDGET_MS}ms)`,+ );++ // The direct measurement of the bug: the longest the server's loop was ever+ // held. In-process that was the whole review; it must now be a hiccup.+ const held = loop.max / 1e6;+ assert.ok(held < BUDGET_MS, `event loop was held for ${held.toFixed(0)}ms — a review is still running on it`);++ // Responsiveness bought by dropping work would not be a fix: the reviews that+ // were accepted still finish, and still publish.+ await app.app.queue.drain();+ assert.equal(completed(app.logs).length, CONCURRENCY);+ assert.equal(app.results.filter((r) => r.viewerUrl !== null).length, CONCURRENCY);+ } finally {+ loop.disable();+ await app.close();+ await app.app.queue.drain();+ }+});++test("reviews: the same blocking work on the server's own loop is what took the pages offline", async () => {+ // The control. Deliberately the old shape — a reviewer called in-process — so the+ // numbers in the test above are a measured difference and not a hope.+ const app = start(async () => {+ const stop = Date.now() + BURN_MS;+ while (Date.now() < stop) {+ /* exactly what the parse loop does to whichever thread it is on */+ }+ return { commentUrl: null, areas: 1, affected: 2, viewerUrl: null };+ }, 1);+ const loop = monitorEventLoopDelay({ resolution: 20 });+ try {+ // The meter has to be armed by a turn of the loop before the loop stops+ // turning, or it measures a block that started before it was watching — and+ // reports a serene zero.+ loop.enable();+ await sleep(50);++ // Scheduled before the block starts and measured against when it was DUE: an+ // outside healthcheck's clock keeps running while this process's does not, and+ // that gap is what Docker saw.+ const due = Date.now() + 100;+ const probe = new Promise<number>((resolve) => {+ setTimeout(() => void timed(`${app.url}/healthz`).then(() => resolve(Date.now() - due)), 100);+ });+ app.app.queue.push("NanoNets/Graft#1", job(1, "in-process"));+ const late = await probe;+ loop.disable();++ const held = loop.max / 1e6;+ assert.ok(+ held > BURN_MS * 0.75,+ `expected the review to hold the loop for ~${BURN_MS}ms, saw ${held.toFixed(0)}ms — the control is not reproducing the bug`,+ );+ assert.ok(late > BUDGET_MS, `expected /healthz to answer ${BUDGET_MS}ms+ after it was due, saw ${late}ms`);+ } finally {+ loop.disable();+ await app.close();+ await app.app.queue.drain();+ }+});++test("reviews: a page published from the review process is served by the server that holds the store", async () => {+ const app = start(childReviewer({ entry: PROBE }));+ try {+ app.app.queue.push("NanoNets/Graft#9", job(9, "ok"));+ await app.app.queue.drain();++ assert.equal(completed(app.logs).length, 1, `expected a completed review, got ${JSON.stringify(app.logs)}`);+ // The child has no page store and no webhook secret to sign a URL with, so the+ // link in its result can only have come back across the IPC channel.+ assert.equal(app.results[0].viewerUrl, linkFor("http://localhost", 9));++ const res = await fetch(linkFor(app.url, 9));+ assert.equal(res.status, 200);+ assert.equal(await res.text(), "<h1>ok</h1>");+ } finally {+ await app.close();+ }+});++test("reviews: a review process that dies without reporting is one failed review, not a dead App", async () => {+ // Before the split this was an outage: an OOM kill, or a segfault in a grammar on+ // a stranger's source, took the HTTP server down with the review. Now it is a+ // rejected promise the queue turns into a log line.+ const reviewer = childReviewer({ entry: PROBE });+ await assert.rejects(() => reviewer(job(1, "crash"), deps()), /exited with code 3 before reporting a result/);++ // A review that fails honestly still says why, in the parent's log.+ await assert.rejects(() => reviewer(job(2, "fail:no diff against main"), deps()), /no diff against main/);++ // And the process is still able to run a review afterwards.+ assert.equal((await reviewer(job(3, "ok"), deps())).areas, 1);+});++test("reviews: a wedged review is killed rather than holding a worker forever", async () => {+ const t0 = Date.now();+ const reviewer = childReviewer({ entry: PROBE, timeoutMs: 400 });+ await assert.rejects(() => reviewer(job(1, "hang"), deps()), /exceeded 400ms and was killed/);+ // The ceiling exists so the queue slot comes back: the next delivery, from some+ // other installation, must not queue behind a review that never ends.+ assert.ok(Date.now() - t0 < 20_000, "the timeout has to actually settle the promise");+});++test("reviews: the real review worker loads the whole graph stack in a forked process", async () => {+ // Everything above forks a probe, which proves the protocol and proves nothing+ // about the module a deployed App actually forks. Two things can only break here:+ // the entry path (`dist/app/review-worker.js`, or the source next door when the+ // App runs from a checkout), and loading the review stack in a child — nine+ // tree-sitter native addons imported at `graph/extract.ts`'s module scope, which+ // also constructs a `Parser` there, plus the WASM grammars behind it. Getting this+ // far is the assertion; no review runs, because a review wants a clone.+ const entry = reviewWorkerEntry();+ assert.ok(existsSync(entry), `${entry} must exist — it is what a deployed App forks`);++ const child = fork(entry, ["load-probe"], { stdio: ["ignore", "pipe", "pipe", "ipc"] });+ let stderr = "";+ child.stderr?.on("data", (c) => {+ stderr += String(c);+ });+ const ended = new Promise<string>((resolve) => {+ child.on("exit", (code, signal) => resolve(signal ? `signal ${signal}` : `code ${code}`));+ });+ try {+ // A grammar that cannot load in a forked child dies during import, in well under+ // this. Still waiting for work at the end of it means the stack came up.+ const outcome = await Promise.race([ended, sleep(2500).then(() => "still waiting for work")]);+ assert.equal(outcome, "still waiting for work", `the review worker did not come up: ${stderr}`);+ assert.equal(stderr, "", "loading the review stack must be silent");+ } finally {+ child.kill("SIGKILL");+ }+});++test("reviews: repeat pushes for one pull request still collapse across the process boundary", async () => {+ // Superseding is why the queue exists and moving the work out of process must not+ // cost it: five pushes to one PR in a minute is normal, and four of those reviews+ // would only be overwritten. One running plus the newest, never five.+ const app = start(childReviewer({ entry: PROBE }), 1);+ try {+ for (let n = 0; n < 5; n++) app.app.queue.push("NanoNets/Graft#4", job(4, "ok"));+ await app.app.queue.drain();+ assert.equal(+ completed(app.logs).length,+ 2,+ `expected the running review plus the newest, got ${JSON.stringify(app.logs)}`,+ );+ } finally {+ await app.close();+ }+});