Skip to content

Open source alternative to Semrush and Ahrefs

Let rendered audits crawl past 1,000 pages within one credit hold (#1085)

Carter Smith689964bf3b72
diff --git a/docs/maintainers/SITE_AUDIT_RENDERING.md b/docs/maintainers/SITE_AUDIT_RENDERING.md
index 3193af1b..512cde1d 100644
--- a/docs/maintainers/SITE_AUDIT_RENDERING.md
+++ b/docs/maintainers/SITE_AUDIT_RENDERING.md
@@ -63,9 +63,10 @@ milliseconds, signed or not, and the refusals are per URL more often than per
site (see `shopify-crawling.md`). When the audited site is a Shopify store (its
`powered-by` header, a saved crawler signature, or a host in Shopify's address
range) and the audit renders JavaScript, a page whose direct fetch returns 429
-goes straight to the renderers instead of the shared backoff. Browser Run is
-usually throttled too; Context fetches from outside Cloudflare and reads the
-page. A page neither renderer reads is given up on as `rate_limited`, which
+goes straight to the renderers instead of the shared backoff. Shopify's edge
+lets Browser Run through, so these pages almost always render on Cloudflare and
+cost no more than any other rendered page; Context reads the few it misses. A
+page neither renderer reads is given up on as `rate_limited`, which
pauses the crawl like any other refusal; a store that refuses several pages in
a row stops the crawl.
@@ -98,10 +99,14 @@ retried finalize cannot charge twice. Unlike a single DataForSEO call, one
audit's hold is split: the monthly balance covers what it can and top-up
covers the rest.
-1. `startAudit` locks the worst case, page limit × the fallback rate, from the
- monthly balance first and top-up for the remainder. Not enough across both
- refuses the audit with `INSUFFICIENT_CREDITS` before the workflow starts.
- The lock ids travel in the workflow params.
+1. `startAudit` locks the estimated cost, twice the Cloudflare-only estimate
+ (`renderingHoldCredits`), from the monthly balance first and top-up for the
+ remainder. Not enough across both refuses the audit with
+ `INSUFFICIENT_CREDITS` before the workflow starts. The lock ids travel in
+ the workflow params. Most pages render on Cloudflare (about 95% of rendered
+ pages in production), so this usually covers the audit without holding the
+ six-times-larger worst case. The launch form and MCP tool quote this amount
+ as what the audit costs at most.
2. Each crawl chunk counts Cloudflare attempts and Context credits and returns
them in its step result. Chunks make no Autumn calls.
3. The workflow sums the checkpointed chunk counts. At the end, whether the
@@ -109,15 +114,23 @@ covers the rest.
monthly first, and releases the rest.
4. Locks expire after at most 24 hours. An audit that dies without settling
returns the hold to the customer on its own.
-5. A rendered audit crawls at most 1,000 pages (`RENDERED_MAX_AUDIT_PAGES`),
- enforced like the plan page limits: `startAudit` refuses more with
- `AUDIT_PAGE_LIMIT_EXCEEDED`, and the launch form caps its page input. At
- the 20 second worst case per Cloudflare attempt, that keeps a rendered
- crawl well inside the lock's 24 hours, so settlement finds its lock.
+5. The crawl stays inside the one hold. A chunk starts a page only while the
+ credits rendered so far, plus every in-flight page and this one at the
+ Context worst case, fit the hold. When they no longer do, the crawl stops
+ and the report gets a `crawl-credits-exhausted` issue. Only a site that
+ falls back to Context heavily reaches it.
+6. A hosted rendered audit stops crawling 22 hours after it starts
+ (`RENDERED_CRAWL_TIME_LIMIT_HOURS`), so settlement still finds its lock.
+ The deadline counts from the workflow's creation, just after the locks are
+ taken, and is the same on every replay. A crawl chunk starts no request
+ after it, and the report gets a `crawl-time-limit` issue. Rendered audits
+ take the plan's page limit. The launch form shows the estimate before a
+ large rendered audit starts.
What we absorb rather than charge: renders in a chunk attempt that failed
before returning its counts, a settlement that Autumn rejects, usage beyond
-the hold (a retried chunk can render a page twice).
+the hold (a retried chunk can render a page twice, outside the chunk's own
+count).
Alternatives considered:
@@ -127,6 +140,15 @@ Alternatives considered:
- **Charging up front with a refund:** needs a manual refund on every failure
path, and overcharges when the workflow dies.
- **Per-page billing:** hundreds of Autumn calls per audit.
+- **Holding the worst case** (the first release): every page at the Context
+ rate is six times what most audits cost, so accounts with a few dollars of
+ credits were refused audits they could easily pay for.
+- **Growing the hold mid-crawl:** one lock per audit keeps settlement a single
+ step, and an audit never ends up holding credits the user did not agree to
+ at the start.
+- **A fixed page cap** (1,000 pages at first): rendered pages take from a few
+ seconds to half a minute, so any cap that fits the slowest site inside the
+ lock refuses large sites that would finish in a few hours.
## Open items
diff --git a/src/client/features/audit/launch/LaunchFormCard.tsx b/src/client/features/audit/launch/LaunchFormCard.tsx
index bd3c7601..a9d54aef 100644
--- a/src/client/features/audit/launch/LaunchFormCard.tsx
+++ b/src/client/features/audit/launch/LaunchFormCard.tsx
@@ -27,7 +27,7 @@ import {
isEveryAppClientAuthMode,
isHostedClientAuthMode,
} from "@/lib/auth-mode";
-import { RENDERED_MAX_AUDIT_PAGES } from "@/shared/audit-limits";
+import { PAID_MAX_AUDIT_PAGES } from "@/shared/audit-limits";
import { renderingEstimateText } from "@/shared/audit-rendering";
import { SUBSCRIBE_ROUTE } from "@/shared/billing";
@@ -35,7 +35,6 @@ type Props = {
launchForm: ReturnType<typeof useLaunchController>["launchForm"];
commitMaxPagesInput: () => number;
maxPagesLimit: number;
- paidMaxPagesLimit: number;
canRenderJavaScript?: boolean;
outOfCredits?: boolean;
};
@@ -44,7 +43,6 @@ export function LaunchFormCard({
commitMaxPagesInput,
launchForm,
maxPagesLimit,
- paidMaxPagesLimit,
canRenderJavaScript,
outOfCredits,
}: Props) {
@@ -103,7 +101,6 @@ export function LaunchFormCard({
launchForm={launchForm}
commitMaxPagesInput={commitMaxPagesInput}
maxPagesLimit={maxPagesLimit}
- paidMaxPagesLimit={paidMaxPagesLimit}
/>
<div className="space-y-4">
<LighthouseOptions launchForm={launchForm} />
@@ -126,7 +123,6 @@ function LaunchOptions({
launchForm,
commitMaxPagesInput,
maxPagesLimit,
- paidMaxPagesLimit,
}: Props) {
return (
<Field className="rounded-lg border border-border p-3">
@@ -154,16 +150,13 @@ function LaunchOptions({
</launchForm.Field>
<FieldDescription>
Enter any value from {MIN_PAGES} to {maxPagesLimit.toLocaleString()}.
- {maxPagesLimit === RENDERED_MAX_AUDIT_PAGES
- ? " Audits that render JavaScript are limited to this many pages."
- : null}
- {maxPagesLimit < paidMaxPagesLimit ? (
+ {maxPagesLimit < PAID_MAX_AUDIT_PAGES ? (
<>
{" "}
<Link to={SUBSCRIBE_ROUTE} search={{ upgrade: true }}>
Upgrade
</Link>{" "}
- to crawl up to {paidMaxPagesLimit.toLocaleString()} pages.
+ to crawl up to {PAID_MAX_AUDIT_PAGES.toLocaleString()} pages.
</>
) : null}
</FieldDescription>
diff --git a/src/client/features/audit/launch/LaunchView.tsx b/src/client/features/audit/launch/LaunchView.tsx
index ff97e815..85872afb 100644
--- a/src/client/features/audit/launch/LaunchView.tsx
+++ b/src/client/features/audit/launch/LaunchView.tsx
@@ -4,6 +4,11 @@ import { AuditHistorySection } from "@/client/features/audit/launch/AuditHistory
import { LaunchFormCard } from "@/client/features/audit/launch/LaunchFormCard";
import { useLaunchController } from "@/client/features/audit/launch/useLaunchController";
import { useHostedPlanGate } from "@/client/features/billing/HostedPlanGate";
+import { isHostedClientAuthMode } from "@/lib/auth-mode";
+import {
+ RENDERING_COST_NOTE,
+ renderingCostText,
+} from "@/shared/audit-rendering";
type LaunchViewProps = {
projectId: string;
@@ -35,7 +40,6 @@ export function LaunchView({
launchForm={controller.launchForm}
commitMaxPagesInput={controller.commitMaxPagesInput}
maxPagesLimit={controller.maxPagesLimit}
- paidMaxPagesLimit={controller.paidMaxPagesLimit}
canRenderJavaScript={controller.canRenderJavaScript}
outOfCredits={controller.outOfCredits}
/>
@@ -47,17 +51,52 @@ export function LaunchView({
/>
{controller.largeCrawlPages != null ? (
- <ConfirmDialog
- title="Start a large audit?"
- confirmLabel="Continue"
+ <LargeCrawlDialog
+ pages={controller.largeCrawlPages}
+ renderJavaScript={controller.renderJavaScript}
onConfirm={controller.confirmLargeCrawl}
onClose={controller.cancelLargeCrawl}
- >
- You are about to crawl {controller.largeCrawlPages.toLocaleString()}{" "}
- pages. This is okay, but it may take a while. Continue?
- </ConfirmDialog>
+ />
) : null}
</div>
</div>
);
}
+
+function LargeCrawlDialog({
+ pages,
+ renderJavaScript,
+ onConfirm,
+ onClose,
+}: {
+ pages: number;
+ renderJavaScript: boolean;
+ onConfirm: () => void;
+ onClose: () => void;
+}) {
+ const pageCount = pages.toLocaleString();
+ // Only hosted plans pay for rendering, so only they see a price.
+ const priced = renderJavaScript && isHostedClientAuthMode();
+ return (
+ <ConfirmDialog
+ title={
+ priced ? `Render up to ${pageCount} pages?` : "Start a large audit?"
+ }
+ confirmLabel={priced ? "Start audit" : "Continue"}
+ onConfirm={onConfirm}
+ onClose={onClose}
+ >
+ {priced ? (
+ <>
+ {RENDERING_COST_NOTE} Rendered audits can take longer than regular
+ audits.
+ <span className="mt-3 block font-semibold text-foreground">
+ {renderingCostText(pages)}
+ </span>
+ </>
+ ) : (
+ `You are about to crawl ${pageCount} pages. This is okay, but it may take a while. Continue?`
+ )}
+ </ConfirmDialog>
+ );
+}
diff --git a/src/client/features/audit/launch/types.ts b/src/client/features/audit/launch/types.ts
index e2b89d0e..92974db1 100644
--- a/src/client/features/audit/launch/types.ts
+++ b/src/client/features/audit/launch/types.ts
@@ -3,19 +3,12 @@ import {
FREE_MAX_AUDIT_PAGES,
MIN_AUDIT_PAGES,
PAID_MAX_AUDIT_PAGES,
- RENDERED_MAX_AUDIT_PAGES,
} from "@/shared/audit-limits";
export const MIN_PAGES = MIN_AUDIT_PAGES;
-export function getMaxPagesLimit(
- isFreePlan: boolean,
- renderJavaScript: boolean,
-) {
- const planLimit = isFreePlan ? FREE_MAX_AUDIT_PAGES : PAID_MAX_AUDIT_PAGES;
- return renderJavaScript
- ? Math.min(planLimit, RENDERED_MAX_AUDIT_PAGES)
- : planLimit;
+export function getMaxPagesLimit(isFreePlan: boolean) {
+ return isFreePlan ? FREE_MAX_AUDIT_PAGES : PAID_MAX_AUDIT_PAGES;
}
/** The page limit an audit runs with for what is typed in the form. */
diff --git a/src/client/features/audit/launch/useLaunchController.ts b/src/client/features/audit/launch/useLaunchController.ts
index d1cde9d2..5e7420fe 100644
--- a/src/client/features/audit/launch/useLaunchController.ts
+++ b/src/client/features/audit/launch/useLaunchController.ts
@@ -83,7 +83,7 @@ export function useLaunchController({
onSubmit: async ({ formApi, value }) => {
const effectiveMaxPages = commitMaxPagesInput(
launchForm,
- getMaxPagesLimit(isFreePlan, value.renderJavaScript),
+ getMaxPagesLimit(isFreePlan),
);
formApi.setErrorMap({ onSubmit: undefined });
@@ -124,17 +124,14 @@ export function useLaunchController({
launchForm.store,
(state) => state.values.renderJavaScript,
);
- const maxPagesLimit = getMaxPagesLimit(isFreePlan, renderJavaScript);
- // What an upgrade offers with the current options, so the upgrade hint
- // never promises pages a rendered audit cannot crawl.
- const paidMaxPagesLimit = getMaxPagesLimit(false, renderJavaScript);
+ const maxPagesLimit = getMaxPagesLimit(isFreePlan);
return {
launchForm,
historyQuery,
canRenderJavaScript: capabilitiesQuery.data?.canRenderJavaScript,
maxPagesLimit,
- paidMaxPagesLimit,
+ renderJavaScript,
commitMaxPagesInput: () => commitMaxPagesInput(launchForm, maxPagesLimit),
deleteAudit: (auditId: string) => deleteMutation.mutate(auditId),
largeCrawlPages,
diff --git a/src/server/features/audit/services/AuditService.rendering.test.ts b/src/server/features/audit/services/AuditService.rendering.test.ts
index 62df8abe..572d09f1 100644
--- a/src/server/features/audit/services/AuditService.rendering.test.ts
+++ b/src/server/features/audit/services/AuditService.rendering.test.ts
@@ -120,16 +120,6 @@ describe("rendered audit start", () => {
);
});
- it("refuses more than 1,000 rendered pages on a paid plan before holding credits", async () => {
- const error = await AuditService.startAudit({
- ...renderedAudit,
- maxPages: 1_001,
- }).catch((reason: unknown) => reason);
- expect(error).toMatchObject({ code: "AUDIT_PAGE_LIMIT_EXCEEDED" });
- expect(String(error)).toContain("limited to 1,000 pages");
- expect(mocks.lock).not.toHaveBeenCalled();
- });
-
it("does not start an audit the organization cannot cover", async () => {
mocks.lock.mockRejectedValue(new AppError("INSUFFICIENT_CREDITS"));
await expect(AuditService.startAudit(renderedAudit)).rejects.toMatchObject({
diff --git a/src/server/features/audit/services/AuditService.ts b/src/server/features/audit/services/AuditService.ts
index f478792a..0ad5fa2c 100644
--- a/src/server/features/audit/services/AuditService.ts
+++ b/src/server/features/audit/services/AuditService.ts
@@ -24,7 +24,6 @@ import {
} from "@/server/features/audit/services/audit-capacity";
import { AppError } from "@/server/lib/errors";
import { deleteR2Prefix } from "@/server/lib/r2";
-import { RENDERED_MAX_AUDIT_PAGES } from "@/shared/audit-limits";
import { AuditProgressKV } from "@/server/lib/audit/progress-kv";
import {
parseAuditConfig,
@@ -67,12 +66,6 @@ async function startAudit(input: {
if (maxPages > limits.maxPagesPerAudit) {
throw new AppError("AUDIT_PAGE_LIMIT_EXCEEDED");
}
- if (renderJavaScript && maxPages > RENDERED_MAX_AUDIT_PAGES) {
- throw new AppError(
- "AUDIT_PAGE_LIMIT_EXCEEDED",
- `Audits that render JavaScript are limited to ${RENDERED_MAX_AUDIT_PAGES.toLocaleString("en-US")} pages.`,
- );
- }
const lighthouseStrategy = input.lighthouseStrategy ?? "auto";
const reservation = getEstimatedAuditCapacity({
diff --git a/src/server/lib/audit/rendering-billing.test.ts b/src/server/lib/audit/rendering-billing.test.ts
index 1e5b1799..ea36c9a3 100644
--- a/src/server/lib/audit/rendering-billing.test.ts
+++ b/src/server/lib/audit/rendering-billing.test.ts
@@ -33,7 +33,7 @@ const customer = {
userEmail: "[email protected]",
organizationId: "org",
};
-// 100 pages hold 384 credits: every page falling back to Context.
+// 100 pages hold 128 credits: twice every page rendering on Cloudflare.
const lockFor100Pages = () =>
lockRenderingCredits({ customer, auditId: "audit", maxPages: 100 });
const monthlyLock = {
@@ -52,7 +52,7 @@ const topupLock = {
beforeEach(() => {
mocks.isHosted.mockResolvedValue(true);
mocks.getUsageCreditsRemaining.mockResolvedValue({
- monthlyRemaining: 300,
+ monthlyRemaining: 100,
topupRemaining: 1_000,
});
mocks.holdCredits.mockImplementation(
@@ -78,13 +78,13 @@ describe("rendering credit lock", () => {
expect(mocks.holdCredits).not.toHaveBeenCalled();
});
- it("holds the worst case from the monthly balance first, then top-up, for at most 24 hours", async () => {
+ it("holds twice the Cloudflare estimate from the monthly balance first, then top-up, for at most 24 hours", async () => {
const locks = await lockFor100Pages();
expect(
locks.map((lock) => [lock.featureId, lock.estimatedCredits]),
).toEqual([
- ["usage_credits", 300],
- ["topup_credits", 84],
+ ["usage_credits", 100],
+ ["topup_credits", 28],
]);
const ttlMs = mocks.holdCredits.mock.calls[0][0].ttlMs ?? 0;
expect(ttlMs).toBeGreaterThan(60 * 60_000);
@@ -93,12 +93,12 @@ describe("rendering credit lock", () => {
it("refuses before holding anything when both balances together fall short", async () => {
mocks.getUsageCreditsRemaining.mockResolvedValue({
- monthlyRemaining: 300,
- topupRemaining: 83,
+ monthlyRemaining: 100,
+ topupRemaining: 27,
});
const error = await lockFor100Pages().catch((reason: unknown) => reason);
expect(error).toMatchObject({ code: "INSUFFICIENT_CREDITS" });
- expect(String(error)).toContain("needs $0.39 of credits");
+ expect(String(error)).toContain("needs $0.13 of credits");
expect(mocks.holdCredits).not.toHaveBeenCalled();
expect(mocks.sendCreditsExhaustedEvent).toHaveBeenCalledWith(customer);
});
diff --git a/src/server/lib/audit/rendering-billing.ts b/src/server/lib/audit/rendering-billing.ts
index 49952bf1..2cf8fda0 100644
--- a/src/server/lib/audit/rendering-billing.ts
+++ b/src/server/lib/audit/rendering-billing.ts
@@ -11,8 +11,8 @@ import { AppError } from "@/server/lib/errors";
import { captureServerEvent } from "@/server/lib/posthog";
import { isHostedServerAuthMode } from "@/server/lib/runtime-env";
import {
- estimateRenderingCredits,
renderingCreditsNeededText,
+ renderingHoldCredits,
renderUsageCredits,
type RenderUsage,
} from "@/shared/audit-rendering";
@@ -29,10 +29,11 @@ const LOCK_TTL_MS = 24 * 60 * 60_000 - 60_000;
export type RenderingLock = UsageCreditHold;
/**
- * Holds the worst-case rendering cost of a hosted audit before it starts:
- * every page falling back to Context. Throws INSUFFICIENT_CREDITS when the
- * monthly and top-up balances together cannot cover it. Returns no locks
- * outside hosted mode, where rendering is unmetered.
+ * Holds a hosted rendered audit's estimated cost before it starts
+ * (`renderingHoldCredits`). A site that falls back to Context more than that
+ * covers stops crawling when the hold runs out. Throws INSUFFICIENT_CREDITS
+ * when the monthly and top-up balances together cannot cover it. Returns no
+ * locks outside hosted mode, where rendering is unmetered.
*
* Unlike a single DataForSEO call, one audit's hold is large enough to split:
* the monthly balance covers what it can and top-up covers the rest.
@@ -46,7 +47,7 @@ export async function lockRenderingCredits(input: {
const { id: customerId } = await getOrCreateOrganizationCustomer(
input.customer,
);
- const required = estimateRenderingCredits(input.maxPages).high;
+ const required = renderingHoldCredits(input.maxPages);
const insufficient = new AppError(
"INSUFFICIENT_CREDITS",
renderingCreditsNeededText(input.maxPages),
diff --git a/src/server/mcp/tools/site-audit-tools.ts b/src/server/mcp/tools/site-audit-tools.ts
index 661189e1..741e8a05 100644
--- a/src/server/mcp/tools/site-audit-tools.ts
+++ b/src/server/mcp/tools/site-audit-tools.ts
@@ -26,7 +26,7 @@ import {
import { withMcpProjectAuth } from "@/server/mcp/project-auth";
import { projectIdSchema } from "@/server/mcp/schemas";
import { renderingEstimateText } from "@/shared/audit-rendering";
-import { RENDERED_MAX_AUDIT_PAGES } from "@/shared/audit-limits";
+import { RENDERED_CRAWL_TIME_LIMIT_HOURS } from "@/shared/audit-limits";
const auditIdSchema = z
.string()
@@ -68,7 +68,7 @@ const runInputSchema = {
.boolean()
.optional()
.describe(
- `Render JavaScript before auditing, for sites whose content or links load client-side or whose bot protection blocks the plain crawler. Slower, and limited to ${RENDERED_MAX_AUDIT_PAGES.toLocaleString("en-US")} pages. On hosted plans: ${renderingEstimateText(100)} Billing counts pages actually rendered. Hosted plans hold the high end until the audit ends; an account that cannot cover it is refused. Default false.`,
+ `Render JavaScript before auditing, for sites whose content or links load client-side or whose bot protection blocks the plain crawler. Slower. On hosted plans, for 100 pages: ${renderingEstimateText(100)} Hosted plans hold that amount until the audit ends; an account that cannot cover it is refused. A hosted rendered crawl stops after ${RENDERED_CRAWL_TIME_LIMIT_HOURS} hours. Default false.`,
),
runLighthouse: z
.boolean()
diff --git a/src/server/workflows/SiteAuditWorkflow.ts b/src/server/workflows/SiteAuditWorkflow.ts
index fe31c543..46a148ba 100644
--- a/src/server/workflows/SiteAuditWorkflow.ts
+++ b/src/server/workflows/SiteAuditWorkflow.ts
@@ -33,6 +33,7 @@ import {
} from "@/server/workflows/auditStepConfigs";
import type { RenderUsage } from "@/shared/audit-rendering";
import { describeError } from "@/server/lib/errors";
+import { RENDERED_CRAWL_TIME_LIMIT_HOURS } from "@/shared/audit-limits";
interface AuditParams {
auditId: string;
@@ -116,6 +117,7 @@ export class SiteAuditWorkflow extends WorkflowEntrypoint<Env, AuditParams> {
config,
access: await CrawlerCredentialService.openCrawlerAccess(access),
renderUsage,
+ renderHold: renderHoldFor(event),
});
} catch (error) {
console.error(`Audit ${auditId} failed: ${describeError(error)}`, error);
@@ -192,3 +194,15 @@ export class SiteAuditWorkflow extends WorkflowEntrypoint<Env, AuditParams> {
}
}
}
+
+function renderHoldFor(event: WorkflowEvent<AuditParams>) {
+ const locks = event.payload.renderLocks ?? [];
+ if (locks.length === 0) return undefined;
+ return {
+ // The locks are taken just before the workflow is created, so its
+ // timestamp dates them, and it is the same on every replay.
+ deadlineAt:
+ event.timestamp.getTime() + RENDERED_CRAWL_TIME_LIMIT_HOURS * 3_600_000,
+ credits: locks.reduce((sum, lock) => sum + lock.estimatedCredits, 0),
+ };
+}
diff --git a/src/server/workflows/siteAuditWorkflowCrawl.test.ts b/src/server/workflows/siteAuditWorkflowCrawl.test.ts
index 8b346f82..34b1407d 100644
--- a/src/server/workflows/siteAuditWorkflowCrawl.test.ts
+++ b/src/server/workflows/siteAuditWorkflowCrawl.test.ts
@@ -46,6 +46,10 @@ vi.mock("@/server/lib/audit/ids", () => ({
import { runCrawlPhase } from "@/server/workflows/siteAuditWorkflowCrawl";
import { parseRobotsTxt } from "@/server/lib/audit/discovery";
import type { RenderedPageService } from "@/server/lib/audit/rendered-page";
+import {
+ estimateRenderingCredits,
+ renderUsageCredits,
+} from "@/shared/audit-rendering";
const ORIGIN = "https://example.com";
const HTML = "<html><title>A page</title><body><h1>A page</h1></body></html>";
@@ -108,6 +112,7 @@ function crawl(
maxPages = 100,
renderJavaScript = false,
sitePlatform?: "shopify",
+ renderHold?: { deadlineAt: number; credits: number },
) {
urls = Array.from({ length: maxPages }, (_, i) => `${ORIGIN}/${i}`);
return runCrawlPhase(
@@ -127,6 +132,7 @@ function crawl(
sitePlatform,
robots: parseRobotsTxt(ORIGIN, ""),
renderUsage,
+ renderHold,
},
);
}
@@ -303,7 +309,10 @@ describe("crawl pacing and cooldowns", () => {
const fetchMock = serve(() => 429);
const result = crawl();
await vi.runAllTimersAsync();
- expect(await result).toMatchObject({ completed: false, rateLimited: true });
+ expect(await result).toMatchObject({
+ completed: false,
+ stoppedBy: "crawl-rate-limited",
+ });
expect(fetchMock).toHaveBeenCalledTimes(6);
expect(saved).toHaveLength(3);
expect(saved.every((page) => page.fetchClass === "rate_limited")).toBe(
@@ -320,7 +329,7 @@ describe("crawl pacing and cooldowns", () => {
expect(await result).toEqual({
pagesCrawled: 1,
completed: false,
- rateLimited: true,
+ stoppedBy: "crawl-rate-limited",
});
expect(fetchMock).toHaveBeenCalledTimes(1);
expect(mocks.sleepUntil).not.toHaveBeenCalled();
@@ -347,6 +356,41 @@ describe("rendering usage", () => {
});
});
+describe("rendering credit hold", () => {
+ it("stops starting pages at the deadline so the hold settles", async () => {
+ const fetchMock = serve(() => 200);
+ mocks.renderPage.mockResolvedValue({ html: HTML, status: 200 });
+ const run = crawl(250, true, undefined, {
+ deadlineAt: 5_000,
+ credits: Infinity,
+ });
+ await vi.runAllTimersAsync();
+ expect(await run).toMatchObject({
+ completed: false,
+ stoppedBy: "crawl-time-limit",
+ });
+ // One request a second from t=0, none started at or after the deadline.
+ expect(fetchMock).toHaveBeenCalledTimes(5);
+ });
+
+ it("stops before rendering could cost more than the hold", async () => {
+ serve(() => 200);
+ mocks.renderPage.mockImplementation(async (_url, { usage }) => {
+ usage.cloudflareAttempts += 1;
+ usage.contextCredits += 1;
+ return { html: HTML, status: 200 };
+ });
+ const credits = estimateRenderingCredits(10).high;
+ const run = crawl(100, true, undefined, { deadlineAt: Infinity, credits });
+ await vi.runAllTimersAsync();
+ expect(await run).toMatchObject({
+ completed: false,
+ stoppedBy: "crawl-credits-exhausted",
+ });
+ expect(renderUsageCredits(renderUsage)).toBeLessThanOrEqual(credits);
+ });
+});
+
describe("throttled Shopify pages", () => {
it("are rendered instead of waiting when the audit renders JavaScript", async () => {
serve(() => 429);
@@ -364,6 +408,9 @@ describe("throttled Shopify pages", () => {
mocks.renderPage.mockRejectedValue(new Error("Unable to render this page"));
const run = crawl(100, true, "shopify");
await vi.runAllTimersAsync();
- expect(await run).toMatchObject({ completed: false, rateLimited: true });
+ expect(await run).toMatchObject({
+ completed: false,
+ stoppedBy: "crawl-rate-limited",
+ });
});
});
diff --git a/src/server/workflows/siteAuditWorkflowCrawl.ts b/src/server/workflows/siteAuditWorkflowCrawl.ts
index 39b3b69d..a9a92fba 100644
--- a/src/server/workflows/siteAuditWorkflowCrawl.ts
+++ b/src/server/workflows/siteAuditWorkflowCrawl.ts
@@ -2,18 +2,11 @@ import { env, type WorkflowStep } from "cloudflare:workers";
import { NonRetryableError } from "cloudflare:workflows";
import type { RobotsResult } from "@/server/lib/audit/discovery";
import type { AuditConfig, CrawledPageResult } from "@/server/lib/audit/types";
-import { isAuditablePageUrl, isSameOrigin } from "@/server/lib/audit/url-utils";
-import { isCrawlableUrl } from "@/server/lib/audit/url-policy";
-import { deterministicAuditRowId } from "@/server/lib/audit/ids";
-import { runPageReporters } from "@/server/lib/audit/issues/page-reporters";
-import { AuditRepository } from "@/server/features/audit/repositories/AuditRepository";
import {
getAuditScratchpad,
type ClaimedUrl,
- type FrontierStats,
- type ScratchpadPageLinksRow,
} from "@/server/features/audit/AuditScratchpad";
-import { AuditProgressKV } from "@/server/lib/audit/progress-kv";
+import { persistCrawledPages } from "@/server/workflows/siteAuditWorkflowPersist";
import {
adjustCrawlWindow,
clampCrawlWindow,
@@ -25,7 +18,11 @@ import {
type CrawlThrottleState,
} from "@/server/lib/audit/crawl-throttle";
import { crawlPage } from "@/server/workflows/site-audit-workflow-helpers";
-import type { RenderUsage } from "@/shared/audit-rendering";
+import {
+ estimateRenderingCredits,
+ renderUsageCredits,
+ type RenderUsage,
+} from "@/shared/audit-rendering";
import type { CrawlerAccess } from "@/shared/crawler-access";
import { pgStep } from "@/server/workflows/pgStep";
import { CRAWL_CHUNK_STEP } from "@/server/workflows/auditStepConfigs";
@@ -60,36 +57,6 @@ const FIRST_PERSIST_BATCH_SIZE = 5;
*/
const MAX_QUEUED_PERSIST_BATCHES = 2;
-/**
- * Mega-menu/footer-heavy sites can carry 1000+ links per page; cap what we
- * record so a 10k-page crawl can't produce tens of millions of link targets
- * to scan at finalize.
- */
-const MAX_STORED_LINKS_PER_PAGE = 500;
-/**
- * Cap newly discovered URLs sent to the scratchpad per persist sub-batch.
- * Serialized RPC messages are limited to 32 MiB; a crawler-trap page family
- * (faceted nav, calendars) can emit tens of thousands of unique URLs per
- * page. Dropped URLs are usually re-discovered from later pages, and a site
- * generating this many is past maxPages anyway.
- */
-const MAX_DISCOVERED_PER_BATCH = 20_000;
-const MAX_PROGRESS_TITLE_CHARS = 300;
-
-function shouldQueueCrawlLink(
- link: string,
- origin: string,
- robots: RobotsResult,
- sitePlatform: AuditConfig["sitePlatform"],
-): boolean {
- return (
- isSameOrigin(link, origin) &&
- isCrawlableUrl(link) &&
- isAuditablePageUrl(link, sitePlatform) &&
- robots.isAllowed(link)
- );
-}
-
type CrawlPhaseParams = {
auditId: string;
workflowInstanceId: string;
@@ -107,13 +74,22 @@ type CrawlPhaseParams = {
renderUsage: RenderUsage;
/** Crawler-access headers for the audited host, when the org has one. */
access?: CrawlerAccess | null;
+ /**
+ * A metered rendered audit's credit hold. No fetch starts after its
+ * deadline or once rendering it could cost more than it holds.
+ */
+ renderHold?: { deadlineAt: number; credits: number };
};
export type CrawlPhaseResult = {
pagesCrawled: number;
/** True when the frontier was exhausted before hitting maxPages. */
completed: boolean;
- rateLimited?: boolean;
+ /** Why the crawl stopped before running out of URLs or pages. */
+ stoppedBy?:
+ | "crawl-rate-limited"
+ | "crawl-time-limit"
+ | "crawl-credits-exhausted";
};
export async function runCrawlPhase(
@@ -155,12 +131,13 @@ export async function runCrawlPhase(
result.renderUsage?.cloudflareAttempts ?? 0;
params.renderUsage.contextCredits +=
result.renderUsage?.contextCredits ?? 0;
- if (result.rateLimited) {
- return {
- pagesCrawled: attemptedTotal,
- completed: false,
- rateLimited: true,
- };
+ const stoppedBy = result.rateLimited
+ ? "crawl-rate-limited"
+ : result.timedOut
+ ? "crawl-time-limit"
+ : result.outOfCredits && "crawl-credits-exhausted";
+ if (stoppedBy) {
+ return { pagesCrawled: attemptedTotal, completed: false, stoppedBy };
}
// `?? initial`: an instance in flight across a deploy replays cached
// step results from before endWindow existed.
@@ -197,6 +174,8 @@ async function runCrawlChunk(
pending: number;
endWindow: number;
rateLimited?: boolean;
+ timedOut?: boolean;
+ outOfCredits?: boolean;
throttleState?: CrawlThrottleState;
resumeAt?: number;
renderUsage?: RenderUsage;
@@ -204,6 +183,7 @@ async function runCrawlChunk(
const { auditId, workflowInstanceId, origin, maxPages, robots, chunkNo } =
input;
const scratchpad = getAuditScratchpad(auditId);
+ const renderDeadlineAt = input.renderHold?.deadlineAt ?? Infinity;
let previousThrottle =
(await scratchpad.getCrawlThrottle()) ?? input.throttleState;
@@ -215,7 +195,10 @@ async function runCrawlChunk(
chunkNo,
claimLimit,
);
- const deadlineAt = Date.now() + CHUNK_SOFT_DEADLINE_MS;
+ const deadlineAt = Math.min(
+ Date.now() + CHUNK_SOFT_DEADLINE_MS,
+ renderDeadlineAt,
+ );
if (isRetry && previousThrottle) {
// Requests made since the last checkpoint may have consumed a slot.
previousThrottle = {
@@ -305,6 +288,18 @@ async function runCrawlChunk(
// This attempt's rendering counts, returned in the step result. The audit's
// credit lock is settled once, when the audit ends.
const renderUsage: RenderUsage = { cloudflareAttempts: 0, contextCredits: 0 };
+ // Whether `pending` in-flight pages and one more, each at the Context worst
+ // case, still fit the hold on top of what this audit has rendered so far.
+ const canAffordRender = (pending: number) =>
+ !input.renderHold ||
+ renderUsageCredits({
+ cloudflareAttempts:
+ input.renderUsage.cloudflareAttempts + renderUsage.cloudflareAttempts,
+ contextCredits:
+ input.renderUsage.contextCredits + renderUsage.contextCredits,
+ }) +
+ estimateRenderingCredits(pending + 1).high <=
+ input.renderHold.credits;
const render = input.renderJavaScript
? async (url: string) => {
const { RenderedPageService } =
@@ -357,7 +352,11 @@ async function runCrawlChunk(
// queuedPersists changes when persistChain settles. Keep it out of the
// loop condition because the type-aware linter cannot see that async
// mutation and flags the otherwise valid backpressure check.
- if (throttle.stopped || queuedPersists > MAX_QUEUED_PERSIST_BATCHES)
+ if (
+ throttle.stopped ||
+ queuedPersists > MAX_QUEUED_PERSIST_BATCHES ||
+ !canAffordRender(inFlight.size)
+ )
break;
launch(claimed[nextIndex]);
nextIndex += 1;
@@ -404,6 +403,16 @@ async function runCrawlChunk(
pending: stats.pending,
endWindow: windowSize,
rateLimited: throttle.stopped,
+ outOfCredits: unattempted.length > 0 && !canAffordRender(0),
+ // Work is left and neither now nor the site's pacing allows another
+ // request before the deadline, so no later chunk could start one either.
+ timedOut:
+ unattempted.length > 0 &&
+ Math.max(
+ Date.now(),
+ throttleState.pausedUntil,
+ throttleState.nextRequestAt,
+ ) >= renderDeadlineAt,
throttleState,
resumeAt:
throttleState.pausedUntil > Date.now()
@@ -412,94 +421,3 @@ async function runCrawlChunk(
renderUsage,
};
}
-
-async function persistCrawledPages(input: {
- auditId: string;
- workflowInstanceId: string;
- origin: string;
- robots: RobotsResult;
- sitePlatform: AuditConfig["sitePlatform"];
- scratchpad: ReturnType<typeof getAuditScratchpad>;
- pages: CrawledPageResult[];
- depthByUrl: Map<string, number | null>;
- maxPages: number;
-}): Promise<FrontierStats> {
- const {
- auditId,
- origin,
- robots,
- sitePlatform,
- scratchpad,
- pages,
- depthByUrl,
- } = input;
-
- // Deterministic ids keep every write idempotent across step retries.
- for (const page of pages) {
- page.id = await deterministicAuditRowId(auditId, page.url);
- }
- const issues = pages.flatMap((page) => runPageReporters(page));
- await AuditRepository.insertCrawledBatch(auditId, pages, issues);
-
- const links: ScratchpadPageLinksRow[] = [];
- const discovered = new Map<string, number | null>();
- for (const page of pages) {
- const pageDepth = depthByUrl.get(page.url) ?? null;
- const childDepth = pageDepth === null ? null : pageDepth + 1;
-
- const targets: string[] = [];
- for (const link of page.links) {
- if (!link.isInternal) continue;
- if (targets.length < MAX_STORED_LINKS_PER_PAGE) {
- targets.push(link.targetUrl);
- }
- if (
- discovered.size < MAX_DISCOVERED_PER_BATCH &&
- !discovered.has(link.targetUrl) &&
- shouldQueueCrawlLink(link.targetUrl, origin, robots, sitePlatform)
- ) {
- discovered.set(link.targetUrl, childDepth);
- }
- }
- if (targets.length > 0) {
- links.push({ pageId: page.id, url: page.url, targets });
- }
-
- // Redirect targets continue the same navigation path: same depth.
- if (
- page.redirectUrl &&
- !discovered.has(page.redirectUrl) &&
- shouldQueueCrawlLink(page.redirectUrl, origin, robots, sitePlatform)
- ) {
- discovered.set(page.redirectUrl, pageDepth);
- }
- }
-
- const stats = await scratchpad.recordBatch({
- crawledUrls: pages.map((page) => page.url),
- pages: pages.map((page) => ({
- pageId: page.id,
- url: page.url,
- statusCode: page.statusCode,
- fetchClass: page.fetchClass,
- redirectUrl: page.redirectUrl,
- })),
- links,
- discovered: Array.from(discovered, ([url, depth]) => ({ url, depth })),
- });
-
- await AuditRepository.updateAuditProgress(auditId, input.workflowInstanceId, {
- pagesCrawled: stats.attempted,
- pagesTotal: Math.min(stats.seen, input.maxPages),
- });
- await AuditProgressKV.pushCrawledUrls(
- auditId,
- pages.map((page) => ({
- url: page.url,
- statusCode: page.statusCode,
- title: page.title.slice(0, MAX_PROGRESS_TITLE_CHARS),
- crawledAt: Date.now(),
- })),
- );
- return stats;
-}
diff --git a/src/server/workflows/siteAuditWorkflowPersist.ts b/src/server/workflows/siteAuditWorkflowPersist.ts
new file mode 100644
index 00000000..1323c430
--- /dev/null
+++ b/src/server/workflows/siteAuditWorkflowPersist.ts
@@ -0,0 +1,138 @@
+import type { RobotsResult } from "@/server/lib/audit/discovery";
+import type { AuditConfig, CrawledPageResult } from "@/server/lib/audit/types";
+import { isAuditablePageUrl, isSameOrigin } from "@/server/lib/audit/url-utils";
+import { isCrawlableUrl } from "@/server/lib/audit/url-policy";
+import { deterministicAuditRowId } from "@/server/lib/audit/ids";
+import { runPageReporters } from "@/server/lib/audit/issues/page-reporters";
+import { AuditRepository } from "@/server/features/audit/repositories/AuditRepository";
+import type {
+ getAuditScratchpad,
+ FrontierStats,
+ ScratchpadPageLinksRow,
+} from "@/server/features/audit/AuditScratchpad";
+import { AuditProgressKV } from "@/server/lib/audit/progress-kv";
+
+/**
+ * Mega-menu/footer-heavy sites can carry 1000+ links per page; cap what we
+ * record so a 10k-page crawl can't produce tens of millions of link targets
+ * to scan at finalize.
+ */
+const MAX_STORED_LINKS_PER_PAGE = 500;
+/**
+ * Cap newly discovered URLs sent to the scratchpad per persist sub-batch.
+ * Serialized RPC messages are limited to 32 MiB; a crawler-trap page family
+ * (faceted nav, calendars) can emit tens of thousands of unique URLs per
+ * page. Dropped URLs are usually re-discovered from later pages, and a site
+ * generating this many is past maxPages anyway.
+ */
+const MAX_DISCOVERED_PER_BATCH = 20_000;
+const MAX_PROGRESS_TITLE_CHARS = 300;
+
+function shouldQueueCrawlLink(
+ link: string,
+ origin: string,
+ robots: RobotsResult,
+ sitePlatform: AuditConfig["sitePlatform"],
+): boolean {
+ return (
+ isSameOrigin(link, origin) &&
+ isCrawlableUrl(link) &&
+ isAuditablePageUrl(link, sitePlatform) &&
+ robots.isAllowed(link)
+ );
+}
+
+/**
+ * Persists one crawled sub-batch: page rows and issues to the app DB, links
+ * and newly discovered URLs to the scratchpad, and progress for the UI.
+ */
+export async function persistCrawledPages(input: {
+ auditId: string;
+ workflowInstanceId: string;
+ origin: string;
+ robots: RobotsResult;
+ sitePlatform: AuditConfig["sitePlatform"];
+ scratchpad: ReturnType<typeof getAuditScratchpad>;
+ pages: CrawledPageResult[];
+ depthByUrl: Map<string, number | null>;
+ maxPages: number;
+}): Promise<FrontierStats> {
+ const {
+ auditId,
+ origin,
+ robots,
+ sitePlatform,
+ scratchpad,
+ pages,
+ depthByUrl,
+ } = input;
+
+ // Deterministic ids keep every write idempotent across step retries.
+ for (const page of pages) {
+ page.id = await deterministicAuditRowId(auditId, page.url);
+ }
+ const issues = pages.flatMap((page) => runPageReporters(page));
+ await AuditRepository.insertCrawledBatch(auditId, pages, issues);
+
+ const links: ScratchpadPageLinksRow[] = [];
+ const discovered = new Map<string, number | null>();
+ for (const page of pages) {
+ const pageDepth = depthByUrl.get(page.url) ?? null;
+ const childDepth = pageDepth === null ? null : pageDepth + 1;
+
+ const targets: string[] = [];
+ for (const link of page.links) {
+ if (!link.isInternal) continue;
+ if (targets.length < MAX_STORED_LINKS_PER_PAGE) {
+ targets.push(link.targetUrl);
+ }
+ if (
+ discovered.size < MAX_DISCOVERED_PER_BATCH &&
+ !discovered.has(link.targetUrl) &&
+ shouldQueueCrawlLink(link.targetUrl, origin, robots, sitePlatform)
+ ) {
+ discovered.set(link.targetUrl, childDepth);
+ }
+ }
+ if (targets.length > 0) {
+ links.push({ pageId: page.id, url: page.url, targets });
+ }
+
+ // Redirect targets continue the same navigation path: same depth.
+ if (
+ page.redirectUrl &&
+ !discovered.has(page.redirectUrl) &&
+ shouldQueueCrawlLink(page.redirectUrl, origin, robots, sitePlatform)
+ ) {
+ discovered.set(page.redirectUrl, pageDepth);
+ }
+ }
+
+ const stats = await scratchpad.recordBatch({
+ crawledUrls: pages.map((page) => page.url),
+ pages: pages.map((page) => ({
+ pageId: page.id,
+ url: page.url,
+ statusCode: page.statusCode,
+ fetchClass: page.fetchClass,
+ redirectUrl: page.redirectUrl,
+ })),
+ links,
+ discovered: Array.from(discovered, ([url, depth]) => ({ url, depth })),
+ });
+
+ await AuditRepository.updateAuditProgress(auditId, input.workflowInstanceId, {
+ pagesCrawled: stats.attempted,
+ pagesTotal: Math.min(stats.seen, input.maxPages),
+ });
+ await AuditProgressKV.pushCrawledUrls(
+ auditId,
+ pages.map((page) => ({
+ url: page.url,
+ statusCode: page.statusCode,
+ title: page.title.slice(0, MAX_PROGRESS_TITLE_CHARS),
+ crawledAt: Date.now(),
+ })),
+ );
+ return stats;
+}
diff --git a/src/server/workflows/siteAuditWorkflowPhases.ts b/src/server/workflows/siteAuditWorkflowPhases.ts
index 1bb36d0d..f91143a1 100644
--- a/src/server/workflows/siteAuditWorkflowPhases.ts
+++ b/src/server/workflows/siteAuditWorkflowPhases.ts
@@ -54,6 +54,8 @@ type AuditPhasesParams = {
config: AuditConfig;
access?: CrawlerAccess | null;
renderUsage: RenderUsage;
+ /** A metered rendered audit's credit hold, which the crawl must stay in. */
+ renderHold?: { deadlineAt: number; credits: number };
};
export async function runAuditPhases(
@@ -378,9 +380,9 @@ async function finalizeAudit(args: {
issueCount: linkIssues.length,
});
issues.push(...linkIssues);
- if (crawl.rateLimited) {
+ if (crawl.stoppedBy) {
issues.push({
- issueType: "crawl-rate-limited",
+ issueType: crawl.stoppedBy,
pageId: null,
pageUrl: startUrl,
});
diff --git a/src/shared/audit-issues.ts b/src/shared/audit-issues.ts
index e99c225f..57484b3a 100644
--- a/src/shared/audit-issues.ts
+++ b/src/shared/audit-issues.ts
@@ -6,6 +6,9 @@
* of these types by id.
*/
+// Relative: tests/badseo type-checks this file without the @ alias.
+import { RENDERED_CRAWL_TIME_LIMIT_HOURS } from "./audit-limits";
+
export type IssueSeverity = "critical" | "warning" | "info";
interface AuditIssueDescriptor {
@@ -48,6 +51,21 @@ export const AUDIT_ISSUE_TYPES = {
howToFix:
"Re-run the audit after the site's rate limit resets, or ask the site owner to allow the OpenSEO-Audit crawler (https://openseo.so/docs/crawler). On Shopify, use Crawler access instead (Online Store → Preferences → Crawler access) and paste the signature into OpenSEO under Settings → Crawler access.",
},
+ "crawl-time-limit": {
+ severity: "warning",
+ title: "Crawl stopped early: time limit",
+ explanation: `Audits that render JavaScript stop crawling after ${RENDERED_CRAWL_TIME_LIMIT_HOURS} hours, and this one reached that limit. This report is incomplete; URLs we did not fetch are not recorded as broken.`,
+ howToFix:
+ "Re-run the audit with a lower page limit, or without Render JavaScript if the site's content and links are in its HTML. Rendering is billed only for pages actually rendered.",
+ },
+ "crawl-credits-exhausted": {
+ severity: "warning",
+ title: "Crawl stopped early: cost limit reached",
+ explanation:
+ "Many pages on this site needed our advanced renderer, so the audit reached its cost limit and stopped. You were only charged for the pages we rendered.",
+ howToFix:
+ "If the site doesn't need JavaScript to show its content, re-run with Render JavaScript off.",
+ },
"server-error": {
severity: "critical",
title: "Server error (5xx)",
diff --git a/src/shared/audit-limits.ts b/src/shared/audit-limits.ts
index 9d1f7610..58469e65 100644
--- a/src/shared/audit-limits.ts
+++ b/src/shared/audit-limits.ts
@@ -4,6 +4,7 @@ export const MIN_AUDIT_PAGES = 10;
export const DEFAULT_AUDIT_PAGES = 50;
export const FREE_MAX_AUDIT_PAGES = 50;
export const PAID_MAX_AUDIT_PAGES = 10_000;
-// A rendered audit's credit hold is an Autumn lock, which lasts at most 24
-// hours. This keeps a rendered crawl well inside that window.
-export const RENDERED_MAX_AUDIT_PAGES = 1_000;
+// A hosted rendered audit's credit hold is an Autumn lock, which lasts at most
+// 24 hours. Its crawl stops after this long, so the audit still settles
+// against the lock with time to spare for Lighthouse and finalization.
+export const RENDERED_CRAWL_TIME_LIMIT_HOURS = 22;
diff --git a/src/shared/audit-rendering.ts b/src/shared/audit-rendering.ts
index 6994beda..08f5d034 100644
--- a/src/shared/audit-rendering.ts
+++ b/src/shared/audit-rendering.ts
@@ -37,8 +37,8 @@ export function renderUsageCredits(usage: RenderUsage): number {
/**
* Usage-credit range for rendering up to `maxPages` pages. The low end renders
- * every page on Cloudflare. The high end is every page falling back to Context
- * after its Cloudflare attempt, which is also what a hosted audit reserves.
+ * every page on Cloudflare, which is what most sites cost. The high end is
+ * every page falling back to Context after its Cloudflare attempt.
*/
export function estimateRenderingCredits(maxPages: number) {
return {
@@ -53,6 +53,15 @@ export function estimateRenderingCredits(maxPages: number) {
};
}
+/**
+ * What a hosted rendered audit holds when it starts, and the most it charges:
+ * twice the Cloudflare-only estimate. Most pages render on Cloudflare, so
+ * this covers almost every audit without holding the Context worst case.
+ */
+export function renderingHoldCredits(maxPages: number) {
+ return estimateRenderingCredits(maxPages).low * 2;
+}
+
/**
* Usage credits as the dollars users see everywhere else. Rounded up to the
* cent, so an account holding the shown amount always covers the hold.
@@ -64,11 +73,16 @@ function formatRenderingUsd(credits: number) {
/** Why a hosted account cannot start a rendered audit of `maxPages` pages. */
export function renderingCreditsNeededText(maxPages: number) {
- const { high } = estimateRenderingCredits(maxPages);
- return `Rendering up to ${maxPages.toLocaleString("en-US")} pages needs ${formatRenderingUsd(high)} of credits available. Lower the page limit or add credits.`;
+ return `Rendering up to ${maxPages.toLocaleString("en-US")} pages needs ${formatRenderingUsd(renderingHoldCredits(maxPages))} of credits available. Lower the page limit or add credits.`;
+}
+
+export const RENDERING_COST_NOTE =
+ "Sites that block our browser need an advanced renderer. If the cost is higher than expected, it will stop.";
+
+export function renderingCostText(maxPages: number) {
+ return `Costs up to ${formatRenderingUsd(renderingHoldCredits(maxPages))}.`;
}
export function renderingEstimateText(maxPages: number) {
- const { low, high } = estimateRenderingCredits(maxPages);
- return `Estimated ${formatRenderingUsd(low)} to ${formatRenderingUsd(high)} for up to ${maxPages.toLocaleString("en-US")} pages. Sites with aggressive bot detection need our more capable renderer and cost more.`;
+ return `${renderingCostText(maxPages)} ${RENDERING_COST_NOTE}`;
}