Open source alternative to Semrush and Ahrefs
TypeScript backlink-analysisgoogle-search-console-mcpkeyword-researchmcpseoseo-agentseo-auditseo-automationseo-skillsseo-toolssite-audit
Site audit P0 (1/3): issue engine, incremental persistence, block detection (#362)
Ben Senescu1c74fded7b59
diff --git a/docs/site-audit-pm-research.md b/docs/site-audit-pm-research.mdnew file mode 100644index 00000000..f4107c55--- /dev/null+++ b/docs/site-audit-pm-research.md@@ -0,0 +1,142 @@+# OpenSEO Site Audit: 80/20 Feature Set & Crawl Architecture Proposal (rev. 3, founder decision applied)++**Author:** PM research synthesis | **Date:** 2026-06-09 | **Inputs:** SEOnaut + LibreCrawl code reviews, OpenSEO baseline review, competitor/market research, architecture research, adversarial panel review. Revision notes: panel critiques accepted are folded in silently; where we overrode the panel, an explicit **Panel pushback** note marks the disagreement. **Rev. 3:** founder decision — build on the in-house Workers crawler, do not adopt DataForSEO OnPage; "we were blocked" flagging accepted as the blocking posture. §4 now weighs that decision's tradeoffs instead of recommending an engine.++---++## 1. How site audits actually work (primer)++Every site audit tool is the same three-stage pipeline:++1. **Crawl.** Start from a URL, fetch robots.txt and sitemaps, fetch pages, extract links, enqueue new same-site URLs (BFS), repeat until a page cap. Two engine choices matter: _plain HTTP fetch_ (fast, cheap, blind to JavaScript) vs _headless rendering_ (sees what Google sees, ~10x cost). JS rendering is table stakes in commercial tools; differentiation is fidelity and whether it's paywalled.+2. **Check.** Per-page rules (missing title, noindex, broken canonical) plus cross-crawl rules (duplicate titles, redirect chains, orphan pages). The cross-page checks carry most of the value, and they're set-queries over crawl data, not parsing.+3. **Report.** Severity grouping, plain-English explanations, drill-down to affected URLs, **export**, and diffs against previous crawls.++**Desktop vs cloud.** Screaming Frog ($259/yr) and Sitebulb run on the user's laptop: residential IP, no bot-blocking, but no scheduling, RAM-limited crawls (the #1 SF complaint), and the machine must be on. Cloud tools (Ahrefs, Semrush, Moz) crawl datacenter IPs on a schedule — and **getting blocked is their #1 recurring complaint**: WAFs 403/challenge their bots, producing phantom 404s and false positives.++**Why blocking happens.** WAFs score IP reputation (datacenter = suspicious) and fingerprint (no JS = bot). It hits every cloud auditor, including ours. The industry playbook: documented UAs, published allowlistable IPs, Cloudflare's Verified Bots program, and ownership verification to justify higher crawl rates (Ahrefs advertises dramatically higher rates for verified owners; the oft-quoted 30→30,000 URLs/min figure is unsourced in our fact base and Ahrefs' verification is its own, not GSC — treat as directional only).++---++## 2. Where OpenSEO's audit stands today++Honest summary: **we have a crawler and a Lighthouse integration, but not an audit product.** We extract per-page fields and show filterable tables, but no rule engine names issues; the only "issues" users see are Lighthouse failures. SEOnaut ships ~88 issue types; LibreCrawl ~26; Screaming Frog ~300. We effectively ship ~2 as table styling.++Structural problems, in priority order:++- **The link graph is thrown away.** Link lists are extracted but only counts persist to D1, so broken links, orphans, redirect chains, and inlink analysis are impossible without schema change.+- **Persistence is finalize-only and Workflow-shaped.** `runCrawlBatch` returns full page arrays (links, images) as durable step state (~1MiB/step cap — already borderline on link-dense sites), `allPages` accumulates everything in 128MB Worker memory, and `batchWriteResults` writes all rows inside the single finalize step (DB_BATCH_SIZE=100 → a 1M-row link graph would mean ~10,000 subrequests in one step, ~10x the limit). **Any link-graph feature requires restructuring persistence first.** This was deferred to P1 in draft 1; the panel correctly identified that as backwards — it is now the P0 prerequisite.+- **Step budget caps crawl size — ledger, not vibes.** Workflows allows ~1,024 steps/instance. Today: 3 steps per 25-page batch → ~341 batches ≈ **~8.5k pages** fits (minus discovery/finalize overhead), so the existing 10,000-page clamp slightly exceeds the budget. The sharper constraint is Lighthouse strategy `all`: 2 steps per 10-URL batch ≈ 1,000 steps at 5k pages — over budget on its own. Politeness delays must live _inside_ steps (wall-clock waits are fine), never as smaller batches. See §4 for the post-refactor ledger and caps.+- **No audit MCP tools** — keywords, SERP, backlinks, rank-tracker, GSC all have them; audit has zero (verified in `src/server/mcp/tools`). Ahrefs, meanwhile, **already ships site-audit MCP tools** (`site-audit-issues`, `site-audit-page-explorer`, etc.); Semrush has an MCP server too. We are behind here, not ahead — see §3.4 for what our actual wedge is.+- **SSRF gap:** `url-policy.ts` validates only the start URL; discovered links are fetched unvalidated. Must be fixed in P0 for as long as the in-house crawler runs.++Condensed check inventory: status/redirects, title/meta, canonicals, indexability, headings, images = **partial** (extracted, unflagged; redirects followed silently; header variants like `X-Robots-Tag` unchecked). Duplicates (30–35% of sites, Semrush study¹), broken links (52% of domains¹), redirect chains (~12%¹), orphans, security headers, hreflang, JS rendering, scheduling/diffs, issue UX, MCP, block resilience = **missing**. Lighthouse/CWV = **strong**, better than both OSS tools. _(¹ Prevalence figures from market research §3; primary sources to be attached as a footnoted appendix before this circulates externally.)_++(Folklore correction stands: the 100k-page lifetime cap settles to actual crawled counts; only failed/stuck audits permanently pay their reservation.)++---++## 3. The 80/20 feature set++**Pricing posture (decides everything downstream).** Audit is the most commoditized feature in SEO — Ahrefs Webmaster Tools and Screaming Frog ≤500 URLs are free. Per our SEO strategy (goal = signups), **audit is acquisition, not monetization**: default 50-page audits are free per site; credits are charged for large crawls, scheduling, and history. The free posture also reinforces the engine decision (§4): with the in-house crawler, a free audit's marginal cost is D1 writes and Workers compute largely inside quotas we already pay for — no per-audit vendor bill, no metered cost on failed runs (DataForSEO charges even for failed tasks).++The prevalence evidence says a credible audit needs ~6 check families, not 300. P0 below runs on the **in-house crawler (founder decision, §4)**; crawler-side hardening is therefore mandatory P0 work, not contingency. Everything except the crawler-side items remains **engine-independent by design** (schema, SQL checks, UX, MCP) — that portability is the cheap hedge that keeps a future vendor or local-agent engine swap on the table.++### P0 — the audit becomes real (~6–8 weeks)++**0. Incremental persistence (prerequisite, was P1).** Write `audit_pages`/`audit_links` rows to D1 _inside each crawl-batch step_ (one extra `db.batch` per batch keeps subrequests bounded), return only slim summaries (id, url, status) as step state, drop the `allPages` accumulator. Finalize becomes: multipage SQL step → completion. Bonus: failed audits keep partial results. Also fix the SSRF gap (run discovered links through url-policy).++**1. Issue engine + schema.** `audit_links` edge table (source, target, anchor, rel, internal) and `audit_issues` table. **Reporter contract (decided now, per panel):** reporters consume a normalized page record — `(record: AuditPageRecord) => boolean` where the record includes response headers captured at crawl time — _not_ SEOnaut's `(page, $, headers)` cheerio signature. Cheerio runs once in `crawlPage`; reporters are DOM-free. This is what makes the engine portable across our crawler, a future local agent, or a vendor feed if the §4 revisit triggers ever fire. Cross-page checks are D1 SQL passes (duplicates = `GROUP BY hash HAVING count > 1`, per SEOnaut's multipage reporters). **Effort: 2–3 weeks including #0's restructure** (was "1 week" — panel was right).++**2. High-prevalence checks** (post-#1; thresholds cribbed from SEOnaut/LibreCrawl):++- _Broken internal links + redirect chains/loops_: switch to `redirect: "manual"`, record hops, enqueue targets. Decision: redirect hops do **not** count against maxPages or billing (they're cheap HEAD-sized fetches; metering/progress semantics budgeted at +1 day). Broken-link checks fire **only for targets we actually fetched or HEAD-checked** — never inferred from absence on partial crawls. ~4 days.+- _Duplicates_: body-hash group-by catches mirrored/parameterized duplicates only (any dynamic token defeats exact hashing); **duplicate titles/descriptions group-by is the workhorse** and carries the prevalence claim. Avoid LibreCrawl's O(n²) similarity scan. ~1–2 days.+- _Title/meta length, missing/multiple H1, heading order_ (SEOnaut thresholds). ~1–2 days.+- _Canonical + indexability with header fallbacks_ (`X-Robots-Tag`, `Link: rel=canonical` — SEOnaut's standout trick; headers are in the normalized record by design). ~2 days.+- _Orphan pages_: reorder seeding to **sitemap-last** (we currently seed sitemap URLs _first_, the opposite of the SEOnaut pattern we cite, so sitemap pages eat the page budget and link discovery never exhausts), and **emit orphan findings only when the crawl completed** (queue exhausted below maxPages) — on truncated crawls "no observed inlinks" is true of nearly everything and would mass-false-positive, the exact Semrush health-score noise we mock. ~3–4 days including reordering and gating.+- _Thin content + crawl depth_: explicit disposition — **build in P0** (word count already stored; BFS depth is one column). _Page size_: P1. ~1 day.++**3. Issues UX.** Critical/Warning/Info grouping, one-paragraph plain-English explanation per type (adapt SEOnaut's YAML copy), affected-URL drill-down, LibreCrawl's ~100-glob default exclusions (wp-admin, checkout, feeds), and **CSV export** (previously unaccounted; it's one of the three core report capabilities from §1). ~4–5 days.++**4. MCP audit tools — reframed.** Draft 1 claimed "a demo no competitor can match." **That was wrong: Ahrefs ships site-audit MCP tools today and Semrush has an MCP server.** Our wedge is not MCP existence; it's **credit-priced, zero-setup, agent-actionable audits**: audit any URL on demand for cents with no project setup or $129+/mo subscription, and return per-issue _fix instructions an agent can execute_ (the issue record carries remediation steps, not just labels). Tool shapes designed against Ahrefs' actual, inspectable MCP audit tools: `run_site_audit`, `get_audit_status`, `get_audit_issues(severity, type)`, `get_audit_pages(filter)`, each issue carrying `how_to_fix`. ~4 days.++**5. Block detection — moved to weeks 1–2 (was week 5).** The panel is right that for our segment (indie founders on Vercel/Cloudflare/Netlify — bot protection there is a one-toggle setting and IP-reputation challenges hit datacenter egress even without it; the week-1 baseline will establish the real rate) blocking is a **first-run activation killer**, not an ops metric — and our own onboarding audit shows activation leaks are the existential problem. Classify 403s/challenge bodies/`cf-mitigated` headers as "we were blocked," say so honestly instead of recording error pages, and show owner-targeted allowlist instructions inline (GSC-verified flow, §4). This layer is response-classification, hence engine-independent, and instruments block-rate telemetry from day one. ~3 days.++### P1 — weeks 8–14++- **Scheduled audits + diffing**: extend the existing cron; store per-run issue counts, show deltas ("12 new broken links since last week"). The retention feature and the reason cloud beats desktop. **Data retention policy (new):** full link graphs kept for the latest 2 runs only; older runs keep issue counts/deltas (a 10k-page graph is ~100–150MB against D1's 10GB cap — weekly large audits would eat gigabytes/year otherwise). Large-audit edge lists optionally archived as compressed R2 blobs.+- **AI-crawler accessibility audit — promoted from P2.** Which AI bots (GPTBot, ClaudeBot) can reach your content, robots/Cloudflare AI-block detection, is-key-content-in-static-HTML. The panel is right that our own words ("open greenfield, perfectly on-brand") contradicted P2 placement: it's mostly computable from data we already collect, and "is ChatGPT blocked from your site?" is the launch/marketing asset for the whole relaunch — nobody switches auditors for HSTS checks. (Skip llms.txt as a headline; no major LLM vendor consumes it.)+- **JS rendering — deferred (consequence of the no-vendor decision, §4).** Plain fetch can't render SPAs. P1 ships the honest mitigation instead: detect client-rendered shells (near-empty body text with a large script payload / root-div-only HTML) and flag "this site renders client-side; HTML-level checks are incomplete" rather than reporting false missing-title/thin-content issues. If demand materializes, the revisit path is Cloudflare Browser Rendering as an opt-in, credit-priced add-on for GSC-verified own-site audits (where the owner can allowlist, neutralizing the fingerprint problem) — not a vendor API.+- **Security/hygiene checks** (HTTPS, mixed content, HSTS/CSP, insecure forms) and **external broken links** (HEAD checks, per-crawl cache) — demoted below AI-crawler work.+- **Cloudflare Verified Bots**: reframed as a **lottery ticket, not a mitigation**. Eligibility requires established, meaningful crawl volume; a new low-volume crawler routinely doesn't qualify, and Web Bot Auth is beta on Cloudflare's side. File opportunistically for our own UA (`OpenSEOBot`); plan zero dependency on it. The owner-targeted allowlist flow is the only mitigation we count on.++### P2 — later++- **`npx openseo-audit` local agent** (see §4 option D) — gated on block-rate telemetry, which now exists from week 1.+- **Crawl snapshots in R2** ("what we saw" evidence; lightweight SEOnaut WACZ analog).+- **Hreflang validation, structured-data validation, page-size checks, link-placement classification.**++**Skip entirely (explicit dispositions):** 300-check parity, crawl visualizations, E-E-A-T scores, multi-language UI.++---++## 4. Architecture: where should the crawl run?++| | **A. In-house Workers crawl + mitigations** | **B. DataForSEO OnPage offload** | **C. Desktop app** | **D. Hybrid: cloud + `npx` local agent** |+| -------------------- | -------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------- | ----------------------------------------------------------- |+| Blocking reliability | Low-medium: no stable IPs, no JS-challenge survival | Medium: still datacenter, but published IPs + named UA are _allowlistable_ | Highest (residential IP, real browser) | High when used; manual trigger only |+| Scheduled audits | Yes (≤ a few k pages — step budget) | Yes | **No** | Yes (cloud path); **no for the blocked sites D exists for** |+| JS rendering | No | Yes (priced) | Yes | Yes (local Playwright) |+| Eng effort | Ongoing hardening forever | Low (vendor/SDK/billing integrated) | **Months** + signing/notarization/3-platform QA | ~3–5 weeks (see below) |+| Cost per 1k pages | **Not $0**: D1 link writes ≈ $0.30/1k (with indexes) + Workers/Workflows | ~$0.125–0.50 base, ~$1.25 JS _(list prices from architecture research — re-verify if the §4 revisit triggers ever fire)_ | $0 marginal | $0 marginal + ingest |+| Max crawl size | ~8.5k today, ~12–25k post-refactor (Lighthouse `all` binds first; instance chaining removes the ceiling) | Vendor-side, no Workflow ceiling | RAM-limited | Local machine-limited |+| MCP / billing fit | Good / fine | Good / per-page cost maps to credits | **Bad** / awkward | Excellent for developers / fine |++**Excluded options, explicitly (panel asked):** _Cloudflare Browser Rendering_ — solves JS but not blocking (egress is still Cloudflare datacenter IPs), adds per-browser-hour cost, and headless-Chrome fingerprints start their own anti-bot treadmill. With the vendor path now declined (see decision below), this becomes the only _cloud_ JS-rendering route (the P2 `npx` agent's local Playwright is the other) — scoped as a P2 opt-in for GSC-verified own-site audits where the owner can allowlist us first. _Residential/ISP proxy egress_ — rejected on principle and practice: it disguises crawler identity, which contradicts our documented-UA/allowlist posture and many proxy networks' sourcing is ethically murky; cost (~$1–8/GB) is also worst-in-class.++**Decision: A — in-house Workers crawler (founder call: avoid DataForSEO vendor costs).** The original recommendation was B, contingent on a week-1 spike; the founder has decided against adding vendor spend, and "we were blocked" honest flagging is accepted as the blocking posture. That is a defensible position, but the proposal's job is to price it accurately. What the decision buys, and what it costs:++**What going in-house buys:**++- **No per-audit vendor bill on a free acquisition feature.** Every OnPage call is marginal cash out (~$0.006–0.025 per free 50-page audit, $1.25–5 per 10k-page crawl at list prices), and DataForSEO meters even failed tasks — a real footgun we've already hit on the SERP side. Crucially, the comparison must not double-count: D1 storage writes are common to **both** engines (vendor results would land in the same `audit_pages`/`audit_links` tables to run the same SQL checks), so the vendor fee is purely additive. In-house wins on marginal cost at every crawl size.+- **No new vendor dependency or pricing-change exposure** on the product's centerpiece free feature; full control of crawl behavior, UA identity, and politeness.+- **One less integration**: no task-queue/webhook polling model, no mapping a vendor's page schema into the normalized record.++**What it costs — priced honestly:**++1. **"Basically free" is true only until the free tier works.** D1 Paid includes 50M row-writes/month _shared with the entire product_. A 50-page free audit writes ~8–15k rows (pages + links + indexes), so **~3–5k free audits/month consumes the whole included allotment**; beyond it, $1/M rows — at 10k free audits/month, ~$50–100/month in-house vs ~$60–250/month vendor-side. A ~2x cost win, not zero, and it eats quota headroom from every other D1 feature. Required companion: a **TTL on free-tier audit data** (e.g., purge one-off free audits' link graphs after 30 days) — the P1 two-run retention policy only covers scheduled audits and doesn't protect the 10GB D1 cap from thousands of one-off free crawls.+2. **JS rendering is off the roadmap** until/unless we build it on Cloudflare Browser Rendering. SPA-heavy sites (a real slice of indie-founder stacks: Next.js CSR pages, Vite SPAs) get degraded audits. Mitigation: detect-and-disclose (§3 P1), never silently mis-report — and count CSR-shell prevalence in the week-1 telemetry so the JS-rendering revisit trigger has a denominator too.+3. **A step-budget ceiling — real, but it's Lighthouse that binds, not the crawl.** The ledger: today's 3 steps per 25-page batch supports ~8.5k pages in the 1,024-step budget; after the P0 #0 refactor (1–2 steps/batch), the crawl alone supports **~12–25k pages** — the 10k clamp can stand. The binding constraint is Lighthouse strategy `all` (2 steps per 10-URL batch ≈ 1,000 steps at 5k pages, over budget by itself): cap `all` at ~2k pages or switch to sampling, and document page caps _coupled to Lighthouse strategy_. Nor is the ceiling permanent: Workflow instances can chain (the P0 #0 refactor already puts frontier state in D1; a continuation instance resumes it) — roughly a week of cross-instance dedupe work, not an engine swap. The agency/large-site segment is deferred, not foreclosed.+4. **Weaker blocking mitigation — and the allowlist fix is unverified on the exact tiers our ICP uses.** Workers egress comes from shared Cloudflare IP ranges — we cannot publish a dedicated allowlistable IP, so the GSC-verified fix flow must be **User-Agent-based** (`OpenSEOBot/1.0` via a WAF custom rule; spoofable, tolerable when the owner allowlists their _own_ site). But: on **Cloudflare Free, Bot Fight Mode supports no exceptions** — a skip-by-UA rule does not bypass it; the owner's only fix there is turning BFM off entirely, a much heavier ask than "add this one rule." Vercel's challenge bypass is plan-gated and may be unavailable on Hobby. **Week 1 therefore includes a 1–2 day validation**: crawl test zones on Cloudflare Free (BFM on), Cloudflare Pro (Super BFM), and Vercel Hobby/Pro (Attack Challenge Mode), and write per-platform fix copy that is honest where the real answer is "disable bot protection." Verified Bots filing (P1) stays a lottery ticket.+5. **The ~1–2 weeks of crawler work inside P0 that the vendor path would have eliminated, plus permanent ownership.** Manual-redirect handling, per-host politeness inside steps, sitemap-last seeding, SSRF fix, step-budget restructuring, timeout/retry edges — already counted inside the 6–8-week P0 estimate (not additional to it), but under B most of it would have been deleted, and the hardening treadmill (new WAF behaviors, new edge cases) is ours indefinitely.+6. **No fallback engine for blocked sites.** Under B, a site blocking us might still be crawlable via the vendor's allowlisted infrastructure. In-house, a blocked site stays blocked until the owner acts. The honest-flagging posture makes this acceptable for owned sites: "we were blocked, here's the fix" instead of pretending.+7. **Third-party-URL audits have no remedy at all.** The MCP wedge includes "audit any URL on demand" (competitor audits, agent workflows) — but the allowlist flow only works for sites the user _owns_. For non-owned URLs, "we were blocked" is a terminal state, exactly where vendor infrastructure would have helped. Disposition: scope the claim — on-demand MCP audits are first-class for owned/verified sites, best-effort for everything else, and the tool response says which one it was.++**Revisit triggers (set now, so the decision is cheap to reverse).** Define "audit blocked" first: start URL blocked, or ≥20% of fetched URLs classified as challenge/`cf-mitigated`. Then: (a) >15–20% of first-run audits blocked _and_ the fix funnel (block shown → instructions viewed → re-run unblocked, instrumented in week 1) shows owners not converting; (b) sustained demand for >10k-page crawls that chaining doesn't satisfy, or for JS rendering (CSR-shell prevalence from week-1 telemetry is the denominator); (c) DataForSEO/competitor pricing shifts. The reporter contract and schema are engine-agnostic by design, so adopting OnPage later — or the `npx` local agent (D) — is an adapter, not a rewrite.++**Implementation notes for A (from §2's constraints):** politeness delays serialized _inside_ steps (wall-clock waits are fine; more steps are not), progress steps collapsed to stay under the 1,024-step budget, incremental per-batch D1 writes (P0 #0), block-rate baseline pulled from existing audit data (status 0/403 rates + PostHog) in week 1 so the revisit triggers have a denominator.++One honest limitation, stated plainly: **for sites that block all datacenter crawlers, every recommended option loses scheduled audits** — D is a manual local run. The cloud-scheduling advantage evaporates for exactly that segment; our remedy there is the GSC-verified allowlist-fix flow, which restores cloud crawling rather than working around it.++**Why not C (the desktop question, directly).** Desktop genuinely solves blocking — it's Screaming Frog's model. But scheduled audits die, server-side MCP can't wake a sleeping laptop, and it's a second product (signing, notarization, auto-update, months of work) whose own failure modes — RAM crashes, machine tie-up — are that model's loudest complaints. Draft 1's "~80% of the benefit for ~5% of the cost" for D was invented precision; the honest version: D recovers desktop's _crawl-origin_ benefit (residential IP, optional local Playwright) but not unattended operation, requires a **new** authenticated bulk-ingest endpoint feeding `audit_pages`/`audit_links` (no ingest path exists today; D depends on the P0 persistence refactor) plus a local crawler — the cheerio extraction logic is plain TS and ports, the Workflow orchestration doesn't. Estimate **~3–5 weeks**. **Panel pushback:** the panel suggested that if D balloons to 6–8 weeks the C-vs-D calculus changes. It doesn't: C strictly contains D's costs (local crawler, Playwright, sync) _plus_ installers/signing/3-platform QA, and still loses scheduling and MCP. Worst-case D remains cheaper than best-case C. Scope honesty: D serves the developer half of the ICP; for SEO freelancers (not terminal-native, and whose clients' sites are exactly the WAF'd ones) the realistic remedy is the GSC-verified allowlist flow — which is why that flow ranks above the CLI.++**The GSC idea, honestly.** GSC verification **does not bypass any WAF** — Cloudflare neither knows nor cares about Google's ownership records. What it buys: (1) **policy cover** to raise crawl _rate_ for verified owners (industry-standard move; the Ahrefs precedent is directional, see §1) — but robots.txt respect stays the default with a per-project owner toggle, not auto-ignored: crawl traps and server load are real, and "verified" doesn't repeal politeness; (2) **the trustworthy block-fix loop** — on a detected block for a GSC-verified site, show the _owner_ the exact Cloudflare/Vercel allowlist rule for our UA (`OpenSEOBot` — UA-based since Workers egress IPs aren't publishable, see decision above). Highest-leverage mitigation available; OAuth already built; now surfaced in onboarding, not post-P0; (3) later, GSC-enriched audits (true orphans = GSC impressions, zero inlinks).++---++## 5. Recommended roadmap++**Week 1:** `audit_pages`/`audit_links`/`audit_issues` schema + engine-agnostic reporter input contract; ship block detection + honest "we were blocked" messaging + telemetry — audit-level blocked definition, fix-funnel events (block shown → instructions viewed → re-run unblocked), CSR-shell prevalence counting, baseline from existing audit data (status 0/403 + PostHog; note raw 403s overcount — classify challenges, don't just count status codes); set the documented UA (`OpenSEOBot/1.0` with a /bot info page); **1–2 day allowlist validation** on Cloudflare Free (BFM), Cloudflare Pro (SBFM), Vercel Hobby/Pro (Attack Challenge) → per-platform fix copy.++**Weeks 2–4:** incremental persistence refactor + SSRF fix; crawler hardening (manual redirects, per-host politeness inside steps, sitemap-last seeding, step collapse; page caps coupled to Lighthouse strategy — `all` capped ~2k or sampled, crawl clamp stays 10k); issue engine + per-page reporters on the normalized record; multipage SQL checks (duplicates, broken links, chains); orphans (sitemap-last + completeness gating).++**Weeks 4–6:** Issues UX (severity, copy, drill-down, exclusions, CSV export); MCP audit tools with `how_to_fix` payloads; GSC-verified allowlist-fix flow in onboarding (UA-based WAF rule instructions).++**Weeks 6–8:** client-side-rendering detection + disclosure; pricing posture shipped (free 50-page audits, credits beyond) with the free-tier TTL/retention policy from §4 cost #1; buffer for the hardening tail that always exists when you own the crawler.++**Weeks 8–14 (P1):** scheduled audits + deltas with the 2-run link-graph retention policy; **AI-crawler accessibility report as the relaunch marketing asset**; thin-content/depth checks if not landed; security + external-link checks; opportunistic Verified Bots filing for `OpenSEOBot`.++**Later (P2):** `npx` local agent (new ingest endpoint; gate on block telemetry vs the §4 triggers); Cloudflare Browser Rendering JS audits for GSC-verified sites if demand shows; R2 crawl snapshots; hreflang/structured-data/page-size.++**The one-sentence version:** keep audits in the cloud on our own crawler (founder call — no vendor spend), give the small ones away to drive signups, spend engineering on a portable issue engine plus agent-actionable MCP output, answer bot-blocking with day-one honest "we were blocked" flagging and owner-targeted allowlist fixes validated per-platform in week 1, accept no JS rendering for now and Lighthouse-coupled page caps — with pre-set, measurable telemetry triggers that tell us if the no-vendor bet needs revisiting, a local CLI later, and a desktop app never.diff --git a/drizzle-pg/0007_same_marvel_zombies.sql b/drizzle-pg/0007_same_marvel_zombies.sqlnew file mode 100644index 00000000..c6ed9bef--- /dev/null+++ b/drizzle-pg/0007_same_marvel_zombies.sql@@ -0,0 +1,36 @@+CREATE TABLE "audit_issues" (+ "id" text PRIMARY KEY NOT NULL,+ "audit_id" text NOT NULL,+ "page_id" text,+ "page_url" text NOT NULL,+ "issue_type" text NOT NULL,+ "severity" text DEFAULT 'info' NOT NULL,+ "details_json" text+);+--> statement-breakpoint+CREATE TABLE "audit_links" (+ "id" text PRIMARY KEY NOT NULL,+ "audit_id" text NOT NULL,+ "source_page_id" text NOT NULL,+ "source_url" text NOT NULL,+ "target_url" text NOT NULL,+ "anchor" text,+ "is_internal" boolean DEFAULT true NOT NULL,+ "is_nofollow" boolean DEFAULT false NOT NULL+);+--> statement-breakpoint+ALTER TABLE "audit_pages" ADD COLUMN "x_robots_tag" text;--> statement-breakpoint+ALTER TABLE "audit_pages" ADD COLUMN "header_canonical_url" text;--> statement-breakpoint+ALTER TABLE "audit_pages" ADD COLUMN "crawl_depth" integer;--> statement-breakpoint+ALTER TABLE "audit_pages" ADD COLUMN "in_sitemap" boolean DEFAULT false NOT NULL;--> statement-breakpoint+ALTER TABLE "audit_pages" ADD COLUMN "content_hash" text;--> statement-breakpoint+ALTER TABLE "audit_pages" ADD COLUMN "fetch_class" text DEFAULT 'ok' NOT NULL;--> statement-breakpoint+ALTER TABLE "audit_issues" ADD CONSTRAINT "audit_issues_audit_id_audits_id_fk" FOREIGN KEY ("audit_id") REFERENCES "public"."audits"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint+ALTER TABLE "audit_issues" ADD CONSTRAINT "audit_issues_page_id_audit_pages_id_fk" FOREIGN KEY ("page_id") REFERENCES "public"."audit_pages"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint+ALTER TABLE "audit_links" ADD CONSTRAINT "audit_links_audit_id_audits_id_fk" FOREIGN KEY ("audit_id") REFERENCES "public"."audits"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint+ALTER TABLE "audit_links" ADD CONSTRAINT "audit_links_source_page_id_audit_pages_id_fk" FOREIGN KEY ("source_page_id") REFERENCES "public"."audit_pages"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint+CREATE INDEX "audit_issues_audit_id_idx" ON "audit_issues" USING btree ("audit_id");--> statement-breakpoint+CREATE INDEX "audit_issues_audit_type_idx" ON "audit_issues" USING btree ("audit_id","issue_type");--> statement-breakpoint+CREATE INDEX "audit_links_audit_id_idx" ON "audit_links" USING btree ("audit_id");--> statement-breakpoint+CREATE INDEX "audit_links_audit_target_idx" ON "audit_links" USING btree ("audit_id","target_url");--> statement-breakpoint+CREATE INDEX "audit_pages_audit_url_idx" ON "audit_pages" USING btree ("audit_id","url");\ No newline at end of filediff --git a/drizzle-pg/meta/0007_snapshot.json b/drizzle-pg/meta/0007_snapshot.jsonnew file mode 100644index 00000000..b608cf7b--- /dev/null+++ b/drizzle-pg/meta/0007_snapshot.json@@ -0,0 +1,3404 @@+{+ "id": "7317f11e-f526-4ac5-be84-0b3c6fad6ab0",+ "prevId": "d7a0c2f9-269d-44eb-bc3e-c13b4bc3cb47",+ "version": "7",+ "dialect": "postgresql",+ "tables": {+ "public.keyword_metrics": {+ "name": "keyword_metrics",+ "schema": "",+ "columns": {+ "id": {+ "name": "id",+ "type": "serial",+ "primaryKey": true,+ "notNull": true+ },+ "project_id": {+ "name": "project_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "keyword": {+ "name": "keyword",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "location_code": {+ "name": "location_code",+ "type": "integer",+ "primaryKey": false,+ "notNull": true+ },+ "language_code": {+ "name": "language_code",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "default": "'en'"+ },+ "search_volume": {+ "name": "search_volume",+ "type": "integer",+ "primaryKey": false,+ "notNull": false+ },+ "cpc": {+ "name": "cpc",+ "type": "real",+ "primaryKey": false,+ "notNull": false+ },+ "competition": {+ "name": "competition",+ "type": "real",+ "primaryKey": false,+ "notNull": false+ },+ "keyword_difficulty": {+ "name": "keyword_difficulty",+ "type": "integer",+ "primaryKey": false,+ "notNull": false+ },+ "intent": {+ "name": "intent",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "monthly_searches": {+ "name": "monthly_searches",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "fetched_at": {+ "name": "fetched_at",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "default": "to_char(now() AT TIME ZONE 'utc', 'YYYY-MM-DD\"T\"HH24:MI:SS.MS\"Z\"')"+ }+ },+ "indexes": {+ "keyword_metrics_unique_project_keyword_location_language": {+ "name": "keyword_metrics_unique_project_keyword_location_language",+ "columns": [+ {+ "expression": "project_id",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ },+ {+ "expression": "keyword",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ },+ {+ "expression": "location_code",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ },+ {+ "expression": "language_code",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ }+ ],+ "isUnique": true,+ "concurrently": false,+ "method": "btree",+ "with": {}+ },+ "keyword_metrics_lookup_idx": {+ "name": "keyword_metrics_lookup_idx",+ "columns": [+ {+ "expression": "project_id",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ },+ {+ "expression": "keyword",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ },+ {+ "expression": "location_code",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ },+ {+ "expression": "language_code",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ },+ {+ "expression": "fetched_at",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ }+ ],+ "isUnique": false,+ "concurrently": false,+ "method": "btree",+ "with": {}+ }+ },+ "foreignKeys": {+ "keyword_metrics_project_id_projects_id_fk": {+ "name": "keyword_metrics_project_id_projects_id_fk",+ "tableFrom": "keyword_metrics",+ "tableTo": "projects",+ "columnsFrom": [+ "project_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ }+ },+ "compositePrimaryKeys": {},+ "uniqueConstraints": {},+ "policies": {},+ "checkConstraints": {},+ "isRLSEnabled": false+ },+ "public.projects": {+ "name": "projects",+ "schema": "",+ "columns": {+ "id": {+ "name": "id",+ "type": "text",+ "primaryKey": true,+ "notNull": true+ },+ "organization_id": {+ "name": "organization_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "name": {+ "name": "name",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "domain": {+ "name": "domain",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "location_code": {+ "name": "location_code",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "default": 2840+ },+ "language_code": {+ "name": "language_code",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "default": "'en'"+ },+ "created_at": {+ "name": "created_at",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "default": "to_char(now() AT TIME ZONE 'utc', 'YYYY-MM-DD\"T\"HH24:MI:SS.MS\"Z\"')"+ },+ "archived_at": {+ "name": "archived_at",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ }+ },+ "indexes": {+ "projects_one_default_per_organization_idx": {+ "name": "projects_one_default_per_organization_idx",+ "columns": [+ {+ "expression": "organization_id",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ }+ ],+ "isUnique": true,+ "where": "\"projects\".\"name\" = 'Default' AND \"projects\".\"domain\" IS NULL AND \"projects\".\"archived_at\" IS NULL",+ "concurrently": false,+ "method": "btree",+ "with": {}+ },+ "projects_organization_id_idx": {+ "name": "projects_organization_id_idx",+ "columns": [+ {+ "expression": "organization_id",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ }+ ],+ "isUnique": false,+ "concurrently": false,+ "method": "btree",+ "with": {}+ }+ },+ "foreignKeys": {+ "projects_organization_id_organization_id_fk": {+ "name": "projects_organization_id_organization_id_fk",+ "tableFrom": "projects",+ "tableTo": "organization",+ "columnsFrom": [+ "organization_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ }+ },+ "compositePrimaryKeys": {},+ "uniqueConstraints": {},+ "policies": {},+ "checkConstraints": {},+ "isRLSEnabled": false+ },+ "public.rank_check_runs": {+ "name": "rank_check_runs",+ "schema": "",+ "columns": {+ "id": {+ "name": "id",+ "type": "text",+ "primaryKey": true,+ "notNull": true+ },+ "config_id": {+ "name": "config_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "project_id": {+ "name": "project_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "status": {+ "name": "status",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "default": "'pending'"+ },+ "keywords_total": {+ "name": "keywords_total",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "default": 0+ },+ "keywords_checked": {+ "name": "keywords_checked",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "default": 0+ },+ "is_subset_run": {+ "name": "is_subset_run",+ "type": "boolean",+ "primaryKey": false,+ "notNull": true,+ "default": false+ },+ "error_message": {+ "name": "error_message",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "started_at": {+ "name": "started_at",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "default": "to_char(now() AT TIME ZONE 'utc', 'YYYY-MM-DD\"T\"HH24:MI:SS.MS\"Z\"')"+ },+ "completed_at": {+ "name": "completed_at",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ }+ },+ "indexes": {+ "rank_check_runs_config_idx": {+ "name": "rank_check_runs_config_idx",+ "columns": [+ {+ "expression": "config_id",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ },+ {+ "expression": "started_at",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ }+ ],+ "isUnique": false,+ "concurrently": false,+ "method": "btree",+ "with": {}+ },+ "rank_check_runs_project_idx": {+ "name": "rank_check_runs_project_idx",+ "columns": [+ {+ "expression": "project_id",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ },+ {+ "expression": "started_at",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ }+ ],+ "isUnique": false,+ "concurrently": false,+ "method": "btree",+ "with": {}+ },+ "rank_check_runs_one_active_per_config_idx": {+ "name": "rank_check_runs_one_active_per_config_idx",+ "columns": [+ {+ "expression": "config_id",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ }+ ],+ "isUnique": true,+ "where": "\"rank_check_runs\".\"status\" IN ('pending', 'running')",+ "concurrently": false,+ "method": "btree",+ "with": {}+ }+ },+ "foreignKeys": {+ "rank_check_runs_config_id_rank_tracking_configs_id_fk": {+ "name": "rank_check_runs_config_id_rank_tracking_configs_id_fk",+ "tableFrom": "rank_check_runs",+ "tableTo": "rank_tracking_configs",+ "columnsFrom": [+ "config_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ },+ "rank_check_runs_project_id_projects_id_fk": {+ "name": "rank_check_runs_project_id_projects_id_fk",+ "tableFrom": "rank_check_runs",+ "tableTo": "projects",+ "columnsFrom": [+ "project_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ }+ },+ "compositePrimaryKeys": {},+ "uniqueConstraints": {},+ "policies": {},+ "checkConstraints": {},+ "isRLSEnabled": false+ },+ "public.rank_snapshots": {+ "name": "rank_snapshots",+ "schema": "",+ "columns": {+ "id": {+ "name": "id",+ "type": "serial",+ "primaryKey": true,+ "notNull": true+ },+ "run_id": {+ "name": "run_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "tracking_keyword_id": {+ "name": "tracking_keyword_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "keyword": {+ "name": "keyword",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "device": {+ "name": "device",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "position": {+ "name": "position",+ "type": "integer",+ "primaryKey": false,+ "notNull": false+ },+ "url": {+ "name": "url",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "serp_features": {+ "name": "serp_features",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "checked_at": {+ "name": "checked_at",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "default": "to_char(now() AT TIME ZONE 'utc', 'YYYY-MM-DD\"T\"HH24:MI:SS.MS\"Z\"')"+ }+ },+ "indexes": {+ "rank_snapshots_keyword_device_idx": {+ "name": "rank_snapshots_keyword_device_idx",+ "columns": [+ {+ "expression": "tracking_keyword_id",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ },+ {+ "expression": "device",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ },+ {+ "expression": "checked_at",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ }+ ],+ "isUnique": false,+ "concurrently": false,+ "method": "btree",+ "with": {}+ },+ "rank_snapshots_run_keyword_device_idx": {+ "name": "rank_snapshots_run_keyword_device_idx",+ "columns": [+ {+ "expression": "run_id",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ },+ {+ "expression": "tracking_keyword_id",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ },+ {+ "expression": "device",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ }+ ],+ "isUnique": true,+ "concurrently": false,+ "method": "btree",+ "with": {}+ }+ },+ "foreignKeys": {+ "rank_snapshots_run_id_rank_check_runs_id_fk": {+ "name": "rank_snapshots_run_id_rank_check_runs_id_fk",+ "tableFrom": "rank_snapshots",+ "tableTo": "rank_check_runs",+ "columnsFrom": [+ "run_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ }+ },+ "compositePrimaryKeys": {},+ "uniqueConstraints": {},+ "policies": {},+ "checkConstraints": {},+ "isRLSEnabled": false+ },+ "public.rank_tracking_configs": {+ "name": "rank_tracking_configs",+ "schema": "",+ "columns": {+ "id": {+ "name": "id",+ "type": "text",+ "primaryKey": true,+ "notNull": true+ },+ "project_id": {+ "name": "project_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "domain": {+ "name": "domain",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "location_code": {+ "name": "location_code",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "default": 2840+ },+ "language_code": {+ "name": "language_code",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "default": "'en'"+ },+ "devices": {+ "name": "devices",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "default": "'both'"+ },+ "serp_depth": {+ "name": "serp_depth",+ "type": "integer",+ "primaryKey": false,+ "notNull": true+ },+ "schedule_interval": {+ "name": "schedule_interval",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "default": "'weekly'"+ },+ "location_name": {+ "name": "location_name",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "is_active": {+ "name": "is_active",+ "type": "boolean",+ "primaryKey": false,+ "notNull": true,+ "default": true+ },+ "last_checked_at": {+ "name": "last_checked_at",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "next_check_at": {+ "name": "next_check_at",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "last_skip_reason": {+ "name": "last_skip_reason",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "created_at": {+ "name": "created_at",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "default": "to_char(now() AT TIME ZONE 'utc', 'YYYY-MM-DD\"T\"HH24:MI:SS.MS\"Z\"')"+ }+ },+ "indexes": {+ "rank_tracking_configs_national_idx": {+ "name": "rank_tracking_configs_national_idx",+ "columns": [+ {+ "expression": "project_id",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ },+ {+ "expression": "domain",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ },+ {+ "expression": "location_code",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ }+ ],+ "isUnique": true,+ "where": "\"rank_tracking_configs\".\"location_name\" IS NULL",+ "concurrently": false,+ "method": "btree",+ "with": {}+ },+ "rank_tracking_configs_local_idx": {+ "name": "rank_tracking_configs_local_idx",+ "columns": [+ {+ "expression": "project_id",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ },+ {+ "expression": "domain",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ },+ {+ "expression": "location_code",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ },+ {+ "expression": "location_name",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ }+ ],+ "isUnique": true,+ "where": "\"rank_tracking_configs\".\"location_name\" IS NOT NULL",+ "concurrently": false,+ "method": "btree",+ "with": {}+ }+ },+ "foreignKeys": {+ "rank_tracking_configs_project_id_projects_id_fk": {+ "name": "rank_tracking_configs_project_id_projects_id_fk",+ "tableFrom": "rank_tracking_configs",+ "tableTo": "projects",+ "columnsFrom": [+ "project_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ }+ },+ "compositePrimaryKeys": {},+ "uniqueConstraints": {},+ "policies": {},+ "checkConstraints": {},+ "isRLSEnabled": false+ },+ "public.rank_tracking_keywords": {+ "name": "rank_tracking_keywords",+ "schema": "",+ "columns": {+ "id": {+ "name": "id",+ "type": "text",+ "primaryKey": true,+ "notNull": true+ },+ "config_id": {+ "name": "config_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "keyword": {+ "name": "keyword",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "search_volume": {+ "name": "search_volume",+ "type": "integer",+ "primaryKey": false,+ "notNull": false+ },+ "keyword_difficulty": {+ "name": "keyword_difficulty",+ "type": "integer",+ "primaryKey": false,+ "notNull": false+ },+ "cpc": {+ "name": "cpc",+ "type": "real",+ "primaryKey": false,+ "notNull": false+ },+ "metrics_fetched_at": {+ "name": "metrics_fetched_at",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "created_at": {+ "name": "created_at",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "default": "to_char(now() AT TIME ZONE 'utc', 'YYYY-MM-DD\"T\"HH24:MI:SS.MS\"Z\"')"+ }+ },+ "indexes": {+ "rank_tracking_keywords_config_keyword_idx": {+ "name": "rank_tracking_keywords_config_keyword_idx",+ "columns": [+ {+ "expression": "config_id",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ },+ {+ "expression": "keyword",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ }+ ],+ "isUnique": true,+ "concurrently": false,+ "method": "btree",+ "with": {}+ }+ },+ "foreignKeys": {+ "rank_tracking_keywords_config_id_rank_tracking_configs_id_fk": {+ "name": "rank_tracking_keywords_config_id_rank_tracking_configs_id_fk",+ "tableFrom": "rank_tracking_keywords",+ "tableTo": "rank_tracking_configs",+ "columnsFrom": [+ "config_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ }+ },+ "compositePrimaryKeys": {},+ "uniqueConstraints": {},+ "policies": {},+ "checkConstraints": {},+ "isRLSEnabled": false+ },+ "public.saved_keyword_tag_assignments": {+ "name": "saved_keyword_tag_assignments",+ "schema": "",+ "columns": {+ "saved_keyword_id": {+ "name": "saved_keyword_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "tag_id": {+ "name": "tag_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "created_at": {+ "name": "created_at",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "default": "to_char(now() AT TIME ZONE 'utc', 'YYYY-MM-DD\"T\"HH24:MI:SS.MS\"Z\"')"+ }+ },+ "indexes": {+ "saved_keyword_tag_assignments_unique_idx": {+ "name": "saved_keyword_tag_assignments_unique_idx",+ "columns": [+ {+ "expression": "saved_keyword_id",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ },+ {+ "expression": "tag_id",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ }+ ],+ "isUnique": true,+ "concurrently": false,+ "method": "btree",+ "with": {}+ },+ "saved_keyword_tag_assignments_tag_idx": {+ "name": "saved_keyword_tag_assignments_tag_idx",+ "columns": [+ {+ "expression": "tag_id",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ }+ ],+ "isUnique": false,+ "concurrently": false,+ "method": "btree",+ "with": {}+ }+ },+ "foreignKeys": {+ "saved_keyword_tag_assignments_saved_keyword_id_saved_keywords_id_fk": {+ "name": "saved_keyword_tag_assignments_saved_keyword_id_saved_keywords_id_fk",+ "tableFrom": "saved_keyword_tag_assignments",+ "tableTo": "saved_keywords",+ "columnsFrom": [+ "saved_keyword_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ },+ "saved_keyword_tag_assignments_tag_id_saved_keyword_tags_id_fk": {+ "name": "saved_keyword_tag_assignments_tag_id_saved_keyword_tags_id_fk",+ "tableFrom": "saved_keyword_tag_assignments",+ "tableTo": "saved_keyword_tags",+ "columnsFrom": [+ "tag_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ }+ },+ "compositePrimaryKeys": {},+ "uniqueConstraints": {},+ "policies": {},+ "checkConstraints": {},+ "isRLSEnabled": false+ },+ "public.saved_keyword_tags": {+ "name": "saved_keyword_tags",+ "schema": "",+ "columns": {+ "id": {+ "name": "id",+ "type": "text",+ "primaryKey": true,+ "notNull": true+ },+ "project_id": {+ "name": "project_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "name": {+ "name": "name",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "normalized_name": {+ "name": "normalized_name",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "color": {+ "name": "color",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "created_at": {+ "name": "created_at",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "default": "to_char(now() AT TIME ZONE 'utc', 'YYYY-MM-DD\"T\"HH24:MI:SS.MS\"Z\"')"+ }+ },+ "indexes": {+ "saved_keyword_tags_project_normalized_name_idx": {+ "name": "saved_keyword_tags_project_normalized_name_idx",+ "columns": [+ {+ "expression": "project_id",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ },+ {+ "expression": "normalized_name",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ }+ ],+ "isUnique": true,+ "concurrently": false,+ "method": "btree",+ "with": {}+ },+ "saved_keyword_tags_project_name_idx": {+ "name": "saved_keyword_tags_project_name_idx",+ "columns": [+ {+ "expression": "project_id",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ },+ {+ "expression": "name",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ }+ ],+ "isUnique": false,+ "concurrently": false,+ "method": "btree",+ "with": {}+ }+ },+ "foreignKeys": {+ "saved_keyword_tags_project_id_projects_id_fk": {+ "name": "saved_keyword_tags_project_id_projects_id_fk",+ "tableFrom": "saved_keyword_tags",+ "tableTo": "projects",+ "columnsFrom": [+ "project_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ }+ },+ "compositePrimaryKeys": {},+ "uniqueConstraints": {},+ "policies": {},+ "checkConstraints": {},+ "isRLSEnabled": false+ },+ "public.saved_keywords": {+ "name": "saved_keywords",+ "schema": "",+ "columns": {+ "id": {+ "name": "id",+ "type": "text",+ "primaryKey": true,+ "notNull": true+ },+ "project_id": {+ "name": "project_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "keyword": {+ "name": "keyword",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "location_code": {+ "name": "location_code",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "default": 2840+ },+ "language_code": {+ "name": "language_code",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "default": "'en'"+ },+ "created_at": {+ "name": "created_at",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "default": "to_char(now() AT TIME ZONE 'utc', 'YYYY-MM-DD\"T\"HH24:MI:SS.MS\"Z\"')"+ }+ },+ "indexes": {+ "saved_keywords_unique_project_keyword_location_language": {+ "name": "saved_keywords_unique_project_keyword_location_language",+ "columns": [+ {+ "expression": "project_id",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ },+ {+ "expression": "keyword",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ },+ {+ "expression": "location_code",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ },+ {+ "expression": "language_code",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ }+ ],+ "isUnique": true,+ "concurrently": false,+ "method": "btree",+ "with": {}+ },+ "saved_keywords_project_created_idx": {+ "name": "saved_keywords_project_created_idx",+ "columns": [+ {+ "expression": "project_id",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ },+ {+ "expression": "created_at",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ }+ ],+ "isUnique": false,+ "concurrently": false,+ "method": "btree",+ "with": {}+ }+ },+ "foreignKeys": {+ "saved_keywords_project_id_projects_id_fk": {+ "name": "saved_keywords_project_id_projects_id_fk",+ "tableFrom": "saved_keywords",+ "tableTo": "projects",+ "columnsFrom": [+ "project_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ }+ },+ "compositePrimaryKeys": {},+ "uniqueConstraints": {},+ "policies": {},+ "checkConstraints": {},+ "isRLSEnabled": false+ },+ "public.user_onboarding_answers": {+ "name": "user_onboarding_answers",+ "schema": "",+ "columns": {+ "user_id": {+ "name": "user_id",+ "type": "text",+ "primaryKey": true,+ "notNull": true+ },+ "organization_id": {+ "name": "organization_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "interested_features": {+ "name": "interested_features",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "default": "'[]'"+ },+ "work_for": {+ "name": "work_for",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "client_website_count": {+ "name": "client_website_count",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "found_via": {+ "name": "found_via",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "mcp_setup_intent": {+ "name": "mcp_setup_intent",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "completed_at": {+ "name": "completed_at",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "gsc_nudge_dismissed_at": {+ "name": "gsc_nudge_dismissed_at",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "created_at": {+ "name": "created_at",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "default": "to_char(now() AT TIME ZONE 'utc', 'YYYY-MM-DD\"T\"HH24:MI:SS.MS\"Z\"')"+ },+ "updated_at": {+ "name": "updated_at",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "default": "to_char(now() AT TIME ZONE 'utc', 'YYYY-MM-DD\"T\"HH24:MI:SS.MS\"Z\"')"+ }+ },+ "indexes": {+ "user_onboarding_answers_organization_idx": {+ "name": "user_onboarding_answers_organization_idx",+ "columns": [+ {+ "expression": "organization_id",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ }+ ],+ "isUnique": false,+ "concurrently": false,+ "method": "btree",+ "with": {}+ }+ },+ "foreignKeys": {+ "user_onboarding_answers_user_id_user_id_fk": {+ "name": "user_onboarding_answers_user_id_user_id_fk",+ "tableFrom": "user_onboarding_answers",+ "tableTo": "user",+ "columnsFrom": [+ "user_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ },+ "user_onboarding_answers_organization_id_organization_id_fk": {+ "name": "user_onboarding_answers_organization_id_organization_id_fk",+ "tableFrom": "user_onboarding_answers",+ "tableTo": "organization",+ "columnsFrom": [+ "organization_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ }+ },+ "compositePrimaryKeys": {},+ "uniqueConstraints": {},+ "policies": {},+ "checkConstraints": {},+ "isRLSEnabled": false+ },+ "public.audit_issues": {+ "name": "audit_issues",+ "schema": "",+ "columns": {+ "id": {+ "name": "id",+ "type": "text",+ "primaryKey": true,+ "notNull": true+ },+ "audit_id": {+ "name": "audit_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "page_id": {+ "name": "page_id",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "page_url": {+ "name": "page_url",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "issue_type": {+ "name": "issue_type",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "severity": {+ "name": "severity",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "default": "'info'"+ },+ "details_json": {+ "name": "details_json",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ }+ },+ "indexes": {+ "audit_issues_audit_id_idx": {+ "name": "audit_issues_audit_id_idx",+ "columns": [+ {+ "expression": "audit_id",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ }+ ],+ "isUnique": false,+ "concurrently": false,+ "method": "btree",+ "with": {}+ },+ "audit_issues_audit_type_idx": {+ "name": "audit_issues_audit_type_idx",+ "columns": [+ {+ "expression": "audit_id",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ },+ {+ "expression": "issue_type",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ }+ ],+ "isUnique": false,+ "concurrently": false,+ "method": "btree",+ "with": {}+ }+ },+ "foreignKeys": {+ "audit_issues_audit_id_audits_id_fk": {+ "name": "audit_issues_audit_id_audits_id_fk",+ "tableFrom": "audit_issues",+ "tableTo": "audits",+ "columnsFrom": [+ "audit_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ },+ "audit_issues_page_id_audit_pages_id_fk": {+ "name": "audit_issues_page_id_audit_pages_id_fk",+ "tableFrom": "audit_issues",+ "tableTo": "audit_pages",+ "columnsFrom": [+ "page_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ }+ },+ "compositePrimaryKeys": {},+ "uniqueConstraints": {},+ "policies": {},+ "checkConstraints": {},+ "isRLSEnabled": false+ },+ "public.audit_lighthouse_results": {+ "name": "audit_lighthouse_results",+ "schema": "",+ "columns": {+ "id": {+ "name": "id",+ "type": "text",+ "primaryKey": true,+ "notNull": true+ },+ "audit_id": {+ "name": "audit_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "page_id": {+ "name": "page_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "strategy": {+ "name": "strategy",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "performance_score": {+ "name": "performance_score",+ "type": "integer",+ "primaryKey": false,+ "notNull": false+ },+ "accessibility_score": {+ "name": "accessibility_score",+ "type": "integer",+ "primaryKey": false,+ "notNull": false+ },+ "best_practices_score": {+ "name": "best_practices_score",+ "type": "integer",+ "primaryKey": false,+ "notNull": false+ },+ "seo_score": {+ "name": "seo_score",+ "type": "integer",+ "primaryKey": false,+ "notNull": false+ },+ "lcp_ms": {+ "name": "lcp_ms",+ "type": "real",+ "primaryKey": false,+ "notNull": false+ },+ "cls": {+ "name": "cls",+ "type": "real",+ "primaryKey": false,+ "notNull": false+ },+ "inp_ms": {+ "name": "inp_ms",+ "type": "real",+ "primaryKey": false,+ "notNull": false+ },+ "ttfb_ms": {+ "name": "ttfb_ms",+ "type": "real",+ "primaryKey": false,+ "notNull": false+ },+ "error_message": {+ "name": "error_message",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "r2_key": {+ "name": "r2_key",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "payload_size_bytes": {+ "name": "payload_size_bytes",+ "type": "integer",+ "primaryKey": false,+ "notNull": false+ }+ },+ "indexes": {+ "audit_lighthouse_results_audit_id_idx": {+ "name": "audit_lighthouse_results_audit_id_idx",+ "columns": [+ {+ "expression": "audit_id",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ }+ ],+ "isUnique": false,+ "concurrently": false,+ "method": "btree",+ "with": {}+ }+ },+ "foreignKeys": {+ "audit_lighthouse_results_audit_id_audits_id_fk": {+ "name": "audit_lighthouse_results_audit_id_audits_id_fk",+ "tableFrom": "audit_lighthouse_results",+ "tableTo": "audits",+ "columnsFrom": [+ "audit_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ },+ "audit_lighthouse_results_page_id_audit_pages_id_fk": {+ "name": "audit_lighthouse_results_page_id_audit_pages_id_fk",+ "tableFrom": "audit_lighthouse_results",+ "tableTo": "audit_pages",+ "columnsFrom": [+ "page_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ }+ },+ "compositePrimaryKeys": {},+ "uniqueConstraints": {},+ "policies": {},+ "checkConstraints": {},+ "isRLSEnabled": false+ },+ "public.audit_links": {+ "name": "audit_links",+ "schema": "",+ "columns": {+ "id": {+ "name": "id",+ "type": "text",+ "primaryKey": true,+ "notNull": true+ },+ "audit_id": {+ "name": "audit_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "source_page_id": {+ "name": "source_page_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "source_url": {+ "name": "source_url",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "target_url": {+ "name": "target_url",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "anchor": {+ "name": "anchor",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "is_internal": {+ "name": "is_internal",+ "type": "boolean",+ "primaryKey": false,+ "notNull": true,+ "default": true+ },+ "is_nofollow": {+ "name": "is_nofollow",+ "type": "boolean",+ "primaryKey": false,+ "notNull": true,+ "default": false+ }+ },+ "indexes": {+ "audit_links_audit_id_idx": {+ "name": "audit_links_audit_id_idx",+ "columns": [+ {+ "expression": "audit_id",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ }+ ],+ "isUnique": false,+ "concurrently": false,+ "method": "btree",+ "with": {}+ },+ "audit_links_audit_target_idx": {+ "name": "audit_links_audit_target_idx",+ "columns": [+ {+ "expression": "audit_id",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ },+ {+ "expression": "target_url",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ }+ ],+ "isUnique": false,+ "concurrently": false,+ "method": "btree",+ "with": {}+ }+ },+ "foreignKeys": {+ "audit_links_audit_id_audits_id_fk": {+ "name": "audit_links_audit_id_audits_id_fk",+ "tableFrom": "audit_links",+ "tableTo": "audits",+ "columnsFrom": [+ "audit_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ },+ "audit_links_source_page_id_audit_pages_id_fk": {+ "name": "audit_links_source_page_id_audit_pages_id_fk",+ "tableFrom": "audit_links",+ "tableTo": "audit_pages",+ "columnsFrom": [+ "source_page_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ }+ },+ "compositePrimaryKeys": {},+ "uniqueConstraints": {},+ "policies": {},+ "checkConstraints": {},+ "isRLSEnabled": false+ },+ "public.audit_pages": {+ "name": "audit_pages",+ "schema": "",+ "columns": {+ "id": {+ "name": "id",+ "type": "text",+ "primaryKey": true,+ "notNull": true+ },+ "audit_id": {+ "name": "audit_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "url": {+ "name": "url",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "status_code": {+ "name": "status_code",+ "type": "integer",+ "primaryKey": false,+ "notNull": false+ },+ "redirect_url": {+ "name": "redirect_url",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "title": {+ "name": "title",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "meta_description": {+ "name": "meta_description",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "canonical_url": {+ "name": "canonical_url",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "robots_meta": {+ "name": "robots_meta",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "og_title": {+ "name": "og_title",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "og_description": {+ "name": "og_description",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "og_image": {+ "name": "og_image",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "h1_count": {+ "name": "h1_count",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "default": 0+ },+ "h2_count": {+ "name": "h2_count",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "default": 0+ },+ "h3_count": {+ "name": "h3_count",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "default": 0+ },+ "h4_count": {+ "name": "h4_count",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "default": 0+ },+ "h5_count": {+ "name": "h5_count",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "default": 0+ },+ "h6_count": {+ "name": "h6_count",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "default": 0+ },+ "heading_order_json": {+ "name": "heading_order_json",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "word_count": {+ "name": "word_count",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "default": 0+ },+ "images_total": {+ "name": "images_total",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "default": 0+ },+ "images_missing_alt": {+ "name": "images_missing_alt",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "default": 0+ },+ "images_json": {+ "name": "images_json",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "internal_link_count": {+ "name": "internal_link_count",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "default": 0+ },+ "external_link_count": {+ "name": "external_link_count",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "default": 0+ },+ "has_structured_data": {+ "name": "has_structured_data",+ "type": "boolean",+ "primaryKey": false,+ "notNull": true,+ "default": false+ },+ "hreflang_tags_json": {+ "name": "hreflang_tags_json",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "is_indexable": {+ "name": "is_indexable",+ "type": "boolean",+ "primaryKey": false,+ "notNull": true,+ "default": true+ },+ "x_robots_tag": {+ "name": "x_robots_tag",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "header_canonical_url": {+ "name": "header_canonical_url",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "crawl_depth": {+ "name": "crawl_depth",+ "type": "integer",+ "primaryKey": false,+ "notNull": false+ },+ "in_sitemap": {+ "name": "in_sitemap",+ "type": "boolean",+ "primaryKey": false,+ "notNull": true,+ "default": false+ },+ "content_hash": {+ "name": "content_hash",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "fetch_class": {+ "name": "fetch_class",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "default": "'ok'"+ },+ "response_time_ms": {+ "name": "response_time_ms",+ "type": "integer",+ "primaryKey": false,+ "notNull": false+ }+ },+ "indexes": {+ "audit_pages_audit_id_idx": {+ "name": "audit_pages_audit_id_idx",+ "columns": [+ {+ "expression": "audit_id",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ }+ ],+ "isUnique": false,+ "concurrently": false,+ "method": "btree",+ "with": {}+ },+ "audit_pages_audit_url_idx": {+ "name": "audit_pages_audit_url_idx",+ "columns": [+ {+ "expression": "audit_id",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ },+ {+ "expression": "url",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ }+ ],+ "isUnique": false,+ "concurrently": false,+ "method": "btree",+ "with": {}+ }+ },+ "foreignKeys": {+ "audit_pages_audit_id_audits_id_fk": {+ "name": "audit_pages_audit_id_audits_id_fk",+ "tableFrom": "audit_pages",+ "tableTo": "audits",+ "columnsFrom": [+ "audit_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ }+ },+ "compositePrimaryKeys": {},+ "uniqueConstraints": {},+ "policies": {},+ "checkConstraints": {},+ "isRLSEnabled": false+ },+ "public.audits": {+ "name": "audits",+ "schema": "",+ "columns": {+ "id": {+ "name": "id",+ "type": "text",+ "primaryKey": true,+ "notNull": true+ },+ "project_id": {+ "name": "project_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "started_by_user_id": {+ "name": "started_by_user_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "start_url": {+ "name": "start_url",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "status": {+ "name": "status",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "default": "'running'"+ },+ "workflow_instance_id": {+ "name": "workflow_instance_id",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "config": {+ "name": "config",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "default": "'{}'"+ },+ "pages_crawled": {+ "name": "pages_crawled",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "default": 0+ },+ "pages_total": {+ "name": "pages_total",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "default": 0+ },+ "lighthouse_total": {+ "name": "lighthouse_total",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "default": 0+ },+ "lighthouse_completed": {+ "name": "lighthouse_completed",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "default": 0+ },+ "lighthouse_failed": {+ "name": "lighthouse_failed",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "default": 0+ },+ "current_phase": {+ "name": "current_phase",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "default": "'discovery'"+ },+ "started_at": {+ "name": "started_at",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "default": "to_char(now() AT TIME ZONE 'utc', 'YYYY-MM-DD\"T\"HH24:MI:SS.MS\"Z\"')"+ },+ "completed_at": {+ "name": "completed_at",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ }+ },+ "indexes": {+ "audits_project_id_idx": {+ "name": "audits_project_id_idx",+ "columns": [+ {+ "expression": "project_id",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ }+ ],+ "isUnique": false,+ "concurrently": false,+ "method": "btree",+ "with": {}+ },+ "audits_started_by_user_id_idx": {+ "name": "audits_started_by_user_id_idx",+ "columns": [+ {+ "expression": "started_by_user_id",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ }+ ],+ "isUnique": false,+ "concurrently": false,+ "method": "btree",+ "with": {}+ }+ },+ "foreignKeys": {+ "audits_project_id_projects_id_fk": {+ "name": "audits_project_id_projects_id_fk",+ "tableFrom": "audits",+ "tableTo": "projects",+ "columnsFrom": [+ "project_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ }+ },+ "compositePrimaryKeys": {},+ "uniqueConstraints": {},+ "policies": {},+ "checkConstraints": {},+ "isRLSEnabled": false+ },+ "public.sam_project_memory": {+ "name": "sam_project_memory",+ "schema": "",+ "columns": {+ "project_id": {+ "name": "project_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "label": {+ "name": "label",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "content": {+ "name": "content",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "updated_at": {+ "name": "updated_at",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "default": "to_char(now() AT TIME ZONE 'utc', 'YYYY-MM-DD\"T\"HH24:MI:SS.MS\"Z\"')"+ }+ },+ "indexes": {},+ "foreignKeys": {+ "sam_project_memory_project_id_projects_id_fk": {+ "name": "sam_project_memory_project_id_projects_id_fk",+ "tableFrom": "sam_project_memory",+ "tableTo": "projects",+ "columnsFrom": [+ "project_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ }+ },+ "compositePrimaryKeys": {+ "sam_project_memory_project_id_label_pk": {+ "name": "sam_project_memory_project_id_label_pk",+ "columns": [+ "project_id",+ "label"+ ]+ }+ },+ "uniqueConstraints": {},+ "policies": {},+ "checkConstraints": {},+ "isRLSEnabled": false+ },+ "public.sam_sessions": {+ "name": "sam_sessions",+ "schema": "",+ "columns": {+ "id": {+ "name": "id",+ "type": "text",+ "primaryKey": true,+ "notNull": true+ },+ "project_id": {+ "name": "project_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "user_id": {+ "name": "user_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "title": {+ "name": "title",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "default": "'New chat'"+ },+ "created_at": {+ "name": "created_at",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "default": "to_char(now() AT TIME ZONE 'utc', 'YYYY-MM-DD\"T\"HH24:MI:SS.MS\"Z\"')"+ },+ "updated_at": {+ "name": "updated_at",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "default": "to_char(now() AT TIME ZONE 'utc', 'YYYY-MM-DD\"T\"HH24:MI:SS.MS\"Z\"')"+ },+ "archived_at": {+ "name": "archived_at",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ }+ },+ "indexes": {+ "sam_sessions_project_updated_idx": {+ "name": "sam_sessions_project_updated_idx",+ "columns": [+ {+ "expression": "project_id",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ },+ {+ "expression": "updated_at",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ }+ ],+ "isUnique": false,+ "concurrently": false,+ "method": "btree",+ "with": {}+ }+ },+ "foreignKeys": {+ "sam_sessions_project_id_projects_id_fk": {+ "name": "sam_sessions_project_id_projects_id_fk",+ "tableFrom": "sam_sessions",+ "tableTo": "projects",+ "columnsFrom": [+ "project_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ },+ "sam_sessions_user_id_user_id_fk": {+ "name": "sam_sessions_user_id_user_id_fk",+ "tableFrom": "sam_sessions",+ "tableTo": "user",+ "columnsFrom": [+ "user_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ }+ },+ "compositePrimaryKeys": {},+ "uniqueConstraints": {},+ "policies": {},+ "checkConstraints": {},+ "isRLSEnabled": false+ },+ "public.account": {+ "name": "account",+ "schema": "",+ "columns": {+ "id": {+ "name": "id",+ "type": "text",+ "primaryKey": true,+ "notNull": true+ },+ "account_id": {+ "name": "account_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "provider_id": {+ "name": "provider_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "user_id": {+ "name": "user_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "access_token": {+ "name": "access_token",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "refresh_token": {+ "name": "refresh_token",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "id_token": {+ "name": "id_token",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "access_token_expires_at": {+ "name": "access_token_expires_at",+ "type": "timestamp with time zone",+ "primaryKey": false,+ "notNull": false+ },+ "refresh_token_expires_at": {+ "name": "refresh_token_expires_at",+ "type": "timestamp with time zone",+ "primaryKey": false,+ "notNull": false+ },+ "scope": {+ "name": "scope",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "password": {+ "name": "password",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "created_at": {+ "name": "created_at",+ "type": "timestamp with time zone",+ "primaryKey": false,+ "notNull": true,+ "default": "now()"+ },+ "updated_at": {+ "name": "updated_at",+ "type": "timestamp with time zone",+ "primaryKey": false,+ "notNull": true+ }+ },+ "indexes": {+ "account_userId_idx": {+ "name": "account_userId_idx",+ "columns": [+ {+ "expression": "user_id",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ }+ ],+ "isUnique": false,+ "concurrently": false,+ "method": "btree",+ "with": {}+ },+ "account_accountId_providerId_idx": {+ "name": "account_accountId_providerId_idx",+ "columns": [+ {+ "expression": "account_id",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ },+ {+ "expression": "provider_id",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ }+ ],+ "isUnique": false,+ "concurrently": false,+ "method": "btree",+ "with": {}+ }+ },+ "foreignKeys": {+ "account_user_id_user_id_fk": {+ "name": "account_user_id_user_id_fk",+ "tableFrom": "account",+ "tableTo": "user",+ "columnsFrom": [+ "user_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ }+ },+ "compositePrimaryKeys": {},+ "uniqueConstraints": {},+ "policies": {},+ "checkConstraints": {},+ "isRLSEnabled": false+ },+ "public.invitation": {+ "name": "invitation",+ "schema": "",+ "columns": {+ "id": {+ "name": "id",+ "type": "text",+ "primaryKey": true,+ "notNull": true+ },+ "organization_id": {+ "name": "organization_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "email": {+ "name": "email",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "role": {+ "name": "role",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "status": {+ "name": "status",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "default": "'pending'"+ },+ "expires_at": {+ "name": "expires_at",+ "type": "timestamp with time zone",+ "primaryKey": false,+ "notNull": true+ },+ "created_at": {+ "name": "created_at",+ "type": "timestamp with time zone",+ "primaryKey": false,+ "notNull": true,+ "default": "now()"+ },+ "inviter_id": {+ "name": "inviter_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ }+ },+ "indexes": {+ "invitation_organizationId_idx": {+ "name": "invitation_organizationId_idx",+ "columns": [+ {+ "expression": "organization_id",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ }+ ],+ "isUnique": false,+ "concurrently": false,+ "method": "btree",+ "with": {}+ },+ "invitation_email_idx": {+ "name": "invitation_email_idx",+ "columns": [+ {+ "expression": "email",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ }+ ],+ "isUnique": false,+ "concurrently": false,+ "method": "btree",+ "with": {}+ }+ },+ "foreignKeys": {+ "invitation_organization_id_organization_id_fk": {+ "name": "invitation_organization_id_organization_id_fk",+ "tableFrom": "invitation",+ "tableTo": "organization",+ "columnsFrom": [+ "organization_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ },+ "invitation_inviter_id_user_id_fk": {+ "name": "invitation_inviter_id_user_id_fk",+ "tableFrom": "invitation",+ "tableTo": "user",+ "columnsFrom": [+ "inviter_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ }+ },+ "compositePrimaryKeys": {},+ "uniqueConstraints": {},+ "policies": {},+ "checkConstraints": {},+ "isRLSEnabled": false+ },+ "public.member": {+ "name": "member",+ "schema": "",+ "columns": {+ "id": {+ "name": "id",+ "type": "text",+ "primaryKey": true,+ "notNull": true+ },+ "organization_id": {+ "name": "organization_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "user_id": {+ "name": "user_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "role": {+ "name": "role",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "default": "'member'"+ },+ "created_at": {+ "name": "created_at",+ "type": "timestamp with time zone",+ "primaryKey": false,+ "notNull": true+ }+ },+ "indexes": {+ "member_organizationId_idx": {+ "name": "member_organizationId_idx",+ "columns": [+ {+ "expression": "organization_id",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ }+ ],+ "isUnique": false,+ "concurrently": false,+ "method": "btree",+ "with": {}+ },+ "member_userId_idx": {+ "name": "member_userId_idx",+ "columns": [+ {+ "expression": "user_id",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ }+ ],+ "isUnique": false,+ "concurrently": false,+ "method": "btree",+ "with": {}+ }+ },+ "foreignKeys": {+ "member_organization_id_organization_id_fk": {+ "name": "member_organization_id_organization_id_fk",+ "tableFrom": "member",+ "tableTo": "organization",+ "columnsFrom": [+ "organization_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ },+ "member_user_id_user_id_fk": {+ "name": "member_user_id_user_id_fk",+ "tableFrom": "member",+ "tableTo": "user",+ "columnsFrom": [+ "user_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ }+ },+ "compositePrimaryKeys": {},+ "uniqueConstraints": {},+ "policies": {},+ "checkConstraints": {},+ "isRLSEnabled": false+ },+ "public.organization": {+ "name": "organization",+ "schema": "",+ "columns": {+ "id": {+ "name": "id",+ "type": "text",+ "primaryKey": true,+ "notNull": true+ },+ "name": {+ "name": "name",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "slug": {+ "name": "slug",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "logo": {+ "name": "logo",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "created_at": {+ "name": "created_at",+ "type": "timestamp with time zone",+ "primaryKey": false,+ "notNull": true+ },+ "metadata": {+ "name": "metadata",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ }+ },+ "indexes": {+ "organization_slug_uidx": {+ "name": "organization_slug_uidx",+ "columns": [+ {+ "expression": "slug",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ }+ ],+ "isUnique": true,+ "concurrently": false,+ "method": "btree",+ "with": {}+ }+ },+ "foreignKeys": {},+ "compositePrimaryKeys": {},+ "uniqueConstraints": {+ "organization_slug_unique": {+ "name": "organization_slug_unique",+ "nullsNotDistinct": false,+ "columns": [+ "slug"+ ]+ }+ },+ "policies": {},+ "checkConstraints": {},+ "isRLSEnabled": false+ },+ "public.session": {+ "name": "session",+ "schema": "",+ "columns": {+ "id": {+ "name": "id",+ "type": "text",+ "primaryKey": true,+ "notNull": true+ },+ "expires_at": {+ "name": "expires_at",+ "type": "timestamp with time zone",+ "primaryKey": false,+ "notNull": true+ },+ "token": {+ "name": "token",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "created_at": {+ "name": "created_at",+ "type": "timestamp with time zone",+ "primaryKey": false,+ "notNull": true,+ "default": "now()"+ },+ "updated_at": {+ "name": "updated_at",+ "type": "timestamp with time zone",+ "primaryKey": false,+ "notNull": true+ },+ "ip_address": {+ "name": "ip_address",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "user_agent": {+ "name": "user_agent",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "user_id": {+ "name": "user_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "active_organization_id": {+ "name": "active_organization_id",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ }+ },+ "indexes": {+ "session_userId_idx": {+ "name": "session_userId_idx",+ "columns": [+ {+ "expression": "user_id",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ }+ ],+ "isUnique": false,+ "concurrently": false,+ "method": "btree",+ "with": {}+ }+ },+ "foreignKeys": {+ "session_user_id_user_id_fk": {+ "name": "session_user_id_user_id_fk",+ "tableFrom": "session",+ "tableTo": "user",+ "columnsFrom": [+ "user_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ }+ },+ "compositePrimaryKeys": {},+ "uniqueConstraints": {+ "session_token_unique": {+ "name": "session_token_unique",+ "nullsNotDistinct": false,+ "columns": [+ "token"+ ]+ }+ },+ "policies": {},+ "checkConstraints": {},+ "isRLSEnabled": false+ },+ "public.user": {+ "name": "user",+ "schema": "",+ "columns": {+ "id": {+ "name": "id",+ "type": "text",+ "primaryKey": true,+ "notNull": true+ },+ "name": {+ "name": "name",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "email": {+ "name": "email",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "email_verified": {+ "name": "email_verified",+ "type": "boolean",+ "primaryKey": false,+ "notNull": true,+ "default": false+ },+ "image": {+ "name": "image",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "created_at": {+ "name": "created_at",+ "type": "timestamp with time zone",+ "primaryKey": false,+ "notNull": true,+ "default": "now()"+ },+ "updated_at": {+ "name": "updated_at",+ "type": "timestamp with time zone",+ "primaryKey": false,+ "notNull": true,+ "default": "now()"+ },+ "analytics_opted_out": {+ "name": "analytics_opted_out",+ "type": "boolean",+ "primaryKey": false,+ "notNull": false+ }+ },+ "indexes": {},+ "foreignKeys": {},+ "compositePrimaryKeys": {},+ "uniqueConstraints": {+ "user_email_unique": {+ "name": "user_email_unique",+ "nullsNotDistinct": false,+ "columns": [+ "email"+ ]+ }+ },+ "policies": {},+ "checkConstraints": {},+ "isRLSEnabled": false+ },+ "public.verification": {+ "name": "verification",+ "schema": "",+ "columns": {+ "id": {+ "name": "id",+ "type": "text",+ "primaryKey": true,+ "notNull": true+ },+ "identifier": {+ "name": "identifier",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "value": {+ "name": "value",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "expires_at": {+ "name": "expires_at",+ "type": "timestamp with time zone",+ "primaryKey": false,+ "notNull": true+ },+ "created_at": {+ "name": "created_at",+ "type": "timestamp with time zone",+ "primaryKey": false,+ "notNull": true,+ "default": "now()"+ },+ "updated_at": {+ "name": "updated_at",+ "type": "timestamp with time zone",+ "primaryKey": false,+ "notNull": true,+ "default": "now()"+ }+ },+ "indexes": {+ "verification_identifier_idx": {+ "name": "verification_identifier_idx",+ "columns": [+ {+ "expression": "identifier",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ }+ ],+ "isUnique": false,+ "concurrently": false,+ "method": "btree",+ "with": {}+ },+ "verification_expiresAt_idx": {+ "name": "verification_expiresAt_idx",+ "columns": [+ {+ "expression": "expires_at",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ }+ ],+ "isUnique": false,+ "concurrently": false,+ "method": "btree",+ "with": {}+ }+ },+ "foreignKeys": {},+ "compositePrimaryKeys": {},+ "uniqueConstraints": {},+ "policies": {},+ "checkConstraints": {},+ "isRLSEnabled": false+ },+ "public.billing_customer_status": {+ "name": "billing_customer_status",+ "schema": "",+ "columns": {+ "organization_id": {+ "name": "organization_id",+ "type": "text",+ "primaryKey": true,+ "notNull": true+ },+ "is_paying": {+ "name": "is_paying",+ "type": "boolean",+ "primaryKey": false,+ "notNull": true,+ "default": false+ },+ "paid_plan_id": {+ "name": "paid_plan_id",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "paid_plan_status": {+ "name": "paid_plan_status",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "customer_json": {+ "name": "customer_json",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "synced_at": {+ "name": "synced_at",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "created_at": {+ "name": "created_at",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "default": "to_char(now() AT TIME ZONE 'utc', 'YYYY-MM-DD\"T\"HH24:MI:SS.MS\"Z\"')"+ },+ "updated_at": {+ "name": "updated_at",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "default": "to_char(now() AT TIME ZONE 'utc', 'YYYY-MM-DD\"T\"HH24:MI:SS.MS\"Z\"')"+ }+ },+ "indexes": {},+ "foreignKeys": {+ "billing_customer_status_organization_id_organization_id_fk": {+ "name": "billing_customer_status_organization_id_organization_id_fk",+ "tableFrom": "billing_customer_status",+ "tableTo": "organization",+ "columnsFrom": [+ "organization_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ }+ },+ "compositePrimaryKeys": {},+ "uniqueConstraints": {},+ "policies": {},+ "checkConstraints": {},+ "isRLSEnabled": false+ },+ "public.gsc_connections": {+ "name": "gsc_connections",+ "schema": "",+ "columns": {+ "id": {+ "name": "id",+ "type": "text",+ "primaryKey": true,+ "notNull": true+ },+ "project_id": {+ "name": "project_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "organization_id": {+ "name": "organization_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "site_url": {+ "name": "site_url",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "connected_by_user_id": {+ "name": "connected_by_user_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "connected_account_email": {+ "name": "connected_account_email",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "created_at": {+ "name": "created_at",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "default": "to_char(now() AT TIME ZONE 'utc', 'YYYY-MM-DD\"T\"HH24:MI:SS.MS\"Z\"')"+ },+ "updated_at": {+ "name": "updated_at",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "default": "to_char(now() AT TIME ZONE 'utc', 'YYYY-MM-DD\"T\"HH24:MI:SS.MS\"Z\"')"+ }+ },+ "indexes": {+ "gsc_connections_project_idx": {+ "name": "gsc_connections_project_idx",+ "columns": [+ {+ "expression": "project_id",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ }+ ],+ "isUnique": true,+ "concurrently": false,+ "method": "btree",+ "with": {}+ },+ "gsc_connections_organization_idx": {+ "name": "gsc_connections_organization_idx",+ "columns": [+ {+ "expression": "organization_id",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ }+ ],+ "isUnique": false,+ "concurrently": false,+ "method": "btree",+ "with": {}+ }+ },+ "foreignKeys": {+ "gsc_connections_project_id_projects_id_fk": {+ "name": "gsc_connections_project_id_projects_id_fk",+ "tableFrom": "gsc_connections",+ "tableTo": "projects",+ "columnsFrom": [+ "project_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ },+ "gsc_connections_organization_id_organization_id_fk": {+ "name": "gsc_connections_organization_id_organization_id_fk",+ "tableFrom": "gsc_connections",+ "tableTo": "organization",+ "columnsFrom": [+ "organization_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ }+ },+ "compositePrimaryKeys": {},+ "uniqueConstraints": {},+ "policies": {},+ "checkConstraints": {},+ "isRLSEnabled": false+ },+ "public.reddit_attributions": {+ "name": "reddit_attributions",+ "schema": "",+ "columns": {+ "id": {+ "name": "id",+ "type": "text",+ "primaryKey": true,+ "notNull": true+ },+ "user_id": {+ "name": "user_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "organization_id": {+ "name": "organization_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true+ },+ "click_id": {+ "name": "click_id",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "uuid": {+ "name": "uuid",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "landing_page": {+ "name": "landing_page",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "referrer": {+ "name": "referrer",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "utm_source": {+ "name": "utm_source",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "utm_medium": {+ "name": "utm_medium",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "utm_campaign": {+ "name": "utm_campaign",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "utm_term": {+ "name": "utm_term",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "utm_content": {+ "name": "utm_content",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "signup_sent_at": {+ "name": "signup_sent_at",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "purchase_sent_at": {+ "name": "purchase_sent_at",+ "type": "text",+ "primaryKey": false,+ "notNull": false+ },+ "created_at": {+ "name": "created_at",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "default": "to_char(now() AT TIME ZONE 'utc', 'YYYY-MM-DD\"T\"HH24:MI:SS.MS\"Z\"')"+ },+ "updated_at": {+ "name": "updated_at",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "default": "to_char(now() AT TIME ZONE 'utc', 'YYYY-MM-DD\"T\"HH24:MI:SS.MS\"Z\"')"+ }+ },+ "indexes": {+ "reddit_attributions_user_idx": {+ "name": "reddit_attributions_user_idx",+ "columns": [+ {+ "expression": "user_id",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ }+ ],+ "isUnique": true,+ "concurrently": false,+ "method": "btree",+ "with": {}+ },+ "reddit_attributions_organization_idx": {+ "name": "reddit_attributions_organization_idx",+ "columns": [+ {+ "expression": "organization_id",+ "isExpression": false,+ "asc": true,+ "nulls": "last"+ }+ ],+ "isUnique": false,+ "concurrently": false,+ "method": "btree",+ "with": {}+ }+ },+ "foreignKeys": {+ "reddit_attributions_user_id_user_id_fk": {+ "name": "reddit_attributions_user_id_user_id_fk",+ "tableFrom": "reddit_attributions",+ "tableTo": "user",+ "columnsFrom": [+ "user_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ },+ "reddit_attributions_organization_id_organization_id_fk": {+ "name": "reddit_attributions_organization_id_organization_id_fk",+ "tableFrom": "reddit_attributions",+ "tableTo": "organization",+ "columnsFrom": [+ "organization_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ }+ },+ "compositePrimaryKeys": {},+ "uniqueConstraints": {},+ "policies": {},+ "checkConstraints": {},+ "isRLSEnabled": false+ }+ },+ "enums": {},+ "schemas": {},+ "sequences": {},+ "roles": {},+ "policies": {},+ "views": {},+ "_meta": {+ "columns": {},+ "schemas": {},+ "tables": {}+ }+}\ No newline at end of filediff --git a/drizzle-pg/meta/_journal.json b/drizzle-pg/meta/_journal.jsonindex d50969bf..5c7196da 100644--- a/drizzle-pg/meta/_journal.json+++ b/drizzle-pg/meta/_journal.json@@ -50,6 +50,13 @@"when": 1783438455708,"tag": "0006_location_name","breakpoints": true+ },+ {+ "idx": 7,+ "version": "7",+ "when": 1783306049167,+ "tag": "0007_same_marvel_zombies",+ "breakpoints": true}]}\ No newline at end of filediff --git a/drizzle/0030_legal_reaper.sql b/drizzle/0030_legal_reaper.sqlnew file mode 100644index 00000000..3bd9775d--- /dev/null+++ b/drizzle/0030_legal_reaper.sql@@ -0,0 +1,36 @@+CREATE TABLE `audit_issues` (+ `id` text PRIMARY KEY NOT NULL,+ `audit_id` text NOT NULL,+ `page_id` text,+ `page_url` text NOT NULL,+ `issue_type` text NOT NULL,+ `severity` text DEFAULT 'info' NOT NULL,+ `details_json` text,+ FOREIGN KEY (`audit_id`) REFERENCES `audits`(`id`) ON UPDATE no action ON DELETE cascade,+ FOREIGN KEY (`page_id`) REFERENCES `audit_pages`(`id`) ON UPDATE no action ON DELETE cascade+);+--> statement-breakpoint+CREATE INDEX `audit_issues_audit_id_idx` ON `audit_issues` (`audit_id`);--> statement-breakpoint+CREATE INDEX `audit_issues_audit_type_idx` ON `audit_issues` (`audit_id`,`issue_type`);--> statement-breakpoint+CREATE TABLE `audit_links` (+ `id` text PRIMARY KEY NOT NULL,+ `audit_id` text NOT NULL,+ `source_page_id` text NOT NULL,+ `source_url` text NOT NULL,+ `target_url` text NOT NULL,+ `anchor` text,+ `is_internal` integer DEFAULT true NOT NULL,+ `is_nofollow` integer DEFAULT false NOT NULL,+ FOREIGN KEY (`audit_id`) REFERENCES `audits`(`id`) ON UPDATE no action ON DELETE cascade,+ FOREIGN KEY (`source_page_id`) REFERENCES `audit_pages`(`id`) ON UPDATE no action ON DELETE cascade+);+--> statement-breakpoint+CREATE INDEX `audit_links_audit_id_idx` ON `audit_links` (`audit_id`);--> statement-breakpoint+CREATE INDEX `audit_links_audit_target_idx` ON `audit_links` (`audit_id`,`target_url`);--> statement-breakpoint+ALTER TABLE `audit_pages` ADD `x_robots_tag` text;--> statement-breakpoint+ALTER TABLE `audit_pages` ADD `header_canonical_url` text;--> statement-breakpoint+ALTER TABLE `audit_pages` ADD `crawl_depth` integer;--> statement-breakpoint+ALTER TABLE `audit_pages` ADD `in_sitemap` integer DEFAULT false NOT NULL;--> statement-breakpoint+ALTER TABLE `audit_pages` ADD `content_hash` text;--> statement-breakpoint+ALTER TABLE `audit_pages` ADD `fetch_class` text DEFAULT 'ok' NOT NULL;--> statement-breakpoint+CREATE INDEX `audit_pages_audit_url_idx` ON `audit_pages` (`audit_id`,`url`);\ No newline at end of filediff --git a/drizzle/meta/0030_snapshot.json b/drizzle/meta/0030_snapshot.jsonnew file mode 100644index 00000000..d5f4d230--- /dev/null+++ b/drizzle/meta/0030_snapshot.json@@ -0,0 +1,3086 @@+{+ "version": "6",+ "dialect": "sqlite",+ "id": "57a79baa-091a-4d26-9324-dd8623e530a9",+ "prevId": "04f10078-feca-42c4-9158-f1ba8df6091f",+ "tables": {+ "keyword_metrics": {+ "name": "keyword_metrics",+ "columns": {+ "id": {+ "name": "id",+ "type": "integer",+ "primaryKey": true,+ "notNull": true,+ "autoincrement": true+ },+ "project_id": {+ "name": "project_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "keyword": {+ "name": "keyword",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "location_code": {+ "name": "location_code",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "language_code": {+ "name": "language_code",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": "'en'"+ },+ "search_volume": {+ "name": "search_volume",+ "type": "integer",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "cpc": {+ "name": "cpc",+ "type": "real",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "competition": {+ "name": "competition",+ "type": "real",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "keyword_difficulty": {+ "name": "keyword_difficulty",+ "type": "integer",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "intent": {+ "name": "intent",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "monthly_searches": {+ "name": "monthly_searches",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "fetched_at": {+ "name": "fetched_at",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": "(current_timestamp)"+ }+ },+ "indexes": {+ "keyword_metrics_unique_project_keyword_location_language": {+ "name": "keyword_metrics_unique_project_keyword_location_language",+ "columns": [+ "project_id",+ "keyword",+ "location_code",+ "language_code"+ ],+ "isUnique": true+ },+ "keyword_metrics_lookup_idx": {+ "name": "keyword_metrics_lookup_idx",+ "columns": [+ "project_id",+ "keyword",+ "location_code",+ "language_code",+ "fetched_at"+ ],+ "isUnique": false+ }+ },+ "foreignKeys": {+ "keyword_metrics_project_id_projects_id_fk": {+ "name": "keyword_metrics_project_id_projects_id_fk",+ "tableFrom": "keyword_metrics",+ "tableTo": "projects",+ "columnsFrom": [+ "project_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ }+ },+ "compositePrimaryKeys": {},+ "uniqueConstraints": {},+ "checkConstraints": {}+ },+ "projects": {+ "name": "projects",+ "columns": {+ "id": {+ "name": "id",+ "type": "text",+ "primaryKey": true,+ "notNull": true,+ "autoincrement": false+ },+ "organization_id": {+ "name": "organization_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "name": {+ "name": "name",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "domain": {+ "name": "domain",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "location_code": {+ "name": "location_code",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": 2840+ },+ "language_code": {+ "name": "language_code",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": "'en'"+ },+ "created_at": {+ "name": "created_at",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": "(current_timestamp)"+ },+ "archived_at": {+ "name": "archived_at",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ }+ },+ "indexes": {+ "projects_one_default_per_organization_idx": {+ "name": "projects_one_default_per_organization_idx",+ "columns": [+ "organization_id"+ ],+ "isUnique": true,+ "where": "\"projects\".\"name\" = 'Default' AND \"projects\".\"domain\" IS NULL AND \"projects\".\"archived_at\" IS NULL"+ },+ "projects_organization_id_idx": {+ "name": "projects_organization_id_idx",+ "columns": [+ "organization_id"+ ],+ "isUnique": false+ }+ },+ "foreignKeys": {+ "projects_organization_id_organization_id_fk": {+ "name": "projects_organization_id_organization_id_fk",+ "tableFrom": "projects",+ "tableTo": "organization",+ "columnsFrom": [+ "organization_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ }+ },+ "compositePrimaryKeys": {},+ "uniqueConstraints": {},+ "checkConstraints": {}+ },+ "rank_check_runs": {+ "name": "rank_check_runs",+ "columns": {+ "id": {+ "name": "id",+ "type": "text",+ "primaryKey": true,+ "notNull": true,+ "autoincrement": false+ },+ "config_id": {+ "name": "config_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "project_id": {+ "name": "project_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "status": {+ "name": "status",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": "'pending'"+ },+ "keywords_total": {+ "name": "keywords_total",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": 0+ },+ "keywords_checked": {+ "name": "keywords_checked",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": 0+ },+ "is_subset_run": {+ "name": "is_subset_run",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": false+ },+ "error_message": {+ "name": "error_message",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "started_at": {+ "name": "started_at",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": "(current_timestamp)"+ },+ "completed_at": {+ "name": "completed_at",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ }+ },+ "indexes": {+ "rank_check_runs_config_idx": {+ "name": "rank_check_runs_config_idx",+ "columns": [+ "config_id",+ "started_at"+ ],+ "isUnique": false+ },+ "rank_check_runs_project_idx": {+ "name": "rank_check_runs_project_idx",+ "columns": [+ "project_id",+ "started_at"+ ],+ "isUnique": false+ },+ "rank_check_runs_one_active_per_config_idx": {+ "name": "rank_check_runs_one_active_per_config_idx",+ "columns": [+ "config_id"+ ],+ "isUnique": true,+ "where": "\"rank_check_runs\".\"status\" IN ('pending', 'running')"+ }+ },+ "foreignKeys": {+ "rank_check_runs_config_id_rank_tracking_configs_id_fk": {+ "name": "rank_check_runs_config_id_rank_tracking_configs_id_fk",+ "tableFrom": "rank_check_runs",+ "tableTo": "rank_tracking_configs",+ "columnsFrom": [+ "config_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ },+ "rank_check_runs_project_id_projects_id_fk": {+ "name": "rank_check_runs_project_id_projects_id_fk",+ "tableFrom": "rank_check_runs",+ "tableTo": "projects",+ "columnsFrom": [+ "project_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ }+ },+ "compositePrimaryKeys": {},+ "uniqueConstraints": {},+ "checkConstraints": {}+ },+ "rank_snapshots": {+ "name": "rank_snapshots",+ "columns": {+ "id": {+ "name": "id",+ "type": "integer",+ "primaryKey": true,+ "notNull": true,+ "autoincrement": true+ },+ "run_id": {+ "name": "run_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "tracking_keyword_id": {+ "name": "tracking_keyword_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "keyword": {+ "name": "keyword",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "device": {+ "name": "device",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "position": {+ "name": "position",+ "type": "integer",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "url": {+ "name": "url",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "serp_features": {+ "name": "serp_features",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "checked_at": {+ "name": "checked_at",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": "(current_timestamp)"+ }+ },+ "indexes": {+ "rank_snapshots_keyword_device_idx": {+ "name": "rank_snapshots_keyword_device_idx",+ "columns": [+ "tracking_keyword_id",+ "device",+ "checked_at"+ ],+ "isUnique": false+ },+ "rank_snapshots_run_keyword_device_idx": {+ "name": "rank_snapshots_run_keyword_device_idx",+ "columns": [+ "run_id",+ "tracking_keyword_id",+ "device"+ ],+ "isUnique": true+ }+ },+ "foreignKeys": {+ "rank_snapshots_run_id_rank_check_runs_id_fk": {+ "name": "rank_snapshots_run_id_rank_check_runs_id_fk",+ "tableFrom": "rank_snapshots",+ "tableTo": "rank_check_runs",+ "columnsFrom": [+ "run_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ }+ },+ "compositePrimaryKeys": {},+ "uniqueConstraints": {},+ "checkConstraints": {}+ },+ "rank_tracking_configs": {+ "name": "rank_tracking_configs",+ "columns": {+ "id": {+ "name": "id",+ "type": "text",+ "primaryKey": true,+ "notNull": true,+ "autoincrement": false+ },+ "project_id": {+ "name": "project_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "domain": {+ "name": "domain",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "location_code": {+ "name": "location_code",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": 2840+ },+ "language_code": {+ "name": "language_code",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": "'en'"+ },+ "devices": {+ "name": "devices",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": "'both'"+ },+ "serp_depth": {+ "name": "serp_depth",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "schedule_interval": {+ "name": "schedule_interval",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": "'weekly'"+ },+ "location_name": {+ "name": "location_name",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "is_active": {+ "name": "is_active",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": true+ },+ "last_checked_at": {+ "name": "last_checked_at",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "next_check_at": {+ "name": "next_check_at",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "last_skip_reason": {+ "name": "last_skip_reason",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "created_at": {+ "name": "created_at",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": "(current_timestamp)"+ }+ },+ "indexes": {+ "rank_tracking_configs_national_idx": {+ "name": "rank_tracking_configs_national_idx",+ "columns": [+ "project_id",+ "domain",+ "location_code"+ ],+ "isUnique": true,+ "where": "\"rank_tracking_configs\".\"location_name\" IS NULL"+ },+ "rank_tracking_configs_local_idx": {+ "name": "rank_tracking_configs_local_idx",+ "columns": [+ "project_id",+ "domain",+ "location_code",+ "location_name"+ ],+ "isUnique": true,+ "where": "\"rank_tracking_configs\".\"location_name\" IS NOT NULL"+ }+ },+ "foreignKeys": {+ "rank_tracking_configs_project_id_projects_id_fk": {+ "name": "rank_tracking_configs_project_id_projects_id_fk",+ "tableFrom": "rank_tracking_configs",+ "tableTo": "projects",+ "columnsFrom": [+ "project_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ }+ },+ "compositePrimaryKeys": {},+ "uniqueConstraints": {},+ "checkConstraints": {}+ },+ "rank_tracking_keywords": {+ "name": "rank_tracking_keywords",+ "columns": {+ "id": {+ "name": "id",+ "type": "text",+ "primaryKey": true,+ "notNull": true,+ "autoincrement": false+ },+ "config_id": {+ "name": "config_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "keyword": {+ "name": "keyword",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "search_volume": {+ "name": "search_volume",+ "type": "integer",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "keyword_difficulty": {+ "name": "keyword_difficulty",+ "type": "integer",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "cpc": {+ "name": "cpc",+ "type": "real",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "metrics_fetched_at": {+ "name": "metrics_fetched_at",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "created_at": {+ "name": "created_at",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": "(current_timestamp)"+ }+ },+ "indexes": {+ "rank_tracking_keywords_config_keyword_idx": {+ "name": "rank_tracking_keywords_config_keyword_idx",+ "columns": [+ "config_id",+ "keyword"+ ],+ "isUnique": true+ }+ },+ "foreignKeys": {+ "rank_tracking_keywords_config_id_rank_tracking_configs_id_fk": {+ "name": "rank_tracking_keywords_config_id_rank_tracking_configs_id_fk",+ "tableFrom": "rank_tracking_keywords",+ "tableTo": "rank_tracking_configs",+ "columnsFrom": [+ "config_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ }+ },+ "compositePrimaryKeys": {},+ "uniqueConstraints": {},+ "checkConstraints": {}+ },+ "saved_keyword_tag_assignments": {+ "name": "saved_keyword_tag_assignments",+ "columns": {+ "saved_keyword_id": {+ "name": "saved_keyword_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "tag_id": {+ "name": "tag_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "created_at": {+ "name": "created_at",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": "(current_timestamp)"+ }+ },+ "indexes": {+ "saved_keyword_tag_assignments_unique_idx": {+ "name": "saved_keyword_tag_assignments_unique_idx",+ "columns": [+ "saved_keyword_id",+ "tag_id"+ ],+ "isUnique": true+ },+ "saved_keyword_tag_assignments_tag_idx": {+ "name": "saved_keyword_tag_assignments_tag_idx",+ "columns": [+ "tag_id"+ ],+ "isUnique": false+ }+ },+ "foreignKeys": {+ "saved_keyword_tag_assignments_saved_keyword_id_saved_keywords_id_fk": {+ "name": "saved_keyword_tag_assignments_saved_keyword_id_saved_keywords_id_fk",+ "tableFrom": "saved_keyword_tag_assignments",+ "tableTo": "saved_keywords",+ "columnsFrom": [+ "saved_keyword_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ },+ "saved_keyword_tag_assignments_tag_id_saved_keyword_tags_id_fk": {+ "name": "saved_keyword_tag_assignments_tag_id_saved_keyword_tags_id_fk",+ "tableFrom": "saved_keyword_tag_assignments",+ "tableTo": "saved_keyword_tags",+ "columnsFrom": [+ "tag_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ }+ },+ "compositePrimaryKeys": {},+ "uniqueConstraints": {},+ "checkConstraints": {}+ },+ "saved_keyword_tags": {+ "name": "saved_keyword_tags",+ "columns": {+ "id": {+ "name": "id",+ "type": "text",+ "primaryKey": true,+ "notNull": true,+ "autoincrement": false+ },+ "project_id": {+ "name": "project_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "name": {+ "name": "name",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "normalized_name": {+ "name": "normalized_name",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "color": {+ "name": "color",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "created_at": {+ "name": "created_at",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": "(current_timestamp)"+ }+ },+ "indexes": {+ "saved_keyword_tags_project_normalized_name_idx": {+ "name": "saved_keyword_tags_project_normalized_name_idx",+ "columns": [+ "project_id",+ "normalized_name"+ ],+ "isUnique": true+ },+ "saved_keyword_tags_project_name_idx": {+ "name": "saved_keyword_tags_project_name_idx",+ "columns": [+ "project_id",+ "name"+ ],+ "isUnique": false+ }+ },+ "foreignKeys": {+ "saved_keyword_tags_project_id_projects_id_fk": {+ "name": "saved_keyword_tags_project_id_projects_id_fk",+ "tableFrom": "saved_keyword_tags",+ "tableTo": "projects",+ "columnsFrom": [+ "project_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ }+ },+ "compositePrimaryKeys": {},+ "uniqueConstraints": {},+ "checkConstraints": {}+ },+ "saved_keywords": {+ "name": "saved_keywords",+ "columns": {+ "id": {+ "name": "id",+ "type": "text",+ "primaryKey": true,+ "notNull": true,+ "autoincrement": false+ },+ "project_id": {+ "name": "project_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "keyword": {+ "name": "keyword",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "location_code": {+ "name": "location_code",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": 2840+ },+ "language_code": {+ "name": "language_code",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": "'en'"+ },+ "created_at": {+ "name": "created_at",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": "(current_timestamp)"+ }+ },+ "indexes": {+ "saved_keywords_unique_project_keyword_location_language": {+ "name": "saved_keywords_unique_project_keyword_location_language",+ "columns": [+ "project_id",+ "keyword",+ "location_code",+ "language_code"+ ],+ "isUnique": true+ },+ "saved_keywords_project_created_idx": {+ "name": "saved_keywords_project_created_idx",+ "columns": [+ "project_id",+ "created_at"+ ],+ "isUnique": false+ }+ },+ "foreignKeys": {+ "saved_keywords_project_id_projects_id_fk": {+ "name": "saved_keywords_project_id_projects_id_fk",+ "tableFrom": "saved_keywords",+ "tableTo": "projects",+ "columnsFrom": [+ "project_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ }+ },+ "compositePrimaryKeys": {},+ "uniqueConstraints": {},+ "checkConstraints": {}+ },+ "user_onboarding_answers": {+ "name": "user_onboarding_answers",+ "columns": {+ "user_id": {+ "name": "user_id",+ "type": "text",+ "primaryKey": true,+ "notNull": true,+ "autoincrement": false+ },+ "organization_id": {+ "name": "organization_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "interested_features": {+ "name": "interested_features",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": "'[]'"+ },+ "work_for": {+ "name": "work_for",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "client_website_count": {+ "name": "client_website_count",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "found_via": {+ "name": "found_via",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "mcp_setup_intent": {+ "name": "mcp_setup_intent",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "completed_at": {+ "name": "completed_at",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "gsc_nudge_dismissed_at": {+ "name": "gsc_nudge_dismissed_at",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "created_at": {+ "name": "created_at",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": "(current_timestamp)"+ },+ "updated_at": {+ "name": "updated_at",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": "(current_timestamp)"+ }+ },+ "indexes": {+ "user_onboarding_answers_organization_idx": {+ "name": "user_onboarding_answers_organization_idx",+ "columns": [+ "organization_id"+ ],+ "isUnique": false+ }+ },+ "foreignKeys": {+ "user_onboarding_answers_user_id_user_id_fk": {+ "name": "user_onboarding_answers_user_id_user_id_fk",+ "tableFrom": "user_onboarding_answers",+ "tableTo": "user",+ "columnsFrom": [+ "user_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ },+ "user_onboarding_answers_organization_id_organization_id_fk": {+ "name": "user_onboarding_answers_organization_id_organization_id_fk",+ "tableFrom": "user_onboarding_answers",+ "tableTo": "organization",+ "columnsFrom": [+ "organization_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ }+ },+ "compositePrimaryKeys": {},+ "uniqueConstraints": {},+ "checkConstraints": {}+ },+ "audit_issues": {+ "name": "audit_issues",+ "columns": {+ "id": {+ "name": "id",+ "type": "text",+ "primaryKey": true,+ "notNull": true,+ "autoincrement": false+ },+ "audit_id": {+ "name": "audit_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "page_id": {+ "name": "page_id",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "page_url": {+ "name": "page_url",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "issue_type": {+ "name": "issue_type",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "severity": {+ "name": "severity",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": "'info'"+ },+ "details_json": {+ "name": "details_json",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ }+ },+ "indexes": {+ "audit_issues_audit_id_idx": {+ "name": "audit_issues_audit_id_idx",+ "columns": [+ "audit_id"+ ],+ "isUnique": false+ },+ "audit_issues_audit_type_idx": {+ "name": "audit_issues_audit_type_idx",+ "columns": [+ "audit_id",+ "issue_type"+ ],+ "isUnique": false+ }+ },+ "foreignKeys": {+ "audit_issues_audit_id_audits_id_fk": {+ "name": "audit_issues_audit_id_audits_id_fk",+ "tableFrom": "audit_issues",+ "tableTo": "audits",+ "columnsFrom": [+ "audit_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ },+ "audit_issues_page_id_audit_pages_id_fk": {+ "name": "audit_issues_page_id_audit_pages_id_fk",+ "tableFrom": "audit_issues",+ "tableTo": "audit_pages",+ "columnsFrom": [+ "page_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ }+ },+ "compositePrimaryKeys": {},+ "uniqueConstraints": {},+ "checkConstraints": {}+ },+ "audit_lighthouse_results": {+ "name": "audit_lighthouse_results",+ "columns": {+ "id": {+ "name": "id",+ "type": "text",+ "primaryKey": true,+ "notNull": true,+ "autoincrement": false+ },+ "audit_id": {+ "name": "audit_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "page_id": {+ "name": "page_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "strategy": {+ "name": "strategy",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "performance_score": {+ "name": "performance_score",+ "type": "integer",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "accessibility_score": {+ "name": "accessibility_score",+ "type": "integer",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "best_practices_score": {+ "name": "best_practices_score",+ "type": "integer",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "seo_score": {+ "name": "seo_score",+ "type": "integer",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "lcp_ms": {+ "name": "lcp_ms",+ "type": "real",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "cls": {+ "name": "cls",+ "type": "real",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "inp_ms": {+ "name": "inp_ms",+ "type": "real",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "ttfb_ms": {+ "name": "ttfb_ms",+ "type": "real",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "error_message": {+ "name": "error_message",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "r2_key": {+ "name": "r2_key",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "payload_size_bytes": {+ "name": "payload_size_bytes",+ "type": "integer",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ }+ },+ "indexes": {+ "audit_lighthouse_results_audit_id_idx": {+ "name": "audit_lighthouse_results_audit_id_idx",+ "columns": [+ "audit_id"+ ],+ "isUnique": false+ }+ },+ "foreignKeys": {+ "audit_lighthouse_results_audit_id_audits_id_fk": {+ "name": "audit_lighthouse_results_audit_id_audits_id_fk",+ "tableFrom": "audit_lighthouse_results",+ "tableTo": "audits",+ "columnsFrom": [+ "audit_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ },+ "audit_lighthouse_results_page_id_audit_pages_id_fk": {+ "name": "audit_lighthouse_results_page_id_audit_pages_id_fk",+ "tableFrom": "audit_lighthouse_results",+ "tableTo": "audit_pages",+ "columnsFrom": [+ "page_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ }+ },+ "compositePrimaryKeys": {},+ "uniqueConstraints": {},+ "checkConstraints": {}+ },+ "audit_links": {+ "name": "audit_links",+ "columns": {+ "id": {+ "name": "id",+ "type": "text",+ "primaryKey": true,+ "notNull": true,+ "autoincrement": false+ },+ "audit_id": {+ "name": "audit_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "source_page_id": {+ "name": "source_page_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "source_url": {+ "name": "source_url",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "target_url": {+ "name": "target_url",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "anchor": {+ "name": "anchor",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "is_internal": {+ "name": "is_internal",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": true+ },+ "is_nofollow": {+ "name": "is_nofollow",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": false+ }+ },+ "indexes": {+ "audit_links_audit_id_idx": {+ "name": "audit_links_audit_id_idx",+ "columns": [+ "audit_id"+ ],+ "isUnique": false+ },+ "audit_links_audit_target_idx": {+ "name": "audit_links_audit_target_idx",+ "columns": [+ "audit_id",+ "target_url"+ ],+ "isUnique": false+ }+ },+ "foreignKeys": {+ "audit_links_audit_id_audits_id_fk": {+ "name": "audit_links_audit_id_audits_id_fk",+ "tableFrom": "audit_links",+ "tableTo": "audits",+ "columnsFrom": [+ "audit_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ },+ "audit_links_source_page_id_audit_pages_id_fk": {+ "name": "audit_links_source_page_id_audit_pages_id_fk",+ "tableFrom": "audit_links",+ "tableTo": "audit_pages",+ "columnsFrom": [+ "source_page_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ }+ },+ "compositePrimaryKeys": {},+ "uniqueConstraints": {},+ "checkConstraints": {}+ },+ "audit_pages": {+ "name": "audit_pages",+ "columns": {+ "id": {+ "name": "id",+ "type": "text",+ "primaryKey": true,+ "notNull": true,+ "autoincrement": false+ },+ "audit_id": {+ "name": "audit_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "url": {+ "name": "url",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "status_code": {+ "name": "status_code",+ "type": "integer",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "redirect_url": {+ "name": "redirect_url",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "title": {+ "name": "title",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "meta_description": {+ "name": "meta_description",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "canonical_url": {+ "name": "canonical_url",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "robots_meta": {+ "name": "robots_meta",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "og_title": {+ "name": "og_title",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "og_description": {+ "name": "og_description",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "og_image": {+ "name": "og_image",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "h1_count": {+ "name": "h1_count",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": 0+ },+ "h2_count": {+ "name": "h2_count",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": 0+ },+ "h3_count": {+ "name": "h3_count",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": 0+ },+ "h4_count": {+ "name": "h4_count",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": 0+ },+ "h5_count": {+ "name": "h5_count",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": 0+ },+ "h6_count": {+ "name": "h6_count",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": 0+ },+ "heading_order_json": {+ "name": "heading_order_json",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "word_count": {+ "name": "word_count",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": 0+ },+ "images_total": {+ "name": "images_total",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": 0+ },+ "images_missing_alt": {+ "name": "images_missing_alt",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": 0+ },+ "images_json": {+ "name": "images_json",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "internal_link_count": {+ "name": "internal_link_count",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": 0+ },+ "external_link_count": {+ "name": "external_link_count",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": 0+ },+ "has_structured_data": {+ "name": "has_structured_data",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": false+ },+ "hreflang_tags_json": {+ "name": "hreflang_tags_json",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "is_indexable": {+ "name": "is_indexable",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": true+ },+ "x_robots_tag": {+ "name": "x_robots_tag",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "header_canonical_url": {+ "name": "header_canonical_url",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "crawl_depth": {+ "name": "crawl_depth",+ "type": "integer",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "in_sitemap": {+ "name": "in_sitemap",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": false+ },+ "content_hash": {+ "name": "content_hash",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "fetch_class": {+ "name": "fetch_class",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": "'ok'"+ },+ "response_time_ms": {+ "name": "response_time_ms",+ "type": "integer",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ }+ },+ "indexes": {+ "audit_pages_audit_id_idx": {+ "name": "audit_pages_audit_id_idx",+ "columns": [+ "audit_id"+ ],+ "isUnique": false+ },+ "audit_pages_audit_url_idx": {+ "name": "audit_pages_audit_url_idx",+ "columns": [+ "audit_id",+ "url"+ ],+ "isUnique": false+ }+ },+ "foreignKeys": {+ "audit_pages_audit_id_audits_id_fk": {+ "name": "audit_pages_audit_id_audits_id_fk",+ "tableFrom": "audit_pages",+ "tableTo": "audits",+ "columnsFrom": [+ "audit_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ }+ },+ "compositePrimaryKeys": {},+ "uniqueConstraints": {},+ "checkConstraints": {}+ },+ "audits": {+ "name": "audits",+ "columns": {+ "id": {+ "name": "id",+ "type": "text",+ "primaryKey": true,+ "notNull": true,+ "autoincrement": false+ },+ "project_id": {+ "name": "project_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "started_by_user_id": {+ "name": "started_by_user_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "start_url": {+ "name": "start_url",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "status": {+ "name": "status",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": "'running'"+ },+ "workflow_instance_id": {+ "name": "workflow_instance_id",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "config": {+ "name": "config",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": "'{}'"+ },+ "pages_crawled": {+ "name": "pages_crawled",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": 0+ },+ "pages_total": {+ "name": "pages_total",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": 0+ },+ "lighthouse_total": {+ "name": "lighthouse_total",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": 0+ },+ "lighthouse_completed": {+ "name": "lighthouse_completed",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": 0+ },+ "lighthouse_failed": {+ "name": "lighthouse_failed",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": 0+ },+ "current_phase": {+ "name": "current_phase",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false,+ "default": "'discovery'"+ },+ "started_at": {+ "name": "started_at",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": "(current_timestamp)"+ },+ "completed_at": {+ "name": "completed_at",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ }+ },+ "indexes": {+ "audits_project_id_idx": {+ "name": "audits_project_id_idx",+ "columns": [+ "project_id"+ ],+ "isUnique": false+ },+ "audits_started_by_user_id_idx": {+ "name": "audits_started_by_user_id_idx",+ "columns": [+ "started_by_user_id"+ ],+ "isUnique": false+ }+ },+ "foreignKeys": {+ "audits_project_id_projects_id_fk": {+ "name": "audits_project_id_projects_id_fk",+ "tableFrom": "audits",+ "tableTo": "projects",+ "columnsFrom": [+ "project_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ }+ },+ "compositePrimaryKeys": {},+ "uniqueConstraints": {},+ "checkConstraints": {}+ },+ "sam_project_memory": {+ "name": "sam_project_memory",+ "columns": {+ "project_id": {+ "name": "project_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "label": {+ "name": "label",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "content": {+ "name": "content",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "updated_at": {+ "name": "updated_at",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": "(current_timestamp)"+ }+ },+ "indexes": {},+ "foreignKeys": {+ "sam_project_memory_project_id_projects_id_fk": {+ "name": "sam_project_memory_project_id_projects_id_fk",+ "tableFrom": "sam_project_memory",+ "tableTo": "projects",+ "columnsFrom": [+ "project_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ }+ },+ "compositePrimaryKeys": {+ "sam_project_memory_project_id_label_pk": {+ "columns": [+ "project_id",+ "label"+ ],+ "name": "sam_project_memory_project_id_label_pk"+ }+ },+ "uniqueConstraints": {},+ "checkConstraints": {}+ },+ "sam_sessions": {+ "name": "sam_sessions",+ "columns": {+ "id": {+ "name": "id",+ "type": "text",+ "primaryKey": true,+ "notNull": true,+ "autoincrement": false+ },+ "project_id": {+ "name": "project_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "user_id": {+ "name": "user_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "title": {+ "name": "title",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": "'New chat'"+ },+ "created_at": {+ "name": "created_at",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": "(current_timestamp)"+ },+ "updated_at": {+ "name": "updated_at",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": "(current_timestamp)"+ },+ "archived_at": {+ "name": "archived_at",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ }+ },+ "indexes": {+ "sam_sessions_project_updated_idx": {+ "name": "sam_sessions_project_updated_idx",+ "columns": [+ "project_id",+ "updated_at"+ ],+ "isUnique": false+ }+ },+ "foreignKeys": {+ "sam_sessions_project_id_projects_id_fk": {+ "name": "sam_sessions_project_id_projects_id_fk",+ "tableFrom": "sam_sessions",+ "tableTo": "projects",+ "columnsFrom": [+ "project_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ },+ "sam_sessions_user_id_user_id_fk": {+ "name": "sam_sessions_user_id_user_id_fk",+ "tableFrom": "sam_sessions",+ "tableTo": "user",+ "columnsFrom": [+ "user_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ }+ },+ "compositePrimaryKeys": {},+ "uniqueConstraints": {},+ "checkConstraints": {}+ },+ "account": {+ "name": "account",+ "columns": {+ "id": {+ "name": "id",+ "type": "text",+ "primaryKey": true,+ "notNull": true,+ "autoincrement": false+ },+ "account_id": {+ "name": "account_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "provider_id": {+ "name": "provider_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "user_id": {+ "name": "user_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "access_token": {+ "name": "access_token",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "refresh_token": {+ "name": "refresh_token",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "id_token": {+ "name": "id_token",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "access_token_expires_at": {+ "name": "access_token_expires_at",+ "type": "integer",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "refresh_token_expires_at": {+ "name": "refresh_token_expires_at",+ "type": "integer",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "scope": {+ "name": "scope",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "password": {+ "name": "password",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "created_at": {+ "name": "created_at",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": "(cast(unixepoch('subsecond') * 1000 as integer))"+ },+ "updated_at": {+ "name": "updated_at",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ }+ },+ "indexes": {+ "account_userId_idx": {+ "name": "account_userId_idx",+ "columns": [+ "user_id"+ ],+ "isUnique": false+ },+ "account_accountId_providerId_idx": {+ "name": "account_accountId_providerId_idx",+ "columns": [+ "account_id",+ "provider_id"+ ],+ "isUnique": false+ }+ },+ "foreignKeys": {+ "account_user_id_user_id_fk": {+ "name": "account_user_id_user_id_fk",+ "tableFrom": "account",+ "tableTo": "user",+ "columnsFrom": [+ "user_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ }+ },+ "compositePrimaryKeys": {},+ "uniqueConstraints": {},+ "checkConstraints": {}+ },+ "invitation": {+ "name": "invitation",+ "columns": {+ "id": {+ "name": "id",+ "type": "text",+ "primaryKey": true,+ "notNull": true,+ "autoincrement": false+ },+ "organization_id": {+ "name": "organization_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "email": {+ "name": "email",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "role": {+ "name": "role",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "status": {+ "name": "status",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": "'pending'"+ },+ "expires_at": {+ "name": "expires_at",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "created_at": {+ "name": "created_at",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": "(cast(unixepoch('subsecond') * 1000 as integer))"+ },+ "inviter_id": {+ "name": "inviter_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ }+ },+ "indexes": {+ "invitation_organizationId_idx": {+ "name": "invitation_organizationId_idx",+ "columns": [+ "organization_id"+ ],+ "isUnique": false+ },+ "invitation_email_idx": {+ "name": "invitation_email_idx",+ "columns": [+ "email"+ ],+ "isUnique": false+ }+ },+ "foreignKeys": {+ "invitation_organization_id_organization_id_fk": {+ "name": "invitation_organization_id_organization_id_fk",+ "tableFrom": "invitation",+ "tableTo": "organization",+ "columnsFrom": [+ "organization_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ },+ "invitation_inviter_id_user_id_fk": {+ "name": "invitation_inviter_id_user_id_fk",+ "tableFrom": "invitation",+ "tableTo": "user",+ "columnsFrom": [+ "inviter_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ }+ },+ "compositePrimaryKeys": {},+ "uniqueConstraints": {},+ "checkConstraints": {}+ },+ "member": {+ "name": "member",+ "columns": {+ "id": {+ "name": "id",+ "type": "text",+ "primaryKey": true,+ "notNull": true,+ "autoincrement": false+ },+ "organization_id": {+ "name": "organization_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "user_id": {+ "name": "user_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "role": {+ "name": "role",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": "'member'"+ },+ "created_at": {+ "name": "created_at",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ }+ },+ "indexes": {+ "member_organizationId_idx": {+ "name": "member_organizationId_idx",+ "columns": [+ "organization_id"+ ],+ "isUnique": false+ },+ "member_userId_idx": {+ "name": "member_userId_idx",+ "columns": [+ "user_id"+ ],+ "isUnique": false+ }+ },+ "foreignKeys": {+ "member_organization_id_organization_id_fk": {+ "name": "member_organization_id_organization_id_fk",+ "tableFrom": "member",+ "tableTo": "organization",+ "columnsFrom": [+ "organization_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ },+ "member_user_id_user_id_fk": {+ "name": "member_user_id_user_id_fk",+ "tableFrom": "member",+ "tableTo": "user",+ "columnsFrom": [+ "user_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ }+ },+ "compositePrimaryKeys": {},+ "uniqueConstraints": {},+ "checkConstraints": {}+ },+ "organization": {+ "name": "organization",+ "columns": {+ "id": {+ "name": "id",+ "type": "text",+ "primaryKey": true,+ "notNull": true,+ "autoincrement": false+ },+ "name": {+ "name": "name",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "slug": {+ "name": "slug",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "logo": {+ "name": "logo",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "created_at": {+ "name": "created_at",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "metadata": {+ "name": "metadata",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ }+ },+ "indexes": {+ "organization_slug_unique": {+ "name": "organization_slug_unique",+ "columns": [+ "slug"+ ],+ "isUnique": true+ },+ "organization_slug_uidx": {+ "name": "organization_slug_uidx",+ "columns": [+ "slug"+ ],+ "isUnique": true+ }+ },+ "foreignKeys": {},+ "compositePrimaryKeys": {},+ "uniqueConstraints": {},+ "checkConstraints": {}+ },+ "session": {+ "name": "session",+ "columns": {+ "id": {+ "name": "id",+ "type": "text",+ "primaryKey": true,+ "notNull": true,+ "autoincrement": false+ },+ "expires_at": {+ "name": "expires_at",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "token": {+ "name": "token",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "created_at": {+ "name": "created_at",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": "(cast(unixepoch('subsecond') * 1000 as integer))"+ },+ "updated_at": {+ "name": "updated_at",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "ip_address": {+ "name": "ip_address",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "user_agent": {+ "name": "user_agent",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "user_id": {+ "name": "user_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "active_organization_id": {+ "name": "active_organization_id",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ }+ },+ "indexes": {+ "session_token_unique": {+ "name": "session_token_unique",+ "columns": [+ "token"+ ],+ "isUnique": true+ },+ "session_userId_idx": {+ "name": "session_userId_idx",+ "columns": [+ "user_id"+ ],+ "isUnique": false+ }+ },+ "foreignKeys": {+ "session_user_id_user_id_fk": {+ "name": "session_user_id_user_id_fk",+ "tableFrom": "session",+ "tableTo": "user",+ "columnsFrom": [+ "user_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ }+ },+ "compositePrimaryKeys": {},+ "uniqueConstraints": {},+ "checkConstraints": {}+ },+ "user": {+ "name": "user",+ "columns": {+ "id": {+ "name": "id",+ "type": "text",+ "primaryKey": true,+ "notNull": true,+ "autoincrement": false+ },+ "name": {+ "name": "name",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "email": {+ "name": "email",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "email_verified": {+ "name": "email_verified",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": false+ },+ "image": {+ "name": "image",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "created_at": {+ "name": "created_at",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": "(cast(unixepoch('subsecond') * 1000 as integer))"+ },+ "updated_at": {+ "name": "updated_at",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": "(cast(unixepoch('subsecond') * 1000 as integer))"+ },+ "analytics_opted_out": {+ "name": "analytics_opted_out",+ "type": "integer",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ }+ },+ "indexes": {+ "user_email_unique": {+ "name": "user_email_unique",+ "columns": [+ "email"+ ],+ "isUnique": true+ }+ },+ "foreignKeys": {},+ "compositePrimaryKeys": {},+ "uniqueConstraints": {},+ "checkConstraints": {}+ },+ "verification": {+ "name": "verification",+ "columns": {+ "id": {+ "name": "id",+ "type": "text",+ "primaryKey": true,+ "notNull": true,+ "autoincrement": false+ },+ "identifier": {+ "name": "identifier",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "value": {+ "name": "value",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "expires_at": {+ "name": "expires_at",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "created_at": {+ "name": "created_at",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": "(cast(unixepoch('subsecond') * 1000 as integer))"+ },+ "updated_at": {+ "name": "updated_at",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": "(cast(unixepoch('subsecond') * 1000 as integer))"+ }+ },+ "indexes": {+ "verification_identifier_idx": {+ "name": "verification_identifier_idx",+ "columns": [+ "identifier"+ ],+ "isUnique": false+ },+ "verification_expiresAt_idx": {+ "name": "verification_expiresAt_idx",+ "columns": [+ "expires_at"+ ],+ "isUnique": false+ }+ },+ "foreignKeys": {},+ "compositePrimaryKeys": {},+ "uniqueConstraints": {},+ "checkConstraints": {}+ },+ "billing_customer_status": {+ "name": "billing_customer_status",+ "columns": {+ "organization_id": {+ "name": "organization_id",+ "type": "text",+ "primaryKey": true,+ "notNull": true,+ "autoincrement": false+ },+ "is_paying": {+ "name": "is_paying",+ "type": "integer",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": false+ },+ "paid_plan_id": {+ "name": "paid_plan_id",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "paid_plan_status": {+ "name": "paid_plan_status",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "customer_json": {+ "name": "customer_json",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "synced_at": {+ "name": "synced_at",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "created_at": {+ "name": "created_at",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": "(current_timestamp)"+ },+ "updated_at": {+ "name": "updated_at",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": "(current_timestamp)"+ }+ },+ "indexes": {},+ "foreignKeys": {+ "billing_customer_status_organization_id_organization_id_fk": {+ "name": "billing_customer_status_organization_id_organization_id_fk",+ "tableFrom": "billing_customer_status",+ "tableTo": "organization",+ "columnsFrom": [+ "organization_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ }+ },+ "compositePrimaryKeys": {},+ "uniqueConstraints": {},+ "checkConstraints": {}+ },+ "gsc_connections": {+ "name": "gsc_connections",+ "columns": {+ "id": {+ "name": "id",+ "type": "text",+ "primaryKey": true,+ "notNull": true,+ "autoincrement": false+ },+ "project_id": {+ "name": "project_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "organization_id": {+ "name": "organization_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "site_url": {+ "name": "site_url",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "connected_by_user_id": {+ "name": "connected_by_user_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "connected_account_email": {+ "name": "connected_account_email",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "created_at": {+ "name": "created_at",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": "(current_timestamp)"+ },+ "updated_at": {+ "name": "updated_at",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": "(current_timestamp)"+ }+ },+ "indexes": {+ "gsc_connections_project_idx": {+ "name": "gsc_connections_project_idx",+ "columns": [+ "project_id"+ ],+ "isUnique": true+ },+ "gsc_connections_organization_idx": {+ "name": "gsc_connections_organization_idx",+ "columns": [+ "organization_id"+ ],+ "isUnique": false+ }+ },+ "foreignKeys": {+ "gsc_connections_project_id_projects_id_fk": {+ "name": "gsc_connections_project_id_projects_id_fk",+ "tableFrom": "gsc_connections",+ "tableTo": "projects",+ "columnsFrom": [+ "project_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ },+ "gsc_connections_organization_id_organization_id_fk": {+ "name": "gsc_connections_organization_id_organization_id_fk",+ "tableFrom": "gsc_connections",+ "tableTo": "organization",+ "columnsFrom": [+ "organization_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ }+ },+ "compositePrimaryKeys": {},+ "uniqueConstraints": {},+ "checkConstraints": {}+ },+ "reddit_attributions": {+ "name": "reddit_attributions",+ "columns": {+ "id": {+ "name": "id",+ "type": "text",+ "primaryKey": true,+ "notNull": true,+ "autoincrement": false+ },+ "user_id": {+ "name": "user_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "organization_id": {+ "name": "organization_id",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false+ },+ "click_id": {+ "name": "click_id",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "uuid": {+ "name": "uuid",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "landing_page": {+ "name": "landing_page",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "referrer": {+ "name": "referrer",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "utm_source": {+ "name": "utm_source",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "utm_medium": {+ "name": "utm_medium",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "utm_campaign": {+ "name": "utm_campaign",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "utm_term": {+ "name": "utm_term",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "utm_content": {+ "name": "utm_content",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "signup_sent_at": {+ "name": "signup_sent_at",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "purchase_sent_at": {+ "name": "purchase_sent_at",+ "type": "text",+ "primaryKey": false,+ "notNull": false,+ "autoincrement": false+ },+ "created_at": {+ "name": "created_at",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": "(current_timestamp)"+ },+ "updated_at": {+ "name": "updated_at",+ "type": "text",+ "primaryKey": false,+ "notNull": true,+ "autoincrement": false,+ "default": "(current_timestamp)"+ }+ },+ "indexes": {+ "reddit_attributions_user_idx": {+ "name": "reddit_attributions_user_idx",+ "columns": [+ "user_id"+ ],+ "isUnique": true+ },+ "reddit_attributions_organization_idx": {+ "name": "reddit_attributions_organization_idx",+ "columns": [+ "organization_id"+ ],+ "isUnique": false+ }+ },+ "foreignKeys": {+ "reddit_attributions_user_id_user_id_fk": {+ "name": "reddit_attributions_user_id_user_id_fk",+ "tableFrom": "reddit_attributions",+ "tableTo": "user",+ "columnsFrom": [+ "user_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ },+ "reddit_attributions_organization_id_organization_id_fk": {+ "name": "reddit_attributions_organization_id_organization_id_fk",+ "tableFrom": "reddit_attributions",+ "tableTo": "organization",+ "columnsFrom": [+ "organization_id"+ ],+ "columnsTo": [+ "id"+ ],+ "onDelete": "cascade",+ "onUpdate": "no action"+ }+ },+ "compositePrimaryKeys": {},+ "uniqueConstraints": {},+ "checkConstraints": {}+ }+ },+ "views": {},+ "enums": {},+ "_meta": {+ "schemas": {},+ "tables": {},+ "columns": {}+ },+ "internal": {+ "indexes": {}+ }+}\ No newline at end of filediff --git a/drizzle/meta/_journal.json b/drizzle/meta/_journal.jsonindex 8c858fda..dd32744a 100644--- a/drizzle/meta/_journal.json+++ b/drizzle/meta/_journal.json@@ -211,6 +211,13 @@"when": 1783438453855,"tag": "0029_location_name","breakpoints": true+ },+ {+ "idx": 30,+ "version": "6",+ "when": 1783306047190,+ "tag": "0030_legal_reaper",+ "breakpoints": true}]}\ No newline at end of filediff --git a/src/db/app.schema.ts b/src/db/app.schema.tsindex 382dd652..a292982c 100644--- a/src/db/app.schema.ts+++ b/src/db/app.schema.ts@@ -345,123 +345,3 @@ export const rankSnapshots = sqliteTable(),],);--// ============================================================================-// Site Audit tables-// ============================================================================--// One row per audit run-export const audits = sqliteTable(- "audits",- {- id: text("id").primaryKey(),- projectId: text("project_id")- .notNull()- .references(() => projects.id, { onDelete: "cascade" }),- startedByUserId: text("started_by_user_id").notNull(),- startUrl: text("start_url").notNull(),- status: text("status", {- enum: ["running", "completed", "failed"],- })- .notNull()- .default("running"),- workflowInstanceId: text("workflow_instance_id"),- // JSON config: { maxPages, lighthouseStrategy }- config: text("config").notNull().default("{}"),- // Progress & summary- pagesCrawled: integer("pages_crawled").notNull().default(0),- pagesTotal: integer("pages_total").notNull().default(0),- lighthouseTotal: integer("lighthouse_total").notNull().default(0),- lighthouseCompleted: integer("lighthouse_completed").notNull().default(0),- lighthouseFailed: integer("lighthouse_failed").notNull().default(0),- currentPhase: text("current_phase").default("discovery"),- startedAt: text("started_at")- .notNull()- .default(sql`(current_timestamp)`),- completedAt: text("completed_at"),- },- (table) => [- index("audits_project_id_idx").on(table.projectId),- index("audits_started_by_user_id_idx").on(table.startedByUserId),- ],-);--// One row per crawled page-export const auditPages = sqliteTable(- "audit_pages",- {- id: text("id").primaryKey(),- auditId: text("audit_id")- .notNull()- .references(() => audits.id, { onDelete: "cascade" }),- url: text("url").notNull(),- statusCode: integer("status_code"),- redirectUrl: text("redirect_url"),- // Metadata- title: text("title"),- metaDescription: text("meta_description"),- canonicalUrl: text("canonical_url"),- robotsMeta: text("robots_meta"),- // Open Graph- ogTitle: text("og_title"),- ogDescription: text("og_description"),- ogImage: text("og_image"),- // Headings- h1Count: integer("h1_count").notNull().default(0),- h2Count: integer("h2_count").notNull().default(0),- h3Count: integer("h3_count").notNull().default(0),- h4Count: integer("h4_count").notNull().default(0),- h5Count: integer("h5_count").notNull().default(0),- h6Count: integer("h6_count").notNull().default(0),- headingOrderJson: text("heading_order_json"),- // Content- wordCount: integer("word_count").notNull().default(0),- // Images- imagesTotal: integer("images_total").notNull().default(0),- imagesMissingAlt: integer("images_missing_alt").notNull().default(0),- imagesJson: text("images_json"),- // Links- internalLinkCount: integer("internal_link_count").notNull().default(0),- externalLinkCount: integer("external_link_count").notNull().default(0),- // Structured data- hasStructuredData: integer("has_structured_data", { mode: "boolean" })- .notNull()- .default(false),- // Hreflang- hreflangTagsJson: text("hreflang_tags_json"),- // Indexability- isIndexable: integer("is_indexable", { mode: "boolean" })- .notNull()- .default(true),- // Performance- responseTimeMs: integer("response_time_ms"),- },- (table) => [index("audit_pages_audit_id_idx").on(table.auditId)],-);--// One row per Lighthouse test (mobile + desktop per page).-export const auditLighthouseResults = sqliteTable(- "audit_lighthouse_results",- {- id: text("id").primaryKey(),- auditId: text("audit_id")- .notNull()- .references(() => audits.id, { onDelete: "cascade" }),- pageId: text("page_id")- .notNull()- .references(() => auditPages.id, { onDelete: "cascade" }),- strategy: text("strategy", { enum: ["mobile", "desktop"] }).notNull(),- performanceScore: integer("performance_score"),- accessibilityScore: integer("accessibility_score"),- bestPracticesScore: integer("best_practices_score"),- seoScore: integer("seo_score"),- lcpMs: real("lcp_ms"),- cls: real("cls"),- inpMs: real("inp_ms"),- ttfbMs: real("ttfb_ms"),- errorMessage: text("error_message"),- r2Key: text("r2_key"),- payloadSizeBytes: integer("payload_size_bytes"),- },- (table) => [index("audit_lighthouse_results_audit_id_idx").on(table.auditId)],-);diff --git a/src/db/audit.schema.ts b/src/db/audit.schema.tsnew file mode 100644index 00000000..55c3fa61--- /dev/null+++ b/src/db/audit.schema.ts@@ -0,0 +1,201 @@+import {+ sqliteTable,+ text,+ integer,+ real,+ index,+} from "drizzle-orm/sqlite-core";+import { sql } from "drizzle-orm";+import { projects } from "./app.schema";++// ============================================================================+// Site Audit tables+// ============================================================================++// One row per audit run+export const audits = sqliteTable(+ "audits",+ {+ id: text("id").primaryKey(),+ projectId: text("project_id")+ .notNull()+ .references(() => projects.id, { onDelete: "cascade" }),+ startedByUserId: text("started_by_user_id").notNull(),+ startUrl: text("start_url").notNull(),+ status: text("status", {+ enum: ["running", "completed", "failed"],+ })+ .notNull()+ .default("running"),+ workflowInstanceId: text("workflow_instance_id"),+ // JSON config: { maxPages, lighthouseStrategy }+ config: text("config").notNull().default("{}"),+ // Progress & summary+ pagesCrawled: integer("pages_crawled").notNull().default(0),+ pagesTotal: integer("pages_total").notNull().default(0),+ lighthouseTotal: integer("lighthouse_total").notNull().default(0),+ lighthouseCompleted: integer("lighthouse_completed").notNull().default(0),+ lighthouseFailed: integer("lighthouse_failed").notNull().default(0),+ currentPhase: text("current_phase").default("discovery"),+ startedAt: text("started_at")+ .notNull()+ .default(sql`(current_timestamp)`),+ completedAt: text("completed_at"),+ },+ (table) => [+ index("audits_project_id_idx").on(table.projectId),+ index("audits_started_by_user_id_idx").on(table.startedByUserId),+ ],+);++// One row per crawled page+export const auditPages = sqliteTable(+ "audit_pages",+ {+ id: text("id").primaryKey(),+ auditId: text("audit_id")+ .notNull()+ .references(() => audits.id, { onDelete: "cascade" }),+ url: text("url").notNull(),+ statusCode: integer("status_code"),+ redirectUrl: text("redirect_url"),+ // Metadata+ title: text("title"),+ metaDescription: text("meta_description"),+ canonicalUrl: text("canonical_url"),+ robotsMeta: text("robots_meta"),+ // Open Graph+ ogTitle: text("og_title"),+ ogDescription: text("og_description"),+ ogImage: text("og_image"),+ // Headings+ h1Count: integer("h1_count").notNull().default(0),+ h2Count: integer("h2_count").notNull().default(0),+ h3Count: integer("h3_count").notNull().default(0),+ h4Count: integer("h4_count").notNull().default(0),+ h5Count: integer("h5_count").notNull().default(0),+ h6Count: integer("h6_count").notNull().default(0),+ headingOrderJson: text("heading_order_json"),+ // Content+ wordCount: integer("word_count").notNull().default(0),+ // Images+ imagesTotal: integer("images_total").notNull().default(0),+ imagesMissingAlt: integer("images_missing_alt").notNull().default(0),+ imagesJson: text("images_json"),+ // Links+ internalLinkCount: integer("internal_link_count").notNull().default(0),+ externalLinkCount: integer("external_link_count").notNull().default(0),+ // Structured data+ hasStructuredData: integer("has_structured_data", { mode: "boolean" })+ .notNull()+ .default(false),+ // Hreflang+ hreflangTagsJson: text("hreflang_tags_json"),+ // Indexability+ isIndexable: integer("is_indexable", { mode: "boolean" })+ .notNull()+ .default(true),+ // Indexability/canonical signals from response headers+ xRobotsTag: text("x_robots_tag"),+ headerCanonicalUrl: text("header_canonical_url"),+ // Crawl metadata+ // null depth = not reached via links (e.g. sitemap-seeded)+ crawlDepth: integer("crawl_depth"),+ inSitemap: integer("in_sitemap", { mode: "boolean" })+ .notNull()+ .default(false),+ // SHA-256 of the visible body text, for duplicate-content grouping+ contentHash: text("content_hash"),+ // How the fetch resolved: ok | blocked (WAF/bot challenge) | error+ fetchClass: text("fetch_class", { enum: ["ok", "blocked", "error"] })+ .notNull()+ .default("ok"),+ // Performance+ responseTimeMs: integer("response_time_ms"),+ },+ (table) => [+ index("audit_pages_audit_id_idx").on(table.auditId),+ index("audit_pages_audit_url_idx").on(table.auditId, table.url),+ ],+);++// One row per unique (source page, target URL) link edge. Currently only+// internal edges are stored (see AuditRepository); isInternal stays so+// external-link checks can start writing rows without a migration.+export const auditLinks = sqliteTable(+ "audit_links",+ {+ id: text("id").primaryKey(),+ auditId: text("audit_id")+ .notNull()+ .references(() => audits.id, { onDelete: "cascade" }),+ sourcePageId: text("source_page_id")+ .notNull()+ .references(() => auditPages.id, { onDelete: "cascade" }),+ sourceUrl: text("source_url").notNull(),+ targetUrl: text("target_url").notNull(),+ anchor: text("anchor"),+ isInternal: integer("is_internal", { mode: "boolean" })+ .notNull()+ .default(true),+ isNofollow: integer("is_nofollow", { mode: "boolean" })+ .notNull()+ .default(false),+ },+ (table) => [+ index("audit_links_audit_id_idx").on(table.auditId),+ index("audit_links_audit_target_idx").on(table.auditId, table.targetUrl),+ ],+);++// One row per (issue type, affected page)+export const auditIssues = sqliteTable(+ "audit_issues",+ {+ id: text("id").primaryKey(),+ auditId: text("audit_id")+ .notNull()+ .references(() => audits.id, { onDelete: "cascade" }),+ pageId: text("page_id").references(() => auditPages.id, {+ onDelete: "cascade",+ }),+ pageUrl: text("page_url").notNull(),+ issueType: text("issue_type").notNull(),+ severity: text("severity", { enum: ["critical", "warning", "info"] })+ .notNull()+ .default("info"),+ // JSON details specific to the issue type (e.g. broken link target)+ detailsJson: text("details_json"),+ },+ (table) => [+ index("audit_issues_audit_id_idx").on(table.auditId),+ index("audit_issues_audit_type_idx").on(table.auditId, table.issueType),+ ],+);++// One row per Lighthouse test (mobile + desktop per page).+export const auditLighthouseResults = sqliteTable(+ "audit_lighthouse_results",+ {+ id: text("id").primaryKey(),+ auditId: text("audit_id")+ .notNull()+ .references(() => audits.id, { onDelete: "cascade" }),+ pageId: text("page_id")+ .notNull()+ .references(() => auditPages.id, { onDelete: "cascade" }),+ strategy: text("strategy", { enum: ["mobile", "desktop"] }).notNull(),+ performanceScore: integer("performance_score"),+ accessibilityScore: integer("accessibility_score"),+ bestPracticesScore: integer("best_practices_score"),+ seoScore: integer("seo_score"),+ lcpMs: real("lcp_ms"),+ cls: real("cls"),+ inpMs: real("inp_ms"),+ ttfbMs: real("ttfb_ms"),+ errorMessage: text("error_message"),+ r2Key: text("r2_key"),+ payloadSizeBytes: integer("payload_size_bytes"),+ },+ (table) => [index("audit_lighthouse_results_audit_id_idx").on(table.auditId)],+);diff --git a/src/db/d1/schema.ts b/src/db/d1/schema.tsindex 74d00809..63ef60b0 100644--- a/src/db/d1/schema.ts+++ b/src/db/d1/schema.ts@@ -2,6 +2,7 @@// which is the provider-aware barrel) so the D1 client always binds to the// SQLite tables regardless of DATABASE_PROVIDER.export * from "../app.schema";+export * from "../audit.schema";export * from "../sam.schema";export * from "../better-auth-schema";export * from "../billing.schema";diff --git a/src/db/pg/app.schema.ts b/src/db/pg/app.schema.tsindex ff427d48..b75fe5be 100644--- a/src/db/pg/app.schema.ts+++ b/src/db/pg/app.schema.ts@@ -338,112 +338,3 @@ export const rankSnapshots = pgTable(),],);--// ============================================================================-// Site Audit tables-// ============================================================================--// One row per audit run-export const audits = pgTable(- "audits",- {- id: text("id").primaryKey(),- projectId: text("project_id")- .notNull()- .references(() => projects.id, { onDelete: "cascade" }),- startedByUserId: text("started_by_user_id").notNull(),- startUrl: text("start_url").notNull(),- status: text("status", {- enum: ["running", "completed", "failed"],- })- .notNull()- .default("running"),- workflowInstanceId: text("workflow_instance_id"),- // JSON config: { maxPages, lighthouseStrategy }- config: text("config").notNull().default("{}"),- // Progress & summary- pagesCrawled: integer("pages_crawled").notNull().default(0),- pagesTotal: integer("pages_total").notNull().default(0),- lighthouseTotal: integer("lighthouse_total").notNull().default(0),- lighthouseCompleted: integer("lighthouse_completed").notNull().default(0),- lighthouseFailed: integer("lighthouse_failed").notNull().default(0),- currentPhase: text("current_phase").default("discovery"),- startedAt: timestampColumn("started_at").notNull().default(isoNow),- completedAt: timestampColumn("completed_at"),- },- (table) => [- index("audits_project_id_idx").on(table.projectId),- index("audits_started_by_user_id_idx").on(table.startedByUserId),- ],-);--// One row per crawled page-export const auditPages = pgTable(- "audit_pages",- {- id: text("id").primaryKey(),- auditId: text("audit_id")- .notNull()- .references(() => audits.id, { onDelete: "cascade" }),- url: text("url").notNull(),- statusCode: integer("status_code"),- redirectUrl: text("redirect_url"),- // Metadata- title: text("title"),- metaDescription: text("meta_description"),- canonicalUrl: text("canonical_url"),- robotsMeta: text("robots_meta"),- // Open Graph- ogTitle: text("og_title"),- ogDescription: text("og_description"),- ogImage: text("og_image"),- // Headings- h1Count: integer("h1_count").notNull().default(0),- h2Count: integer("h2_count").notNull().default(0),- h3Count: integer("h3_count").notNull().default(0),- h4Count: integer("h4_count").notNull().default(0),- h5Count: integer("h5_count").notNull().default(0),- h6Count: integer("h6_count").notNull().default(0),- headingOrderJson: text("heading_order_json"),- // Content- wordCount: integer("word_count").notNull().default(0),- // Images- imagesTotal: integer("images_total").notNull().default(0),- imagesMissingAlt: integer("images_missing_alt").notNull().default(0),- imagesJson: text("images_json"),- internalLinkCount: integer("internal_link_count").notNull().default(0),- externalLinkCount: integer("external_link_count").notNull().default(0),- hasStructuredData: boolean("has_structured_data").notNull().default(false),- hreflangTagsJson: text("hreflang_tags_json"),- isIndexable: boolean("is_indexable").notNull().default(true),- responseTimeMs: integer("response_time_ms"),- },- (table) => [index("audit_pages_audit_id_idx").on(table.auditId)],-);--// One row per Lighthouse test (mobile + desktop per page).-export const auditLighthouseResults = pgTable(- "audit_lighthouse_results",- {- id: text("id").primaryKey(),- auditId: text("audit_id")- .notNull()- .references(() => audits.id, { onDelete: "cascade" }),- pageId: text("page_id")- .notNull()- .references(() => auditPages.id, { onDelete: "cascade" }),- strategy: text("strategy", { enum: ["mobile", "desktop"] }).notNull(),- performanceScore: integer("performance_score"),- accessibilityScore: integer("accessibility_score"),- bestPracticesScore: integer("best_practices_score"),- seoScore: integer("seo_score"),- lcpMs: real("lcp_ms"),- cls: real("cls"),- inpMs: real("inp_ms"),- ttfbMs: real("ttfb_ms"),- errorMessage: text("error_message"),- r2Key: text("r2_key"),- payloadSizeBytes: integer("payload_size_bytes"),- },- (table) => [index("audit_lighthouse_results_audit_id_idx").on(table.auditId)],-);diff --git a/src/db/pg/audit.schema.ts b/src/db/pg/audit.schema.tsnew file mode 100644index 00000000..5f57113c--- /dev/null+++ b/src/db/pg/audit.schema.ts@@ -0,0 +1,196 @@+import { sql } from "drizzle-orm";+import {+ boolean,+ index,+ integer,+ pgTable,+ real,+ text,+} from "drizzle-orm/pg-core";+import { projects } from "./app.schema";++// Timestamps are stored as *text* (same column shape as the SQLite schema); see+// the note in pg/app.schema.ts. `isoNow` matches `new Date().toISOString()` so+// DB-defaulted and app-written values sort together lexicographically.+const isoNow = sql`to_char(now() AT TIME ZONE 'utc', 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"')`;+const timestampColumn = (name: string) => text(name);++// ============================================================================+// Site Audit tables+// ============================================================================++// One row per audit run+export const audits = pgTable(+ "audits",+ {+ id: text("id").primaryKey(),+ projectId: text("project_id")+ .notNull()+ .references(() => projects.id, { onDelete: "cascade" }),+ startedByUserId: text("started_by_user_id").notNull(),+ startUrl: text("start_url").notNull(),+ status: text("status", {+ enum: ["running", "completed", "failed"],+ })+ .notNull()+ .default("running"),+ workflowInstanceId: text("workflow_instance_id"),+ // JSON config: { maxPages, lighthouseStrategy }+ config: text("config").notNull().default("{}"),+ // Progress & summary+ pagesCrawled: integer("pages_crawled").notNull().default(0),+ pagesTotal: integer("pages_total").notNull().default(0),+ lighthouseTotal: integer("lighthouse_total").notNull().default(0),+ lighthouseCompleted: integer("lighthouse_completed").notNull().default(0),+ lighthouseFailed: integer("lighthouse_failed").notNull().default(0),+ currentPhase: text("current_phase").default("discovery"),+ startedAt: timestampColumn("started_at").notNull().default(isoNow),+ completedAt: timestampColumn("completed_at"),+ },+ (table) => [+ index("audits_project_id_idx").on(table.projectId),+ index("audits_started_by_user_id_idx").on(table.startedByUserId),+ ],+);++// One row per crawled page+export const auditPages = pgTable(+ "audit_pages",+ {+ id: text("id").primaryKey(),+ auditId: text("audit_id")+ .notNull()+ .references(() => audits.id, { onDelete: "cascade" }),+ url: text("url").notNull(),+ statusCode: integer("status_code"),+ redirectUrl: text("redirect_url"),+ // Metadata+ title: text("title"),+ metaDescription: text("meta_description"),+ canonicalUrl: text("canonical_url"),+ robotsMeta: text("robots_meta"),+ // Open Graph+ ogTitle: text("og_title"),+ ogDescription: text("og_description"),+ ogImage: text("og_image"),+ // Headings+ h1Count: integer("h1_count").notNull().default(0),+ h2Count: integer("h2_count").notNull().default(0),+ h3Count: integer("h3_count").notNull().default(0),+ h4Count: integer("h4_count").notNull().default(0),+ h5Count: integer("h5_count").notNull().default(0),+ h6Count: integer("h6_count").notNull().default(0),+ headingOrderJson: text("heading_order_json"),+ // Content+ wordCount: integer("word_count").notNull().default(0),+ // Images+ imagesTotal: integer("images_total").notNull().default(0),+ imagesMissingAlt: integer("images_missing_alt").notNull().default(0),+ imagesJson: text("images_json"),+ // Links+ internalLinkCount: integer("internal_link_count").notNull().default(0),+ externalLinkCount: integer("external_link_count").notNull().default(0),+ // Structured data+ hasStructuredData: boolean("has_structured_data").notNull().default(false),+ // Hreflang+ hreflangTagsJson: text("hreflang_tags_json"),+ // Indexability+ isIndexable: boolean("is_indexable").notNull().default(true),+ // Indexability/canonical signals from response headers+ xRobotsTag: text("x_robots_tag"),+ headerCanonicalUrl: text("header_canonical_url"),+ // Crawl metadata+ // null depth = not reached via links (e.g. sitemap-seeded)+ crawlDepth: integer("crawl_depth"),+ inSitemap: boolean("in_sitemap").notNull().default(false),+ // SHA-256 of the visible body text, for duplicate-content grouping+ contentHash: text("content_hash"),+ // How the fetch resolved: ok | blocked (WAF/bot challenge) | error+ fetchClass: text("fetch_class", { enum: ["ok", "blocked", "error"] })+ .notNull()+ .default("ok"),+ // Performance+ responseTimeMs: integer("response_time_ms"),+ },+ (table) => [+ index("audit_pages_audit_id_idx").on(table.auditId),+ index("audit_pages_audit_url_idx").on(table.auditId, table.url),+ ],+);++// One row per unique (source page, target URL) link edge. Currently only+// internal edges are stored (see AuditRepository); isInternal stays so+// external-link checks can start writing rows without a migration.+export const auditLinks = pgTable(+ "audit_links",+ {+ id: text("id").primaryKey(),+ auditId: text("audit_id")+ .notNull()+ .references(() => audits.id, { onDelete: "cascade" }),+ sourcePageId: text("source_page_id")+ .notNull()+ .references(() => auditPages.id, { onDelete: "cascade" }),+ sourceUrl: text("source_url").notNull(),+ targetUrl: text("target_url").notNull(),+ anchor: text("anchor"),+ isInternal: boolean("is_internal").notNull().default(true),+ isNofollow: boolean("is_nofollow").notNull().default(false),+ },+ (table) => [+ index("audit_links_audit_id_idx").on(table.auditId),+ index("audit_links_audit_target_idx").on(table.auditId, table.targetUrl),+ ],+);++// One row per (issue type, affected page)+export const auditIssues = pgTable(+ "audit_issues",+ {+ id: text("id").primaryKey(),+ auditId: text("audit_id")+ .notNull()+ .references(() => audits.id, { onDelete: "cascade" }),+ pageId: text("page_id").references(() => auditPages.id, {+ onDelete: "cascade",+ }),+ pageUrl: text("page_url").notNull(),+ issueType: text("issue_type").notNull(),+ severity: text("severity", { enum: ["critical", "warning", "info"] })+ .notNull()+ .default("info"),+ // JSON details specific to the issue type (e.g. broken link target)+ detailsJson: text("details_json"),+ },+ (table) => [+ index("audit_issues_audit_id_idx").on(table.auditId),+ index("audit_issues_audit_type_idx").on(table.auditId, table.issueType),+ ],+);++// One row per Lighthouse test (mobile + desktop per page).+export const auditLighthouseResults = pgTable(+ "audit_lighthouse_results",+ {+ id: text("id").primaryKey(),+ auditId: text("audit_id")+ .notNull()+ .references(() => audits.id, { onDelete: "cascade" }),+ pageId: text("page_id")+ .notNull()+ .references(() => auditPages.id, { onDelete: "cascade" }),+ strategy: text("strategy", { enum: ["mobile", "desktop"] }).notNull(),+ performanceScore: integer("performance_score"),+ accessibilityScore: integer("accessibility_score"),+ bestPracticesScore: integer("best_practices_score"),+ seoScore: integer("seo_score"),+ lcpMs: real("lcp_ms"),+ cls: real("cls"),+ inpMs: real("inp_ms"),+ ttfbMs: real("ttfb_ms"),+ errorMessage: text("error_message"),+ r2Key: text("r2_key"),+ payloadSizeBytes: integer("payload_size_bytes"),+ },+ (table) => [index("audit_lighthouse_results_audit_id_idx").on(table.auditId)],+);diff --git a/src/db/pg/schema.ts b/src/db/pg/schema.tsindex 9d03f35f..db15b8c8 100644--- a/src/db/pg/schema.ts+++ b/src/db/pg/schema.ts@@ -1,4 +1,5 @@export * from "./app.schema";+export * from "./audit.schema";export * from "./sam.schema";export * from "./better-auth-schema";export * from "./billing.schema";diff --git a/src/db/schema.ts b/src/db/schema.tsindex 75f40d33..2e21d2d4 100644--- a/src/db/schema.ts+++ b/src/db/schema.ts@@ -1,11 +1,13 @@import { getDatabaseProvider } from "./provider";import * as sqliteApp from "./app.schema";+import * as sqliteAudit from "./audit.schema";import * as sqliteSam from "./sam.schema";import * as sqliteAuth from "./better-auth-schema";import * as sqliteBilling from "./billing.schema";import * as sqliteGsc from "./gsc.schema";import * as sqliteReddit from "./reddit-attribution.schema";import * as pgApp from "./pg/app.schema";+import * as pgAudit from "./pg/audit.schema";import * as pgSam from "./pg/sam.schema";import * as pgAuth from "./pg/better-auth-schema";import * as pgBilling from "./pg/billing.schema";@@ -23,6 +25,7 @@ import * as pgReddit from "./pg/reddit-attribution.schema";// schema is the one structural artifact NOT regenerated by `db:generate`, so the// parity test is its drift guard.type AppSchema = typeof sqliteApp &+ typeof sqliteAudit &typeof sqliteSam &typeof sqliteAuth &typeof sqliteBilling &@@ -31,9 +34,18 @@ type AppSchema = typeof sqliteApp &const runtimeSchema =getDatabaseProvider() === "postgres"- ? { ...pgApp, ...pgSam, ...pgAuth, ...pgBilling, ...pgGsc, ...pgReddit }+ ? {+ ...pgApp,+ ...pgAudit,+ ...pgSam,+ ...pgAuth,+ ...pgBilling,+ ...pgGsc,+ ...pgReddit,+ }: {...sqliteApp,+ ...sqliteAudit,...sqliteSam,...sqliteAuth,...sqliteBilling,@@ -57,6 +69,8 @@ export const {rankSnapshots,audits,auditPages,+ auditLinks,+ auditIssues,auditLighthouseResults,samSessions,samProjectMemory,diff --git a/src/server/features/audit/repositories/AuditRepository.ts b/src/server/features/audit/repositories/AuditRepository.tsindex a01a0751..9fa0ad0f 100644--- a/src/server/features/audit/repositories/AuditRepository.ts+++ b/src/server/features/audit/repositories/AuditRepository.ts@@ -1,17 +1,34 @@/*** Data access layer for site audit tables.- * Provider-aware (D1 or Postgres) via the `@/db` handle.+ * Provider-aware (D1 or Postgres) via the `@/db` handle. Covers audits,+ * audit_pages, audit_links, audit_issues, and stored Lighthouse results.*/import { and, desc, eq } from "drizzle-orm";import { db } from "@/db";-import { audits, auditLighthouseResults, auditPages } from "@/db/schema";+import {+ audits,+ auditIssues,+ auditLighthouseResults,+ auditLinks,+ auditPages,+} from "@/db/schema";import { executeInBatches } from "@/db/runBatch";+import { AUDIT_ISSUE_TYPES } from "@/shared/audit-issues";+import { deterministicAuditRowId } from "@/server/lib/audit/ids";+import type { DetectedIssue } from "@/server/lib/audit/issues/page-reporters";import type {AuditConfig,+ CrawledPageResult,LighthouseResult,- StepPageResult,} from "@/server/lib/audit/types";+// Only internal links are stored: both consumers (broken-internal-link and+// orphan checks) filter on isInternal, and per-page external counts already+// live on audit_pages. External rows come back when P1 adds external-link+// checks. Mega-menu/footer-heavy sites can carry 1000+ links per page; cap+// what we store so a 10k-page crawl can't write tens of millions of link rows.+const MAX_STORED_LINKS_PER_PAGE = 500;+async function createAudit(data: {id: string;projectId: string;@@ -84,6 +101,9 @@ async function completeAudit(}async function failAudit(auditId: string, workflowInstanceId: string) {+ // Only a running audit can transition to failed: the getStatus reconciler+ // races the workflow's own finalize, and without this guard it could flip+ // a just-completed audit to failed.await db.update(audits).set({@@ -95,6 +115,7 @@ async function failAudit(auditId: string, workflowInstanceId: string) {and(eq(audits.id, auditId),eq(audits.workflowInstanceId, workflowInstanceId),+ eq(audits.status, "running"),),);}@@ -111,25 +132,24 @@ async function getAuditForWorkflow(});}-async function batchWriteResults(+/**+ * Persist one crawl batch (pages + link edges + per-page issues).+ * Called inside the crawl-batch Workflow step so results land in D1+ * incrementally instead of accumulating in memory until finalize.+ *+ * Idempotent on step retry: callers assign deterministic page ids+ * (deterministicAuditRowId) and link/issue ids are derived from stable+ * content. Page rows upsert (a retried fetch may legitimately differ — last+ * attempt wins, matching what the step returns); links and issues are+ * insert-or-ignore.+ */+async function insertCrawledBatch(auditId: string,- pages: StepPageResult[],- lighthouseResults: LighthouseResult[],+ pages: CrawledPageResult[],+ issues: DetectedIssue[],) {- // The `finalize` workflow step can retry after a partial write (multi-chunk- // inserts aren't atomic, and steps after the insert can throw). Clear any- // rows from a prior attempt first so the re-run is idempotent — otherwise- // stable page ids collide on the PK and lighthouse rows silently duplicate.- // audit_lighthouse_results.page_id FKs audit_pages, so delete it first.- await db- .delete(auditLighthouseResults)- .where(eq(auditLighthouseResults.auditId, auditId));- await db.delete(auditPages).where(eq(auditPages.auditId, auditId));-- await executeInBatches(pages, (tx, page) =>- tx.insert(auditPages).values({- id: page.id,- auditId,+ await executeInBatches(pages, (tx, page) => {+ const dataColumns = {url: page.url,statusCode: page.statusCode,redirectUrl: page.redirectUrl,@@ -137,6 +157,8 @@ async function batchWriteResults(metaDescription: page.metaDescription,canonicalUrl: page.canonicalUrl,robotsMeta: page.robotsMeta,+ xRobotsTag: page.xRobotsTag,+ headerCanonicalUrl: page.headerCanonicalUrl,ogTitle: page.ogTitle,ogDescription: page.ogDescription,ogImage: page.ogImage,@@ -148,25 +170,83 @@ async function batchWriteResults(h6Count: page.h6Count,headingOrderJson: JSON.stringify(page.headingOrder),wordCount: page.wordCount,+ contentHash: page.contentHash,imagesTotal: page.imagesTotal,imagesMissingAlt: page.imagesMissingAlt,imagesJson: JSON.stringify(page.images),- internalLinkCount: page.internalLinks.length,- externalLinkCount: page.externalLinks.length,+ internalLinkCount: page.links.filter((l) => l.isInternal).length,+ externalLinkCount: page.links.filter((l) => !l.isInternal).length,hasStructuredData: page.hasStructuredData,hreflangTagsJson: JSON.stringify(page.hreflangTags),isIndexable: page.isIndexable,+ fetchClass: page.fetchClass,+ crawlDepth: page.crawlDepth,+ inSitemap: page.inSitemap,responseTimeMs: page.responseTimeMs,- }),+ };+ return tx+ .insert(auditPages)+ .values({ id: page.id, auditId, ...dataColumns })+ .onConflictDoUpdate({ target: auditPages.id, set: dataColumns });+ });++ const linkRows = await Promise.all(+ pages.flatMap((page) =>+ page.links+ .filter((link) => link.isInternal)+ .slice(0, MAX_STORED_LINKS_PER_PAGE)+ .map(async (link) => ({+ id: await deterministicAuditRowId(auditId, page.url, link.targetUrl),+ auditId,+ sourcePageId: page.id,+ sourceUrl: page.url,+ targetUrl: link.targetUrl,+ anchor: link.anchor,+ isInternal: link.isInternal,+ isNofollow: link.isNofollow,+ })),+ ),+ );+ await executeInBatches(linkRows, (tx, row) =>+ tx.insert(auditLinks).values(row).onConflictDoNothing(),);- if (lighthouseResults.length === 0) {- return;- }+ await insertIssues(auditId, issues);+}- await executeInBatches(lighthouseResults, (tx, result) =>- tx.insert(auditLighthouseResults).values({- id: crypto.randomUUID(),+async function insertIssues(auditId: string, issues: DetectedIssue[]) {+ const issueRows = await Promise.all(+ issues.map(async (issue) => ({+ id: await deterministicAuditRowId(+ auditId,+ issue.pageUrl,+ issue.issueType,+ issue.dedupeKey ?? "",+ ),+ auditId,+ pageId: issue.pageId,+ pageUrl: issue.pageUrl,+ issueType: issue.issueType,+ severity: AUDIT_ISSUE_TYPES[issue.issueType].severity,+ detailsJson: issue.details ? JSON.stringify(issue.details) : null,+ })),+ );+ await executeInBatches(issueRows, (tx, row) =>+ tx.insert(auditIssues).values(row).onConflictDoNothing(),+ );+}++async function insertLighthouseResults(+ auditId: string,+ lighthouseResults: LighthouseResult[],+) {+ const rows = await Promise.all(+ lighthouseResults.map(async (result) => ({+ id: await deterministicAuditRowId(+ auditId,+ result.pageId,+ result.strategy,+ ),auditId,pageId: result.pageId,strategy: result.strategy,@@ -181,8 +261,17 @@ async function batchWriteResults(errorMessage: result.errorMessage ?? null,r2Key: result.r2Key ?? null,payloadSizeBytes: result.payloadSizeBytes ?? null,- }),+ })),);+ // Upsert: a step retry can charge a second DataForSEO call whose result+ // must not be silently dropped in favor of a failed first attempt.+ await executeInBatches(rows, (tx, row) => {+ const { id: _id, auditId: _auditId, ...dataColumns } = row;+ return tx.insert(auditLighthouseResults).values(row).onConflictDoUpdate({+ target: auditLighthouseResults.id,+ set: dataColumns,+ });+ });}async function getAuditForProject(auditId: string, projectId: string) {@@ -191,6 +280,58 @@ async function getAuditForProject(auditId: string, projectId: string) {});}+async function getLatestAuditForProject(projectId: string) {+ return db.query.audits.findFirst({+ where: eq(audits.projectId, projectId),+ orderBy: desc(audits.startedAt),+ });+}++async function getIssuesForAudit(+ auditId: string,+ filters: { severity?: "critical" | "warning" | "info"; issueType?: string },+) {+ return db.query.auditIssues.findMany({+ where: and(+ eq(auditIssues.auditId, auditId),+ filters.severity ? eq(auditIssues.severity, filters.severity) : undefined,+ filters.issueType+ ? eq(auditIssues.issueType, filters.issueType)+ : undefined,+ ),+ });+}++async function getPagesForAudit(auditId: string) {+ return db+ .select({+ id: auditPages.id,+ url: auditPages.url,+ statusCode: auditPages.statusCode,+ fetchClass: auditPages.fetchClass,+ redirectUrl: auditPages.redirectUrl,+ title: auditPages.title,+ metaDescription: auditPages.metaDescription,+ wordCount: auditPages.wordCount,+ isIndexable: auditPages.isIndexable,+ crawlDepth: auditPages.crawlDepth,+ inSitemap: auditPages.inSitemap,+ internalLinkCount: auditPages.internalLinkCount,+ responseTimeMs: auditPages.responseTimeMs,+ })+ .from(auditPages)+ .where(eq(auditPages.auditId, auditId));+}++async function hasPagesForAudit(auditId: string): Promise<boolean> {+ const rows = await db+ .select({ id: auditPages.id })+ .from(auditPages)+ .where(eq(auditPages.auditId, auditId))+ .limit(1);+ return rows.length > 0;+}+async function getAuditsByProject(projectId: string) {const rows = await db.select({ audit: audits })@@ -223,19 +364,22 @@ async function getAuditUsageForUser(userId: string) {async function getAuditResultsForProject(auditId: string, projectId: string) {const audit = await getAuditForProject(auditId, projectId);if (!audit) {- return { audit: null, pages: [], lighthouse: [] };+ return { audit: null, pages: [], lighthouse: [], issues: [] };}- const [pages, lighthouse] = await Promise.all([+ const [pages, lighthouse, issues] = await Promise.all([db.query.auditPages.findMany({where: eq(auditPages.auditId, auditId),}),db.query.auditLighthouseResults.findMany({where: eq(auditLighthouseResults.auditId, auditId),}),+ db.query.auditIssues.findMany({+ where: eq(auditIssues.auditId, auditId),+ }),]);- return { audit, pages, lighthouse };+ return { audit, pages, lighthouse, issues };}async function getLighthouseResultById(input: {@@ -285,8 +429,14 @@ export const AuditRepository = {completeAudit,failAudit,getAuditForWorkflow,- batchWriteResults,+ insertCrawledBatch,+ insertIssues,+ insertLighthouseResults,getAuditForProject,+ getLatestAuditForProject,+ getIssuesForAudit,+ getPagesForAudit,+ hasPagesForAudit,getAuditsByProject,getAuditUsageForUser,getAuditResultsForProject,diff --git a/src/server/features/audit/services/AuditService.ts b/src/server/features/audit/services/AuditService.tsindex d0ff340b..f36146c9 100644--- a/src/server/features/audit/services/AuditService.ts+++ b/src/server/features/audit/services/AuditService.ts@@ -1,5 +1,9 @@import { env } from "cloudflare:workers";-import type { BillingCustomerContext } from "@/server/billing/subscription";+import {+ customerHasManagedAccess,+ customerHasPaidPlan,+ type BillingCustomerContext,+} from "@/server/billing/subscription";import { AuditRepository } from "@/server/features/audit/repositories/AuditRepository";import {AUDIT_LIMITS,@@ -15,6 +19,24 @@ import {type LighthouseStrategy,} from "@/server/lib/audit/types";import { normalizeAndValidateStartUrl } from "@/server/lib/audit/url-policy";+import { isHostedServerAuthMode } from "@/server/lib/runtime-env";++// Plan-tier limits are the abuse bound in hosted mode: free accounts get one+// small audit at a time, paid keeps the full limits, and customers with no+// Autumn product at all are turned away. Self-hosted isn't gated.+async function resolveAuditLimitTier(+ organizationId: string,+): Promise<AuditLimitTier> {+ if (!(await isHostedServerAuthMode())) return "paid";+ const [hasManagedAccess, hasPaidPlan] = await Promise.all([+ customerHasManagedAccess(organizationId),+ customerHasPaidPlan(organizationId),+ ]);+ if (!hasManagedAccess) {+ throw new AppError("PAYMENT_REQUIRED", "Subscribe to run site audits");+ }+ return hasPaidPlan ? "paid" : "free";+}async function startAudit(input: {actorUserId: string;@@ -98,8 +120,29 @@ async function startAudit(input: {}async function getStatus(auditId: string, projectId: string) {- const audit = await AuditRepository.getAuditForProject(auditId, projectId);- if (!audit) throw new AppError("NOT_FOUND");+ let audit = await AuditRepository.getAuditForProject(auditId, projectId);+ if (!audit)+ throw new AppError("NOT_FOUND", "Audit not found in this project.");++ // Self-heal audits whose workflow died without reaching the mark-failed+ // step (instance terminated, mark-failed itself failed, deploys, ...).+ // Without this they stay "running" forever and hold capacity.+ if (audit.status === "running" && audit.workflowInstanceId) {+ try {+ const instance = await env.SITE_AUDIT_WORKFLOW.get(+ audit.workflowInstanceId,+ );+ const { status } = await instance.status();+ if (status === "errored" || status === "terminated") {+ await AuditRepository.failAudit(audit.id, audit.workflowInstanceId);+ audit =+ (await AuditRepository.getAuditForProject(auditId, projectId)) ??+ audit;+ }+ } catch {+ // Instance not found or status unavailable — leave the audit as-is.+ }+ }return {id: audit.id,@@ -117,7 +160,7 @@ async function getStatus(auditId: string, projectId: string) {}async function getResults(auditId: string, projectId: string) {- const { audit, pages, lighthouse } =+ const { audit, pages, lighthouse, issues } =await AuditRepository.getAuditResultsForProject(auditId, projectId);if (!audit) throw new AppError("NOT_FOUND");@@ -140,6 +183,7 @@ async function getResults(auditId: string, projectId: string) {},pages,lighthouse,+ issues,};}@@ -216,6 +260,7 @@ async function remove(auditId: string, projectId: string) {}export const AuditService = {+ resolveAuditLimitTier,startAudit,getStatus,getCrawlProgress,diff --git a/src/server/features/audit/services/audit-capacity.test.ts b/src/server/features/audit/services/audit-capacity.test.tsindex adc64327..238991c7 100644--- a/src/server/features/audit/services/audit-capacity.test.ts+++ b/src/server/features/audit/services/audit-capacity.test.ts@@ -21,16 +21,6 @@ describe("audit capacity helpers", () => {lighthouseTotal: 0,total: 100,});- expect(- getEstimatedAuditCapacity({- maxPages: 100,- lighthouseStrategy: "manual",- }),- ).toEqual({- pagesTotal: 100,- lighthouseTotal: 0,- total: 100,- });expect(getEstimatedAuditCapacity({ maxPages: 100, lighthouseStrategy: "auto" }),).toEqual({diff --git a/src/server/lib/audit/discovery.ts b/src/server/lib/audit/discovery.tsindex f37b0a44..68bd387b 100644--- a/src/server/lib/audit/discovery.ts+++ b/src/server/lib/audit/discovery.ts@@ -22,39 +22,39 @@ export interface RobotsResult {}/**- * Fetch and parse robots.txt for a given origin.- * Returns a helper to check if URLs are allowed + discovered sitemap URLs.+ * Fetch the raw robots.txt body (null = missing/unreachable). Kept separate+ * from parsing so Workflows can checkpoint the text as durable step state and+ * re-derive the parsed result deterministically on replay.*/-export async function fetchRobotsTxt(origin: string): Promise<RobotsResult> {- const robotsUrl = `${origin}/robots.txt`;+async function fetchRobotsTxtText(origin: string): Promise<string | null> {try {- const response = await fetch(robotsUrl, {+ const response = await fetch(`${origin}/robots.txt`, {headers: { "User-Agent": "OpenSEO-Audit/1.0" },signal: AbortSignal.timeout(10_000),});- if (!response.ok) {- // No robots.txt = everything allowed- return {- isAllowed: () => true,- sitemapUrls: [],- };- }-- const text = await response.text();- const robots = robotsParser(robotsUrl, text);-- return {- isAllowed: (url: string) => robots.isAllowed(url) ?? true,- sitemapUrls: robots.getSitemaps(),- };+ if (!response.ok) return null;+ return await response.text();} catch (error) {console.warn("Failed to fetch robots.txt:", error);- return {- isAllowed: () => true,- sitemapUrls: [],- };+ return null;+ }+}++/** Deterministic: same text in, same result out. Null = everything allowed. */+export function parseRobotsTxt(+ origin: string,+ text: string | null,+): RobotsResult {+ if (text === null) {+ return { isAllowed: () => true, sitemapUrls: [] };}++ const robots = robotsParser(`${origin}/robots.txt`, text);+ return {+ isAllowed: (url: string) => robots.isAllowed(url) ?? true,+ sitemapUrls: robots.getSitemaps(),+ };}/**@@ -184,8 +184,9 @@ async function fetchSitemapDocumentWithRetry(sitemapUrl: string): Promise<{export async function discoverUrls(origin: string,maxPages = 50,-): Promise<{ urls: string[]; robots: RobotsResult; sitemapUrls: Set<string> }> {- const robots = await fetchRobotsTxt(origin);+): Promise<{ urls: string[]; robotsText: string | null }> {+ const robotsText = await fetchRobotsTxtText(origin);+ const robots = parseRobotsTxt(origin, robotsText);// Collect sitemap URLs: from robots.txt + default locationconst sitemapSources = new Set(robots.sitemapUrls);@@ -262,9 +263,10 @@ export async function discoverUrls();}+ // Cap at the crawl's page budget: these are seeds, the crawl can never use+ // more — and an uncapped list can blow the ~1MiB Workflow step-state limit.return {- urls: Array.from(allUrls),- robots,- sitemapUrls: allUrls,+ urls: Array.from(allUrls).slice(0, maxPages),+ robotsText,};}diff --git a/src/server/lib/audit/ids.ts b/src/server/lib/audit/ids.tsnew file mode 100644index 00000000..6dc30f27--- /dev/null+++ b/src/server/lib/audit/ids.ts@@ -0,0 +1,24 @@+/**+ * Deterministic row ids for audit data.+ *+ * Crawl/lighthouse persistence happens inside Workflow steps, which retry on+ * failure. Deriving ids from stable content (audit id + URL + ...) combined+ * with `onConflictDoNothing` makes those writes idempotent across retries —+ * a partially-written batch is simply completed on the next attempt instead+ * of duplicated under fresh random ids.+ */+export async function sha256Hex(text: string): Promise<string> {+ const digest = await crypto.subtle.digest(+ "SHA-256",+ new TextEncoder().encode(text),+ );+ return Array.from(new Uint8Array(digest))+ .map((byte) => byte.toString(16).padStart(2, "0"))+ .join("");+}++export async function deterministicAuditRowId(+ ...parts: string[]+): Promise<string> {+ return (await sha256Hex(parts.join("|"))).slice(0, 36);+}diff --git a/src/server/lib/audit/issues/multipage-checks.ts b/src/server/lib/audit/issues/multipage-checks.tsnew file mode 100644index 00000000..5858db5e--- /dev/null+++ b/src/server/lib/audit/issues/multipage-checks.ts@@ -0,0 +1,180 @@+/**+ * Pure cross-page checks (no database access): duplicate grouping and+ * redirect chain/loop detection. The D1-backed checks (broken links,+ * orphans) live in multipage.ts.+ */+import type { DetectedIssue } from "@/server/lib/audit/issues/page-reporters";++const DUPLICATE_GROUP_SAMPLE = 3;++export interface SlimPage {+ id: string;+ url: string;+ statusCode: number | null;+ fetchClass: "ok" | "blocked" | "error";+ title: string | null;+ metaDescription: string | null;+ contentHash: string | null;+ redirectUrl: string | null;+ wordCount: number;+ isIndexable: boolean;+ canonicalUrl: string | null;+ headerCanonicalUrl: string | null;+}++function isOkHtmlPage(page: SlimPage): boolean {+ return (+ page.fetchClass === "ok" &&+ page.statusCode !== null &&+ page.statusCode >= 200 &&+ page.statusCode < 300+ );+}++/**+ * Pages the owner already de-duplicated (noindex, or canonicalized to+ * another URL) don't belong in duplicate groups — flagging them tells the+ * user to fix something they already fixed.+ */+function isDuplicateCandidate(page: SlimPage): boolean {+ if (!isOkHtmlPage(page) || !page.isIndexable) return false;+ const effectiveCanonical = page.canonicalUrl ?? page.headerCanonicalUrl;+ return !effectiveCanonical || effectiveCanonical === page.url;+}++export function findDuplicates(pages: SlimPage[]): DetectedIssue[] {+ const okPages = pages.filter(isDuplicateCandidate);++ const groupBy = (+ keyOf: (page: SlimPage) => string | null,+ ): Map<string, SlimPage[]> => {+ const groups = new Map<string, SlimPage[]>();+ for (const page of okPages) {+ const key = keyOf(page);+ if (!key) continue;+ const group = groups.get(key);+ if (group) group.push(page);+ else groups.set(key, [page]);+ }+ return groups;+ };++ const issues: DetectedIssue[] = [];+ const emitGroups = (+ groups: Map<string, SlimPage[]>,+ issueType: DetectedIssue["issueType"],+ ) => {+ for (const group of groups.values()) {+ if (group.length < 2) continue;+ for (const page of group) {+ issues.push({+ issueType,+ pageId: page.id,+ pageUrl: page.url,+ details: {+ groupSize: group.length,+ otherUrls: group+ .filter((other) => other.id !== page.id)+ .slice(0, DUPLICATE_GROUP_SAMPLE)+ .map((other) => other.url),+ },+ });+ }+ }+ };++ emitGroups(+ groupBy((page) => page.title || null),+ "duplicate-title",+ );+ emitGroups(+ groupBy((page) => page.metaDescription || null),+ "duplicate-meta-description",+ );+ emitGroups(+ groupBy((page) => (page.wordCount > 0 ? page.contentHash : null)),+ "duplicate-content",+ );+ return issues;+}++export function findRedirectChainsAndLoops(pages: SlimPage[]): DetectedIssue[] {+ const redirects = new Map<string, SlimPage>();+ for (const page of pages) {+ const isRedirect =+ page.statusCode !== null &&+ page.statusCode >= 300 &&+ page.statusCode < 400 &&+ page.redirectUrl;+ if (isRedirect) redirects.set(page.url, page);+ }++ const redirectTargets = new Set(+ Array.from(redirects.values(), (page) => page.redirectUrl!),+ );++ const issues: DetectedIssue[] = [];+ const walked = new Set<string>();++ // Walk from chain heads (redirects nothing else redirects to), so a 5-hop+ // chain yields one issue, not five.+ for (const [url, head] of redirects) {+ if (redirectTargets.has(url)) continue;++ const hops: string[] = [url];+ const seen = new Set(hops);+ walked.add(url);+ let current = head.redirectUrl;+ let isLoop = false;+ while (current) {+ if (seen.has(current)) {+ isLoop = true;+ hops.push(current);+ break;+ }+ hops.push(current);+ seen.add(current);+ if (redirects.has(current)) walked.add(current);+ current = redirects.get(current)?.redirectUrl ?? null;+ }++ if (isLoop) {+ issues.push({+ issueType: "redirect-loop",+ pageId: head.id,+ pageUrl: url,+ details: { hops },+ });+ } else if (hops.length > 2) {+ // url -> a -> b: two redirects before content = a chain+ issues.push({+ issueType: "redirect-chain",+ pageId: head.id,+ pageUrl: url,+ details: { hops, finalUrl: hops[hops.length - 1] },+ });+ }+ }++ // Headless cycles (every member is also a target — e.g. a↔b, or a→a) are+ // never reached from a head; emit one loop issue per cycle.+ for (const [url, page] of redirects) {+ if (walked.has(url)) continue;++ const cycle: string[] = [];+ let current: string | null = url;+ while (current && !walked.has(current)) {+ walked.add(current);+ cycle.push(current);+ current = redirects.get(current)?.redirectUrl ?? null;+ }+ issues.push({+ issueType: "redirect-loop",+ pageId: page.id,+ pageUrl: url,+ details: { hops: [...cycle, url] },+ });+ }++ return issues;+}diff --git a/src/server/lib/audit/issues/multipage.ts b/src/server/lib/audit/issues/multipage.tsnew file mode 100644index 00000000..b089e50c--- /dev/null+++ b/src/server/lib/audit/issues/multipage.ts@@ -0,0 +1,154 @@+/**+ * Cross-page (multipage) issue checks.+ *+ * These run once after the crawl, against the rows the crawl wrote to D1:+ * duplicates, broken internal links, redirect chains/loops, orphan pages.+ * Pure set-queries over crawl data — no fetching, no DOM.+ */+import { and, eq, gte, lt, ne, notExists, sql } from "drizzle-orm";+import { alias } from "drizzle-orm/sqlite-core";+import { db } from "@/db";+import { auditLinks, auditPages } from "@/db/schema";+import { normalizeUrl } from "@/server/lib/audit/url-utils";+import {+ findDuplicates,+ findRedirectChainsAndLoops,+ type SlimPage,+} from "@/server/lib/audit/issues/multipage-checks";+import type { DetectedIssue } from "@/server/lib/audit/issues/page-reporters";++const BROKEN_LINK_ISSUE_CAP = 2_000;++export async function runMultipageChecks(input: {+ auditId: string;+ startUrl: string;+ /** Orphan detection only makes sense when the crawl wasn't truncated. */+ crawlCompleted: boolean;+}): Promise<DetectedIssue[]> {+ const pages: SlimPage[] = await db+ .select({+ id: auditPages.id,+ url: auditPages.url,+ statusCode: auditPages.statusCode,+ fetchClass: auditPages.fetchClass,+ title: auditPages.title,+ metaDescription: auditPages.metaDescription,+ contentHash: auditPages.contentHash,+ redirectUrl: auditPages.redirectUrl,+ wordCount: auditPages.wordCount,+ isIndexable: auditPages.isIndexable,+ canonicalUrl: auditPages.canonicalUrl,+ headerCanonicalUrl: auditPages.headerCanonicalUrl,+ })+ .from(auditPages)+ .where(eq(auditPages.auditId, input.auditId));++ const issues: DetectedIssue[] = [+ ...findDuplicates(pages),+ ...findRedirectChainsAndLoops(pages),+ ...(await findBrokenInternalLinks(input.auditId)),+ ];++ if (input.crawlCompleted) {+ // Page rows store normalized URLs; normalize the start URL the same way+ // so the orphan exclusion matches.+ const normalizedStart = normalizeUrl(input.startUrl) ?? input.startUrl;+ issues.push(...(await findOrphanPages(input.auditId, normalizedStart)));+ }++ return issues;+}++async function findBrokenInternalLinks(+ auditId: string,+): Promise<DetectedIssue[]> {+ // Only flag targets we actually crawled and saw fail — never inferred from+ // absence. Blocked targets (WAF challenges) are excluded: a 403 from bot+ // protection is not evidence of a broken link.+ const rows = await db+ .select({+ sourcePageId: auditLinks.sourcePageId,+ sourceUrl: auditLinks.sourceUrl,+ targetUrl: auditLinks.targetUrl,+ targetStatus: auditPages.statusCode,+ })+ .from(auditLinks)+ .innerJoin(+ auditPages,+ and(+ eq(auditPages.auditId, auditLinks.auditId),+ eq(auditPages.url, auditLinks.targetUrl),+ ),+ )+ .where(+ and(+ eq(auditLinks.auditId, auditId),+ eq(auditLinks.isInternal, true),+ gte(auditPages.statusCode, 400),+ eq(auditPages.fetchClass, "ok"),+ ),+ )+ .limit(BROKEN_LINK_ISSUE_CAP);++ return rows.map((row) => ({+ issueType: "broken-internal-link" as const,+ pageId: row.sourcePageId,+ pageUrl: row.sourceUrl,+ dedupeKey: row.targetUrl,+ details: { targetUrl: row.targetUrl, targetStatus: row.targetStatus },+ }));+}++async function findOrphanPages(+ auditId: string,+ startUrl: string,+): Promise<DetectedIssue[]> {+ // A live 2xx page is an orphan when no OTHER crawled page links to it and+ // nothing redirects to it. Only meaningful on a completed crawl: on a+ // truncated one, "no observed inlinks" is true of nearly everything.+ // Error/redirect rows aren't orphans — they already get their own issues.+ const inlink = db+ .select({ one: sql`1` })+ .from(auditLinks)+ .where(+ and(+ eq(auditLinks.auditId, auditId),+ eq(auditLinks.isInternal, true),+ eq(auditLinks.targetUrl, auditPages.url),+ // Self-links (breadcrumbs, anchors) don't make a page reachable.+ ne(auditLinks.sourcePageId, auditPages.id),+ ),+ );++ const redirectSourcePages = alias(auditPages, "redirect_sources");+ const redirectSources = db+ .select({ one: sql`1` })+ .from(redirectSourcePages)+ .where(+ and(+ eq(redirectSourcePages.auditId, auditId),+ eq(redirectSourcePages.redirectUrl, auditPages.url),+ ),+ );++ const rows = await db+ .select({ id: auditPages.id, url: auditPages.url })+ .from(auditPages)+ .where(+ and(+ eq(auditPages.auditId, auditId),+ ne(auditPages.url, startUrl),+ eq(auditPages.fetchClass, "ok"),+ gte(auditPages.statusCode, 200),+ lt(auditPages.statusCode, 300),+ notExists(inlink),+ notExists(redirectSources),+ ),+ );++ return rows.map((row) => ({+ issueType: "orphan-page" as const,+ pageId: row.id,+ pageUrl: row.url,+ }));+}diff --git a/src/server/lib/audit/issues/page-reporters.test.ts b/src/server/lib/audit/issues/page-reporters.test.tsnew file mode 100644index 00000000..576328f2--- /dev/null+++ b/src/server/lib/audit/issues/page-reporters.test.ts@@ -0,0 +1,308 @@+import { describe, expect, it } from "vitest";+import { runPageReporters } from "@/server/lib/audit/issues/page-reporters";+import {+ findDuplicates,+ findRedirectChainsAndLoops,+ type SlimPage,+} from "@/server/lib/audit/issues/multipage-checks";+import type { CrawledPageResult } from "@/server/lib/audit/types";++function makePage(overrides: Partial<CrawledPageResult>): CrawledPageResult {+ return {+ id: "page-1",+ url: "https://example.com/a",+ statusCode: 200,+ fetchClass: "ok",+ redirectUrl: null,+ title: "A perfectly reasonable page title",+ metaDescription:+ "A reasonable meta description that says something useful about the page.",+ canonicalUrl: null,+ robotsMeta: null,+ xRobotsTag: null,+ headerCanonicalUrl: null,+ ogTitle: null,+ ogDescription: null,+ ogImage: null,+ h1Count: 1,+ h2Count: 0,+ h3Count: 0,+ h4Count: 0,+ h5Count: 0,+ h6Count: 0,+ headingOrder: [1, 2, 3],+ wordCount: 500,+ contentHash: "abc123",+ isHtml: true,+ imagesTotal: 0,+ imagesMissingAlt: 0,+ images: [],+ links: [],+ hasStructuredData: false,+ hreflangTags: [],+ isIndexable: true,+ responseTimeMs: 200,+ crawlDepth: 1,+ inSitemap: true,+ ...overrides,+ };+}++function issueTypes(page: CrawledPageResult): string[] {+ return runPageReporters(page).map((issue) => issue.issueType);+}++describe("runPageReporters", () => {+ it("reports nothing for a healthy page", () => {+ expect(issueTypes(makePage({}))).toEqual([]);+ });++ it("reports only blocked-page for a blocked fetch", () => {+ expect(+ issueTypes(makePage({ fetchClass: "blocked", statusCode: 403 })),+ ).toEqual(["blocked-page"]);+ });++ it("reports nothing for a fetch error", () => {+ expect(+ issueTypes(makePage({ fetchClass: "error", statusCode: 0 })),+ ).toEqual([]);+ });++ it("classifies error statuses by range", () => {+ expect(issueTypes(makePage({ statusCode: 500 }))).toEqual(["server-error"]);+ expect(issueTypes(makePage({ statusCode: 404 }))).toEqual(["broken-page"]);+ expect(+ issueTypes(+ makePage({+ statusCode: 301,+ redirectUrl: "https://example.com/b",+ }),+ ),+ ).toEqual([]);+ });++ it("checks titles and meta descriptions", () => {+ expect(issueTypes(makePage({ title: "" }))).toContain("missing-title");+ expect(issueTypes(makePage({ title: "x".repeat(70) }))).toContain(+ "title-too-long",+ );+ expect(issueTypes(makePage({ title: "Tiny" }))).toContain(+ "title-too-short",+ );+ expect(issueTypes(makePage({ metaDescription: "" }))).toContain(+ "missing-meta-description",+ );+ expect(+ issueTypes(makePage({ metaDescription: "x".repeat(200) })),+ ).toContain("meta-description-too-long");+ });++ it("checks headings", () => {+ expect(issueTypes(makePage({ h1Count: 0 }))).toContain("missing-h1");+ expect(issueTypes(makePage({ h1Count: 3 }))).toContain("multiple-h1");+ expect(issueTypes(makePage({ headingOrder: [1, 2, 4] }))).toContain(+ "heading-order-skip",+ );+ });++ it("skips content checks for non-HTML responses", () => {+ const nonHtml = makePage({+ isHtml: false,+ title: "",+ metaDescription: "",+ h1Count: 0,+ headingOrder: [],+ wordCount: 0,+ contentHash: null,+ });+ expect(issueTypes(nonHtml)).toEqual([]);+ });++ it("still checks empty-shell HTML pages", () => {+ const shell = makePage({+ isHtml: true,+ title: "",+ metaDescription: "",+ h1Count: 0,+ headingOrder: [],+ wordCount: 0,+ contentHash: null,+ });+ const types = issueTypes(shell);+ expect(types).toContain("missing-title");+ expect(types).toContain("missing-h1");+ expect(types).toContain("thin-content");+ });++ it("flags indexability and canonical signals", () => {+ expect(+ issueTypes(makePage({ isIndexable: false, robotsMeta: "noindex" })),+ ).toContain("noindex-page");++ const conflicted = issueTypes(+ makePage({+ canonicalUrl: "https://example.com/canonical-a",+ headerCanonicalUrl: "https://example.com/canonical-b",+ }),+ );+ expect(conflicted).toContain("canonical-conflict");+ expect(conflicted).toContain("canonicalized-page");++ expect(+ issueTypes(makePage({ canonicalUrl: "https://example.com/a" })),+ ).not.toContain("canonicalized-page");+ });++ it("flags thin content only on indexable pages", () => {+ expect(issueTypes(makePage({ wordCount: 50 }))).toContain("thin-content");+ expect(+ issueTypes(+ makePage({ wordCount: 50, isIndexable: false, robotsMeta: "noindex" }),+ ),+ ).not.toContain("thin-content");+ });++ it("flags slow responses and deep pages", () => {+ expect(issueTypes(makePage({ responseTimeMs: 3000 }))).toContain(+ "slow-response",+ );+ expect(issueTypes(makePage({ crawlDepth: 6 }))).toContain("deep-page");+ expect(issueTypes(makePage({ crawlDepth: null }))).not.toContain(+ "deep-page",+ );+ });+});++function makeSlimPage(overrides: Partial<SlimPage>): SlimPage {+ return {+ id: overrides.url ?? "page",+ url: "https://example.com/a",+ statusCode: 200,+ fetchClass: "ok",+ title: null,+ metaDescription: null,+ contentHash: null,+ redirectUrl: null,+ wordCount: 100,+ isIndexable: true,+ canonicalUrl: null,+ headerCanonicalUrl: null,+ ...overrides,+ };+}++describe("findDuplicates", () => {+ it("flags duplicate titles across pages and includes the other URLs", () => {+ const issues = findDuplicates([+ makeSlimPage({ url: "https://example.com/a", title: "Same" }),+ makeSlimPage({ url: "https://example.com/b", title: "Same" }),+ makeSlimPage({ url: "https://example.com/c", title: "Different" }),+ ]);+ const duplicateTitles = issues.filter(+ (issue) => issue.issueType === "duplicate-title",+ );+ expect(duplicateTitles).toHaveLength(2);+ expect(duplicateTitles[0].details?.otherUrls).toEqual([+ "https://example.com/b",+ ]);+ });++ it("excludes noindexed and canonicalized pages from duplicate groups", () => {+ const issues = findDuplicates([+ makeSlimPage({ url: "https://example.com/a", title: "Same" }),+ makeSlimPage({+ url: "https://example.com/b",+ title: "Same",+ canonicalUrl: "https://example.com/a",+ }),+ makeSlimPage({+ url: "https://example.com/c",+ title: "Same",+ isIndexable: false,+ }),+ ]);+ expect(issues).toHaveLength(0);+ });++ it("ignores non-2xx and blocked pages", () => {+ const issues = findDuplicates([+ makeSlimPage({ url: "https://example.com/a", title: "Same" }),+ makeSlimPage({+ url: "https://example.com/b",+ title: "Same",+ fetchClass: "blocked",+ statusCode: 403,+ }),+ ]);+ expect(issues).toHaveLength(0);+ });++ it("groups duplicate content by hash only when there is text", () => {+ const issues = findDuplicates([+ makeSlimPage({ url: "https://example.com/a", contentHash: "h1" }),+ makeSlimPage({ url: "https://example.com/b", contentHash: "h1" }),+ makeSlimPage({+ url: "https://example.com/empty-1",+ contentHash: "h2",+ wordCount: 0,+ }),+ makeSlimPage({+ url: "https://example.com/empty-2",+ contentHash: "h2",+ wordCount: 0,+ }),+ ]);+ expect(+ issues.filter((issue) => issue.issueType === "duplicate-content"),+ ).toHaveLength(2);+ });+});++describe("findRedirectChainsAndLoops", () => {+ const redirect = (url: string, target: string) =>+ makeSlimPage({ url, statusCode: 301, redirectUrl: target });++ it("ignores single redirects", () => {+ expect(+ findRedirectChainsAndLoops([+ redirect("https://example.com/a", "https://example.com/b"),+ makeSlimPage({ url: "https://example.com/b" }),+ ]),+ ).toHaveLength(0);+ });++ it("flags a chain once, on its head", () => {+ const issues = findRedirectChainsAndLoops([+ redirect("https://example.com/a", "https://example.com/b"),+ redirect("https://example.com/b", "https://example.com/c"),+ makeSlimPage({ url: "https://example.com/c" }),+ ]);+ expect(issues).toHaveLength(1);+ expect(issues[0].issueType).toBe("redirect-chain");+ expect(issues[0].pageUrl).toBe("https://example.com/a");+ expect(issues[0].details?.hops).toEqual([+ "https://example.com/a",+ "https://example.com/b",+ "https://example.com/c",+ ]);+ });++ it("flags loops", () => {+ const issues = findRedirectChainsAndLoops([+ redirect("https://example.com/a", "https://example.com/b"),+ redirect("https://example.com/b", "https://example.com/a"),+ ]);+ expect(+ issues.filter((issue) => issue.issueType === "redirect-loop").length,+ ).toBeGreaterThan(0);+ });++ it("flags self-loops", () => {+ const issues = findRedirectChainsAndLoops([+ redirect("https://example.com/a", "https://example.com/a"),+ ]);+ expect(issues).toHaveLength(1);+ expect(issues[0].issueType).toBe("redirect-loop");+ });+});diff --git a/src/server/lib/audit/issues/page-reporters.ts b/src/server/lib/audit/issues/page-reporters.tsnew file mode 100644index 00000000..b9886784--- /dev/null+++ b/src/server/lib/audit/issues/page-reporters.ts@@ -0,0 +1,146 @@+/**+ * Per-page issue reporters.+ *+ * Each reporter is a pure function over a single crawled page record —+ * DOM-free by design (cheerio runs once in crawlPage), so the engine works+ * over any crawl source that can produce a CrawledPageResult.+ *+ * Cross-page checks (duplicates, broken links, orphans, redirect chains)+ * live in multipage.ts and run over D1 after the crawl.+ */+import type { AuditIssueType } from "@/shared/audit-issues";+import type { CrawledPageResult } from "@/server/lib/audit/types";++export interface DetectedIssue {+ issueType: AuditIssueType;+ pageId: string | null;+ pageUrl: string;+ details?: Record<string, unknown>;+ /**+ * Distinguishes multiple issues of the same type on the same page+ * (e.g. one broken-internal-link issue per target). Part of the+ * deterministic row id, so step retries don't duplicate issues.+ */+ dedupeKey?: string;+}++const TITLE_MAX_CHARS = 60;+const TITLE_MIN_CHARS = 10;+const META_DESCRIPTION_MAX_CHARS = 160;+const THIN_CONTENT_WORDS = 150;+const SLOW_RESPONSE_MS = 1500;+const DEEP_PAGE_DEPTH = 5;++function hasHeadingLevelSkip(headingOrder: number[]): boolean {+ for (let i = 1; i < headingOrder.length; i++) {+ if (headingOrder[i] > headingOrder[i - 1] + 1) return true;+ }+ return false;+}++export function runPageReporters(page: CrawledPageResult): DetectedIssue[] {+ const issues: DetectedIssue[] = [];+ const report = (+ issueType: AuditIssueType,+ details?: Record<string, unknown>,+ ) => issues.push({ issueType, pageId: page.id, pageUrl: page.url, details });++ if (page.fetchClass === "blocked") {+ report("blocked-page", { statusCode: page.statusCode });+ return issues;+ }+ if (page.fetchClass === "error") {+ return issues;+ }++ if (page.statusCode >= 500) {+ report("server-error", { statusCode: page.statusCode });+ return issues;+ }+ if (page.statusCode >= 400) {+ report("broken-page", { statusCode: page.statusCode });+ return issues;+ }+ // Redirects are normal on their own; chains/loops are flagged in multipage.+ if (page.statusCode >= 300) {+ return issues;+ }++ if (page.responseTimeMs > SLOW_RESPONSE_MS) {+ report("slow-response", { responseTimeMs: page.responseTimeMs });+ }++ // Content checks only make sense for analyzed HTML documents (a PDF has no+ // title tag to miss; an empty-shell HTML page very much does).+ if (!page.isHtml) {+ return issues;+ }++ // Titles+ if (!page.title) {+ report("missing-title");+ } else if (page.title.length > TITLE_MAX_CHARS) {+ report("title-too-long", { length: page.title.length });+ } else if (page.title.length < TITLE_MIN_CHARS) {+ report("title-too-short", { length: page.title.length });+ }++ // Meta description+ if (!page.metaDescription) {+ report("missing-meta-description");+ } else if (page.metaDescription.length > META_DESCRIPTION_MAX_CHARS) {+ report("meta-description-too-long", {+ length: page.metaDescription.length,+ });+ }++ // Headings+ if (page.h1Count === 0) {+ report("missing-h1");+ } else if (page.h1Count > 1) {+ report("multiple-h1", { h1Count: page.h1Count });+ }+ if (hasHeadingLevelSkip(page.headingOrder)) {+ report("heading-order-skip");+ }++ // Indexability + canonical signals+ if (!page.isIndexable) {+ report("noindex-page", {+ robotsMeta: page.robotsMeta,+ xRobotsTag: page.xRobotsTag,+ });+ }+ if (+ page.canonicalUrl &&+ page.headerCanonicalUrl &&+ page.canonicalUrl !== page.headerCanonicalUrl+ ) {+ report("canonical-conflict", {+ htmlCanonical: page.canonicalUrl,+ headerCanonical: page.headerCanonicalUrl,+ });+ }+ const effectiveCanonical = page.canonicalUrl ?? page.headerCanonicalUrl;+ if (effectiveCanonical && effectiveCanonical !== page.url) {+ report("canonicalized-page", { canonicalUrl: effectiveCanonical });+ }++ // Content quality+ if (page.isIndexable && page.wordCount < THIN_CONTENT_WORDS) {+ report("thin-content", { wordCount: page.wordCount });+ }+ if (page.imagesMissingAlt > 0) {+ report("images-missing-alt", {+ imagesMissingAlt: page.imagesMissingAlt,+ imagesTotal: page.imagesTotal,+ });+ }++ // Structure+ if (page.crawlDepth !== null && page.crawlDepth >= DEEP_PAGE_DEPTH) {+ report("deep-page", { crawlDepth: page.crawlDepth });+ }++ return issues;+}diff --git a/src/server/lib/audit/lighthouse.ts b/src/server/lib/audit/lighthouse.tsindex 5194ed0c..50976f85 100644--- a/src/server/lib/audit/lighthouse.ts+++ b/src/server/lib/audit/lighthouse.ts@@ -1,4 +1,4 @@-import { detectUrlTemplate } from "./url-utils";+import { detectUrlTemplate, normalizeUrl } from "./url-utils";import type { BillingCustomerContext } from "@/server/billing/subscription";import { createDataforseoClient } from "@/server/lib/dataforseo";import type { LighthouseResult, LighthouseStrategy } from "./types";@@ -127,16 +127,13 @@ export function selectLighthouseSample((p) => p.statusCode >= 200 && p.statusCode < 300,);- if (strategy === "manual") {- // manual = user picks after crawl; for now return empty- return [];- }-// strategy === "auto": homepage + 1 per URL pattern, capped at 10const selected = new Set<string>();- // Always include the start URL / homepage- const startPage = validPages.find((p) => p.url === startUrl);+ // Always include the start URL / homepage. Page URLs are normalized;+ // normalize the start URL the same way or the comparison silently misses.+ const normalizedStart = normalizeUrl(startUrl) ?? startUrl;+ const startPage = validPages.find((p) => p.url === normalizedStart);if (startPage) selected.add(startPage.url);// Group by URL template patterndiff --git a/src/server/lib/audit/page-analyzer.ts b/src/server/lib/audit/page-analyzer.tsindex e853ae0c..8c424c88 100644--- a/src/server/lib/audit/page-analyzer.ts+++ b/src/server/lib/audit/page-analyzer.ts@@ -7,7 +7,7 @@*/import * as cheerio from "cheerio";import { normalizeUrl, isSameOrigin } from "./url-utils";-import type { PageAnalysis } from "./types";+import type { PageAnalysis, PageLink } from "./types";/*** Analyze an HTML string and extract all SEO-relevant data.@@ -71,9 +71,8 @@ export function analyzeHtml(});});- // --- Links ---- const internalLinks: string[] = [];- const externalLinks: string[] = [];+ // --- Links (deduped by target URL; first anchor wins) ---+ const linksByTarget = new Map<string, PageLink>();$("a[href]").each((_, el) => {const href = $(el).attr("href");@@ -84,13 +83,18 @@ export function analyzeHtml(const resolved = normalizeUrl(href, pageUrl);if (!resolved) return;-- if (isSameOrigin(resolved, pageUrl)) {- internalLinks.push(resolved);- } else {- externalLinks.push(resolved);- }+ if (linksByTarget.has(resolved)) return;++ const anchor = $(el).text().replace(/\s+/g, " ").trim().slice(0, 200);+ const rel = $(el).attr("rel")?.toLowerCase() ?? "";+ linksByTarget.set(resolved, {+ targetUrl: resolved,+ anchor: anchor || null,+ isInternal: isSameOrigin(resolved, pageUrl),+ isNofollow: rel.split(/\s+/).includes("nofollow"),+ });});+ const links = Array.from(linksByTarget.values());// --- Structured data (JSON-LD) ---let hasStructuredData = false;@@ -119,9 +123,9 @@ export function analyzeHtml(h1s,headingOrder,wordCount,+ bodyText,images,- internalLinks,- externalLinks,+ links,hasStructuredData,hreflangTags,};diff --git a/src/server/lib/audit/types.ts b/src/server/lib/audit/types.tsindex bb2bf157..e7ca4028 100644--- a/src/server/lib/audit/types.ts+++ b/src/server/lib/audit/types.ts@@ -6,7 +6,7 @@ import { z } from "zod";import { MIN_AUDIT_PAGES, PAID_MAX_AUDIT_PAGES } from "@/shared/audit-limits";import { jsonCodec } from "@/shared/json";-export type LighthouseStrategy = "auto" | "manual" | "none";+export type LighthouseStrategy = "auto" | "none";export interface AuditConfig {maxPages: number;@@ -14,11 +14,21 @@ export interface AuditConfig {}// Read-side only (writes stringify a typed AuditConfig). Stored rows may hold-// retired strategies (e.g. "all"); fall back to "auto" instead of failing the-// whole config parse and making the audit's results unviewable.+// retired strategies ("all", "manual") from older audits; map them onto the+// closest surviving strategy — and fall back to "auto" on anything unknown —+// instead of failing the whole config parse and making the audit's results+// unviewable.+const lighthouseStrategySchema = z+ .enum(["auto", "all", "manual", "none"])+ .transform(+ (value): LighthouseStrategy =>+ value === "all" ? "auto" : value === "manual" ? "none" : value,+ )+ .catch("auto");+const auditConfigSchema = z.object({maxPages: z.number().int().min(MIN_AUDIT_PAGES).max(PAID_MAX_AUDIT_PAGES),- lighthouseStrategy: z.enum(["auto", "manual", "none"]).catch("auto"),+ lighthouseStrategy: lighthouseStrategySchema,});const auditConfigCodec = jsonCodec(auditConfigSchema);@@ -29,6 +39,17 @@ export function parseAuditConfig(configRaw: string | null): AuditConfig | null {return result.success ? result.data : null;}+/** How a page fetch resolved. "blocked" = WAF/bot challenge stood in the way. */+export type PageFetchClass = "ok" | "blocked" | "error";++/** One outgoing link edge, deduped by target URL within a page. */+export interface PageLink {+ targetUrl: string;+ anchor: string | null;+ isInternal: boolean;+ isNofollow: boolean;+}+/** Data extracted from a single page via cheerio. */export interface PageAnalysis {url: string;@@ -51,13 +72,13 @@ export interface PageAnalysis {// ContentwordCount: number;+ bodyText: string;// Imagesimages: Array<{ src: string | null; alt: string | null }>;- // Links (raw href values from the HTML)- internalLinks: string[];- externalLinks: string[];+ // Links (normalized, deduped by target)+ links: PageLink[];// Structured datahasStructuredData: boolean;@@ -84,15 +105,22 @@ export interface LighthouseResult {payloadSizeBytes?: number | null;}-export interface StepPageResult {+/**+ * Full result of crawling one page. Persisted to D1 inside the crawl-batch+ * step; never accumulated in memory or returned as durable step state.+ */+export interface CrawledPageResult {id: string;url: string;statusCode: number;+ fetchClass: PageFetchClass;redirectUrl: string | null;title: string;metaDescription: string;canonicalUrl: string | null;robotsMeta: string | null;+ xRobotsTag: string | null;+ headerCanonicalUrl: string | null;ogTitle: string | null;ogDescription: string | null;ogImage: string | null;@@ -104,13 +132,37 @@ export interface StepPageResult {h6Count: number;headingOrder: number[];wordCount: number;+ contentHash: string | null;+ /**+ * True when an HTML document was fetched and analyzed. Gates the content+ * checks in page reporters (an empty-shell HTML page must still be+ * checked; a PDF must not). Transient — not persisted.+ */+ isHtml: boolean;imagesTotal: number;imagesMissingAlt: number;images: Array<{ src: string | null; alt: string | null }>;- internalLinks: string[];- externalLinks: string[];+ links: PageLink[];hasStructuredData: boolean;hreflangTags: string[];isIndexable: boolean;responseTimeMs: number;+ /** null = not reached via links (e.g. sitemap-seeded). */+ crawlDepth: number | null;+ inSitemap: boolean;+}++/**+ * Slim per-page summary returned as durable step state from a crawl batch.+ * Keep this small: full page data lives in D1, not in Workflow step state.+ */+export interface StepPageSummary {+ id: string;+ url: string;+ statusCode: number;+ fetchClass: PageFetchClass;+ redirectUrl: string | null;+ title: string;+ /** Normalized same-origin link targets, for frontier expansion. */+ internalLinks: string[];}diff --git a/src/server/lib/audit/url-policy.ts b/src/server/lib/audit/url-policy.tsindex f5ddbb31..0c386fd7 100644--- a/src/server/lib/audit/url-policy.ts+++ b/src/server/lib/audit/url-policy.ts@@ -186,6 +186,26 @@ async function hostnameResolvesToBlockedAddress(}}+/**+ * Synchronous SSRF check for URLs discovered mid-crawl (links, redirect+ * targets, sitemap entries). Blocks non-http(s) schemes, private/loopback IP+ * literals, and internal hostnames. DNS resolution is only performed for the+ * start URL (see normalizeAndValidateStartUrl); per-link DoH lookups would be+ * prohibitively slow.+ */+export function isCrawlableUrl(url: string): boolean {+ let parsed: URL;+ try {+ parsed = new URL(url);+ } catch {+ return false;+ }+ if (parsed.protocol !== "http:" && parsed.protocol !== "https:") {+ return false;+ }+ return !isBlockedHost(parsed.hostname);+}+export async function normalizeAndValidateStartUrl(input: string,): Promise<string> {diff --git a/src/server/mcp/server.ts b/src/server/mcp/server.tsindex 22d224cb..3d2d8d6b 100644--- a/src/server/mcp/server.ts+++ b/src/server/mcp/server.ts@@ -22,6 +22,12 @@ import {getSearchConsolePerformanceTool,inspectUrlsTool,} from "@/server/mcp/tools/search-console-tools";+import {+ getAuditIssuesTool,+ getAuditPagesTool,+ getAuditStatusTool,+ runSiteAuditTool,+} from "@/server/mcp/tools/site-audit-tools";import { whoamiTool } from "@/server/mcp/tools/whoami";// Each handler is wrapped with instrumentMcpToolHandler so failures reach@@ -201,4 +207,24 @@ export function registerOpenSeoMcpTools(server: McpServer) {inspectUrlsTool.handler,),);+ server.registerTool(+ runSiteAuditTool.name,+ runSiteAuditTool.config,+ runSiteAuditTool.handler,+ );+ server.registerTool(+ getAuditStatusTool.name,+ getAuditStatusTool.config,+ getAuditStatusTool.handler,+ );+ server.registerTool(+ getAuditIssuesTool.name,+ getAuditIssuesTool.config,+ getAuditIssuesTool.handler,+ );+ server.registerTool(+ getAuditPagesTool.name,+ getAuditPagesTool.config,+ getAuditPagesTool.handler,+ );}diff --git a/src/server/mcp/tools/site-audit-tools.ts b/src/server/mcp/tools/site-audit-tools.tsnew file mode 100644index 00000000..488bc787--- /dev/null+++ b/src/server/mcp/tools/site-audit-tools.ts@@ -0,0 +1,399 @@+import { z } from "zod";+import { AuditRepository } from "@/server/features/audit/repositories/AuditRepository";+import { AuditService } from "@/server/features/audit/services/AuditService";+import { AppError } from "@/server/lib/errors";+import { captureServerEvent } from "@/server/lib/posthog";+import {+ AUDIT_ISSUE_TYPES,+ getIssueDescriptor,+ ISSUE_SEVERITY_ORDER,+} from "@/shared/audit-issues";+import { mcpResponse } from "@/server/mcp/formatters";+import { buildProjectMeta } from "@/server/mcp/context";+import {+ looseObjectOutputSchema,+ optionalMetaOutputSchema,+} from "@/server/mcp/output-schemas";+import { withMcpProjectAuth } from "@/server/mcp/project-auth";+import { projectIdSchema } from "@/server/mcp/schemas";++const auditIdSchema = z+ .string()+ .optional()+ .describe("Audit ID. If omitted, uses the project's most recent audit.");++async function resolveAudit(projectId: string, auditId?: string) {+ const audit = auditId+ ? await AuditRepository.getAuditForProject(auditId, projectId)+ : await AuditRepository.getLatestAuditForProject(projectId);+ if (!audit) {+ throw new AppError(+ "NOT_FOUND",+ auditId+ ? `Audit ${auditId} not found in this project.`+ : "No audits exist for this project yet. Start one with run_site_audit.",+ );+ }+ return audit;+}++function auditPath(projectId: string, auditId: string) {+ return `/p/${projectId}/audit?auditId=${auditId}`;+}++// ─── run_site_audit ──────────────────────────────────────────────────────────++const runInputSchema = {+ projectId: projectIdSchema,+ url: z.string().min(1).max(2048).describe("Start URL to crawl."),+ maxPages: z+ .number()+ .int()+ .min(10)+ .max(10_000)+ .optional()+ .describe("Page budget for the crawl (default 50)."),+ runLighthouse: z+ .boolean()+ .optional()+ .describe(+ "Run Lighthouse on a sample of up to 10 representative pages (default true).",+ ),+} as const;++type RunArgs = z.infer<z.ZodObject<typeof runInputSchema>>;++export const runSiteAuditTool = {+ name: "run_site_audit",+ config: {+ title: "Run site audit",+ description:+ "Start a site audit: crawls the site (robots.txt-aware, same-origin), checks every page for SEO issues (broken links, duplicate/missing titles and descriptions, redirect chains, orphan pages, canonical problems, thin content, and more), and optionally runs Lighthouse on a sample of pages. Runs in the background — poll get_audit_status, then read get_audit_issues. If the site blocks our crawler, pages are honestly flagged as blocked rather than misreported.",+ inputSchema: runInputSchema,+ outputSchema: z+ .object({+ auditId: z.string(),+ ...optionalMetaOutputSchema,+ })+ .passthrough(),+ annotations: {+ readOnlyHint: false,+ openWorldHint: true,+ destructiveHint: false,+ },+ },+ handler: withMcpProjectAuth(async (args: RunArgs, context) => {+ const lighthouseStrategy = (args.runLighthouse ?? true) ? "auto" : "none";+ const limitTier = await AuditService.resolveAuditLimitTier(+ context.auth.organizationId,+ );+ let auditId: string;+ try {+ ({ auditId } = await AuditService.startAudit({+ actorUserId: context.auth.userId,+ billingCustomer: context.billing,+ projectId: args.projectId,+ startUrl: args.url,+ maxPages: args.maxPages,+ lighthouseStrategy,+ limitTier,+ }));+ } catch (error) {+ if (+ error instanceof AppError &&+ error.code === "AUDIT_CAPACITY_REACHED"+ ) {+ return mcpResponse({+ text: "Audit capacity reached for this account — delete old audits in the dashboard to free capacity, then try again.",+ meta: buildProjectMeta(+ context,+ args.projectId,+ `/p/${args.projectId}/audit`,+ ),+ });+ }+ throw error;+ }++ await captureServerEvent({+ distinctId: context.auth.userId,+ event: "site_audit:start",+ organizationId: context.auth.organizationId,+ properties: {+ project_id: args.projectId,+ max_pages: args.maxPages ?? 50,+ run_lighthouse: lighthouseStrategy !== "none",+ source: "mcp",+ },+ });++ return mcpResponse({+ text: `Audit ${auditId} started for ${args.url}. Poll get_audit_status until it completes, then call get_audit_issues for the prioritized issue report.`,+ meta: buildProjectMeta(+ context,+ args.projectId,+ auditPath(args.projectId, auditId),+ ),+ structuredContent: { auditId },+ });+ }),+};++// ─── get_audit_status ────────────────────────────────────────────────────────++const statusInputSchema = {+ projectId: projectIdSchema,+ auditId: auditIdSchema,+} as const;++type StatusArgs = z.infer<z.ZodObject<typeof statusInputSchema>>;++export const getAuditStatusTool = {+ name: "get_audit_status",+ config: {+ title: "Get site audit status",+ description:+ "Check the progress of a site audit (phase, pages crawled, Lighthouse progress). Free — reads OpenSEO state. Omit auditId for the most recent audit.",+ inputSchema: statusInputSchema,+ outputSchema: z+ .object({+ status: looseObjectOutputSchema,+ ...optionalMetaOutputSchema,+ })+ .passthrough(),+ annotations: {+ readOnlyHint: true,+ openWorldHint: false,+ destructiveHint: false,+ },+ },+ handler: withMcpProjectAuth(async (args: StatusArgs, context) => {+ // getStatus fetches (and self-heals) the audit row itself; only hit the+ // DB here when we need to default to the most recent audit.+ const auditId = args.auditId ?? (await resolveAudit(args.projectId)).id;+ const status = await AuditService.getStatus(auditId, args.projectId);++ const lighthouseNote =+ status.lighthouseTotal > 0+ ? `, lighthouse ${status.lighthouseCompleted + status.lighthouseFailed}/${status.lighthouseTotal}`+ : "";+ return mcpResponse({+ text: `Audit ${status.id} (${status.startUrl}): ${status.status} — phase ${status.currentPhase}, ${status.pagesCrawled}/${status.pagesTotal} pages${lighthouseNote}.${status.status === "completed" ? " Call get_audit_issues for the issue report." : ""}`,+ meta: buildProjectMeta(+ context,+ args.projectId,+ auditPath(args.projectId, status.id),+ ),+ structuredContent: { status },+ });+ }),+};++// ─── get_audit_issues ────────────────────────────────────────────────────────++const issuesInputSchema = {+ projectId: projectIdSchema,+ auditId: auditIdSchema,+ severity: z+ .enum(["critical", "warning", "info"])+ .optional()+ .describe("Only return issues of this severity."),+ issueType: z+ .string()+ .optional()+ .describe(+ `Only return issues of this type. One of: ${Object.keys(AUDIT_ISSUE_TYPES).join(", ")}`,+ ),+ limit: z+ .number()+ .int()+ .min(1)+ .max(1_000)+ .optional()+ .describe("Max issues to return (default 200)."),+} as const;++type IssuesArgs = z.infer<z.ZodObject<typeof issuesInputSchema>>;++export const getAuditIssuesTool = {+ name: "get_audit_issues",+ config: {+ title: "Get site audit issues",+ description:+ "Read the prioritized issue report from a completed site audit. Every issue carries a how_to_fix with concrete remediation steps an agent can act on. Free — reads OpenSEO state. Omit auditId for the most recent audit.",+ inputSchema: issuesInputSchema,+ outputSchema: z+ .object({+ summary: z.array(looseObjectOutputSchema),+ issues: z.array(looseObjectOutputSchema),+ ...optionalMetaOutputSchema,+ })+ .passthrough(),+ annotations: {+ readOnlyHint: true,+ openWorldHint: false,+ destructiveHint: false,+ },+ },+ handler: withMcpProjectAuth(async (args: IssuesArgs, context) => {+ const audit = await resolveAudit(args.projectId, args.auditId);+ const unsorted = await AuditRepository.getIssuesForAudit(audit.id, {+ severity: args.severity,+ issueType: args.issueType,+ });+ // Severity-first so truncation drops info rows, never critical ones.+ const rows = unsorted.toSorted(+ (a, b) =>+ ISSUE_SEVERITY_ORDER[a.severity] - ISSUE_SEVERITY_ORDER[b.severity] ||+ a.issueType.localeCompare(b.issueType),+ );++ const counts = new Map<string, number>();+ for (const row of rows) {+ counts.set(row.issueType, (counts.get(row.issueType) ?? 0) + 1);+ }+ const summary = Array.from(counts.entries())+ .map(([issueType, count]) => {+ const descriptor = getIssueDescriptor(issueType);+ return {+ issueType,+ title: descriptor?.title ?? issueType,+ severity: descriptor?.severity ?? "info",+ count,+ };+ })+ .toSorted(+ (a, b) =>+ ISSUE_SEVERITY_ORDER[a.severity] - ISSUE_SEVERITY_ORDER[b.severity] ||+ b.count - a.count,+ );++ const limit = args.limit ?? 200;+ const issues = rows.slice(0, limit).map((row) => {+ const descriptor = getIssueDescriptor(row.issueType);+ return {+ severity: row.severity,+ issueType: row.issueType,+ title: descriptor?.title ?? row.issueType,+ url: row.pageUrl,+ details: row.detailsJson+ ? (JSON.parse(row.detailsJson) as unknown)+ : null,+ howToFix: descriptor?.howToFix ?? null,+ };+ });++ const text =+ rows.length === 0+ ? args.severity || args.issueType+ ? `No issues found for audit ${audit.id} matching the given filters.`+ : `No issues recorded for audit ${audit.id}. Note: audits run before issue checks existed have no issue data — re-run the audit with run_site_audit to get a real report.`+ : [+ `Audit ${audit.id} (${audit.startUrl}): ${rows.length} issues${rows.length > limit ? ` (showing ${limit})` : ""}.`,+ "By type:",+ ...summary.map(+ (entry) =>+ `- [${entry.severity}] ${entry.title} (${entry.issueType}): ${entry.count}`,+ ),+ "Full issue rows with how_to_fix instructions are in structuredContent.issues.",+ ].join("\n");++ return mcpResponse({+ text,+ meta: buildProjectMeta(+ context,+ args.projectId,+ auditPath(args.projectId, audit.id),+ ),+ structuredContent: { summary, issues },+ });+ }),+};++// ─── get_audit_pages ─────────────────────────────────────────────────────────++const pagesInputSchema = {+ projectId: projectIdSchema,+ auditId: auditIdSchema,+ fetchClass: z+ .enum(["ok", "blocked", "error"])+ .optional()+ .describe(+ 'Filter by fetch outcome ("blocked" = the site\'s bot protection challenged the crawler).',+ ),+ statusCode: z+ .number()+ .int()+ .optional()+ .describe("Filter by exact HTTP status code."),+ urlContains: z+ .string()+ .optional()+ .describe("Filter to URLs containing this substring."),+ limit: z+ .number()+ .int()+ .min(1)+ .max(1_000)+ .optional()+ .describe("Max pages to return (default 100)."),+} as const;++type PagesArgs = z.infer<z.ZodObject<typeof pagesInputSchema>>;++export const getAuditPagesTool = {+ name: "get_audit_pages",+ config: {+ title: "Get site audit pages",+ description:+ "List crawled pages from a site audit with per-page SEO data (status, title, description, word count, indexability, crawl depth, link counts). Free — reads OpenSEO state. Omit auditId for the most recent audit.",+ inputSchema: pagesInputSchema,+ outputSchema: z+ .object({+ pages: z.array(looseObjectOutputSchema),+ total: z.number(),+ ...optionalMetaOutputSchema,+ })+ .passthrough(),+ annotations: {+ readOnlyHint: true,+ openWorldHint: false,+ destructiveHint: false,+ },+ },+ handler: withMcpProjectAuth(async (args: PagesArgs, context) => {+ const audit = await resolveAudit(args.projectId, args.auditId);+ const allPages = await AuditRepository.getPagesForAudit(audit.id);++ const filtered = allPages.filter(+ (page) =>+ (!args.fetchClass || page.fetchClass === args.fetchClass) &&+ (args.statusCode === undefined ||+ page.statusCode === args.statusCode) &&+ (!args.urlContains || page.url.includes(args.urlContains)),+ );+ const limit = args.limit ?? 100;+ const pages = filtered.slice(0, limit);++ const text = [+ `Audit ${audit.id}: ${filtered.length} pages${filtered.length > limit ? ` (showing ${limit})` : ""}.`,+ ...pages+ .slice(0, 25)+ .map(+ (page) =>+ `- ${page.statusCode} ${page.url}${page.fetchClass !== "ok" ? ` [${page.fetchClass}]` : ""} "${page.title ?? ""}"`,+ ),+ "Full rows are in structuredContent.pages.",+ ].join("\n");++ return mcpResponse({+ text,+ meta: buildProjectMeta(+ context,+ args.projectId,+ auditPath(args.projectId, audit.id),+ ),+ structuredContent: { pages, total: filtered.length },+ });+ }),+};diff --git a/src/server/workflows/SiteAuditWorkflow.ts b/src/server/workflows/SiteAuditWorkflow.tsindex 2ca39245..b6fe8986 100644--- a/src/server/workflows/SiteAuditWorkflow.ts+++ b/src/server/workflows/SiteAuditWorkflow.ts@@ -40,20 +40,25 @@ export class SiteAuditWorkflow extends WorkflowEntrypoint<Env, AuditParams> {const { auditId, billingCustomer, projectId, startUrl, config } =event.payload;- const audit = await AuditRepository.getAuditForWorkflow(- auditId,- event.instanceId,- );+ try {+ // Inside a step so the D1 read is retried and replay-cached; a bare+ // read here would re-execute on every replay and a transient failure+ // would kill the instance before the catch below exists.+ await pgStep(step, "validate-context", undefined, async () => {+ const audit = await AuditRepository.getAuditForWorkflow(+ auditId,+ event.instanceId,+ );- if (!audit) {- throw new Error("Audit workflow context mismatch");- }+ if (!audit) {+ throw new Error("Audit workflow context mismatch");+ }- if (audit.projectId !== projectId) {- throw new Error("Audit workflow project mismatch");- }+ if (audit.projectId !== projectId) {+ throw new Error("Audit workflow project mismatch");+ }+ });- try {await runAuditPhases(step, {auditId,workflowInstanceId: event.instanceId,diff --git a/src/server/workflows/site-audit-workflow-helpers.ts b/src/server/workflows/site-audit-workflow-helpers.tsindex 5b616024..aac85068 100644--- a/src/server/workflows/site-audit-workflow-helpers.ts+++ b/src/server/workflows/site-audit-workflow-helpers.ts@@ -1,110 +1,258 @@-import type { StepPageResult } from "@/server/lib/audit/types";-import { isSameOrigin, normalizeUrl } from "@/server/lib/audit/url-utils";+import type {+ CrawledPageResult,+ PageFetchClass,+} from "@/server/lib/audit/types";+import { sha256Hex } from "@/server/lib/audit/ids";+import { normalizeUrl } from "@/server/lib/audit/url-utils";++const CRAWL_USER_AGENT = "OpenSEO-Audit/1.0";++/**+ * Markers of a bot-mitigation challenge page. We classify these honestly as+ * "blocked" instead of recording the challenge HTML as if it were the page.+ */+const CHALLENGE_BODY_MARKERS = [+ "just a moment...",+ "challenge-platform",+ "cf-browser-verification",+ "attention required! | cloudflare",+ "verifying you are human",+];++function classifyFetch(+ statusCode: number,+ headers: Headers,+ bodySnippet: string,+): PageFetchClass {+ if (statusCode === 0) return "error";+ if (headers.get("cf-mitigated")) return "blocked";+ if (statusCode === 401 || statusCode === 403 || statusCode === 429) {+ return "blocked";+ }+ if (statusCode === 503) {+ const snippet = bodySnippet.toLowerCase();+ if (CHALLENGE_BODY_MARKERS.some((marker) => snippet.includes(marker))) {+ return "blocked";+ }+ }+ return "ok";+}++/** Resolve a Location header against its base without normalizing. */+function resolveRawUrl(location: string, base: string): string | null {+ try {+ return new URL(location, base).toString();+ } catch {+ return null;+ }+}++/** Parse `Link: <url>; rel="canonical"` response headers. */+function parseLinkHeaderCanonical(+ linkHeader: string | null,+ pageUrl: string,+): string | null {+ if (!linkHeader) return null;+ for (const part of linkHeader.split(",")) {+ const match = part.match(/<([^>]+)>\s*;([^]*)/);+ if (!match) continue;+ if (/rel\s*=\s*"?canonical"?/i.test(match[2])) {+ return normalizeUrl(match[1].trim(), pageUrl);+ }+ }+ return null;+}export async function crawlPage(url: string,- crawlOrigin: string,-): Promise<StepPageResult | null> {+ crawlDepth: number | null,+ inSitemap: boolean,+): Promise<CrawledPageResult> {const startTime = Date.now();try {- const response = await fetch(url, {- headers: {- "User-Agent": "OpenSEO-Audit/1.0",- Accept: "text/html,application/xhtml+xml",- },- redirect: "follow",- signal: AbortSignal.timeout(15_000),- });+ // Manual redirect handling: each hop is recorded as its own page row and+ // the target is enqueued by the frontier, so chains/loops are detectable.+ // Exception: redirects whose target normalizes to this same URL (e.g.+ // /docs -> /docs/ on slash-canonical sites — our normalizer strips the+ // slash) are followed inline; recording them would create self-redirect+ // rows the frontier can never resolve.+ let fetchUrl = url;+ let response: Response;+ let hops = 0;+ for (;;) {+ response = await fetch(fetchUrl, {+ headers: {+ "User-Agent": CRAWL_USER_AGENT,+ Accept: "text/html,application/xhtml+xml",+ },+ redirect: "manual",+ signal: AbortSignal.timeout(15_000),+ });++ if (response.status < 300 || response.status >= 400) break;++ const location = response.headers.get("location");+ const rawTarget = location ? resolveRawUrl(location, fetchUrl) : null;+ const normalizedTarget = location+ ? normalizeUrl(location, fetchUrl)+ : null;+ const isSelfAfterNormalization =+ normalizedTarget === url &&+ rawTarget !== null &&+ rawTarget !== fetchUrl;+ if (!isSelfAfterNormalization || hops >= 3) break;++ fetchUrl = rawTarget;+ hops += 1;+ }const responseTimeMs = Date.now() - startTime;const statusCode = response.status;- const finalUrl = normalizeUrl(response.url) ?? response.url;- if (!isSameOrigin(finalUrl, crawlOrigin)) return null;+ const xRobotsTag = response.headers.get("x-robots-tag");+ const headerCanonicalUrl = parseLinkHeaderCanonical(+ response.headers.get("link"),+ fetchUrl,+ );++ if (statusCode >= 300 && statusCode < 400) {+ const location = response.headers.get("location");+ const redirectUrl = location ? normalizeUrl(location, fetchUrl) : null;+ return emptyPageResult({+ url,+ statusCode,+ fetchClass: "ok",+ redirectUrl,+ responseTimeMs,+ xRobotsTag,+ headerCanonicalUrl,+ crawlDepth,+ inSitemap,+ });+ }- const redirectUrl =- response.redirected && response.url !== url ? response.url : null;const contentType = response.headers.get("content-type") ?? "";- if (!contentType.includes("text/html")) {- return emptyPageResult(finalUrl, statusCode, redirectUrl, responseTimeMs);+ const isHtml = contentType.includes("text/html");+ const body = isHtml ? await response.text() : "";+ const fetchClass = classifyFetch(+ statusCode,+ response.headers,+ body.slice(0, 4_000),+ );++ if (!isHtml || fetchClass !== "ok" || statusCode >= 400) {+ return emptyPageResult({+ url,+ statusCode,+ fetchClass,+ redirectUrl: null,+ responseTimeMs,+ xRobotsTag,+ headerCanonicalUrl,+ crawlDepth,+ inSitemap,+ });}- const html = await response.text();+ // Resolve links/canonical against the URL that actually served the+ // content (it may carry a trailing slash the recorded URL doesn't).// Dynamic import keeps cheerio (page-analyzer's HTML parser) out of the// worker's startup module graph: SiteAuditWorkflow is re-exported from// src/server.ts, so a static import would evaluate cheerio in every// isolate's baseline heap, not just when an audit actually crawls.const { analyzeHtml } = await import("@/server/lib/audit/page-analyzer");- const analysis = analyzeHtml(- html,- finalUrl,- statusCode,- responseTimeMs,- redirectUrl,- );- const isIndexable = !(- analysis.robotsMeta?.toLowerCase().includes("noindex") ?? false- );- const h2Count = analysis.headingOrder.filter((h) => h === 2).length;- const h3Count = analysis.headingOrder.filter((h) => h === 3).length;- const h4Count = analysis.headingOrder.filter((h) => h === 4).length;- const h5Count = analysis.headingOrder.filter((h) => h === 5).length;- const h6Count = analysis.headingOrder.filter((h) => h === 6).length;+ const analysis = analyzeHtml(body, fetchUrl, statusCode, responseTimeMs);+ const robotsDirectives = [analysis.robotsMeta, xRobotsTag]+ .filter(Boolean)+ .join(",")+ .toLowerCase();+ const isIndexable = !robotsDirectives.includes("noindex");+ const headingCount = (level: number) =>+ analysis.headingOrder.filter((h) => h === level).length;return {id: crypto.randomUUID(),- url: finalUrl,+ url,statusCode,- redirectUrl,+ fetchClass,+ redirectUrl: null,title: analysis.title,metaDescription: analysis.metaDescription,- canonicalUrl: analysis.canonical,+ canonicalUrl: analysis.canonical+ ? (normalizeUrl(analysis.canonical, fetchUrl) ?? analysis.canonical)+ : null,robotsMeta: analysis.robotsMeta,+ xRobotsTag,+ headerCanonicalUrl,ogTitle: analysis.ogTitle,ogDescription: analysis.ogDescription,ogImage: analysis.ogImage,h1Count: analysis.h1s.length,- h2Count,- h3Count,- h4Count,- h5Count,- h6Count,+ h2Count: headingCount(2),+ h3Count: headingCount(3),+ h4Count: headingCount(4),+ h5Count: headingCount(5),+ h6Count: headingCount(6),headingOrder: analysis.headingOrder,wordCount: analysis.wordCount,+ contentHash: analysis.bodyText+ ? await sha256Hex(analysis.bodyText)+ : null,+ isHtml: true,imagesTotal: analysis.images.length,- imagesMissingAlt: analysis.images.filter(- (img) => !img.alt || img.alt === "",- ).length,+ // Only a truly absent alt attribute counts: alt="" is the correct+ // markup for decorative images.+ imagesMissingAlt: analysis.images.filter((img) => img.alt === null)+ .length,images: analysis.images,- internalLinks: analysis.internalLinks,- externalLinks: analysis.externalLinks,+ links: analysis.links,hasStructuredData: analysis.hasStructuredData,hreflangTags: analysis.hreflangTags,isIndexable,responseTimeMs,+ crawlDepth,+ inSitemap,};} catch (error) {const responseTimeMs = Date.now() - startTime;console.warn(`Failed to crawl ${url}:`, error);- return emptyPageResult(url, 0, null, responseTimeMs);+ return emptyPageResult({+ url,+ statusCode: 0,+ fetchClass: "error",+ redirectUrl: null,+ responseTimeMs,+ xRobotsTag: null,+ headerCanonicalUrl: null,+ crawlDepth,+ inSitemap,+ });}}-function emptyPageResult(- url: string,- statusCode: number,- redirectUrl: string | null,- responseTimeMs: number,-): StepPageResult {+function emptyPageResult(input: {+ url: string;+ statusCode: number;+ fetchClass: PageFetchClass;+ redirectUrl: string | null;+ responseTimeMs: number;+ xRobotsTag: string | null;+ headerCanonicalUrl: string | null;+ crawlDepth: number | null;+ inSitemap: boolean;+}): CrawledPageResult {return {id: crypto.randomUUID(),- url,- statusCode,- redirectUrl,+ url: input.url,+ statusCode: input.statusCode,+ fetchClass: input.fetchClass,+ redirectUrl: input.redirectUrl,title: "",metaDescription: "",canonicalUrl: null,robotsMeta: null,+ xRobotsTag: input.xRobotsTag,+ headerCanonicalUrl: input.headerCanonicalUrl,ogTitle: null,ogDescription: null,ogImage: null,@@ -116,14 +264,17 @@ function emptyPageResult(h6Count: 0,headingOrder: [],wordCount: 0,+ contentHash: null,+ isHtml: false,imagesTotal: 0,imagesMissingAlt: 0,images: [],- internalLinks: [],- externalLinks: [],+ links: [],hasStructuredData: false,hreflangTags: [],isIndexable: false,- responseTimeMs,+ responseTimeMs: input.responseTimeMs,+ crawlDepth: input.crawlDepth,+ inSitemap: input.inSitemap,};}diff --git a/src/server/workflows/siteAuditWorkflowCrawl.ts b/src/server/workflows/siteAuditWorkflowCrawl.tsindex a915e34c..08b9871f 100644--- a/src/server/workflows/siteAuditWorkflowCrawl.ts+++ b/src/server/workflows/siteAuditWorkflowCrawl.ts@@ -1,13 +1,21 @@import type { WorkflowStep } from "cloudflare:workers";import type { RobotsResult } from "@/server/lib/audit/discovery";-import type { StepPageResult } from "@/server/lib/audit/types";+import type { StepPageSummary } from "@/server/lib/audit/types";import { isSameOrigin, normalizeUrl } from "@/server/lib/audit/url-utils";+import { isCrawlableUrl } from "@/server/lib/audit/url-policy";+import { deterministicAuditRowId } from "@/server/lib/audit/ids";+import { runPageReporters } from "@/server/lib/audit/issues/page-reporters";import { AuditRepository } from "@/server/features/audit/repositories/AuditRepository";import { AuditProgressKV } from "@/server/lib/audit/progress-kv";import { crawlPage } from "@/server/workflows/site-audit-workflow-helpers";import { pgStep } from "@/server/workflows/pgStep";const CRAWL_CONCURRENCY = 25;+// Keep durable step state under the ~1MiB limit: full link lists live in D1;+// the step return only carries new-to-the-frontier targets, deduped across+// the batch and capped.+const MAX_FRONTIER_LINKS_PER_BATCH = 2_000;+const MAX_SUMMARY_TITLE_CHARS = 300;function shouldQueueCrawlLink(link: string,@@ -18,12 +26,19 @@ function shouldQueueCrawlLink(): boolean {return (isSameOrigin(link, origin) &&+ isCrawlableUrl(link) &&robots.isAllowed(link) &&!visited.has(link) &&!queued.has(link));}+interface QueueEntry {+ url: string;+ /** Clicks from the start URL; null when only reachable via sitemap. */+ depth: number | null;+}+type CrawlPhaseParams = {auditId: string;workflowInstanceId: string;@@ -34,10 +49,19 @@ type CrawlPhaseParams = {sitemapUrls: string[];};+/** What later phases need per page — no link lists (those stay in D1). */+export type CrawledPageSummary = Omit<StepPageSummary, "internalLinks">;++export type CrawlPhaseResult = {+ pages: CrawledPageSummary[];+ /** True when the frontier was exhausted before hitting maxPages. */+ completed: boolean;+};+export async function runCrawlPhase(step: WorkflowStep,params: CrawlPhaseParams,-): Promise<StepPageResult[]> {+): Promise<CrawlPhaseResult> {const {auditId,workflowInstanceId,@@ -48,43 +72,69 @@ export async function runCrawlPhase(sitemapUrls,} = params;const visited = new Set<string>();- const queue: string[] = [];const queued = new Set<string>();- const allPages: StepPageResult[] = [];+ // Link-discovered URLs crawl first (BFS from the start URL); sitemap-only+ // URLs drain last so link discovery isn't starved of page budget and+ // orphan detection stays meaningful.+ const linkQueue: QueueEntry[] = [];+ const sitemapQueue: QueueEntry[] = [];+ const sitemapSet = new Set<string>();+ const summaries: CrawledPageSummary[] = [];- seedCrawlQueue({- startUrl,- origin,- robots,- sitemapUrls,- visited,- queued,- queue,- });+ const normalizedStart = normalizeUrl(startUrl) ?? startUrl;+ if (+ robots.isAllowed(normalizedStart) &&+ isSameOrigin(normalizedStart, origin)+ ) {+ linkQueue.push({ url: normalizedStart, depth: 0 });+ queued.add(normalizedStart);+ }++ for (const sitemapUrl of sitemapUrls) {+ const normalized = normalizeUrl(sitemapUrl);+ if (!normalized) continue;+ sitemapSet.add(normalized);+ if (!shouldQueueCrawlLink(normalized, origin, robots, visited, queued)) {+ continue;+ }+ sitemapQueue.push({ url: normalized, depth: null });+ queued.add(normalized);+ }let crawlBatchIndex = 0;- while (queue.length > 0 && allPages.length < maxPages) {- const urlsToCrawl = selectNextCrawlBatch(- queue,+ while (+ (linkQueue.length > 0 || sitemapQueue.length > 0) &&+ summaries.length < maxPages+ ) {+ const batchEntries = selectNextCrawlBatch({+ linkQueue,+ sitemapQueue,queued,visited,robots,- maxPages - allPages.length,- );- if (urlsToCrawl.length === 0) continue;+ remaining: maxPages - summaries.length,+ });+ if (batchEntries.length === 0) continue;crawlBatchIndex += 1;- const crawledBatch = await runCrawlBatch(- step,+ const crawledBatch = await runCrawlBatch(step, {crawlBatchIndex,- urlsToCrawl,- origin,+ auditId,+ batchEntries,+ sitemapSet,+ visited,+ queued,+ });+ // Keep only the slim summary in memory: at 10k pages, retaining link+ // lists for the whole crawl would not fit in the 128MB Worker heap.+ summaries.push(+ ...crawledBatch.map(({ internalLinks: _links, ...summary }) => summary),);- allPages.push(...crawledBatch);enqueueDiscoveredLinks({crawledBatch,- queue,+ batchEntries,+ linkQueue,queued,visited,origin,@@ -96,110 +146,151 @@ export async function runCrawlPhase(auditId,workflowInstanceId,crawledBatch,- pagesCrawled: allPages.length,+ pagesCrawled: summaries.length,visitedCount: visited.size,- queueLength: queue.length,+ queueLength: linkQueue.length + sitemapQueue.length,maxPages,});}- return allPages;+ return {+ pages: summaries,+ completed: linkQueue.length === 0 && sitemapQueue.length === 0,+ };}-function seedCrawlQueue({- startUrl,- origin,- robots,- sitemapUrls,- visited,- queued,- queue,-}: {- startUrl: string;- origin: string;- robots: RobotsResult;- sitemapUrls: string[];- visited: Set<string>;+function selectNextCrawlBatch(params: {+ linkQueue: QueueEntry[];+ sitemapQueue: QueueEntry[];queued: Set<string>;- queue: string[];+ visited: Set<string>;+ robots: RobotsResult;+ remaining: number;}) {- const normalizedStart = normalizeUrl(startUrl) ?? startUrl;- if (- robots.isAllowed(normalizedStart) &&- isSameOrigin(normalizedStart, origin)- ) {- queue.push(normalizedStart);- queued.add(normalizedStart);- }-- for (const sitemapUrl of sitemapUrls) {- const normalized = normalizeUrl(sitemapUrl);- if (!normalized) continue;- if (!shouldQueueCrawlLink(normalized, origin, robots, visited, queued)) {- continue;- }- queue.push(normalized);- queued.add(normalized);- }-}--function selectNextCrawlBatch(- queue: string[],- queued: Set<string>,- visited: Set<string>,- robots: RobotsResult,- remaining: number,-) {+ const { linkQueue, sitemapQueue, queued, visited, robots, remaining } =+ params;const batchSize = Math.min(CRAWL_CONCURRENCY, remaining);- const urlsToCrawl: string[] = [];-- while (queue.length > 0 && urlsToCrawl.length < batchSize) {- const url = queue.shift()!;- queued.delete(url);- if (visited.has(url)) continue;- if (!robots.isAllowed(url)) continue;- visited.add(url);- urlsToCrawl.push(url);+ const batchEntries: QueueEntry[] = [];++ while (+ (linkQueue.length > 0 || sitemapQueue.length > 0) &&+ batchEntries.length < batchSize+ ) {+ const entry = (linkQueue.length > 0 ? linkQueue : sitemapQueue).shift()!;+ queued.delete(entry.url);+ if (visited.has(entry.url)) continue;+ if (!robots.isAllowed(entry.url)) continue;+ visited.add(entry.url);+ batchEntries.push(entry);}- return urlsToCrawl;+ return batchEntries;}async function runCrawlBatch(step: WorkflowStep,- crawlBatchIndex: number,- urlsToCrawl: string[],- origin: string,-): Promise<StepPageResult[]> {- return step.do(`crawl-batch-${crawlBatchIndex}`, async () => {- const settled = await Promise.allSettled(- urlsToCrawl.map((url) => crawlPage(url, origin)),+ input: {+ crawlBatchIndex: number;+ auditId: string;+ batchEntries: QueueEntry[];+ sitemapSet: Set<string>;+ visited: Set<string>;+ queued: Set<string>;+ },+): Promise<StepPageSummary[]> {+ const {+ crawlBatchIndex,+ auditId,+ batchEntries,+ sitemapSet,+ visited,+ queued,+ } = input;+ return pgStep(step, `crawl-batch-${crawlBatchIndex}`, undefined, async () => {+ const pages = await Promise.all(+ batchEntries.map((entry) =>+ crawlPage(entry.url, entry.depth, sitemapSet.has(entry.url)),+ ),);- return settled.flatMap((result) => {- if (result.status === "fulfilled" && result.value) {- return [result.value];++ // Deterministic ids keep the D1 writes idempotent across step retries.+ for (const page of pages) {+ page.id = await deterministicAuditRowId(auditId, page.url);+ }++ const issues = pages.flatMap((page) => runPageReporters(page));+ await AuditRepository.insertCrawledBatch(auditId, pages, issues);++ // Frontier candidates only: drop targets already visited/queued and+ // dedupe across the batch, so the step return stays far under the+ // ~1MiB durable-state limit even on mega-menu sites.+ const seenTargets = new Set<string>();+ return pages.map((page) => {+ const internalLinks: string[] = [];+ for (const link of page.links) {+ if (!link.isInternal) continue;+ if (seenTargets.size >= MAX_FRONTIER_LINKS_PER_BATCH) break;+ if (visited.has(link.targetUrl) || queued.has(link.targetUrl)) continue;+ if (seenTargets.has(link.targetUrl)) continue;+ seenTargets.add(link.targetUrl);+ internalLinks.push(link.targetUrl);}- return [];+ return {+ id: page.id,+ url: page.url,+ statusCode: page.statusCode,+ fetchClass: page.fetchClass,+ redirectUrl: page.redirectUrl,+ title: page.title.slice(0, MAX_SUMMARY_TITLE_CHARS),+ internalLinks,+ };});});}function enqueueDiscoveredLinks(params: {- crawledBatch: StepPageResult[];- queue: string[];+ crawledBatch: StepPageSummary[];+ batchEntries: QueueEntry[];+ linkQueue: QueueEntry[];queued: Set<string>;visited: Set<string>;origin: string;robots: RobotsResult;}) {- const { crawledBatch, queue, queued, visited, origin, robots } = params;+ const {+ crawledBatch,+ batchEntries,+ linkQueue,+ queued,+ visited,+ origin,+ robots,+ } = params;+ const depthByUrl = new Map(+ batchEntries.map((entry) => [entry.url, entry.depth]),+ );+for (const pageResult of crawledBatch) {- for (const link of pageResult.internalLinks.filter((candidate) =>- shouldQueueCrawlLink(candidate, origin, robots, visited, queued),- )) {- queue.push(link);+ const pageDepth = depthByUrl.get(pageResult.url) ?? null;+ const childDepth = pageDepth === null ? null : pageDepth + 1;++ for (const link of pageResult.internalLinks) {+ if (!shouldQueueCrawlLink(link, origin, robots, visited, queued)) {+ continue;+ }+ linkQueue.push({ url: link, depth: childDepth });queued.add(link);}++ // Redirect targets continue the same navigation path: same depth.+ const redirectTarget = pageResult.redirectUrl;+ if (+ redirectTarget &&+ shouldQueueCrawlLink(redirectTarget, origin, robots, visited, queued)+ ) {+ linkQueue.push({ url: redirectTarget, depth: pageDepth });+ queued.add(redirectTarget);+ }}}@@ -208,7 +299,7 @@ async function persistCrawlProgress(params: {crawlBatchIndex: number;auditId: string;workflowInstanceId: string;- crawledBatch: StepPageResult[];+ crawledBatch: StepPageSummary[];pagesCrawled: number;visitedCount: number;queueLength: number;@@ -225,23 +316,24 @@ async function persistCrawlProgress(params: {queueLength,maxPages,} = params;- await step.do(`kv-progress-batch-${crawlBatchIndex}`, async () => {- await AuditProgressKV.pushCrawledUrls(- auditId,- crawledBatch.map((pageResult) => ({- url: pageResult.url,- statusCode: pageResult.statusCode,- title: pageResult.title,- crawledAt: Date.now(),- })),- );- });-+ // KV push + D1 progress in one step — merging them halves the per-batch+ // step count against the ~1k step budget. The D1 update is idempotent; the+ // KV push can duplicate entries on a partial retry, which is acceptable for+ // an ephemeral progress feed (capped list, short TTL).await pgStep(step,`progress-batch-${crawlBatchIndex}`,undefined,async () => {+ await AuditProgressKV.pushCrawledUrls(+ auditId,+ crawledBatch.map((pageResult) => ({+ url: pageResult.url,+ statusCode: pageResult.statusCode,+ title: pageResult.title,+ crawledAt: Date.now(),+ })),+ );await AuditRepository.updateAuditProgress(auditId, workflowInstanceId, {pagesCrawled,pagesTotal: Math.min(visitedCount + queueLength, maxPages),diff --git a/src/server/workflows/siteAuditWorkflowPhases.ts b/src/server/workflows/siteAuditWorkflowPhases.tsindex 034a481f..19468025 100644--- a/src/server/workflows/siteAuditWorkflowPhases.ts+++ b/src/server/workflows/siteAuditWorkflowPhases.ts@@ -1,6 +1,6 @@import type { WorkflowStep } from "cloudflare:workers";import type { BillingCustomerContext } from "@/server/billing/subscription";-import { discoverUrls, fetchRobotsTxt } from "@/server/lib/audit/discovery";+import { discoverUrls, parseRobotsTxt } from "@/server/lib/audit/discovery";import {fetchAndStoreLighthouseResult,selectLighthouseSample,@@ -8,13 +8,14 @@ import {import { getOrigin } from "@/server/lib/audit/url-utils";import { AuditRepository } from "@/server/features/audit/repositories/AuditRepository";import { AuditProgressKV } from "@/server/lib/audit/progress-kv";-import type {- AuditConfig,- LighthouseResult,- StepPageResult,-} from "@/server/lib/audit/types";+import { runMultipageChecks } from "@/server/lib/audit/issues/multipage";+import type { AuditConfig } from "@/server/lib/audit/types";import { captureServerEvent } from "@/server/lib/posthog";-import { runCrawlPhase } from "@/server/workflows/siteAuditWorkflowCrawl";+import {+ runCrawlPhase,+ type CrawledPageSummary,+ type CrawlPhaseResult,+} from "@/server/workflows/siteAuditWorkflowCrawl";import { pgStep } from "@/server/workflows/pgStep";const LIGHTHOUSE_URL_BATCH_SIZE = 10;@@ -35,22 +36,6 @@ function capSitemapSeeds(urls: string[], maxPages: number): string[] {return seeds;}-function countLighthouseBatchResults(results: LighthouseResult[]): {- completed: number;- failed: number;-} {- let completed = 0;- let failed = 0;- for (const result of results) {- if (result.errorMessage) {- failed += 1;- continue;- }- completed += 1;- }- return { completed, failed };-}-type AuditPhasesParams = {auditId: string;workflowInstanceId: string;@@ -82,8 +67,11 @@ export async function runAuditPhases(origin,maxPages,);- const robots = await fetchRobotsTxt(origin);- const allPages = await runCrawlPhase(step, {+ // Parsed outside the step from checkpointed text, so replays see the exact+ // robots rules the original run used (a live re-fetch could differ and+ // desync the frontier from already-persisted crawl batches).+ const robots = parseRobotsTxt(origin, discovery.robotsText);+ const crawl = await runCrawlPhase(step, {auditId,workflowInstanceId,origin,@@ -92,14 +80,14 @@ export async function runAuditPhases(robots,sitemapUrls: discovery.sitemapUrls,});- const lighthouseResults = await runLighthousePhase(step, {+ await runLighthousePhase(step, {auditId,workflowInstanceId,billingCustomer,projectId,startUrl,config,- allPages,+ pages: crawl.pages,});await finalizeAudit({step,@@ -107,9 +95,9 @@ export async function runAuditPhases(workflowInstanceId,billingCustomer,projectId,+ startUrl,config,- allPages,- lighthouseResults,+ crawl,});}@@ -126,7 +114,10 @@ async function runDiscoveryPhase(pagesTotal: Math.min(result.urls.length + 1, maxPages),currentPhase: "crawling",});- return { sitemapUrls: capSitemapSeeds(result.urls, maxPages) };+ return {+ sitemapUrls: capSitemapSeeds(result.urls, maxPages),+ robotsText: result.robotsText,+ };});}@@ -137,13 +128,13 @@ type LighthousePhaseParams = {projectId: string;startUrl: string;config: AuditConfig;- allPages: StepPageResult[];+ pages: CrawledPageSummary[];};async function runLighthousePhase(step: WorkflowStep,params: LighthousePhaseParams,-): Promise<LighthouseResult[]> {+) {const {auditId,workflowInstanceId,@@ -151,20 +142,19 @@ async function runLighthousePhase(projectId,startUrl,config,- allPages,+ pages,} = params;- if (config.lighthouseStrategy === "none") return [];+ if (config.lighthouseStrategy === "none") return;const lighthouseWork = await selectLighthousePages({step,auditId,workflowInstanceId,- allPages,+ pages,startUrl,strategy: config.lighthouseStrategy,});- const lighthouseResults: LighthouseResult[] = [];let completedChecks = 0;let failedChecks = 0;let lighthouseBatchIndex = 0;@@ -172,47 +162,69 @@ async function runLighthousePhase(for (let i = 0; i < lighthouseWork.length; i += LIGHTHOUSE_URL_BATCH_SIZE) {const batch = lighthouseWork.slice(i, i + LIGHTHOUSE_URL_BATCH_SIZE);lighthouseBatchIndex += 1;- const lighthouseBatchResults = await runLighthouseBatch({- step,- lighthouseBatchIndex,- batch,- billingCustomer,- projectId,- auditId,- });+ const priorCompleted = completedChecks;+ const priorFailed = failedChecks;- lighthouseResults.push(...lighthouseBatchResults);- const counts = countLighthouseBatchResults(lighthouseBatchResults);- failedChecks += counts.failed;- completedChecks += counts.completed;- await pgStep(+ // Fetch, store (R2 + D1) and update progress inside one step. The step+ // returns only counts; full results live in D1.+ const counts = await pgStep(step,- `lighthouse-progress-batch-${lighthouseBatchIndex}`,+ `lighthouse-batch-${lighthouseBatchIndex}`,undefined,async () => {+ const perUrlResults = await Promise.all(+ batch.map(async ({ url, pageId }) => {+ const [mobileResult, desktopResult] = await Promise.all([+ fetchAndStoreLighthouseResult({+ url,+ pageId,+ strategy: "mobile",+ billingCustomer,+ projectId,+ auditId,+ }),+ fetchAndStoreLighthouseResult({+ url,+ pageId,+ strategy: "desktop",+ billingCustomer,+ projectId,+ auditId,+ }),+ ]);+ return [mobileResult, desktopResult];+ }),+ );+ const results = perUrlResults.flat();+ await AuditRepository.insertLighthouseResults(auditId, results);++ const failed = results.filter((result) => result.errorMessage).length;+ const completed = results.length - failed;await AuditRepository.updateAuditProgress(auditId, workflowInstanceId, {- lighthouseCompleted: completedChecks,- lighthouseFailed: failedChecks,+ lighthouseCompleted: priorCompleted + completed,+ lighthouseFailed: priorFailed + failed,});+ return { completed, failed };},);- }- return lighthouseResults;+ completedChecks += counts.completed;+ failedChecks += counts.failed;+ }}async function selectLighthousePages(params: {step: WorkflowStep;auditId: string;workflowInstanceId: string;- allPages: StepPageResult[];+ pages: CrawledPageSummary[];startUrl: string;strategy: AuditConfig["lighthouseStrategy"];}) {- const { step, auditId, workflowInstanceId, allPages, startUrl, strategy } =+ const { step, auditId, workflowInstanceId, pages, startUrl, strategy } =params;return pgStep(step, "select-lighthouse-sample", undefined, async () => {- const sample = selectLighthouseSample(allPages, startUrl, strategy);+ const sample = selectLighthouseSample(pages, startUrl, strategy);const selectedUrls = new Set(sample);await AuditRepository.updateAuditProgress(auditId, workflowInstanceId, {@@ -221,66 +233,21 @@ async function selectLighthousePages(params: {lighthouseCompleted: 0,lighthouseFailed: 0,});- return allPages.flatMap((page) =>+ return pages.flatMap((page) =>selectedUrls.has(page.url) ? [{ url: page.url, pageId: page.id }] : [],);});}-async function runLighthouseBatch(params: {- step: WorkflowStep;- lighthouseBatchIndex: number;- batch: Array<{ url: string; pageId: string }>;- billingCustomer: BillingCustomerContext;- projectId: string;- auditId: string;-}) {- const {- step,- lighthouseBatchIndex,- batch,- billingCustomer,- projectId,- auditId,- } = params;- return step.do(`lighthouse-batch-${lighthouseBatchIndex}`, async () => {- const perUrlResults = await Promise.all(- batch.map(async ({ url, pageId }) => {- const [mobileResult, desktopResult] = await Promise.all([- fetchAndStoreLighthouseResult({- url,- pageId,- strategy: "mobile",- billingCustomer,- projectId,- auditId,- }),- fetchAndStoreLighthouseResult({- url,- pageId,- strategy: "desktop",- billingCustomer,- projectId,- auditId,- }),- ]);- return [mobileResult, desktopResult];- }),- );-- return perUrlResults.flat();- });-}-async function finalizeAudit(args: {step: WorkflowStep;auditId: string;workflowInstanceId: string;billingCustomer: BillingCustomerContext;projectId: string;+ startUrl: string;config: AuditConfig;- allPages: StepPageResult[];- lighthouseResults: LighthouseResult[];+ crawl: CrawlPhaseResult;}) {const {step,@@ -288,23 +255,42 @@ async function finalizeAudit(args: {workflowInstanceId,billingCustomer,projectId,+ startUrl,config,- allPages,- lighthouseResults,+ crawl,} = args;- await pgStep(step, "finalize", undefined, async () => {+ await pgStep(step, "multipage-checks", undefined, async () => {await AuditRepository.updateAuditProgress(auditId, workflowInstanceId, {currentPhase: "finalizing",});- await AuditRepository.batchWriteResults(++ // Integrity guard: pages are persisted inside crawl-batch steps. If the+ // crawl claims pages but D1 has none (e.g. an instance started under the+ // pre-incremental-persistence code was replayed under this code), fail+ // loudly instead of completing with an empty audit.+ if (+ crawl.pages.length > 0 &&+ !(await AuditRepository.hasPagesForAudit(auditId))+ ) {+ throw new Error(+ `Audit ${auditId}: crawl reported ${crawl.pages.length} pages but none were persisted`,+ );+ }++ const issues = await runMultipageChecks({auditId,- allPages,- lighthouseResults,- );+ startUrl,+ crawlCompleted: crawl.completed,+ });+ await AuditRepository.insertIssues(auditId, issues);+ return { issueCount: issues.length };+ });++ await pgStep(step, "finalize", undefined, async () => {await AuditRepository.completeAudit(auditId, workflowInstanceId, {- pagesCrawled: allPages.length,- pagesTotal: allPages.length,+ pagesCrawled: crawl.pages.length,+ pagesTotal: crawl.pages.length,});await captureServerEvent({distinctId: billingCustomer.userId,@@ -313,8 +299,12 @@ async function finalizeAudit(args: {properties: {project_id: projectId,status: "completed",- pages_crawled: allPages.length,- pages_total: allPages.length,+ pages_crawled: crawl.pages.length,+ pages_total: crawl.pages.length,+ crawl_completed: crawl.completed,+ pages_blocked: crawl.pages.filter(+ (page) => page.fetchClass === "blocked",+ ).length,run_lighthouse: config.lighthouseStrategy !== "none",},});diff --git a/src/serverFunctions/audit.ts b/src/serverFunctions/audit.tsindex 050573e8..eb56fefc 100644--- a/src/serverFunctions/audit.ts+++ b/src/serverFunctions/audit.ts@@ -1,14 +1,7 @@import { createServerFn } from "@tanstack/react-start";import { waitUntil } from "cloudflare:workers";import { AuditService } from "@/server/features/audit/services/AuditService";-import type { AuditLimitTier } from "@/server/features/audit/services/audit-capacity";-import {- customerHasManagedAccess,- customerHasPaidPlan,-} from "@/server/billing/subscription";-import { AppError } from "@/server/lib/errors";import { captureServerEvent } from "@/server/lib/posthog";-import { isHostedServerAuthMode } from "@/server/lib/runtime-env";import { requireProjectContext } from "@/serverFunctions/middleware";import {deleteAuditSchema,@@ -23,21 +16,9 @@ export const startAudit = createServerFn({ method: "POST" }).middleware(requireProjectContext).validator(startAuditSchema).handler(async ({ data, context }) => {- // The crawler runs on our Workers compute and isn't credit-metered, so- // plan-tier limits are the abuse bound in hosted mode: free accounts get- // one small audit at a time, paid keeps the full limits, and customers- // with no Autumn product at all are turned away. Self-hosted isn't gated.- let limitTier: AuditLimitTier = "paid";- if (await isHostedServerAuthMode()) {- const [hasManagedAccess, hasPaidPlan] = await Promise.all([- customerHasManagedAccess(context.organizationId),- customerHasPaidPlan(context.organizationId),- ]);- if (!hasManagedAccess) {- throw new AppError("PAYMENT_REQUIRED", "Subscribe to run site audits");- }- limitTier = hasPaidPlan ? "paid" : "free";- }+ const limitTier = await AuditService.resolveAuditLimitTier(+ context.organizationId,+ );const result = await AuditService.startAudit({actorUserId: context.userId,diff --git a/src/shared/audit-issues.ts b/src/shared/audit-issues.tsnew file mode 100644index 00000000..c3f13bad--- /dev/null+++ b/src/shared/audit-issues.ts@@ -0,0 +1,235 @@+/**+ * Registry of site-audit issue types.+ *+ * Shared between the server (issue engine, MCP tools) and the client+ * (issues UI, CSV export). Each issue row in `audit_issues` references one+ * of these types by id.+ */++type IssueSeverity = "critical" | "warning" | "info";++interface AuditIssueDescriptor {+ severity: IssueSeverity;+ title: string;+ explanation: string;+ howToFix: string;+}++export const AUDIT_ISSUE_TYPES = {+ "blocked-page": {+ severity: "critical",+ title: "Crawler was blocked",+ explanation:+ "The site returned a bot challenge or access denial (e.g. a Cloudflare challenge, 403, or 429) instead of the page. We report this honestly rather than pretending the page is broken — but it means this page could not be audited, and other crawlers like search engines may face similar friction.",+ howToFix:+ 'If you own this site, allowlist the "OpenSEO-Audit" user agent in your WAF/bot-protection settings (on Cloudflare: a WAF custom rule that skips bot protection when the user agent contains "OpenSEO-Audit"; on some free tiers you may need to relax bot protection). Then re-run the audit.',+ },+ "server-error": {+ severity: "critical",+ title: "Server error (5xx)",+ explanation:+ "The page returned a 5xx server error. Search engines that repeatedly see server errors will crawl the site less and may drop the page from the index.",+ howToFix:+ "Check the server logs for this URL and fix the underlying error. If the page is gone, return a 404/410 or redirect it to a relevant page instead of erroring.",+ },+ "broken-internal-link": {+ severity: "critical",+ title: "Broken internal link",+ explanation:+ "This page links to an internal URL that returns an error status (4xx/5xx). Broken links waste crawl budget, leak link equity, and frustrate users — they are among the most common and most damaging technical SEO issues.",+ howToFix:+ "Update the link to point at the correct live URL, or remove it. If the target was moved, prefer linking directly to the new URL rather than relying on a redirect.",+ },+ "missing-title": {+ severity: "critical",+ title: "Missing title tag",+ explanation:+ "The page has no <title>. The title is the strongest on-page relevance signal and the headline shown in search results; without it search engines generate one themselves, usually badly.",+ howToFix:+ "Add a unique, descriptive <title> of roughly 50–60 characters that includes the page's primary topic.",+ },+ "broken-page": {+ severity: "warning",+ title: "Page returns an error (4xx)",+ explanation:+ "This crawled URL returned a client error (e.g. 404). If it is referenced from your sitemap or other pages, crawlers keep wasting requests on it.",+ howToFix:+ "If the page should exist, restore it. If it is intentionally gone, remove it from the sitemap and internal links, and consider a 301 redirect to the closest live page.",+ },+ "duplicate-title": {+ severity: "warning",+ title: "Duplicate title",+ explanation:+ "Multiple pages share the same title tag. Search engines use titles to differentiate pages; duplicates make pages compete with each other and depress click-through rates.",+ howToFix:+ "Write a unique title for each page describing its specific content. For templated pages, include the distinguishing attribute (name, category, location) in the template.",+ },+ "duplicate-meta-description": {+ severity: "warning",+ title: "Duplicate meta description",+ explanation:+ "Multiple pages share the same meta description, so search results show identical snippets and users cannot tell the pages apart.",+ howToFix:+ "Write a unique meta description per page, or remove the duplicated one entirely — search engines will generate a snippet from page content, which beats a wrong duplicate.",+ },+ "duplicate-content": {+ severity: "warning",+ title: "Duplicate page content",+ explanation:+ "Two or more URLs serve byte-identical visible text. Search engines pick one version to index and ignore the rest, and ranking signals get split across the duplicates.",+ howToFix:+ "Consolidate duplicates: pick the canonical URL, add rel=canonical from the others, and 301-redirect duplicate URLs where possible (common causes: trailing-slash variants, URL parameters, http/https or www variants).",+ },+ "missing-meta-description": {+ severity: "warning",+ title: "Missing meta description",+ explanation:+ "The page has no meta description. Search engines will assemble a snippet from page text, which is often less compelling and hurts click-through rate.",+ howToFix:+ "Add a meta description of roughly 70–160 characters that summarizes the page and gives a reason to click.",+ },+ "missing-h1": {+ severity: "warning",+ title: "Missing H1 heading",+ explanation:+ "The page has no H1. The H1 tells users and search engines what the page is about; pages without one tend to have weaker topical clarity.",+ howToFix:+ "Add a single H1 that states the page's main topic, consistent with the title tag.",+ },+ "multiple-h1": {+ severity: "warning",+ title: "Multiple H1 headings",+ explanation:+ "The page has more than one H1, which dilutes the main-topic signal and usually indicates a templating mistake (e.g. a logo and a headline both marked up as H1).",+ howToFix:+ "Keep one H1 for the page's main heading and demote the others to H2/H3 (or unstyled elements for non-headings like logos).",+ },+ "redirect-chain": {+ severity: "warning",+ title: "Redirect chain",+ explanation:+ "Reaching the final page requires two or more consecutive redirects. Each hop adds latency, leaks link equity, and burns crawl budget; long chains may not be followed at all.",+ howToFix:+ "Point the first URL (and any internal links) directly at the final destination so there is at most one redirect.",+ },+ "redirect-loop": {+ severity: "warning",+ title: "Redirect loop",+ explanation:+ "This redirect eventually points back to itself, so the URL never resolves. Browsers and crawlers give up with an error.",+ howToFix:+ "Trace the redirect rules for this URL and break the cycle so the chain terminates at a real 200 page.",+ },+ "canonical-conflict": {+ severity: "warning",+ title: "Conflicting canonical signals",+ explanation:+ "The page declares different canonical URLs in its HTML <link rel=canonical> and its HTTP Link header. When signals conflict, search engines ignore both and choose their own canonical.",+ howToFix:+ "Pick one canonical URL and declare it in exactly one place (HTML head is the most common); remove or align the other declaration.",+ },+ "thin-content": {+ severity: "warning",+ title: "Thin content",+ explanation:+ "The page has very little visible text. Thin pages rarely rank, can drag down sitewide quality assessments, and (if the site renders client-side) may indicate content invisible to plain-HTML crawlers.",+ howToFix:+ "Either expand the page with genuinely useful content, noindex it, or consolidate it into a stronger page. If the content exists but is rendered by JavaScript, ensure it is server-rendered or pre-rendered.",+ },+ "images-missing-alt": {+ severity: "warning",+ title: "Images missing alt text",+ explanation:+ "One or more images on the page lack alt attributes. Alt text is an accessibility requirement and the main way search engines understand images.",+ howToFix:+ 'Add descriptive alt text to meaningful images; use an empty alt (alt="") only for purely decorative ones.',+ },+ "orphan-page": {+ severity: "warning",+ title: "Orphan page",+ explanation:+ "No crawled page links to this URL — it was only discoverable via the sitemap. Pages without internal links receive little crawl attention and no internal link equity, and users can't find them by browsing.",+ howToFix:+ "Link to this page from relevant pages (navigation, related content, hub pages), or remove it from the sitemap if it shouldn't be indexed.",+ },+ "title-too-long": {+ severity: "info",+ title: "Title too long",+ explanation:+ "The title exceeds ~60 characters, so search results will truncate it and the ending may be cut off mid-phrase.",+ howToFix:+ "Shorten the title to roughly 50–60 characters, front-loading the most important words.",+ },+ "title-too-short": {+ severity: "info",+ title: "Title too short",+ explanation:+ "The title is under ~10 characters, which is usually too generic to describe the page or attract clicks.",+ howToFix:+ "Expand the title into a descriptive phrase (roughly 30–60 characters) that states what the page offers.",+ },+ "meta-description-too-long": {+ severity: "info",+ title: "Meta description too long",+ explanation:+ "The meta description exceeds ~160 characters, so search engines will truncate the snippet.",+ howToFix:+ "Trim the description to roughly 70–160 characters while keeping the core message and call to action.",+ },+ "heading-order-skip": {+ severity: "info",+ title: "Heading levels skip",+ explanation:+ "The heading hierarchy skips levels (e.g. an H4 directly after an H2). This weakens document structure for accessibility tools and content parsing.",+ howToFix:+ "Adjust heading levels so they descend one step at a time (H1 → H2 → H3) without skipping.",+ },+ "slow-response": {+ severity: "info",+ title: "Slow server response",+ explanation:+ "The HTML response took over 1.5 seconds. Slow time-to-first-byte drags down every downstream performance metric and reduces crawl rate on large sites.",+ howToFix:+ "Investigate server/database time and caching for this route; serving cached or statically generated HTML usually fixes it.",+ },+ "noindex-page": {+ severity: "info",+ title: "Page is noindex",+ explanation:+ "The page asks search engines not to index it (via robots meta tag or X-Robots-Tag header). That's often intentional — this is a heads-up, not an error.",+ howToFix:+ "If this page should rank, remove the noindex directive. If it's intentional (admin, thank-you, filter pages), no action is needed.",+ },+ "canonicalized-page": {+ severity: "info",+ title: "Canonicalized to another URL",+ explanation:+ "The page declares a different URL as its canonical, telling search engines to index that URL instead. Fine when intentional (parameter pages, syndication) — a problem if this page was meant to rank.",+ howToFix:+ "If this page should rank on its own, set its canonical to itself. Otherwise no action is needed.",+ },+ "deep-page": {+ severity: "info",+ title: "Page is deep in the site structure",+ explanation:+ "The page is 5+ clicks from the homepage. Deep pages get crawled less often and receive less link equity.",+ howToFix:+ "Add links from higher-level pages (hubs, category pages, navigation) to flatten the path to this page.",+ },+} as const satisfies Record<string, AuditIssueDescriptor>;++export type AuditIssueType = keyof typeof AUDIT_ISSUE_TYPES;++export const ISSUE_SEVERITY_ORDER: Record<IssueSeverity, number> = {+ critical: 0,+ warning: 1,+ info: 2,+};++const issueRegistry: Record<string, AuditIssueDescriptor> = AUDIT_ISSUE_TYPES;++export function getIssueDescriptor(+ issueType: string,+): AuditIssueDescriptor | null {+ return issueRegistry[issueType] ?? null;+}diff --git a/src/types/schemas/audit.ts b/src/types/schemas/audit.tsindex b12c783c..864f3f4b 100644--- a/src/types/schemas/audit.ts+++ b/src/types/schemas/audit.ts@@ -17,10 +17,7 @@ export const startAuditSchema = z.object({.max(PAID_MAX_AUDIT_PAGES).optional().default(DEFAULT_AUDIT_PAGES),- lighthouseStrategy: z- .enum(["auto", "manual", "none"])- .optional()- .default("auto"),+ lighthouseStrategy: z.enum(["auto", "none"]).optional().default("auto"),});export const getAuditStatusSchema = z.object({