Skip to content

Open-source SEO, GEO, and marketing skills for AI agents.

feat(seo): integrate PageSpeed Insights API for performance monitoring (#33)

Tong Chen519de413306c
diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json
index 0162167c..a14cd744 100644
--- a/.claude-plugin/marketplace.json
+++ b/.claude-plugin/marketplace.json
@@ -5,14 +5,14 @@
},
"metadata": {
"description": "SEO and Google Ads skills for Claude Code",
- "version": "0.9.9"
+ "version": "0.10.0"
},
"plugins": [
{
"name": "toprank",
"source": "./",
"description": "SEO analysis, Google Ads management, keyword research, content optimization, schema markup, and meta tags optimization.",
- "version": "0.9.9",
+ "version": "0.10.0",
"author": {
"name": "nowork-studio"
},
diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.json
index 80637edf..0c2212a7 100644
--- a/.claude-plugin/plugin.json
+++ b/.claude-plugin/plugin.json
@@ -1,6 +1,6 @@
{
"name": "toprank",
- "version": "0.9.9",
+ "version": "0.10.0",
"description": "SEO analysis, Google Ads management, keyword research, content optimization, and schema markup tools powered by Google Search Console and Google Ads APIs.",
"author": {
"name": "nowork-studio"
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 9d333d03..f2696a13 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -6,6 +6,16 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
---
+## [0.10.0] — 2026-04-08
+
+### Added
+- **PageSpeed Insights API integration** — New `pagespeed.py` script calls the Google PageSpeed Insights API for multiple URLs with concurrent execution. Collects Lighthouse performance scores, Core Web Vitals (LCP, INP, CLS, FCP, TTFB), CrUX field data, optimization opportunities, and diagnostics. Supports both mobile and desktop strategies.
+- **PageSpeed display script** — New `show_pagespeed.py` renders PageSpeed results in a terminal-friendly format with scores, metrics, opportunities, and diagnostics.
+- **Phase 5.5 in SEO analysis** — PageSpeed analysis runs in parallel alongside URL Inspection and CMS detection during audits. Results feed into the report with a dedicated "PageSpeed & Core Web Vitals" section and are logged in the audit history for tracking performance over time.
+- **Preflight PageSpeed checks** — `preflight.py` now auto-enables the PageSpeed Insights API and checks for a `PAGESPEED_API_KEY` in the environment or `~/.toprank/.env`.
+
+---
+
## [0.9.9] — 2026-04-08
### Changed
diff --git a/VERSION b/VERSION
index 7e310bae..78bc1abd 100644
--- a/VERSION
+++ b/VERSION
@@ -1 +1 @@
-0.9.9
+0.10.0
diff --git a/seo/seo-analysis/SKILL.md b/seo/seo-analysis/SKILL.md
index e6b831d4..939fce83 100644
--- a/seo/seo-analysis/SKILL.md
+++ b/seo/seo-analysis/SKILL.md
@@ -2,21 +2,25 @@
name: seo-analysis
argument-hint: "<URL to audit, e.g. https://example.com>"
description: >
- Full SEO audit: Google Search Console data + URL Inspection API + technical
- crawl + keyword research + metadata audit + schema markup audit + search
- intent analysis. Feeds real GSC data into AI to surface quick wins, diagnose
- traffic drops, find content gaps, identify metadata mismatches, detect schema
- gaps, and produce an actionable 30-day plan. Use this skill whenever the user
- asks about SEO, search rankings, organic traffic, Google Search Console,
- keyword performance, traffic drops, content gaps, search visibility, technical
- SEO, meta tags, schema markup, structured data, URL indexing, keyword research,
- or indexing issues. Also trigger on: "why is my traffic down", "what keywords
- am I ranking for", "improve my rankings", "check my search console", "SEO
- audit", "analyze my SEO", "technical SEO", "meta tags", "indexing issues",
- "crawl errors", "content strategy", "keyword cannibalization", "search intent",
- "schema markup", "structured data", "URL inspection", or any organic search
- question. If in doubt, trigger. This skill handles everything from quick GSC
- checks to deep technical audits.
+ Full SEO audit: Google Search Console data + URL Inspection API + PageSpeed
+ Insights API + technical crawl + keyword research + metadata audit + schema
+ markup audit + search intent analysis + Core Web Vitals monitoring. Feeds real
+ GSC data and PageSpeed metrics into AI to surface quick wins, diagnose traffic
+ drops, find content gaps, identify metadata mismatches, detect schema gaps,
+ monitor page performance, and produce an actionable 30-day plan. Use this skill
+ whenever the user asks about SEO, search rankings, organic traffic, Google
+ Search Console, keyword performance, traffic drops, content gaps, search
+ visibility, technical SEO, meta tags, schema markup, structured data, URL
+ indexing, keyword research, indexing issues, page speed, performance, Core Web
+ Vitals, LCP, INP, CLS, or Lighthouse scores. Also trigger on: "why is my
+ traffic down", "what keywords am I ranking for", "improve my rankings", "check
+ my search console", "SEO audit", "analyze my SEO", "technical SEO", "meta
+ tags", "indexing issues", "crawl errors", "content strategy", "keyword
+ cannibalization", "search intent", "schema markup", "structured data", "URL
+ inspection", "page speed", "performance score", "core web vitals", "lighthouse",
+ or any organic search question. If in doubt, trigger. This skill handles
+ everything from quick GSC checks to deep technical audits with performance
+ monitoring.
---
# SEO Analysis
@@ -115,7 +119,16 @@ Do NOT pause for user confirmation — just show the one-liner and continue.
Read and follow `../shared/preamble.md` — it handles script discovery, gcloud auth, and GSC API setup. If credentials are already cached, this is instant.
-If the user has no gcloud and wants to skip GSC, jump directly to Phase 5 for a technical-only audit (crawl, meta tags, schema, indexing).
+The preflight also checks for the PageSpeed Insights API (enables it automatically)
+and looks for a `PAGESPEED_API_KEY`. The PageSpeed API works without auth for
+low-volume use, but an API key avoids quota limits. If the preflight reports no
+API key, suggest:
+
+> "For reliable PageSpeed analysis, create an API key at
+> https://console.cloud.google.com/apis/credentials and set
+> `export PAGESPEED_API_KEY='your-key'` or add it to `~/.toprank/.env`."
+
+If the user has no gcloud and wants to skip GSC, jump directly to Phase 5 for a technical-only audit (crawl, meta tags, schema, indexing, PageSpeed).
> **Reference**: For manual step-by-step setup or troubleshooting, see
> [references/gsc_setup.md](references/gsc_setup.md).
@@ -233,16 +246,17 @@ This pulls:
## ⚡ Parallel Data Collection (after Phase 3 completes)
-**Do not run Phase 3.5, 3.6, and 5 sequentially — run them all at once.**
+**Do not run Phase 3.5, 3.6, 5, and 5.5 sequentially — run them all at once.**
As soon as Phase 3's `analyze_gsc.py` finishes and you have the top pages list,
-launch all three of these in a single turn using parallel tool calls:
+launch all four of these in a single turn using parallel tool calls:
1. **Phase 3.5**: run `url_inspection.py` (Bash tool)
2. **Phase 3.6**: detect CMS with `cms_detect.py`, then run the appropriate preflight + fetch if configured (Bash tool)
3. **Phase 5 pre-fetch**: fetch `robots.txt`, the homepage, and up to 4 top pages via WebFetch — all in parallel
+4. **Phase 5.5**: run `pagespeed.py` for the homepage + top pages by clicks (Bash tool) — this calls the PageSpeed Insights API which is independent of GSC auth
-This is safe because all three only need the target URL and top pages list, which
+This is safe because all four only need the target URL and top pages list, which
Phase 3 has already produced. Running them in parallel cuts ~3-5 minutes off the
total audit time. Start them all in the same response before reading any results.
@@ -913,6 +927,75 @@ After presenting the schema audit, offer:
---
+## Phase 5.5 — PageSpeed Insights (Performance Monitoring)
+
+Run the PageSpeed Insights API on the homepage + top 4 pages by clicks from
+Phase 3. This provides both **lab data** (Lighthouse synthetic test) and **field
+data** (Chrome UX Report real-user metrics) for Core Web Vitals.
+
+**⚡ Speed note**: This should already be running in parallel from the Parallel
+Data Collection step. If not, run it now.
+
+```bash
+python3 "$SKILL_SCRIPTS/pagespeed.py" \
+ --urls "$TARGET_URL,https://example.com/page2,https://example.com/page3" \
+ --both-strategies
+```
+
+Replace the example URLs with the actual homepage and top pages from Phase 3.
+Use `--both-strategies` to get both mobile and desktop scores. If the user has
+set `PAGESPEED_API_KEY` in their environment, the script uses it automatically
+for higher rate limits.
+
+After `pagespeed.py` completes, run the display utility:
+
+```bash
+python3 "$SKILL_SCRIPTS/show_pagespeed.py"
+```
+
+### Analyze the Results
+
+**1. Performance Scores** — Lighthouse scores 0-100 per page:
+- **90-100 (Good)**: No action needed.
+- **50-89 (Needs Work)**: Flag the top opportunities. These pages are losing
+ rankings due to performance — Google uses Core Web Vitals as a ranking signal.
+- **0-49 (Poor)**: Critical. These pages are actively penalized in rankings.
+ Flag as a Priority Action if the page has significant organic traffic.
+
+**2. Core Web Vitals (Field Data)** — Real-user metrics from Chrome UX Report:
+- **LCP** (Largest Contentful Paint): Good < 2.5s, Poor > 4.0s
+- **INP** (Interaction to Next Paint): Good < 200ms, Poor > 500ms
+- **CLS** (Cumulative Layout Shift): Good < 0.1, Poor > 0.25
+
+Field data is more authoritative than lab data for SEO — Google uses CrUX data
+for rankings. If field data is available, lead with it. If not (low-traffic
+sites often lack CrUX data), use lab data and note it's synthetic.
+
+**3. Cross-Reference with Other Phases**:
+- **Phase 3 device split**: If mobile performance score is significantly lower
+ than desktop, and Phase 3 shows mobile traffic underperforming, the
+ performance gap is likely a contributing factor.
+- **Phase 5 technical audit**: Correlate specific opportunities (e.g.,
+ "Eliminate render-blocking resources") with the technical findings (e.g.,
+ render-blocking scripts in `<head>`). This gives concrete evidence for
+ technical fixes.
+- **Phase 3.5 URL Inspection**: Pages flagged as mobile-unfriendly that also
+ have poor mobile PageSpeed scores need urgent attention.
+
+**4. Top Opportunities** — The script extracts Lighthouse optimization
+opportunities sorted by potential time savings. For each, note:
+- What the opportunity is (e.g., "Properly size images", "Remove unused JavaScript")
+- Estimated savings in milliseconds
+- Which specific page(s) are affected
+- Whether it's a site-wide template issue or page-specific
+
+**5. Origin-Level Data** — If available, the origin (site-wide) CrUX data shows
+the overall performance health of the entire domain. Compare individual page
+scores against the origin average to identify outlier pages dragging down the
+site's overall performance profile.
+
+---
+
## Phase 6 — Report
**The goal of this report is not comprehensiveness — it is clarity.** The user needs to know exactly what to do next, in what order, and why. Lead with the highest-impact actions. Put supporting data after. Omit anything that doesn't change what the user should do.
@@ -1015,6 +1098,17 @@ This section exists to back up the Priority Actions and surface anything else th
| Issue | Pages Affected | Fix | Severity |
|-------|---------------|-----|----------|
+### PageSpeed & Core Web Vitals
+*(From Phase 5.5. Only include if issues found. Lead with field data if available, fall back to lab data.)*
+
+**Site-wide (Origin)**: [Overall CrUX rating if available]
+
+| Page | Score | LCP | INP | CLS | Top Opportunity |
+|------|-------|-----|-----|-----|-----------------|
+| / | [score] | [value] [rating] | [value] [rating] | [value] [rating] | [top opportunity title + savings] |
+
+*(If any page scores below 50, flag it as a Priority Action candidate — poor Core Web Vitals directly hurt rankings.)*
+
### Traffic Drops
*(Pages/queries with >30% decline. Only include if not already in Priority Actions.)*
| Page / Query | Change | Hypothesis | Next Step |
@@ -1064,9 +1158,18 @@ existing.append({
"avg_ctr_pct": <FILL>,
"avg_position": <FILL>
},
+ "pagespeed_snapshot": {
+ "avg_score_mobile": <FILL or null>,
+ "avg_score_desktop": <FILL or null>,
+ "homepage_score_mobile": <FILL or null>,
+ "cwv_lcp_ms": <FILL or null>,
+ "cwv_inp_ms": <FILL or null>,
+ "cwv_cls": <FILL or null>,
+ "cwv_source": "<FILL: field|lab>" # "field" if CrUX data available, else "lab"
+ },
"top_issues": [
# One entry per Priority Action (max 5), in priority order
- {"rank": 1, "title": "<FILL>", "type": "<FILL: title_tag|indexing|cannibalization|schema|content_gap>", "page": "<FILL>", "metric": "<FILL>", "expected_impact": "<FILL>", "status": "open"}
+ {"rank": 1, "title": "<FILL>", "type": "<FILL: title_tag|indexing|cannibalization|schema|content_gap|performance>", "page": "<FILL>", "metric": "<FILL>", "expected_impact": "<FILL>", "status": "open"}
],
"resolved_from_previous": [] # populated on next audit from Audit History comparison
})
diff --git a/seo/seo-analysis/scripts/pagespeed.py b/seo/seo-analysis/scripts/pagespeed.py
new file mode 100644
index 00000000..4c3c8078
--- /dev/null
+++ b/seo/seo-analysis/scripts/pagespeed.py
@@ -0,0 +1,336 @@
+#!/usr/bin/env python3
+"""
+Pull PageSpeed Insights data for one or more URLs.
+Outputs structured JSON with Core Web Vitals, performance scores,
+and optimization opportunities.
+
+Usage:
+ python3 pagespeed.py --urls "https://example.com,https://example.com/about"
+ python3 pagespeed.py --urls "https://example.com" --strategy mobile
+ python3 pagespeed.py --urls "https://example.com" --api-key "YOUR_KEY"
+"""
+
+import argparse
+import json
+import os
+import sys
+import tempfile
+import urllib.parse
+import urllib.request
+import urllib.error
+from concurrent.futures import ThreadPoolExecutor, as_completed
+
+
+PSI_API = "https://www.googleapis.com/pagespeedonline/v5/runPagespeed"
+
+
+def _load_api_key_from_env_files():
+ """Try to load PAGESPEED_API_KEY from .env files if not in environment."""
+ for env_file in [".env", ".env.local", os.path.expanduser("~/.toprank/.env")]:
+ if os.path.isfile(env_file):
+ try:
+ with open(env_file) as f:
+ for line in f:
+ line = line.strip()
+ if line.startswith("PAGESPEED_API_KEY=") and not line.startswith("#"):
+ val = line.split("=", 1)[1].strip().strip("'\"")
+ if val:
+ return val
+ except OSError:
+ pass
+ return None
+
+
+def run_pagespeed(url, strategy="mobile", api_key=None):
+ """Call PageSpeed Insights API for a single URL and strategy."""
+ params = {
+ "url": url,
+ "strategy": strategy,
+ "category": "PERFORMANCE",
+ }
+ if api_key:
+ params["key"] = api_key
+
+ api_url = f"{PSI_API}?{urllib.parse.urlencode(params)}"
+ req = urllib.request.Request(api_url)
+ try:
+ with urllib.request.urlopen(req, timeout=60) as resp:
+ return json.loads(resp.read())
+ except urllib.error.HTTPError as e:
+ err_body = e.read().decode() if e.fp else "(no body)"
+ print(f"PSI API error {e.code} for {url}: {err_body}", file=sys.stderr)
+ if e.code == 429:
+ print("", file=sys.stderr)
+ print(" HINT: Quota exceeded. Fix options:", file=sys.stderr)
+ print(" 1. Create an API key: https://console.cloud.google.com/apis/credentials", file=sys.stderr)
+ print(" Then: export PAGESPEED_API_KEY='your-key'", file=sys.stderr)
+ print(" 2. Enable the API: gcloud services enable pagespeedonline.googleapis.com", file=sys.stderr)
+ return None
+ except urllib.error.URLError as e:
+ print(f"PSI API network error for {url}: {e.reason}", file=sys.stderr)
+ return None
+ except Exception as e:
+ print(f"PSI API unexpected error for {url}: {e}", file=sys.stderr)
+ return None
+
+
+def extract_crux_metric(metric_data):
+ """Extract Chrome UX Report (field data) metric value and rating."""
+ if not metric_data:
+ return None
+ percentile = metric_data.get("percentile")
+ category = metric_data.get("category")
+ distributions = metric_data.get("distributions", [])
+ return {
+ "value": percentile,
+ "rating": category, # FAST, AVERAGE, SLOW
+ "distributions": [
+ {"min": d.get("min", 0), "max": d.get("max"), "proportion": round(d.get("proportion", 0), 4)}
+ for d in distributions
+ ] if distributions else []
+ }
+
+
+def extract_field_data(loading_experience):
+ """Extract CrUX field data from the loading experience object."""
+ if not loading_experience or not loading_experience.get("metrics"):
+ return None
+
+ metrics = loading_experience["metrics"]
+ result = {
+ "overall_category": loading_experience.get("overall_category"),
+ }
+
+ metric_map = {
+ "LARGEST_CONTENTFUL_PAINT_MS": "lcp",
+ "INTERACTION_TO_NEXT_PAINT": "inp",
+ "CUMULATIVE_LAYOUT_SHIFT_SCORE": "cls",
+ "FIRST_CONTENTFUL_PAINT_MS": "fcp",
+ "EXPERIMENTAL_TIME_TO_FIRST_BYTE": "ttfb",
+ }
+
+ for api_key, short_key in metric_map.items():
+ if api_key in metrics:
+ result[short_key] = extract_crux_metric(metrics[api_key])
+
+ return result
+
+
+def extract_lab_data(lighthouse_result):
+ """Extract Lighthouse lab data (synthetic test results)."""
+ if not lighthouse_result:
+ return None
+
+ audits = lighthouse_result.get("audits", {})
+ categories = lighthouse_result.get("categories", {})
+
+ perf_score = None
+ perf_category = categories.get("performance", {})
+ if perf_category:
+ perf_score = perf_category.get("score")
+ if perf_score is not None:
+ perf_score = round(perf_score * 100)
+
+ def get_audit_value(audit_id, field="numericValue"):
+ audit = audits.get(audit_id, {})
+ val = audit.get(field)
+ display = audit.get("displayValue", "")
+ score = audit.get("score")
+ return {"value": val, "display": display, "score": score}
+
+ lab = {
+ "performance_score": perf_score,
+ "fcp": get_audit_value("first-contentful-paint"),
+ "lcp": get_audit_value("largest-contentful-paint"),
+ "cls": get_audit_value("cumulative-layout-shift"),
+ "tbt": get_audit_value("total-blocking-time"),
+ "si": get_audit_value("speed-index"),
+ "tti": get_audit_value("interactive"),
+ }
+
+ return lab
+
+
+def extract_opportunities(lighthouse_result, max_items=10):
+ """Extract top optimization opportunities from Lighthouse."""
+ if not lighthouse_result:
+ return []
+
+ audits = lighthouse_result.get("audits", {})
+ categories = lighthouse_result.get("categories", {})
+ perf = categories.get("performance", {})
+ audit_refs = perf.get("auditRefs", [])
+
+ # Collect opportunity-type audits that failed
+ opportunities = []
+ for ref in audit_refs:
+ if ref.get("group") != "opportunity":
+ continue
+ audit_id = ref.get("id", "")
+ audit = audits.get(audit_id, {})
+ score = audit.get("score")
+ if score is not None and score >= 0.9:
+ continue # already passing
+ details = audit.get("details", {})
+ overallSavingsMs = details.get("overallSavingsMs") or audit.get("numericValue", 0)
+ if not overallSavingsMs or overallSavingsMs <= 0:
+ continue
+ overallSavingsBytes = details.get("overallSavingsBytes", 0)
+
+ opportunities.append({
+ "id": audit_id,
+ "title": audit.get("title", audit_id),
+ "description": audit.get("description", ""),
+ "savings_ms": round(overallSavingsMs),
+ "savings_bytes": overallSavingsBytes,
+ "score": score,
+ "display": audit.get("displayValue", ""),
+ })
+
+ opportunities.sort(key=lambda x: x["savings_ms"], reverse=True)
+ return opportunities[:max_items]
+
+
+def extract_diagnostics(lighthouse_result, max_items=10):
+ """Extract diagnostic audit findings from Lighthouse."""
+ if not lighthouse_result:
+ return []
+
+ audits = lighthouse_result.get("audits", {})
+ categories = lighthouse_result.get("categories", {})
+ perf = categories.get("performance", {})
+ audit_refs = perf.get("auditRefs", [])
+
+ diagnostics = []
+ for ref in audit_refs:
+ if ref.get("group") != "diagnostics":
+ continue
+ audit_id = ref.get("id", "")
+ audit = audits.get(audit_id, {})
+ score = audit.get("score")
+ if score is not None and score >= 0.9:
+ continue
+
+ diagnostics.append({
+ "id": audit_id,
+ "title": audit.get("title", audit_id),
+ "description": audit.get("description", ""),
+ "display": audit.get("displayValue", ""),
+ "score": score,
+ })
+
+ diagnostics.sort(key=lambda x: (x["score"] or 0))
+ return diagnostics[:max_items]
+
+
+def analyze_url(url, strategy, api_key):
+ """Run PageSpeed analysis for a single URL and return structured results."""
+ print(f" Analyzing {url} ({strategy})...", file=sys.stderr)
+ raw = run_pagespeed(url, strategy=strategy, api_key=api_key)
+ if not raw:
+ return {"url": url, "strategy": strategy, "error": "API call failed"}
+
+ lighthouse = raw.get("lighthouseResult", {})
+
+ result = {
+ "url": url,
+ "strategy": strategy,
+ "field_data": extract_field_data(raw.get("loadingExperience")),
+ "origin_field_data": extract_field_data(raw.get("originLoadingExperience")),
+ "lab_data": extract_lab_data(lighthouse),
+ "opportunities": extract_opportunities(lighthouse),
+ "diagnostics": extract_diagnostics(lighthouse),
+ }
+
+ score = (result["lab_data"] or {}).get("performance_score")
+ if score is not None:
+ print(f" \u2713 {url}: score {score}/100", file=sys.stderr)
+ else:
+ print(f" \u2713 {url}: done (no score)", file=sys.stderr)
+
+ return result
+
+
+def main():
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--urls", required=True,
+ help="Comma-separated URLs to analyze")
+ parser.add_argument("--strategy", default="mobile",
+ choices=["mobile", "desktop"],
+ help="Test strategy (default: mobile)")
+ parser.add_argument("--both-strategies", action="store_true",
+ help="Run both mobile and desktop")
+ parser.add_argument("--api-key", default=os.environ.get("PAGESPEED_API_KEY", ""),
+ help="Google API key (optional, increases rate limits)")
+ _default_out = os.path.join(tempfile.gettempdir(), f"pagespeed_{os.getuid()}.json")
+ parser.add_argument("--output", default=_default_out, help="Output file")
+ args = parser.parse_args()
+
+ urls = [u.strip() for u in args.urls.split(",") if u.strip()]
+ if not urls:
+ print("ERROR: No URLs provided.", file=sys.stderr)
+ sys.exit(1)
+
+ strategies = ["mobile", "desktop"] if args.both_strategies else [args.strategy]
+ api_key = args.api_key or _load_api_key_from_env_files()
+ if api_key:
+ print("Using PageSpeed API key.", file=sys.stderr)
+ else:
+ print("No API key found. Requests may hit quota limits.", file=sys.stderr)
+ print(" Set PAGESPEED_API_KEY or add to ~/.toprank/.env", file=sys.stderr)
+
+ tasks = []
+ for url in urls:
+ for strategy in strategies:
+ tasks.append((url, strategy))
+
+ print(f"Running PageSpeed analysis for {len(urls)} URL(s), "
+ f"strategy: {', '.join(strategies)}...", file=sys.stderr)
+
+ results = []
+ # PSI API has rate limits; use modest concurrency
+ max_workers = min(len(tasks), 4)
+ with ThreadPoolExecutor(max_workers=max_workers) as pool:
+ futures = {
+ pool.submit(analyze_url, url, strategy, api_key): (url, strategy)
+ for url, strategy in tasks
+ }
+ for future in as_completed(futures):
+ url, strategy = futures[future]
+ try:
+ result = future.result()
+ results.append(result)
+ except Exception as exc:
+ print(f" \u2717 {url} ({strategy}): {exc}", file=sys.stderr)
+ results.append({"url": url, "strategy": strategy, "error": str(exc)})
+
+ # Sort results: by URL then strategy for consistent output
+ results.sort(key=lambda x: (x["url"], x.get("strategy", "")))
+
+ # Build summary
+ scored = [r for r in results if (r.get("lab_data") or {}).get("performance_score") is not None]
+ summary = {
+ "urls_tested": len(urls),
+ "strategies": strategies,
+ "results_count": len(results),
+ "avg_performance_score": (
+ round(sum(r["lab_data"]["performance_score"] for r in scored) / len(scored))
+ if scored else None
+ ),
+ }
+
+ output = {
+ "summary": summary,
+ "results": results,
+ }
+
+ with open(args.output, "w") as f:
+ json.dump(output, f, indent=2)
+
+ print(f"\nDone. Results saved to {args.output}", file=sys.stderr)
+ if summary["avg_performance_score"] is not None:
+ print(f"Average performance score: {summary['avg_performance_score']}/100", file=sys.stderr)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/seo/seo-analysis/scripts/preflight.py b/seo/seo-analysis/scripts/preflight.py
index b15b7f49..3bb51e8d 100644
--- a/seo/seo-analysis/scripts/preflight.py
+++ b/seo/seo-analysis/scripts/preflight.py
@@ -316,13 +316,92 @@ def check_quota_project():
return False
+def check_pagespeed_api():
+ """Ensure the PageSpeed Insights API is enabled in the active project.
+ Non-fatal — PageSpeed is optional but recommended."""
+ try:
+ result = subprocess.run(
+ ["gcloud", "services", "list", "--enabled",
+ "--filter=config.name:pagespeedonline.googleapis.com",
+ "--format=value(config.name)"],
+ capture_output=True, text=True, timeout=30,
+ )
+ except subprocess.TimeoutExpired:
+ print("WARNING: Timed out checking PageSpeed API status.", file=sys.stderr)
+ return
+
+ if result.returncode != 0:
+ print("WARNING: Could not check PageSpeed API status (gcloud error).", file=sys.stderr)
+ print(" PageSpeed analysis will still work with an API key.", file=sys.stderr)
+ return
+
+ if "pagespeedonline.googleapis.com" in result.stdout:
+ print("PageSpeed Insights API: enabled", file=sys.stderr)
+ return
+
+ print("PageSpeed Insights API is not enabled. Enabling it now...", file=sys.stderr)
+ try:
+ enable_result = subprocess.run(
+ ["gcloud", "services", "enable", "pagespeedonline.googleapis.com"],
+ capture_output=True, text=True, timeout=60,
+ )
+ except subprocess.TimeoutExpired:
+ print("WARNING: Timed out enabling PageSpeed API.", file=sys.stderr)
+ print(" To enable manually: gcloud services enable pagespeedonline.googleapis.com", file=sys.stderr)
+ return
+ if enable_result.returncode == 0:
+ print("PageSpeed Insights API: enabled", file=sys.stderr)
+ else:
+ print("WARNING: Could not enable PageSpeed Insights API.", file=sys.stderr)
+ print(" PageSpeed analysis will still work with an API key.", file=sys.stderr)
+ print(" To enable manually: gcloud services enable pagespeedonline.googleapis.com", file=sys.stderr)
+
+
+def check_pagespeed_api_key():
+ """Check if a PageSpeed API key is available. Suggest creating one if not.
+ The PSI API requires an API key for reliable access (without one, requests
+ may be rejected with quota errors)."""
+ if os.environ.get("PAGESPEED_API_KEY"):
+ print("PageSpeed API key: found in environment", file=sys.stderr)
+ return
+
+ # Check .env and .env.local files in common locations
+ for env_file in [".env", ".env.local", os.path.expanduser("~/.toprank/.env")]:
+ if os.path.isfile(env_file):
+ try:
+ with open(env_file) as f:
+ for line in f:
+ stripped = line.strip()
+ if stripped.startswith("PAGESPEED_API_KEY=") and not stripped.startswith("#"):
+ val = stripped.split("=", 1)[1].strip().strip("'\"")
+ if val:
+ print(f"PageSpeed API key: found in {env_file}", file=sys.stderr)
+ return
+ except OSError:
+ pass
+
+ print("", file=sys.stderr)
+ print("NOTE: No PageSpeed API key found.", file=sys.stderr)
+ print(" The PageSpeed Insights API works best with an API key.", file=sys.stderr)
+ print(" Without one, requests may hit quota limits.", file=sys.stderr)
+ print("", file=sys.stderr)
+ print(" To create one:", file=sys.stderr)
+ print(" 1. Go to https://console.cloud.google.com/apis/credentials", file=sys.stderr)
+ print(" 2. Click 'Create Credentials' > 'API key'", file=sys.stderr)
+ print(" 3. Set: export PAGESPEED_API_KEY='your-key-here'", file=sys.stderr)
+ print(" Or add to ~/.toprank/.env: PAGESPEED_API_KEY=your-key-here", file=sys.stderr)
+ print("", file=sys.stderr)
+
+
def main():
check_python_version()
check_gcloud()
check_gcloud_project()
check_search_console_api()
+ check_pagespeed_api()
check_adc_credentials()
quota_ok = check_quota_project()
+ check_pagespeed_api_key()
if quota_ok:
print("OK: All dependencies ready.", file=sys.stderr)
else:
diff --git a/seo/seo-analysis/scripts/show_pagespeed.py b/seo/seo-analysis/scripts/show_pagespeed.py
new file mode 100644
index 00000000..431826ee
--- /dev/null
+++ b/seo/seo-analysis/scripts/show_pagespeed.py
@@ -0,0 +1,238 @@
+#!/usr/bin/env python3
+"""
+Display PageSpeed Insights results in a terminal-friendly format.
+Reads the JSON output from pagespeed.py.
+
+Usage:
+ python3 show_pagespeed.py
+ python3 show_pagespeed.py --input /path/to/pagespeed.json
+"""
+
+import argparse
+import json
+import os
+import sys
+import tempfile
+
+
+def rating_indicator(rating):
+ """Return a visual indicator for CrUX rating."""
+ if not rating:
+ return "?"
+ return {"FAST": "GOOD", "AVERAGE": "NEEDS WORK", "SLOW": "POOR"}.get(rating, rating)
+
+
+def score_indicator(score):
+ """Return a visual indicator for Lighthouse score."""
+ if score is None:
+ return "N/A"
+ if score >= 90:
+ return f"{score} (Good)"
+ elif score >= 50:
+ return f"{score} (Needs Work)"
+ else:
+ return f"{score} (Poor)"
+
+
+def format_ms(val):
+ """Format milliseconds for display."""
+ if val is None:
+ return "N/A"
+ if val >= 1000:
+ return f"{val / 1000:.1f} s"
+ return f"{val:.0f} ms"
+
+
+def format_bytes(val):
+ """Format bytes for display."""
+ if not val:
+ return ""
+ if val >= 1_048_576:
+ return f"{val / 1_048_576:.1f} MB"
+ if val >= 1024:
+ return f"{val / 1024:.0f} KB"
+ return f"{val} B"
+
+
+def format_cls(val, is_crux=False):
+ """Format CLS value.
+ CrUX API returns CLS as an integer (CLS * 100), e.g. 10 means 0.10.
+ Lighthouse returns CLS as a float, e.g. 0.10."""
+ if val is None:
+ return "N/A"
+ if isinstance(val, (int, float)):
+ if is_crux:
+ return f"{val / 100:.3f}"
+ return f"{val:.3f}"
+ return str(val)
+
+
+def print_section(title):
+ print(f"\n{'=' * 60}")
+ print(f" {title}")
+ print(f"{'=' * 60}")
+
+
+def print_result(result):
+ """Print a single URL's PageSpeed result."""
+ url = result.get("url", "?")
+ strategy = result.get("strategy", "?")
+
+ if result.get("error"):
+ print(f"\n {url} ({strategy}): ERROR - {result['error']}")
+ return
+
+ print(f"\n URL: {url}")
+ print(f" Strategy: {strategy}")
+
+ # Lab data (Lighthouse)
+ lab = result.get("lab_data")
+ if lab:
+ score = lab.get("performance_score")
+ print(f"\n Performance Score: {score_indicator(score)}")
+ print(f" {'─' * 40}")
+
+ metrics = [
+ ("First Contentful Paint", lab.get("fcp", {})),
+ ("Largest Contentful Paint", lab.get("lcp", {})),
+ ("Total Blocking Time", lab.get("tbt", {})),
+ ("Cumulative Layout Shift", lab.get("cls", {})),
+ ("Speed Index", lab.get("si", {})),
+ ("Time to Interactive", lab.get("tti", {})),
+ ]
+
+ print(f" {'Metric':<30} {'Value':<15} {'Score'}")
+ print(f" {'─' * 55}")
+ for name, data in metrics:
+ if not data:
+ continue
+ display = data.get("display", "")
+ metric_score = data.get("score")
+ if metric_score is not None:
+ metric_score = f"{round(metric_score * 100)}/100"
+ else:
+ metric_score = "N/A"
+ print(f" {name:<30} {display:<15} {metric_score}")
+
+ # Field data (CrUX - real user data)
+ field = result.get("field_data")
+ if field:
+ print(f"\n Real-User Data (Chrome UX Report)")
+ print(f" {'─' * 40}")
+ overall = field.get("overall_category")
+ if overall:
+ print(f" Overall: {rating_indicator(overall)}")
+
+ crux_metrics = [
+ ("LCP", field.get("lcp"), "ms", False),
+ ("INP", field.get("inp"), "ms", False),
+ ("CLS", field.get("cls"), "", True),
+ ("FCP", field.get("fcp"), "ms", False),
+ ("TTFB", field.get("ttfb"), "ms", False),
+ ]
+
+ print(f" {'Metric':<8} {'Value':<12} {'Rating'}")
+ print(f" {'─' * 35}")
+ for name, data, unit, is_cls in crux_metrics:
+ if not data:
+ continue
+ val = data.get("value")
+ rating = data.get("rating")
+ if is_cls:
+ display = format_cls(val, is_crux=True)
+ elif unit == "ms":
+ display = format_ms(val)
+ else:
+ display = str(val) if val is not None else "N/A"
+ print(f" {name:<8} {display:<12} {rating_indicator(rating)}")
+
+ # Origin field data (site-wide CrUX)
+ origin = result.get("origin_field_data")
+ if origin and origin.get("overall_category"):
+ print(f"\n Origin (Site-Wide) Real-User Data")
+ print(f" {'─' * 40}")
+ print(f" Overall: {rating_indicator(origin.get('overall_category'))}")
+
+ crux_metrics = [
+ ("LCP", origin.get("lcp"), "ms", False),
+ ("INP", origin.get("inp"), "ms", False),
+ ("CLS", origin.get("cls"), "", True),
+ ]
+ for name, data, unit, is_cls in crux_metrics:
+ if not data:
+ continue
+ val = data.get("value")
+ rating = data.get("rating")
+ if is_cls:
+ display = format_cls(val, is_crux=True)
+ elif unit == "ms":
+ display = format_ms(val)
+ else:
+ display = str(val) if val is not None else "N/A"
+ print(f" {name:<8} {display:<12} {rating_indicator(rating)}")
+
+ # Opportunities
+ opportunities = result.get("opportunities", [])
+ if opportunities:
+ print(f"\n Top Optimization Opportunities")
+ print(f" {'─' * 55}")
+ print(f" {'Opportunity':<40} {'Savings'}")
+ print(f" {'─' * 55}")
+ for opp in opportunities:
+ title = opp.get("title", "?")
+ if len(title) > 38:
+ title = title[:35] + "..."
+ savings = format_ms(opp.get("savings_ms"))
+ bytes_saved = opp.get("savings_bytes", 0)
+ extra = f" ({format_bytes(bytes_saved)})" if bytes_saved else ""
+ print(f" {title:<40} {savings}{extra}")
+
+ # Diagnostics
+ diagnostics = result.get("diagnostics", [])
+ if diagnostics:
+ print(f"\n Diagnostics")
+ print(f" {'─' * 55}")
+ for diag in diagnostics:
+ title = diag.get("title", "?")
+ display = diag.get("display", "")
+ score = diag.get("score")
+ indicator = ""
+ if score is not None:
+ indicator = f" (score: {round(score * 100)}/100)"
+ suffix = f" — {display}" if display else ""
+ print(f" - {title}{suffix}{indicator}")
+
+
+def main():
+ parser = argparse.ArgumentParser()
+ _default_in = os.path.join(tempfile.gettempdir(), f"pagespeed_{os.getuid()}.json")
+ parser.add_argument("--input", default=_default_in, help="Input JSON file")
+ args = parser.parse_args()
+
+ if not os.path.isfile(args.input):
+ print(f"ERROR: File not found: {args.input}", file=sys.stderr)
+ print("Run pagespeed.py first to generate the data.", file=sys.stderr)
+ sys.exit(1)
+
+ with open(args.input) as f:
+ data = json.load(f)
+
+ summary = data.get("summary", {})
+ results = data.get("results", [])
+
+ print_section("PageSpeed Insights Report")
+ print(f" URLs tested: {summary.get('urls_tested', '?')}")
+ print(f" Strategies: {', '.join(summary.get('strategies', []))}")
+ avg = summary.get("avg_performance_score")
+ if avg is not None:
+ print(f" Average score: {score_indicator(avg)}")
+
+ for result in results:
+ print(f"\n{'─' * 60}")
+ print_result(result)
+
+ print(f"\n{'=' * 60}")
+
+
+if __name__ == "__main__":
+ main()