Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
TypeScript aiai-agentsai-crawlerai-scrapingai-searchcrawlerdata-extractionhtml-to-markdownllmmarkdownscraperscrapingweb-crawlerweb-dataweb-data-extractionweb-scraperweb-scrapingweb-searchwebscraping
feat: add engine forcing by domain pattern (#2371)
devin-ai-integration[bot]706cbffad9b2
diff --git a/apps/api/src/index.ts b/apps/api/src/index.tsindex 47ce622d8..ff5366612 100644--- a/apps/api/src/index.ts+++ b/apps/api/src/index.ts@@ -34,6 +34,7 @@ import domainFrequencyRouter from "./routes/domain-frequency";import { nuqShutdown } from "./services/worker/nuq";import { getErrorContactMessage } from "./lib/deployment";import { initializeBlocklist } from "./scraper/WebScraper/utils/blocklist";+import { initializeEngineForcing } from "./scraper/WebScraper/utils/engine-forcing";import responseTime from "response-time";const { createBullBoard } = require("@bull-board/api");@@ -110,8 +111,11 @@ const HOST = process.env.HOST ?? "localhost";async function startServer(port = DEFAULT_PORT) {try {await initializeBlocklist();+ initializeEngineForcing();} catch (error) {- logger.error("Failed to initialize blocklist", { error });+ logger.error("Failed to initialize blocklist and engine forcing", {+ error,+ });throw error;}diff --git a/apps/api/src/scraper/WebScraper/utils/ENGINE_FORCING.md b/apps/api/src/scraper/WebScraper/utils/ENGINE_FORCING.mdnew file mode 100644index 000000000..5dfcbf155--- /dev/null+++ b/apps/api/src/scraper/WebScraper/utils/ENGINE_FORCING.md@@ -0,0 +1,112 @@+# Engine Forcing++This feature allows you to force specific scraping engines for certain domains based on URL patterns. This is useful when you know that certain websites work better with specific engines.++## Configuration++The engine forcing is configured via the `FORCED_ENGINE_DOMAINS` environment variable. This should be a JSON object mapping domain patterns to engines.++### Environment Variable Format++```json+{+ "example.com": "playwright",+ "test.com": "fetch",+ "*.subdomain.com": "fire-engine;chrome-cdp",+ "google.com": ["fire-engine;chrome-cdp", "playwright"]+}+```++### Supported Patterns++1. **Exact domain match**: `"example.com"` matches `example.com` and all its subdomains (`www.example.com`, `api.example.com`, etc.)+2. **Wildcard pattern**: `"*.subdomain.com"` matches only subdomains of `subdomain.com` (e.g., `api.subdomain.com`, `www.subdomain.com`) but NOT the base domain itself+3. **Single engine**: `"playwright"` forces a single engine+4. **Multiple engines**: `["fire-engine;chrome-cdp", "playwright"]` provides a fallback list of engines to try in order++### Available Engines++- `fire-engine;chrome-cdp` - Advanced browser with Chrome DevTools Protocol+- `fire-engine;playwright` - Playwright-based browser automation+- `fire-engine;tlsclient` - TLS fingerprinting for anti-bot bypass+- `fire-engine;chrome-cdp;stealth` - Chrome CDP with stealth mode+- `fire-engine;playwright;stealth` - Playwright with stealth mode+- `fire-engine;tlsclient;stealth` - TLS client with stealth mode+- `playwright` - Direct Playwright integration+- `fetch` - Simple HTTP requests+- `pdf` - PDF document parsing+- `document` - Office document handling++## How It Works++1. When a scrape request is made, the system checks if the URL matches any domain pattern in `FORCED_ENGINE_DOMAINS`+2. If a match is found, the specified engine(s) are used instead of the default engine selection logic+3. If no match is found, the normal engine selection waterfall is used+4. The engine forcing only applies if `forceEngine` is not already set in the internal options++## Example Configuration++### Example 1: Force Playwright for specific domains++```bash+export FORCED_ENGINE_DOMAINS='{"linkedin.com":"playwright","twitter.com":"playwright"}'+```++This forces Playwright for LinkedIn and Twitter URLs.++### Example 2: Use fetch for simple sites++```bash+export FORCED_ENGINE_DOMAINS='{"example.com":"fetch","httpbin.org":"fetch"}'+```++This uses the simple fetch engine for example.com and httpbin.org.++### Example 3: Complex configuration with wildcards++```bash+export FORCED_ENGINE_DOMAINS='{+ "google.com": ["fire-engine;chrome-cdp", "playwright"],+ "*.cloudflare.com": "fire-engine;tlsclient;stealth",+ "wikipedia.org": "fetch"+}'+```++This configuration:++- Uses fire-engine with Chrome CDP for Google, falling back to Playwright if needed+- Uses fire-engine with TLS client in stealth mode for Cloudflare subdomains+- Uses simple fetch for Wikipedia++## Implementation Details++The engine forcing logic is implemented in:++- `apps/api/src/scraper/WebScraper/utils/engine-forcing.ts` - Core logic+- `apps/api/src/scraper/scrapeURL/index.ts` - Integration into scraping pipeline++The system is initialized at startup in:++- `apps/api/src/index.ts` - Main API server+- `apps/api/src/services/queue-worker.ts` - Queue worker+- `apps/api/src/services/extract-worker.ts` - Extract worker+- `apps/api/src/services/worker/nuq-worker.ts` - NuQ worker++## Precedence++The engine forcing has the following precedence:++1. If `forceEngine` is already set in `InternalOptions`, it takes precedence (engine forcing is skipped)+2. If a URL matches an engine forcing pattern, that engine is used+3. Otherwise, the normal engine selection waterfall is used++## Testing++Unit tests are available in `apps/api/src/scraper/WebScraper/utils/__tests__/engine-forcing.test.ts`.++## Notes++- Domain matching is case-insensitive+- The system handles invalid URLs gracefully by returning `undefined`+- If the JSON in `FORCED_ENGINE_DOMAINS` is invalid, the system logs an error and continues with empty mappings+- The feature is similar to the blocked domains logic but for engine selection instead of blockingdiff --git a/apps/api/src/scraper/WebScraper/utils/__tests__/engine-forcing.test.ts b/apps/api/src/scraper/WebScraper/utils/__tests__/engine-forcing.test.tsnew file mode 100644index 000000000..a0972f73f--- /dev/null+++ b/apps/api/src/scraper/WebScraper/utils/__tests__/engine-forcing.test.ts@@ -0,0 +1,105 @@+import { initializeEngineForcing, getEngineForUrl } from "../engine-forcing";++describe("Engine Forcing", () => {+ beforeEach(() => {+ delete process.env.FORCED_ENGINE_DOMAINS;+ });++ describe("initializeEngineForcing", () => {+ it("should initialize with empty mappings when env var is not set", () => {+ initializeEngineForcing();+ expect(getEngineForUrl("https://example.com")).toBeUndefined();+ });++ it("should initialize with empty mappings when env var is empty", () => {+ process.env.FORCED_ENGINE_DOMAINS = "";+ initializeEngineForcing();+ expect(getEngineForUrl("https://example.com")).toBeUndefined();+ });++ it("should parse valid JSON mappings", () => {+ process.env.FORCED_ENGINE_DOMAINS = JSON.stringify({+ "example.com": "playwright",+ });+ initializeEngineForcing();+ expect(getEngineForUrl("https://example.com")).toBe("playwright");+ });++ it("should handle invalid JSON gracefully", () => {+ process.env.FORCED_ENGINE_DOMAINS = "invalid json";+ initializeEngineForcing();+ expect(getEngineForUrl("https://example.com")).toBeUndefined();+ });+ });++ describe("getEngineForUrl", () => {+ beforeEach(() => {+ process.env.FORCED_ENGINE_DOMAINS = JSON.stringify({+ "example.com": "playwright",+ "test.com": "fetch",+ "*.subdomain.com": "fire-engine;chrome-cdp",+ "google.com": ["fire-engine;chrome-cdp", "playwright"],+ });+ initializeEngineForcing();+ });++ it("should return undefined for non-matching domains", () => {+ expect(getEngineForUrl("https://nomatch.com")).toBeUndefined();+ });++ it("should match exact domain", () => {+ expect(getEngineForUrl("https://example.com")).toBe("playwright");+ expect(getEngineForUrl("https://test.com")).toBe("fetch");+ });++ it("should match subdomains of exact domain", () => {+ expect(getEngineForUrl("https://www.example.com")).toBe("playwright");+ expect(getEngineForUrl("https://api.example.com")).toBe("playwright");+ });++ it("should match wildcard patterns", () => {+ expect(getEngineForUrl("https://api.subdomain.com")).toBe(+ "fire-engine;chrome-cdp",+ );+ expect(getEngineForUrl("https://www.subdomain.com")).toBe(+ "fire-engine;chrome-cdp",+ );+ });++ it("should not match wildcard pattern for base domain", () => {+ expect(getEngineForUrl("https://subdomain.com")).toBeUndefined();+ });++ it("should return array of engines when configured", () => {+ const result = getEngineForUrl("https://google.com");+ expect(Array.isArray(result)).toBe(true);+ expect(result).toEqual(["fire-engine;chrome-cdp", "playwright"]);+ });++ it("should handle URLs with paths and query strings", () => {+ expect(getEngineForUrl("https://example.com/path?query=1")).toBe(+ "playwright",+ );+ });++ it("should be case-insensitive", () => {+ expect(getEngineForUrl("https://EXAMPLE.COM")).toBe("playwright");+ expect(getEngineForUrl("https://Example.Com/path")).toBe("playwright");+ });++ it("should handle invalid URLs gracefully", () => {+ expect(getEngineForUrl("not-a-url")).toBeUndefined();+ expect(getEngineForUrl("")).toBeUndefined();+ });++ it("should throw error if not initialized", () => {+ delete process.env.FORCED_ENGINE_DOMAINS;+ expect(() => {+ const {+ getEngineForUrl: freshGetEngineForUrl,+ } = require("../engine-forcing");+ freshGetEngineForUrl("https://example.com");+ }).toThrow("Engine forcing not initialized");+ });+ });+});diff --git a/apps/api/src/scraper/WebScraper/utils/engine-forcing.ts b/apps/api/src/scraper/WebScraper/utils/engine-forcing.tsnew file mode 100644index 000000000..89c972978--- /dev/null+++ b/apps/api/src/scraper/WebScraper/utils/engine-forcing.ts@@ -0,0 +1,92 @@+import { configDotenv } from "dotenv";+import { parse } from "tldts";+import { Engine } from "../../scrapeURL/engines";+import { logger } from "../../../lib/logger";++configDotenv();++type EngineForcingMapping = {+ [domain: string]: Engine | Engine[];+};++let engineMappings: EngineForcingMapping | null = null;++/**+ * Initialize the engine forcing mappings from environment variable+ * Expected format: JSON object with domain patterns as keys and engines as values+ * Example: {"example.com": "playwright", "*.google.com": ["fire-engine;chrome-cdp", "playwright"]}+ */+export function initializeEngineForcing() {+ const envVar = process.env.FORCED_ENGINE_DOMAINS;++ if (!envVar || envVar.trim() === "") {+ engineMappings = {};+ return;+ }++ try {+ engineMappings = JSON.parse(envVar);+ } catch (error) {+ logger.error("Error parsing FORCED_ENGINE_DOMAINS environment variable", {+ error,+ });+ engineMappings = {};+ }+}++/**+ * Check if a domain matches a pattern (supports wildcards)+ * @param domain The domain to check (e.g., "sub.example.com")+ * @param pattern The pattern to match against (e.g., "example.com" or "*.example.com")+ * @returns true if the domain matches the pattern+ */+function domainMatchesPattern(domain: string, pattern: string): boolean {+ if (domain === pattern) {+ return true;+ }++ if (pattern.startsWith("*.")) {+ const basePattern = pattern.substring(2); // Remove "*."+ return domain === basePattern || domain.endsWith(`.${basePattern}`);+ }++ return domain === pattern || domain.endsWith(`.${pattern}`);+}++/**+ * Get the forced engine(s) for a given URL based on domain mappings+ * @param url The URL to check+ * @returns The forced engine(s) if a match is found, undefined otherwise+ */+export function getEngineForUrl(url: string): Engine | Engine[] | undefined {+ if (engineMappings === null) {+ return undefined;+ }++ if (Object.keys(engineMappings).length === 0) {+ return undefined;+ }++ const lowerCaseUrl = url.trim().toLowerCase();++ let parsedUrl: any;+ try {+ parsedUrl = parse(lowerCaseUrl);+ } catch (error) {+ logger.warn("Error parsing URL for engine forcing", { url, error });+ return undefined;+ }++ const domain = parsedUrl.domain;+ if (!domain) {+ return undefined;+ }++ for (const [pattern, engine] of Object.entries(engineMappings)) {+ if (domainMatchesPattern(domain, pattern.toLowerCase())) {+ return engine;+ }+ }++ return undefined;+}diff --git a/apps/api/src/scraper/scrapeURL/engines/index.ts b/apps/api/src/scraper/scrapeURL/engines/index.tsindex 357788beb..4e29065b2 100644--- a/apps/api/src/scraper/scrapeURL/engines/index.ts+++ b/apps/api/src/scraper/scrapeURL/engines/index.ts@@ -596,9 +596,16 @@ export async function scrapeURLWithEngine(method: fn.name ?? "scrapeURLWithEngine",engine,});++ const featureFlags = new Set(meta.featureFlags);+ if (engineOptions[engine].features.stealthProxy) {+ featureFlags.add("stealthProxy");+ }+const _meta = {...meta,logger,+ featureFlags,};return await fn(_meta);diff --git a/apps/api/src/scraper/scrapeURL/index.ts b/apps/api/src/scraper/scrapeURL/index.tsindex 57b43ce19..fe92000a8 100644--- a/apps/api/src/scraper/scrapeURL/index.ts+++ b/apps/api/src/scraper/scrapeURL/index.ts@@ -49,6 +49,7 @@ import { LLMRefusalError } from "./transformers/llmExtract";import { urlSpecificParams } from "./lib/urlSpecificParams";import { loadMock, MockState } from "./lib/mock";import { CostTracking } from "../../lib/cost-tracking";+import { getEngineForUrl } from "../WebScraper/utils/engine-forcing";import {addIndexRFInsertJob,generateDomainSplits,@@ -258,6 +259,15 @@ async function buildMetaObject();}+ if (internalOptions.forceEngine === undefined) {+ const forcedEngine = getEngineForUrl(url);+ if (forcedEngine !== undefined) {+ internalOptions = Object.assign(internalOptions, {+ forceEngine: forcedEngine,+ });+ }+ }+const logger = _logger.child({module: "ScrapeURL",scrapeId: id,@@ -736,7 +746,7 @@ async function scrapeURLLoop(meta: Meta): Promise<ScrapeUrlResponse> {? { title: engineResult.pdfMetadata.title }: {}),contentType: engineResult.contentType,- proxyUsed: meta.featureFlags.has("stealthProxy") ? "stealth" : "basic",+ proxyUsed: engineResult.proxyUsed ?? "basic",...(fallbackList.find(x =>["index", "index;documents"].includes(x.engine),)diff --git a/apps/api/src/services/extract-worker.ts b/apps/api/src/services/extract-worker.tsindex c0a435478..fa0b7989b 100644--- a/apps/api/src/services/extract-worker.ts+++ b/apps/api/src/services/extract-worker.ts@@ -20,6 +20,7 @@ import { robustFetch } from "../scraper/scrapeURL/lib/fetch";import { BullMQOtel } from "bullmq-otel";import { getErrorContactMessage } from "../lib/deployment";import { initializeBlocklist } from "../scraper/WebScraper/utils/blocklist";+import { initializeEngineForcing } from "../scraper/WebScraper/utils/engine-forcing";configDotenv();@@ -297,6 +298,8 @@ app.listen(workerPort, () => {process.exit(1);});+ initializeEngineForcing();+await Promise.all([workerFun(getExtractQueue(), processExtractJobInternal)]);_logger.info("All workers exited. Waiting for all jobs to finish...");diff --git a/apps/api/src/services/queue-worker.ts b/apps/api/src/services/queue-worker.tsindex 2df91f172..c1a7cb076 100644--- a/apps/api/src/services/queue-worker.ts+++ b/apps/api/src/services/queue-worker.ts@@ -20,6 +20,7 @@ import Express from "express";import { robustFetch } from "../scraper/scrapeURL/lib/fetch";import { BullMQOtel } from "bullmq-otel";import { initializeBlocklist } from "../scraper/WebScraper/utils/blocklist";+import { initializeEngineForcing } from "../scraper/WebScraper/utils/engine-forcing";import { crawlFinishedQueue, NuQJob, scrapeQueue } from "./worker/nuq";import { finishCrawlSuper } from "./worker/crawl-logic";import { getCrawl } from "../lib/crawl-redis";@@ -443,6 +444,8 @@ app.listen(workerPort, () => {process.exit(1);});+ initializeEngineForcing();+await Promise.all([workerFun(getDeepResearchQueue(), processDeepResearchJobInternal),workerFun(getGenerateLlmsTxtQueue(), processGenerateLlmsTxtJobInternal),diff --git a/apps/api/src/services/worker/nuq-worker.ts b/apps/api/src/services/worker/nuq-worker.tsindex 2bf4a3f59..250b65960 100644--- a/apps/api/src/services/worker/nuq-worker.ts+++ b/apps/api/src/services/worker/nuq-worker.ts@@ -5,12 +5,16 @@ import { scrapeQueue, nuqGetLocalMetrics, nuqHealthCheck } from "./nuq";import Express from "express";import { _ } from "ajv";import { initializeBlocklist } from "../../scraper/WebScraper/utils/blocklist";+import { initializeEngineForcing } from "../../scraper/WebScraper/utils/engine-forcing";(async () => {try {await initializeBlocklist();+ initializeEngineForcing();} catch (error) {- _logger.error("Failed to initialize blocklist", { error });+ _logger.error("Failed to initialize blocklist and engine forcing", {+ error,+ });process.exit(1);}