Skip to content

Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥

feat: add engine forcing by domain pattern (#2371)

devin-ai-integration[bot]706cbffad9b2
diff --git a/apps/api/src/index.ts b/apps/api/src/index.ts
index 47ce622d8..ff5366612 100644
--- a/apps/api/src/index.ts
+++ b/apps/api/src/index.ts
@@ -34,6 +34,7 @@ import domainFrequencyRouter from "./routes/domain-frequency";
import { nuqShutdown } from "./services/worker/nuq";
import { getErrorContactMessage } from "./lib/deployment";
import { initializeBlocklist } from "./scraper/WebScraper/utils/blocklist";
+import { initializeEngineForcing } from "./scraper/WebScraper/utils/engine-forcing";
import responseTime from "response-time";
const { createBullBoard } = require("@bull-board/api");
@@ -110,8 +111,11 @@ const HOST = process.env.HOST ?? "localhost";
async function startServer(port = DEFAULT_PORT) {
try {
await initializeBlocklist();
+ initializeEngineForcing();
} catch (error) {
- logger.error("Failed to initialize blocklist", { error });
+ logger.error("Failed to initialize blocklist and engine forcing", {
+ error,
+ });
throw error;
}
diff --git a/apps/api/src/scraper/WebScraper/utils/ENGINE_FORCING.md b/apps/api/src/scraper/WebScraper/utils/ENGINE_FORCING.md
new file mode 100644
index 000000000..5dfcbf155
--- /dev/null
+++ b/apps/api/src/scraper/WebScraper/utils/ENGINE_FORCING.md
@@ -0,0 +1,112 @@
+# Engine Forcing
+
+This feature allows you to force specific scraping engines for certain domains based on URL patterns. This is useful when you know that certain websites work better with specific engines.
+
+## Configuration
+
+The engine forcing is configured via the `FORCED_ENGINE_DOMAINS` environment variable. This should be a JSON object mapping domain patterns to engines.
+
+### Environment Variable Format
+
+```json
+{
+ "example.com": "playwright",
+ "test.com": "fetch",
+ "*.subdomain.com": "fire-engine;chrome-cdp",
+ "google.com": ["fire-engine;chrome-cdp", "playwright"]
+}
+```
+
+### Supported Patterns
+
+1. **Exact domain match**: `"example.com"` matches `example.com` and all its subdomains (`www.example.com`, `api.example.com`, etc.)
+2. **Wildcard pattern**: `"*.subdomain.com"` matches only subdomains of `subdomain.com` (e.g., `api.subdomain.com`, `www.subdomain.com`) but NOT the base domain itself
+3. **Single engine**: `"playwright"` forces a single engine
+4. **Multiple engines**: `["fire-engine;chrome-cdp", "playwright"]` provides a fallback list of engines to try in order
+
+### Available Engines
+
+- `fire-engine;chrome-cdp` - Advanced browser with Chrome DevTools Protocol
+- `fire-engine;playwright` - Playwright-based browser automation
+- `fire-engine;tlsclient` - TLS fingerprinting for anti-bot bypass
+- `fire-engine;chrome-cdp;stealth` - Chrome CDP with stealth mode
+- `fire-engine;playwright;stealth` - Playwright with stealth mode
+- `fire-engine;tlsclient;stealth` - TLS client with stealth mode
+- `playwright` - Direct Playwright integration
+- `fetch` - Simple HTTP requests
+- `pdf` - PDF document parsing
+- `document` - Office document handling
+
+## How It Works
+
+1. When a scrape request is made, the system checks if the URL matches any domain pattern in `FORCED_ENGINE_DOMAINS`
+2. If a match is found, the specified engine(s) are used instead of the default engine selection logic
+3. If no match is found, the normal engine selection waterfall is used
+4. The engine forcing only applies if `forceEngine` is not already set in the internal options
+
+## Example Configuration
+
+### Example 1: Force Playwright for specific domains
+
+```bash
+export FORCED_ENGINE_DOMAINS='{"linkedin.com":"playwright","twitter.com":"playwright"}'
+```
+
+This forces Playwright for LinkedIn and Twitter URLs.
+
+### Example 2: Use fetch for simple sites
+
+```bash
+export FORCED_ENGINE_DOMAINS='{"example.com":"fetch","httpbin.org":"fetch"}'
+```
+
+This uses the simple fetch engine for example.com and httpbin.org.
+
+### Example 3: Complex configuration with wildcards
+
+```bash
+export FORCED_ENGINE_DOMAINS='{
+ "google.com": ["fire-engine;chrome-cdp", "playwright"],
+ "*.cloudflare.com": "fire-engine;tlsclient;stealth",
+ "wikipedia.org": "fetch"
+}'
+```
+
+This configuration:
+
+- Uses fire-engine with Chrome CDP for Google, falling back to Playwright if needed
+- Uses fire-engine with TLS client in stealth mode for Cloudflare subdomains
+- Uses simple fetch for Wikipedia
+
+## Implementation Details
+
+The engine forcing logic is implemented in:
+
+- `apps/api/src/scraper/WebScraper/utils/engine-forcing.ts` - Core logic
+- `apps/api/src/scraper/scrapeURL/index.ts` - Integration into scraping pipeline
+
+The system is initialized at startup in:
+
+- `apps/api/src/index.ts` - Main API server
+- `apps/api/src/services/queue-worker.ts` - Queue worker
+- `apps/api/src/services/extract-worker.ts` - Extract worker
+- `apps/api/src/services/worker/nuq-worker.ts` - NuQ worker
+
+## Precedence
+
+The engine forcing has the following precedence:
+
+1. If `forceEngine` is already set in `InternalOptions`, it takes precedence (engine forcing is skipped)
+2. If a URL matches an engine forcing pattern, that engine is used
+3. Otherwise, the normal engine selection waterfall is used
+
+## Testing
+
+Unit tests are available in `apps/api/src/scraper/WebScraper/utils/__tests__/engine-forcing.test.ts`.
+
+## Notes
+
+- Domain matching is case-insensitive
+- The system handles invalid URLs gracefully by returning `undefined`
+- If the JSON in `FORCED_ENGINE_DOMAINS` is invalid, the system logs an error and continues with empty mappings
+- The feature is similar to the blocked domains logic but for engine selection instead of blocking
diff --git a/apps/api/src/scraper/WebScraper/utils/__tests__/engine-forcing.test.ts b/apps/api/src/scraper/WebScraper/utils/__tests__/engine-forcing.test.ts
new file mode 100644
index 000000000..a0972f73f
--- /dev/null
+++ b/apps/api/src/scraper/WebScraper/utils/__tests__/engine-forcing.test.ts
@@ -0,0 +1,105 @@
+import { initializeEngineForcing, getEngineForUrl } from "../engine-forcing";
+
+describe("Engine Forcing", () => {
+ beforeEach(() => {
+ delete process.env.FORCED_ENGINE_DOMAINS;
+ });
+
+ describe("initializeEngineForcing", () => {
+ it("should initialize with empty mappings when env var is not set", () => {
+ initializeEngineForcing();
+ expect(getEngineForUrl("https://example.com")).toBeUndefined();
+ });
+
+ it("should initialize with empty mappings when env var is empty", () => {
+ process.env.FORCED_ENGINE_DOMAINS = "";
+ initializeEngineForcing();
+ expect(getEngineForUrl("https://example.com")).toBeUndefined();
+ });
+
+ it("should parse valid JSON mappings", () => {
+ process.env.FORCED_ENGINE_DOMAINS = JSON.stringify({
+ "example.com": "playwright",
+ });
+ initializeEngineForcing();
+ expect(getEngineForUrl("https://example.com")).toBe("playwright");
+ });
+
+ it("should handle invalid JSON gracefully", () => {
+ process.env.FORCED_ENGINE_DOMAINS = "invalid json";
+ initializeEngineForcing();
+ expect(getEngineForUrl("https://example.com")).toBeUndefined();
+ });
+ });
+
+ describe("getEngineForUrl", () => {
+ beforeEach(() => {
+ process.env.FORCED_ENGINE_DOMAINS = JSON.stringify({
+ "example.com": "playwright",
+ "test.com": "fetch",
+ "*.subdomain.com": "fire-engine;chrome-cdp",
+ "google.com": ["fire-engine;chrome-cdp", "playwright"],
+ });
+ initializeEngineForcing();
+ });
+
+ it("should return undefined for non-matching domains", () => {
+ expect(getEngineForUrl("https://nomatch.com")).toBeUndefined();
+ });
+
+ it("should match exact domain", () => {
+ expect(getEngineForUrl("https://example.com")).toBe("playwright");
+ expect(getEngineForUrl("https://test.com")).toBe("fetch");
+ });
+
+ it("should match subdomains of exact domain", () => {
+ expect(getEngineForUrl("https://www.example.com")).toBe("playwright");
+ expect(getEngineForUrl("https://api.example.com")).toBe("playwright");
+ });
+
+ it("should match wildcard patterns", () => {
+ expect(getEngineForUrl("https://api.subdomain.com")).toBe(
+ "fire-engine;chrome-cdp",
+ );
+ expect(getEngineForUrl("https://www.subdomain.com")).toBe(
+ "fire-engine;chrome-cdp",
+ );
+ });
+
+ it("should not match wildcard pattern for base domain", () => {
+ expect(getEngineForUrl("https://subdomain.com")).toBeUndefined();
+ });
+
+ it("should return array of engines when configured", () => {
+ const result = getEngineForUrl("https://google.com");
+ expect(Array.isArray(result)).toBe(true);
+ expect(result).toEqual(["fire-engine;chrome-cdp", "playwright"]);
+ });
+
+ it("should handle URLs with paths and query strings", () => {
+ expect(getEngineForUrl("https://example.com/path?query=1")).toBe(
+ "playwright",
+ );
+ });
+
+ it("should be case-insensitive", () => {
+ expect(getEngineForUrl("https://EXAMPLE.COM")).toBe("playwright");
+ expect(getEngineForUrl("https://Example.Com/path")).toBe("playwright");
+ });
+
+ it("should handle invalid URLs gracefully", () => {
+ expect(getEngineForUrl("not-a-url")).toBeUndefined();
+ expect(getEngineForUrl("")).toBeUndefined();
+ });
+
+ it("should throw error if not initialized", () => {
+ delete process.env.FORCED_ENGINE_DOMAINS;
+ expect(() => {
+ const {
+ getEngineForUrl: freshGetEngineForUrl,
+ } = require("../engine-forcing");
+ freshGetEngineForUrl("https://example.com");
+ }).toThrow("Engine forcing not initialized");
+ });
+ });
+});
diff --git a/apps/api/src/scraper/WebScraper/utils/engine-forcing.ts b/apps/api/src/scraper/WebScraper/utils/engine-forcing.ts
new file mode 100644
index 000000000..89c972978
--- /dev/null
+++ b/apps/api/src/scraper/WebScraper/utils/engine-forcing.ts
@@ -0,0 +1,92 @@
+import { configDotenv } from "dotenv";
+import { parse } from "tldts";
+import { Engine } from "../../scrapeURL/engines";
+import { logger } from "../../../lib/logger";
+
+configDotenv();
+
+type EngineForcingMapping = {
+ [domain: string]: Engine | Engine[];
+};
+
+let engineMappings: EngineForcingMapping | null = null;
+
+/**
+ * Initialize the engine forcing mappings from environment variable
+ * Expected format: JSON object with domain patterns as keys and engines as values
+ * Example: {"example.com": "playwright", "*.google.com": ["fire-engine;chrome-cdp", "playwright"]}
+ */
+export function initializeEngineForcing() {
+ const envVar = process.env.FORCED_ENGINE_DOMAINS;
+
+ if (!envVar || envVar.trim() === "") {
+ engineMappings = {};
+ return;
+ }
+
+ try {
+ engineMappings = JSON.parse(envVar);
+ } catch (error) {
+ logger.error("Error parsing FORCED_ENGINE_DOMAINS environment variable", {
+ error,
+ });
+ engineMappings = {};
+ }
+}
+
+/**
+ * Check if a domain matches a pattern (supports wildcards)
+ * @param domain The domain to check (e.g., "sub.example.com")
+ * @param pattern The pattern to match against (e.g., "example.com" or "*.example.com")
+ * @returns true if the domain matches the pattern
+ */
+function domainMatchesPattern(domain: string, pattern: string): boolean {
+ if (domain === pattern) {
+ return true;
+ }
+
+ if (pattern.startsWith("*.")) {
+ const basePattern = pattern.substring(2); // Remove "*."
+ return domain === basePattern || domain.endsWith(`.${basePattern}`);
+ }
+
+ return domain === pattern || domain.endsWith(`.${pattern}`);
+}
+
+/**
+ * Get the forced engine(s) for a given URL based on domain mappings
+ * @param url The URL to check
+ * @returns The forced engine(s) if a match is found, undefined otherwise
+ */
+export function getEngineForUrl(url: string): Engine | Engine[] | undefined {
+ if (engineMappings === null) {
+ return undefined;
+ }
+
+ if (Object.keys(engineMappings).length === 0) {
+ return undefined;
+ }
+
+ const lowerCaseUrl = url.trim().toLowerCase();
+
+ let parsedUrl: any;
+ try {
+ parsedUrl = parse(lowerCaseUrl);
+ } catch (error) {
+ logger.warn("Error parsing URL for engine forcing", { url, error });
+ return undefined;
+ }
+
+ const domain = parsedUrl.domain;
+ if (!domain) {
+ return undefined;
+ }
+
+ for (const [pattern, engine] of Object.entries(engineMappings)) {
+ if (domainMatchesPattern(domain, pattern.toLowerCase())) {
+ return engine;
+ }
+ }
+
+ return undefined;
+}
diff --git a/apps/api/src/scraper/scrapeURL/engines/index.ts b/apps/api/src/scraper/scrapeURL/engines/index.ts
index 357788beb..4e29065b2 100644
--- a/apps/api/src/scraper/scrapeURL/engines/index.ts
+++ b/apps/api/src/scraper/scrapeURL/engines/index.ts
@@ -596,9 +596,16 @@ export async function scrapeURLWithEngine(
method: fn.name ?? "scrapeURLWithEngine",
engine,
});
+
+ const featureFlags = new Set(meta.featureFlags);
+ if (engineOptions[engine].features.stealthProxy) {
+ featureFlags.add("stealthProxy");
+ }
+
const _meta = {
...meta,
logger,
+ featureFlags,
};
return await fn(_meta);
diff --git a/apps/api/src/scraper/scrapeURL/index.ts b/apps/api/src/scraper/scrapeURL/index.ts
index 57b43ce19..fe92000a8 100644
--- a/apps/api/src/scraper/scrapeURL/index.ts
+++ b/apps/api/src/scraper/scrapeURL/index.ts
@@ -49,6 +49,7 @@ import { LLMRefusalError } from "./transformers/llmExtract";
import { urlSpecificParams } from "./lib/urlSpecificParams";
import { loadMock, MockState } from "./lib/mock";
import { CostTracking } from "../../lib/cost-tracking";
+import { getEngineForUrl } from "../WebScraper/utils/engine-forcing";
import {
addIndexRFInsertJob,
generateDomainSplits,
@@ -258,6 +259,15 @@ async function buildMetaObject(
);
}
+ if (internalOptions.forceEngine === undefined) {
+ const forcedEngine = getEngineForUrl(url);
+ if (forcedEngine !== undefined) {
+ internalOptions = Object.assign(internalOptions, {
+ forceEngine: forcedEngine,
+ });
+ }
+ }
+
const logger = _logger.child({
module: "ScrapeURL",
scrapeId: id,
@@ -736,7 +746,7 @@ async function scrapeURLLoop(meta: Meta): Promise<ScrapeUrlResponse> {
? { title: engineResult.pdfMetadata.title }
: {}),
contentType: engineResult.contentType,
- proxyUsed: meta.featureFlags.has("stealthProxy") ? "stealth" : "basic",
+ proxyUsed: engineResult.proxyUsed ?? "basic",
...(fallbackList.find(x =>
["index", "index;documents"].includes(x.engine),
)
diff --git a/apps/api/src/services/extract-worker.ts b/apps/api/src/services/extract-worker.ts
index c0a435478..fa0b7989b 100644
--- a/apps/api/src/services/extract-worker.ts
+++ b/apps/api/src/services/extract-worker.ts
@@ -20,6 +20,7 @@ import { robustFetch } from "../scraper/scrapeURL/lib/fetch";
import { BullMQOtel } from "bullmq-otel";
import { getErrorContactMessage } from "../lib/deployment";
import { initializeBlocklist } from "../scraper/WebScraper/utils/blocklist";
+import { initializeEngineForcing } from "../scraper/WebScraper/utils/engine-forcing";
configDotenv();
@@ -297,6 +298,8 @@ app.listen(workerPort, () => {
process.exit(1);
});
+ initializeEngineForcing();
+
await Promise.all([workerFun(getExtractQueue(), processExtractJobInternal)]);
_logger.info("All workers exited. Waiting for all jobs to finish...");
diff --git a/apps/api/src/services/queue-worker.ts b/apps/api/src/services/queue-worker.ts
index 2df91f172..c1a7cb076 100644
--- a/apps/api/src/services/queue-worker.ts
+++ b/apps/api/src/services/queue-worker.ts
@@ -20,6 +20,7 @@ import Express from "express";
import { robustFetch } from "../scraper/scrapeURL/lib/fetch";
import { BullMQOtel } from "bullmq-otel";
import { initializeBlocklist } from "../scraper/WebScraper/utils/blocklist";
+import { initializeEngineForcing } from "../scraper/WebScraper/utils/engine-forcing";
import { crawlFinishedQueue, NuQJob, scrapeQueue } from "./worker/nuq";
import { finishCrawlSuper } from "./worker/crawl-logic";
import { getCrawl } from "../lib/crawl-redis";
@@ -443,6 +444,8 @@ app.listen(workerPort, () => {
process.exit(1);
});
+ initializeEngineForcing();
+
await Promise.all([
workerFun(getDeepResearchQueue(), processDeepResearchJobInternal),
workerFun(getGenerateLlmsTxtQueue(), processGenerateLlmsTxtJobInternal),
diff --git a/apps/api/src/services/worker/nuq-worker.ts b/apps/api/src/services/worker/nuq-worker.ts
index 2bf4a3f59..250b65960 100644
--- a/apps/api/src/services/worker/nuq-worker.ts
+++ b/apps/api/src/services/worker/nuq-worker.ts
@@ -5,12 +5,16 @@ import { scrapeQueue, nuqGetLocalMetrics, nuqHealthCheck } from "./nuq";
import Express from "express";
import { _ } from "ajv";
import { initializeBlocklist } from "../../scraper/WebScraper/utils/blocklist";
+import { initializeEngineForcing } from "../../scraper/WebScraper/utils/engine-forcing";
(async () => {
try {
await initializeBlocklist();
+ initializeEngineForcing();
} catch (error) {
- _logger.error("Failed to initialize blocklist", { error });
+ _logger.error("Failed to initialize blocklist and engine forcing", {
+ error,
+ });
process.exit(1);
}