Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

fix(core): keep same-domain filtering after an off-domain redirect (#3923)

Martin Adámek9b312efd14d6
diff --git a/packages/core/src/enqueue_links/enqueue_links.ts b/packages/core/src/enqueue_links/enqueue_links.ts
index b13f09508..c87a29428 100644
--- a/packages/core/src/enqueue_links/enqueue_links.ts
+++ b/packages/core/src/enqueue_links/enqueue_links.ts
@@ -489,8 +489,8 @@ export function resolveBaseUrlForEnqueueLinksFiltering({
}
// If the user wants to ensure the same domain is accessed, regardless of subdomains, we check to ensure the domains match
- // Returning undefined here is intentional! If the domains don't match, having no baseUrl in enqueueLinks will cause it to not enqueue anything
- // which is the intended behavior (since we went off domain)
+ // If they don't (we went off domain via a redirect), we keep filtering against the original domain - returning
+ // undefined here would disable the filtering entirely and enqueue every link on the redirected page
if (enqueueStrategy === EnqueueStrategy.SameDomain) {
const originalHostname = getDomain(originalUrlOrigin, { mixedInputs: false })!;
const finalHostname = getDomain(finalUrlOrigin, { mixedInputs: false })!;
@@ -499,7 +499,7 @@ export function resolveBaseUrlForEnqueueLinksFiltering({
return finalUrlOrigin;
}
- return undefined;
+ return originalUrlOrigin;
}
// Always enqueue urls that are from the same origin in all other cases, as the filtering happens on the original request url, even if there was a redirect
diff --git a/test/core/enqueue_links/enqueue_links.test.ts b/test/core/enqueue_links/enqueue_links.test.ts
index 3d15f86d7..a4231d7e4 100644
--- a/test/core/enqueue_links/enqueue_links.test.ts
+++ b/test/core/enqueue_links/enqueue_links.test.ts
@@ -911,6 +911,23 @@ describe('enqueueLinks()', () => {
expect(enqueued[2].userData).toEqual({});
});
+ test('keeps filtering by the original domain with the strategy of same-domain after an off-domain redirect', async () => {
+ const { enqueued, requestQueue } = createRequestQueueMock();
+ await cheerioCrawlerEnqueueLinks({
+ options: { strategy: EnqueueStrategy.SameDomain },
+ $,
+ requestQueue,
+ originalRequestUrl: 'https://example.com',
+ finalRequestUrl: 'https://another.com/',
+ });
+
+ expect(enqueued).toHaveLength(3);
+
+ expect(enqueued[0].url).toBe('https://example.com/a/b/first');
+ expect(enqueued[1].url).toBe('https://example.com/a/second');
+ expect(enqueued[2].url).toBe('https://example.com/a/b/third');
+ });
+
test('correctly resolves relative URLs with `urls` option', async () => {
const { enqueued, requestQueue } = createRequestQueueMock();
await cheerioCrawlerEnqueueLinks({