Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

fix: improve error message when no default route provided

Martin Adámek04c3b6ac2fd1
diff --git a/packages/core/src/router.ts b/packages/core/src/router.ts
index 68dae4f83..98c138075 100644
--- a/packages/core/src/router.ts
+++ b/packages/core/src/router.ts
@@ -88,7 +88,8 @@ export class Router<Context extends CrawlingContext> {
}
if (!label) {
- throw new MissingRouteError(`No default route set up!`);
+ // eslint-disable-next-line max-len
+ throw new MissingRouteError(`No default route set up. Please specify 'requestHandler' option or provide default route via 'crawler.router.addDefaultRoute()'.`);
}
throw new MissingRouteError(`Route not found for label '${String(label)}' and no default route set up!`);
diff --git a/test/core/crawlers/cheerio_crawler.test.ts b/test/core/crawlers/cheerio_crawler.test.ts
index b7e5687b1..42082bf0a 100644
--- a/test/core/crawlers/cheerio_crawler.test.ts
+++ b/test/core/crawlers/cheerio_crawler.test.ts
@@ -268,6 +268,19 @@ describe('CheerioCrawler', () => {
});
});
+ test('should throw when no requestHandler nor default route provided', async () => {
+ const requestList = await getRequestListForMirror(port);
+
+ const cheerioCrawler = new CheerioCrawler({
+ requestList,
+ minConcurrency: 2,
+ maxConcurrency: 2,
+ });
+
+ // eslint-disable-next-line max-len
+ await expect(cheerioCrawler.run()).rejects.toThrow(`No default route set up. Please specify 'requestHandler' option or provide default route via 'crawler.router.addDefaultRoute()'.`);
+ });
+
test('should ignore ssl by default', async () => {
const sources = [
{ url: 'http://example.com/?q=1' },