Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

899 B
Raw Blame
import { expect, getActorTestDir, hasNestedKey, initialize, runActor } from '../tools.mjs';

const testActorDirname = getActorTestDir(import.meta.url);
await initialize(testActorDirname);

const { stats, defaultKeyValueStoreItems } = await runActor(testActorDirname);

// All requests should fail to test the error snapshots
await expect(stats.requestsFailed === 4, 'All requests failed');

let totalErrorHtmlFiles = 0;
for (const error of Object.values(stats.errors)) {
    if (hasNestedKey(error, 'firstErrorHtmlUrl')) {
        totalErrorHtmlFiles++;
    }
}

// Count of error HTML files stored in the stats to make sure they are saved
await expect(totalErrorHtmlFiles === 3, 'Number of HTML error files in stats should be 3');
// Count of error HTML files stored in the Key-Value store
await expect(defaultKeyValueStoreItems.length === 3, 'Number of HTML error files in KV store should be 3');