Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
add more comments + save results to dataset
Tobey6cf404a46e99
diff --git a/examples/forms.js b/examples/forms.jsindex 25b357b54..ccd2e1d34 100644--- a/examples/forms.js+++ b/examples/forms.js@@ -6,6 +6,9 @@* using headless Chrome / Puppeteer.* The actor first fills in the search term, repository owner, start date and* language of the repository, then submits the form and prints out the results.+ * Finally, the results are saved either on the Apify platform to the default+ * <a href="https://sdk.apify.com/docs/api/dataset" target="_blank">dataset</a>+ * or on the local machine as JSON files in ./apify_storage/datasets/default.** To run this example on the Apify Platform, select the `Node.js 8 + Chrome on Debian (apify/actor-node-chrome)` base image* on the source tab of your actor configuration.@@ -18,15 +21,18 @@ Apify.main(async () => {const browser = await Apify.launchPuppeteer();// Create and navigate new page+ console.log('Fill in search form');const page = await browser.newPage();await page.goto('https://github.com/search/advanced');+ // Fill form fields and select desired search optionsconsole.log('Fill in search form');await page.type('#adv_code_search input.js-advanced-search-input', 'apify-js');await page.type('#search_from', 'apifytech');await page.type('#search_date', '>2015');await page.select('select#search_language', 'JavaScript');+ // Submit the form and wait for full load of next pageconsole.log('Submit search form');await Promise.all([page.waitForNavigation(),@@ -34,6 +40,12 @@ Apify.main(async () => {]);// Obtain and print list of search results- const results = await page.$$eval('div.codesearch-results > div h3 a', nodes => nodes.map(node => node.innerText));+ const results = await page.$$eval('div.codesearch-results ul.repo-list li h3 a', nodes => nodes.map(node => ({+ url: node.href,+ name: node.innerText+ })));console.log('Results:', results);++ // Store data in default dataset+ await Apify.pushData(results);});