Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
TypeScript aiai-agentsai-crawlerai-scrapingai-searchcrawlerdata-extractionhtml-to-markdownllmmarkdownscraperscrapingweb-crawlerweb-dataweb-data-extractionweb-scraperweb-scrapingweb-searchwebscraping
Commits on main
Merge branch 'nsc/extract-queue' of https://github.com/mendableai/firecrawl into nsc/extract-queue
5163635 Nicolas committed 1 year ago
fix(crawl): never invalidate first crawl scrape if redirects
7d73ebd Móricz Gergő committed 1 year ago
fix(crawler): also take the hostname of the base url when determining isInternalLink
35d1d85 Móricz Gergő committed 1 year ago
Merge pull request #1038 from 1101-1/add_actual_random_useragent
b82cfa8 Nicolas committed 1 year ago
Merge pull request #1034 from mendableai/sdk/fixed-none-undefined-on-response
ceb2104 Nicolas committed 1 year ago
fix(v1/crawl-status): handle job's returnvalue being explicitly null (db race)
461842f Gergő Móricz committed 1 year ago
fix(queue-worker): only do redirect handling logic on crawls, not batch scrape
b92a4eb Gergő Móricz committed 1 year ago
Merge pull request #1037 from mendableai/nsc/semantic-index-extract
6bfdbcd Nicolas committed 1 year ago