Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
TypeScript aiai-agentsai-crawlerai-scrapingai-searchcrawlerdata-extractionhtml-to-markdownllmmarkdownscraperscrapingweb-crawlerweb-dataweb-data-extractionweb-scraperweb-scrapingweb-searchwebscraping
Commits on main
fix(queue-worker/crawl): only report successful page count in num_docs (#1179)
73e7884 Gergő Móricz committed 1 year ago
feat(v1/checkCredits): say "tokens" instead of "credits" if out of tokens (#1178)
1d9a0b9 Gergő Móricz committed 1 year ago
Merge pull request #1175 from mayooear/feat/gemini-sub-links
582bbf8 Eric Ciarla committed 1 year ago
fix: update gemini library. extract pdf links from markdowncontent
32f9897 mayo committed 1 year ago
Merge pull request #1173 from mayooear/feat/gemini-sub-links
a1b7d6e Eric Ciarla committed 1 year ago
fix(html-transformer): Update free_string function parameter type (#1163)
78094e2 Carter LaSalle committed 1 year ago
chore: add Serper and Search API envs to docker-compose (#1147)
52c0f78 Luke Martin committed 1 year ago
Merge pull request #1161 from aparupganguly/feature/gemini-2.0-crawler
ef9aad8 Eric Ciarla committed 1 year ago
Merge pull request #1146 from mendableai/feat/wait-validation
3608256 AdemÃlson Tonato committed 1 year ago
[FIR-796] feat(api/types): Add action and wait time validation for scrape requests
843cec9 AdemÃlson F. Tonato committed 1 year ago
Merge pull request #1144 from aparupganguly/feature/o3-mini-job-resource
2b7b740 Eric Ciarla committed 1 year ago
Merge pull request #1139 from aparupganguly/feature/o3-mini-review-summarizer
42f4f7e Eric Ciarla committed 1 year ago
fix(scrape): allow getting valid JSON via rawHtml (FIR-852) (#1138)
892f3a4 Gergő Móricz committed 1 year ago
Merge pull request #1117 from mendableai/feat/blocked-domains
996352c AdemÃlson Tonato committed 1 year ago
fix(crawl-redis/generateURLPermutations): dedupe index.html/index.php/slash/bare URL ends (FIR-827) (#1134)
34fe360 Gergő Móricz committed 1 year ago