Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
TypeScript aiai-agentsai-crawlerai-scrapingai-searchcrawlerdata-extractionhtml-to-markdownllmmarkdownscraperscrapingweb-crawlerweb-dataweb-data-extractionweb-scraperweb-scrapingweb-searchwebscraping
Commits on main
Honor Exchange terms overrides while preserving universal scrape gates (#5026)
f0dd059 Developers Digest committed 1 month ago
feat(api): make checkPromptInjection a top-level scrape option (#5024)
fa7ec5f Gergő Móricz committed 1 month ago
fix(api): send Retry-After on per-minute rate limit 429s (#5025)
ec849f3 Gaurav Chadha committed 1 month ago
fix(api): reject deeply nested sitemaps instead of crashing the process (#5016)
8a0e91b Gaurav Chadha committed 1 month ago
fix(security): remove braces, sprintf-js, csv-parse and postcss-selector-parser from every lockfile (#5012)
244f83b Gergő Móricz committed 1 month ago
feat(api): point scrapes of search results pages to /search (#5003)
b00172b David Tippett committed 1 month ago
refactor(api): remove built-in agent hints (#4992)
7cca3ed Rakshith Ramprakash committed 1 month ago
fix(api): keep blocklist keyword exemptions off URLs the Exchange claims (#5011)
f9f2e3d Gergő Móricz committed 1 month ago
fix(api): check robots.txt before fetching a crawl's start URL (#5005)
8d737f1 Gergő Móricz committed 1 month ago
feat(api): log every auth denial as one canonical auth/denied line (#5008)
e3986eb Gergő Móricz committed 1 month ago
fix(security): override geoip-country's ip-address to 10.7.3 in apps/api (#5007)
27220e8 Gergő Móricz committed 1 month ago
fix(security): patch open Dependabot alerts in examples, Rust SDK and Go html-to-md (#5006)
2a48715 Gergő Móricz committed 1 month ago
feat(js-sdk): abortable crawl() that cancels its job; cancelCrawl returns false on 409 (#4898)
23b68ae claude[bot] committed 1 month ago
fix(python-sdk): cancel the crawl when async crawl() is cancelled; cancel_crawl returns False on 409 (#4897)
a73af1d claude[bot] committed 1 month ago
fix(api): record robots.txt denials on crawls so the status warning and robotsBlocked work (#4900)
9788805 claude[bot] committed 1 month ago
fix(api): report crawl completedAt correctly when no page succeeds (#4899)
924298d claude[bot] committed 1 month ago
feat(api): record token usage and team on TypeSafe spans (#5004)
0010c5f Abimael Martell committed 1 month ago
fix(api): name the option that keeps a provider out of a blocklisted URL (#5002)
6c673c9 Gergő Móricz committed 1 month ago
fix(api): admit Exchange-served blocklisted URLs in v2 batch scrape (#4999)
bd654c5 Gergő Móricz committed 1 month ago
feat(rust-sdk): add DocumentMetadata::provider for Exchange scrapes (#5001)
7599de4 Gergő Móricz committed 1 month ago
feat(api): report the serving provider as metadata.provider on Exchange scrapes (#5000)
a283324 Gergő Móricz committed 1 month ago
fix(api): return 4xx for Exchange not_found and invalid_exchange_url refusals (#4998)
ecd7934 Gergő Móricz committed 1 month ago
fix(sdks): surface success:false scrape errors in Go, Java, .NET, Ruby and PHP (#4971)
88569f4 Gaurav Chadha committed 1 month ago
fix(sdks): match the Alexandria client timeout to the API's 120 s cap (#4995)
9ae2451 claude[bot] committed 1 month ago
feat(api): evaluate provider-supplied response guidance rules (#4991)
0b25e17 Rakshith Ramprakash committed 1 month ago
feat(elixir-sdk): agent thread_id, mode and exchange (#4990)
1bf1215 Gergő Móricz committed 1 month ago
feat(php-sdk): add agent exchange options, threadId and mode (#4989)
36f31d9 Gergő Móricz committed 1 month ago
feat(rust-sdk): add agent exchange, threadId and mode options (#4988)
776a87f Gergő Móricz committed 1 month ago