Skip to content

Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥

Commits on main

fix(crawl-status): improve finished checking
Gergő Móricz committed 1 year ago
078c067
feat(v1): crawl/batch scrape errors route
Gergő Móricz committed 1 year ago
e653127
fix(kickoff): mark as finished if it errors out
Gergő Móricz committed 1 year ago
dcd3d6d
fix(crawler): bad urls from sitemap
Gergő Móricz committed 1 year ago
5992c57
fix(requests.http): map
Gergő Móricz committed 1 year ago
237d0dc
805bfa4
fix(crawler): recognize sitemaps in robots.txt
Gergő Móricz committed 1 year ago
23bb172
faf58df
feat: adjust CI testing
Móricz Gergő committed 1 year ago
de08b37
fix(queue-worker): fill out time taken on failure too
Móricz Gergő committed 1 year ago
4a947e3
Update package.json
Nicolas committed 1 year ago
8e57fde
Nick: error details for extract
Nicolas committed 1 year ago
80e5acf
fix(html,markdown): always get absolute links
Gergő Móricz committed 1 year ago
6c94db7
feat(html): always pick largest image from srcset
Gergő Móricz committed 1 year ago
e824303
fix(url): allow domains with ports
Gergő Móricz committed 1 year ago
655753c
Update model-prices.ts
Nicolas committed 1 year ago
ca14c65
Nick: introduce llm-usage cost analysis
Nicolas committed 1 year ago
4db0232
Update package.json
Nicolas committed 1 year ago
de14c0a
feat(js-sdk): add further options to checkxstatus
Gergő Móricz committed 1 year ago
c393799
feat(queue-worker): proactive job cancel
Gergő Móricz committed 1 year ago
cbe67d8
fix(blocklist): unblock
Gergő Móricz committed 1 year ago
ec039dc
fix(v1/crawl-status): fix stuck on 0 jobs
Gergő Móricz committed 1 year ago
dde3aeb
fix(queue-worker/billing): fix crawl overbilling
Gergő Móricz committed 1 year ago
ce2f6ff
Update check-fire-engine.ts
Nicolas committed 1 year ago
db89e36
Nick: streaming callback experimental
Nicolas committed 1 year ago
61e6af2
23d3257
Update extract-redis.ts
Nicolas committed 1 year ago
c323c64
Update extraction-service.ts
Nicolas committed 1 year ago
2dc87a2