Skip to content

Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥

Commits on main

feat(crawl): allowSubdomain
Gergő Móricz committed 2 years ago
79a75e0
fix schema
rafaelmmiller committed 2 years ago
2fb8a3c
Rafa: removed throw error and added map to requests
rafaelmmiller committed 2 years ago
53134b7
36cf49c
Update CONTRIBUTING.md
Nicolas committed 2 years ago
91caa01
Update README.md
Nicolas committed 2 years ago
1328ae0
Create README.md
Eric Ciarla committed 2 years ago
a313367
added team_id to scrape-status endpoint
rafaelmmiller committed 2 years ago
77e152c
fix(crawl-redis): ordered push to wrong side of list
Gergő Móricz committed 2 years ago
31a0471
fix(webhook): add logging
Gergő Móricz committed 2 years ago
1a0f13c
fix(map): make sitemapOnly simpler
Gergő Móricz committed 2 years ago
1b032b0
fix(map): ignore limit when using sitemapOnly
Gergő Móricz committed 2 years ago
a4d3dba
4cddcd5
fix(crawler): treat XML files as sitemaps (temporarily)
Gergő Móricz committed 2 years ago
350d00d
ca2e33d
fix(v1/types): better timeout primitives
Gergő Móricz committed 2 years ago
7b02c45
fix(v1/batch/scrape): raise default timeout
Gergő Móricz committed 2 years ago
c95a4a2
fix(scrapeURL/playwright): JSON body fix
Móricz Gergő committed 2 years ago
3a342bf
Update map.ts
Nicolas committed 2 years ago
3c1b190
Merge branch 'nsc/sitemap-only'
Nicolas committed 2 years ago
9519897
Nick:
Nicolas committed 2 years ago
7f084c6
Update fireEngine.ts
Nicolas committed 2 years ago
3fcdf57
Nick: moved away from axios
Nicolas committed 2 years ago
d62f12c
Nick: sitemap only
Nicolas committed 2 years ago
f155449
fix(batch/scrape/webhook): add batch_scrape.started
Móricz Gergő committed 2 years ago
431e64e
Update package.json
Nicolas committed 2 years ago
7bca448
feat(v1/batch/scrape): webhooks
Móricz Gergő committed 2 years ago
df05124
Update extract.ts
Nicolas committed 2 years ago
91f4fd8