Skip to content

Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥

Commits on main

added today to extract prompts
rafaelmmiller committed 1 year ago
f1cd891
fix(scrapeQueue): change expiry to 1 hour
Gergő Móricz committed 1 year ago
a1efe33
feat(crawl-status): same for v0
Gergő Móricz committed 1 year ago
a7b56ab
95ce3c3
feat(extract): add log on 0 links
Gergő Móricz committed 1 year ago
6f696d3
5d56627
feat(extract): add original schema to logs
Móricz Gergő committed 1 year ago
9da51a7
fix build error
Móricz Gergő committed 1 year ago
561f018
feat(sitemap): change sitemap logging
Móricz Gergő committed 1 year ago
6557365
feat(extract): add logging
Móricz Gergő committed 1 year ago
d3518e8
fix(sitemap): increase limit to 20
Móricz Gergő committed 1 year ago
434a435
fix(sitemap): increase limit
Móricz Gergő committed 1 year ago
1e28ba2
fix(sitemap): better ordering
Móricz Gergő committed 1 year ago
bee2b28
feat(sitemap): reenable fallback to tlsclient
Móricz Gergő committed 1 year ago
3761eb1
feat(sitemap): propagate crawlid
Móricz Gergő committed 1 year ago
aa2c369
fix(crawler): dumb sitemap limit
Móricz Gergő committed 1 year ago
a922aac
fix(sitemap): temporarily disable tlsclient
Móricz Gergő committed 1 year ago
51a0e23
Update cache.ts
Nicolas committed 1 year ago
d162247
Nick: formatting done
Nicolas committed 1 year ago
498558d
Nick: rm logs
Nicolas committed 1 year ago
994e1eb
Reapply "Nick:"
Nicolas committed 1 year ago
56f048a
Revert "Nick:"
Nicolas committed 1 year ago
4b4385c
Nick:
Nicolas committed 1 year ago
6718ce8
fix(extraction-service): marginally improve logging
Gergő Móricz committed 1 year ago
208bd4c
feat(sitemap): switch around engine order
Gergő Móricz committed 1 year ago
ed92922
fix(v1/map): add wrapper around tryGetSitemap
Gergő Móricz committed 1 year ago
5a039e7