Skip to content

Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥

Commits on main

Update types.ts
Nicolas committed 1 year ago
dd14744
Update index.ts
Nicolas committed 1 year ago
9fdcfb9
11af214
Update apps/api/src/lib/extract/extraction-service.ts
Gergő Móricz committed 1 year ago
1f2a76f
Nick:
Nicolas committed 1 year ago
eb25454
crawl incomplete issues
Gergő Móricz committed 1 year ago
c6a6379
various queue fixes
Gergő Móricz committed 1 year ago
ccfada9
Nick: wip
Nicolas committed 1 year ago
86e34d7
add comment
Móricz Gergő committed 1 year ago
7a03275
fix(crawl): never invalidate first crawl scrape if redirects
Móricz Gergő committed 1 year ago
7d73ebd
fix(crawl): don't push rawhtml to db unless requested
Móricz Gergő committed 1 year ago
b96b97e
Merge branch 'main' into nsc/extract-queue
Nicolas committed 1 year ago
bb27594
736c367
Update canonical-url.test.ts
Nicolas committed 1 year ago
d48ddb8
Nick: url normalization
Nicolas committed 1 year ago
f2e0bfb
Nick: added canonical tests
Nicolas committed 1 year ago
f25c0c6
Nick: from cache fixes
Nicolas committed 1 year ago
aef040b
Update cache.ts
Nicolas committed 1 year ago
05e845a
Nick: fixed
Nicolas committed 1 year ago
c655c68
Nick: fixed
Nicolas committed 1 year ago
a4f7c38
Update queue-worker.ts
Nicolas committed 1 year ago
8df1c67
Update url-processor.ts
Nicolas committed 1 year ago
499479c