AutoGPT is the vision of accessible AI for everyone, to use and to build on. Our mission is to provide the tools, so that you can focus on what matters.
Tavily Crawl
Blocks for crawling a website and extracting page content with Tavily.
Tavily Crawl
What it is
Crawls a website with Tavily, following links from the root URL and extracting page content
How it works
Starting from a root URL, the block follows links up to max_depth and extracts the content of each page it visits, stopping at limit pages. Optional natural-language instructions steer which pages to follow (for example, "Find all the API reference pages"), and extract_depth and format control how much content is captured and whether it comes back as markdown or text.
Each crawled page is returned on results (and one at a time on result) with its extracted content. A crawl combines mapping and extraction, so its credit cost is the sum of both per Tavily's schedule; actual spend is read from the API's usage report.
Inputs
| Input | Description | Type | Required |
|---|---|---|---|
| url | The root URL to start crawling from | str | Yes |
| limit | Maximum number of pages to crawl | int | No |
| max_depth | Maximum link depth from the root URL | int | No |
| instructions | Natural language instructions guiding which pages to crawl (e.g. 'Find all the API reference pages') | str | No |
| extract_depth | Depth of the extraction: basic or advanced (retrieves more data, including tables and embedded content) | "basic" | "advanced" | No |
| format | The format of the extracted content | "markdown" | "text" | No |
Outputs
| Output | Description | Type |
|---|---|---|
| error | Error message if the crawl failed | str |
| results | List of crawled pages with their content | List[TavilyPageContent] |
| result | Single crawled page | TavilyPageContent |
Possible use case
Documentation Ingestion: Crawl a docs site and extract every page to build a knowledge base or RAG pipeline.
Targeted Site Harvesting: Use instructions to gather only pricing, product, or reference pages from a large site.
Competitive Content Snapshots: Capture the content of a section of a site in one pass for analysis or change tracking.