Skip to content

AutoGPT is the vision of accessible AI for everyone, to use and to build on. Our mission is to provide the tools, so that you can focus on what matters.

2.2 KB
Raw Blame

Tavily Crawl

Blocks for crawling a website and extracting page content with Tavily.

Tavily Crawl

What it is

Crawls a website with Tavily, following links from the root URL and extracting page content

How it works

Starting from a root URL, the block follows links up to max_depth and extracts the content of each page it visits, stopping at limit pages. Optional natural-language instructions steer which pages to follow (for example, "Find all the API reference pages"), and extract_depth and format control how much content is captured and whether it comes back as markdown or text.

Each crawled page is returned on results (and one at a time on result) with its extracted content. A crawl combines mapping and extraction, so its credit cost is the sum of both per Tavily's schedule; actual spend is read from the API's usage report.

Inputs

Input Description Type Required
url The root URL to start crawling from str Yes
limit Maximum number of pages to crawl int No
max_depth Maximum link depth from the root URL int No
instructions Natural language instructions guiding which pages to crawl (e.g. 'Find all the API reference pages') str No
extract_depth Depth of the extraction: basic or advanced (retrieves more data, including tables and embedded content) "basic" | "advanced" No
format The format of the extracted content "markdown" | "text" No

Outputs

Output Description Type
error Error message if the crawl failed str
results List of crawled pages with their content List[TavilyPageContent]
result Single crawled page TavilyPageContent

Possible use case

Documentation Ingestion: Crawl a docs site and extract every page to build a knowledge base or RAG pipeline.

Targeted Site Harvesting: Use instructions to gather only pricing, product, or reference pages from a large site.

Competitive Content Snapshots: Capture the content of a section of a site in one pass for analysis or change tracking.