Skip to content

armgit / camel

Public Imported from GitHub
0

🐫 CAMEL: The first and the best multi-agent framework. Finding the Scaling Law of Agents. https://www.camel-ai.org

15.7 KB
Raw Blame

title: "Loaders" icon: loader

doc_code_map: - "camel/loaders/**/*.py"

CAMEL’s Loaders provide flexible ways to ingest and process all kinds of data structured files, unstructured text, web content, and even OCR from images. They power your agent’s ability to interact with the outside world. Additionally, several data readers were added, including `Apify Reader`, `Chunkr Reader`, `Firecrawl Reader`, `Jina_url Reader`, and `Mistral Reader`, which enable retrieval of external data for improved data integration and analysis. ## Types

{" "} Handles core file input/output for formats like PDF, DOCX, HTML, and more.

Lets you represent, read, and process structured files.

{" "} Powerful ETL for parsing, cleaning, extracting, chunking, and staging unstructured data.

Perfect for RAG pipelines and pre-processing.

{" "} Integrates with Apify to automate web workflows and scraping.

Supports authentication, actor management, and dataset operations via API.

{" "} Connects to the Chunkr API for document chunking, segmentation, and OCR.

Handles everything from simple docs to scanned PDFs.

{" "} Converts entire websites into LLM-ready markdown using the Firecrawl API.

Useful for quickly ingesting web content as clean text.

{" "} Uses Jina AI’s URL reading service to cleanly extract web content.

Designed for LLM-friendly extraction from any URL.

{" "} Lightweight tool to convert files (HTML, DOCX, PDF, etc.) into Markdown.

Ideal for prepping documents for LLM ingestion or analysis.

{" "} Integrates Mistral AI’s OCR service for extracting text from images and PDFs.

Supports both local and remote file processing for various formats.

Get Started

This module is designed to read files of various formats, extract their contents, and represent them as `File` objects, each tailored to handle a specific file type. ```python base_io_example.py from io import BytesIO from camel.loaders import create_file_from_raw_bytes

Read a pdf file from disk

with open("test.pdf", "rb") as file: file_content = file.read()

Use the create_file function to create an object based on the file extension

file_obj = create_file_from_raw_bytes(file_content, "test.pdf")

Once you have the File object, you can access its content

print(file_obj.docs[0]["page_content"])

</CodeGroup>
</Card>

---

<Card title="Using Unstructured IO" icon="puzzle">
To get started with the <code>Unstructured IO</code> module, just import and initialize it. You can parse, clean, extract, chunk, and stage data from files or URLs. Here’s how you use it step by step:

<br />

<strong>1. Parse unstructured data from a file or URL:</strong>
<CodeGroup>
```python unstructured_io_parse.py
from camel.loaders import UnstructuredIO

uio = UnstructuredIO()
example_url = (
    "https://www.cnn.com/2023/01/30/sport/empire-state-building-green-"
    "philadelphia-eagles-spt-intl/index.html"
)
elements = uio.parse_file_or_url(example_url)
print(("\n\n".join([str(el) for el in elements])))
> > > The Empire State Building was lit in green and white to celebrate the Philadelphia Eagles’ victory in the NFC Championship game on Sunday – a decision that’s sparked a bit of a backlash in the Big Apple.
> > > The Eagles advanced to the Super Bowl for the first time since 2018 after defeating the San Francisco 49ers 31-7, and the Empire State Building later tweeted how it was marking the occasion.
> > > Fly @Eagles Fly! We’re going Green and White in honor of the Eagles NFC Championship Victory. pic.twitter.com/RNiwbCIkt7— Empire State Building (@EmpireStateBldg)
> > > January 29, 2023...

2. Clean unstructured text data:

example_dirty_text = "\u201cSome dirty text\u201d with  extra spaces and \u2013 dashes."
options = [
    ("replace_unicode_quotes", {}),
    ("clean_dashes", {}),
    ("clean_non_ascii_chars", {}),
    ("clean_extra_whitespace", {}),
]
cleaned_text = uio.clean_text_data(
    text=example_dirty_text,
    clean_options=options,
)
print(cleaned_text)
>>> "Some dirty text" with extra spaces and dashes.

3. Extract data from text (for example, emails):

example_email_text = "Contact me at [email protected]."

extracted_text = uio.extract_data_from_text(
text=example_email_text,
extract_type="extract_email_address"
)
print(extracted_text)
>>> ['[email protected]']

4. Chunk content by title:

chunks = uio.chunk_elements(elements=elements, chunk_type="chunk_by_title")

for chunk in chunks:
    print(chunk)
    print("\n" + "-" * 80)
>>> The Empire State Building was lit in green and white to celebrate the Philadelphia Eagles’ victory in the NFC Championship game on Sunday – a decision that’s sparked a bit of a backlash in the Big Apple.
>>> --------------------------------------------------------------------------------
>>> Fly @Eagles Fly! We’re going Green and White in honor of the Eagles NFC Championship Victory. pic.twitter.com/RNiwbCIkt7— Empire State Building (@EmpireStateBldg)
>>> --------------------------------------------------------------------------------
>>> January 29, 2023

5. Stage elements for use with other platforms:

staged_element = uio.stage_elements(elements=elements, stage_type="stage_for_baseplate")
print(staged_element)
>>> {'rows': [{'data': {'type': 'UncategorizedText', 'element_id': 'e78902d05b0cb1e4c38fc7a79db450d5', 'text': 'CNN\n        \xa0—'}, 'metadata': {'filetype': 'text/html', 'languages': ['eng'], 'page_number': 1, 'url': 'https://www.cnn.com/2023/01/30/sport/empire-state-building-green-philadelphia-eagles-spt-intl/index.html', 'emphasized_text_contents': ['CNN'], 'emphasized_text_tags': ['span']}}, ...

This guide gets you started with Unstructured IO. For more, see the Unstructured IO Documentation.
Initialize the Apify client, set up the required actors and parameters, and run the actor. ```python apify_reader.py from camel.loaders import Apify

apify = Apify()

run_input = { "startUrls": [{"url": "https://www.camel-ai.org/"}], "maxCrawlDepth": 0, "maxCrawlPages": 1, } actor_result = apify.run_actor( actor_id="apify/website-content-crawler", run_input=run_input ) dataset_result = apify.get_dataset_items( dataset_id=actor_result["defaultDatasetId"] ) print(dataset_result)

```markdown apify_output.md
>>>[{'url': 'https://www.camel-ai.org/', 'crawl': {'loadedUrl': 'https://www.camel-ai.org/', ...}, 'metadata': {'canonicalUrl': 'https://www.camel-ai.org/', ...}, ... }]

Firecrawl Reader provides a simple way to turn any website into LLM-ready markdown format. Here’s how you can use it step by step: First, create a Firecrawl client and crawl a specific URL.
<CodeGroup>
```python firecrawl_crawl.py
from camel.loaders import Firecrawl

firecrawl = Firecrawl()
response = firecrawl.crawl(url="https://www.camel-ai.org/about")
print(response["status"])  # Should print "completed" when done
```
```markdown crawl_status.md
>>>completed
```
</CodeGroup>
When the status is <code>"completed"</code>, the content extraction is done and you can retrieve the results.
Once finished, access the LLM-ready markdown directly from the response:
<CodeGroup>
```python firecrawl_markdown.py
print(response["data"][0]["markdown"])
```
```markdown extracted_markdown.md
>>>Camel-AI Team

We are finding the
scaling law of agent

🐫 CAMEL is an open-source library designed for the study of autonomous and
communicative agents. We believe that studying these agents on a large scale
offers valuable insights into their behaviors, capabilities, and potential
risks. To facilitate research in this field, we implement and support various
types of agents, tasks, prompts, models, and simulated environments.

**We are** always looking for more **contributors** and **collaborators**.
Contact us to join forces via [Slack](https://join.slack.com/t/camel-kwr1314/
shared_invite/zt-1vy8u9lbo-ZQmhIAyWSEfSwLCl2r2eKA)
or [Discord](https://discord.gg/CNcNpquyDc)...
```
</CodeGroup>

That’s it. With just a couple of lines, you can turn any website into clean markdown, ready for LLM pipelines or further processing.
Chunkr Reader allows you to process PDFs (and other docs) in chunks, with built-in OCR and format control. Below is a basic usage pattern:

Initialize the ChunkrReader and ChunkrReaderConfig, set the file path and chunking options, then submit your task and fetch results:

import asyncio
from camel.loaders import ChunkrReader, ChunkrReaderConfig

async def main():
    chunkr = ChunkrReader()

    config = ChunkrReaderConfig(
        chunk_processing=512,      # Example: target chunk length
        ocr_strategy="Auto",       # Example: OCR strategy
        high_resolution=False      # False for faster processing (old "Fast" model)
    )

    # Replace with your actual file path.
    file_path = "/path/to/your/document.pdf"
    try:
        task_id = await chunkr.submit_task(
            file_path=file_path,
            chunkr_config=config,
        )
        print(f"Task ID: {task_id}")

        # Poll and fetch the output.
        if task_id:
            task_output_json_str = await chunkr.get_task_output(task_id=task_id)
            if task_output_json_str:
                print("Task Output:")
                print(task_output_json_str)
            else:
                print(f"Failed to get output for task {task_id}, or task did not succeed/was cancelled.")
    except ValueError as e:
        print(f"An error occurred during task submission or retrieval: {e}")
    except FileNotFoundError:
        print(f"Error: File not found at {file_path}. Please check the path.")
    except Exception as e:
        print(f"An unexpected error occurred: {e}")

if __name__ == "__main__":
    print("To run this example, replace '/path/to/your/document.pdf' with a real file path, ensure CHUNKR_API_KEY is set, and uncomment 'asyncio.run(main())'.")
    # asyncio.run(main()) # Uncomment to run the example

A successful task returns a chunked structure like this:

> > > Task ID: 7becf001-6f07-4f63-bddf-5633df363bbb
> > > Task Output:
> > > { "task_id": "7becf001-6f07-4f63-bddf-5633df363bbb", "status": "Succeeded", "created_at": "2024-11-08T12:45:04.260765Z", "finished_at": "2024-11-08T12:45:48.942365Z", "expires_at": null, "message": "Task succeeded", "output": { "chunks": [ { "segments": [ { "segment_id": "d53ec931-3779-41be-a220-3fe4da2770c5", "bbox": { "left": 224.16666, "top": 370.0, "width": 2101.6665, "height": 64.166664 }, "page_number": 1, "page_width": 2550.0, "page_height": 3300.0, "content": "Large Language Model based Multi-Agents: A Survey of Progress and Challenges", "segment_type": "Title", "ocr": null, "image": "https://chunkmydocs-bucket-prod.storage.googleapis.com/.../d53ec931-3779-41be-a220-3fe4da2770c5.jpg?...", "html": "<h1>Large Language Model based Multi-Agents: A Survey of Progress and Challenges</h1>", "markdown": "# Large Language Model based Multi-Agents: A Survey of Progress and Challenges\n\n" } ], "chunk_length": 11 }, { "segments": [ { "segment_id": "7bb38fc7-c1b3-4153-a3cc-116c0b9caa0a", "bbox": { "left": 432.49997, "top": 474.16666, "width": 1659.9999, "height": 122.49999 }, "page_number": 1, "page_width": 2550.0, "page_height": 3300.0, "content": "Taicheng Guo 1 , Xiuying Chen 2 , Yaqi Wang 3 \u2217 , Ruidi Chang , Shichao Pei 4 , Nitesh V. Chawla 1 , Olaf Wiest 1 , Xiangliang Zhang 1 \u2020", "segment_type": "Text", "ocr": null, "image": "https://chunkmydocs-bucket-prod.storage.googleapis.com/.../7bb38fc7-c1b3-4153-a3cc-116c0b9caa0a.jpg?...", "html": "<p>Taicheng Guo 1 , Xiuying Chen 2 , Yaqi Wang 3 \u2217 , Ruidi Chang , Shichao Pei 4 , Nitesh V. Chawla 1 , Olaf Wiest 1 , Xiangliang Zhang 1 \u2020</p>", "markdown": "Taicheng Guo 1 , Xiuying Chen 2 , Yaqi Wang 3 \u2217 , Ruidi Chang , Shichao Pei 4 , Nitesh V. Chawla 1 , Olaf Wiest 1 , Xiangliang Zhang 1 \u2020\n\n" } ], "chunk_length": 100 } ] } }

Jina Reader provides a convenient interface to extract clean, LLM-friendly content from any URL in a chosen format (like markdown):
from camel.loaders import JinaURLReader
from camel.types.enums import JinaReturnFormat

jina_reader = JinaURLReader(return_format=JinaReturnFormat.MARKDOWN)
response = jina_reader.read_content("https://docs.camel-ai.org/")
print(response)

MarkitDown Reader lets you convert files (like HTML or docs) into LLM-ready markdown with a single line.
from camel.loaders import MarkItDownLoader

loader = MarkItDownLoader()
response = loader.convert_file("demo.html")

print(response)

Example output:

> > > Welcome to CAMEL’s documentation! — CAMEL 0.2.61 documentation

[Skip to main content](https://docs.camel-ai.org/#main-content)
...

Mistral Reader offers OCR and text extraction from both PDFs and images, whether local or remote. Just specify the file path or URL:
from camel.loaders import MistralReader

mistral_reader = MistralReader()

# Extract text from a PDF URL
url_ocr_response = mistral_reader.extract_text(
    file_path="https://arxiv.org/pdf/2201.04234", pages=[5]
)
print(url_ocr_response)

You can also extract from images or local files:

# Extract text from an image URL
image_ocr_response = mistral_reader.extract_text(
    file_path="https://raw.githubusercontent.com/mistralai/cookbook/refs/heads/main/mistral/ocr/receipt.png",
    is_image=True,
)
print(image_ocr_response)
# Extract text from a local PDF file
local_ocr_response = mistral_reader.extract_text("path/to/your/document.pdf")
print(local_ocr_response)

Response includes structured page data, markdown content, and usage details.

> > > pages=[OCRPageObject(index=5, markdown='![img-0.jpeg](./images/img-0.jpeg)\n\nFigure 2: Scatter plot of predicted accuracy versus (true) OOD accuracy. Each point denotes a dif...',
> > > images=[OCRImageObject(id='img-0.jpeg', ...)], dimensions=OCRPageDimensions(...))] model='mistral-ocr-2505-completion' usage_info=...