Skip to content

A curated collection of practical AI projects implementing OCR systems, RAG, AI agents, and other AI use cases.

Update: Switch OCR engine to Mistral Large 3 and implement Collection schemas

cyberholics2cdf76aa570d
diff --git a/OCR/image_to_structured_data/.env.example b/OCR/image_to_structured_data/.env.example
index 41f8834..e6c1b53 100644
--- a/OCR/image_to_structured_data/.env.example
+++ b/OCR/image_to_structured_data/.env.example
@@ -1 +1 @@
-GROQ_API_KEY=gsk_your_key_here
\ No newline at end of file
+MISTRAL_API_KEY=your_mistral_key_here
\ No newline at end of file
diff --git a/OCR/image_to_structured_data/README.md b/OCR/image_to_structured_data/README.md
index 50b9f4e..8ba4702 100644
--- a/OCR/image_to_structured_data/README.md
+++ b/OCR/image_to_structured_data/README.md
@@ -1,32 +1,39 @@
-# Image-to-Structured-Data Extractor (Llama 4 Scout + Instructor)
+You are absolutely right. Since we switched the engine to Mistral, we need to update the documentation so it doesn't confuse users.
-> A high-speed, visual OCR framework that converts any image into validated JSON using Llama 4 Scout and Pydantic.
+Here is the fully updated **README.md** reflecting the Mistral Large 3 integration and the new "Collection" schema logic.
+
+***
+
+# Image-to-Structured-Data Extractor (Mistral Large 3 + Instructor)
+
+> A high-fidelity, visual extraction framework that converts images into validated, structured JSON using Mistral Large 3 and Pydantic.
## Overview
This project demonstrates how to move beyond raw text OCR to **Structured Visual Extraction**. Instead of getting a messy block of text, this tool uses the `instructor` library to force the VLM (Vision Language Model) to output data that fits a strict Pydantic schema.
-**Problem it solves:** Traditionally, extracting specific fields from varying layouts (like different receipt formats or product labels) required complex regex or custom OCR training. This project solves that by using a multimodal LLM to "understand" the visual context and map it directly to code-ready objects.
+**Problem it solves:** Traditionally, extracting specific fields from varying layouts (like different receipt formats or product labels) required complex regex or custom OCR training. This project solves that by using Mistral's flagship multimodal model to "understand" the visual context and map it directly to code-ready objects.
**Who is this for:** AI Engineers building automated data entry pipelines, invoice processing systems, or cataloging tools.
## Features
- **Schema-First Extraction:** Define your desired output in Python (Pydantic), and the model follows it.
-- **SOTA OSS Model:** Uses Llama 4 Scout via Groq for sub-second inference.
-- **Auto-Image Optimization:** Intelligent resizing and compression to handle high-res smartphone photos while staying within API pixel limits.
+- **Flagship Multimodal Model:** Uses Mistral Large 3 (`mistral-large-latest`) for state-of-the-art visual reasoning.
+- **Parallel Extraction:** Capable of extracting multiple items (e.g., all products on a receipt) in a single pass using Pydantic Collections.
+- **Auto-Image Optimization:** Intelligent resizing and compression to handle high-res photos while ensuring the model sees the sharpest possible text.
- **Multimodal Validation:** Leverages `instructor` to ensure the JSON output is valid and matches the schema before returning.
## Tech Stack
**Frameworks & Libraries:**
- **Instructor:** For structured data extraction and validation.
+- **Mistral Python SDK:** For multimodal inference.
- **Pydantic v2:** For defining data schemas.
-- **Groq SDK:** For high-speed inference.
- **Pillow (PIL):** For image processing and resizing.
**Additional Tools:**
-- **Model:** `meta-llama/llama-4-scout-17b-16e-instruct`
+- **Model:** `mistral-large-latest` (Mistral Large 3)
- **Web Framework:** Streamlit (UI)
- **Environment Management:** python-dotenv
@@ -36,7 +43,7 @@ Before you begin, ensure you have:
- Python 3.10 or higher (Recommended)
- API keys for:
- - [ ] [Groq Cloud Console](https://console.groq.com/)
+ - [ ] [Mistral AI Console](https://console.mistral.ai/)
- Basic understanding of Pydantic and Multimodal LLMs.
## Installation
@@ -69,7 +76,10 @@ Create a `.env` file in the project directory:
cp .env.example .env
```
-Edit `.env` and add your Groq API key.
+Edit `.env` and add your **Mistral API key**:
+```bash
+MISTRAL_API_KEY=your_key_here
+```
## Usage
@@ -79,23 +89,32 @@ Edit `.env` and add your Groq API key.
streamlit run app.py
```
+### Example Extraction
+
+**Input:**
![Sample Input Receipt](assets/payment-receipt.png)
**Output (Structured JSON):**
-
```json
{
- "name": "Organic Shampoo",
- "brand": null,
- "price": 22.0,
- "currency": "USD",
- "attributes": [
+ "products": [
{
- "key": "Quantity",
- "value": "3"
+ "name": "Organic Shampoo",
+ "brand": null,
+ "price": 22.0,
+ "currency": "USD",
+ "attributes": [{"key": "Quantity", "value": "3"}],
+ "summary": "Personal care product"
+ },
+ {
+ "name": "Whitening Toothpaste",
+ "brand": null,
+ "price": 28.0,
+ "currency": "USD",
+ "attributes": [{"key": "Quantity", "value": "1"}],
+ "summary": "Dental care product"
}
- ],
- "summary": "Personal care product"
+ ]
}
```
@@ -104,8 +123,8 @@ streamlit run app.py
```
image_to_structured_data/
├── app.py # Streamlit UI Layer
-├── processor.py # Logic for image resizing & Instructor calls
-├── schemas.py # Pydantic models (Add new schemas here!)
+├── processor.py # Logic for image resizing & Mistral + Instructor calls
+├── schemas.py # Pydantic models (Includes Collection wrappers)
├── requirements.txt # Dependencies
├── .env.example # Environment template
└── README.md # This file
@@ -113,10 +132,12 @@ image_to_structured_data/
## How It Works
-1. **Pre-processing:** The system accepts an image and uses Pillow to check its resolution. If it exceeds Groq's pixel limit (approx. 33M pixels), it resizes the image while maintaining aspect ratio and converts it to a high-quality JPEG to reduce payload size.
+1. **Pre-processing:** The system accepts an image and uses Pillow to check its resolution. It resizes the image to a maximum of 2048px (maintaining aspect ratio) to stay within optimal processing limits while keeping text sharp.
+
+2. **Instructor Integration:** It wraps the Mistral AI client with `instructor`, allowing the model to target a specific `response_model` via Mistral's native tool-calling capability.
-2. **Instructor Integration:** It "patches" the Groq client with instructor, enabling `response_model` support.
+3. **Structured Prompting:** The image is sent as a base64 encoded string to Mistral Large 3. By using "Collection" models (e.g., `ProductCollection`), the model is instructed to find *every* instance of a product and return them as a list.
-3. **Structured Prompting:** The image is sent as a base64 encoded string to Llama 4 Scout with a system instruction to map visual cues to the provided Pydantic schema.
+4. **Validation:** Pydantic validates the JSON returned. If Mistral hallucinates a field type (e.g., a string where a float should be), Instructor can catch this during the validation phase.
-4. **Validation:** Pydantic validates the JSON returned by the model. If a field like price is expected as a float but the model returns a string, Pydantic catches the error.
+[⬆ Back to Top](#image-to-structured-data-extractor-mistral-large-3--instructor)
\ No newline at end of file
diff --git a/OCR/image_to_structured_data/app.py b/OCR/image_to_structured_data/app.py
index 3f1ffab..43dd8b3 100644
--- a/OCR/image_to_structured_data/app.py
+++ b/OCR/image_to_structured_data/app.py
@@ -1,24 +1,25 @@
import streamlit as st
from processor import extract_structured_data
-from schemas import StructuredProduct, InvoiceData
+from schemas import ProductCollection, InvoiceCollection
import os
from dotenv import load_dotenv
load_dotenv()
-st.set_page_config(page_title="OSS Vision Extractor", layout="wide")
+st.set_page_config(page_title="Mistral Vision Extractor", layout="wide")
-st.title("📸Image-to-Structured-Data")
-st.write("Using **Llama 4 Scout** on Groq for sub-second OCR extraction.")
+st.title("📸 Image-to-Structured-Data")
+st.write("Using **Mistral Large 3** for high-fidelity visual OCR and structured extraction.")
with st.sidebar:
- # Use GROQ_API_KEY
- api_key = st.text_input("Groq API Key", value=os.getenv("GROQ_API_KEY", ""), type="password")
+ # Use the MISTRAL_API_KEY from .env if available
+ api_key = st.text_input("Mistral API Key", value=os.getenv("MISTRAL_API_KEY", ""), type="password")
schema_choice = st.selectbox("Select Extraction Schema", ["Product", "Invoice"])
+ # These names now match the updated import above
schema_map = {
- "Product": StructuredProduct,
- "Invoice": InvoiceData
+ "Product": ProductCollection,
+ "Invoice": InvoiceCollection
}
uploaded_file = st.file_uploader("Upload an image...", type=["jpg", "jpeg", "png"])
@@ -30,7 +31,7 @@ if uploaded_file and api_key:
with col2:
if st.button("Extract Data"):
- with st.spinner("Groq is thinking..."):
+ with st.spinner("Mistral is analyzing the image..."):
try:
uploaded_file.seek(0)
result = extract_structured_data(uploaded_file, schema_map[schema_choice], api_key)
diff --git a/OCR/image_to_structured_data/assets/payment-receipt.png b/OCR/image_to_structured_data/assets/payment-receipt.png
index a5d5189..81ff550 100644
Binary files a/OCR/image_to_structured_data/assets/payment-receipt.png and b/OCR/image_to_structured_data/assets/payment-receipt.png differ
diff --git a/OCR/image_to_structured_data/processor.py b/OCR/image_to_structured_data/processor.py
index 5b1a401..3786aaa 100644
--- a/OCR/image_to_structured_data/processor.py
+++ b/OCR/image_to_structured_data/processor.py
@@ -1,59 +1,36 @@
import base64
import instructor
-from groq import Groq
+from mistralai import Mistral
from PIL import Image
import io
def process_and_encode_image(image_file, max_size=(2048, 2048)):
- """
- Resizes the image if it's too large and converts to base64.
- 2048x2048 is a 'sweet spot' for OCR: high enough to read small text,
- but well within the pixel limits of Groq/Llama.
- """
- # Open the image using PIL
+ """Resizes image to fit API limits and converts to base64."""
img = Image.open(image_file)
-
- # Convert RGBA to RGB (in case of PNGs with transparency)
if img.mode in ("RGBA", "P"):
img = img.convert("RGB")
- # Resize image while maintaining aspect ratio if it exceeds max_size
img.thumbnail(max_size, Image.Resampling.LANCZOS)
- # Save to a bytes buffer
buffered = io.BytesIO()
- img.save(buffered, format="JPEG", quality=90) # JPEG at 90 quality is efficient
-
- # Encode to base64
+ img.save(buffered, format="JPEG", quality=90)
return base64.b64encode(buffered.getvalue()).decode('utf-8')
def extract_structured_data(image_file, schema_model, api_key: str):
- # Initialize the Groq client wrapped with Instructor
- client = instructor.from_groq(
- Groq(api_key=api_key),
- mode=instructor.Mode.JSON
- )
-
- # Process and encode the image
+ client = instructor.from_mistral(Mistral(api_key=api_key))
base64_image = process_and_encode_image(image_file)
return client.chat.completions.create(
- model="meta-llama/llama-4-scout-17b-16e-instruct",
+ model="mistral-large-latest",
response_model=schema_model,
+ max_retries=1, # Set retries to 1 to avoid hitting rate limits on errors
messages=[
{
"role": "user",
"content": [
- {
- "type": "text",
- "text": "Extract all relevant information from this image into the requested structured JSON format. Ensure high accuracy for numbers and dates."
- },
- {
- "type": "image_url",
- "image_url": {"url": f"data:image/jpeg;base64,{base64_image}"}
- },
+ {"type": "text", "text": "Extract all items found in this image into the requested structure."},
+ {"type": "image_url", "image_url": f"data:image/jpeg;base64,{base64_image}"}
],
}
],
- temperature=0.1,
)
\ No newline at end of file
diff --git a/OCR/image_to_structured_data/requirements.txt b/OCR/image_to_structured_data/requirements.txt
index ff620de..f0edf29 100644
--- a/OCR/image_to_structured_data/requirements.txt
+++ b/OCR/image_to_structured_data/requirements.txt
@@ -1,6 +1,7 @@
-groq
+mistralai>=1.5.0
instructor
-pydantic
+pydantic>=2.0.0
streamlit
python-dotenv
-pillow
\ No newline at end of file
+pillow
+eval_type_backport
\ No newline at end of file
diff --git a/OCR/image_to_structured_data/schemas.py b/OCR/image_to_structured_data/schemas.py
index d2842dd..84f1f26 100644
--- a/OCR/image_to_structured_data/schemas.py
+++ b/OCR/image_to_structured_data/schemas.py
@@ -2,21 +2,28 @@ from pydantic import BaseModel, Field
from typing import List, Optional
class ProductAttribute(BaseModel):
- key: str = Field(description="The attribute name, e.g., Color, Material, Size")
+ key: str
value: str
class StructuredProduct(BaseModel):
- """Extraction schema for product labels or advertisements."""
- name: str = Field(description="The primary name of the product")
+ name: str
brand: Optional[str]
price: Optional[float]
- currency: Optional[str] = Field(description="ISO currency code")
+ currency: Optional[str]
attributes: List[ProductAttribute]
- summary: str = Field(description="A brief 1-sentence description of the visual style")
+ summary: str
+
+# These are the ones the error is looking for:
+class ProductCollection(BaseModel):
+ """A collection of all products found in the image."""
+ products: List[StructuredProduct]
class InvoiceData(BaseModel):
- """Extraction schema for invoices or receipts."""
vendor_name: str
date: str
total_amount: float
- items: List[str]
\ No newline at end of file
+ items: List[str]
+
+class InvoiceCollection(BaseModel):
+ """A collection of invoices or line items found."""
+ invoices: List[InvoiceData]
\ No newline at end of file