A curated collection of practical AI projects implementing OCR systems, RAG, AI agents, and other AI use cases.
Add: Image-to-Structured-Data Extractor using Llama 4 Scout
cyberholicsea5c9de49555
diff --git a/.gitignore b/.gitignoreindex 134ea14..f65d529 100644--- a/.gitignore+++ b/.gitignore@@ -1,4 +1,6 @@-.venv/+venv/+__pycache__/+*.pyc.claude/.idea/node_modules/diff --git a/OCR/image_to_structured_data/.env.example b/OCR/image_to_structured_data/.env.examplenew file mode 100644index 0000000..41f8834--- /dev/null+++ b/OCR/image_to_structured_data/.env.example@@ -0,0 +1 @@+GROQ_API_KEY=gsk_your_key_here\ No newline at end of filediff --git a/OCR/image_to_structured_data/README.md b/OCR/image_to_structured_data/README.mdnew file mode 100644index 0000000..50b9f4e--- /dev/null+++ b/OCR/image_to_structured_data/README.md@@ -0,0 +1,122 @@+# Image-to-Structured-Data Extractor (Llama 4 Scout + Instructor)++> A high-speed, visual OCR framework that converts any image into validated JSON using Llama 4 Scout and Pydantic.++## Overview++This project demonstrates how to move beyond raw text OCR to **Structured Visual Extraction**. Instead of getting a messy block of text, this tool uses the `instructor` library to force the VLM (Vision Language Model) to output data that fits a strict Pydantic schema.++**Problem it solves:** Traditionally, extracting specific fields from varying layouts (like different receipt formats or product labels) required complex regex or custom OCR training. This project solves that by using a multimodal LLM to "understand" the visual context and map it directly to code-ready objects.++**Who is this for:** AI Engineers building automated data entry pipelines, invoice processing systems, or cataloging tools.++## Features++- **Schema-First Extraction:** Define your desired output in Python (Pydantic), and the model follows it.+- **SOTA OSS Model:** Uses Llama 4 Scout via Groq for sub-second inference.+- **Auto-Image Optimization:** Intelligent resizing and compression to handle high-res smartphone photos while staying within API pixel limits.+- **Multimodal Validation:** Leverages `instructor` to ensure the JSON output is valid and matches the schema before returning.++## Tech Stack++**Frameworks & Libraries:**+- **Instructor:** For structured data extraction and validation.+- **Pydantic v2:** For defining data schemas.+- **Groq SDK:** For high-speed inference.+- **Pillow (PIL):** For image processing and resizing.++**Additional Tools:**+- **Model:** `meta-llama/llama-4-scout-17b-16e-instruct`+- **Web Framework:** Streamlit (UI)+- **Environment Management:** python-dotenv++## Prerequisites++Before you begin, ensure you have:++- Python 3.10 or higher (Recommended)+- API keys for:+ - [ ] [Groq Cloud Console](https://console.groq.com/)+- Basic understanding of Pydantic and Multimodal LLMs.++## Installation++### 1. Clone the Repository++```bash+git clone https://github.com/Sumanth077/Hands-On-AI-Engineering.git+cd Hands-On-AI-Engineering/OCR/image_to_structured_data+```++### 2. Create Virtual Environment++```bash+python -m venv venv+source venv/bin/activate # On Windows: venv\Scripts\activate+```++### 3. Install Dependencies++```bash+pip install -r requirements.txt+```++### 4. Set Up Environment Variables++Create a `.env` file in the project directory:++```bash+cp .env.example .env+```++Edit `.env` and add your Groq API key.++## Usage++### Running the Application++```bash+streamlit run app.py+```++++**Output (Structured JSON):**++```json+{+ "name": "Organic Shampoo",+ "brand": null,+ "price": 22.0,+ "currency": "USD",+ "attributes": [+ {+ "key": "Quantity",+ "value": "3"+ }+ ],+ "summary": "Personal care product"+}+```++## Project Structure++```+image_to_structured_data/+├── app.py # Streamlit UI Layer+├── processor.py # Logic for image resizing & Instructor calls+├── schemas.py # Pydantic models (Add new schemas here!)+├── requirements.txt # Dependencies+├── .env.example # Environment template+└── README.md # This file+```++## How It Works++1. **Pre-processing:** The system accepts an image and uses Pillow to check its resolution. If it exceeds Groq's pixel limit (approx. 33M pixels), it resizes the image while maintaining aspect ratio and converts it to a high-quality JPEG to reduce payload size.++2. **Instructor Integration:** It "patches" the Groq client with instructor, enabling `response_model` support.++3. **Structured Prompting:** The image is sent as a base64 encoded string to Llama 4 Scout with a system instruction to map visual cues to the provided Pydantic schema.++4. **Validation:** Pydantic validates the JSON returned by the model. If a field like price is expected as a float but the model returns a string, Pydantic catches the error.diff --git a/OCR/image_to_structured_data/app.py b/OCR/image_to_structured_data/app.pynew file mode 100644index 0000000..3f1ffab--- /dev/null+++ b/OCR/image_to_structured_data/app.py@@ -0,0 +1,40 @@+import streamlit as st+from processor import extract_structured_data+from schemas import StructuredProduct, InvoiceData+import os+from dotenv import load_dotenv++load_dotenv()++st.set_page_config(page_title="OSS Vision Extractor", layout="wide")++st.title("📸Image-to-Structured-Data")+st.write("Using **Llama 4 Scout** on Groq for sub-second OCR extraction.")++with st.sidebar:+ # Use GROQ_API_KEY+ api_key = st.text_input("Groq API Key", value=os.getenv("GROQ_API_KEY", ""), type="password")+ schema_choice = st.selectbox("Select Extraction Schema", ["Product", "Invoice"])++ schema_map = {+ "Product": StructuredProduct,+ "Invoice": InvoiceData+ }++uploaded_file = st.file_uploader("Upload an image...", type=["jpg", "jpeg", "png"])++if uploaded_file and api_key:+ col1, col2 = st.columns(2)+ with col1:+ st.image(uploaded_file, caption="Source Image")++ with col2:+ if st.button("Extract Data"):+ with st.spinner("Groq is thinking..."):+ try:+ uploaded_file.seek(0)+ result = extract_structured_data(uploaded_file, schema_map[schema_choice], api_key)+ st.success("Extracted Successfully!")+ st.json(result.model_dump())+ except Exception as e:+ st.error(f"Error: {e}")\ No newline at end of filediff --git a/OCR/image_to_structured_data/assets/payment-receipt.png b/OCR/image_to_structured_data/assets/payment-receipt.pngnew file mode 100644index 0000000..a5d5189Binary files /dev/null and b/OCR/image_to_structured_data/assets/payment-receipt.png differdiff --git a/OCR/image_to_structured_data/processor.py b/OCR/image_to_structured_data/processor.pynew file mode 100644index 0000000..5b1a401--- /dev/null+++ b/OCR/image_to_structured_data/processor.py@@ -0,0 +1,59 @@+import base64+import instructor+from groq import Groq+from PIL import Image+import io++def process_and_encode_image(image_file, max_size=(2048, 2048)):+ """+ Resizes the image if it's too large and converts to base64.+ 2048x2048 is a 'sweet spot' for OCR: high enough to read small text,+ but well within the pixel limits of Groq/Llama.+ """+ # Open the image using PIL+ img = Image.open(image_file)++ # Convert RGBA to RGB (in case of PNGs with transparency)+ if img.mode in ("RGBA", "P"):+ img = img.convert("RGB")++ # Resize image while maintaining aspect ratio if it exceeds max_size+ img.thumbnail(max_size, Image.Resampling.LANCZOS)++ # Save to a bytes buffer+ buffered = io.BytesIO()+ img.save(buffered, format="JPEG", quality=90) # JPEG at 90 quality is efficient++ # Encode to base64+ return base64.b64encode(buffered.getvalue()).decode('utf-8')++def extract_structured_data(image_file, schema_model, api_key: str):+ # Initialize the Groq client wrapped with Instructor+ client = instructor.from_groq(+ Groq(api_key=api_key),+ mode=instructor.Mode.JSON+ )++ # Process and encode the image+ base64_image = process_and_encode_image(image_file)++ return client.chat.completions.create(+ model="meta-llama/llama-4-scout-17b-16e-instruct",+ response_model=schema_model,+ messages=[+ {+ "role": "user",+ "content": [+ {+ "type": "text",+ "text": "Extract all relevant information from this image into the requested structured JSON format. Ensure high accuracy for numbers and dates."+ },+ {+ "type": "image_url",+ "image_url": {"url": f"data:image/jpeg;base64,{base64_image}"}+ },+ ],+ }+ ],+ temperature=0.1,+ )\ No newline at end of filediff --git a/OCR/image_to_structured_data/requirements.txt b/OCR/image_to_structured_data/requirements.txtnew file mode 100644index 0000000..ff620de--- /dev/null+++ b/OCR/image_to_structured_data/requirements.txt@@ -0,0 +1,6 @@+groq+instructor+pydantic+streamlit+python-dotenv+pillow\ No newline at end of filediff --git a/OCR/image_to_structured_data/schemas.py b/OCR/image_to_structured_data/schemas.pynew file mode 100644index 0000000..d2842dd--- /dev/null+++ b/OCR/image_to_structured_data/schemas.py@@ -0,0 +1,22 @@+from pydantic import BaseModel, Field+from typing import List, Optional++class ProductAttribute(BaseModel):+ key: str = Field(description="The attribute name, e.g., Color, Material, Size")+ value: str++class StructuredProduct(BaseModel):+ """Extraction schema for product labels or advertisements."""+ name: str = Field(description="The primary name of the product")+ brand: Optional[str]+ price: Optional[float]+ currency: Optional[str] = Field(description="ISO currency code")+ attributes: List[ProductAttribute]+ summary: str = Field(description="A brief 1-sentence description of the visual style")++class InvoiceData(BaseModel):+ """Extraction schema for invoices or receipts."""+ vendor_name: str+ date: str+ total_amount: float+ items: List[str]\ No newline at end of file