Skip to content

100+ AI Agents, Agent Skills and RAG Apps - Free and Open Source.

feat: add toonify and scrapegraphai sdk

Marco Vinciguerra782ddfe31390
diff --git a/README.md b/README.md
index e6cd599d..eed5ddda 100644
--- a/README.md
+++ b/README.md
@@ -105,7 +105,7 @@ A curated collection of **Awesome LLM apps built with RAG, AI Agents, Multi-agen
* [πŸ”„ Mixture of Agents](starter_ai_agents/mixture_of_agents/)
* [πŸ“Š xAI Finance Agent](starter_ai_agents/xai_finance_agent/)
* [πŸ” OpenAI Research Agent](starter_ai_agents/opeani_research_agent/)
-* [πŸ•ΈοΈ Web Scrapping AI Agent (Local & Cloud)](starter_ai_agents/web_scrapping_ai_agent/)
+* [πŸ•ΈοΈ Web Scraping AI Agent (Local & Cloud SDK)](starter_ai_agents/web_scrapping_ai_agent/)
### πŸš€ Advanced AI Agents
* [🏚️ 🍌 AI Home Renovation Agent with Nano Banana](advanced_ai_agents/multi_agent_apps/ai_home_renovation_agent)
@@ -196,6 +196,10 @@ A curated collection of **Awesome LLM apps built with RAG, AI Agents, Multi-agen
* [πŸ“ Chat with Substack](advanced_llm_apps/chat_with_X_tutorials/chat_with_substack/)
* [πŸ“½οΈ Chat with YouTube Videos](advanced_llm_apps/chat_with_X_tutorials/chat_with_youtube_videos/)
+### 🎯 LLM Optimization Tools
+
+* [🎯 Toonify Token Optimization](advanced_llm_apps/llm_optimization_tools/toonify_token_optimization/) - Reduce LLM API costs by 30-60% using TOON format
+
### πŸ”§ LLM Fine-tuning Tutorials
* <img src="https://cdn.simpleicons.org/google" alt="google logo" width="20" height="15"> [Gemma 3 Fine-tuning](advanced_llm_apps/llm_finetuning_tutorials/gemma3_finetuning/)
diff --git a/advanced_llm_apps/llm_optimization_tools/README.md b/advanced_llm_apps/llm_optimization_tools/README.md
new file mode 100644
index 00000000..f1c87d2d
--- /dev/null
+++ b/advanced_llm_apps/llm_optimization_tools/README.md
@@ -0,0 +1,186 @@
+# 🎯 LLM Optimization Tools
+
+A collection of tools and techniques to optimize your LLM applications - reduce costs, improve performance, and maximize efficiency.
+
+---
+
+## πŸ“š Tools Available
+
+### 🎯 [Toonify Token Optimization](toonify_token_optimization/)
+
+**Reduce LLM API costs by 30-60%** using TOON (Token-Oriented Object Notation) format.
+
+#### What it does:
+- Converts JSON data to compact TOON format
+- Reduces token usage significantly
+- Maintains data structure and readability
+- Saves money on API calls
+
+#### Key Features:
+- βœ… **63.9% average token reduction** vs JSON
+- βœ… **Up to 73.4% savings** for tabular data
+- βœ… Human-readable format
+- βœ… Roundtrip conversion (JSON ↔ TOON)
+- βœ… Schema validation support
+- βœ… Interactive Streamlit app
+
+#### Quick Example:
+```python
+from toon import encode, decode
+
+# Your data (247 bytes as JSON)
+data = {
+ "products": [
+ {"id": 101, "name": "Laptop Pro", "price": 1299},
+ {"id": 102, "name": "Magic Mouse", "price": 79}
+ ]
+}
+
+# Convert to TOON (98 bytes - 60% reduction!)
+toon_str = encode(data)
+# products[2]{id,name,price}:
+# 101,Laptop Pro,1299
+# 102,Magic Mouse,79
+
+# Pass to LLM with reduced cost
+response = llm.complete(f"Analyze: {toon_str}")
+```
+
+#### Use Cases:
+- πŸ“Š Pass large datasets to LLMs
+- πŸ’° Reduce API costs significantly
+- πŸ”„ Optimize context window usage
+- πŸ“ˆ Improve response times
+
+#### Get Started:
+```bash
+cd toonify_token_optimization/
+pip install -r requirements.txt
+python quick_test.py
+```
+
+**πŸ“– [Full Documentation β†’](toonify_token_optimization/README.md)**
+
+---
+
+## πŸ’‘ Why Optimize?
+
+### Cost Savings
+LLM API costs are based on token count. Reducing tokens = saving money!
+
+**Example Savings (GPT-4)**:
+- 1,000 API calls: **$2.15 saved**
+- 100,000 API calls: **$214.70 saved**
+- 1M API calls: **$2,147.00 saved** πŸ’°
+
+### Performance
+Fewer tokens = faster processing and better efficiency.
+
+### Context Window
+Maximize what you can fit in your context window by using compact formats.
+
+---
+
+## 🎯 Best Practices
+
+### 1. Use Compact Formats for Structured Data
+When passing data to LLMs, use efficient serialization:
+- βœ… TOON for tabular/structured data
+- βœ… CSV for simple datasets
+- ❌ Avoid verbose JSON with excessive whitespace
+
+### 2. Optimize Prompts
+- Be concise and clear
+- Remove unnecessary examples
+- Use structured formats
+
+### 3. Batch Processing
+- Group similar requests
+- Reuse context when possible
+- Cache frequent responses
+
+### 4. Choose the Right Model
+- Use smaller models for simple tasks
+- Reserve GPT-4 for complex reasoning
+- Consider fine-tuned models
+
+---
+
+## πŸ“Š Comparison Table
+
+| Format | Size | Tokens | Cost (per 1M calls) | Best For |
+|--------|------|--------|---------------------|----------|
+| **JSON (verbose)** | 247 B | 85 | $2,550 | Compatibility |
+| **JSON (compact)** | 189 B | 67 | $2,010 | Standard use |
+| **TOON** | 98 B | 39 | $1,170 | Structured data |
+| **CSV** | 112 B | 42 | $1,260 | Simple tables |
+
+*Based on GPT-4 pricing ($0.03/1K input tokens)*
+
+---
+
+## πŸš€ Future Tools (Coming Soon)
+
+### Planned Additions:
+
+#### πŸ“¦ Prompt Compression
+Automatically compress long prompts while preserving meaning.
+
+#### πŸ—œοΈ Context Optimization
+Smart context window management for long conversations.
+
+#### πŸ“ˆ Token Analytics
+Track and analyze token usage across your applications.
+
+#### πŸ’Ύ Response Caching
+Intelligent caching to avoid redundant API calls.
+
+---
+
+## 🀝 Contributing
+
+Have an optimization technique to share? We'd love to include it!
+
+**How to contribute:**
+1. Fork the repository
+2. Create a new folder for your tool
+3. Include README, code, and examples
+4. Submit a pull request
+
+**Guidelines:**
+- Must significantly reduce costs or improve performance
+- Include benchmarks and comparisons
+- Provide clear documentation
+- Add usage examples
+
+---
+
+## πŸ“– Additional Resources
+
+### Learning Resources
+- [LLM Token Basics](https://platform.openai.com/tokenizer)
+- [Cost Optimization Guide](https://openai.com/pricing)
+- [Best Practices for Production](https://platform.openai.com/docs/guides/production-best-practices)
+
+### Related Projects
+- [TOON Format Spec](https://github.com/toon-format/toon)
+- [Toonify Library](https://github.com/ScrapeGraphAI/toonify)
+
+---
+
+## πŸ’¬ Support
+
+- πŸ“§ Questions? Open an issue on GitHub
+- πŸ’‘ Suggestions? We're always looking for new optimization techniques!
+- 🌟 Find this useful? Star the repository!
+
+---
+
+## πŸ“„ License
+
+Tools in this collection may have different licenses. Check each tool's folder for specific license information.
+
+---
+
+**Save money, go faster, build better! πŸš€πŸ’°**
+
diff --git a/advanced_llm_apps/llm_optimization_tools/toonify_token_optimization/README.md b/advanced_llm_apps/llm_optimization_tools/toonify_token_optimization/README.md
new file mode 100644
index 00000000..7d87135c
--- /dev/null
+++ b/advanced_llm_apps/llm_optimization_tools/toonify_token_optimization/README.md
@@ -0,0 +1,148 @@
+# 🎯 Toonify Token Optimization
+
+Reduce LLM API costs by 30-60% using TOON (Token-Oriented Object Notation) format for structured data serialization.
+
+## πŸ“‹ Overview
+
+This app demonstrates how to use [Toonify](https://github.com/ScrapeGraphAI/toonify) to dramatically reduce token usage when passing structured data to Large Language Models. TOON format achieves CSV-like compactness while maintaining explicit structure and human readability.
+
+### Key Benefits
+
+- **πŸ’° 63.9% average token reduction** compared to JSON
+- **🎯 Up to 73.4% savings** for optimal use cases (tabular data)
+- **πŸ’΅ Saves $2,147 per million API requests** at GPT-4 pricing
+- **πŸ“– Human-readable** format
+- **⚑ Minimal overhead** (<1ms for typical payloads)
+
+## πŸš€ Features
+
+- **JSON vs TOON Comparison**: See the size difference in action
+- **Token Cost Calculator**: Calculate savings for your use cases
+- **LLM Integration Example**: Pass optimized data to GPT/Claude
+- **Real-world Examples**: Product catalogs, surveys, analytics data
+- **Benchmarking**: Measure compression ratios for your data
+
+## πŸ“¦ Installation
+
+1. Install required dependencies:
+
+```bash
+pip install -r requirements.txt
+```
+
+2. Set up your API key (optional, for LLM integration demo):
+
+```bash
+export OPENAI_API_KEY='your-api-key-here'
+```
+
+## πŸ’» Usage
+
+### Basic Example
+
+Run the basic comparison demo:
+
+```bash
+python toonify_demo.py
+```
+
+### Interactive Demo
+
+Run the interactive Streamlit app:
+
+```bash
+streamlit run toonify_app.py
+```
+
+## πŸ“Š Format Comparison
+
+### JSON (247 bytes)
+```json
+{
+ "products": [
+ {"id": 101, "name": "Laptop Pro", "price": 1299},
+ {"id": 102, "name": "Magic Mouse", "price": 79},
+ {"id": 103, "name": "USB-C Cable", "price": 19}
+ ]
+}
+```
+
+### TOON (98 bytes, 60% reduction)
+```
+products[3]{id,name,price}:
+ 101,Laptop Pro,1299
+ 102,Magic Mouse,79
+ 103,USB-C Cable,19
+```
+
+## 🎯 Best Use Cases
+
+**Use TOON when:**
+- βœ… Passing data to LLM APIs (reduce token costs)
+- βœ… Working with uniform tabular data
+- βœ… Context window is limited
+- βœ… Human readability matters
+
+**Use JSON when:**
+- ❌ Maximum compatibility is required
+- ❌ Data is highly irregular/nested
+- ❌ Working with existing JSON-only tools
+
+## πŸ’‘ Example: E-commerce Product Analysis
+
+```python
+from toonify import encode
+import openai
+
+# Your product data (could be hundreds of products)
+products = [
+ {"id": 1, "name": "Laptop", "price": 1299, "stock": 45},
+ {"id": 2, "name": "Mouse", "price": 79, "stock": 120},
+ # ... many more products
+]
+
+# Convert to TOON format (saves 60% tokens)
+toon_data = encode(products)
+
+# Send to LLM with reduced token cost
+response = openai.chat.completions.create(
+ model="gpt-4",
+ messages=[{
+ "role": "user",
+ "content": f"Analyze this product data:\n{toon_data}"
+ }]
+)
+```
+
+## πŸ“ˆ Performance
+
+**Benchmarked across 50 real-world datasets:**
+- 63.9% average size reduction vs JSON
+- 54.1% average token reduction
+- 98% of datasets achieve 40%+ savings
+- Minimal overhead (<1ms encoding/decoding)
+
+## πŸ”— Resources
+
+- **Toonify GitHub**: https://github.com/ScrapeGraphAI/toonify
+- **PyPI**: https://pypi.org/project/toonify/
+- **Documentation**: https://docs.scrapegraphai.com/services/toonify
+- **Format Spec**: https://github.com/toon-format/toon
+
+## 🀝 Contributing
+
+Contributions are welcome! Feel free to:
+- Report bugs
+- Suggest new examples
+- Add benchmarks
+- Improve documentation
+
+## πŸ“„ License
+
+This example is provided as-is for educational purposes.
+Toonify library is licensed under MIT License.
+
+## πŸ™ Credits
+
+Built with [Toonify](https://github.com/ScrapeGraphAI/toonify) by the ScrapeGraphAI team.
+
diff --git a/advanced_llm_apps/llm_optimization_tools/toonify_token_optimization/quick_test.py b/advanced_llm_apps/llm_optimization_tools/toonify_token_optimization/quick_test.py
new file mode 100644
index 00000000..04200558
--- /dev/null
+++ b/advanced_llm_apps/llm_optimization_tools/toonify_token_optimization/quick_test.py
@@ -0,0 +1,74 @@
+"""
+Quick test to verify Toonify installation and basic functionality
+Run this to quickly see the token savings in action!
+"""
+
+import json
+from toon import encode, decode
+
+
+def quick_test():
+ """Quick demonstration of Toonify savings."""
+
+ print("🎯 TOONIFY QUICK TEST")
+ print("=" * 60)
+
+ # Sample data: Product catalog
+ products = {
+ "products": [
+ {"id": 1, "name": "Laptop", "price": 1299, "stock": 45},
+ {"id": 2, "name": "Mouse", "price": 79, "stock": 120},
+ {"id": 3, "name": "Keyboard", "price": 89, "stock": 85},
+ {"id": 4, "name": "Monitor", "price": 399, "stock": 32},
+ {"id": 5, "name": "Webcam", "price": 129, "stock": 67}
+ ]
+ }
+
+ # Convert to JSON
+ json_str = json.dumps(products, indent=2)
+ json_size = len(json_str)
+
+ # Convert to TOON
+ toon_str = encode(products)
+ toon_size = len(toon_str)
+
+ # Calculate savings
+ reduction = ((json_size - toon_size) / json_size) * 100
+
+ # Display results
+ print(f"\nπŸ“„ JSON ({json_size} bytes):")
+ print(json_str)
+
+ print(f"\n🎯 TOON ({toon_size} bytes):")
+ print(toon_str)
+
+ print(f"\nπŸ’° Size Reduction: {reduction:.1f}%")
+ print(f" Saved: {json_size - toon_size} bytes")
+
+ # Verify roundtrip
+ decoded = decode(toon_str)
+ if decoded == products:
+ print("\nβœ… Roundtrip verification: PASSED")
+ else:
+ print("\n❌ Roundtrip verification: FAILED")
+ return False
+
+ print("\n" + "=" * 60)
+ print("✨ Installation verified! Run toonify_demo.py for full demo.")
+ print("=" * 60)
+
+ return True
+
+
+if __name__ == "__main__":
+ try:
+ success = quick_test()
+ exit(0 if success else 1)
+ except ImportError as e:
+ print(f"\n❌ Error: {e}")
+ print("\nπŸ’‘ Install dependencies with: pip install -r requirements.txt")
+ exit(1)
+ except Exception as e:
+ print(f"\n❌ Unexpected error: {e}")
+ exit(1)
+
diff --git a/advanced_llm_apps/llm_optimization_tools/toonify_token_optimization/requirements.txt b/advanced_llm_apps/llm_optimization_tools/toonify_token_optimization/requirements.txt
new file mode 100644
index 00000000..8ca80e3f
--- /dev/null
+++ b/advanced_llm_apps/llm_optimization_tools/toonify_token_optimization/requirements.txt
@@ -0,0 +1,7 @@
+toonify>=1.4.0
+openai>=1.0.0
+anthropic>=0.40.0
+streamlit>=1.28.0
+pandas>=2.0.0
+tiktoken>=0.5.0
+
diff --git a/advanced_llm_apps/llm_optimization_tools/toonify_token_optimization/toonify_app.py b/advanced_llm_apps/llm_optimization_tools/toonify_token_optimization/toonify_app.py
new file mode 100644
index 00000000..a00292b5
--- /dev/null
+++ b/advanced_llm_apps/llm_optimization_tools/toonify_token_optimization/toonify_app.py
@@ -0,0 +1,307 @@
+"""
+Toonify Interactive Streamlit App
+Visualize token savings and test TOON format with your own data
+"""
+
+import streamlit as st
+import json
+from toon import encode, decode
+import tiktoken
+import pandas as pd
+
+
+def count_tokens(text: str, model: str = "gpt-4") -> int:
+ """Count tokens in text."""
+ encoding = tiktoken.encoding_for_model(model)
+ return len(encoding.encode(text))
+
+
+def calculate_cost(tokens: int, model: str = "gpt-4") -> float:
+ """Calculate API cost based on token count."""
+ pricing = {
+ "gpt-4": 0.03 / 1000, # $0.03 per 1K tokens
+ "gpt-4-turbo": 0.01 / 1000,
+ "gpt-3.5-turbo": 0.0015 / 1000,
+ "claude-3-opus": 0.015 / 1000,
+ "claude-3-sonnet": 0.003 / 1000,
+ }
+ return tokens * pricing.get(model, 0.03 / 1000)
+
+
+def main():
+ st.set_page_config(
+ page_title="Toonify Token Optimizer",
+ page_icon="🎯",
+ layout="wide"
+ )
+
+ st.title("🎯 Toonify Token Optimization")
+ st.markdown("""
+ Reduce your LLM API costs by **30-60%** using TOON format for structured data!
+
+ [GitHub](https://github.com/ScrapeGraphAI/toonify) |
+ [Documentation](https://docs.scrapegraphai.com/services/toonify)
+ """)
+
+ # Sidebar
+ with st.sidebar:
+ st.header("βš™οΈ Settings")
+
+ model = st.selectbox(
+ "LLM Model",
+ ["gpt-4", "gpt-4-turbo", "gpt-3.5-turbo", "claude-3-opus", "claude-3-sonnet"],
+ help="Select model for token counting and cost calculation"
+ )
+
+ delimiter = st.selectbox(
+ "TOON Delimiter",
+ ["comma", "tab", "pipe"],
+ help="Choose delimiter for array elements"
+ )
+
+ key_folding = st.selectbox(
+ "Key Folding",
+ ["off", "safe"],
+ help="Collapse nested single-key chains into dotted paths"
+ )
+
+ st.markdown("---")
+ st.markdown("### πŸ’‘ Quick Tips")
+ st.info("""
+ **Best for:**
+ - Tabular data
+ - Product catalogs
+ - Survey responses
+ - Analytics data
+
+ **Avoid for:**
+ - Highly nested data
+ - Irregular structures
+ """)
+
+ # Main content
+ tab1, tab2, tab3 = st.tabs(["πŸ“Š Comparison", "✍️ Custom Data", "πŸ“ˆ Benchmark"])
+
+ with tab1:
+ st.header("JSON vs TOON Comparison")
+
+ # Example data selector
+ example = st.selectbox(
+ "Choose example dataset",
+ [
+ "E-commerce Products",
+ "Customer Orders",
+ "Survey Responses",
+ "Analytics Data"
+ ]
+ )
+
+ # Load example data
+ examples = {
+ "E-commerce Products": {
+ "products": [
+ {"id": 1, "name": "Laptop Pro", "price": 1299, "stock": 45, "rating": 4.5},
+ {"id": 2, "name": "Magic Mouse", "price": 79, "stock": 120, "rating": 4.2},
+ {"id": 3, "name": "USB-C Cable", "price": 19, "stock": 350, "rating": 4.8},
+ {"id": 4, "name": "Keyboard", "price": 89, "stock": 85, "rating": 4.6},
+ {"id": 5, "name": "Monitor Stand", "price": 45, "stock": 60, "rating": 4.3}
+ ]
+ },
+ "Customer Orders": {
+ "orders": [
+ {"order_id": "ORD001", "customer": "Alice", "total": 299.99, "status": "shipped"},
+ {"order_id": "ORD002", "customer": "Bob", "total": 149.50, "status": "processing"},
+ {"order_id": "ORD003", "customer": "Charlie", "total": 449.99, "status": "delivered"}
+ ]
+ },
+ "Survey Responses": {
+ "responses": [
+ {"id": 1, "age": 25, "satisfaction": 4, "recommend": True, "comment": "Great service!"},
+ {"id": 2, "age": 34, "satisfaction": 5, "recommend": True, "comment": "Excellent!"},
+ {"id": 3, "age": 42, "satisfaction": 3, "recommend": False, "comment": "Could be better"}
+ ]
+ },
+ "Analytics Data": {
+ "pageviews": [
+ {"page": "/home", "views": 1523, "avg_time": 45, "bounce_rate": 0.32},
+ {"page": "/products", "views": 892, "avg_time": 120, "bounce_rate": 0.45},
+ {"page": "/about", "views": 234, "avg_time": 60, "bounce_rate": 0.28}
+ ]
+ }
+ }
+
+ data = examples[example]
+
+ # Convert formats
+ json_str = json.dumps(data, indent=2)
+ toon_options = {
+ 'delimiter': delimiter,
+ 'key_folding': key_folding
+ }
+ toon_str = encode(data, toon_options)
+
+ # Calculate metrics
+ json_size = len(json_str.encode('utf-8'))
+ toon_size = len(toon_str.encode('utf-8'))
+ json_tokens = count_tokens(json_str, model)
+ toon_tokens = count_tokens(toon_str, model)
+
+ size_reduction = ((json_size - toon_size) / json_size) * 100
+ token_reduction = ((json_tokens - toon_tokens) / json_tokens) * 100
+
+ json_cost = calculate_cost(json_tokens, model)
+ toon_cost = calculate_cost(toon_tokens, model)
+ cost_savings = json_cost - toon_cost
+
+ # Display metrics
+ col1, col2, col3, col4 = st.columns(4)
+
+ with col1:
+ st.metric("Size Reduction", f"{size_reduction:.1f}%")
+ with col2:
+ st.metric("Token Reduction", f"{token_reduction:.1f}%")
+ with col3:
+ st.metric("Cost per Call", f"${toon_cost:.6f}", f"-${cost_savings:.6f}")
+ with col4:
+ savings_1k = cost_savings * 1000
+ st.metric("Savings per 1K calls", f"${savings_1k:.2f}")
+
+ # Side-by-side comparison
+ col1, col2 = st.columns(2)
+
+ with col1:
+ st.subheader("πŸ“„ JSON Format")
+ st.code(json_str, language="json")
+ st.caption(f"Size: {json_size} bytes | Tokens: {json_tokens}")
+
+ with col2:
+ st.subheader("🎯 TOON Format")
+ st.code(toon_str, language="text")
+ st.caption(f"Size: {toon_size} bytes | Tokens: {toon_tokens}")
+
+ # Cost projection
+ st.subheader("πŸ’° Cost Savings Projection")
+
+ calls = [100, 1_000, 10_000, 100_000, 1_000_000]
+ json_costs = [json_cost * n for n in calls]
+ toon_costs = [toon_cost * n for n in calls]
+ savings = [json_costs[i] - toon_costs[i] for i in range(len(calls))]
+
+ df = pd.DataFrame({
+ "API Calls": [f"{n:,}" for n in calls],
+ "JSON Cost": [f"${c:.2f}" for c in json_costs],
+ "TOON Cost": [f"${c:.2f}" for c in toon_costs],
+ "Savings": [f"${s:.2f}" for s in savings],
+ "Savings %": [f"{token_reduction:.1f}%" for _ in calls]
+ })
+
+ st.dataframe(df, use_container_width=True)
+
+ with tab2:
+ st.header("Test Your Own Data")
+
+ st.markdown("Paste your JSON data below to see how much you can save:")
+
+ user_json = st.text_area(
+ "JSON Data",
+ value='{\n "items": [\n {"id": 1, "name": "Example", "value": 100}\n ]\n}',
+ height=300
+ )
+
+ if st.button("🎯 Convert to TOON"):
+ try:
+ # Parse JSON
+ data = json.loads(user_json)
+
+ # Convert to TOON
+ toon_options = {
+ 'delimiter': delimiter,
+ 'key_folding': key_folding
+ }
+ toon_str = encode(data, toon_options)
+
+ # Calculate savings
+ json_size = len(user_json.encode('utf-8'))
+ toon_size = len(toon_str.encode('utf-8'))
+ json_tokens = count_tokens(user_json, model)
+ toon_tokens = count_tokens(toon_str, model)
+
+ size_reduction = ((json_size - toon_size) / json_size) * 100
+ token_reduction = ((json_tokens - toon_tokens) / json_tokens) * 100
+
+ # Display results
+ st.success("βœ… Conversion successful!")
+
+ col1, col2, col3 = st.columns(3)
+ with col1:
+ st.metric("Size Reduction", f"{size_reduction:.1f}%")
+ with col2:
+ st.metric("Token Reduction", f"{token_reduction:.1f}%")
+ with col3:
+ cost_savings = calculate_cost(json_tokens - toon_tokens, model)
+ st.metric("Savings per call", f"${cost_savings:.6f}")
+
+ st.subheader("🎯 TOON Output")
+ st.code(toon_str, language="text")
+
+ # Verify roundtrip
+ decoded = decode(toon_str)
+ if decoded == data:
+ st.success("βœ… Roundtrip verification passed!")
+ else:
+ st.warning("⚠️ Roundtrip verification failed")
+
+ except json.JSONDecodeError as e:
+ st.error(f"❌ Invalid JSON: {e}")
+ except Exception as e:
+ st.error(f"❌ Error: {e}")
+
+ with tab3:
+ st.header("πŸ“ˆ Format Benchmark")
+
+ st.markdown("""
+ Based on benchmarks across 50 real-world datasets:
+ """)
+
+ # Benchmark stats
+ col1, col2, col3 = st.columns(3)
+
+ with col1:
+ st.metric("Avg Size Reduction", "63.9%")
+ with col2:
+ st.metric("Avg Token Reduction", "54.1%")
+ with col3:
+ st.metric("Best Case", "73.4%")
+
+ # Data type performance
+ st.subheader("Performance by Data Type")
+
+ performance_data = {
+ "Data Type": ["Tabular", "E-commerce", "Analytics", "Surveys", "Mixed"],
+ "Token Reduction": [73.4, 68.2, 65.1, 61.5, 48.3],
+ "Use Case": ["βœ… Excellent", "βœ… Excellent", "βœ… Great", "βœ… Great", "βœ… Good"]
+ }
+
+ df = pd.DataFrame(performance_data)
+ st.dataframe(df, use_container_width=True)
+
+ st.info("""
+ **πŸ’‘ Optimization Tips:**
+ - Use TOON for uniform, structured data
+ - Enable key folding for deeply nested objects
+ - Choose appropriate delimiter based on your data
+ - Test with your actual data for best results
+ """)
+
+ st.markdown("---")
+ st.markdown("""
+ ### πŸ”— Learn More
+ - [GitHub Repository](https://github.com/ScrapeGraphAI/toonify)
+ - [Documentation](https://docs.scrapegraphai.com/services/toonify)
+ - [Format Specification](https://github.com/toon-format/toon)
+ """)
+
+
+if __name__ == "__main__":
+ main()
+
diff --git a/advanced_llm_apps/llm_optimization_tools/toonify_token_optimization/toonify_demo.py b/advanced_llm_apps/llm_optimization_tools/toonify_token_optimization/toonify_demo.py
new file mode 100644
index 00000000..f3edbfe2
--- /dev/null
+++ b/advanced_llm_apps/llm_optimization_tools/toonify_token_optimization/toonify_demo.py
@@ -0,0 +1,307 @@
+"""
+Toonify Token Optimization Demo
+Demonstrates how to reduce LLM API costs by 30-60% using TOON format
+"""
+
+import json
+from toon import encode, decode
+import tiktoken
+from openai import OpenAI
+from anthropic import Anthropic
+import os
+
+
+def count_tokens(text: str, model: str = "gpt-4") -> int:
+ """Count the number of tokens in a text string."""
+ encoding = tiktoken.encoding_for_model(model)
+ return len(encoding.encode(text))
+
+
+def format_comparison_demo():
+ """Compare JSON vs TOON format sizes and token counts."""
+
+ print("=" * 80)
+ print("🎯 TOONIFY TOKEN OPTIMIZATION DEMO")
+ print("=" * 80)
+
+ # Example: E-commerce product catalog
+ products_data = {
+ "products": [
+ {
+ "id": 101,
+ "name": "Laptop Pro 15",
+ "category": "Electronics",
+ "price": 1299.99,
+ "stock": 45,
+ "rating": 4.5
+ },
+ {
+ "id": 102,
+ "name": "Magic Mouse",
+ "category": "Electronics",
+ "price": 79.99,
+ "stock": 120,
+ "rating": 4.2
+ },
+ {
+ "id": 103,
+ "name": "USB-C Cable",
+ "category": "Accessories",
+ "price": 19.99,
+ "stock": 350,
+ "rating": 4.8
+ },
+ {
+ "id": 104,
+ "name": "Wireless Keyboard",
+ "category": "Electronics",
+ "price": 89.99,
+ "stock": 85,
+ "rating": 4.6
+ },
+ {
+ "id": 105,
+ "name": "Monitor Stand",
+ "category": "Accessories",
+ "price": 45.99,
+ "stock": 60,
+ "rating": 4.3
+ }
+ ]
+ }
+
+ # Convert to JSON
+ json_str = json.dumps(products_data, indent=2)
+ json_size = len(json_str.encode('utf-8'))
+ json_tokens = count_tokens(json_str)
+
+ # Convert to TOON
+ toon_str = encode(products_data)
+ toon_size = len(toon_str.encode('utf-8'))
+ toon_tokens = count_tokens(toon_str)
+
+ # Calculate savings
+ size_reduction = ((json_size - toon_size) / json_size) * 100
+ token_reduction = ((json_tokens - toon_tokens) / json_tokens) * 100
+
+ print("\nπŸ“Š FORMAT COMPARISON")
+ print("-" * 80)
+
+ print("\nπŸ“„ JSON Format:")
+ print(json_str)
+ print(f"\nSize: {json_size} bytes")
+ print(f"Tokens: {json_tokens}")
+
+ print("\n" + "=" * 80)
+
+ print("\n🎯 TOON Format:")
+ print(toon_str)
+ print(f"\nSize: {toon_size} bytes")
+ print(f"Tokens: {toon_tokens}")
+
+ print("\n" + "=" * 80)
+ print("\nπŸ’° SAVINGS")
+ print("-" * 80)
+ print(f"Size Reduction: {size_reduction:.1f}%")
+ print(f"Token Reduction: {token_reduction:.1f}%")
+
+ # Calculate cost savings
+ # GPT-4 pricing: $0.03 per 1K tokens (input)
+ cost_per_token = 0.03 / 1000
+ json_cost = json_tokens * cost_per_token
+ toon_cost = toon_tokens * cost_per_token
+ savings_per_call = json_cost - toon_cost
+
+ print(f"\nπŸ’΅ Cost per API call:")
+ print(f" JSON: ${json_cost:.6f}")
+ print(f" TOON: ${toon_cost:.6f}")
+ print(f" Savings: ${savings_per_call:.6f} ({token_reduction:.1f}%)")
+
+ print(f"\nπŸ“ˆ Projected savings:")
+ print(f" Per 1,000 calls: ${savings_per_call * 1000:.2f}")
+ print(f" Per 1M calls: ${savings_per_call * 1_000_000:.2f}")
+
+ print("\n" + "=" * 80)
+
+ return toon_str, products_data
+
+
+def llm_integration_demo():
+ """Demonstrate using TOON format with LLM APIs."""
+
+ print("\nπŸ€– LLM INTEGRATION DEMO")
+ print("=" * 80)
+
+ # Create sample data
+ customer_orders = {
+ "orders": [
+ {"order_id": "ORD001", "customer": "Alice", "total": 299.99, "status": "shipped"},
+ {"order_id": "ORD002", "customer": "Bob", "total": 149.50, "status": "processing"},
+ {"order_id": "ORD003", "customer": "Charlie", "total": 449.99, "status": "delivered"},
+ {"order_id": "ORD004", "customer": "Diana", "total": 89.99, "status": "pending"},
+ ]
+ }
+
+ # Convert to TOON
+ toon_data = encode(customer_orders)
+ json_data = json.dumps(customer_orders, indent=2)
+
+ print("\nπŸ“¦ Data to analyze:")
+ print(toon_data)
+
+ # Check if API keys are available
+ openai_key = os.getenv("OPENAI_API_KEY")
+ anthropic_key = os.getenv("ANTHROPIC_API_KEY")
+
+ if openai_key:
+ try:
+ print("\nπŸ”΅ Testing with OpenAI GPT-4...")
+ client = OpenAI(api_key=openai_key)
+
+ prompt = f"""Analyze these customer orders and provide a brief summary:
+
+{toon_data}
+
+Provide: 1) Total revenue, 2) Orders by status, 3) Average order value"""
+
+ response = client.chat.completions.create(
+ model="gpt-4",
+ messages=[{"role": "user", "content": prompt}],
+ max_tokens=200
+ )
+
+ print("\nβœ… GPT-4 Response:")
+ print(response.choices[0].message.content)
+
+ # Show token usage
+ print(f"\nπŸ“Š Token Usage:")
+ print(f" Input tokens: {response.usage.prompt_tokens}")
+ print(f" Output tokens: {response.usage.completion_tokens}")
+ print(f" Total tokens: {response.usage.total_tokens}")
+
+ # Compare with JSON
+ json_tokens = count_tokens(prompt.replace(toon_data, json_data))
+ toon_tokens = response.usage.prompt_tokens
+ savings = ((json_tokens - toon_tokens) / json_tokens) * 100
+ print(f"\nπŸ’° Token Savings: {savings:.1f}% (vs JSON)")
+
+ except Exception as e:
+ print(f"❌ OpenAI Error: {e}")
+ else:
+ print("\n⚠️ Set OPENAI_API_KEY to test with GPT-4")
+
+ if anthropic_key:
+ try:
+ print("\n🟣 Testing with Anthropic Claude...")
+ client = Anthropic(api_key=anthropic_key)
+
+ response = client.messages.create(
+ model="claude-3-5-sonnet-20241022",
+ max_tokens=200,
+ messages=[{
+ "role": "user",
+ "content": f"""Analyze these customer orders and provide a brief summary:
+
+{toon_data}
+
+Provide: 1) Total revenue, 2) Orders by status, 3) Average order value"""
+ }]
+ )
+
+ print("\nβœ… Claude Response:")
+ print(response.content[0].text)
+
+ # Show token usage
+ print(f"\nπŸ“Š Token Usage:")
+ print(f" Input tokens: {response.usage.input_tokens}")
+ print(f" Output tokens: {response.usage.output_tokens}")
+
+ except Exception as e:
+ print(f"❌ Anthropic Error: {e}")
+ else:
+ print("\n⚠️ Set ANTHROPIC_API_KEY to test with Claude")
+
+
+def advanced_features_demo():
+ """Demonstrate advanced TOON features."""
+
+ print("\nβš™οΈ ADVANCED FEATURES")
+ print("=" * 80)
+
+ # Key folding example
+ nested_data = {
+ 'api': {
+ 'response': {
+ 'product': {
+ 'title': 'Wireless Keyboard',
+ 'specs': {
+ 'battery': '6 months',
+ 'connectivity': 'Bluetooth 5.0'
+ }
+ }
+ }
+ }
+ }
+
+ print("\n1️⃣ Key Folding (collapse nested paths)")
+ print("-" * 80)
+
+ # Without key folding
+ normal_toon = encode(nested_data)
+ print("Without key folding:")
+ print(normal_toon)
+
+ # With key folding
+ folded_toon = encode(nested_data, {'key_folding': 'safe'})
+ print("\nWith key folding:")
+ print(folded_toon)
+
+ print(f"\nSavings: {len(normal_toon)} β†’ {len(folded_toon)} bytes")
+
+ # Custom delimiters
+ print("\n2️⃣ Custom Delimiters")
+ print("-" * 80)
+
+ data = {
+ "items": [
+ ["Product A", "Description with, commas", 29.99],
+ ["Product B", "Another, description", 39.99]
+ ]
+ }
+
+ print("Tab delimiter (for data with commas):")
+ tab_toon = encode(data, {'delimiter': 'tab'})
+ print(tab_toon)
+
+ print("\nPipe delimiter:")
+ pipe_toon = encode(data, {'delimiter': 'pipe'})
+ print(pipe_toon)
+
+
+def main():
+ """Run all demos."""
+
+ # Basic comparison
+ toon_str, original_data = format_comparison_demo()
+
+ # Verify roundtrip
+ decoded_data = decode(toon_str)
+ assert decoded_data == original_data, "Roundtrip failed!"
+ print("\nβœ… Roundtrip verification: PASSED")
+
+ # LLM integration (optional, requires API keys)
+ llm_integration_demo()
+
+ # Advanced features
+ advanced_features_demo()
+
+ print("\n" + "=" * 80)
+ print("πŸŽ‰ Demo completed!")
+ print("πŸ’‘ Set OPENAI_API_KEY or ANTHROPIC_API_KEY to test LLM integration")
+ print("πŸ”— Learn more: https://github.com/ScrapeGraphAI/toonify")
+ print("=" * 80)
+
+
+if __name__ == "__main__":
+ main()
+
diff --git a/starter_ai_agents/web_scrapping_ai_agent/QUICKSTART.md b/starter_ai_agents/web_scrapping_ai_agent/QUICKSTART.md
new file mode 100644
index 00000000..4ce07a47
--- /dev/null
+++ b/starter_ai_agents/web_scrapping_ai_agent/QUICKSTART.md
@@ -0,0 +1,224 @@
+# πŸš€ Quick Start Guide - Web Scraping AI Agent
+
+Choose your version and get started in minutes!
+
+---
+
+## πŸ“‹ Two Versions Available
+
+### 1️⃣ **Local Library** (Free, Open Source)
+βœ… Best for: Learning, privacy, full control
+πŸ“ Files: `ai_scrapper.py`, `local_ai_scrapper.py`
+
+### 2️⃣ **Cloud SDK** (Managed Service)
+βœ… Best for: Production, scalability, advanced features
+πŸ“ Folder: `scrapegraph_ai_sdk/`
+
+---
+
+## πŸƒβ€β™‚οΈ Quick Start - Local Version
+
+### Step 1: Install Dependencies
+```bash
+pip install -r requirements.txt
+```
+
+### Step 2: Get OpenAI API Key
+πŸ‘‰ Sign up at [platform.openai.com](https://platform.openai.com/)
+
+### Step 3: Run the App
+```bash
+streamlit run ai_scrapper.py
+```
+
+### Step 4: Use the App
+1. Enter your OpenAI API key
+2. Select model (GPT-4o recommended)
+3. Enter website URL
+4. Describe what to extract
+5. Click "Scrape"!
+
+**⏱️ Setup Time: ~2 minutes**
+
+---
+
+## ☁️ Quick Start - Cloud SDK
+
+### Step 1: Navigate to SDK Folder
+```bash
+cd scrapegraph_ai_sdk/
+```
+
+### Step 2: Install Dependencies
+```bash
+pip install -r requirements.txt
+```
+
+### Step 3: Get ScrapeGraph AI API Key
+πŸ‘‰ Sign up at [scrapegraphai.com](https://scrapegraphai.com)
+
+### Step 4: Set API Key
+```bash
+export SGAI_API_KEY='your-api-key-here'
+```
+
+### Step 5: Quick Test
+```bash
+python quickstart.py
+```
+
+### Step 6: Try Different Features
+
+**SmartScraper** (extract structured data):
+```bash
+python smart_scraper_demo.py
+```
+
+**SearchScraper** (AI web search):
+```bash
+python search_scraper_demo.py
+```
+
+**SmartCrawler** (multi-page scraping):
+```bash
+python smart_crawler_demo.py
+```
+
+**Interactive App**:
+```bash
+streamlit run scrapegraph_app.py
+```
+
+**⏱️ Setup Time: ~1 minute**
+
+---
+
+## 🎯 Which One Should I Choose?
+
+### Choose **Local Library** if you want:
+- πŸ†“ Free solution (pay only for OpenAI API)
+- πŸ”’ Privacy (all processing on your machine)
+- πŸŽ“ Learning and experimentation
+- πŸ› οΈ Full control over the process
+
+### Choose **Cloud SDK** if you want:
+- ⚑ Quick setup (no dependencies)
+- πŸš€ Production-ready scalability
+- ✨ Advanced features (SmartCrawler, SearchScraper, Markdownify)
+- πŸ”„ Automatic updates
+
+---
+
+## πŸ’‘ Example Use Cases
+
+### E-commerce Price Monitoring
+```python
+# Local Version
+prompt = "Extract product name, price, and availability"
+
+# Cloud SDK
+response = client.smartscraper(
+ website_url="https://shop.example.com/product",
+ user_prompt="Extract product name, price, and availability"
+)
+```
+
+### News Aggregation
+```python
+# Cloud SDK only
+response = client.smartscraper(
+ user_prompt="Find top 5 tech news articles with titles and summaries",
+ num_results=5
+)
+```
+
+### Documentation Scraping
+```python
+# Cloud SDK SmartCrawler
+request_id = client.smartcrawler(
+ url="https://docs.example.com",
+ user_prompt="Extract all API endpoints and descriptions",
+ max_pages=50
+)
+```
+
+---
+
+## πŸ†˜ Troubleshooting
+
+### Local Version Issues
+
+**Problem**: `ModuleNotFoundError: No module named 'scrapegraphai'`
+```bash
+# Solution
+pip install scrapegraphai
+```
+
+**Problem**: Scraping is slow
+```
+# Solution: Try a lighter model
+Use GPT-3.5-turbo instead of GPT-4
+```
+
+### Cloud SDK Issues
+
+**Problem**: `SGAI_API_KEY not found`
+```bash
+# Solution
+export SGAI_API_KEY='your-actual-key'
+# Or add to your .bashrc/.zshrc
+```
+
+**Problem**: `Insufficient credits`
+```bash
+# Check balance
+python -c "from scrapegraph_py import Client; print(Client('your-key').get_credits())"
+# Then recharge at scrapegraphai.com
+```
+
+---
+
+## πŸ“š Next Steps
+
+### After Getting Started:
+
+1. **Read Full Documentation**
+ - Local: Main README.md
+ - Cloud SDK: `scrapegraph_ai_sdk/README.md`
+
+2. **Try Advanced Features**
+ - Custom output schemas
+ - JavaScript rendering
+ - Scheduled jobs (SDK only)
+
+3. **Integrate into Your Project**
+ - Use as a library in your Python code
+ - Build custom workflows
+ - Automate data collection
+
+---
+
+## πŸ”— Useful Links
+
+### Local Library
+- [ScrapeGraph AI GitHub](https://github.com/VinciGit00/Scrapegraph-ai)
+- [Documentation](https://scrapegraph-ai.readthedocs.io/)
+
+### Cloud SDK
+- [SDK GitHub](https://github.com/ScrapeGraphAI/scrapegraph-sdk)
+- [API Documentation](https://docs.scrapegraphai.com)
+- [Get API Key](https://scrapegraphai.com)
+
+---
+
+## πŸ’¬ Need Help?
+
+- πŸ“– Check the README files
+- πŸ’‘ Look at example code in demos
+- πŸ› Report issues on GitHub
+- πŸ“§ Contact: [email protected]
+
+---
+
+**Happy Scraping! πŸ•·οΈβœ¨**
+
diff --git a/starter_ai_agents/web_scrapping_ai_agent/README.md b/starter_ai_agents/web_scrapping_ai_agent/README.md
index faff623d..e60ca1d2 100644
--- a/starter_ai_agents/web_scrapping_ai_agent/README.md
+++ b/starter_ai_agents/web_scrapping_ai_agent/README.md
@@ -1,39 +1,270 @@
-## πŸ’» Web Scrapping AI Agent
-This Streamlit app allows you to scrape a website using OpenAI API and the scrapegraphai library. Simply provide your OpenAI API key, enter the URL of the website you want to scrape, and specify what you want the AI agent to extract from the website.
+# πŸ•·οΈ Web Scraping AI Agent
-### Features
-- Scrape any website by providing the URL
-- Utilize OpenAI's LLMs (GPT-4o or GPT-5) for intelligent scraping
-- Customize the scraping task by specifying what you want the AI agent to extract
+AI-powered web scraping using **ScrapeGraph AI** - extract structured data from websites using natural language prompts. This folder contains two implementations:
-### How to get Started?
+1. **🏠 Local Library** - Using `scrapegraphai` library (runs locally)
+2. **☁️ Cloud SDK** - Using ScrapeGraph AI API (managed service)
-1. Clone the GitHub repository
+---
+## πŸ“ What's Inside
+
+### 🏠 Local Library Version
+**Files**: `ai_scrapper.py`, `local_ai_scrapper.py`
+
+Use the open-source ScrapeGraph AI library that runs on your local machine.
+
+**βœ… Pros:**
+- Free to use (no API costs)
+- Full control over execution
+- Privacy-friendly (all data stays local)
+
+**❌ Cons:**
+- Requires local installation and dependencies
+- Limited by your hardware
+- Need to manage updates
+
+**Quick Start:**
+```bash
+pip install -r requirements.txt
+streamlit run ai_scrapper.py
+```
+
+---
+
+### ☁️ Cloud SDK Version
+**Folder**: `scrapegraph_ai_sdk/`
+
+Use the managed ScrapeGraph AI API with advanced features and no setup required.
+
+**βœ… Pros:**
+- No setup required (just API key)
+- Scalable and fast
+- Advanced features (SmartCrawler, SearchScraper, Markdownify)
+- Always up-to-date
+
+**❌ Cons:**
+- Pay-per-use (credit-based)
+- Requires internet connection
+
+**Quick Start:**
+```bash
+cd scrapegraph_ai_sdk/
+pip install -r requirements.txt
+export SGAI_API_KEY='your-api-key'
+python quickstart.py
+```
+
+**πŸ“– Full Documentation**: [See scrapegraph_ai_sdk/README.md](scrapegraph_ai_sdk/README.md)
+
+---
+
+## πŸš€ Getting Started
+
+### Local Library Version
+
+1. **Clone the repository**
```bash
git clone https://github.com/Shubhamsaboo/awesome-llm-apps.git
-cd awesome-llm-apps/advanced_tools_frameworks/web_scrapping_ai_agent
+cd awesome-llm-apps/starter_ai_agents/web_scrapping_ai_agent
```
-2. Install the required dependencies:
+2. **Install dependencies**
```bash
pip install -r requirements.txt
```
-3. Get your OpenAI API Key
-- Sign up for an [OpenAI account](https://platform.openai.com/) (or the LLM provider of your choice) and obtain your API key.
+3. **Get your OpenAI API Key**
+- Sign up for an [OpenAI account](https://platform.openai.com/)
+- Obtain your API key
-4. Run the Streamlit App
+4. **Run the Streamlit App**
```bash
streamlit run ai_scrapper.py
+# Or for local models:
+streamlit run local_ai_scrapper.py
+```
+
+### Cloud SDK Version
+
+1. **Navigate to SDK folder**
+```bash
+cd scrapegraph_ai_sdk/
+```
+
+2. **Install dependencies**
+```bash
+pip install -r requirements.txt
```
-### How it Works?
+3. **Get your ScrapeGraph AI API Key**
+- Sign up at [scrapegraphai.com](https://scrapegraphai.com)
+- Get your API key
+
+4. **Set API key**
+```bash
+export SGAI_API_KEY='your-api-key-here'
+```
+
+5. **Run demos**
+```bash
+# Quick test
+python quickstart.py
+
+# SmartScraper demo
+python smart_scraper_demo.py
+
+# Interactive app
+streamlit run scrapegraph_app.py
+```
+
+---
+
+## πŸ“Š Feature Comparison
+
+| Feature | Local Library | Cloud SDK |
+|---------|--------------|-----------|
+| **Setup** | Install dependencies | API key only |
+| **Cost** | Free (+ LLM costs) | Pay-per-use |
+| **Processing** | Your hardware | Cloud-based |
+| **Speed** | Depends on hardware | Fast & optimized |
+| **SmartScraper** | βœ… | βœ… |
+| **SearchScraper** | ❌ | βœ… |
+| **SmartCrawler** | ❌ | βœ… |
+| **Markdownify** | ❌ | βœ… |
+| **Scheduled Jobs** | ❌ | βœ… |
+| **Scalability** | Limited | Unlimited |
+| **Maintenance** | Self-managed | Fully managed |
+
+---
+
+## πŸ’‘ Use Cases
+
+### E-commerce Scraping
+```python
+# Extract product information
+prompt = "Extract product names, prices, and availability"
+```
+
+### Content Aggregation
+```python
+# Convert articles to structured data
+prompt = "Extract article title, author, date, and main content"
+```
+
+### Competitive Intelligence
+```python
+# Monitor competitor websites
+prompt = "Extract pricing, features, and updates"
+```
+
+### Lead Generation
+```python
+# Extract contact information
+prompt = "Find company names, emails, and phone numbers"
+```
+
+---
+
+## πŸ”§ How It Works
+
+### Local Library
+1. You provide your OpenAI API key
+2. Select the model (GPT-4o, GPT-5, or local models)
+3. Enter the URL and extraction prompt
+4. The app uses ScrapeGraphAI to scrape and extract data locally
+5. Results are displayed in the app
+
+### Cloud SDK
+1. You provide your ScrapeGraph AI API key
+2. Choose the scraping method (SmartScraper, SearchScraper, etc.)
+3. Define extraction prompt and optional output schema
+4. API processes the request in the cloud
+5. Structured results are returned
+
+---
+
+## 🌟 Cloud SDK Features
+
+### πŸ€– SmartScraper
+Extract structured data using natural language:
+```python
+response = client.smartscraper(
+ website_url="https://example.com",
+ user_prompt="Extract all products with prices"
+)
+```
+
+### πŸ” SearchScraper
+AI-powered web search with structured results:
+```python
+response = client.smartscraper(
+ user_prompt="Find top 5 AI news websites",
+ num_results=5
+)
+```
+
+### πŸ“ Markdownify
+Convert webpages to clean markdown:
+```python
+response = client.markdownify(
+ website_url="https://example.com/article"
+)
+```
+
+### πŸ•·οΈ SmartCrawler
+Crawl multiple pages intelligently:
+```python
+request_id = client.smartcrawler(
+ url="https://docs.example.com",
+ user_prompt="Extract all API endpoints",
+ max_pages=50
+)
+```
+
+---
+
+## πŸ“– Documentation
+
+- **Local Library**: [ScrapeGraphAI GitHub](https://github.com/VinciGit00/Scrapegraph-ai)
+- **Cloud SDK**: [See scrapegraph_ai_sdk/README.md](scrapegraph_ai_sdk/README.md)
+- **API Docs**: https://docs.scrapegraphai.com
+
+---
+
+## 🀝 Which Version Should I Use?
+
+**Use Local Library if:**
+- βœ… You want free, open-source solution
+- βœ… You have good hardware
+- βœ… You need full control
+- βœ… Privacy is critical
+
+**Use Cloud SDK if:**
+- βœ… You want quick setup
+- βœ… You need advanced features
+- βœ… You want scalability
+- βœ… You prefer managed service
+
+**πŸ’‘ Pro Tip**: Start with the local version to learn, then switch to SDK for production!
+
+---
+
+## πŸ”— Resources
+
+- **ScrapeGraph AI Website**: https://scrapegraphai.com
+- **GitHub Repository**: https://github.com/ScrapeGraphAI/scrapegraph-sdk
+- **Documentation**: https://docs.scrapegraphai.com
+- **Get API Key**: https://scrapegraphai.com
+
+---
+
+## πŸ’¬ Support
+
+- πŸ“§ Email: [email protected]
+- πŸ’» GitHub Issues
+- 🌟 Star the repo if you find it useful!
+
+---
-- The app prompts you to enter your OpenAI API key, which is used to authenticate and access the OpenAI language models.
-- You can select the desired language model (GPT-3.5-turbo or GPT-4) for the scraping task.
-- Enter the URL of the website you want to scrape in the provided text input field.
-- Specify what you want the AI agent to extract from the website by entering a user prompt.
-- The app creates a SmartScraperGraph object using the provided URL, user prompt, and OpenAI configuration.
-- The SmartScraperGraph object scrapes the website and extracts the requested information using the specified language model.
-- The scraped results are displayed in the app for you to view
\ No newline at end of file
+Made with ❀️ by the ScrapeGraph AI team
diff --git a/starter_ai_agents/web_scrapping_ai_agent/scrapegraph_ai_sdk/README.md b/starter_ai_agents/web_scrapping_ai_agent/scrapegraph_ai_sdk/README.md
new file mode 100644
index 00000000..07188963
--- /dev/null
+++ b/starter_ai_agents/web_scrapping_ai_agent/scrapegraph_ai_sdk/README.md
@@ -0,0 +1,433 @@
+# πŸ•·οΈ ScrapeGraph AI SDK - Intelligent Web Scraping
+
+Powerful AI-powered web scraping using the official [ScrapeGraph AI SDK](https://github.com/ScrapeGraphAI/scrapegraph-sdk). Extract structured data, convert to markdown, crawl multiple pages, and perform AI-powered web searches.
+
+## πŸ“‹ Overview
+
+ScrapeGraph AI SDK provides cloud-based AI scraping capabilities with natural language prompts. No complex selectors or XPath needed - just describe what you want to extract!
+
+### 🌟 Key Features
+
+- πŸ€– **SmartScraper**: Extract structured data with natural language
+- πŸ” **SearchScraper**: AI-powered web search with structured results
+- πŸ“ **Markdownify**: Convert webpages to clean markdown
+- πŸ•·οΈ **SmartCrawler**: Intelligent multi-page crawling
+- πŸ€– **AgenticScraper**: Automated browser actions
+- ⏰ **Scheduled Jobs**: Cron-based scraping workflows
+- πŸ’³ **Credits Management**: Monitor API usage
+
+## πŸš€ Features
+
+- **Natural Language Extraction**: No CSS selectors needed
+- **Schema Validation**: Structured output with Pydantic
+- **Multiple Formats**: JSON, Markdown, custom schemas
+- **High Performance**: Concurrent processing
+- **Enterprise Ready**: Production-grade reliability
+
+## πŸ“¦ Installation
+
+Install the required dependencies:
+
+```bash
+pip install -r requirements.txt
+```
+
+## πŸ”‘ Setup
+
+1. Get your API key from [ScrapeGraph AI](https://scrapegraphai.com)
+
+2. Set your API key as environment variable:
+
+```bash
+export SGAI_API_KEY='your-api-key-here'
+```
+
+## πŸ’» Usage
+
+### Basic Smart Scraping
+
+Run the basic scraper demo:
+
+```bash
+python smart_scraper_demo.py
+```
+
+### Interactive Web App
+
+Run the Streamlit app for interactive scraping:
+
+```bash
+streamlit run scrapegraph_app.py
+```
+
+### Search Scraping
+
+Run the search scraper:
+
+```bash
+python search_scraper_demo.py
+```
+
+### Smart Crawler
+
+Run the multi-page crawler:
+
+```bash
+python smart_crawler_demo.py
+```
+
+## πŸ“š Examples
+
+### 1. SmartScraper - Extract Product Data
+
+```python
+from scrapegraph_py import Client
+
+client = Client(api_key="your-api-key")
+
+# Extract product information
+response = client.smartscraper(
+ website_url="https://example.com/product",
+ user_prompt="Extract product name, price, description, and availability"
+)
+
+print(response.result)
+```
+
+### 2. SearchScraper - AI Web Search
+
+```python
+# Search and extract structured data
+response = client.smartscraper(
+ user_prompt="Find the top 5 AI news websites with their names, URLs, and descriptions",
+ num_results=5
+)
+
+for result in response.result:
+ print(f"{result['name']}: {result['url']}")
+```
+
+### 3. Markdownify - Convert to Markdown
+
+```python
+# Convert webpage to markdown
+response = client.markdownify(
+ website_url="https://example.com/article"
+)
+
+print(response.result) # Clean markdown output
+```
+
+### 4. SmartCrawler - Multi-page Extraction
+
+```python
+# Crawl and extract from multiple pages
+request_id = client.smartcrawler(
+ website_url="https://docs.example.com",
+ user_prompt="Extract all API endpoints with their methods and descriptions",
+ max_pages=10,
+ depth=2
+)
+
+# Fetch results
+results = client.smartcrawler_get(request_id)
+```
+
+### 5. Schema Validation with Pydantic
+
+```python
+from pydantic import BaseModel, Field
+
+class Product(BaseModel):
+ name: str
+ price: float
+ in_stock: bool
+ rating: float = Field(ge=0, le=5)
+
+# Extract with schema validation
+response = client.smartscraper(
+ website_url="https://example.com/product",
+ user_prompt="Extract product details",
+ output_schema=Product.model_json_schema()
+)
+
+product = Product(**response.result)
+```
+
+## 🎯 Use Cases
+
+### E-commerce Data Extraction
+
+```python
+# Extract product catalog
+products = client.smartscraper(
+ website_url="https://shop.example.com",
+ user_prompt="Extract all products with name, price, image, and availability"
+)
+```
+
+### News & Content Aggregation
+
+```python
+# Convert articles to markdown for processing
+article_md = client.markdownify(
+ website_url="https://news.example.com/article"
+)
+```
+
+### Competitive Intelligence
+
+```python
+# Search and analyze competitor data
+competitors = client.smartscraper(
+ user_prompt="Find top 10 SaaS companies in AI space with pricing info",
+ num_results=10
+)
+```
+
+### Documentation Scraping
+
+```python
+# Crawl entire documentation site
+request_id = client.smartcrawler(
+ website_url="https://docs.example.com",
+ user_prompt="Extract all function signatures, parameters, and examples",
+ max_pages=50
+)
+```
+
+## πŸ”§ Advanced Features
+
+### Custom Output Schemas
+
+Define exactly what data you want:
+
+```python
+schema = {
+ "type": "object",
+ "properties": {
+ "title": {"type": "string"},
+ "author": {"type": "string"},
+ "date": {"type": "string", "format": "date"},
+ "content": {"type": "string"},
+ "tags": {"type": "array", "items": {"type": "string"}}
+ }
+}
+
+response = client.smartscraper(
+ website_url="https://blog.example.com/post",
+ user_prompt="Extract blog post information",
+ output_schema=schema
+)
+```
+
+### JavaScript Rendering
+
+For dynamic Single Page Applications:
+
+```python
+response = client.smartscraper(
+ website_url="https://spa.example.com",
+ user_prompt="Extract data",
+ render_heavy_js=True # Enable JS rendering
+)
+```
+
+### Stealth Mode
+
+Avoid bot detection:
+
+```python
+response = client.smartscraper(
+ website_url="https://protected-site.com",
+ user_prompt="Extract data",
+ stealth=True # Enable stealth mode
+)
+```
+
+### Scheduled Jobs
+
+Create recurring scraping tasks:
+
+```python
+# Create scheduled job (daily at 9 AM)
+job = client.create_scheduled_job(
+ name="Daily Product Scrape",
+ schedule="0 9 * * *", # Cron expression
+ scraper_type="smartscraper",
+ config={
+ "website_url": "https://shop.example.com",
+ "user_prompt": "Extract all products"
+ }
+)
+
+# List all scheduled jobs
+jobs = client.list_scheduled_jobs()
+
+# Get job execution history
+history = client.get_job_history(job.id)
+```
+
+## πŸ“Š Comparison with Local Library
+
+| Feature | SDK (Cloud API) | Local Library |
+|---------|----------------|---------------|
+| Setup | API key only | Full installation |
+| Processing | Cloud-based | Local compute |
+| Scalability | Unlimited | Hardware limited |
+| Maintenance | Managed | Self-managed |
+| Credits | Pay-per-use | Free (local) |
+| Speed | Very fast | Depends on hardware |
+| Features | All features | Core features |
+
+**Use SDK when:**
+- βœ… You want managed infrastructure
+- βœ… You need high scalability
+- βœ… You prefer pay-per-use pricing
+- βœ… You want latest features
+
+**Use Local Library when:**
+- βœ… You have sensitive data
+- βœ… You prefer self-hosted
+- βœ… You have available compute
+- βœ… You want full control
+
+## πŸ’° Credits & Pricing
+
+Different operations use different credit amounts:
+
+- **SmartScraper**: 10 credits per page
+- **Markdownify**: 2 credits per page
+- **SearchScraper**: 10 credits per result
+- **SmartCrawler**: Variable (based on pages)
+- **Scrape**: 1 credit per page
+
+Check your credits:
+
+```python
+balance = client.get_credits()
+print(f"Available credits: {balance.credits}")
+```
+
+## πŸ”— Integrations
+
+ScrapeGraph AI SDK integrates with:
+
+- **LLM Frameworks**: Langchain, LlamaIndex, Crew.ai, CamelAI
+- **Low-code**: Pipedream, Bubble, Zapier, n8n, LangFlow
+- **MCP Server**: Model Context Protocol support
+
+See the [integration docs](https://docs.scrapegraphai.com/integrations) for details.
+
+## πŸ“– API Reference
+
+### Client
+
+```python
+from scrapegraph_py import Client
+
+client = Client(api_key="your-api-key")
+```
+
+### SmartScraper
+
+```python
+response = client.smartscraper(
+ website_url: str | None = None,
+ website_html: str | None = None,
+ user_prompt: str,
+ output_schema: dict | None = None,
+ render_heavy_js: bool = False,
+ stealth: bool = False
+)
+```
+
+### SearchScraper
+
+```python
+response = client.smartscraper(
+ user_prompt: str,
+ num_results: int = 3,
+ output_schema: dict | None = None
+)
+```
+
+### Markdownify
+
+```python
+response = client.markdownify(
+ website_url: str,
+ render_heavy_js: bool = False
+)
+```
+
+### SmartCrawler
+
+```python
+request_id = client.smartcrawler(
+ url: str,
+ user_prompt: str,
+ max_pages: int | None = None,
+ depth: int | None = None,
+ same_domain_only: bool = True
+)
+
+results = client.smartcrawler_get(request_id)
+```
+
+## πŸ› Troubleshooting
+
+### Common Issues
+
+**Issue**: `Authentication failed`
+```bash
+# Solution: Check your API key
+export SGAI_API_KEY='your-actual-key'
+```
+
+**Issue**: `Insufficient credits`
+```python
+# Solution: Check balance and recharge
+balance = client.get_credits()
+print(f"Credits: {balance.credits}")
+```
+
+**Issue**: `Rate limit exceeded`
+```python
+# Solution: Add delays between requests
+import time
+time.sleep(1) # Wait 1 second between calls
+```
+
+## 🀝 Contributing
+
+Contributions welcome! Feel free to:
+- Report bugs
+- Suggest features
+- Submit pull requests
+- Improve documentation
+
+## πŸ“„ License
+
+This example code is provided as-is for educational purposes.
+ScrapeGraph AI SDK is licensed under MIT License.
+
+## πŸ”— Resources
+
+- **GitHub**: https://github.com/ScrapeGraphAI/scrapegraph-sdk
+- **Documentation**: https://docs.scrapegraphai.com
+- **API Docs**: https://docs.scrapegraphai.com/api
+- **Python SDK**: https://docs.scrapegraphai.com/sdk/python
+- **JavaScript SDK**: https://docs.scrapegraphai.com/sdk/javascript
+- **Get API Key**: https://scrapegraphai.com
+
+## πŸ’¬ Support
+
+- πŸ“§ Email: [email protected]
+- πŸ’» GitHub Issues: [Create an issue](https://github.com/ScrapeGraphAI/scrapegraph-sdk/issues)
+- 🌟 Feature Requests: [Request a feature](https://github.com/ScrapeGraphAI/scrapegraph-sdk/issues/new)
+
+---
+
+Made with ❀️ by [ScrapeGraph AI](https://scrapegraphai.com)
+
diff --git a/starter_ai_agents/web_scrapping_ai_agent/scrapegraph_ai_sdk/config.example b/starter_ai_agents/web_scrapping_ai_agent/scrapegraph_ai_sdk/config.example
new file mode 100644
index 00000000..6c2cb80d
--- /dev/null
+++ b/starter_ai_agents/web_scrapping_ai_agent/scrapegraph_ai_sdk/config.example
@@ -0,0 +1,7 @@
+# ScrapeGraph AI Configuration Example
+# Copy this to .env and add your API key
+
+SGAI_API_KEY=your-api-key-here
+
+# Get your API key at: https://scrapegraphai.com
+
diff --git a/starter_ai_agents/web_scrapping_ai_agent/scrapegraph_ai_sdk/quickstart.py b/starter_ai_agents/web_scrapping_ai_agent/scrapegraph_ai_sdk/quickstart.py
new file mode 100644
index 00000000..2c0b21a7
--- /dev/null
+++ b/starter_ai_agents/web_scrapping_ai_agent/scrapegraph_ai_sdk/quickstart.py
@@ -0,0 +1,146 @@
+"""
+ScrapeGraph AI Quick Start
+Quick test to verify installation and basic functionality
+"""
+
+import os
+
+
+def check_installation():
+ """Verify ScrapeGraph AI SDK is installed."""
+ try:
+ from scrapegraph_py import Client
+ print("βœ… scrapegraph-py is installed")
+ return True
+ except ImportError:
+ print("❌ scrapegraph-py not found")
+ print("πŸ’‘ Install with: pip install scrapegraph-py")
+ return False
+
+
+def check_api_key():
+ """Check if API key is configured."""
+ api_key = os.getenv("SGAI_API_KEY")
+ if api_key:
+ print(f"βœ… API key found: {api_key[:10]}...")
+ return api_key
+ else:
+ print("❌ SGAI_API_KEY not found")
+ print("πŸ’‘ Get your API key at: https://scrapegraphai.com")
+ print(" Then set it: export SGAI_API_KEY='your-key'")
+ return None
+
+
+def test_connection(api_key):
+ """Test API connection."""
+ from scrapegraph_py import Client
+
+ try:
+ client = Client(api_key=api_key)
+ balance = client.get_credits()
+ print(f"βœ… Connection successful!")
+ print(f"πŸ’³ Available credits: {balance.credits}")
+ return client
+ except Exception as e:
+ print(f"❌ Connection failed: {e}")
+ return None
+
+
+def quick_scrape_test(client):
+ """Run a quick scraping test."""
+ print("\n" + "=" * 60)
+ print("Running quick scrape test...")
+ print("=" * 60)
+
+ # Simple HTML to scrape
+ html = """
+ <html>
+ <body>
+ <h1>Welcome to ScrapeGraph AI</h1>
+ <div class="features">
+ <div class="feature">
+ <h3>SmartScraper</h3>
+ <p>Extract structured data with natural language</p>
+ </div>
+ <div class="feature">
+ <h3>SearchScraper</h3>
+ <p>AI-powered web search</p>
+ </div>
+ </div>
+ </body>
+ </html>
+ """
+
+ try:
+ response = client.smartscraper(
+ website_html=html,
+ user_prompt="Extract all feature names and descriptions"
+ )
+
+ print("\nβœ… Scrape successful!")
+ print("\nπŸ“Š Results:")
+ import json
+ print(json.dumps(response.result, indent=2))
+
+ return True
+
+ except Exception as e:
+ print(f"\n❌ Scrape failed: {e}")
+ return False
+
+
+def main():
+ """Run quick start checks."""
+ print("=" * 60)
+ print("πŸ•·οΈ SCRAPEGRAPH AI - QUICK START")
+ print("=" * 60)
+ print()
+
+ # 1. Check installation
+ if not check_installation():
+ return False
+
+ print()
+
+ # 2. Check API key
+ api_key = check_api_key()
+ if not api_key:
+ return False
+
+ print()
+
+ # 3. Test connection
+ client = test_connection(api_key)
+ if not client:
+ return False
+
+ # 4. Quick scrape test
+ success = quick_scrape_test(client)
+
+ print("\n" + "=" * 60)
+ if success:
+ print("✨ Quick start completed successfully!")
+ print("\nπŸ’‘ Next steps:")
+ print(" 1. Run full demo: python smart_scraper_demo.py")
+ print(" 2. Try search: python search_scraper_demo.py")
+ print(" 3. Launch app: streamlit run scrapegraph_app.py")
+ else:
+ print("⚠️ Quick start completed with errors")
+ print("πŸ’‘ Check the error messages above")
+
+ print("=" * 60)
+
+ return success
+
+
+if __name__ == "__main__":
+ try:
+ success = main()
+ exit(0 if success else 1)
+ except KeyboardInterrupt:
+ print("\n\n⚠️ Interrupted by user")
+ exit(1)
+ except Exception as e:
+ print(f"\n❌ Unexpected error: {e}")
+ exit(1)
+
diff --git a/starter_ai_agents/web_scrapping_ai_agent/scrapegraph_ai_sdk/requirements.txt b/starter_ai_agents/web_scrapping_ai_agent/scrapegraph_ai_sdk/requirements.txt
new file mode 100644
index 00000000..003d4527
--- /dev/null
+++ b/starter_ai_agents/web_scrapping_ai_agent/scrapegraph_ai_sdk/requirements.txt
@@ -0,0 +1,6 @@
+scrapegraph-py>=1.41.0
+streamlit>=1.28.0
+pydantic>=2.0.0
+python-dotenv>=1.0.0
+pandas>=2.0.0
+
diff --git a/starter_ai_agents/web_scrapping_ai_agent/scrapegraph_ai_sdk/scrapegraph_app.py b/starter_ai_agents/web_scrapping_ai_agent/scrapegraph_ai_sdk/scrapegraph_app.py
new file mode 100644
index 00000000..33627b74
--- /dev/null
+++ b/starter_ai_agents/web_scrapping_ai_agent/scrapegraph_ai_sdk/scrapegraph_app.py
@@ -0,0 +1,392 @@
+"""
+ScrapeGraph AI Interactive Streamlit App
+Comprehensive web scraping interface with all features
+"""
+
+import streamlit as st
+from scrapegraph_py import Client
+import json
+import os
+from typing import Optional
+import time
+
+
+def init_client() -> Optional[Client]:
+ """Initialize ScrapeGraph AI client."""
+ api_key = st.session_state.get("api_key") or os.getenv("SGAI_API_KEY")
+
+ if not api_key:
+ st.error("❌ Please enter your ScrapeGraph AI API key")
+ st.info("Get your API key at: https://scrapegraphai.com")
+ return None
+
+ try:
+ client = Client(api_key=api_key)
+ # Try to get credits to verify key
+ balance = client.get_credits()
+ st.sidebar.success(f"πŸ’³ Credits: {balance.credits}")
+ return client
+ except Exception as e:
+ st.error(f"❌ Error initializing client: {e}")
+ return None
+
+
+def smart_scraper_tab(client: Client):
+ """SmartScraper interface."""
+ st.header("πŸ€– SmartScraper")
+ st.markdown("Extract structured data from websites using natural language")
+
+ col1, col2 = st.columns([2, 1])
+
+ with col1:
+ url = st.text_input("Website URL", placeholder="https://example.com")
+
+ with col2:
+ render_js = st.checkbox("Render JavaScript", help="Enable for SPAs")
+
+ prompt = st.text_area(
+ "What to extract?",
+ placeholder="Extract product names, prices, and availability",
+ height=100
+ )
+
+ # Advanced options
+ with st.expander("βš™οΈ Advanced Options"):
+ use_html = st.checkbox("Use HTML input instead of URL")
+
+ if use_html:
+ html_content = st.text_area("HTML Content", height=200)
+
+ use_schema = st.checkbox("Define output schema")
+
+ if use_schema:
+ schema_input = st.text_area(
+ "JSON Schema",
+ value='{\n "type": "object",\n "properties": {\n "items": {"type": "array"}\n }\n}',
+ height=200
+ )
+
+ stealth = st.checkbox("Enable stealth mode")
+
+ if st.button("πŸš€ Extract Data", key="smart_scraper_btn"):
+ if not prompt:
+ st.error("Please enter an extraction prompt")
+ return
+
+ if not use_html and not url:
+ st.error("Please enter a URL")
+ return
+
+ if use_html and not html_content:
+ st.error("Please enter HTML content")
+ return
+
+ with st.spinner("Extracting data..."):
+ try:
+ # Prepare parameters
+ params = {
+ "user_prompt": prompt,
+ "render_heavy_js": render_js,
+ }
+
+ if use_html:
+ params["website_html"] = html_content
+ else:
+ params["website_url"] = url
+
+ if use_schema:
+ try:
+ params["output_schema"] = json.loads(schema_input)
+ except json.JSONDecodeError:
+ st.error("Invalid JSON schema")
+ return
+
+ if stealth:
+ params["stealth"] = True
+
+ # Make request
+ response = client.smartscraper(**params)
+
+ # Display results
+ st.success("βœ… Data extracted successfully!")
+
+ st.subheader("πŸ“Š Results")
+ st.json(response.result)
+
+ # Download button
+ st.download_button(
+ label="πŸ’Ύ Download JSON",
+ data=json.dumps(response.result, indent=2),
+ file_name="scraped_data.json",
+ mime="application/json"
+ )
+
+ except Exception as e:
+ st.error(f"❌ Error: {e}")
+
+
+def search_scraper_tab(client: Client):
+ """SearchScraper interface."""
+ st.header("πŸ” SearchScraper")
+ st.markdown("AI-powered web search with structured results")
+
+ query = st.text_area(
+ "Search Query",
+ placeholder="Find the top 5 AI news websites with descriptions",
+ height=100
+ )
+
+ col1, col2 = st.columns(2)
+
+ with col1:
+ num_results = st.slider("Number of results", 1, 10, 5)
+
+ with col2:
+ use_schema = st.checkbox("Define result schema")
+
+ if use_schema:
+ schema_input = st.text_area(
+ "JSON Schema",
+ value='{\n "type": "object",\n "properties": {\n "results": {"type": "array"}\n }\n}',
+ height=200
+ )
+
+ if st.button("πŸ” Search", key="search_btn"):
+ if not query:
+ st.error("Please enter a search query")
+ return
+
+ with st.spinner("Searching..."):
+ try:
+ params = {
+ "user_prompt": query,
+ "num_results": num_results
+ }
+
+ if use_schema:
+ try:
+ params["output_schema"] = json.loads(schema_input)
+ except json.JSONDecodeError:
+ st.error("Invalid JSON schema")
+ return
+
+ response = client.smartscraper(**params)
+
+ st.success("βœ… Search completed!")
+
+ st.subheader("πŸ“Š Results")
+ st.json(response.result)
+
+ st.download_button(
+ label="πŸ’Ύ Download Results",
+ data=json.dumps(response.result, indent=2),
+ file_name="search_results.json",
+ mime="application/json"
+ )
+
+ except Exception as e:
+ st.error(f"❌ Error: {e}")
+
+
+def markdownify_tab(client: Client):
+ """Markdownify interface."""
+ st.header("πŸ“ Markdownify")
+ st.markdown("Convert any webpage to clean markdown")
+
+ url = st.text_input("Website URL", placeholder="https://example.com/article")
+
+ render_js = st.checkbox("Render JavaScript", help="Enable for SPAs")
+
+ if st.button("πŸ”„ Convert to Markdown", key="markdown_btn"):
+ if not url:
+ st.error("Please enter a URL")
+ return
+
+ with st.spinner("Converting..."):
+ try:
+ response = client.markdownify(
+ website_url=url,
+ render_heavy_js=render_js
+ )
+
+ st.success("βœ… Converted successfully!")
+
+ st.subheader("πŸ“„ Markdown Output")
+ st.code(response.result, language="markdown")
+
+ st.download_button(
+ label="πŸ’Ύ Download Markdown",
+ data=response.result,
+ file_name="content.md",
+ mime="text/markdown"
+ )
+
+ except Exception as e:
+ st.error(f"❌ Error: {e}")
+
+
+def smart_crawler_tab(client: Client):
+ """SmartCrawler interface."""
+ st.header("πŸ•·οΈ SmartCrawler")
+ st.markdown("Intelligently crawl and extract from multiple pages")
+
+ st.warning("⚠️ SmartCrawler is asynchronous and may take several minutes")
+
+ url = st.text_input("Start URL", placeholder="https://example.com")
+
+ prompt = st.text_area(
+ "What to extract?",
+ placeholder="Extract all page titles and descriptions",
+ height=100
+ )
+
+ col1, col2, col3 = st.columns(3)
+
+ with col1:
+ max_pages = st.number_input("Max pages", 1, 100, 10)
+
+ with col2:
+ depth = st.number_input("Crawl depth", 1, 5, 2)
+
+ with col3:
+ same_domain = st.checkbox("Same domain only", value=True)
+
+ col_mode1, col_mode2 = st.columns(2)
+
+ with col_mode1:
+ extraction_mode = st.selectbox("Mode", ["ai", "markdown"])
+
+ if st.button("πŸš€ Start Crawler", key="crawler_btn"):
+ if not url:
+ st.error("Please enter a URL")
+ return
+
+ with st.spinner("Starting crawler..."):
+ try:
+ params = {
+ "url": url,
+ "extraction_mode": extraction_mode,
+ "max_pages": max_pages,
+ "depth": depth,
+ "same_domain_only": same_domain
+ }
+
+ if extraction_mode == "ai" and prompt:
+ params["prompt"] = prompt
+
+ request_id = client.smartcrawler(**params)
+
+ st.success(f"βœ… Crawler started!")
+ st.info(f"Request ID: `{request_id}`")
+
+ st.markdown("### ⏳ Fetching results...")
+ progress_bar = st.progress(0)
+ status_text = st.empty()
+
+ # Poll for results
+ max_attempts = 30
+ for i in range(max_attempts):
+ try:
+ results = client.smartcrawler_get(request_id)
+
+ if results:
+ progress_bar.progress(100)
+ status_text.success("βœ… Crawl completed!")
+
+ st.subheader("πŸ“Š Results")
+ st.json(results)
+
+ st.download_button(
+ label="πŸ’Ύ Download Results",
+ data=json.dumps(results, indent=2),
+ file_name="crawl_results.json",
+ mime="application/json"
+ )
+ break
+ except Exception as e:
+ if "not ready" in str(e).lower():
+ progress = int((i + 1) / max_attempts * 100)
+ progress_bar.progress(progress)
+ status_text.text(f"Processing... ({i + 1}/{max_attempts})")
+ time.sleep(2)
+ else:
+ raise
+ else:
+ st.warning("⏰ Crawler is still processing. Check back later with the request ID.")
+
+ except Exception as e:
+ st.error(f"❌ Error: {e}")
+
+
+def main():
+ st.set_page_config(
+ page_title="ScrapeGraph AI SDK",
+ page_icon="πŸ•·οΈ",
+ layout="wide"
+ )
+
+ st.title("πŸ•·οΈ ScrapeGraph AI SDK")
+ st.markdown("Intelligent web scraping powered by AI")
+
+ # Sidebar
+ with st.sidebar:
+ st.header("βš™οΈ Configuration")
+
+ api_key = st.text_input(
+ "API Key",
+ type="password",
+ value=os.getenv("SGAI_API_KEY", ""),
+ help="Get your API key at scrapegraphai.com"
+ )
+
+ if api_key:
+ st.session_state.api_key = api_key
+
+ st.markdown("---")
+
+ st.markdown("### πŸ”— Links")
+ st.markdown("[GitHub](https://github.com/ScrapeGraphAI/scrapegraph-sdk)")
+ st.markdown("[Documentation](https://docs.scrapegraphai.com)")
+ st.markdown("[Get API Key](https://scrapegraphai.com)")
+
+ st.markdown("---")
+
+ st.markdown("### πŸ’‘ Features")
+ st.markdown("""
+ - πŸ€– SmartScraper
+ - πŸ” SearchScraper
+ - πŸ“ Markdownify
+ - πŸ•·οΈ SmartCrawler
+ """)
+
+ # Initialize client
+ client = init_client()
+
+ if not client:
+ st.info("πŸ‘† Please enter your API key in the sidebar to get started")
+ return
+
+ # Main tabs
+ tab1, tab2, tab3, tab4 = st.tabs([
+ "πŸ€– SmartScraper",
+ "πŸ” SearchScraper",
+ "πŸ“ Markdownify",
+ "πŸ•·οΈ SmartCrawler"
+ ])
+
+ with tab1:
+ smart_scraper_tab(client)
+
+ with tab2:
+ search_scraper_tab(client)
+
+ with tab3:
+ markdownify_tab(client)
+
+ with tab4:
+ smart_crawler_tab(client)
+
+
+if __name__ == "__main__":
+ main()
+
diff --git a/starter_ai_agents/web_scrapping_ai_agent/scrapegraph_ai_sdk/search_scraper_demo.py b/starter_ai_agents/web_scrapping_ai_agent/scrapegraph_ai_sdk/search_scraper_demo.py
new file mode 100644
index 00000000..534c63bd
--- /dev/null
+++ b/starter_ai_agents/web_scrapping_ai_agent/scrapegraph_ai_sdk/search_scraper_demo.py
@@ -0,0 +1,296 @@
+"""
+ScrapeGraph AI SearchScraper Demo
+AI-powered web search with structured data extraction
+"""
+
+import os
+from scrapegraph_py import Client
+from pydantic import BaseModel
+from typing import List
+import json
+
+
+class SearchResult(BaseModel):
+ """Schema for search results."""
+ title: str
+ url: str
+ description: str
+
+
+class SearchResults(BaseModel):
+ """Schema for multiple search results."""
+ results: List[SearchResult]
+
+
+def basic_search_example():
+ """Basic AI-powered web search."""
+
+ print("=" * 80)
+ print("πŸ” SCRAPEGRAPH AI - SEARCH SCRAPER DEMO")
+ print("=" * 80)
+
+ api_key = os.getenv("SGAI_API_KEY")
+ if not api_key:
+ print("\n❌ Error: SGAI_API_KEY environment variable not set")
+ print("πŸ’‘ Get your API key at: https://scrapegraphai.com")
+ return
+
+ client = Client(api_key=api_key)
+
+ # Check credits
+ try:
+ balance = client.get_credits()
+ print(f"\nπŸ’³ Available credits: {balance.credits}")
+ except Exception as e:
+ print(f"\n⚠️ Could not check credits: {e}")
+
+ print("\n" + "=" * 80)
+ print("Example 1: Basic Web Search")
+ print("=" * 80)
+
+ query = "Top 5 AI news websites"
+ print(f"\nπŸ” Query: {query}")
+
+ try:
+ response = client.smartscraper(
+ user_prompt=query,
+ num_results=5
+ )
+
+ print("\nβœ… Results:")
+ print(json.dumps(response.result, indent=2))
+
+ except Exception as e:
+ print(f"\n❌ Error: {e}")
+
+
+def structured_search_example():
+ """Search with structured output schema."""
+
+ print("\n" + "=" * 80)
+ print("Example 2: Structured Search Results")
+ print("=" * 80)
+
+ api_key = os.getenv("SGAI_API_KEY")
+ if not api_key:
+ return
+
+ client = Client(api_key=api_key)
+
+ query = "Find the best Python web scraping libraries with their GitHub stars and documentation links"
+ print(f"\nπŸ” Query: {query}")
+
+ schema = {
+ "type": "object",
+ "properties": {
+ "libraries": {
+ "type": "array",
+ "items": {
+ "type": "object",
+ "properties": {
+ "name": {"type": "string"},
+ "description": {"type": "string"},
+ "github_url": {"type": "string"},
+ "stars": {"type": "string"},
+ "docs_url": {"type": "string"}
+ }
+ }
+ }
+ }
+ }
+
+ try:
+ response = client.smartscraper(
+ user_prompt=query,
+ num_results=5,
+ output_schema=schema
+ )
+
+ print("\nβœ… Structured Results:")
+ print(json.dumps(response.result, indent=2))
+
+ except Exception as e:
+ print(f"\n❌ Error: {e}")
+
+
+def competitive_intelligence_example():
+ """Use search for competitive intelligence."""
+
+ print("\n" + "=" * 80)
+ print("Example 3: Competitive Intelligence")
+ print("=" * 80)
+
+ api_key = os.getenv("SGAI_API_KEY")
+ if not api_key:
+ return
+
+ client = Client(api_key=api_key)
+
+ query = "Top SaaS companies in the web scraping space with pricing and features"
+ print(f"\nπŸ” Query: {query}")
+
+ schema = {
+ "type": "object",
+ "properties": {
+ "companies": {
+ "type": "array",
+ "items": {
+ "type": "object",
+ "properties": {
+ "company_name": {"type": "string"},
+ "website": {"type": "string"},
+ "pricing_model": {"type": "string"},
+ "key_features": {"type": "array", "items": {"type": "string"}},
+ "target_market": {"type": "string"}
+ }
+ }
+ }
+ }
+ }
+
+ try:
+ response = client.smartscraper(
+ user_prompt=query,
+ num_results=5,
+ output_schema=schema
+ )
+
+ print("\nβœ… Competitive Analysis:")
+ if "companies" in response.result:
+ for company in response.result["companies"]:
+ print(f"\n 🏒 {company.get('company_name', 'N/A')}")
+ print(f" Website: {company.get('website', 'N/A')}")
+ print(f" Pricing: {company.get('pricing_model', 'N/A')}")
+ if 'key_features' in company:
+ print(f" Features: {', '.join(company['key_features'][:3])}")
+ else:
+ print(json.dumps(response.result, indent=2))
+
+ except Exception as e:
+ print(f"\n❌ Error: {e}")
+
+
+def news_aggregation_example():
+ """Aggregate news articles on a topic."""
+
+ print("\n" + "=" * 80)
+ print("Example 4: News Aggregation")
+ print("=" * 80)
+
+ api_key = os.getenv("SGAI_API_KEY")
+ if not api_key:
+ return
+
+ client = Client(api_key=api_key)
+
+ query = "Latest AI breakthroughs in 2024 with article titles, dates, and summaries"
+ print(f"\nπŸ” Query: {query}")
+
+ schema = {
+ "type": "object",
+ "properties": {
+ "articles": {
+ "type": "array",
+ "items": {
+ "type": "object",
+ "properties": {
+ "title": {"type": "string"},
+ "source": {"type": "string"},
+ "url": {"type": "string"},
+ "date": {"type": "string"},
+ "summary": {"type": "string"}
+ }
+ }
+ }
+ }
+ }
+
+ try:
+ response = client.smartscraper(
+ user_prompt=query,
+ num_results=5,
+ output_schema=schema
+ )
+
+ print("\nβœ… News Articles:")
+ if "articles" in response.result:
+ for i, article in enumerate(response.result["articles"], 1):
+ print(f"\n πŸ“° Article {i}: {article.get('title', 'N/A')}")
+ print(f" Source: {article.get('source', 'N/A')}")
+ print(f" Date: {article.get('date', 'N/A')}")
+ print(f" URL: {article.get('url', 'N/A')}")
+ summary = article.get('summary', 'N/A')
+ if len(summary) > 100:
+ summary = summary[:100] + "..."
+ print(f" Summary: {summary}")
+ else:
+ print(json.dumps(response.result, indent=2))
+
+ except Exception as e:
+ print(f"\n❌ Error: {e}")
+
+
+def market_research_example():
+ """Perform market research using search."""
+
+ print("\n" + "=" * 80)
+ print("Example 5: Market Research")
+ print("=" * 80)
+
+ api_key = os.getenv("SGAI_API_KEY")
+ if not api_key:
+ return
+
+ client = Client(api_key=api_key)
+
+ query = "E-commerce platforms for small businesses with features and pricing"
+ print(f"\nπŸ” Query: {query}")
+
+ try:
+ response = client.smartscraper(
+ user_prompt=query,
+ num_results=5
+ )
+
+ print("\nβœ… Market Research Results:")
+ print(json.dumps(response.result, indent=2))
+
+ except Exception as e:
+ print(f"\n❌ Error: {e}")
+
+
+def main():
+ """Run all search examples."""
+
+ # Basic search
+ basic_search_example()
+
+ # Structured search
+ structured_search_example()
+
+ # Competitive intelligence
+ competitive_intelligence_example()
+
+ # News aggregation
+ news_aggregation_example()
+
+ # Market research
+ market_research_example()
+
+ print("\n" + "=" * 80)
+ print("πŸŽ‰ Search Demo completed!")
+ print("\nπŸ’‘ Use Cases:")
+ print(" - Competitive analysis")
+ print(" - Market research")
+ print(" - News aggregation")
+ print(" - Lead generation")
+ print(" - Content discovery")
+ print("\nπŸ”— Resources:")
+ print(" - GitHub: https://github.com/ScrapeGraphAI/scrapegraph-sdk")
+ print(" - Docs: https://docs.scrapegraphai.com")
+ print("=" * 80)
+
+
+if __name__ == "__main__":
+ main()
+
diff --git a/starter_ai_agents/web_scrapping_ai_agent/scrapegraph_ai_sdk/smart_crawler_demo.py b/starter_ai_agents/web_scrapping_ai_agent/scrapegraph_ai_sdk/smart_crawler_demo.py
new file mode 100644
index 00000000..5cbb9767
--- /dev/null
+++ b/starter_ai_agents/web_scrapping_ai_agent/scrapegraph_ai_sdk/smart_crawler_demo.py
@@ -0,0 +1,298 @@
+"""
+ScrapeGraph AI SmartCrawler Demo
+Intelligently crawl and extract data from multiple pages
+"""
+
+import os
+from scrapegraph_py import Client
+import json
+import time
+
+
+def basic_crawler_example():
+ """Basic multi-page crawling."""
+
+ print("=" * 80)
+ print("πŸ•·οΈ SCRAPEGRAPH AI - SMART CRAWLER DEMO")
+ print("=" * 80)
+
+ api_key = os.getenv("SGAI_API_KEY")
+ if not api_key:
+ print("\n❌ Error: SGAI_API_KEY environment variable not set")
+ print("πŸ’‘ Get your API key at: https://scrapegraphai.com")
+ return
+
+ client = Client(api_key=api_key)
+
+ # Check credits
+ try:
+ balance = client.get_credits()
+ print(f"\nπŸ’³ Available credits: {balance.credits}")
+ except Exception as e:
+ print(f"\n⚠️ Could not check credits: {e}")
+
+ print("\n" + "=" * 80)
+ print("Example 1: Basic Website Crawling")
+ print("=" * 80)
+
+ # Start crawling
+ url = "https://www.scrapethissite.com/pages/"
+ prompt = "Extract all page titles and descriptions"
+
+ print(f"\nπŸ” Starting crawler...")
+ print(f" URL: {url}")
+ print(f" Prompt: {prompt}")
+ print(f" Max pages: 5")
+ print(f" Depth: 1")
+
+ try:
+ # Initiate crawl
+ request_id = client.smartcrawler(
+ url=url,
+ user_prompt=prompt,
+ max_pages=5,
+ depth=1,
+ same_domain_only=True
+ )
+
+ print(f"\nβœ… Crawler started!")
+ print(f" Request ID: {request_id}")
+ print("\n⏳ Waiting for results...")
+
+ # Poll for results
+ max_attempts = 30
+ for attempt in range(max_attempts):
+ try:
+ results = client.smartcrawler_get(request_id)
+
+ if results:
+ print("\nβœ… Crawl completed!")
+ print(f"\nπŸ“Š Results:")
+ print(json.dumps(results, indent=2))
+ break
+
+ except Exception as e:
+ if "not ready" in str(e).lower() or "pending" in str(e).lower():
+ print(f" Attempt {attempt + 1}/{max_attempts}...")
+ time.sleep(2)
+ else:
+ raise
+ else:
+ print("\n⏰ Timeout waiting for results")
+
+ except Exception as e:
+ print(f"\n❌ Error: {e}")
+
+
+def documentation_crawler_example():
+ """Crawl documentation sites."""
+
+ print("\n" + "=" * 80)
+ print("Example 2: Documentation Crawling")
+ print("=" * 80)
+
+ api_key = os.getenv("SGAI_API_KEY")
+ if not api_key:
+ return
+
+ client = Client(api_key=api_key)
+
+ # Example: Crawl a documentation site
+ url = "https://docs.python.org/3/"
+ prompt = "Extract all function names, descriptions, and code examples"
+
+ print(f"\nπŸ” Crawling documentation...")
+ print(f" URL: {url}")
+ print(f" Max pages: 10")
+ print(f" Depth: 2")
+
+ try:
+ request_id = client.smartcrawler(
+ url=url,
+ user_prompt=prompt,
+ max_pages=10,
+ depth=2,
+ same_domain_only=True
+ )
+
+ print(f"\nβœ… Crawler initiated: {request_id}")
+ print("⏳ Crawling in progress...")
+ print("πŸ’‘ This may take a few minutes for multiple pages")
+
+ # Note: In production, you'd poll for results
+ # For this demo, we just show how to initiate
+
+ except Exception as e:
+ print(f"\n❌ Error: {e}")
+
+
+def product_catalog_crawler():
+ """Crawl e-commerce product catalogs."""
+
+ print("\n" + "=" * 80)
+ print("Example 3: Product Catalog Crawling")
+ print("=" * 80)
+
+ api_key = os.getenv("SGAI_API_KEY")
+ if not api_key:
+ return
+
+ client = Client(api_key=api_key)
+
+ url = "https://www.scrapethissite.com/pages/simple/"
+ prompt = "Extract all country information including name, capital, population, and area"
+
+ print(f"\nπŸ” Crawling product catalog...")
+ print(f" URL: {url}")
+ print(f" Extraction mode: AI")
+
+ try:
+ request_id = client.smartcrawler(
+ url=url,
+ prompt=prompt,
+ extraction_mode="ai",
+ max_pages=10,
+ depth=1
+ )
+
+ print(f"\nβœ… Crawler started: {request_id}")
+ print("⏳ Processing...")
+
+ # Poll for results
+ for i in range(20):
+ try:
+ results = client.smartcrawler_get(request_id)
+ if results:
+ print("\nβœ… Results ready!")
+ print(json.dumps(results, indent=2)[:500] + "...")
+ break
+ except:
+ time.sleep(3)
+
+ except Exception as e:
+ print(f"\n❌ Error: {e}")
+
+
+def markdown_crawler_example():
+ """Crawl and convert pages to markdown."""
+
+ print("\n" + "=" * 80)
+ print("Example 4: Markdown Conversion Crawler")
+ print("=" * 80)
+
+ api_key = os.getenv("SGAI_API_KEY")
+ if not api_key:
+ return
+
+ client = Client(api_key=api_key)
+
+ url = "https://www.scrapethissite.com/pages/"
+
+ print(f"\nπŸ” Converting pages to markdown...")
+ print(f" URL: {url}")
+ print(f" Mode: Markdown")
+
+ try:
+ request_id = client.smartcrawler(
+ url=url,
+ extraction_mode="markdown",
+ max_pages=5,
+ depth=1
+ )
+
+ print(f"\nβœ… Crawler initiated: {request_id}")
+ print("⏳ Converting to markdown...")
+
+ except Exception as e:
+ print(f"\n❌ Error: {e}")
+
+
+def custom_depth_crawl():
+ """Demonstrate different crawl depths."""
+
+ print("\n" + "=" * 80)
+ print("Example 5: Custom Depth Crawling")
+ print("=" * 80)
+
+ api_key = os.getenv("SGAI_API_KEY")
+ if not api_key:
+ return
+
+ client = Client(api_key=api_key)
+
+ url = "https://www.scrapethissite.com/"
+ prompt = "Extract page titles and main content"
+
+ print("\nπŸ” Comparing different crawl depths:")
+
+ # Depth 1: Only linked pages from start URL
+ print("\n Depth 1: Direct links only")
+
+ try:
+ request_id_1 = client.smartcrawler(
+ url=url,
+ user_prompt=prompt,
+ depth=1,
+ max_pages=5
+ )
+ print(f" Request ID: {request_id_1}")
+
+ except Exception as e:
+ print(f" Error: {e}")
+
+ # Depth 2: Two levels deep
+ print("\n Depth 2: Two levels deep")
+
+ try:
+ request_id_2 = client.smartcrawler(
+ url=url,
+ user_prompt=prompt,
+ depth=2,
+ max_pages=10
+ )
+ print(f" Request ID: {request_id_2}")
+
+ except Exception as e:
+ print(f" Error: {e}")
+
+ print("\nπŸ’‘ Higher depth = more pages discovered")
+ print("πŸ’‘ Use max_pages to control costs")
+
+
+def main():
+ """Run all crawler examples."""
+
+ print("\n⚠️ Note: SmartCrawler is asynchronous and may take time")
+ print("πŸ’‘ Results are fetched via polling or webhooks")
+
+ # Basic crawler
+ basic_crawler_example()
+
+ # Documentation crawler
+ documentation_crawler_example()
+
+ # Product catalog
+ product_catalog_crawler()
+
+ # Markdown crawler
+ markdown_crawler_example()
+
+ # Custom depth
+ custom_depth_crawl()
+
+ print("\n" + "=" * 80)
+ print("πŸŽ‰ Crawler Demo completed!")
+ print("\nπŸ’‘ Best Practices:")
+ print(" - Start with small max_pages to test")
+ print(" - Use same_domain_only to avoid following external links")
+ print(" - Set appropriate depth based on site structure")
+ print(" - Monitor credit usage for large crawls")
+ print("\nπŸ”— Resources:")
+ print(" - GitHub: https://github.com/ScrapeGraphAI/scrapegraph-sdk")
+ print(" - Docs: https://docs.scrapegraphai.com")
+ print("=" * 80)
+
+
+if __name__ == "__main__":
+ main()
+
diff --git a/starter_ai_agents/web_scrapping_ai_agent/scrapegraph_ai_sdk/smart_scraper_demo.py b/starter_ai_agents/web_scrapping_ai_agent/scrapegraph_ai_sdk/smart_scraper_demo.py
new file mode 100644
index 00000000..6a11d35b
--- /dev/null
+++ b/starter_ai_agents/web_scrapping_ai_agent/scrapegraph_ai_sdk/smart_scraper_demo.py
@@ -0,0 +1,279 @@
+"""
+ScrapeGraph AI SmartScraper Demo
+Extract structured data from websites using natural language prompts
+"""
+
+import os
+from scrapegraph_py import Client
+from pydantic import BaseModel, Field
+from typing import List, Optional
+import json
+
+
+def basic_scraping_example():
+ """Basic web scraping with natural language."""
+
+ print("=" * 80)
+ print("πŸ•·οΈ SCRAPEGRAPH AI - SMART SCRAPER DEMO")
+ print("=" * 80)
+
+ # Initialize client
+ api_key = os.getenv("SGAI_API_KEY")
+ if not api_key:
+ print("\n❌ Error: SGAI_API_KEY environment variable not set")
+ print("πŸ’‘ Get your API key at: https://scrapegraphai.com")
+ print(" Then run: export SGAI_API_KEY='your-key'")
+ return
+
+ client = Client(api_key=api_key)
+
+ # Check credits
+ try:
+ balance = client.get_credits()
+ print(f"\nπŸ’³ Available credits: {balance.credits}")
+ except Exception as e:
+ print(f"\n⚠️ Could not check credits: {e}")
+
+ print("\n" + "=" * 80)
+ print("Example 1: Extract Product Information")
+ print("=" * 80)
+
+ # Example website (you can change this)
+ url = "https://www.scrapethissite.com/pages/simple/"
+
+ print(f"\nπŸ” Scraping: {url}")
+ print("πŸ“‹ Prompt: Extract country names, capitals, and population")
+
+ try:
+ response = client.smartscraper(
+ website_url=url,
+ user_prompt="Extract the first 5 countries with their names, capitals, and population"
+ )
+
+ print("\nβœ… Results:")
+ print(json.dumps(response.result, indent=2))
+
+ except Exception as e:
+ print(f"\n❌ Error: {e}")
+
+
+def schema_validation_example():
+ """Scraping with Pydantic schema validation."""
+
+ print("\n" + "=" * 80)
+ print("Example 2: Schema Validation with Pydantic")
+ print("=" * 80)
+
+ # Define data structure
+ class Country(BaseModel):
+ name: str
+ capital: str
+ population: int
+ area: Optional[float] = None
+
+ class Countries(BaseModel):
+ countries: List[Country]
+
+ api_key = os.getenv("SGAI_API_KEY")
+ if not api_key:
+ return
+
+ client = Client(api_key=api_key)
+
+ url = "https://www.scrapethissite.com/pages/simple/"
+
+ print(f"\nπŸ” Scraping: {url}")
+ print("πŸ“‹ Using Pydantic schema for validation")
+
+ try:
+ response = client.smartscraper(
+ website_url=url,
+ user_prompt="Extract the first 3 countries with name, capital, and population",
+ output_schema=Countries.model_json_schema()
+ )
+
+ # Validate with Pydantic
+ countries = Countries(**response.result)
+
+ print("\nβœ… Validated Results:")
+ for country in countries.countries:
+ print(f"\n 🌍 {country.name}")
+ print(f" Capital: {country.capital}")
+ print(f" Population: {country.population:,}")
+
+ except Exception as e:
+ print(f"\n❌ Error: {e}")
+
+
+def html_scraping_example():
+ """Scrape from HTML string instead of URL."""
+
+ print("\n" + "=" * 80)
+ print("Example 3: Scraping from HTML String")
+ print("=" * 80)
+
+ api_key = os.getenv("SGAI_API_KEY")
+ if not api_key:
+ return
+
+ client = Client(api_key=api_key)
+
+ # Sample HTML
+ html_content = """
+ <html>
+ <body>
+ <div class="products">
+ <div class="product">
+ <h2>Laptop Pro</h2>
+ <p class="price">$1,299</p>
+ <p class="stock">In Stock</p>
+ </div>
+ <div class="product">
+ <h2>Magic Mouse</h2>
+ <p class="price">$79</p>
+ <p class="stock">Out of Stock</p>
+ </div>
+ <div class="product">
+ <h2>Wireless Keyboard</h2>
+ <p class="price">$89</p>
+ <p class="stock">In Stock</p>
+ </div>
+ </div>
+ </body>
+ </html>
+ """
+
+ print("\nπŸ“„ Scraping from HTML content")
+
+ try:
+ response = client.smartscraper(
+ website_html=html_content,
+ user_prompt="Extract all products with their names, prices, and stock status"
+ )
+
+ print("\nβœ… Results:")
+ print(json.dumps(response.result, indent=2))
+
+ except Exception as e:
+ print(f"\n❌ Error: {e}")
+
+
+def custom_schema_example():
+ """Use custom JSON schema for precise output structure."""
+
+ print("\n" + "=" * 80)
+ print("Example 4: Custom JSON Schema")
+ print("=" * 80)
+
+ api_key = os.getenv("SGAI_API_KEY")
+ if not api_key:
+ return
+
+ client = Client(api_key=api_key)
+
+ # Define custom schema
+ schema = {
+ "type": "object",
+ "properties": {
+ "countries": {
+ "type": "array",
+ "items": {
+ "type": "object",
+ "properties": {
+ "country_name": {"type": "string"},
+ "capital_city": {"type": "string"},
+ "population_count": {"type": "number"}
+ },
+ "required": ["country_name", "capital_city", "population_count"]
+ }
+ }
+ },
+ "required": ["countries"]
+ }
+
+ url = "https://www.scrapethissite.com/pages/simple/"
+
+ print(f"\nπŸ” Scraping: {url}")
+ print("πŸ“‹ Using custom JSON schema")
+
+ try:
+ response = client.smartscraper(
+ website_url=url,
+ user_prompt="Extract first 3 countries",
+ output_schema=schema
+ )
+
+ print("\nβœ… Results:")
+ print(json.dumps(response.result, indent=2))
+
+ except Exception as e:
+ print(f"\n❌ Error: {e}")
+
+
+def spa_rendering_example():
+ """Scrape Single Page Applications with JavaScript rendering."""
+
+ print("\n" + "=" * 80)
+ print("Example 5: JavaScript Rendering for SPAs")
+ print("=" * 80)
+
+ api_key = os.getenv("SGAI_API_KEY")
+ if not api_key:
+ return
+
+ client = Client(api_key=api_key)
+
+ # Example SPA site
+ url = "https://www.scrapethissite.com/pages/ajax-javascript/"
+
+ print(f"\nπŸ” Scraping SPA: {url}")
+ print("βš™οΈ JavaScript rendering: ENABLED")
+
+ try:
+ response = client.smartscraper(
+ website_url=url,
+ user_prompt="Extract the years and number of Oscar wins",
+ render_heavy_js=True # Enable JS rendering
+ )
+
+ print("\nβœ… Results:")
+ print(json.dumps(response.result, indent=2))
+
+ except Exception as e:
+ print(f"\n❌ Error: {e}")
+
+
+def main():
+ """Run all examples."""
+
+ # Basic scraping
+ basic_scraping_example()
+
+ # Schema validation
+ schema_validation_example()
+
+ # HTML scraping
+ html_scraping_example()
+
+ # Custom schema
+ custom_schema_example()
+
+ # SPA rendering
+ spa_rendering_example()
+
+ print("\n" + "=" * 80)
+ print("πŸŽ‰ Demo completed!")
+ print("\nπŸ’‘ Next steps:")
+ print(" - Try with your own URLs")
+ print(" - Customize the prompts")
+ print(" - Define your own schemas")
+ print(" - Run the Streamlit app: streamlit run scrapegraph_app.py")
+ print("\nπŸ”— Resources:")
+ print(" - GitHub: https://github.com/ScrapeGraphAI/scrapegraph-sdk")
+ print(" - Docs: https://docs.scrapegraphai.com")
+ print("=" * 80)
+
+
+if __name__ == "__main__":
+ main()
+