🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Python aiai-scrapingautomationcrawlercrawlingcrawling-pythondatadata-extractionmcpmcp-serverplaywrightpythonscrapingselectorsstealthweb-scraperweb-scrapingweb-scraping-pythonwebscrapingxpath
docs: simplify the architecture mermaid
Karim shoaird0b2cec84991
diff --git a/docs/spiders/architecture.md b/docs/spiders/architecture.mdindex 70fba8c..412243b 100644--- a/docs/spiders/architecture.md+++ b/docs/spiders/architecture.md@@ -15,57 +15,23 @@ The diagram below shows how data flows through the spider system when a crawl is```mermaidgraph TB- subgraph Spider["🕷️ Spider"]- direction TB- SR["start_requests()"]- Parse["parse() / callbacks"]- Hooks["Lifecycle hooks<br/><small>on_start · on_close · on_error<br/>is_blocked · on_scraped_item</small>"]- end-- subgraph Engine["⚙️ Crawler Engine"]- direction TB- Loop["Crawl loop"]- Conc["Concurrency control<br/><small>Global limiter · Per-domain limiter<br/>Download delay</small>"]- BlockDet["Blocked request<br/>detection & retry"]- end-- subgraph Scheduler["📋 Scheduler"]- PQ["Priority queue"]- Dedup["Fingerprint<br/>deduplication"]- end-- subgraph SessionMgr["🔌 Session Manager"]- direction TB- Router["Request router<br/><small>Routes by session ID</small>"]- subgraph Sessions["Sessions"]- direction LR- HTTP["FetcherSession<br/><small>HTTP requests</small>"]- Dynamic["AsyncDynamicSession<br/><small>Browser automation</small>"]- Stealth["AsyncStealthySession<br/><small>Anti-bot bypass</small>"]- end- end-- subgraph Output["📦 Output"]- Items["ItemList<br/><small>to_json() · to_jsonl()</small>"]- Stream["Streaming<br/><small>async for item in spider.stream()</small>"]- Stats["CrawlStats"]- end-- Checkpoint["💾 Checkpoint<br/><small>Pause/Resume</small>"]-- %% Data flow- SR -- "1. Initial<br/>Requests" --> Scheduler- Scheduler -- "2. Next<br/>Request" --> Engine+ Spider["Spider"]+ Scheduler["Scheduler"]+ Engine["Crawler Engine"]+ SessionMgr["Session Manager"]+ Output["Output"]+ Checkpoint["Checkpoint"]++ Spider -- "1. Initial Requests" --> Scheduler+ Scheduler -- "2. Next Request" --> EngineEngine -- "3. Fetch" --> SessionMgr- Router --> SessionsSessionMgr -- "4. Response" --> Engine- Engine -- "5. Response" --> Parse- Parse -- "6a. New Requests" --> Scheduler- Parse -- "6b. Items (dict)" --> Output- Engine -. "Periodic save" .-> Checkpoint+ Engine -- "5. Response to callback" --> Spider+ Spider -- "6. New Requests" --> Scheduler+ Spider -- "7. Items" --> Output+ Engine -. "Save" .-> CheckpointCheckpoint -. "Resume" .-> Scheduler- %% StylingclassDef spiderClass fill:#7c4dff,stroke:#333,color:#fffclassDef engineClass fill:#00897b,stroke:#333,color:#fffclassDef schedulerClass fill:#1565c0,stroke:#333,color:#fff