Skip to content

🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

docs: add new api references

Karim shoairdddb179c50a2
diff --git a/docs/api-reference/proxy-rotation.md b/docs/api-reference/proxy-rotation.md
new file mode 100644
index 0000000..61a8fad
--- /dev/null
+++ b/docs/api-reference/proxy-rotation.md
@@ -0,0 +1,18 @@
+---
+search:
+ exclude: true
+---
+
+# Proxy Rotation
+
+The `ProxyRotator` class provides thread-safe proxy rotation for any fetcher or session.
+
+You can import it directly like below:
+
+```python
+from scrapling.fetchers import ProxyRotator
+```
+
+## ::: scrapling.engines.toolbelt.proxy_rotation.ProxyRotator
+ handler: python
+ :docstring:
diff --git a/docs/api-reference/response.md b/docs/api-reference/response.md
new file mode 100644
index 0000000..26f6895
--- /dev/null
+++ b/docs/api-reference/response.md
@@ -0,0 +1,18 @@
+---
+search:
+ exclude: true
+---
+
+# Response Class
+
+The `Response` class wraps HTTP responses returned by all fetchers, providing access to status, headers, body, cookies, and a `Selector` for parsing.
+
+You can import the `Response` class like below:
+
+```python
+from scrapling.engines.toolbelt.custom import Response
+```
+
+## ::: scrapling.engines.toolbelt.custom.Response
+ handler: python
+ :docstring:
diff --git a/docs/api-reference/spiders.md b/docs/api-reference/spiders.md
new file mode 100644
index 0000000..0a69abb
--- /dev/null
+++ b/docs/api-reference/spiders.md
@@ -0,0 +1,42 @@
+---
+search:
+ exclude: true
+---
+
+# Spider Classes
+
+Here's the reference information for the spider framework classes' parameters, attributes, and methods.
+
+You can import them directly like below:
+
+```python
+from scrapling.spiders import Spider, Request, CrawlResult, SessionManager, Response
+```
+
+## ::: scrapling.spiders.Spider
+ handler: python
+ :docstring:
+
+## ::: scrapling.spiders.Request
+ handler: python
+ :docstring:
+
+## Result Classes
+
+## ::: scrapling.spiders.result.CrawlResult
+ handler: python
+ :docstring:
+
+## ::: scrapling.spiders.result.CrawlStats
+ handler: python
+ :docstring:
+
+## ::: scrapling.spiders.result.ItemList
+ handler: python
+ :docstring:
+
+## Session Management
+
+## ::: scrapling.spiders.session.SessionManager
+ handler: python
+ :docstring:
diff --git a/zensical.toml b/zensical.toml
index b19a6ab..231d82f 100644
--- a/zensical.toml
+++ b/zensical.toml
@@ -54,7 +54,10 @@ nav = [
{Selector = "api-reference/selector.md"},
{Fetchers = "api-reference/fetchers.md"},
{"MCP Server" = "api-reference/mcp-server.md"},
- {"Custom Types" = "api-reference/custom-types.md"}
+ {"Custom Types" = "api-reference/custom-types.md"},
+ {Response = "api-reference/response.md"},
+ {Spiders = "api-reference/spiders.md"},
+ {"Proxy Rotation" = "api-reference/proxy-rotation.md"}
]},
{"Writing your retrieval system" = "development/adaptive_storage_system.md"},
{"Using Scrapling's custom types" = "development/scrapling_custom_types.md"}