Skip to content

🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

docs: add a page for the agent skill and restructure the website sections to clearer

Karim shoair6815ca4f4fde
diff --git a/docs/ai/agent-skill.md b/docs/ai/agent-skill.md
new file mode 100644
index 0000000..c108539
--- /dev/null
+++ b/docs/ai/agent-skill.md
@@ -0,0 +1,37 @@
+# Scrapling Agent Skill
+
+While the [MCP server](mcp-server.md) gives AI chatbots tools to scrape websites for you, the **Scrapling Agent Skill** teaches coding agents the library itself, so the code they write with Scrapling matches the current API instead of guessing from outdated training data.
+
+The skill follows the [AgentSkills](https://agentskills.io/specification) specification, so it's readable by [Claude Code](https://claude.com/product/claude-code), [OpenClaw](https://github.com/openclaw/openclaw), and other agentic tools. It packages almost all of this documentation website's content as Markdown references that the agent loads on demand, together with ready-to-run examples, so it can answer most Scrapling questions and write correct scraping code without leaving your editor.
+
+The skill lives in the [agent-skill](https://github.com/D4Vinci/Scrapling/tree/main/agent-skill) directory of the repository under the name `scrapling-official`, and its version always tracks the library version.
+
+## Installation
+
+### Skills.sh
+
+Install the skill from the repository with the [skills.sh](https://skills.sh) CLI:
+```bash
+npx skills add D4Vinci/Scrapling --skill scrapling-official
+```
+The CLI detects the agents you have installed and adds the skill to them.
+
+### Clawhub
+
+If you use OpenClaw, you can install the skill through [Clawhub](https://docs.openclaw.ai/tools/clawhub):
+```bash
+clawhub install scrapling-official
+```
+Or install it from its [Clawhub page](https://clawhub.ai/D4Vinci/scrapling-official).
+
+### Manual download
+
+Download the skill's ZIP file [directly from the repository](https://github.com/D4Vinci/Scrapling/raw/refs/heads/main/agent-skill/Scrapling-Skill.zip) and extract it into your agent's skills directory (for Claude Code, that's `~/.claude/skills/`).
+
+## What's inside
+
+- **`SKILL.md`**: The entry point the agent reads first. It covers setup, which fetcher to choose, and the safety rules the agent must follow, like always passing `--ai-targeted` to the CLI commands for prompt injection protection.
+- **`references/`**: Markdown mirrors of the documentation pages (parsing, fetching, spiders, CLI, MCP server, and integrations), loaded on demand so they don't consume the agent's context until needed.
+- **`examples/`**: Ready-to-run scripts the agent can copy from.
+
+We tested the skill on OpenClaw and Claude Code. If you face any issues with it, open a [ticket](https://github.com/D4Vinci/Scrapling/issues/new/choose) or reach out on our [Discord server](https://discord.gg/EMgGbDceNQ).
diff --git a/docs/assets/star-history.png b/docs/assets/star-history.png
index dc11a53..e911617 100644
Binary files a/docs/assets/star-history.png and b/docs/assets/star-history.png differ
diff --git a/docs/integrations/scrapy.md b/docs/integrations/scrapy.md
index 9036626..1ecf357 100644
--- a/docs/integrations/scrapy.md
+++ b/docs/integrations/scrapy.md
@@ -1,4 +1,4 @@
-# Scrapy
+# Integrating Scrapling into Scrapy
If you have an existing Scrapy project, you don't need to rewrite it to enjoy Scrapling's parsing API. The Scrapy integration converts Scrapy responses to Scrapling [Response](../fetching/choosing.md#response-object) objects right inside your spider callbacks, so Scrapy keeps handling the crawling while Scrapling handles the parsing.
diff --git a/zensical.toml b/zensical.toml
index 550bb11..2553502 100644
--- a/zensical.toml
+++ b/zensical.toml
@@ -44,14 +44,17 @@ nav = [
{"Interactive shell" = "cli/interactive-shell.md"},
{"Extract commands" = "cli/extract-commands.md"}
]},
- {Integrations = [
- {"AI MCP server" = "ai/mcp-server.md"},
- {Scrapy = "integrations/scrapy.md"}
+ {"Using with AI" = [
+ {"MCP server" = "ai/mcp-server.md"},
+ {"Agent skill" = "ai/agent-skill.md"}
+ ]},
+ {"Integrations and migrations" = [
+ {"Integrating into Scrapy" = "integrations/scrapy.md"},
+ {"Migrating from BeautifulSoup" = "tutorials/migrating_from_beautifulsoup.md"}
]}
]},
{Tutorials = [
- {"A Free Alternative to AI for Robust Web Scraping" = "tutorials/replacing_ai.md"},
- {"Migrating from BeautifulSoup" = "tutorials/migrating_from_beautifulsoup.md"}
+ {"A Free Alternative to AI for Robust Web Scraping" = "tutorials/replacing_ai.md"}
]},
{Development = [
{"API Reference" = [