🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Python aiai-scrapingautomationcrawlercrawlingcrawling-pythondatadata-extractionmcpmcp-serverplaywrightpythonscrapingselectorsstealthweb-scraperweb-scrapingweb-scraping-pythonwebscrapingxpath
docs: add a page for the agent skill and restructure the website sections to clearer
Karim shoair6815ca4f4fde
diff --git a/docs/ai/agent-skill.md b/docs/ai/agent-skill.mdnew file mode 100644index 0000000..c108539--- /dev/null+++ b/docs/ai/agent-skill.md@@ -0,0 +1,37 @@+# Scrapling Agent Skill++While the [MCP server](mcp-server.md) gives AI chatbots tools to scrape websites for you, the **Scrapling Agent Skill** teaches coding agents the library itself, so the code they write with Scrapling matches the current API instead of guessing from outdated training data.++The skill follows the [AgentSkills](https://agentskills.io/specification) specification, so it's readable by [Claude Code](https://claude.com/product/claude-code), [OpenClaw](https://github.com/openclaw/openclaw), and other agentic tools. It packages almost all of this documentation website's content as Markdown references that the agent loads on demand, together with ready-to-run examples, so it can answer most Scrapling questions and write correct scraping code without leaving your editor.++The skill lives in the [agent-skill](https://github.com/D4Vinci/Scrapling/tree/main/agent-skill) directory of the repository under the name `scrapling-official`, and its version always tracks the library version.++## Installation++### Skills.sh++Install the skill from the repository with the [skills.sh](https://skills.sh) CLI:+```bash+npx skills add D4Vinci/Scrapling --skill scrapling-official+```+The CLI detects the agents you have installed and adds the skill to them.++### Clawhub++If you use OpenClaw, you can install the skill through [Clawhub](https://docs.openclaw.ai/tools/clawhub):+```bash+clawhub install scrapling-official+```+Or install it from its [Clawhub page](https://clawhub.ai/D4Vinci/scrapling-official).++### Manual download++Download the skill's ZIP file [directly from the repository](https://github.com/D4Vinci/Scrapling/raw/refs/heads/main/agent-skill/Scrapling-Skill.zip) and extract it into your agent's skills directory (for Claude Code, that's `~/.claude/skills/`).++## What's inside++- **`SKILL.md`**: The entry point the agent reads first. It covers setup, which fetcher to choose, and the safety rules the agent must follow, like always passing `--ai-targeted` to the CLI commands for prompt injection protection.+- **`references/`**: Markdown mirrors of the documentation pages (parsing, fetching, spiders, CLI, MCP server, and integrations), loaded on demand so they don't consume the agent's context until needed.+- **`examples/`**: Ready-to-run scripts the agent can copy from.++We tested the skill on OpenClaw and Claude Code. If you face any issues with it, open a [ticket](https://github.com/D4Vinci/Scrapling/issues/new/choose) or reach out on our [Discord server](https://discord.gg/EMgGbDceNQ).diff --git a/docs/assets/star-history.png b/docs/assets/star-history.pngindex dc11a53..e911617 100644Binary files a/docs/assets/star-history.png and b/docs/assets/star-history.png differdiff --git a/docs/integrations/scrapy.md b/docs/integrations/scrapy.mdindex 9036626..1ecf357 100644--- a/docs/integrations/scrapy.md+++ b/docs/integrations/scrapy.md@@ -1,4 +1,4 @@-# Scrapy+# Integrating Scrapling into ScrapyIf you have an existing Scrapy project, you don't need to rewrite it to enjoy Scrapling's parsing API. The Scrapy integration converts Scrapy responses to Scrapling [Response](../fetching/choosing.md#response-object) objects right inside your spider callbacks, so Scrapy keeps handling the crawling while Scrapling handles the parsing.diff --git a/zensical.toml b/zensical.tomlindex 550bb11..2553502 100644--- a/zensical.toml+++ b/zensical.toml@@ -44,14 +44,17 @@ nav = [{"Interactive shell" = "cli/interactive-shell.md"},{"Extract commands" = "cli/extract-commands.md"}]},- {Integrations = [- {"AI MCP server" = "ai/mcp-server.md"},- {Scrapy = "integrations/scrapy.md"}+ {"Using with AI" = [+ {"MCP server" = "ai/mcp-server.md"},+ {"Agent skill" = "ai/agent-skill.md"}+ ]},+ {"Integrations and migrations" = [+ {"Integrating into Scrapy" = "integrations/scrapy.md"},+ {"Migrating from BeautifulSoup" = "tutorials/migrating_from_beautifulsoup.md"}]}]},{Tutorials = [- {"A Free Alternative to AI for Robust Web Scraping" = "tutorials/replacing_ai.md"},- {"Migrating from BeautifulSoup" = "tutorials/migrating_from_beautifulsoup.md"}+ {"A Free Alternative to AI for Robust Web Scraping" = "tutorials/replacing_ai.md"}]},{Development = [{"API Reference" = [