Skip to content

🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

docs: add a new page for the Shopify template

Karim shoairb1c250fb2928
diff --git a/agent-skill/Scrapling-Skill.zip b/agent-skill/Scrapling-Skill.zip
index fb35f39..5a50283 100644
Binary files a/agent-skill/Scrapling-Skill.zip and b/agent-skill/Scrapling-Skill.zip differ
diff --git a/agent-skill/Scrapling-Skill/SKILL.md b/agent-skill/Scrapling-Skill/SKILL.md
index 04f77c9..980a319 100644
--- a/agent-skill/Scrapling-Skill/SKILL.md
+++ b/agent-skill/Scrapling-Skill/SKILL.md
@@ -325,6 +325,8 @@ class BlogCrawler(CrawlSpider):
```
For sitemap-driven crawls, use `SitemapSpider` with the same `rules()` API. It fetches `sitemap_urls`, descends into sitemap indexes, and dispatches each URL through your rules. Put a `robots.txt` URL directly in `sitemap_urls` and the spider extracts each `Sitemap:` directive from it automatically. See `references/spiders/generic-templates.md` for the full reference, including `LinkExtractor`'s allow/deny/restrict_css/canonicalize options.
+For Shopify-powered stores, subclass `ShopifySpider` and set `target_website` to the store's domain; it extracts every product variant through Shopify's JSON API without touching the HTML. See `references/spiders/platform-templates.md`.
+
### Advanced Parsing & Navigation
```python
from scrapling.fetchers import Fetcher
diff --git a/agent-skill/Scrapling-Skill/references/spiders/platform-templates.md b/agent-skill/Scrapling-Skill/references/spiders/platform-templates.md
new file mode 100644
index 0000000..8f67f67
--- /dev/null
+++ b/agent-skill/Scrapling-Skill/references/spiders/platform-templates.md
@@ -0,0 +1,54 @@
+# Platform Spider Templates
+
+Generic templates cover crawl *patterns* (follow links, walk sitemaps). Platform templates cover *platforms*: site builders that expose the same machine-readable structure across many independent websites, so the spider already knows where the data lives. You only point it at a domain.
+
+## ShopifySpider
+
+`ShopifySpider` extracts every product from any Shopify-powered store through Shopify's JSON API, without touching the website's HTML.
+
+```python
+from scrapling.spiders import ShopifySpider
+
+class MyStore(ShopifySpider):
+ target_website = "example.com"
+
+result = MyStore().start()
+print(result.items[0])
+```
+
+Set `target_website` to the store's domain. When it's empty, the spider falls back to the first entry in `start_urls`, then `allowed_domains`, and raises `ValueError` if all three are empty. Full URLs are normalized to their domain automatically.
+
+### How it works
+
+1. Pages through `https://<store>/collections.json` (250 collections per page, the platform's cap).
+2. For every collection that reports products, pages through `/collections/<handle>/products.json`.
+3. Yields one item per product variant, deduplicating variants that appear in multiple collections.
+
+### Item fields
+
+| Field | Source |
+|---------------|---------------------------------------------------------------------------|
+| `name` | Product title, plus the variant title when it isn't the default one |
+| `price` | Variant price (a string, exactly as Shopify returns it) |
+| `category` | Collection handle, title-cased (`summer-sale` becomes `Summer Sale`) |
+| `brand` | Product vendor |
+| `identifier` | Variant id |
+| `sku` | Variant SKU, or `""` when the store doesn't set one |
+| `stock` | `None` when the variant is available, `0` when it's out of stock |
+| `image_url` | First product image, or `""` |
+| `url` | The product's page inside the collection |
+| `description` | Product `body_html` with the HTML tags stripped |
+| `old_price` | `compare_at_price` when it's a real pre-sale price, else `""` |
+| `barcode` | Variant barcode, or `""` (most stores don't expose it in these endpoints) |
+
+These fields are not mandatory; override the `_process_product()` method in your subclass to change the item structure or extract different fields from the product data.
+
+### Notes and limits
+
+- The JSON endpoints only expose products published to the online-store channel, so a collection's `products_count` can be higher than what's actually retrievable. The spider treats it as "nonzero means fetch this collection", never as an expected total.
+- This template most likely won't work on stores behind extra protections or password-protected ones, and if it can work at all, it will need a lot of overrides from your side.
+- Everything from `Spider` still applies: concurrency settings, delays, robots.txt compliance, checkpoints, and the lifecycle hooks.
+
+### What qualifies as a platform template
+
+Platform templates are accepted for platforms that expose a uniform, machine-readable structure across many independent websites, like Shopify does. Spiders for one specific website don't belong in the library, no matter how popular the website is.
diff --git a/docs/spiders/platform-templates.md b/docs/spiders/platform-templates.md
new file mode 100644
index 0000000..8f67f67
--- /dev/null
+++ b/docs/spiders/platform-templates.md
@@ -0,0 +1,54 @@
+# Platform Spider Templates
+
+Generic templates cover crawl *patterns* (follow links, walk sitemaps). Platform templates cover *platforms*: site builders that expose the same machine-readable structure across many independent websites, so the spider already knows where the data lives. You only point it at a domain.
+
+## ShopifySpider
+
+`ShopifySpider` extracts every product from any Shopify-powered store through Shopify's JSON API, without touching the website's HTML.
+
+```python
+from scrapling.spiders import ShopifySpider
+
+class MyStore(ShopifySpider):
+ target_website = "example.com"
+
+result = MyStore().start()
+print(result.items[0])
+```
+
+Set `target_website` to the store's domain. When it's empty, the spider falls back to the first entry in `start_urls`, then `allowed_domains`, and raises `ValueError` if all three are empty. Full URLs are normalized to their domain automatically.
+
+### How it works
+
+1. Pages through `https://<store>/collections.json` (250 collections per page, the platform's cap).
+2. For every collection that reports products, pages through `/collections/<handle>/products.json`.
+3. Yields one item per product variant, deduplicating variants that appear in multiple collections.
+
+### Item fields
+
+| Field | Source |
+|---------------|---------------------------------------------------------------------------|
+| `name` | Product title, plus the variant title when it isn't the default one |
+| `price` | Variant price (a string, exactly as Shopify returns it) |
+| `category` | Collection handle, title-cased (`summer-sale` becomes `Summer Sale`) |
+| `brand` | Product vendor |
+| `identifier` | Variant id |
+| `sku` | Variant SKU, or `""` when the store doesn't set one |
+| `stock` | `None` when the variant is available, `0` when it's out of stock |
+| `image_url` | First product image, or `""` |
+| `url` | The product's page inside the collection |
+| `description` | Product `body_html` with the HTML tags stripped |
+| `old_price` | `compare_at_price` when it's a real pre-sale price, else `""` |
+| `barcode` | Variant barcode, or `""` (most stores don't expose it in these endpoints) |
+
+These fields are not mandatory; override the `_process_product()` method in your subclass to change the item structure or extract different fields from the product data.
+
+### Notes and limits
+
+- The JSON endpoints only expose products published to the online-store channel, so a collection's `products_count` can be higher than what's actually retrievable. The spider treats it as "nonzero means fetch this collection", never as an expected total.
+- This template most likely won't work on stores behind extra protections or password-protected ones, and if it can work at all, it will need a lot of overrides from your side.
+- Everything from `Spider` still applies: concurrency settings, delays, robots.txt compliance, checkpoints, and the lifecycle hooks.
+
+### What qualifies as a platform template
+
+Platform templates are accepted for platforms that expose a uniform, machine-readable structure across many independent websites, like Shopify does. Spiders for one specific website don't belong in the library, no matter how popular the website is.
diff --git a/zensical.toml b/zensical.toml
index 13550cd..550bb11 100644
--- a/zensical.toml
+++ b/zensical.toml
@@ -36,6 +36,7 @@ nav = [
{"Sessions" = "spiders/sessions.md"},
{"Proxy management & Blocking" = "spiders/proxy-blocking.md"},
{"Generic crawlers" = "spiders/generic-templates.md"},
+ {"Platform crawlers" = "spiders/platform-templates.md"},
{"Advanced features" = "spiders/advanced.md"}
]},
{"Command Line Interface" = [