Skip to content

🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

feat(spiders): Add exporters to CSV/XML

Karim shoair0a7fec8f864f
diff --git a/README.md b/README.md
index cec3704..63a453f 100644
--- a/README.md
+++ b/README.md
@@ -216,7 +216,7 @@ MySpider().start()
- 🧪 **Development Mode**: Cache responses to disk on the first run and replay them on subsequent runs - iterate on your `parse()` logic without re-hitting the target servers.
- 🧩 **Ready-made Spider Templates**: Skip the boilerplate with `CrawlSpider` for rule-based link following, `SitemapSpider` for sitemap/robots.txt-driven crawls, and `ShopifySpider` to pull every product out of any Shopify store through its JSON API, one item per variant.
- 🔗 **Link Extraction**: A standalone `LinkExtractor` primitive with allow/deny patterns, domain filters, CSS/XPath scoping, extension filtering, and canonicalization - use it inside the templates or on its own.
-- 📦 **Built-in Export**: Export results through hooks and your own pipeline or the built-in JSON/JSONL with `result.items.to_json()` / `result.items.to_jsonl()` respectively.
+- 📦 **Built-in Export**: Export results through hooks and your own pipeline or the built-in JSON/JSONL/CSV/XML exporters with `result.items.to_json()`, `to_jsonl()`, `to_csv()`, and `to_xml()`.
### Advanced Websites Fetching with Session Support
- **HTTP Requests**: Fast and stealthy HTTP requests with the `Fetcher` class. Can impersonate browsers' TLS fingerprint, headers, and use HTTP/3.
diff --git a/agent-skill/Scrapling-Skill.zip b/agent-skill/Scrapling-Skill.zip
index 3cb75e7..b82cc6f 100644
Binary files a/agent-skill/Scrapling-Skill.zip and b/agent-skill/Scrapling-Skill.zip differ
diff --git a/agent-skill/Scrapling-Skill/references/spiders/architecture.md b/agent-skill/Scrapling-Skill/references/spiders/architecture.md
index 54186a1..68ea7b8 100644
--- a/agent-skill/Scrapling-Skill/references/spiders/architecture.md
+++ b/agent-skill/Scrapling-Skill/references/spiders/architecture.md
@@ -66,7 +66,7 @@ An optional cache that, when development mode is enabled, stores every fetched r
### Output
-Scraped items are collected in an `ItemList` (a list subclass with `to_json()` and `to_jsonl()` export methods). Crawl statistics are tracked in a `CrawlStats` dataclass which contains a lot of useful info.
+Scraped items are collected in an `ItemList` (a list subclass with `to_json()`, `to_jsonl()`, `to_csv()`, and `to_xml()` export methods). Crawl statistics are tracked in a `CrawlStats` dataclass which contains a lot of useful info.
## Comparison with Scrapy
@@ -88,7 +88,7 @@ If you're coming from Scrapy, here's how Scrapling's spider system maps:
| Domain filtering | `allowed_domains` | `allowed_domains` |
| Robots.txt | `ROBOTSTXT_OBEY` setting | `robots_txt_obey` class attribute |
| Pause/Resume | `JOBDIR` setting | `crawldir` constructor argument |
-| Export | Feed exports | `result.items.to_json()` / `to_jsonl()` or custom through hooks |
+| Export | Feed exports | `result.items.to_json()` / `to_jsonl()` / `to_csv()` / `to_xml()` or custom through hooks |
| Running | `scrapy crawl spider_name` | `MySpider().start()` |
| Streaming | N/A | `async for item in spider.stream()` |
| Multi-session | N/A | Multiple sessions with different types per spider |
\ No newline at end of file
diff --git a/agent-skill/Scrapling-Skill/references/spiders/getting-started.md b/agent-skill/Scrapling-Skill/references/spiders/getting-started.md
index e553b7b..f18073e 100644
--- a/agent-skill/Scrapling-Skill/references/spiders/getting-started.md
+++ b/agent-skill/Scrapling-Skill/references/spiders/getting-started.md
@@ -113,9 +113,17 @@ result.items.to_json("quotes.json", indent=True)
# Export as JSON Lines (one JSON object per line)
result.items.to_jsonl("quotes.jsonl")
+
+# Export as CSV or XML
+result.items.to_csv("quotes.csv")
+result.items.to_xml("quotes.xml")
```
-Both methods create parent directories automatically if they don't exist.
+All of them create parent directories automatically if they don't exist.
+
+`to_csv()` writes a column for every key found across your items, so items that don't all share the same keys are still exported with the missing cells left empty. Pass `fields=[...]` to pick the columns and their order yourself, or `delimiter="\t"` for a TSV file. `to_xml()` wraps each item in an `<item>` element inside an `<items>` root, both renameable through `root_tag` and `item_tag`.
+
+In both formats, values that aren't simple scalars (a nested dictionary or a list) are written as JSON so nothing is silently dropped.
## Filtering Domains
diff --git a/docs/README_AR.md b/docs/README_AR.md
index 1224119..7a50a10 100644
--- a/docs/README_AR.md
+++ b/docs/README_AR.md
@@ -212,7 +212,7 @@ MySpider().start()
- 🧪 **وضع التطوير**: تخزين الاستجابات على القرص في التشغيل الأول وإعادة تشغيلها في التشغيلات اللاحقة - كرّر العمل على منطق `parse()` دون الحاجة لإرسال طلبات جديدة إلى الخوادم المستهدفة.
- 🧩 **قوالب Spider جاهزة**: تخطَّ الكود المتكرر مع `CrawlSpider` لتتبع الروابط بقواعد، و`SitemapSpider` للزحف المعتمد على sitemap/robots.txt، و`ShopifySpider` لسحب كل منتج من أي متجر Shopify عبر واجهة JSON الخاصة به، مع عنصر لكل variant.
- 🔗 **استخراج الروابط**: أداة `LinkExtractor` مستقلة مع أنماط allow/deny، وفلاتر النطاقات، وتحديد النطاق عبر CSS/XPath، وتصفية الامتدادات، وتوحيد الروابط - استخدمها داخل القوالب أو بمفردها.
-- 📦 **تصدير مدمج**: صدّر النتائج عبر الخطافات وخط الأنابيب الخاص بك أو JSON/JSONL المدمج مع `result.items.to_json()` / `result.items.to_jsonl()` على التوالي.
+- 📦 **تصدير مدمج**: صدّر النتائج عبر الخطافات وخط الأنابيب الخاص بك أو المصدّرات المدمجة JSON/JSONL/CSV/XML عبر `result.items.to_json()` و`to_jsonl()` و`to_csv()` و`to_xml()`.
### جلب متقدم للمواقع مع دعم الجلسات
- **طلبات HTTP**: طلبات HTTP سريعة وخفية مع فئة `Fetcher`. يمكنها تقليد بصمة TLS للمتصفح والرؤوس واستخدام HTTP/3.
diff --git a/docs/README_CN.md b/docs/README_CN.md
index 7b525b7..deb4840 100644
--- a/docs/README_CN.md
+++ b/docs/README_CN.md
@@ -212,7 +212,7 @@ MySpider().start()
- 🧪 **开发模式**:首次运行时将响应缓存到磁盘,后续运行时直接回放 - 在不重新请求目标服务器的情况下迭代你的 `parse()` 逻辑。
- 🧩 **现成的 Spider 模板**:跳过样板代码--使用 `CrawlSpider` 基于规则跟踪链接,`SitemapSpider` 基于 sitemap/robots.txt 爬取,以及 `ShopifySpider` 通过 JSON API 抓取任意 Shopify 商店的全部商品,每个变体一条数据。
- 🔗 **链接提取**:独立的 `LinkExtractor` 组件,支持 allow/deny 模式、域名过滤、CSS/XPath 范围限定、扩展名过滤和链接规范化--可在模板中使用,也可单独使用。
-- 📦 **内置导出**:通过钩子和您自己的管道导出结果,或使用内置的 JSON/JSONL,分别通过 `result.items.to_json()`/`result.items.to_jsonl()`。
+- 📦 **内置导出**:通过钩子和您自己的管道导出结果,或使用内置的 JSON/JSONL/CSV/XML 导出器:`result.items.to_json()`、`to_jsonl()`、`to_csv()` 和 `to_xml()`。
### 支持 Session 的高级网站获取
- **HTTP 请求**:使用 `Fetcher` 类进行快速和隐秘的 HTTP 请求。可以模拟浏览器的 TLS fingerprint、标头并使用 HTTP/3。
diff --git a/docs/README_DE.md b/docs/README_DE.md
index 27fb549..ec3038a 100644
--- a/docs/README_DE.md
+++ b/docs/README_DE.md
@@ -212,7 +212,7 @@ MySpider().start()
- 🧪 **Entwicklungsmodus**: Antworten beim ersten Lauf auf der Festplatte zwischenspeichern und bei weiteren Läufen erneut abspielen - iterieren Sie an Ihrer `parse()`-Logik, ohne die Zielserver erneut abzufragen.
- 🧩 **Fertige Spider-Vorlagen**: Sparen Sie sich den Boilerplate-Code mit `CrawlSpider` für regelbasiertes Link-Following, `SitemapSpider` für Sitemap-/robots.txt-gesteuerte Crawls und `ShopifySpider`, um über die JSON-API jedes Produkt aus einem beliebigen Shopify-Shop zu ziehen -- ein Item pro Variante.
- 🔗 **Link-Extraktion**: Ein eigenständiges `LinkExtractor`-Primitiv mit Allow-/Deny-Mustern, Domain-Filtern, CSS-/XPath-Eingrenzung, Erweiterungsfilterung und Kanonisierung -- nutzbar innerhalb der Vorlagen oder für sich allein.
-- 📦 **Integrierter Export**: Ergebnisse über Hooks und Ihre eigene Pipeline oder den integrierten JSON/JSONL-Export mit `result.items.to_json()` / `result.items.to_jsonl()` exportieren.
+- 📦 **Integrierter Export**: Ergebnisse über Hooks und Ihre eigene Pipeline oder die integrierten JSON/JSONL/CSV/XML-Exporte mit `result.items.to_json()`, `to_jsonl()`, `to_csv()` und `to_xml()` exportieren.
### Erweitertes Website-Abrufen mit Session-Unterstützung
- **HTTP-Anfragen**: Schnelle und heimliche HTTP-Anfragen mit der `Fetcher`-Klasse. Kann Browser-TLS-Fingerprints und Header imitieren und HTTP/3 verwenden.
diff --git a/docs/README_ES.md b/docs/README_ES.md
index e3f475e..632654e 100644
--- a/docs/README_ES.md
+++ b/docs/README_ES.md
@@ -212,7 +212,7 @@ MySpider().start()
- 🧪 **Modo de Desarrollo**: Almacena las respuestas en disco en la primera ejecución y las reproduce en ejecuciones posteriores - itera sobre tu lógica de `parse()` sin volver a consultar los servidores objetivo.
- 🧩 **Plantillas de Spider Listas para Usar**: Olvídate del código repetitivo con `CrawlSpider` para seguir enlaces por reglas, `SitemapSpider` para rastreos guiados por sitemap/robots.txt, y `ShopifySpider` para extraer todos los productos de cualquier tienda Shopify a través de su API JSON, un ítem por variante.
- 🔗 **Extracción de Enlaces**: Un primitivo `LinkExtractor` independiente con patrones allow/deny, filtros de dominio, delimitación por CSS/XPath, filtrado de extensiones y canonicalización - úsalo dentro de las plantillas o por su cuenta.
-- 📦 **Exportación Integrada**: Exporta resultados a través de hooks y tu propio pipeline o el JSON/JSONL integrado con `result.items.to_json()` / `result.items.to_jsonl()` respectivamente.
+- 📦 **Exportación Integrada**: Exporta resultados a través de hooks y tu propio pipeline o los exportadores integrados JSON/JSONL/CSV/XML con `result.items.to_json()`, `to_jsonl()`, `to_csv()` y `to_xml()`.
### Obtención Avanzada de Sitios Web con Soporte de Session
- **Solicitudes HTTP**: Solicitudes HTTP rápidas y sigilosas con la clase `Fetcher`. Puede imitar el fingerprint TLS de los navegadores, encabezados y usar HTTP/3.
diff --git a/docs/README_FR.md b/docs/README_FR.md
index d964a4a..776abae 100644
--- a/docs/README_FR.md
+++ b/docs/README_FR.md
@@ -212,7 +212,7 @@ MySpider().start()
- 🧪 **Mode développement** : Mettez les réponses en cache sur le disque lors de la première exécution et rejouez-les lors des exécutions suivantes - itérez sur votre logique `parse()` sans solliciter à nouveau les serveurs cibles.
- 🧩 **Modèles de Spider prêts à l'emploi** : Évitez le code répétitif avec `CrawlSpider` pour le suivi de liens par règles, `SitemapSpider` pour les crawls pilotés par sitemap/robots.txt, et `ShopifySpider` pour récupérer tous les produits de n'importe quelle boutique Shopify via son API JSON, un item par variante.
- 🔗 **Extraction de liens** : Un primitif `LinkExtractor` autonome avec motifs allow/deny, filtres de domaines, délimitation CSS/XPath, filtrage des extensions et canonicalisation - utilisable dans les modèles ou seul.
-- 📦 **Export intégré** : Exportez les résultats via des hooks et votre propre pipeline ou l'export JSON/JSONL intégré avec `result.items.to_json()` / `result.items.to_jsonl()` respectivement.
+- 📦 **Export intégré** : Exportez les résultats via des hooks et votre propre pipeline ou les exports JSON/JSONL/CSV/XML intégrés avec `result.items.to_json()`, `to_jsonl()`, `to_csv()` et `to_xml()`.
### Récupération avancée de sites web avec support de sessions
- **Requêtes HTTP** : Requêtes HTTP rapides et furtives avec la classe `Fetcher`. Peut imiter l'empreinte TLS des navigateurs, les headers et utiliser HTTP/3.
diff --git a/docs/README_JP.md b/docs/README_JP.md
index dcd6071..4d317cd 100644
--- a/docs/README_JP.md
+++ b/docs/README_JP.md
@@ -212,7 +212,7 @@ MySpider().start()
- 🧪 **開発モード**:初回実行時にレスポンスをディスクにキャッシュし、以降の実行ではそれを再生 - ターゲットサーバーに再リクエストすることなく `parse()` ロジックを反復開発できます。
- 🧩 **すぐに使える Spider テンプレート**:定型コードは不要です。ルールベースのリンク追跡には `CrawlSpider`、sitemap/robots.txt を起点としたクロールには `SitemapSpider`、そして `ShopifySpider` を使えば任意の Shopify ストアの全商品を JSON API 経由で取得できます(バリエーションごとに 1 アイテム)。
- 🔗 **リンク抽出**:allow/deny パターン、ドメインフィルタ、CSS/XPath による範囲指定、拡張子フィルタ、正規化に対応した独立した `LinkExtractor` を提供。テンプレート内でも単体でも使用できます。
-- 📦 **組み込みエクスポート**:フックや独自のパイプライン、または組み込みの JSON/JSONL で結果をエクスポート。それぞれ`result.items.to_json()` / `result.items.to_jsonl()`を使用。
+- 📦 **組み込みエクスポート**:フックや独自のパイプライン、または組み込みの JSON/JSONL/CSV/XML で結果をエクスポート。`result.items.to_json()`、`to_jsonl()`、`to_csv()`、`to_xml()` が使えます。
### Session サポート付き高度なウェブサイト取得
- **HTTP リクエスト**:`Fetcher` クラスで高速かつステルスな HTTP リクエスト。ブラウザの TLS fingerprint、ヘッダーを模倣し、HTTP/3 を使用可能。
diff --git a/docs/README_KR.md b/docs/README_KR.md
index 2935e9f..84f4a38 100644
--- a/docs/README_KR.md
+++ b/docs/README_KR.md
@@ -212,7 +212,7 @@ MySpider().start()
- 🧪 **개발 모드**: 첫 실행 시 응답을 디스크에 캐싱하고 이후 실행에서는 캐시된 응답을 재생합니다 - 대상 서버에 다시 요청하지 않고 `parse()` 로직을 반복 개발할 수 있습니다.
- 🧩 **바로 쓰는 Spider 템플릿**: 반복적인 코드는 건너뛰세요. 규칙 기반 링크 추적에는 `CrawlSpider`, sitemap/robots.txt 기반 크롤링에는 `SitemapSpider`, 그리고 `ShopifySpider`로 어떤 Shopify 스토어든 JSON API를 통해 모든 상품을 가져올 수 있습니다(변형별로 항목 하나).
- 🔗 **링크 추출**: allow/deny 패턴, 도메인 필터, CSS/XPath 범위 지정, 확장자 필터링, 정규화를 지원하는 독립형 `LinkExtractor`. 템플릿 안에서도, 단독으로도 사용할 수 있습니다.
-- 📦 **내장 내보내기**: 훅이나 자체 파이프라인, 또는 내장 JSON/JSONL로 결과를 내보냅니다. 각각 `result.items.to_json()` / `result.items.to_jsonl()`을 사용합니다.
+- 📦 **내장 내보내기**: 훅이나 자체 파이프라인, 또는 내장 JSON/JSONL/CSV/XML로 결과를 내보냅니다. `result.items.to_json()`, `to_jsonl()`, `to_csv()`, `to_xml()`을 사용합니다.
### 세션을 지원하는 고급 웹사이트 가져오기
- **HTTP 요청**: `Fetcher` 클래스로 빠르고 은밀한 HTTP 요청. 브라우저의 TLS fingerprint, 헤더를 모방하고, HTTP/3를 사용할 수 있습니다.
diff --git a/docs/README_PT_BR.md b/docs/README_PT_BR.md
index ba8282c..285751d 100644
--- a/docs/README_PT_BR.md
+++ b/docs/README_PT_BR.md
@@ -214,7 +214,7 @@ MySpider().start()
- 🧪 **Modo de Desenvolvimento**: Armazene respostas em disco na primeira execução e reproduza-as nas seguintes - itere sobre sua lógica de `parse()` sem reenviar requisições aos servidores-alvo.
- 🧩 **Templates de Spider Prontos**: Pule o código repetitivo com `CrawlSpider` para seguir links por regras, `SitemapSpider` para crawls guiados por sitemap/robots.txt e `ShopifySpider` para extrair todos os produtos de qualquer loja Shopify através da sua API JSON, um item por variação.
- 🔗 **Extração de Links**: Um primitivo `LinkExtractor` independente com padrões allow/deny, filtros de domínio, delimitação por CSS/XPath, filtragem de extensões e canonicalização - use dentro dos templates ou sozinho.
-- 📦 **Exportação Nativa**: Exporte resultados via hooks, seu próprio pipeline ou JSON/JSONL nativos com `result.items.to_json()` / `result.items.to_jsonl()` respectivamente.
+- 📦 **Exportação Nativa**: Exporte resultados via hooks, seu próprio pipeline ou os exportadores nativos JSON/JSONL/CSV/XML com `result.items.to_json()`, `to_jsonl()`, `to_csv()` e `to_xml()`.
### Busca Avançada de Sites com Suporte a Sessões
- **Requisições HTTP**: Requisições HTTP rápidas e furtivas com a classe `Fetcher`. Pode imitar fingerprint TLS de navegadores, cabeçalhos e usar HTTP/3.
diff --git a/docs/README_RU.md b/docs/README_RU.md
index 4f223c9..57dd334 100644
--- a/docs/README_RU.md
+++ b/docs/README_RU.md
@@ -214,7 +214,7 @@ MySpider().start()
- 🧪 **Режим разработки**: Кэшируйте ответы на диск при первом запуске и воспроизводите их при последующих запусках - итерируйте над логикой `parse()`, не отправляя повторные запросы к целевым серверам.
- 🧩 **Готовые шаблоны Spider**: Обойдитесь без шаблонного кода - `CrawlSpider` для перехода по ссылкам на основе правил, `SitemapSpider` для обхода на основе sitemap/robots.txt и `ShopifySpider`, чтобы вытащить все товары любого магазина Shopify через его JSON API, по одному элементу на вариант.
- 🔗 **Извлечение ссылок**: Отдельный примитив `LinkExtractor` с шаблонами allow/deny, фильтрами доменов, ограничением области через CSS/XPath, фильтрацией расширений и канонизацией - используйте его внутри шаблонов или сам по себе.
-- 📦 **Встроенный экспорт**: Экспортируйте результаты через хуки и собственный конвейер или встроенный JSON/JSONL с `result.items.to_json()` / `result.items.to_jsonl()` соответственно.
+- 📦 **Встроенный экспорт**: Экспортируйте результаты через хуки и собственный конвейер или встроенные экспортеры JSON/JSONL/CSV/XML с `result.items.to_json()`, `to_jsonl()`, `to_csv()` и `to_xml()`.
### Продвинутая загрузка сайтов с поддержкой Session
- **HTTP-запросы**: Быстрые и скрытные HTTP-запросы с классом `Fetcher`. Может имитировать TLS fingerprint браузера, заголовки и использовать HTTP/3.
diff --git a/docs/index.md b/docs/index.md
index b537ef1..a96f155 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -104,7 +104,7 @@ MySpider().start()
- 🧪 **Development Mode**: Cache responses to disk on the first run and replay them on subsequent runs - iterate on your `parse()` logic without re-hitting the target servers.
- 🧩 **Ready-made Spider Templates**: Skip the boilerplate with `CrawlSpider` for rule-based link following, `SitemapSpider` for sitemap/robots.txt-driven crawls, and `ShopifySpider` to pull every product out of any Shopify store through its JSON API, one item per variant.
- 🔗 **Link Extraction**: A standalone `LinkExtractor` primitive with allow/deny patterns, domain filters, CSS/XPath scoping, extension filtering, and canonicalization - use it inside the templates or on its own.
-- 📦 **Built-in Export**: Export results through hooks and your own pipeline or the built-in JSON/JSONL with `result.items.to_json()` / `result.items.to_jsonl()` respectively.
+- 📦 **Built-in Export**: Export results through hooks and your own pipeline or the built-in JSON/JSONL/CSV/XML exporters with `result.items.to_json()`, `to_jsonl()`, `to_csv()`, and `to_xml()`.
### Advanced Websites Fetching with Session Support
- **HTTP Requests**: Fast and stealthy HTTP requests with the `Fetcher` class. Can impersonate browsers' TLS fingerprint, headers, and use HTTP/3.
diff --git a/docs/spiders/architecture.md b/docs/spiders/architecture.md
index dc360b8..d6f6d65 100644
--- a/docs/spiders/architecture.md
+++ b/docs/spiders/architecture.md
@@ -75,7 +75,7 @@ An optional cache that, when development mode is enabled, stores every fetched r
### Output
-Scraped items are collected in an `ItemList` (a list subclass with `to_json()` and `to_jsonl()` export methods). Crawl statistics are tracked in a `CrawlStats` dataclass which contains a lot of useful info.
+Scraped items are collected in an `ItemList` (a list subclass with `to_json()`, `to_jsonl()`, `to_csv()`, and `to_xml()` export methods). Crawl statistics are tracked in a `CrawlStats` dataclass which contains a lot of useful info.
## Comparison with Scrapy
@@ -97,7 +97,7 @@ If you're coming from Scrapy, here's how Scrapling's spider system maps:
| Domain filtering | `allowed_domains` | `allowed_domains` |
| Robots.txt | `ROBOTSTXT_OBEY` setting | `robots_txt_obey` class attribute |
| Pause/Resume | `JOBDIR` setting | `crawldir` constructor argument |
-| Export | Feed exports | `result.items.to_json()` / `to_jsonl()` or custom through hooks |
+| Export | Feed exports | `result.items.to_json()` / `to_jsonl()` / `to_csv()` / `to_xml()` or custom through hooks |
| Running | `scrapy crawl spider_name` | `MySpider().start()` |
| Streaming | N/A | `async for item in spider.stream()` |
| Multi-session | N/A | Multiple sessions with different types per spider |
\ No newline at end of file
diff --git a/docs/spiders/getting-started.md b/docs/spiders/getting-started.md
index 197828f..7f6b523 100644
--- a/docs/spiders/getting-started.md
+++ b/docs/spiders/getting-started.md
@@ -125,9 +125,17 @@ result.items.to_json("quotes.json", indent=True)
# Export as JSON Lines (one JSON object per line)
result.items.to_jsonl("quotes.jsonl")
+
+# Export as CSV or XML
+result.items.to_csv("quotes.csv")
+result.items.to_xml("quotes.xml")
```
-Both methods create parent directories automatically if they don't exist.
+All of them create parent directories automatically if they don't exist.
+
+`to_csv()` writes a column for every key found across your items, so items that don't all share the same keys are still exported with the missing cells left empty. Pass `fields=[...]` to pick the columns and their order yourself, or `delimiter="\t"` for a TSV file. `to_xml()` wraps each item in an `<item>` element inside an `<items>` root, both renameable through `root_tag` and `item_tag`.
+
+In both formats, values that aren't simple scalars (a nested dictionary or a list) are written as JSON so nothing is silently dropped.
## Filtering Domains
diff --git a/scrapling/spiders/result.py b/scrapling/spiders/result.py
index 796bbc2..29bda89 100644
--- a/scrapling/spiders/result.py
+++ b/scrapling/spiders/result.py
@@ -1,10 +1,37 @@
+import csv
+import re
from pathlib import Path
from dataclasses import dataclass, field
+from xml.etree.ElementTree import ( # nosec B405 - we only serialize items, nothing here parses XML
+ Element,
+ ElementTree,
+ SubElement,
+ indent as indent_tree,
+)
import orjson
from scrapling.core.utils import log
-from scrapling.core._types import Any, Iterator, Dict, List, Tuple, Union
+from scrapling.core._types import Any, Iterable, Iterator, Dict, List, Optional, Tuple, Union
+
+# Anything outside these ranges can't be represented in XML, and scraped pages are full of control characters
+_XML_FORBIDDEN_CHARS = re.compile(r"[^\x09\x0a\x0d\x20-퟿-�\U00010000-\U0010ffff]")
+_XML_FORBIDDEN_TAG_CHARS = re.compile(r"[^\w.-]", re.UNICODE)
+
+
+def _stringify(value: Any) -> str:
+ """Turn an item's value into text, serializing containers to JSON so no data is silently dropped."""
+ if value is None:
+ return ""
+ if isinstance(value, (dict, list, tuple)):
+ return orjson.dumps(value, option=orjson.OPT_SERIALIZE_NUMPY).decode()
+ return str(value)
+
+
+def _xml_tag(key: Any) -> str:
+ """Turn an item's key into a usable XML tag name."""
+ tag = _XML_FORBIDDEN_TAG_CHARS.sub("_", str(key))
+ return tag if tag and (tag[0].isalpha() or tag[0] == "_") else f"_{tag}"
class ItemList(list):
@@ -37,6 +64,59 @@ class ItemList(list):
f.write(b"\n")
log.info("Saved %d items to %s", len(self), path)
+ def to_csv(self, path: Union[str, Path], *, fields: Optional[Iterable[str]] = None, delimiter: str = ","):
+ """Export items to a CSV file.
+
+ Items that don't share the same keys are still written, with the missing cells left empty, and any value
+ that isn't a scalar (a nested dictionary or a list) is written as JSON.
+
+ :param path: Path to the output file
+ :param fields: The columns to write, defaulting to every key found in the items, in the order they appeared
+ :param delimiter: The character separating the columns
+ """
+ columns = list(fields) if fields is not None else list({key: None for item in self for key in item})
+
+ file = Path(path)
+ file.parent.mkdir(parents=True, exist_ok=True)
+ with open(file, "w", newline="", encoding="utf-8") as f:
+ writer = csv.DictWriter(f, fieldnames=columns, delimiter=delimiter, extrasaction="ignore")
+ writer.writeheader()
+ for item in self:
+ writer.writerow({column: _stringify(item.get(column)) for column in columns})
+
+ log.info("Saved %d items to %s", len(self), path)
+
+ def to_xml(self, path: Union[str, Path], *, root_tag: str = "items", item_tag: str = "item", indent: bool = True):
+ """Export items to an XML file.
+
+ Each item becomes an element whose children are named after the item's keys. Keys that aren't valid XML
+ names are rewritten and keep the original in a `name` attribute, and any value that isn't a scalar
+ (a nested dictionary or a list) is written as JSON.
+
+ :param path: Path to the output file
+ :param root_tag: The name of the element wrapping all the items
+ :param item_tag: The name of the element wrapping every item
+ :param indent: Pretty-print the file instead of writing it on a single line
+ """
+ root = Element(root_tag)
+ for item in self:
+ element = SubElement(root, item_tag)
+ for key, value in item.items():
+ tag = _xml_tag(key)
+ child = SubElement(element, tag)
+ if tag != str(key):
+ child.set("name", _XML_FORBIDDEN_CHARS.sub("", str(key)))
+ child.text = _XML_FORBIDDEN_CHARS.sub("", _stringify(value))
+
+ tree = ElementTree(root)
+ if indent:
+ indent_tree(tree, space=" ")
+
+ file = Path(path)
+ file.parent.mkdir(parents=True, exist_ok=True)
+ tree.write(file, encoding="utf-8", xml_declaration=True)
+ log.info("Saved %d items to %s", len(self), path)
+
@dataclass
class CrawlStats:
diff --git a/tests/spiders/test_result.py b/tests/spiders/test_result.py
index f59f236..cd8da61 100644
--- a/tests/spiders/test_result.py
+++ b/tests/spiders/test_result.py
@@ -1,14 +1,21 @@
"""Tests for the result module (ItemList, CrawlStats, CrawlResult)."""
+import csv
import json
import tempfile
from pathlib import Path
+from xml.etree import ElementTree # nosec B405 - only used to read back files these tests just wrote
import pytest
from scrapling.spiders.result import ItemList, CrawlStats, CrawlResult
+def _read_xml(path):
+ """Parse a file the test itself just wrote."""
+ return ElementTree.parse(path).getroot() # nosec B314 - trusted local file, not untrusted input
+
+
class TestItemList:
"""Test ItemList functionality."""
@@ -68,6 +75,174 @@ class TestItemList:
# Indented JSON should have newlines
assert "\n" in content
+ def test_to_csv_creates_file(self):
+ """Test to_csv writes a header and one row per item."""
+ items = ItemList()
+ items.append({"name": "first", "price": 10})
+ items.append({"name": "second", "price": 20})
+
+ with tempfile.TemporaryDirectory() as tmpdir:
+ path = Path(tmpdir) / "output.csv"
+ items.to_csv(path)
+
+ assert path.exists()
+
+ rows = list(csv.DictReader(path.open(newline="", encoding="utf-8")))
+ assert len(rows) == 2
+ assert rows[0] == {"name": "first", "price": "10"}
+ assert rows[1]["name"] == "second"
+
+ def test_to_csv_unions_the_keys_of_every_item(self):
+ """Items with different keys must all be written, with the missing cells left empty."""
+ items = ItemList()
+ items.append({"name": "first"})
+ items.append({"price": 20, "name": "second"})
+
+ with tempfile.TemporaryDirectory() as tmpdir:
+ path = Path(tmpdir) / "output.csv"
+ items.to_csv(path)
+
+ reader = csv.DictReader(path.open(newline="", encoding="utf-8"))
+ assert reader.fieldnames == ["name", "price"]
+ rows = list(reader)
+ assert rows[0] == {"name": "first", "price": ""}
+ assert rows[1] == {"name": "second", "price": "20"}
+
+ def test_to_csv_serializes_nested_values(self):
+ items = ItemList()
+ items.append({"name": "first", "tags": ["a", "b"], "meta": {"x": 1}, "empty": None})
+
+ with tempfile.TemporaryDirectory() as tmpdir:
+ path = Path(tmpdir) / "output.csv"
+ items.to_csv(path)
+
+ row = next(csv.DictReader(path.open(newline="", encoding="utf-8")))
+ assert json.loads(row["tags"]) == ["a", "b"]
+ assert json.loads(row["meta"]) == {"x": 1}
+ assert row["empty"] == ""
+
+ def test_to_csv_accepts_explicit_fields_and_delimiter(self):
+ items = ItemList()
+ items.append({"name": "first", "price": 10, "skipped": "yes"})
+
+ with tempfile.TemporaryDirectory() as tmpdir:
+ path = Path(tmpdir) / "output.tsv"
+ items.to_csv(path, fields=["price", "name"], delimiter="\t")
+
+ lines = path.read_text(encoding="utf-8").splitlines()
+ assert lines[0] == "price\tname"
+ assert lines[1] == "10\tfirst"
+
+ def test_to_csv_handles_no_items(self):
+ with tempfile.TemporaryDirectory() as tmpdir:
+ path = Path(tmpdir) / "empty.csv"
+ ItemList().to_csv(path)
+
+ assert path.exists()
+ assert path.read_text(encoding="utf-8").strip() == ""
+
+ def test_to_csv_creates_parent_directory(self):
+ items = ItemList()
+ items.append({"data": "test"})
+
+ with tempfile.TemporaryDirectory() as tmpdir:
+ path = Path(tmpdir) / "nested" / "dir" / "output.csv"
+ items.to_csv(path)
+
+ assert path.exists()
+
+ def test_to_xml_creates_file(self):
+ """Test to_xml writes one element per item with the keys as children."""
+ items = ItemList()
+ items.append({"name": "first", "price": 10})
+ items.append({"name": "second", "price": 20})
+
+ with tempfile.TemporaryDirectory() as tmpdir:
+ path = Path(tmpdir) / "output.xml"
+ items.to_xml(path)
+
+ assert path.exists()
+
+ root = _read_xml(path)
+ assert root.tag == "items"
+ assert len(root) == 2
+ assert [child.tag for child in root] == ["item", "item"]
+ assert root[0].find("name").text == "first"
+ assert root[0].find("price").text == "10"
+
+ def test_to_xml_custom_tags(self):
+ items = ItemList()
+ items.append({"name": "first"})
+
+ with tempfile.TemporaryDirectory() as tmpdir:
+ path = Path(tmpdir) / "output.xml"
+ items.to_xml(path, root_tag="products", item_tag="product")
+
+ root = _read_xml(path)
+ assert root.tag == "products"
+ assert root[0].tag == "product"
+
+ def test_to_xml_sanitizes_keys_that_are_not_valid_tags(self):
+ """Scraped keys can be anything, but XML names can't"""
+ items = ItemList()
+ items.append({"product name": "first", "2nd": "x", "ok_key": "y"})
+
+ with tempfile.TemporaryDirectory() as tmpdir:
+ path = Path(tmpdir) / "output.xml"
+ items.to_xml(path)
+
+ item = _read_xml(path)[0]
+ tags = {child.tag: child for child in item}
+ assert tags["product_name"].text == "first"
+ assert tags["product_name"].get("name") == "product name"
+ assert tags["_2nd"].get("name") == "2nd"
+ assert tags["ok_key"].get("name") is None
+
+ def test_to_xml_strips_characters_that_xml_forbids(self):
+ """Control characters in scraped text would produce a file no parser can read"""
+ items = ItemList()
+ items.append({"text": "bad\x08char\x0bhere", "kept": "tab\tnewline\n"})
+
+ with tempfile.TemporaryDirectory() as tmpdir:
+ path = Path(tmpdir) / "output.xml"
+ items.to_xml(path)
+
+ item = _read_xml(path)[0]
+ assert item.find("text").text == "badcharhere"
+ assert item.find("kept").text == "tab\tnewline\n"
+
+ def test_to_xml_serializes_nested_values(self):
+ items = ItemList()
+ items.append({"tags": ["a", "b"], "meta": {"x": 1}, "empty": None})
+
+ with tempfile.TemporaryDirectory() as tmpdir:
+ path = Path(tmpdir) / "output.xml"
+ items.to_xml(path)
+
+ item = _read_xml(path)[0]
+ assert json.loads(item.find("tags").text) == ["a", "b"]
+ assert json.loads(item.find("meta").text) == {"x": 1}
+ assert item.find("empty").text in (None, "")
+
+ def test_to_xml_handles_no_items(self):
+ with tempfile.TemporaryDirectory() as tmpdir:
+ path = Path(tmpdir) / "empty.xml"
+ ItemList().to_xml(path)
+
+ root = _read_xml(path)
+ assert root.tag == "items"
+ assert len(root) == 0
+
+ def test_to_xml_creates_parent_directory(self):
+ items = ItemList()
+ items.append({"data": "test"})
+
+ with tempfile.TemporaryDirectory() as tmpdir:
+ path = Path(tmpdir) / "nested" / "dir" / "output.xml"
+ items.to_xml(path)
+
+ assert path.exists()
+
def test_to_jsonl_creates_file(self):
"""Test to_jsonl creates JSON Lines file."""
items = ItemList()