🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Python aiai-scrapingautomationcrawlercrawlingcrawling-pythondatadata-extractionmcpmcp-serverplaywrightpythonscrapingselectorsstealthweb-scraperweb-scrapingweb-scraping-pythonwebscrapingxpath
feat(spiders): Add exporters to CSV/XML
Karim shoair0a7fec8f864f
diff --git a/README.md b/README.mdindex cec3704..63a453f 100644--- a/README.md+++ b/README.md@@ -216,7 +216,7 @@ MySpider().start()- 🧪 **Development Mode**: Cache responses to disk on the first run and replay them on subsequent runs - iterate on your `parse()` logic without re-hitting the target servers.- 🧩 **Ready-made Spider Templates**: Skip the boilerplate with `CrawlSpider` for rule-based link following, `SitemapSpider` for sitemap/robots.txt-driven crawls, and `ShopifySpider` to pull every product out of any Shopify store through its JSON API, one item per variant.- 🔗 **Link Extraction**: A standalone `LinkExtractor` primitive with allow/deny patterns, domain filters, CSS/XPath scoping, extension filtering, and canonicalization - use it inside the templates or on its own.-- 📦 **Built-in Export**: Export results through hooks and your own pipeline or the built-in JSON/JSONL with `result.items.to_json()` / `result.items.to_jsonl()` respectively.+- 📦 **Built-in Export**: Export results through hooks and your own pipeline or the built-in JSON/JSONL/CSV/XML exporters with `result.items.to_json()`, `to_jsonl()`, `to_csv()`, and `to_xml()`.### Advanced Websites Fetching with Session Support- **HTTP Requests**: Fast and stealthy HTTP requests with the `Fetcher` class. Can impersonate browsers' TLS fingerprint, headers, and use HTTP/3.diff --git a/agent-skill/Scrapling-Skill.zip b/agent-skill/Scrapling-Skill.zipindex 3cb75e7..b82cc6f 100644Binary files a/agent-skill/Scrapling-Skill.zip and b/agent-skill/Scrapling-Skill.zip differdiff --git a/agent-skill/Scrapling-Skill/references/spiders/architecture.md b/agent-skill/Scrapling-Skill/references/spiders/architecture.mdindex 54186a1..68ea7b8 100644--- a/agent-skill/Scrapling-Skill/references/spiders/architecture.md+++ b/agent-skill/Scrapling-Skill/references/spiders/architecture.md@@ -66,7 +66,7 @@ An optional cache that, when development mode is enabled, stores every fetched r### Output-Scraped items are collected in an `ItemList` (a list subclass with `to_json()` and `to_jsonl()` export methods). Crawl statistics are tracked in a `CrawlStats` dataclass which contains a lot of useful info.+Scraped items are collected in an `ItemList` (a list subclass with `to_json()`, `to_jsonl()`, `to_csv()`, and `to_xml()` export methods). Crawl statistics are tracked in a `CrawlStats` dataclass which contains a lot of useful info.## Comparison with Scrapy@@ -88,7 +88,7 @@ If you're coming from Scrapy, here's how Scrapling's spider system maps:| Domain filtering | `allowed_domains` | `allowed_domains` || Robots.txt | `ROBOTSTXT_OBEY` setting | `robots_txt_obey` class attribute || Pause/Resume | `JOBDIR` setting | `crawldir` constructor argument |-| Export | Feed exports | `result.items.to_json()` / `to_jsonl()` or custom through hooks |+| Export | Feed exports | `result.items.to_json()` / `to_jsonl()` / `to_csv()` / `to_xml()` or custom through hooks || Running | `scrapy crawl spider_name` | `MySpider().start()` || Streaming | N/A | `async for item in spider.stream()` || Multi-session | N/A | Multiple sessions with different types per spider |\ No newline at end of filediff --git a/agent-skill/Scrapling-Skill/references/spiders/getting-started.md b/agent-skill/Scrapling-Skill/references/spiders/getting-started.mdindex e553b7b..f18073e 100644--- a/agent-skill/Scrapling-Skill/references/spiders/getting-started.md+++ b/agent-skill/Scrapling-Skill/references/spiders/getting-started.md@@ -113,9 +113,17 @@ result.items.to_json("quotes.json", indent=True)# Export as JSON Lines (one JSON object per line)result.items.to_jsonl("quotes.jsonl")++# Export as CSV or XML+result.items.to_csv("quotes.csv")+result.items.to_xml("quotes.xml")```-Both methods create parent directories automatically if they don't exist.+All of them create parent directories automatically if they don't exist.++`to_csv()` writes a column for every key found across your items, so items that don't all share the same keys are still exported with the missing cells left empty. Pass `fields=[...]` to pick the columns and their order yourself, or `delimiter="\t"` for a TSV file. `to_xml()` wraps each item in an `<item>` element inside an `<items>` root, both renameable through `root_tag` and `item_tag`.++In both formats, values that aren't simple scalars (a nested dictionary or a list) are written as JSON so nothing is silently dropped.## Filtering Domainsdiff --git a/docs/README_AR.md b/docs/README_AR.mdindex 1224119..7a50a10 100644--- a/docs/README_AR.md+++ b/docs/README_AR.md@@ -212,7 +212,7 @@ MySpider().start()- 🧪 **وضع التطوير**: تخزين الاستجابات على القرص في التشغيل الأول وإعادة تشغيلها في التشغيلات اللاحقة - كرّر العمل على منطق `parse()` دون الحاجة لإرسال طلبات جديدة إلى الخوادم المستهدفة.- 🧩 **قوالب Spider جاهزة**: تخطَّ الكود المتكرر مع `CrawlSpider` لتتبع الروابط بقواعد، و`SitemapSpider` للزحف المعتمد على sitemap/robots.txt، و`ShopifySpider` لسحب كل منتج من أي متجر Shopify عبر واجهة JSON الخاصة به، مع عنصر لكل variant.- 🔗 **استخراج الروابط**: أداة `LinkExtractor` مستقلة مع أنماط allow/deny، وفلاتر النطاقات، وتحديد النطاق عبر CSS/XPath، وتصفية الامتدادات، وتوحيد الروابط - استخدمها داخل القوالب أو بمفردها.-- 📦 **تصدير مدمج**: صدّر النتائج عبر الخطافات وخط الأنابيب الخاص بك أو JSON/JSONL المدمج مع `result.items.to_json()` / `result.items.to_jsonl()` على التوالي.+- 📦 **تصدير مدمج**: صدّر النتائج عبر الخطافات وخط الأنابيب الخاص بك أو المصدّرات المدمجة JSON/JSONL/CSV/XML عبر `result.items.to_json()` و`to_jsonl()` و`to_csv()` و`to_xml()`.### جلب متقدم للمواقع مع دعم الجلسات- **طلبات HTTP**: طلبات HTTP سريعة وخفية مع فئة `Fetcher`. يمكنها تقليد بصمة TLS للمتصفح والرؤوس واستخدام HTTP/3.diff --git a/docs/README_CN.md b/docs/README_CN.mdindex 7b525b7..deb4840 100644--- a/docs/README_CN.md+++ b/docs/README_CN.md@@ -212,7 +212,7 @@ MySpider().start()- 🧪 **开发模式**:首次运行时将响应缓存到磁盘,后续运行时直接回放 - 在不重新请求目标服务器的情况下迭代你的 `parse()` 逻辑。- 🧩 **现成的 Spider 模板**:跳过样板代码--使用 `CrawlSpider` 基于规则跟踪链接,`SitemapSpider` 基于 sitemap/robots.txt 爬取,以及 `ShopifySpider` 通过 JSON API 抓取任意 Shopify 商店的全部商品,每个变体一条数据。- 🔗 **链接提取**:独立的 `LinkExtractor` 组件,支持 allow/deny 模式、域名过滤、CSS/XPath 范围限定、扩展名过滤和链接规范化--可在模板中使用,也可单独使用。-- 📦 **内置导出**:通过钩子和您自己的管道导出结果,或使用内置的 JSON/JSONL,分别通过 `result.items.to_json()`/`result.items.to_jsonl()`。+- 📦 **内置导出**:通过钩子和您自己的管道导出结果,或使用内置的 JSON/JSONL/CSV/XML 导出器:`result.items.to_json()`、`to_jsonl()`、`to_csv()` 和 `to_xml()`。### 支持 Session 的高级网站获取- **HTTP 请求**:使用 `Fetcher` 类进行快速和隐秘的 HTTP 请求。可以模拟浏览器的 TLS fingerprint、标头并使用 HTTP/3。diff --git a/docs/README_DE.md b/docs/README_DE.mdindex 27fb549..ec3038a 100644--- a/docs/README_DE.md+++ b/docs/README_DE.md@@ -212,7 +212,7 @@ MySpider().start()- 🧪 **Entwicklungsmodus**: Antworten beim ersten Lauf auf der Festplatte zwischenspeichern und bei weiteren Läufen erneut abspielen - iterieren Sie an Ihrer `parse()`-Logik, ohne die Zielserver erneut abzufragen.- 🧩 **Fertige Spider-Vorlagen**: Sparen Sie sich den Boilerplate-Code mit `CrawlSpider` für regelbasiertes Link-Following, `SitemapSpider` für Sitemap-/robots.txt-gesteuerte Crawls und `ShopifySpider`, um über die JSON-API jedes Produkt aus einem beliebigen Shopify-Shop zu ziehen -- ein Item pro Variante.- 🔗 **Link-Extraktion**: Ein eigenständiges `LinkExtractor`-Primitiv mit Allow-/Deny-Mustern, Domain-Filtern, CSS-/XPath-Eingrenzung, Erweiterungsfilterung und Kanonisierung -- nutzbar innerhalb der Vorlagen oder für sich allein.-- 📦 **Integrierter Export**: Ergebnisse über Hooks und Ihre eigene Pipeline oder den integrierten JSON/JSONL-Export mit `result.items.to_json()` / `result.items.to_jsonl()` exportieren.+- 📦 **Integrierter Export**: Ergebnisse über Hooks und Ihre eigene Pipeline oder die integrierten JSON/JSONL/CSV/XML-Exporte mit `result.items.to_json()`, `to_jsonl()`, `to_csv()` und `to_xml()` exportieren.### Erweitertes Website-Abrufen mit Session-Unterstützung- **HTTP-Anfragen**: Schnelle und heimliche HTTP-Anfragen mit der `Fetcher`-Klasse. Kann Browser-TLS-Fingerprints und Header imitieren und HTTP/3 verwenden.diff --git a/docs/README_ES.md b/docs/README_ES.mdindex e3f475e..632654e 100644--- a/docs/README_ES.md+++ b/docs/README_ES.md@@ -212,7 +212,7 @@ MySpider().start()- 🧪 **Modo de Desarrollo**: Almacena las respuestas en disco en la primera ejecución y las reproduce en ejecuciones posteriores - itera sobre tu lógica de `parse()` sin volver a consultar los servidores objetivo.- 🧩 **Plantillas de Spider Listas para Usar**: Olvídate del código repetitivo con `CrawlSpider` para seguir enlaces por reglas, `SitemapSpider` para rastreos guiados por sitemap/robots.txt, y `ShopifySpider` para extraer todos los productos de cualquier tienda Shopify a través de su API JSON, un ítem por variante.- 🔗 **Extracción de Enlaces**: Un primitivo `LinkExtractor` independiente con patrones allow/deny, filtros de dominio, delimitación por CSS/XPath, filtrado de extensiones y canonicalización - úsalo dentro de las plantillas o por su cuenta.-- 📦 **Exportación Integrada**: Exporta resultados a través de hooks y tu propio pipeline o el JSON/JSONL integrado con `result.items.to_json()` / `result.items.to_jsonl()` respectivamente.+- 📦 **Exportación Integrada**: Exporta resultados a través de hooks y tu propio pipeline o los exportadores integrados JSON/JSONL/CSV/XML con `result.items.to_json()`, `to_jsonl()`, `to_csv()` y `to_xml()`.### Obtención Avanzada de Sitios Web con Soporte de Session- **Solicitudes HTTP**: Solicitudes HTTP rápidas y sigilosas con la clase `Fetcher`. Puede imitar el fingerprint TLS de los navegadores, encabezados y usar HTTP/3.diff --git a/docs/README_FR.md b/docs/README_FR.mdindex d964a4a..776abae 100644--- a/docs/README_FR.md+++ b/docs/README_FR.md@@ -212,7 +212,7 @@ MySpider().start()- 🧪 **Mode développement** : Mettez les réponses en cache sur le disque lors de la première exécution et rejouez-les lors des exécutions suivantes - itérez sur votre logique `parse()` sans solliciter à nouveau les serveurs cibles.- 🧩 **Modèles de Spider prêts à l'emploi** : Évitez le code répétitif avec `CrawlSpider` pour le suivi de liens par règles, `SitemapSpider` pour les crawls pilotés par sitemap/robots.txt, et `ShopifySpider` pour récupérer tous les produits de n'importe quelle boutique Shopify via son API JSON, un item par variante.- 🔗 **Extraction de liens** : Un primitif `LinkExtractor` autonome avec motifs allow/deny, filtres de domaines, délimitation CSS/XPath, filtrage des extensions et canonicalisation - utilisable dans les modèles ou seul.-- 📦 **Export intégré** : Exportez les résultats via des hooks et votre propre pipeline ou l'export JSON/JSONL intégré avec `result.items.to_json()` / `result.items.to_jsonl()` respectivement.+- 📦 **Export intégré** : Exportez les résultats via des hooks et votre propre pipeline ou les exports JSON/JSONL/CSV/XML intégrés avec `result.items.to_json()`, `to_jsonl()`, `to_csv()` et `to_xml()`.### Récupération avancée de sites web avec support de sessions- **Requêtes HTTP** : Requêtes HTTP rapides et furtives avec la classe `Fetcher`. Peut imiter l'empreinte TLS des navigateurs, les headers et utiliser HTTP/3.diff --git a/docs/README_JP.md b/docs/README_JP.mdindex dcd6071..4d317cd 100644--- a/docs/README_JP.md+++ b/docs/README_JP.md@@ -212,7 +212,7 @@ MySpider().start()- 🧪 **開発モード**:初回実行時にレスポンスをディスクにキャッシュし、以降の実行ではそれを再生 - ターゲットサーバーに再リクエストすることなく `parse()` ロジックを反復開発できます。- 🧩 **すぐに使える Spider テンプレート**:定型コードは不要です。ルールベースのリンク追跡には `CrawlSpider`、sitemap/robots.txt を起点としたクロールには `SitemapSpider`、そして `ShopifySpider` を使えば任意の Shopify ストアの全商品を JSON API 経由で取得できます(バリエーションごとに 1 アイテム)。- 🔗 **リンク抽出**:allow/deny パターン、ドメインフィルタ、CSS/XPath による範囲指定、拡張子フィルタ、正規化に対応した独立した `LinkExtractor` を提供。テンプレート内でも単体でも使用できます。-- 📦 **組み込みエクスポート**:フックや独自のパイプライン、または組み込みの JSON/JSONL で結果をエクスポート。それぞれ`result.items.to_json()` / `result.items.to_jsonl()`を使用。+- 📦 **組み込みエクスポート**:フックや独自のパイプライン、または組み込みの JSON/JSONL/CSV/XML で結果をエクスポート。`result.items.to_json()`、`to_jsonl()`、`to_csv()`、`to_xml()` が使えます。### Session サポート付き高度なウェブサイト取得- **HTTP リクエスト**:`Fetcher` クラスで高速かつステルスな HTTP リクエスト。ブラウザの TLS fingerprint、ヘッダーを模倣し、HTTP/3 を使用可能。diff --git a/docs/README_KR.md b/docs/README_KR.mdindex 2935e9f..84f4a38 100644--- a/docs/README_KR.md+++ b/docs/README_KR.md@@ -212,7 +212,7 @@ MySpider().start()- 🧪 **개발 모드**: 첫 실행 시 응답을 디스크에 캐싱하고 이후 실행에서는 캐시된 응답을 재생합니다 - 대상 서버에 다시 요청하지 않고 `parse()` 로직을 반복 개발할 수 있습니다.- 🧩 **바로 쓰는 Spider 템플릿**: 반복적인 코드는 건너뛰세요. 규칙 기반 링크 추적에는 `CrawlSpider`, sitemap/robots.txt 기반 크롤링에는 `SitemapSpider`, 그리고 `ShopifySpider`로 어떤 Shopify 스토어든 JSON API를 통해 모든 상품을 가져올 수 있습니다(변형별로 항목 하나).- 🔗 **링크 추출**: allow/deny 패턴, 도메인 필터, CSS/XPath 범위 지정, 확장자 필터링, 정규화를 지원하는 독립형 `LinkExtractor`. 템플릿 안에서도, 단독으로도 사용할 수 있습니다.-- 📦 **내장 내보내기**: 훅이나 자체 파이프라인, 또는 내장 JSON/JSONL로 결과를 내보냅니다. 각각 `result.items.to_json()` / `result.items.to_jsonl()`을 사용합니다.+- 📦 **내장 내보내기**: 훅이나 자체 파이프라인, 또는 내장 JSON/JSONL/CSV/XML로 결과를 내보냅니다. `result.items.to_json()`, `to_jsonl()`, `to_csv()`, `to_xml()`을 사용합니다.### 세션을 지원하는 고급 웹사이트 가져오기- **HTTP 요청**: `Fetcher` 클래스로 빠르고 은밀한 HTTP 요청. 브라우저의 TLS fingerprint, 헤더를 모방하고, HTTP/3를 사용할 수 있습니다.diff --git a/docs/README_PT_BR.md b/docs/README_PT_BR.mdindex ba8282c..285751d 100644--- a/docs/README_PT_BR.md+++ b/docs/README_PT_BR.md@@ -214,7 +214,7 @@ MySpider().start()- 🧪 **Modo de Desenvolvimento**: Armazene respostas em disco na primeira execução e reproduza-as nas seguintes - itere sobre sua lógica de `parse()` sem reenviar requisições aos servidores-alvo.- 🧩 **Templates de Spider Prontos**: Pule o código repetitivo com `CrawlSpider` para seguir links por regras, `SitemapSpider` para crawls guiados por sitemap/robots.txt e `ShopifySpider` para extrair todos os produtos de qualquer loja Shopify através da sua API JSON, um item por variação.- 🔗 **Extração de Links**: Um primitivo `LinkExtractor` independente com padrões allow/deny, filtros de domínio, delimitação por CSS/XPath, filtragem de extensões e canonicalização - use dentro dos templates ou sozinho.-- 📦 **Exportação Nativa**: Exporte resultados via hooks, seu próprio pipeline ou JSON/JSONL nativos com `result.items.to_json()` / `result.items.to_jsonl()` respectivamente.+- 📦 **Exportação Nativa**: Exporte resultados via hooks, seu próprio pipeline ou os exportadores nativos JSON/JSONL/CSV/XML com `result.items.to_json()`, `to_jsonl()`, `to_csv()` e `to_xml()`.### Busca Avançada de Sites com Suporte a Sessões- **Requisições HTTP**: Requisições HTTP rápidas e furtivas com a classe `Fetcher`. Pode imitar fingerprint TLS de navegadores, cabeçalhos e usar HTTP/3.diff --git a/docs/README_RU.md b/docs/README_RU.mdindex 4f223c9..57dd334 100644--- a/docs/README_RU.md+++ b/docs/README_RU.md@@ -214,7 +214,7 @@ MySpider().start()- 🧪 **Режим разработки**: Кэшируйте ответы на диск при первом запуске и воспроизводите их при последующих запусках - итерируйте над логикой `parse()`, не отправляя повторные запросы к целевым серверам.- 🧩 **Готовые шаблоны Spider**: Обойдитесь без шаблонного кода - `CrawlSpider` для перехода по ссылкам на основе правил, `SitemapSpider` для обхода на основе sitemap/robots.txt и `ShopifySpider`, чтобы вытащить все товары любого магазина Shopify через его JSON API, по одному элементу на вариант.- 🔗 **Извлечение ссылок**: Отдельный примитив `LinkExtractor` с шаблонами allow/deny, фильтрами доменов, ограничением области через CSS/XPath, фильтрацией расширений и канонизацией - используйте его внутри шаблонов или сам по себе.-- 📦 **Встроенный экспорт**: Экспортируйте результаты через хуки и собственный конвейер или встроенный JSON/JSONL с `result.items.to_json()` / `result.items.to_jsonl()` соответственно.+- 📦 **Встроенный экспорт**: Экспортируйте результаты через хуки и собственный конвейер или встроенные экспортеры JSON/JSONL/CSV/XML с `result.items.to_json()`, `to_jsonl()`, `to_csv()` и `to_xml()`.### Продвинутая загрузка сайтов с поддержкой Session- **HTTP-запросы**: Быстрые и скрытные HTTP-запросы с классом `Fetcher`. Может имитировать TLS fingerprint браузера, заголовки и использовать HTTP/3.diff --git a/docs/index.md b/docs/index.mdindex b537ef1..a96f155 100644--- a/docs/index.md+++ b/docs/index.md@@ -104,7 +104,7 @@ MySpider().start()- 🧪 **Development Mode**: Cache responses to disk on the first run and replay them on subsequent runs - iterate on your `parse()` logic without re-hitting the target servers.- 🧩 **Ready-made Spider Templates**: Skip the boilerplate with `CrawlSpider` for rule-based link following, `SitemapSpider` for sitemap/robots.txt-driven crawls, and `ShopifySpider` to pull every product out of any Shopify store through its JSON API, one item per variant.- 🔗 **Link Extraction**: A standalone `LinkExtractor` primitive with allow/deny patterns, domain filters, CSS/XPath scoping, extension filtering, and canonicalization - use it inside the templates or on its own.-- 📦 **Built-in Export**: Export results through hooks and your own pipeline or the built-in JSON/JSONL with `result.items.to_json()` / `result.items.to_jsonl()` respectively.+- 📦 **Built-in Export**: Export results through hooks and your own pipeline or the built-in JSON/JSONL/CSV/XML exporters with `result.items.to_json()`, `to_jsonl()`, `to_csv()`, and `to_xml()`.### Advanced Websites Fetching with Session Support- **HTTP Requests**: Fast and stealthy HTTP requests with the `Fetcher` class. Can impersonate browsers' TLS fingerprint, headers, and use HTTP/3.diff --git a/docs/spiders/architecture.md b/docs/spiders/architecture.mdindex dc360b8..d6f6d65 100644--- a/docs/spiders/architecture.md+++ b/docs/spiders/architecture.md@@ -75,7 +75,7 @@ An optional cache that, when development mode is enabled, stores every fetched r### Output-Scraped items are collected in an `ItemList` (a list subclass with `to_json()` and `to_jsonl()` export methods). Crawl statistics are tracked in a `CrawlStats` dataclass which contains a lot of useful info.+Scraped items are collected in an `ItemList` (a list subclass with `to_json()`, `to_jsonl()`, `to_csv()`, and `to_xml()` export methods). Crawl statistics are tracked in a `CrawlStats` dataclass which contains a lot of useful info.## Comparison with Scrapy@@ -97,7 +97,7 @@ If you're coming from Scrapy, here's how Scrapling's spider system maps:| Domain filtering | `allowed_domains` | `allowed_domains` || Robots.txt | `ROBOTSTXT_OBEY` setting | `robots_txt_obey` class attribute || Pause/Resume | `JOBDIR` setting | `crawldir` constructor argument |-| Export | Feed exports | `result.items.to_json()` / `to_jsonl()` or custom through hooks |+| Export | Feed exports | `result.items.to_json()` / `to_jsonl()` / `to_csv()` / `to_xml()` or custom through hooks || Running | `scrapy crawl spider_name` | `MySpider().start()` || Streaming | N/A | `async for item in spider.stream()` || Multi-session | N/A | Multiple sessions with different types per spider |\ No newline at end of filediff --git a/docs/spiders/getting-started.md b/docs/spiders/getting-started.mdindex 197828f..7f6b523 100644--- a/docs/spiders/getting-started.md+++ b/docs/spiders/getting-started.md@@ -125,9 +125,17 @@ result.items.to_json("quotes.json", indent=True)# Export as JSON Lines (one JSON object per line)result.items.to_jsonl("quotes.jsonl")++# Export as CSV or XML+result.items.to_csv("quotes.csv")+result.items.to_xml("quotes.xml")```-Both methods create parent directories automatically if they don't exist.+All of them create parent directories automatically if they don't exist.++`to_csv()` writes a column for every key found across your items, so items that don't all share the same keys are still exported with the missing cells left empty. Pass `fields=[...]` to pick the columns and their order yourself, or `delimiter="\t"` for a TSV file. `to_xml()` wraps each item in an `<item>` element inside an `<items>` root, both renameable through `root_tag` and `item_tag`.++In both formats, values that aren't simple scalars (a nested dictionary or a list) are written as JSON so nothing is silently dropped.## Filtering Domainsdiff --git a/scrapling/spiders/result.py b/scrapling/spiders/result.pyindex 796bbc2..29bda89 100644--- a/scrapling/spiders/result.py+++ b/scrapling/spiders/result.py@@ -1,10 +1,37 @@+import csv+import refrom pathlib import Pathfrom dataclasses import dataclass, field+from xml.etree.ElementTree import ( # nosec B405 - we only serialize items, nothing here parses XML+ Element,+ ElementTree,+ SubElement,+ indent as indent_tree,+)import orjsonfrom scrapling.core.utils import log-from scrapling.core._types import Any, Iterator, Dict, List, Tuple, Union+from scrapling.core._types import Any, Iterable, Iterator, Dict, List, Optional, Tuple, Union++# Anything outside these ranges can't be represented in XML, and scraped pages are full of control characters+_XML_FORBIDDEN_CHARS = re.compile(r"[^\x09\x0a\x0d\x20--�\U00010000-\U0010ffff]")+_XML_FORBIDDEN_TAG_CHARS = re.compile(r"[^\w.-]", re.UNICODE)+++def _stringify(value: Any) -> str:+ """Turn an item's value into text, serializing containers to JSON so no data is silently dropped."""+ if value is None:+ return ""+ if isinstance(value, (dict, list, tuple)):+ return orjson.dumps(value, option=orjson.OPT_SERIALIZE_NUMPY).decode()+ return str(value)+++def _xml_tag(key: Any) -> str:+ """Turn an item's key into a usable XML tag name."""+ tag = _XML_FORBIDDEN_TAG_CHARS.sub("_", str(key))+ return tag if tag and (tag[0].isalpha() or tag[0] == "_") else f"_{tag}"class ItemList(list):@@ -37,6 +64,59 @@ class ItemList(list):f.write(b"\n")log.info("Saved %d items to %s", len(self), path)+ def to_csv(self, path: Union[str, Path], *, fields: Optional[Iterable[str]] = None, delimiter: str = ","):+ """Export items to a CSV file.++ Items that don't share the same keys are still written, with the missing cells left empty, and any value+ that isn't a scalar (a nested dictionary or a list) is written as JSON.++ :param path: Path to the output file+ :param fields: The columns to write, defaulting to every key found in the items, in the order they appeared+ :param delimiter: The character separating the columns+ """+ columns = list(fields) if fields is not None else list({key: None for item in self for key in item})++ file = Path(path)+ file.parent.mkdir(parents=True, exist_ok=True)+ with open(file, "w", newline="", encoding="utf-8") as f:+ writer = csv.DictWriter(f, fieldnames=columns, delimiter=delimiter, extrasaction="ignore")+ writer.writeheader()+ for item in self:+ writer.writerow({column: _stringify(item.get(column)) for column in columns})++ log.info("Saved %d items to %s", len(self), path)++ def to_xml(self, path: Union[str, Path], *, root_tag: str = "items", item_tag: str = "item", indent: bool = True):+ """Export items to an XML file.++ Each item becomes an element whose children are named after the item's keys. Keys that aren't valid XML+ names are rewritten and keep the original in a `name` attribute, and any value that isn't a scalar+ (a nested dictionary or a list) is written as JSON.++ :param path: Path to the output file+ :param root_tag: The name of the element wrapping all the items+ :param item_tag: The name of the element wrapping every item+ :param indent: Pretty-print the file instead of writing it on a single line+ """+ root = Element(root_tag)+ for item in self:+ element = SubElement(root, item_tag)+ for key, value in item.items():+ tag = _xml_tag(key)+ child = SubElement(element, tag)+ if tag != str(key):+ child.set("name", _XML_FORBIDDEN_CHARS.sub("", str(key)))+ child.text = _XML_FORBIDDEN_CHARS.sub("", _stringify(value))++ tree = ElementTree(root)+ if indent:+ indent_tree(tree, space=" ")++ file = Path(path)+ file.parent.mkdir(parents=True, exist_ok=True)+ tree.write(file, encoding="utf-8", xml_declaration=True)+ log.info("Saved %d items to %s", len(self), path)+@dataclassclass CrawlStats:diff --git a/tests/spiders/test_result.py b/tests/spiders/test_result.pyindex f59f236..cd8da61 100644--- a/tests/spiders/test_result.py+++ b/tests/spiders/test_result.py@@ -1,14 +1,21 @@"""Tests for the result module (ItemList, CrawlStats, CrawlResult)."""+import csvimport jsonimport tempfilefrom pathlib import Path+from xml.etree import ElementTree # nosec B405 - only used to read back files these tests just wroteimport pytestfrom scrapling.spiders.result import ItemList, CrawlStats, CrawlResult+def _read_xml(path):+ """Parse a file the test itself just wrote."""+ return ElementTree.parse(path).getroot() # nosec B314 - trusted local file, not untrusted input++class TestItemList:"""Test ItemList functionality."""@@ -68,6 +75,174 @@ class TestItemList:# Indented JSON should have newlinesassert "\n" in content+ def test_to_csv_creates_file(self):+ """Test to_csv writes a header and one row per item."""+ items = ItemList()+ items.append({"name": "first", "price": 10})+ items.append({"name": "second", "price": 20})++ with tempfile.TemporaryDirectory() as tmpdir:+ path = Path(tmpdir) / "output.csv"+ items.to_csv(path)++ assert path.exists()++ rows = list(csv.DictReader(path.open(newline="", encoding="utf-8")))+ assert len(rows) == 2+ assert rows[0] == {"name": "first", "price": "10"}+ assert rows[1]["name"] == "second"++ def test_to_csv_unions_the_keys_of_every_item(self):+ """Items with different keys must all be written, with the missing cells left empty."""+ items = ItemList()+ items.append({"name": "first"})+ items.append({"price": 20, "name": "second"})++ with tempfile.TemporaryDirectory() as tmpdir:+ path = Path(tmpdir) / "output.csv"+ items.to_csv(path)++ reader = csv.DictReader(path.open(newline="", encoding="utf-8"))+ assert reader.fieldnames == ["name", "price"]+ rows = list(reader)+ assert rows[0] == {"name": "first", "price": ""}+ assert rows[1] == {"name": "second", "price": "20"}++ def test_to_csv_serializes_nested_values(self):+ items = ItemList()+ items.append({"name": "first", "tags": ["a", "b"], "meta": {"x": 1}, "empty": None})++ with tempfile.TemporaryDirectory() as tmpdir:+ path = Path(tmpdir) / "output.csv"+ items.to_csv(path)++ row = next(csv.DictReader(path.open(newline="", encoding="utf-8")))+ assert json.loads(row["tags"]) == ["a", "b"]+ assert json.loads(row["meta"]) == {"x": 1}+ assert row["empty"] == ""++ def test_to_csv_accepts_explicit_fields_and_delimiter(self):+ items = ItemList()+ items.append({"name": "first", "price": 10, "skipped": "yes"})++ with tempfile.TemporaryDirectory() as tmpdir:+ path = Path(tmpdir) / "output.tsv"+ items.to_csv(path, fields=["price", "name"], delimiter="\t")++ lines = path.read_text(encoding="utf-8").splitlines()+ assert lines[0] == "price\tname"+ assert lines[1] == "10\tfirst"++ def test_to_csv_handles_no_items(self):+ with tempfile.TemporaryDirectory() as tmpdir:+ path = Path(tmpdir) / "empty.csv"+ ItemList().to_csv(path)++ assert path.exists()+ assert path.read_text(encoding="utf-8").strip() == ""++ def test_to_csv_creates_parent_directory(self):+ items = ItemList()+ items.append({"data": "test"})++ with tempfile.TemporaryDirectory() as tmpdir:+ path = Path(tmpdir) / "nested" / "dir" / "output.csv"+ items.to_csv(path)++ assert path.exists()++ def test_to_xml_creates_file(self):+ """Test to_xml writes one element per item with the keys as children."""+ items = ItemList()+ items.append({"name": "first", "price": 10})+ items.append({"name": "second", "price": 20})++ with tempfile.TemporaryDirectory() as tmpdir:+ path = Path(tmpdir) / "output.xml"+ items.to_xml(path)++ assert path.exists()++ root = _read_xml(path)+ assert root.tag == "items"+ assert len(root) == 2+ assert [child.tag for child in root] == ["item", "item"]+ assert root[0].find("name").text == "first"+ assert root[0].find("price").text == "10"++ def test_to_xml_custom_tags(self):+ items = ItemList()+ items.append({"name": "first"})++ with tempfile.TemporaryDirectory() as tmpdir:+ path = Path(tmpdir) / "output.xml"+ items.to_xml(path, root_tag="products", item_tag="product")++ root = _read_xml(path)+ assert root.tag == "products"+ assert root[0].tag == "product"++ def test_to_xml_sanitizes_keys_that_are_not_valid_tags(self):+ """Scraped keys can be anything, but XML names can't"""+ items = ItemList()+ items.append({"product name": "first", "2nd": "x", "ok_key": "y"})++ with tempfile.TemporaryDirectory() as tmpdir:+ path = Path(tmpdir) / "output.xml"+ items.to_xml(path)++ item = _read_xml(path)[0]+ tags = {child.tag: child for child in item}+ assert tags["product_name"].text == "first"+ assert tags["product_name"].get("name") == "product name"+ assert tags["_2nd"].get("name") == "2nd"+ assert tags["ok_key"].get("name") is None++ def test_to_xml_strips_characters_that_xml_forbids(self):+ """Control characters in scraped text would produce a file no parser can read"""+ items = ItemList()+ items.append({"text": "bad\x08char\x0bhere", "kept": "tab\tnewline\n"})++ with tempfile.TemporaryDirectory() as tmpdir:+ path = Path(tmpdir) / "output.xml"+ items.to_xml(path)++ item = _read_xml(path)[0]+ assert item.find("text").text == "badcharhere"+ assert item.find("kept").text == "tab\tnewline\n"++ def test_to_xml_serializes_nested_values(self):+ items = ItemList()+ items.append({"tags": ["a", "b"], "meta": {"x": 1}, "empty": None})++ with tempfile.TemporaryDirectory() as tmpdir:+ path = Path(tmpdir) / "output.xml"+ items.to_xml(path)++ item = _read_xml(path)[0]+ assert json.loads(item.find("tags").text) == ["a", "b"]+ assert json.loads(item.find("meta").text) == {"x": 1}+ assert item.find("empty").text in (None, "")++ def test_to_xml_handles_no_items(self):+ with tempfile.TemporaryDirectory() as tmpdir:+ path = Path(tmpdir) / "empty.xml"+ ItemList().to_xml(path)++ root = _read_xml(path)+ assert root.tag == "items"+ assert len(root) == 0++ def test_to_xml_creates_parent_directory(self):+ items = ItemList()+ items.append({"data": "test"})++ with tempfile.TemporaryDirectory() as tmpdir:+ path = Path(tmpdir) / "nested" / "dir" / "output.xml"+ items.to_xml(path)++ assert path.exists()+def test_to_jsonl_creates_file(self):"""Test to_jsonl creates JSON Lines file."""items = ItemList()