🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Python aiai-scrapingautomationcrawlercrawlingcrawling-pythondatadata-extractionmcpmcp-serverplaywrightpythonscrapingselectorsstealthweb-scraperweb-scrapingweb-scraping-pythonwebscrapingxpath
docs: update the changelog
Karim shoaird04613e52e9c
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 35d41ff..a608a82 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -2,6 +2,66 @@All notable changes to Scrapling, taken from the [GitHub releases](https://github.com/D4Vinci/Scrapling/releases).+## [v0.4.15](https://github.com/D4Vinci/Scrapling/releases/tag/v0.4.15) - 2026-08-23++**One of the biggest releases this year: a reworked MCP server, RAG-ready Markdown in one line, an improved Cloudflare solver, and browser tabs that stay open for automation 🚀**++> [!WARNING]+> **This release introduces breaking changes to the MCP server. Check the [breaking changes](https://scrapling.readthedocs.io/en/latest/ai/mcp-server.html#breaking-changes) section before updating.**++### 🚀 New Stuff and quality of life changes++- **Browser tabs now stay open and get reused across requests** (Check the [docs](https://scrapling.readthedocs.io/en/latest/fetching/dynamic.html#session-management)):+ - All browser sessions keep their tabs after a request, and the next request reuses a free tab instead of opening a new one.+ - Every request re-applies its own settings (timeouts, headers, resource blocking) to the tab it gets, so nothing leaks between requests.+ - Tabs that hit an error are closed and replaced, and the new `close_pages()` method closes every open tab.+ - The page you fetched stays loaded, so a `page_setup` function on the next request runs on it before navigating away. That's the building block for chaining automation across requests.+- **Turn any page into clean, LLM-ready Markdown with `Response.markdown()`** (Check the [docs](https://scrapling.readthedocs.io/en/latest/ai/building-rag-systems.html)):+ ```python+ from scrapling.fetchers import Fetcher++ markdown = Fetcher.get("https://example.com").markdown(main_content_only=True)+ ```+ - Scripts, styles, and hidden prompt-injection content are always stripped first, the same cleaning the MCP server does.+ - Pass `css_selector` to convert only the elements you need.+ - Available through the new `rag` extra (`pip install "scrapling[rag]"`), which the `ai`/`shell`/`all` extras include too.+- **New `SiteToMarkdownSpider` template to crawl a whole website into a Markdown corpus for RAG pipelines** (Check the [docs](https://scrapling.readthedocs.io/en/latest/ai/building-rag-systems.html)):+ ```python+ from scrapling.spiders import SiteToMarkdownSpider++ class DocsSpider(SiteToMarkdownSpider):+ name = "docs"+ start_urls = ["https://example.com/docs/"]+ allowed_domains = {"example.com"}+ output_dir = "docs_markdown"++ result = DocsSpider().start()+ result.items.to_jsonl("docs.jsonl")+ ```+ - Yields one item per page with `url`/`title`/`markdown`, and the optional `output_dir` writes one Markdown file per page.+ - `max_pages` caps the crawl, and since it builds on `CrawlSpider`, overriding `rules()` gives you full control over which links get followed.+- **The MCP server is reworked (breaking)** (Check the [breaking changes](https://scrapling.readthedocs.io/en/latest/ai/mcp-server.html#breaking-changes) and the [docs](https://scrapling.readthedocs.io/en/latest/ai/mcp-server.html)):+ - The 13 tools are now split into two modes: one-shot tools (`fetch`, `bulk_fetch`, `stealthy_fetch`, `bulk_stealthy_fetch`) that always launch their own browser and show their real defaults, and session tools that work through a session opened once.+ - The new `session_fetch` tool fetches through a browser session, while `open_session` now holds the browser-level settings only and returns the session's effective settings for the AI agent.+ - The `get` tool is renamed to `make_request`, and it now supports any HTTP method.+ - The new `open_request_session` and `session_make_request` tools give the AI persistent HTTP sessions that keep cookies and the browser fingerprint between requests.+ - This also ends fetches resetting the session's settings, first fixed by @Yigtwxx in [#418](https://github.com/D4Vinci/Scrapling/pull/418).+- **The MCP server's HTTP transport now requires authentication and binds to localhost by default (breaking)** by @yamantaka-singh in [#414](https://github.com/D4Vinci/Scrapling/pull/414) (Fixes [#413](https://github.com/D4Vinci/Scrapling/issues/413), check the [docs](https://scrapling.readthedocs.io/en/latest/ai/mcp-server.html#authentication)):+ - Pass `--auth-token` (or the `SCRAPLING_MCP_AUTH_TOKEN` environment variable) to require a bearer token, or `--no-auth` to serve it unauthenticated on purpose.+ - Pass `--host 0.0.0.0` to accept connections from the network.++### 🐛 Bug Fixes++- **Cloudflare Turnstile/Interstitial solving now works regardless of the browser locale, and no longer loops forever on interactive challenges in headless mode**. Stealth pages also stop crashing mid-solve, with contributions by @subediparas5 in [#412](https://github.com/D4Vinci/Scrapling/pull/412). (Fixes [#411](https://github.com/D4Vinci/Scrapling/issues/411) and [#422](https://github.com/D4Vinci/Scrapling/issues/422))+- **Fixed `find`/`find_all` with `class_` silently missing multi-class elements** by @yetval in [#410](https://github.com/D4Vinci/Scrapling/pull/410), **and blank `class_` values and unescaped CSS string values** by @yamantaka-singh in [#417](https://github.com/D4Vinci/Scrapling/pull/417).+- **Fixed cached responses in `development_mode` losing the request meta on replay** by @Yigtwxx in [#419](https://github.com/D4Vinci/Scrapling/pull/419).+- **Fixed HTTP requests with `retries` below 1 failing without sending the request** by @Yigtwxx in [#420](https://github.com/D4Vinci/Scrapling/pull/420).++### Docs++- **The website sections are restructured**: a new "Using with AI" section holds the MCP server, the new [Agent skill](https://scrapling.readthedocs.io/en/latest/ai/agent-skill.html) page, and the [Building RAG systems](https://scrapling.readthedocs.io/en/latest/ai/building-rag-systems.html) guide, and the BeautifulSoup migration guide moved next to the Scrapy integration under "Integrations and migrations".+- **Added a [CHANGELOG.md](https://github.com/D4Vinci/Scrapling/blob/main/CHANGELOG.md)** to the repository with the notes of every release so far.+## [v0.4.14](https://github.com/D4Vinci/Scrapling/releases/tag/v0.4.14) - 2026-08-10**A quick maintenance release to fix installation with `uv` 🔧**