Soul-driven AI agent with permission-hardened tools, token budgets, and multi-channel access. Runs 24/7 from CLI, Telegram or More.
feat(search): three web sources (Parallel, DuckDuckGo html, lite), DuckDuckGo parsed exactly
Salman Qureshiff7e7f166b05
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex ba63b5c7..3c9dfd55 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -3,11 +3,12 @@## Unreleased### Web search-- **Parallel search first.** `web_search` now asks [Parallel's public search](https://search.parallel.ai/mcp) first, for web and news: a keyless MCP endpoint built for agents that answers in about a second with excerpts and publish dates. Brave, DuckDuckGo, Wikipedia, Hacker News and the news feeds stay as fallbacks, in that order. News keeps only links to individual articles, so section fronts like "/kashmir" are dropped and Bing News fills in.+- **Three web sources, the ones that work.** `web_search` asks [Parallel's public search](https://search.parallel.ai/mcp) first (keyless MCP built for agents, about a second, with excerpts and dates), then DuckDuckGo's html page, then its lite page. Brave, Wikipedia, Hacker News and Bing's web feed are gone from web search: Brave refuses automated requests and the others answered general questions badly. News keeps Parallel (article links only, section fronts dropped), then Bing News, Google News and Hacker News.+- **DuckDuckGo parsed exactly.** Both pages are read from their real markup: each result's title, the real address behind DuckDuckGo's redirect, and the snippet; ads (`result--ad`, sponsored rows) are dropped. A challenge is retried once after a short pause, and a page that is neither results nor "No results" is reported as "page not recognised" instead of silently empty. Tests run against saved copies of both real pages.- **News links open the article.** News search used Google News first, whose links are Google redirects (`news.google.com/rss/articles/…`) that can't be turned into the publisher's address. Bing News now comes first and its links are unwrapped to the real article; Google News is the fallback, and its links are marked as redirects.-- **Two web indexes, combined.** Brave and DuckDuckGo (its html and lite pages) both answer some automated requests and block others, by client fingerprint and rate. `web_search` now asks them in turn, retries a DuckDuckGo challenge once after a short pause, and combines relevant results until it has enough, instead of stopping at the first engine with one hit.- **Search never depends on the model.** Results come only from Mercury's own search tools, never from a provider's built-in web search, so search works the same with every model.-- **You can see what happened.** Every result ends with the trail, for example `Searched: Brave: blocked (HTTP 429) → DuckDuckGo: 5 results`, so "nothing found" and "everything was blocked" read differently.+- **The web-search skill (1.3.0) explains the sources** and tells the model to report which ones failed and why when a search comes back empty.+- **You can see what happened.** Every result ends with the trail, for example `Searched: Parallel: timed out → DuckDuckGo: 8 results`, so "nothing found" and "everything was blocked" read differently.### Chat- **The research prompt asks only when you want research.** It used to fire on words ("latest", "recent", "explain", "today's news"), so "convert the latest epub" or "explain how my build works and fix it" got a research question. The words now only decide whether to check; one short model call decides whether you asked for an in-depth, multi-source write-up. Instructions, quick questions and today's news never get the prompt, and a failed or slow check means no prompt.diff --git a/src/capabilities/web/fixtures/ddg-html.html b/src/capabilities/web/fixtures/ddg-html.htmlnew file mode 100644index 00000000..0e99a3a6--- /dev/null+++ b/src/capabilities/web/fixtures/ddg-html.html@@ -0,0 +1,626 @@+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">++<!--[if IE 6]><html class="ie6" xmlns="http://www.w3.org/1999/xhtml"><![endif]-->+<!--[if IE 7]><html class="lt-ie8 lt-ie9" xmlns="http://www.w3.org/1999/xhtml"><![endif]-->+<!--[if IE 8]><html class="lt-ie9" xmlns="http://www.w3.org/1999/xhtml"><![endif]-->+<!--[if gt IE 8]><!--><html xmlns="http://www.w3.org/1999/xhtml"><!--<![endif]-->+<head>+ <meta http-equiv="content-type" content="text/html; charset=UTF-8" />+ <meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=3.0, user-scalable=1" />+ <meta name="referrer" content="origin" />+ <meta name="HandheldFriendly" content="true" />+ <meta name="robots" content="noindex, nofollow" />+ <title>pandoc epub to markdown at DuckDuckGo</title>+ <link title="DuckDuckGo (HTML)" type="application/opensearchdescription+xml" rel="search" href="//duckduckgo.com/opensearch_html_v2.xml" />+ <link href="//duckduckgo.com/favicon.ico" rel="shortcut icon" />+ <link rel="icon" href="//duckduckgo.com/favicon.ico" type="image/x-icon" />+ <link id="icon60" rel="apple-touch-icon" href="//duckduckgo.com/assets/icons/meta/DDG-iOS-icon_60x60.png?v=2"/>+ <link id="icon76" rel="apple-touch-icon" sizes="76x76" href="//duckduckgo.com/assets/icons/meta/DDG-iOS-icon_76x76.png?v=2"/>+ <link id="icon120" rel="apple-touch-icon" sizes="120x120" href="//duckduckgo.com/assets/icons/meta/DDG-iOS-icon_120x120.png?v=2"/>+ <link id="icon152" rel="apple-touch-icon" sizes="152x152" href="//duckduckgo.com/assets/icons/meta/DDG-iOS-icon_152x152.png?v=2"/>+ <link rel="image_src" href="//duckduckgo.com/assets/icons/meta/DDG-icon_256x256.png">+ <link rel="stylesheet" media="handheld, all" href="//duckduckgo.com/dist/h.12e359033b8c1146e5f7.css" type="text/css"/>+</head>++<body class="body--html">+ <a name="top" id="top"></a>++ <form action="/html/" method="post">+ <input type="text" name="state_hidden" id="state_hidden" />+ </form>++ <div>+ <div class="site-wrapper-border"></div>++ <div id="header" class="header cw header--html">+ <a title="DuckDuckGo" href="/html/" class="header__logo-wrap"></a>++ <form name="x" class="header__form" action="/html/" method="post">+ <div class="search search--header">+ <input name="q" autocomplete="off" class="search__input" id="search_form_input_homepage" type="text" value="pandoc epub to markdown" />+ <input name="b" id="search_button_homepage" class="search__button search__button--html" value="" title="Search" alt="Search" type="submit" />+ </div>+++++++ <div class="frm__select">+ <select name="kl">++ <option value="" >All Regions</option>++ <option value="ar-es" >Argentina</option>++ <option value="au-en" >Australia</option>++ <option value="at-de" >Austria</option>++ <option value="be-fr" >Belgium (fr)</option>++ <option value="be-nl" >Belgium (nl)</option>++ <option value="br-pt" >Brazil</option>++ <option value="bg-bg" >Bulgaria</option>++ <option value="ca-en" >Canada (en)</option>++ <option value="ca-fr" >Canada (fr)</option>++ <option value="ct-ca" >Catalonia</option>++ <option value="cl-es" >Chile</option>++ <option value="cn-zh" >China</option>++ <option value="co-es" >Colombia</option>++ <option value="hr-hr" >Croatia</option>++ <option value="cz-cs" >Czech Republic</option>++ <option value="dk-da" >Denmark</option>++ <option value="ee-et" >Estonia</option>++ <option value="fi-fi" >Finland</option>++ <option value="fr-fr" >France</option>++ <option value="de-de" >Germany</option>++ <option value="gr-el" >Greece</option>++ <option value="hk-tzh" >Hong Kong</option>++ <option value="hu-hu" >Hungary</option>++ <option value="is-is" >Iceland</option>++ <option value="in-en" >India (en)</option>++ <option value="id-en" >Indonesia (en)</option>++ <option value="ie-en" >Ireland</option>++ <option value="il-en" >Israel (en)</option>++ <option value="it-it" >Italy</option>++ <option value="jp-jp" >Japan</option>++ <option value="kr-kr" >Korea</option>++ <option value="lv-lv" >Latvia</option>++ <option value="lt-lt" >Lithuania</option>++ <option value="my-en" >Malaysia (en)</option>++ <option value="mx-es" >Mexico</option>++ <option value="nl-nl" >Netherlands</option>++ <option value="nz-en" >New Zealand</option>++ <option value="no-no" >Norway</option>++ <option value="pk-en" >Pakistan (en)</option>++ <option value="pe-es" >Peru</option>++ <option value="ph-en" >Philippines (en)</option>++ <option value="pl-pl" >Poland</option>++ <option value="pt-pt" >Portugal</option>++ <option value="ro-ro" >Romania</option>++ <option value="ru-ru" >Russia</option>++ <option value="xa-ar" >Saudi Arabia</option>++ <option value="sg-en" >Singapore</option>++ <option value="sk-sk" >Slovakia</option>++ <option value="sl-sl" >Slovenia</option>++ <option value="za-en" >South Africa</option>++ <option value="es-ca" >Spain (ca)</option>++ <option value="es-es" >Spain (es)</option>++ <option value="se-sv" >Sweden</option>++ <option value="ch-de" >Switzerland (de)</option>++ <option value="ch-fr" >Switzerland (fr)</option>++ <option value="tw-tzh" >Taiwan</option>++ <option value="th-en" >Thailand (en)</option>++ <option value="tr-tr" >Turkey</option>++ <option value="us-en" >US (English)</option>++ <option value="us-es" >US (Spanish)</option>++ <option value="ua-uk" >Ukraine</option>++ <option value="uk-en" >United Kingdom</option>++ <option value="vn-en" >Vietnam (en)</option>++ </select>+ </div>++ <div class="frm__select frm__select--last">+ <select class="" name="df">++ <option value="" selected>Any Time</option>++ <option value="d" >Past Day</option>++ <option value="w" >Past Week</option>++ <option value="m" >Past Month</option>++ <option value="y" >Past Year</option>++ </select>+ </div>+ </form>+ </div>+++++ <!-- Web results are present -->+ <div>+ <div class="serp__results">+ <div id="links" class="results">+++++ <div class="result results_links results_links_deep web-result ">+ <div class="links_main links_deep result__body"> <!-- This is the visible part -->++ <h2 class="result__title">+ <a rel="nofollow" class="result__a" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fwww.reddit.com%2Fr%2Fpandoc%2Fcomments%2Fsg6sqx%2Fconvert_epub_to_markdown%2F&rut=77770eaab0e7a6560c5e5f6bbbe12bcb2c8b4338d157bd0f6381eb95a455fd1d">Convert epub to markdown : r/pandoc - Reddit</a>+ </h2>+++++ <div class="result__extras">+ <div class="result__extras__url">+ <span class="result__icon">+ <a rel="nofollow" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fwww.reddit.com%2Fr%2Fpandoc%2Fcomments%2Fsg6sqx%2Fconvert_epub_to_markdown%2F&rut=77770eaab0e7a6560c5e5f6bbbe12bcb2c8b4338d157bd0f6381eb95a455fd1d">+ <img class="result__icon__img" width="16" height="16" alt="" src="//external-content.duckduckgo.com/ip3/www.reddit.com.ico" name="i15" />+ </a>+ </span>+ <a class="result__url" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fwww.reddit.com%2Fr%2Fpandoc%2Fcomments%2Fsg6sqx%2Fconvert_epub_to_markdown%2F&rut=77770eaab0e7a6560c5e5f6bbbe12bcb2c8b4338d157bd0f6381eb95a455fd1d">+ www.reddit.com/r/pandoc/comments/sg6sqx/convert_epub_to_markdown/+ </a>++ </div>+ </div>+++++ <a class="result__snippet" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fwww.reddit.com%2Fr%2Fpandoc%2Fcomments%2Fsg6sqx%2Fconvert_epub_to_markdown%2F&rut=77770eaab0e7a6560c5e5f6bbbe12bcb2c8b4338d157bd0f6381eb95a455fd1d">Hi, I want to convert an ebook in <b>epub</b> format to <b>markdown</b>. I want to improve text formatting, add syntax highlighting for source code and improve other stuff. Than I want to convert it back to <b>epub</b>. How can I do this in <b>Pandoc</b>?</a>++++ <div class="clear"></div>+ </div>+ </div>++++ <div class="result results_links results_links_deep web-result ">+ <div class="links_main links_deep result__body"> <!-- This is the visible part -->++ <h2 class="result__title">+ <a rel="nofollow" class="result__a" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fpandoc.org%2Fepub.html&rut=8ae184922fd55e2ca39c4dcc6050bdb50aa30b59fdc4146d85ba38160c8a5da6">Pandoc - Creating an ebook with pandoc</a>+ </h2>+++++ <div class="result__extras">+ <div class="result__extras__url">+ <span class="result__icon">+ <a rel="nofollow" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fpandoc.org%2Fepub.html&rut=8ae184922fd55e2ca39c4dcc6050bdb50aa30b59fdc4146d85ba38160c8a5da6">+ <img class="result__icon__img" width="16" height="16" alt="" src="//external-content.duckduckgo.com/ip3/pandoc.org.ico" name="i15" />+ </a>+ </span>+ <a class="result__url" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fpandoc.org%2Fepub.html&rut=8ae184922fd55e2ca39c4dcc6050bdb50aa30b59fdc4146d85ba38160c8a5da6">+ pandoc.org/epub.html+ </a>++ </div>+ </div>+++++ <a class="result__snippet" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fpandoc.org%2Fepub.html&rut=8ae184922fd55e2ca39c4dcc6050bdb50aa30b59fdc4146d85ba38160c8a5da6">Creating an ebook with <b>pandoc</b> Starting with version 1.6, <b>pandoc</b> can produce output in the <b>EPUB</b> electronic book format. <b>EPUB</b> books can be viewed on iPads, Nooks, and other electronic book readers, including many smart phones. (They can also be converted to Kindle books using the GUI only KindlePreviewer on Windows and Mac OSX.</a>++++ <div class="clear"></div>+ </div>+ </div>++++ <div class="result results_links results_links_deep web-result ">+ <div class="links_main links_deep result__body"> <!-- This is the visible part -->++ <h2 class="result__title">+ <a rel="nofollow" class="result__a" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fgithub.com%2Fadamdexter%2Fepub2md%2Dpandoc&rut=c5fb60367c86f2835a4eef2b4229e6bd6b50e89ce62759409fe4a6c38e9f6972">GitHub - adamdexter/epub2md-pandoc: A simple tool that converts epub ...</a>+ </h2>+++++ <div class="result__extras">+ <div class="result__extras__url">+ <span class="result__icon">+ <a rel="nofollow" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fgithub.com%2Fadamdexter%2Fepub2md%2Dpandoc&rut=c5fb60367c86f2835a4eef2b4229e6bd6b50e89ce62759409fe4a6c38e9f6972">+ <img class="result__icon__img" width="16" height="16" alt="" src="//external-content.duckduckgo.com/ip3/github.com.ico" name="i15" />+ </a>+ </span>+ <a class="result__url" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fgithub.com%2Fadamdexter%2Fepub2md%2Dpandoc&rut=c5fb60367c86f2835a4eef2b4229e6bd6b50e89ce62759409fe4a6c38e9f6972">+ github.com/adamdexter/epub2md-pandoc+ </a>++ </div>+ </div>+++++ <a class="result__snippet" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fgithub.com%2Fadamdexter%2Fepub2md%2Dpandoc&rut=c5fb60367c86f2835a4eef2b4229e6bd6b50e89ce62759409fe4a6c38e9f6972">A simple tool that converts <b>epub</b> files to <b>mark</b> <b>down,</b> optimizing for use as Claude AI project knowledge and RAG. - adamdexter/epub2md-<b>pandoc</b></a>++++ <div class="clear"></div>+ </div>+ </div>++++ <div class="result results_links results_links_deep web-result ">+ <div class="links_main links_deep result__body"> <!-- This is the visible part -->++ <h2 class="result__title">+ <a rel="nofollow" class="result__a" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fezmarkdown.com%2Fepub%2Dto%2Dmarkdown&rut=74d41cc52b469f5b6b2aef8f8274be33b77f6c75893ff0076fec3cc38946b378">EPUB to Markdown Converter Online | EZMarkdown</a>+ </h2>+++++ <div class="result__extras">+ <div class="result__extras__url">+ <span class="result__icon">+ <a rel="nofollow" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fezmarkdown.com%2Fepub%2Dto%2Dmarkdown&rut=74d41cc52b469f5b6b2aef8f8274be33b77f6c75893ff0076fec3cc38946b378">+ <img class="result__icon__img" width="16" height="16" alt="" src="//external-content.duckduckgo.com/ip3/ezmarkdown.com.ico" name="i15" />+ </a>+ </span>+ <a class="result__url" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fezmarkdown.com%2Fepub%2Dto%2Dmarkdown&rut=74d41cc52b469f5b6b2aef8f8274be33b77f6c75893ff0076fec3cc38946b378">+ ezmarkdown.com/epub-to-markdown+ </a>++ </div>+ </div>+++++ <a class="result__snippet" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fezmarkdown.com%2Fepub%2Dto%2Dmarkdown&rut=74d41cc52b469f5b6b2aef8f8274be33b77f6c75893ff0076fec3cc38946b378">Recover editable <b>Markdown</b> from <b>EPUB</b> ebook content. Does this run locally in my browser? This conversion uses the server-side <b>Pandoc</b> engine. File size, request frequency, and execution time are limited, and temporary files are cleaned up after conversion.</a>++++ <div class="clear"></div>+ </div>+ </div>++++ <div class="result results_links results_links_deep web-result ">+ <div class="links_main links_deep result__body"> <!-- This is the visible part -->++ <h2 class="result__title">+ <a rel="nofollow" class="result__a" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fpandoc.org%2FMANUAL.html&rut=a1ad61b56e5cbe61eef13c19bb1f138ef42e6086b919173b29c82cfd0e9420c4">Pandoc - Pandoc User's Guide</a>+ </h2>+++++ <div class="result__extras">+ <div class="result__extras__url">+ <span class="result__icon">+ <a rel="nofollow" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fpandoc.org%2FMANUAL.html&rut=a1ad61b56e5cbe61eef13c19bb1f138ef42e6086b919173b29c82cfd0e9420c4">+ <img class="result__icon__img" width="16" height="16" alt="" src="//external-content.duckduckgo.com/ip3/pandoc.org.ico" name="i15" />+ </a>+ </span>+ <a class="result__url" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fpandoc.org%2FMANUAL.html&rut=a1ad61b56e5cbe61eef13c19bb1f138ef42e6086b919173b29c82cfd0e9420c4">+ pandoc.org/MANUAL.html+ </a>++ </div>+ </div>+++++ <a class="result__snippet" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fpandoc.org%2FMANUAL.html&rut=a1ad61b56e5cbe61eef13c19bb1f138ef42e6086b919173b29c82cfd0e9420c4"><b>Pandoc</b> User's Guide Synopsis <b>pandoc</b> [options] [input-file]… Description <b>Pandoc</b> is a Haskell library for converting from one markup format to another, and a command-line tool that uses this library. <b>Pandoc</b> can convert between numerous markup and word processing formats, including, but not limited <b>to</b>, various flavors of <b>Markdown</b>, HTML, LaTeX and Word docx. For the full lists of input and ...</a>++++ <div class="clear"></div>+ </div>+ </div>++++ <div class="result results_links results_links_deep web-result ">+ <div class="links_main links_deep result__body"> <!-- This is the visible part -->++ <h2 class="result__title">+ <a rel="nofollow" class="result__a" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fskills.rest%2Fskill%2Fepub%2Dto%2Dmarkdown&rut=b7bb374cafac6b375bc3be4241705f4a0f65d1a4c80a7500623f7f469d894c28">epub-to-markdown: Convert EPUB ebook files to Markdown using pandoc ...</a>+ </h2>+++++ <div class="result__extras">+ <div class="result__extras__url">+ <span class="result__icon">+ <a rel="nofollow" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fskills.rest%2Fskill%2Fepub%2Dto%2Dmarkdown&rut=b7bb374cafac6b375bc3be4241705f4a0f65d1a4c80a7500623f7f469d894c28">+ <img class="result__icon__img" width="16" height="16" alt="" src="//external-content.duckduckgo.com/ip3/skills.rest.ico" name="i15" />+ </a>+ </span>+ <a class="result__url" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fskills.rest%2Fskill%2Fepub%2Dto%2Dmarkdown&rut=b7bb374cafac6b375bc3be4241705f4a0f65d1a4c80a7500623f7f469d894c28">+ skills.rest/skill/epub-to-markdown+ </a>++ </div>+ </div>+++++ <a class="result__snippet" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fskills.rest%2Fskill%2Fepub%2Dto%2Dmarkdown&rut=b7bb374cafac6b375bc3be4241705f4a0f65d1a4c80a7500623f7f469d894c28">Convert <b>EPUB</b> ebook files to <b>Markdown</b> using <b>pandoc</b> with gfm-raw_html output. Browse installation instructions, technical dependencies, and related skills by zocomputer.</a>++++ <div class="clear"></div>+ </div>+ </div>++++ <div class="result results_links results_links_deep web-result ">+ <div class="links_main links_deep result__body"> <!-- This is the visible part -->++ <h2 class="result__title">+ <a rel="nofollow" class="result__a" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fgithub.com%2Fuxiew%2Fepub2MD&rut=ae73de5c4ab2198b823ec9f2bfe711280afeeb4445e7629c5f302fcef54c0339">GitHub - uxiew/epub2MD: convert epub file to markdown files</a>+ </h2>+++++ <div class="result__extras">+ <div class="result__extras__url">+ <span class="result__icon">+ <a rel="nofollow" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fgithub.com%2Fuxiew%2Fepub2MD&rut=ae73de5c4ab2198b823ec9f2bfe711280afeeb4445e7629c5f302fcef54c0339">+ <img class="result__icon__img" width="16" height="16" alt="" src="//external-content.duckduckgo.com/ip3/github.com.ico" name="i15" />+ </a>+ </span>+ <a class="result__url" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fgithub.com%2Fuxiew%2Fepub2MD&rut=ae73de5c4ab2198b823ec9f2bfe711280afeeb4445e7629c5f302fcef54c0339">+ github.com/uxiew/epub2MD+ </a>++ </div>+ </div>+++++ <a class="result__snippet" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fgithub.com%2Fuxiew%2Fepub2MD&rut=ae73de5c4ab2198b823ec9f2bfe711280afeeb4445e7629c5f302fcef54c0339">convert <b>epub</b> file to <b>markdown</b> files. Contribute to uxiew/epub2MD development by creating an account on GitHub.</a>++++ <div class="clear"></div>+ </div>+ </div>++++ <div class="result results_links results_links_deep web-result ">+ <div class="links_main links_deep result__body"> <!-- This is the visible part -->++ <h2 class="result__title">+ <a rel="nofollow" class="result__a" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fcommunity.obsidian.md%2Fplugins%2Fepub%2Dreading%2Dimporter&rut=1b4b47c30121be800fa6fbe6db817faf39d2a670217001550f672913d0a03641">EPUB Transfer to Markdown - Obsidian Plugin</a>+ </h2>+++++ <div class="result__extras">+ <div class="result__extras__url">+ <span class="result__icon">+ <a rel="nofollow" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fcommunity.obsidian.md%2Fplugins%2Fepub%2Dreading%2Dimporter&rut=1b4b47c30121be800fa6fbe6db817faf39d2a670217001550f672913d0a03641">+ <img class="result__icon__img" width="16" height="16" alt="" src="//external-content.duckduckgo.com/ip3/community.obsidian.md.ico" name="i15" />+ </a>+ </span>+ <a class="result__url" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fcommunity.obsidian.md%2Fplugins%2Fepub%2Dreading%2Dimporter&rut=1b4b47c30121be800fa6fbe6db817faf39d2a670217001550f672913d0a03641">+ community.obsidian.md/plugins/epub-reading-importer+ </a>++ </div>+ </div>+++++ <a class="result__snippet" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fcommunity.obsidian.md%2Fplugins%2Fepub%2Dreading%2Dimporter&rut=1b4b47c30121be800fa6fbe6db817faf39d2a670217001550f672913d0a03641">It does use local filesystem access to read the selected <b>EPUB</b>, write <b>Markdown</b> files, and export <b>EPUB</b> files into your vault. It also runs local command-line tools such as <b>pandoc</b>, and on Windows it may run PowerShell for <b>EPUB</b> extraction or Outlook draft creation.</a>++++ <div class="clear"></div>+ </div>+ </div>++++ <div class="result results_links results_links_deep web-result ">+ <div class="links_main links_deep result__body"> <!-- This is the visible part -->++ <h2 class="result__title">+ <a rel="nofollow" class="result__a" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fwww.zamzar.com%2Fconvert%2Fepub%2Dto%2Dmd%2F&rut=6b2b736760912eda87501188d6f0ab96bc0822bcddcdbbac68b5a9600a3edcfc">EPUB to MD - Convert your EPUB to MD for Free Online - Zamzar</a>+ </h2>+++++ <div class="result__extras">+ <div class="result__extras__url">+ <span class="result__icon">+ <a rel="nofollow" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fwww.zamzar.com%2Fconvert%2Fepub%2Dto%2Dmd%2F&rut=6b2b736760912eda87501188d6f0ab96bc0822bcddcdbbac68b5a9600a3edcfc">+ <img class="result__icon__img" width="16" height="16" alt="" src="//external-content.duckduckgo.com/ip3/www.zamzar.com.ico" name="i15" />+ </a>+ </span>+ <a class="result__url" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fwww.zamzar.com%2Fconvert%2Fepub%2Dto%2Dmd%2F&rut=6b2b736760912eda87501188d6f0ab96bc0822bcddcdbbac68b5a9600a3edcfc">+ www.zamzar.com/convert/epub-to-md/+ </a>++ </div>+ </div>+++++ <a class="result__snippet" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fwww.zamzar.com%2Fconvert%2Fepub%2Dto%2Dmd%2F&rut=6b2b736760912eda87501188d6f0ab96bc0822bcddcdbbac68b5a9600a3edcfc">Do you want to convert a <b>EPUB</b> file to a MD file ? Don't download software - use Zamzar to convert it for free online. Click to convert your <b>EPUB</b> file now.</a>++++ <div class="clear"></div>+ </div>+ </div>++++ <div class="result results_links results_links_deep web-result ">+ <div class="links_main links_deep result__body"> <!-- This is the visible part -->++ <h2 class="result__title">+ <a rel="nofollow" class="result__a" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fuseotools.com%2Fepub%2Dto%2Dmarkdown&rut=7a6f1d7b9df5b6c6d28eb4567a15a9fafd5f2a31e4d235ab3319b15c2eb33846">EPUB to Markdown Converter - Free, Formatting Kept</a>+ </h2>+++++ <div class="result__extras">+ <div class="result__extras__url">+ <span class="result__icon">+ <a rel="nofollow" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fuseotools.com%2Fepub%2Dto%2Dmarkdown&rut=7a6f1d7b9df5b6c6d28eb4567a15a9fafd5f2a31e4d235ab3319b15c2eb33846">+ <img class="result__icon__img" width="16" height="16" alt="" src="//external-content.duckduckgo.com/ip3/useotools.com.ico" name="i15" />+ </a>+ </span>+ <a class="result__url" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fuseotools.com%2Fepub%2Dto%2Dmarkdown&rut=7a6f1d7b9df5b6c6d28eb4567a15a9fafd5f2a31e4d235ab3319b15c2eb33846">+ useotools.com/epub-to-markdown+ </a>++ </div>+ </div>+++++ <a class="result__snippet" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fuseotools.com%2Fepub%2Dto%2Dmarkdown&rut=7a6f1d7b9df5b6c6d28eb4567a15a9fafd5f2a31e4d235ab3319b15c2eb33846">The words of a book as plain text An <b>EPUB</b> is a zipped set of web pages, awkward to edit or search outside a reader. <b>Pandoc</b> reads the chapters in order and writes one GitHub-flavoured <b>Markdown</b> file with headings, paragraphs, block quotes, lists and emphasis, which opens in any text editor, note-taking app or writing tool.</a>++++ <div class="clear"></div>+ </div>+ </div>+++++++ <div class="nav-link">+ <form action="/html/" method="post">+ <input type="submit" class='btn btn--alt' value="Next" />+ <input type="hidden" name="q" value="pandoc epub to markdown" />+ <input type="hidden" name="s" value="10" />+ <input type="hidden" name="nextParams" value="" />+ <input type="hidden" name="v" value="l" />+ <input type="hidden" name="o" value="json" />+ <input type="hidden" name="dc" value="11" />+ <input type="hidden" name="api" value="d.js" />+ <input type="hidden" name="vqd" value="4-142623474753119617845835499511333817539" />+++++ <input name="kl" value="wt-wt" type="hidden" />+++++ </form>+ </div>++++ <div class="feedback-btn">+ <a rel="nofollow" href="//duckduckgo.com/feedback.html" target="_new">Feedback</a>+ </div>+ <div class="clear"></div>+ </div>+ </div>+ </div> <!-- links wrapper //-->++ </div>++ <div id="bottom_spacing2"></div>+++ <img src="//duckduckgo.com/t/sl_h"/>++</body>+</html>\ No newline at end of filediff --git a/src/capabilities/web/fixtures/ddg-lite.html b/src/capabilities/web/fixtures/ddg-lite.htmlnew file mode 100644index 00000000..df2c7d29--- /dev/null+++ b/src/capabilities/web/fixtures/ddg-lite.html@@ -0,0 +1,723 @@+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">+<html>++<head>+ <meta http-equiv="content-type" content="text/html; charset=UTF-8">+ <meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=3.0, user-scalable=1;">+ <meta name="referrer" content="origin">+ <meta name="HandheldFriendly" content="true" />+ <meta name="robots" content="noindex, nofollow" />+ <title>pandoc epub to markdown at DuckDuckGo</title>+ <link title="DuckDuckGo (Lite)" type="application/opensearchdescription+xml" rel="search" href="//duckduckgo.com/opensearch_lite_v2.xml">+ <link href="//duckduckgo.com/favicon.ico" rel="shortcut icon" />+ <link rel="icon" href="//duckduckgo.com/favicon.ico" type="image/x-icon" />+ <link id="icon60" rel="apple-touch-icon" href="//duckduckgo.com/assets/icons/meta/DDG-iOS-icon_60x60.png?v=2"/>+ <link id="icon76" rel="apple-touch-icon" sizes="76x76" href="//duckduckgo.com/assets/icons/meta/DDG-iOS-icon_76x76.png?v=2"/>+ <link id="icon120" rel="apple-touch-icon" sizes="120x120" href="//duckduckgo.com/assets/icons/meta/DDG-iOS-icon_120x120.png?v=2"/>+ <link id="icon152" rel="apple-touch-icon" sizes="152x152" href="//duckduckgo.com/assets/icons/meta/DDG-iOS-icon_152x152.png?v=2"/>+ <link rel="image_src" href="//duckduckgo.com/assets/icons/meta/DDG-icon_256x256.png">+ <link rel="stylesheet" media="handheld, all" href="//duckduckgo.com/dist/lr.48ddfe4eadf6a534e93f.css" type="text/css"/>+</head>++<body>+ <p class='extra'> </p>+ <div class="header">+ DuckDuckGo+ </div>+ <p class='extra'> </p>++ <form action="/lite/" method="post">+ <input class='query' type="text" size="40" name="q" value="pandoc epub to markdown">+ <input class='submit' type="submit" value="Search">++++++ <div class="filters">+ <select class="submit" name="kl">++ <option value="" >All Regions</option>++ <option value="ar-es" >Argentina</option>++ <option value="au-en" >Australia</option>++ <option value="at-de" >Austria</option>++ <option value="be-fr" >Belgium (fr)</option>++ <option value="be-nl" >Belgium (nl)</option>++ <option value="br-pt" >Brazil</option>++ <option value="bg-bg" >Bulgaria</option>++ <option value="ca-en" >Canada (en)</option>++ <option value="ca-fr" >Canada (fr)</option>++ <option value="ct-ca" >Catalonia</option>++ <option value="cl-es" >Chile</option>++ <option value="cn-zh" >China</option>++ <option value="co-es" >Colombia</option>++ <option value="hr-hr" >Croatia</option>++ <option value="cz-cs" >Czech Republic</option>++ <option value="dk-da" >Denmark</option>++ <option value="ee-et" >Estonia</option>++ <option value="fi-fi" >Finland</option>++ <option value="fr-fr" >France</option>++ <option value="de-de" >Germany</option>++ <option value="gr-el" >Greece</option>++ <option value="hk-tzh" >Hong Kong</option>++ <option value="hu-hu" >Hungary</option>++ <option value="is-is" >Iceland</option>++ <option value="in-en" >India (en)</option>++ <option value="id-en" >Indonesia (en)</option>++ <option value="ie-en" >Ireland</option>++ <option value="il-en" >Israel (en)</option>++ <option value="it-it" >Italy</option>++ <option value="jp-jp" >Japan</option>++ <option value="kr-kr" >Korea</option>++ <option value="lv-lv" >Latvia</option>++ <option value="lt-lt" >Lithuania</option>++ <option value="my-en" >Malaysia (en)</option>++ <option value="mx-es" >Mexico</option>++ <option value="nl-nl" >Netherlands</option>++ <option value="nz-en" >New Zealand</option>++ <option value="no-no" >Norway</option>++ <option value="pk-en" >Pakistan (en)</option>++ <option value="pe-es" >Peru</option>++ <option value="ph-en" >Philippines (en)</option>++ <option value="pl-pl" >Poland</option>++ <option value="pt-pt" >Portugal</option>++ <option value="ro-ro" >Romania</option>++ <option value="ru-ru" >Russia</option>++ <option value="xa-ar" >Saudi Arabia</option>++ <option value="sg-en" >Singapore</option>++ <option value="sk-sk" >Slovakia</option>++ <option value="sl-sl" >Slovenia</option>++ <option value="za-en" >South Africa</option>++ <option value="es-ca" >Spain (ca)</option>++ <option value="es-es" >Spain (es)</option>++ <option value="se-sv" >Sweden</option>++ <option value="ch-de" >Switzerland (de)</option>++ <option value="ch-fr" >Switzerland (fr)</option>++ <option value="tw-tzh" >Taiwan</option>++ <option value="th-en" >Thailand (en)</option>++ <option value="tr-tr" >Turkey</option>++ <option value="us-en" >US (English)</option>++ <option value="us-es" >US (Spanish)</option>++ <option value="ua-uk" >Ukraine</option>++ <option value="uk-en" >United Kingdom</option>++ <option value="vn-en" >Vietnam (en)</option>++ </select>++ <select class="submit" name="df">++ <option value="" selected>Any Time</option>++ <option value="d" >Past Day</option>++ <option value="w" >Past Week</option>++ <option value="m" >Past Month</option>++ <option value="y" >Past Year</option>++ </select>+ </div>+ </form>+++ <p class="extra"> </p>+ <table border="0">+ <tr>+ <td>++ </td>+ <td>++ <!-- Next Page Button Sub-template -->+<form class="next_form" action="/lite/" method="post">+ <input type="submit" class='navbutton' value="Next Page >">+ <input type="hidden" name="q" value="pandoc epub to markdown">+ <input type="hidden" name="s" value="10">+ <input type="hidden" name="nextParams" value="">+ <input type="hidden" name="v" value="l">+ <input type="hidden" name="o" value="json">+ <input type="hidden" name="dc" value="11">+ <input type="hidden" name="api" value="d.js">+ <input type="hidden" name="vqd" value="4-142623474753119617845835499511333817539">+ ++++ <input name="kl" value="wt-wt" type="hidden">+++++</form>++ </td>+ </tr>+ </table>+++++ <p class='extra'> </p>++ <table border="0">++ </table>++ <table border="0">+++ <!-- Web results are present -->+++ <tr>++ <td valign="top">++ 1. ++ </td>+ <td>+ <a rel="nofollow" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fwww.reddit.com%2Fr%2Fpandoc%2Fcomments%2Fsg6sqx%2Fconvert_epub_to_markdown%2F&rut=77770eaab0e7a6560c5e5f6bbbe12bcb2c8b4338d157bd0f6381eb95a455fd1d" class='result-link'>Convert epub to markdown : r/pandoc - Reddit</a>+ </td>++ </tr>++++ <!-- Only show abstract separately if there's a click URL (not EOF) -->+ <tr>+ <td> </td>+ <td class='result-snippet'>+ Hi, I want to convert an ebook in <b>epub</b> format to <b>markdown</b>. I want to improve text formatting, add syntax highlighting for source code and improve other stuff. Than I want to convert it back to <b>epub</b>. How can I do this in <b>Pandoc</b>?+ </td>+ </tr>+++++ <tr>+ <td> </td>+ <td>+ <span class='link-text'>www.reddit.com/r/pandoc/comments/sg6sqx/convert_epub_to_markdown/</span>++ </td>+ </tr>+++ <tr>+ <td> </td>+ <td> </td>+ </tr>+++++ <tr>++ <td valign="top">++ 2. ++ </td>+ <td>+ <a rel="nofollow" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fpandoc.org%2Fepub.html&rut=8ae184922fd55e2ca39c4dcc6050bdb50aa30b59fdc4146d85ba38160c8a5da6" class='result-link'>Pandoc - Creating an ebook with pandoc</a>+ </td>++ </tr>++++ <!-- Only show abstract separately if there's a click URL (not EOF) -->+ <tr>+ <td> </td>+ <td class='result-snippet'>+ Creating an ebook with <b>pandoc</b> Starting with version 1.6, <b>pandoc</b> can produce output in the <b>EPUB</b> electronic book format. <b>EPUB</b> books can be viewed on iPads, Nooks, and other electronic book readers, including many smart phones. (They can also be converted to Kindle books using the GUI only KindlePreviewer on Windows and Mac OSX.+ </td>+ </tr>+++++ <tr>+ <td> </td>+ <td>+ <span class='link-text'>pandoc.org/epub.html</span>++ </td>+ </tr>+++ <tr>+ <td> </td>+ <td> </td>+ </tr>+++++ <tr>++ <td valign="top">++ 3. ++ </td>+ <td>+ <a rel="nofollow" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fgithub.com%2Fadamdexter%2Fepub2md%2Dpandoc&rut=c5fb60367c86f2835a4eef2b4229e6bd6b50e89ce62759409fe4a6c38e9f6972" class='result-link'>GitHub - adamdexter/epub2md-pandoc: A simple tool that converts epub ...</a>+ </td>++ </tr>++++ <!-- Only show abstract separately if there's a click URL (not EOF) -->+ <tr>+ <td> </td>+ <td class='result-snippet'>+ A simple tool that converts <b>epub</b> files to <b>mark</b> <b>down,</b> optimizing for use as Claude AI project knowledge and RAG. - adamdexter/epub2md-<b>pandoc</b>+ </td>+ </tr>+++++ <tr>+ <td> </td>+ <td>+ <span class='link-text'>github.com/adamdexter/epub2md-pandoc</span>++ </td>+ </tr>+++ <tr>+ <td> </td>+ <td> </td>+ </tr>+++++ <tr>++ <td valign="top">++ 4. ++ </td>+ <td>+ <a rel="nofollow" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fezmarkdown.com%2Fepub%2Dto%2Dmarkdown&rut=74d41cc52b469f5b6b2aef8f8274be33b77f6c75893ff0076fec3cc38946b378" class='result-link'>EPUB to Markdown Converter Online | EZMarkdown</a>+ </td>++ </tr>++++ <!-- Only show abstract separately if there's a click URL (not EOF) -->+ <tr>+ <td> </td>+ <td class='result-snippet'>+ Recover editable <b>Markdown</b> from <b>EPUB</b> ebook content. Does this run locally in my browser? This conversion uses the server-side <b>Pandoc</b> engine. File size, request frequency, and execution time are limited, and temporary files are cleaned up after conversion.+ </td>+ </tr>+++++ <tr>+ <td> </td>+ <td>+ <span class='link-text'>ezmarkdown.com/epub-to-markdown</span>++ </td>+ </tr>+++ <tr>+ <td> </td>+ <td> </td>+ </tr>+++++ <tr>++ <td valign="top">++ 5. ++ </td>+ <td>+ <a rel="nofollow" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fpandoc.org%2FMANUAL.html&rut=a1ad61b56e5cbe61eef13c19bb1f138ef42e6086b919173b29c82cfd0e9420c4" class='result-link'>Pandoc - Pandoc User's Guide</a>+ </td>++ </tr>++++ <!-- Only show abstract separately if there's a click URL (not EOF) -->+ <tr>+ <td> </td>+ <td class='result-snippet'>+ <b>Pandoc</b> User's Guide Synopsis <b>pandoc</b> [options] [input-file]… Description <b>Pandoc</b> is a Haskell library for converting from one markup format to another, and a command-line tool that uses this library. <b>Pandoc</b> can convert between numerous markup and word processing formats, including, but not limited <b>to</b>, various flavors of <b>Markdown</b>, HTML, LaTeX and Word docx. For the full lists of input and ...+ </td>+ </tr>+++++ <tr>+ <td> </td>+ <td>+ <span class='link-text'>pandoc.org/MANUAL.html</span>++ </td>+ </tr>+++ <tr>+ <td> </td>+ <td> </td>+ </tr>+++++ <tr>++ <td valign="top">++ 6. ++ </td>+ <td>+ <a rel="nofollow" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fskills.rest%2Fskill%2Fepub%2Dto%2Dmarkdown&rut=b7bb374cafac6b375bc3be4241705f4a0f65d1a4c80a7500623f7f469d894c28" class='result-link'>epub-to-markdown: Convert EPUB ebook files to Markdown using pandoc ...</a>+ </td>++ </tr>++++ <!-- Only show abstract separately if there's a click URL (not EOF) -->+ <tr>+ <td> </td>+ <td class='result-snippet'>+ Convert <b>EPUB</b> ebook files to <b>Markdown</b> using <b>pandoc</b> with gfm-raw_html output. Browse installation instructions, technical dependencies, and related skills by zocomputer.+ </td>+ </tr>+++++ <tr>+ <td> </td>+ <td>+ <span class='link-text'>skills.rest/skill/epub-to-markdown</span>++ </td>+ </tr>+++ <tr>+ <td> </td>+ <td> </td>+ </tr>+++++ <tr>++ <td valign="top">++ 7. ++ </td>+ <td>+ <a rel="nofollow" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fcommunity.obsidian.md%2Fplugins%2Fepub%2Dreading%2Dimporter&rut=1b4b47c30121be800fa6fbe6db817faf39d2a670217001550f672913d0a03641" class='result-link'>EPUB Transfer to Markdown - Obsidian Plugin</a>+ </td>++ </tr>++++ <!-- Only show abstract separately if there's a click URL (not EOF) -->+ <tr>+ <td> </td>+ <td class='result-snippet'>+ It does use local filesystem access to read the selected <b>EPUB</b>, write <b>Markdown</b> files, and export <b>EPUB</b> files into your vault. It also runs local command-line tools such as <b>pandoc</b>, and on Windows it may run PowerShell for <b>EPUB</b> extraction or Outlook draft creation.+ </td>+ </tr>+++++ <tr>+ <td> </td>+ <td>+ <span class='link-text'>community.obsidian.md/plugins/epub-reading-importer</span>++ </td>+ </tr>+++ <tr>+ <td> </td>+ <td> </td>+ </tr>+++++ <tr>++ <td valign="top">++ 8. ++ </td>+ <td>+ <a rel="nofollow" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fgithub.com%2Fuxiew%2Fepub2MD&rut=ae73de5c4ab2198b823ec9f2bfe711280afeeb4445e7629c5f302fcef54c0339" class='result-link'>GitHub - uxiew/epub2MD: convert epub file to markdown files</a>+ </td>++ </tr>++++ <!-- Only show abstract separately if there's a click URL (not EOF) -->+ <tr>+ <td> </td>+ <td class='result-snippet'>+ convert <b>epub</b> file to <b>markdown</b> files. Contribute to uxiew/epub2MD development by creating an account on GitHub.+ </td>+ </tr>+++++ <tr>+ <td> </td>+ <td>+ <span class='link-text'>github.com/uxiew/epub2MD</span>++ </td>+ </tr>+++ <tr>+ <td> </td>+ <td> </td>+ </tr>+++++ <tr>++ <td valign="top">++ 9. ++ </td>+ <td>+ <a rel="nofollow" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fwww.zamzar.com%2Fconvert%2Fepub%2Dto%2Dmd%2F&rut=6b2b736760912eda87501188d6f0ab96bc0822bcddcdbbac68b5a9600a3edcfc" class='result-link'>EPUB to MD - Convert your EPUB to MD for Free Online - Zamzar</a>+ </td>++ </tr>++++ <!-- Only show abstract separately if there's a click URL (not EOF) -->+ <tr>+ <td> </td>+ <td class='result-snippet'>+ Do you want to convert a <b>EPUB</b> file to a MD file ? Don't download software - use Zamzar to convert it for free online. Click to convert your <b>EPUB</b> file now.+ </td>+ </tr>+++++ <tr>+ <td> </td>+ <td>+ <span class='link-text'>www.zamzar.com/convert/epub-to-md/</span>++ </td>+ </tr>+++ <tr>+ <td> </td>+ <td> </td>+ </tr>+++++ <tr>++ <td valign="top">++ 10. ++ </td>+ <td>+ <a rel="nofollow" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fuseotools.com%2Fepub%2Dto%2Dmarkdown&rut=7a6f1d7b9df5b6c6d28eb4567a15a9fafd5f2a31e4d235ab3319b15c2eb33846" class='result-link'>EPUB to Markdown Converter - Free, Formatting Kept</a>+ </td>++ </tr>++++ <!-- Only show abstract separately if there's a click URL (not EOF) -->+ <tr>+ <td> </td>+ <td class='result-snippet'>+ The words of a book as plain text An <b>EPUB</b> is a zipped set of web pages, awkward to edit or search outside a reader. <b>Pandoc</b> reads the chapters in order and writes one GitHub-flavoured <b>Markdown</b> file with headings, paragraphs, block quotes, lists and emphasis, which opens in any text editor, note-taking app or writing tool.+ </td>+ </tr>+++++ <tr>+ <td> </td>+ <td>+ <span class='link-text'>useotools.com/epub-to-markdown</span>++ </td>+ </tr>+++ <tr>+ <td> </td>+ <td> </td>+ </tr>++++++++ <tr>+ <td colspan=2>+++ <!-- Next Page Button Sub-template -->+<form class="next_form" action="/lite/" method="post">+ <input type="submit" class='navbutton' value="Next Page >">+ <input type="hidden" name="q" value="pandoc epub to markdown">+ <input type="hidden" name="s" value="10">+ <input type="hidden" name="nextParams" value="">+ <input type="hidden" name="v" value="l">+ <input type="hidden" name="o" value="json">+ <input type="hidden" name="dc" value="11">+ <input type="hidden" name="api" value="d.js">+ <input type="hidden" name="vqd" value="4-142623474753119617845835499511333817539">+ ++++ <input name="kl" value="wt-wt" type="hidden">+++++</form>+++ </td>+ </tr>++ </table>++ <p class='extra'> </p>++ <form action="/lite/" method="post">+ <input class='query' type="text" size="40" name="q" value="pandoc epub to markdown" >+ <input class='submit' type="submit" value="Search">++++++ <input name="kl" value="wt-wt" type="hidden">++ </form>++ <p class='extra'> </p>+++ <img src="//duckduckgo.com/t/sl_l"/>++ <div id='end-spacer'> </div>++</body>+</html>\ No newline at end of filediff --git a/src/capabilities/web/search-redirect.test.ts b/src/capabilities/web/search-redirect.test.tsindex b99b8be0..d4bf997b 100644--- a/src/capabilities/web/search-redirect.test.ts+++ b/src/capabilities/web/search-redirect.test.ts@@ -1,16 +1,14 @@import { describe, expect, it, vi } from 'vitest';-const hit = (url: string, title: string) => `<a href="${url}" target="_self" class="l1"><div class="title search-snippet-title" title="${title}">x</div></a><div class="content">AI news today.</div>`;-const BRAVE = [- hit('https://www.theverge.com/ai', 'Latest AI news - The Verge'),- hit('https://www.reuters.com/technology/ai', 'AI News | Reuters'),- hit('https://www.technologyreview.com/ai', 'AI - MIT Technology Review'),- hit('https://techcrunch.com/ai', 'AI news | TechCrunch'),-].join('');+const PARALLEL_REPLY = JSON.stringify({ jsonrpc: '2.0', id: 1, result: { structuredContent: { results: [+ { url: 'https://www.theverge.com/ai', title: 'Latest AI news - The Verge', excerpts: ['All the AI news.'] },+ { url: 'https://www.reuters.com/technology/ai', title: 'AI News | Reuters', excerpts: ['AI news today.'] },+ { url: 'https://www.technologyreview.com/ai', title: 'AI - MIT Technology Review', excerpts: ['AI news today.'] },+ { url: 'https://techcrunch.com/ai', title: 'AI news | TechCrunch', excerpts: ['AI news today.'] },+] } } });vi.mock('../../utils/ssrf.js', () => ({guardedFetch: vi.fn(async (url: string) => {- if (url.includes('search.brave.com')) return new Response(BRAVE, { status: 200 });if (url.includes('duckduckgo.com')) return new Response('<html>anomaly challenge</html>', { status: 202 });return new Response('<html><body><p>An article</p></body></html>', { status: 200, headers: { 'content-type': 'text/html' } });}),@@ -18,8 +16,8 @@ vi.mock('../../utils/ssrf.js', () => ({import { createFetchUrlTool } from './fetch-url.js';-// Parallel, the first search engine, uses the global fetch: unavailable here.-vi.stubGlobal('fetch', vi.fn(async () => new Response('', { status: 503 })));+// Parallel, the first search engine, uses the global fetch: it answers here.+vi.stubGlobal('fetch', vi.fn(async () => new Response(PARALLEL_REPLY, { status: 200 })));import { guardedFetch } from '../../utils/ssrf.js';import { isBrokenSearchSkill } from '../../skills/loader.js';@@ -31,7 +29,7 @@ describe('fetch_url on a search results page', () => {it('answers with web_search results instead of the bot challenge page', async () => {const t = createFetchUrlTool() as unknown as { execute: (a: unknown, o: unknown) => Promise<string> };const out = await t.execute({ url: 'https://html.duckduckgo.com/html/?q=latest+AI+news' }, {});- expect(out).toMatch(/^Web results from Brave for "latest AI news":/);+ expect(out).toMatch(/^Web results from Parallel for "latest AI news":/);expect(out).toContain('1. Latest AI news - The Verge');expect(out).toContain('Call web_search directly next time');const urls = vi.mocked(guardedFetch).mock.calls.map((c) => String(c[0]));diff --git a/src/capabilities/web/web-search.test.ts b/src/capabilities/web/web-search.test.tsindex b3aa782c..5ec27939 100644--- a/src/capabilities/web/web-search.test.ts+++ b/src/capabilities/web/web-search.test.ts@@ -1,251 +1,204 @@+import { readFileSync } from 'node:fs';+import { join } from 'node:path';import { afterEach, describe, expect, it, vi } from 'vitest';+/** Real pages, saved from html.duckduckgo.com and lite.duckduckgo.com for "pandoc epub to markdown". */+const DDG_HTML_PAGE = readFileSync(join(__dirname, 'fixtures', 'ddg-html.html'), 'utf8');+const DDG_LITE_PAGE = readFileSync(join(__dirname, 'fixtures', 'ddg-lite.html'), 'utf8');+const GOOGLE_NEWS = `<?xml version="1.0"?><rss><channel><title>Google News</title><item><title>Envoy meets leader in Moscow - BBC</title><link>https://news.google.com/rss/articles/abc</link><pubDate>Wed, 21 Oct 2026 08:00:00 GMT</pubDate><description><a href="x">Envoy meets leader</a>&nbsp;BBC</description><source url="https://www.bbc.co.uk">BBC</source></item><item><title>Markets rally - Reuters</title><link>https://news.google.com/rss/articles/def</link><pubDate>Wed, 21 Oct 2026 07:00:00 GMT</pubDate><source url="https://reuters.com">Reuters</source></item></channel></rss>`;-const BING = `<?xml version="1.0"?><rss><channel>-<item><title>what is the capital of australia - Bing</title><link>http://www.bing.com:80/search?q=x</link></item>-<item><title>Canberra - Wikipedia</title><link>https://en.wikipedia.org/wiki/Canberra</link><description>Canberra is the capital city of Australia.</description></item>-</channel></rss>`;--/** What Bing's web RSS answers automated requests with today: pages unrelated to the query. */-const BING_JUNK = `<?xml version="1.0"?><rss><channel>-<item><title>SonyLIV Coupons, Promo Code & Subscription Offers</title><link>https://example.com/sonyliv</link><description>Save on SonyLIV.</description></item>-<item><title>Jammu to Chandigarh Trains - redBus</title><link>https://example.com/trains</link><description>Train timings.</description></item>-</channel></rss>`;--/** Two results trimmed from a real Brave Search page (markup as served in October 2026). */-const BRAVE = `<!doctype html><html><body><div class="snippet">-<a href="https://search.brave.com/settings" class="nav">Settings</a>-<a href="https://www.typescriptlang.org/docs/handbook/release-notes/typescript-4-9.html" target="_self" class="svelte-14r20fy l1"><div class="site-name-wrapper"><cite class="snippet-url">typescriptlang.org</cite></div> <div class="title search-snippet-title line-clamp-1 svelte-14r20fy" title="TypeScript: Documentation - TypeScript 4.9">TypeScript: Documentation - TypeScript 4.9</div></a><!----> <div class="generic-snippet svelte-1cwdgg3"><div class="content desktop-default-regular t-primary line-clamp-dynamic svelte-1cwdgg3"><!---->The new satisfies operator <strong>lets us validate that the type of an expression matches some type</strong>.<!----></div></div>-<a href="https://stackoverflow.com/questions/75000000/satisfies-vs-as" target="_self" class="svelte-14r20fy l1"><div class="title search-snippet-title line-clamp-1 svelte-14r20fy" title="What are the differences between TypeScript's satisfies operator and type assertions? - Stack Overflow">x</div></a><div class="generic-snippet"><div class="content desktop-default-regular">satisfies checks without widening; as asserts.</div></div>-</div></body></html>`;--const WIKI = JSON.stringify({ query: { search: [- { title: 'Canberra', snippet: '<span class="searchmatch">Canberra</span> is the capital city of Australia.', timestamp: '2026-10-01T00:00:00Z' },- { title: 'Australian Capital Territory', snippet: 'A federal territory around Canberra, the capital of Australia', timestamp: '2026-09-01T00:00:00Z' },-] } });--const HN = JSON.stringify({ hits: [- { title: "TypeScript's satisfies operator", url: 'https://example.dev/satisfies', objectID: '41', points: 120, created_at: '2026-10-02T00:00:00Z' },- { title: 'Ask HN: TypeScript satisfies vs as?', url: null, objectID: '42', points: 30 },-] });-const routes = (map: Record<string, string | number>) => async (url: string) => {for (const [host, body] of Object.entries(map)) {- if (url.includes(host)) return typeof body === 'number' ? new Response('', { status: body }) : new Response(body, { status: 200 });+ if (!url.includes(host)) continue;+ return typeof body === 'number' ? new Response('', { status: body }) : new Response(body, { status: 200 });}return new Response('', { status: 404 });};-const defaultFetch = routes({ 'search.brave.com': BRAVE, 'news.google.com': GOOGLE_NEWS, 'bing.com/news': GOOGLE_NEWS, 'bing.com': BING, 'wikipedia.org': WIKI, 'hn.algolia.com': HN });+const defaultFetch = routes({ 'news.google.com': GOOGLE_NEWS });vi.mock('../../utils/ssrf.js', () => ({ guardedFetch: vi.fn() }));-import { createWebSearchTool, directLink, looksLikeArticle, parseBrave, parseDdg, parseParallel, parseRss, relevant, resetBraveCooldown, searchQueryFromUrl, webSearch } from './web-search.js';+import { createWebSearchTool, ddgTarget, directLink, looksLikeArticle, parseDdg, parseDdgHtml, parseDdgLite, parseParallel, parseRss, relevant, searchQueryFromUrl, webSearch } from './web-search.js';import { guardedFetch } from '../../utils/ssrf.js';vi.mocked(guardedFetch).mockImplementation(defaultFetch as never);// Parallel (the first engine) is called with the global fetch. By default it-// is unavailable here, so the older tests exercise the fallback chain; the-// Parallel tests below answer it.+// is unavailable here, so tests reach DuckDuckGo; Parallel tests answer it.const parallelDown = async () => new Response('', { status: 503 });const fetchMock = vi.fn(parallelDown);vi.stubGlobal('fetch', fetchMock);const parallelReply = (results: unknown[]) => new Response(JSON.stringify({ jsonrpc: '2.0', id: 1, result: { content: [{ type: 'text', text: '{}' }], structuredContent: { search_id: 's', results } } }), { status: 200 });-// Tests that swap the fetch mock must not leak it (or a Brave cooldown) into the next test.-afterEach(() => { vi.mocked(guardedFetch).mockClear(); vi.mocked(guardedFetch).mockImplementation(defaultFetch as never); resetBraveCooldown(); fetchMock.mockReset(); fetchMock.mockImplementation(parallelDown); });-describe('web_search', () => {- it('parses Google News items, strips the source suffix, keeps the date', () => {- const r = parseRss(GOOGLE_NEWS, 5);- expect(r).toHaveLength(2);- expect(r[0]).toMatchObject({ title: 'Envoy meets leader in Moscow', source: 'BBC', date: 'Wed, 21 Oct 2026 08:00:00 GMT' });- expect(r[0].snippet).toBe('Envoy meets leader BBC');- });+afterEach(() => {+ vi.mocked(guardedFetch).mockClear();+ vi.mocked(guardedFetch).mockImplementation(defaultFetch as never);+ fetchMock.mockReset();+ fetchMock.mockImplementation(parallelDown);+});- it('drops Bing\'s self-link item', () => {- expect(parseRss(BING, 5).map((x) => x.url)).toEqual(['https://en.wikipedia.org/wiki/Canberra']);- });+const ddgHtmlHit = (target: string, title: string, snippet: string, cls = 'result results_links results_links_deep web-result ') =>+ `<div class="${cls}"><div class="links_main links_deep result__body"><h2 class="result__title"><a rel="nofollow" class="result__a" href="//duckduckgo.com/l/?uddg=${encodeURIComponent(target)}&rut=abc">${title}</a></h2><a class="result__snippet" href="//duckduckgo.com/l/?uddg=${encodeURIComponent(target)}">${snippet}</a></div></div>`;- it('parses Brave results: link, title, snippet; skips Brave\'s own links', () => {- const r = parseBrave(BRAVE, 5);- expect(r).toHaveLength(2);+describe('DuckDuckGo parsing (exact markup)', () => {+ it('reads all ten results from a real html page: titles, real links, snippets', () => {+ const r = parseDdgHtml(DDG_HTML_PAGE, 20);+ expect(r).toHaveLength(10);expect(r[0]).toEqual({- title: 'TypeScript: Documentation - TypeScript 4.9',- url: 'https://www.typescriptlang.org/docs/handbook/release-notes/typescript-4-9.html',- snippet: 'The new satisfies operator lets us validate that the type of an expression matches some type .',+ title: 'Convert epub to markdown : r/pandoc - Reddit',+ url: 'https://www.reddit.com/r/pandoc/comments/sg6sqx/convert_epub_to_markdown/',+ snippet: expect.stringMatching(/^Hi, I want to convert an ebook in epub format to markdown/),});- expect(r[1].title).toBe("What are the differences between TypeScript's satisfies operator and type assertions? - Stack Overflow");- expect(r[1].snippet).toBe('satisfies checks without widening; as asserts.');- });-- it('web uses Brave first; news uses Google News; all through the SSRF guard', async () => {- const web = await webSearch('typescript satisfies operator');- expect(web.engine).toMatch(/^Brave/); // two hits here, so Wikipedia tops it up- expect(web.results[0].url).toContain('typescriptlang.org');- expect(String(vi.mocked(guardedFetch).mock.calls[0][0])).toMatch(/^https:\/\/search\.brave\.com\/search\?q=typescript%20satisfies%20operator/);- expect((await webSearch('', { kind: 'news' })).engine).toBe('Google News');+ expect(r.map((x) => x.url)).toContain('https://pandoc.org/epub.html');+ for (const x of r) {+ expect(x.url).toMatch(/^https:\/\//);+ expect(x.url).not.toMatch(/duckduckgo\.com/);+ expect(x.title).not.toMatch(/<|&/);+ }});- it('an engine answering with unrelated pages counts as empty: Bing\'s junk never reaches the model', async () => {- vi.mocked(guardedFetch).mockImplementation(routes({ 'search.brave.com': 503, 'wikipedia.org': 503, 'hn.algolia.com': 503, 'news.google.com': 503, 'bing.com/news': 503, 'bing.com': BING_JUNK }) as never);- await expect(webSearch('typescript satisfies operator')).rejects.toThrow(/Bing: only unrelated results/);+ it('reads all ten results from a real lite page, the same pages as the html one', () => {+ const r = parseDdgLite(DDG_LITE_PAGE, 20);+ expect(r).toHaveLength(10);+ expect(r[0].url).toBe('https://www.reddit.com/r/pandoc/comments/sg6sqx/convert_epub_to_markdown/');+ expect(r[0].snippet).toMatch(/^Hi, I want to convert an ebook in epub format to markdown/);+ expect(new Set(r.map((x) => x.url))).toEqual(new Set(parseDdgHtml(DDG_HTML_PAGE, 20).map((x) => x.url)));});- it('relevance: one or two words must all appear, longer queries need half', () => {- const r = (title: string) => ({ title, url: 'https://x.dev', snippet: '' });- expect(relevant('best pizza in srinagar', [r('Best pizza places in Srinagar'), r('Srinagar war history'), r('Pizza in Rome')]).map((x) => x.title)).toEqual(['Best pizza places in Srinagar']);- expect(relevant('typescript satisfies operator', [r('The satisfies operator'), r('TypeScript handbook'), r('Train timings')]).map((x) => x.title)).toEqual(['The satisfies operator']);- expect(relevant('', [r('anything')])).toHaveLength(1);+ it('parseDdg picks the right parser for either page', () => {+ expect(parseDdg(DDG_HTML_PAGE, 3)).toEqual(parseDdgHtml(DDG_HTML_PAGE, 3));+ expect(parseDdg(DDG_LITE_PAGE, 3)).toEqual(parseDdgLite(DDG_LITE_PAGE, 3));});- it('keeps going down the keyless chain: Wikipedia, then Hacker News', async () => {- vi.mocked(guardedFetch).mockImplementation(routes({ 'search.brave.com': 429, 'wikipedia.org': WIKI, 'hn.algolia.com': HN }) as never);- const wiki = await webSearch('canberra');- expect(wiki.engine).toBe('Wikipedia');- expect(wiki.results[0]).toMatchObject({ title: 'Canberra', url: 'https://en.wikipedia.org/wiki/Canberra', snippet: 'Canberra is the capital city of Australia.', source: 'Wikipedia' });-- resetBraveCooldown();- vi.mocked(guardedFetch).mockImplementation(routes({ 'hn.algolia.com': HN }) as never);- const hn = await webSearch('typescript satisfies');- expect(hn.engine).toBe('Hacker News');- expect(hn.results[0]).toMatchObject({ title: "TypeScript's satisfies operator", url: 'https://example.dev/satisfies', source: 'Hacker News' });- // A story without its own link points at the discussion.- expect(hn.results[1].url).toBe('https://news.ycombinator.com/item?id=42');+ it('drops ads on both pages', () => {+ const ad = `<div class="result results_links results_links_deep result--ad "><h2 class="result__title"><a class="result__a" href="https://duckduckgo.com/y.js?ad_domain=shop.example&u3=x">Buy now</a></h2><a class="result__snippet" href="https://duckduckgo.com/y.js?x">Sponsored</a></div>`;+ const html = ad + ddgHtmlHit('https://pandoc.org/epub.html', 'Pandoc EPUB', 'Creating an ebook with pandoc');+ expect(parseDdgHtml(html, 5).map((x) => x.url)).toEqual(['https://pandoc.org/epub.html']);+ const lite = `<table><tr class="result-sponsored"><td><a rel="nofollow" href="https://duckduckgo.com/y.js?ad=1" class='result-link'>Ad</a></td></tr>+ <tr><td><a rel="nofollow" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fpandoc.org%2Fepub.html&rut=x" class='result-link'>Pandoc EPUB</a></td></tr>+ <tr><td class='result-snippet'>Creating an ebook with <b>pandoc</b>.</td></tr></table>`;+ expect(parseDdgLite(lite, 5)).toEqual([{ title: 'Pandoc EPUB', url: 'https://pandoc.org/epub.html', snippet: 'Creating an ebook with pandoc .' }]);});- it('after a 429, Brave rests for a minute instead of being tried on every query', async () => {- vi.mocked(guardedFetch).mockImplementation(routes({ 'search.brave.com': 429, 'wikipedia.org': WIKI }) as never);- await webSearch('canberra');- await webSearch('canberra capital');- const braveCalls = vi.mocked(guardedFetch).mock.calls.filter((c) => String(c[0]).includes('search.brave.com'));- expect(braveCalls).toHaveLength(1);- });-- it('tops a thin answer up with Wikipedia and lists each page once', async () => {- const oneHit = `<a href="https://en.wikipedia.org/wiki/Canberra" target="_self" class="l1"><div class="title search-snippet-title" title="Canberra, capital of Australia">x</div></a><div class="content">The capital of Australia.</div>`;- vi.mocked(guardedFetch).mockImplementation(routes({ 'search.brave.com': oneHit, 'wikipedia.org': WIKI }) as never);- const r = await webSearch('canberra australia');- expect(r.engine).toBe('Brave + Wikipedia');- expect(r.results.filter((x) => /wiki\/Canberra$/.test(x.url))).toHaveLength(1);- expect(r.results.map((x) => x.title)).toContain('Australian Capital Territory');+ it('unwraps the redirect, keeps direct links, and rejects DuckDuckGo\'s own pages', () => {+ expect(ddgTarget('//duckduckgo.com/l/?uddg=https%3A%2F%2Fpandoc.org%2Fepub.html&rut=abc')).toBe('https://pandoc.org/epub.html');+ expect(ddgTarget('https://example.dev/page')).toBe('https://example.dev/page');+ expect(ddgTarget('https://duckduckgo.com/y.js?ad_domain=x')).toBeNull();+ expect(ddgTarget('//duckduckgo.com/l/?uddg=https%3A%2F%2Fduckduckgo.com%2Fsettings')).toBeNull();+ expect(ddgTarget('/html/?q=next')).toBeNull();+ expect(ddgTarget('javascript:void(0)')).toBeNull();});+});- it('recognises search-engine result URLs, so fetch_url can answer them with real results', () => {- expect(searchQueryFromUrl('https://html.duckduckgo.com/html/?q=latest+AI+news')).toBe('latest AI news');- expect(searchQueryFromUrl('https://lite.duckduckgo.com/lite/?q=x')).toBe('x');- expect(searchQueryFromUrl('https://duckduckgo.com/?q=pandoc%20epub')).toBe('pandoc epub');- expect(searchQueryFromUrl('https://www.google.com/search?q=canberra')).toBe('canberra');- expect(searchQueryFromUrl('https://www.google.co.uk/search?q=canberra')).toBe('canberra');- expect(searchQueryFromUrl('https://www.bing.com/search?q=canberra')).toBe('canberra');- expect(searchQueryFromUrl('https://search.yahoo.com/search?p=canberra')).toBe('canberra');- // Not search pages: the feed web_search itself uses, articles, empty queries.- expect(searchQueryFromUrl('https://www.bing.com/search?q=canberra&format=rss')).toBeNull();- expect(searchQueryFromUrl('https://duckduckgo.com/about')).toBeNull();- expect(searchQueryFromUrl('https://en.wikipedia.org/wiki/Canberra')).toBeNull();- expect(searchQueryFromUrl('https://html.duckduckgo.com/html/')).toBeNull();- expect(searchQueryFromUrl('not a url')).toBeNull();+describe('web_search chain: Parallel, then DuckDuckGo html, then lite', () => {+ it('Parallel answers first: one keyless MCP tools/call, results with excerpts and dates', async () => {+ fetchMock.mockImplementation(async () => parallelReply([+ { url: 'https://www.linkedin.com/company/cosmic-stack', title: 'Cosmic Stack', publish_date: '2026-05-14', excerpts: ['* Name: Cosmic Stack\n * description\n - Cosmic Stack is a technology company focused on agentic AI, building [Mercury Agent](https://mercuryagent.sh).'] },+ { url: 'https://cosmicstack.ai', title: 'Cosmic Stack', publish_date: null, excerpts: ['Cosmic Stack builds open agentic AI.'] },+ { url: 'https://github.com/cosmicstack-labs', title: 'Cosmic Stack Labs', excerpts: [] },+ { url: 'https://x.com/cosmicstack', title: 'Cosmic Stack on X', excerpts: ['Cosmic Stack news'] },+ ]));+ const r = await webSearch('cosmic stack linkedin');+ expect(r.engine).toBe('Parallel');+ expect(r.trail).toEqual(['Parallel: 4 results']);+ expect(r.results[0]).toEqual({ title: 'Cosmic Stack', url: 'https://www.linkedin.com/company/cosmic-stack', date: '2026-05-14', snippet: 'Name: Cosmic Stack description - Cosmic Stack is a technology company focused on agentic AI, building Mercury Agent.' });+ expect(vi.mocked(guardedFetch)).not.toHaveBeenCalled(); // Parallel had enough+ const [url, init] = fetchMock.mock.calls[0] as unknown as [string, RequestInit];+ expect(url).toBe('https://search.parallel.ai/mcp');+ expect(JSON.parse(String(init.body))).toMatchObject({ jsonrpc: '2.0', method: 'tools/call', params: { name: 'web_search', arguments: { objective: 'cosmic stack linkedin', search_queries: ['cosmic stack linkedin'] } } });});- it('the tool returns a readable list, or a clear error', async () => {- const t = createWebSearchTool() as unknown as { execute: (a: unknown, o: unknown) => Promise<string> };- const out = await t.execute({ query: '', kind: 'news' }, {});- expect(out).toMatch(/News results from Google News[\s\S]*1\. Envoy meets leader in Moscow \(BBC · Wed, 21 Oct 2026\)/);- vi.mocked(guardedFetch).mockImplementation(async () => new Response('', { status: 500 }));- expect(await t.execute({ query: 'x' }, {})).toMatch(/^Error: web search found nothing: no source found anything \(Parallel: HTTP 503; Brave: HTTP 500; DuckDuckGo: HTTP 500/);+ it('Parallel down: DuckDuckGo\'s html page answers', async () => {+ vi.mocked(guardedFetch).mockImplementation(routes({ 'html.duckduckgo.com': DDG_HTML_PAGE }) as never);+ const r = await webSearch('pandoc epub to markdown');+ expect(r.engine).toBe('DuckDuckGo');+ expect(r.trail[0]).toBe('Parallel: HTTP 503');+ expect(r.trail[1]).toMatch(/^DuckDuckGo: \d+ results$/);+ expect(r.results[0].url).toBe('https://www.reddit.com/r/pandoc/comments/sg6sqx/convert_epub_to_markdown/');+ expect(String(vi.mocked(guardedFetch).mock.calls[0][0])).toBe('https://html.duckduckgo.com/html/?q=pandoc%20epub%20to%20markdown');});- it('parses DuckDuckGo html and lite pages: real links out of the redirect, ads dropped', () => {- const html = `<a rel="nofollow" class="result__a" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fpandoc.org%2Fepub.html&rut=abc">Pandoc - Creating an ebook with pandoc</a>- <a class="result__snippet" href="x">Starting with version 1.6, <b>pandoc</b> can produce EPUB.</a>- <a rel="nofollow" class="result__a" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fduckduckgo.com%2Fy.js%3Fad%3D1">Sponsored</a>`;- expect(parseDdg(html, 5)).toEqual([{ title: 'Pandoc - Creating an ebook with pandoc', url: 'https://pandoc.org/epub.html', snippet: 'Starting with version 1.6, pandoc can produce EPUB.' }]);- const lite = `<a rel="nofollow" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fwww.linkedin.com%2Fcompany%2Fcosmic-stack&rut=x" class='result-link'>Cosmic Stack - LinkedIn</a>- <td class='result-snippet'>Cosmic Stack builds Mercury.</td>`;- expect(parseDdg(lite, 5)[0]).toEqual({ title: 'Cosmic Stack - LinkedIn', url: 'https://www.linkedin.com/company/cosmic-stack', snippet: 'Cosmic Stack builds Mercury.' });+ it('a challenged html page is retried once, then the lite page answers', async () => {+ let htmlCalls = 0;+ vi.mocked(guardedFetch).mockImplementation((async (url: string) => {+ if (url.includes('html.duckduckgo.com')) { htmlCalls++; return new Response('<html>anomaly</html>', { status: 202 }); }+ if (url.includes('lite.duckduckgo.com')) return new Response(DDG_LITE_PAGE, { status: 200 });+ return new Response('', { status: 404 });+ }) as never);+ const r = await webSearch('pandoc epub to markdown');+ expect(htmlCalls).toBe(2);+ expect(r.engine).toBe('DuckDuckGo Lite');+ expect(r.trail.slice(0, 2)).toEqual(['Parallel: HTTP 503', 'DuckDuckGo: blocked (bot challenge)']);+ expect(r.trail[2]).toMatch(/^DuckDuckGo Lite: \d+ results$/);});- it('news links open the article: Bing News redirects are unwrapped', () => {- expect(directLink('http://www.bing.com/news/apiclick.aspx?ref=FexRss&aid=&tid=1&url=https%3a%2f%2fkashmirlife.net%2fstory&c=2')).toBe('https://kashmirlife.net/story');- expect(directLink('https://kashmirlife.net/story')).toBe('https://kashmirlife.net/story');- const feed = `<rss><channel><item><title>Kashmir story</title><link>http://www.bing.com/news/apiclick.aspx?ref=FexRss&url=https%3a%2f%2fkashmirlife.net%2fstory</link></item></channel></rss>`;- expect(parseRss(feed, 3)[0].url).toBe('https://kashmirlife.net/story');+ it('a challenge then a normal page counts as a success', async () => {+ let calls = 0;+ vi.mocked(guardedFetch).mockImplementation((async (url: string) => {+ if (url.includes('html.duckduckgo.com')) return ++calls === 1 ? new Response('<html>anomaly</html>', { status: 202 }) : new Response(DDG_HTML_PAGE, { status: 200 });+ return new Response('', { status: 404 });+ }) as never);+ const r = await webSearch('pandoc epub');+ expect(calls).toBe(2);+ expect(r.engine).toBe('DuckDuckGo');});- it('news asks Bing News first, so links are direct; Google News is the fallback', async () => {- const bingNewsFeed = `<rss><channel><item><title>Kashmir story</title><link>http://www.bing.com/news/apiclick.aspx?url=https%3a%2f%2fkashmirlife.net%2fstory</link></item></channel></rss>`;- vi.mocked(guardedFetch).mockImplementation(routes({ 'bing.com/news': bingNewsFeed, 'news.google.com': GOOGLE_NEWS }) as never);- const r = await webSearch('kashmir', { kind: 'news' });- expect(r.engine).toBe('Bing News');- expect(r.results[0].url).toBe('https://kashmirlife.net/story');+ it('Parallel and DuckDuckGo are combined until there are enough results', async () => {+ fetchMock.mockImplementation(async () => parallelReply([{ url: 'https://pandoc.org/MANUAL.html', title: 'Pandoc manual: EPUB to Markdown', excerpts: ['pandoc converts epub to markdown'] }]));+ vi.mocked(guardedFetch).mockImplementation(routes({ 'html.duckduckgo.com': DDG_HTML_PAGE }) as never);+ const r = await webSearch('pandoc epub to markdown');+ expect(r.engine).toBe('Parallel + DuckDuckGo');+ expect(r.results[0].url).toBe('https://pandoc.org/MANUAL.html');+ expect(r.trail[0]).toBe('Parallel: 1 result');});- it('combines engines until there are enough results, and records what each one did', async () => {- const oneHit = `<a href="https://www.typescriptlang.org/satisfies" target="_self" class="l1"><div class="title search-snippet-title" title="TypeScript satisfies operator">x</div></a><div class="content">satisfies</div>`;- const ddgPage = `<a class="result__a" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fexample.dev%2Fa">TypeScript satisfies explained</a><a class="result__a" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fexample.dev%2Fb">The satisfies operator in TypeScript</a><a class="result__a" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fexample.dev%2Fc">TypeScript satisfies vs as</a>`;- vi.mocked(guardedFetch).mockImplementation(routes({ 'search.brave.com': oneHit, 'html.duckduckgo.com': ddgPage }) as never);- const r = await webSearch('typescript satisfies operator', { max: 8 });- expect(r.engine).toBe('Brave + DuckDuckGo');- expect(r.results).toHaveLength(4);- expect(r.trail).toEqual(['Parallel: HTTP 503', 'Brave: 1 result', 'DuckDuckGo: 3 results']);+ it('a real "no results" page is no results; an unrecognised page says so', async () => {+ vi.mocked(guardedFetch).mockImplementation(routes({ 'html.duckduckgo.com': '<html><div class="no-results">No results.</div></html>', 'lite.duckduckgo.com': '<html><body>something new</body></html>' }) as never);+ await expect(webSearch('zzqqxx unfindable')).rejects.toThrow('no source found anything (Parallel: HTTP 503; DuckDuckGo: no results; DuckDuckGo Lite: page not recognised)');});- it('a bot challenge reads as "blocked" in the trail', async () => {- vi.mocked(guardedFetch).mockImplementation(routes({ 'search.brave.com': 429, 'duckduckgo.com': '<html>anomaly</html>', 'wikipedia.org': WIKI }) as never);- const r = await webSearch('canberra');- expect(r.trail.slice(0, 5)).toEqual(['Parallel: HTTP 503', 'Brave: blocked (HTTP 429)', 'DuckDuckGo: blocked (bot challenge)', 'DuckDuckGo Lite: blocked (bot challenge)', 'Wikipedia: 2 results']);+ it('only results that mention the query count', () => {+ const r = (title: string) => ({ title, url: 'https://x.dev', snippet: '' });+ expect(relevant('best pizza in srinagar', [r('Best pizza places in Srinagar'), r('Srinagar war history'), r('Pizza in Rome')]).map((x) => x.title)).toEqual(['Best pizza places in Srinagar']);+ expect(relevant('typescript satisfies operator', [r('The satisfies operator'), r('TypeScript handbook'), r('Train timings')]).map((x) => x.title)).toEqual(['The satisfies operator']);+ expect(relevant('', [r('anything')])).toHaveLength(1);});--- it('a DuckDuckGo challenge is retried once after a short pause', async () => {- const page = `<a class="result__a" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fexample.dev%2Fcanberra">Canberra guide</a><a class="result__a" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fexample.dev%2Fcanberra2">Visit Canberra</a><a class="result__a" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fexample.dev%2Fcanberra3">Canberra history</a><a class="result__a" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Fexample.dev%2Fcanberra4">Canberra map</a>`;- let ddgCalls = 0;- vi.mocked(guardedFetch).mockImplementation((async (url: string) => {- if (url.includes('search.brave.com')) return new Response('', { status: 429 });- if (url.includes('html.duckduckgo.com')) return ++ddgCalls === 1 ? new Response('<html>anomaly</html>', { status: 202 }) : new Response(page, { status: 200 });- return new Response('', { status: 404 });- }) as never);- const r = await webSearch('canberra');- expect(ddgCalls).toBe(2);- expect(r.engine).toBe('DuckDuckGo');- expect(r.trail).toEqual(['Parallel: HTTP 503', 'Brave: blocked (HTTP 429)', 'DuckDuckGo: 4 results']);+ it('the tool returns a readable list with the trail, or a clear error', async () => {+ vi.mocked(guardedFetch).mockImplementation(routes({ 'html.duckduckgo.com': DDG_HTML_PAGE }) as never);+ const t = createWebSearchTool() as unknown as { execute: (a: unknown, o: unknown) => Promise<string> };+ const out = await t.execute({ query: 'pandoc epub' }, {});+ expect(out).toMatch(/^Web results from DuckDuckGo for "pandoc epub":\n\n1\. Convert epub to markdown : r\/pandoc - Reddit/);+ expect(out).toMatch(/Searched: Parallel: HTTP 503 → DuckDuckGo: \d+ results$/);+ vi.mocked(guardedFetch).mockImplementation(async () => new Response('', { status: 500 }));+ expect(await t.execute({ query: 'x' }, {})).toMatch(/^Error: web search found nothing: no source found anything \(Parallel: HTTP 503; DuckDuckGo: HTTP 500; DuckDuckGo Lite: HTTP 500\)/);});+});- it('Parallel answers first: one keyless MCP tools/call, results with excerpts and dates', async () => {- fetchMock.mockImplementation(async () => parallelReply([- { url: 'https://www.linkedin.com/company/cosmic-stack', title: 'Cosmic Stack', publish_date: '2026-05-14', excerpts: ['* Name: Cosmic Stack\n * description\n - Cosmic Stack is a technology company focused on agentic AI, building [Mercury Agent](https://mercuryagent.sh).'] },- { url: 'https://cosmicstack.ai', title: 'Cosmic Stack', publish_date: null, excerpts: ['Cosmic Stack builds open agentic AI.'] },- { url: 'https://github.com/cosmicstack-labs', title: 'Cosmic Stack Labs', excerpts: [] },- { url: 'https://x.com/cosmicstack', title: 'Cosmic Stack on X', excerpts: ['Cosmic Stack news'] },- ]));- const r = await webSearch('cosmic stack linkedin');- expect(r.engine).toBe('Parallel');- expect(r.trail).toEqual(['Parallel: 4 results']);- expect(r.results[0]).toEqual({ title: 'Cosmic Stack', url: 'https://www.linkedin.com/company/cosmic-stack', date: '2026-05-14', snippet: 'Name: Cosmic Stack description - Cosmic Stack is a technology company focused on agentic AI, building Mercury Agent.' });- // Nothing else was asked: Parallel had enough.- expect(vi.mocked(guardedFetch)).not.toHaveBeenCalled();- const [url, init] = fetchMock.mock.calls[0] as unknown as [string, RequestInit];- expect(url).toBe('https://search.parallel.ai/mcp');- const body = JSON.parse(String(init.body));- expect(body).toMatchObject({ jsonrpc: '2.0', method: 'tools/call', params: { name: 'web_search', arguments: { objective: 'cosmic stack linkedin', search_queries: ['cosmic stack linkedin'] } } });+describe('news', () => {+ it('parses Google News items, strips the source suffix, keeps the date', () => {+ const r = parseRss(GOOGLE_NEWS, 5);+ expect(r).toHaveLength(2);+ expect(r[0]).toMatchObject({ title: 'Envoy meets leader in Moscow', source: 'BBC', date: 'Wed, 21 Oct 2026 08:00:00 GMT' });+ expect(r[0].snippet).toBe('Envoy meets leader BBC');});- it('parses Parallel replies sent as server-sent events, and surfaces MCP errors', () => {- const sse = `event: message\ndata: ${JSON.stringify({ jsonrpc: '2.0', id: 1, result: { structuredContent: { results: [{ url: 'https://nodejs.org/en/blog/release', title: 'Node.js releases' }] } } })}\n\n`;- expect(parseParallel(sse, 5)).toEqual([{ title: 'Node.js releases', url: 'https://nodejs.org/en/blog/release', snippet: '', date: undefined }]);- expect(() => parseParallel(JSON.stringify({ jsonrpc: '2.0', id: 1, error: { code: -32000, message: 'rate limited' } }), 5)).toThrow('rate limited');+ it('Bing News redirects are unwrapped to the article', () => {+ expect(directLink('http://www.bing.com/news/apiclick.aspx?ref=FexRss&aid=&tid=1&url=https%3a%2f%2fkashmirlife.net%2fstory&c=2')).toBe('https://kashmirlife.net/story');+ expect(directLink('https://kashmirlife.net/story')).toBe('https://kashmirlife.net/story');+ const feed = `<rss><channel><item><title>Kashmir story</title><link>http://www.bing.com/news/apiclick.aspx?ref=FexRss&url=https%3a%2f%2fkashmirlife.net%2fstory</link></item></channel></rss>`;+ expect(parseRss(feed, 3)[0].url).toBe('https://kashmirlife.net/story');});- it('news from Parallel keeps article links and drops section fronts; Bing News fills in', async () => {+ it('Parallel keeps article links and drops section fronts; Bing News fills in', async () => {expect(looksLikeArticle('https://www.greaterkashmir.com/kashmir')).toBe(false);expect(looksLikeArticle('https://www.greaterkashmir.com/latest-news')).toBe(false);expect(looksLikeArticle('https://www.democracynow.org/2026/10/23/headlines')).toBe(true);- expect(looksLikeArticle('https://kashmirlife.net/jammu-kashmir-government-warns-officers-over-delays')).toBe(false);expect(looksLikeArticle('https://kashmirlife.net/news/jammu-kashmir-government-warns-officers-over-delays')).toBe(true);expect(looksLikeArticle('https://www.greaterkashmir.com/kashmir/attacks-on-infrastructure-unacceptable-12742822')).toBe(true);fetchMock.mockImplementation(async () => parallelReply([@@ -258,4 +211,28 @@ describe('web_search', () => {expect(r.engine).toBe('Parallel + Bing News');expect(r.results.map((x) => x.url)).toEqual(['https://www.greaterkashmir.com/kashmir/power-shutdown-in-kashmir-announced-12742577', 'https://kashmirlife.net/story']);});++ it('top stories with no query come from Google News when Parallel is down', async () => {+ const r = await webSearch('', { kind: 'news' });+ expect(r.engine).toBe('Google News');+ });+});++describe('MCP reply parsing and search-page URLs', () => {+ it('parses Parallel replies sent as server-sent events, and surfaces MCP errors', () => {+ const sse = `event: message\ndata: ${JSON.stringify({ jsonrpc: '2.0', id: 1, result: { structuredContent: { results: [{ url: 'https://nodejs.org/en/blog/release', title: 'Node.js releases' }] } } })}\n\n`;+ expect(parseParallel(sse, 5)).toEqual([{ title: 'Node.js releases', url: 'https://nodejs.org/en/blog/release', snippet: '', date: undefined }]);+ expect(() => parseParallel(JSON.stringify({ jsonrpc: '2.0', id: 1, error: { code: -32000, message: 'rate limited' } }), 5)).toThrow('rate limited');+ });++ it('recognises search-engine result URLs, so fetch_url can answer them with real results', () => {+ expect(searchQueryFromUrl('https://html.duckduckgo.com/html/?q=latest+AI+news')).toBe('latest AI news');+ expect(searchQueryFromUrl('https://lite.duckduckgo.com/lite/?q=x')).toBe('x');+ expect(searchQueryFromUrl('https://duckduckgo.com/?q=pandoc%20epub')).toBe('pandoc epub');+ expect(searchQueryFromUrl('https://www.google.com/search?q=canberra')).toBe('canberra');+ expect(searchQueryFromUrl('https://www.bing.com/search?q=canberra')).toBe('canberra');+ expect(searchQueryFromUrl('https://www.bing.com/search?q=canberra&format=rss')).toBeNull();+ expect(searchQueryFromUrl('https://en.wikipedia.org/wiki/Canberra')).toBeNull();+ expect(searchQueryFromUrl('not a url')).toBeNull();+ });});diff --git a/src/capabilities/web/web-search.ts b/src/capabilities/web/web-search.tsindex 62a9f5de..5c85edca 100644--- a/src/capabilities/web/web-search.ts+++ b/src/capabilities/web/web-search.ts@@ -5,9 +5,9 @@ import { guardedFetch } from '../../utils/ssrf.js';/*** web_search: free, keyless search for the model.*- * web → Parallel (search.parallel.ai, keyless MCP), then Brave Search,- * DuckDuckGo (html, then lite), Wikipedia, Hacker News, Bing News,- * Google News, Bing RSS+ * web → Parallel (search.parallel.ai, keyless MCP), then DuckDuckGo's+ * html page, then its lite page: the three keyless web indexes+ * that answer automated requests* news → Parallel, then Bing News (direct article links), Google News (its* links are Google redirects), Hacker News*@@ -16,8 +16,8 @@ import { guardedFetch } from '../../utils/ssrf.js';** Read-only and through the same SSRF guard as fetch_url. Returns titles,* links, snippets and dates; the model follows up with fetch_url to read a- * page. (DuckDuckGo's HTML endpoint answers automated requests with a 202- * challenge, which is why the old search skill often came back empty.)+ * page. Brave, Mojeek and Startpage refuse automated requests, and Bing's+ * web RSS serves unrelated pages, so none of them are used.*/export interface SearchResult {@@ -105,43 +105,12 @@ async function fetchText(url: string): Promise<string> {}}-/**- * Bing's web RSS feed. Kept as a late fallback only: since autumn 2026 it- * answers automated requests with results unrelated to the query, which the- * relevance check below throws away.- */-async function bing(query: string, max: number): Promise<SearchResult[]> {- return parseRss(await fetchText(`https://www.bing.com/search?q=${encodeURIComponent(query)}&format=rss&count=${Math.min(20, max + 2)}`), max);-}-/** Bing News RSS: relevant, recent, keyless. */async function bingNews(query: string, max: number): Promise<SearchResult[]> {if (!query.trim()) return [];return parseRss(await fetchText(`https://www.bing.com/news/search?q=${encodeURIComponent(query)}&format=rss`), max);}-/**- * Brave Search's results page: a real, independent web index that answers a- * plain browser request without a key or a challenge. Parsed from the HTML,- * so a markup change shows up as "no results" and the chain moves on.- */-export function parseBrave(html: string, max: number): SearchResult[] {- const out: SearchResult[] = [];- const anchor = /<a href="(https?:\/\/[^"]+)"[^>]*>([\s\S]*?)<\/a>/g;- let m: RegExpExecArray | null;- while ((m = anchor.exec(html)) && out.length < max) {- const [, url, inner] = m;- const title = /class="title search-snippet-title[^"]*"[^>]*title="([^"]*)"/.exec(inner)?.[1];- if (!title || /^https?:\/\/([a-z]+\.)?search\.brave\.com/.test(url)) continue;- const after = html.slice(anchor.lastIndex, anchor.lastIndex + 4000);- const nextResult = after.search(/<a href="https?:\/\/[^"]+"[^>]*>[\s\S]{0,3000}?search-snippet-title/);- const scope = nextResult > 0 ? after.slice(0, nextResult) : after;- const snippet = /class="content[^"]*">([\s\S]*?)<\/div>/.exec(scope)?.[1] ?? '';- out.push({ title: decode(title), url: unescape(url), snippet: decode(snippet).slice(0, 300) });- }- return out;-}-/*** Parallel's public search over MCP (https://search.parallel.ai/mcp): a real* web index built for agents, keyless, answering in about a second with@@ -226,52 +195,105 @@ async function parallelNews(query: string, max: number): Promise<SearchResult[]>}/**- * DuckDuckGo's no-JavaScript result pages (html. and lite.). Links come as- * duckduckgo.com/l/?uddg=<the page>; ads point back at duckduckgo.com and- * are dropped. DuckDuckGo answers some automated requests with a challenge- * page (HTTP 202, "anomaly"), which counts as a failure so the chain moves on.+ * DuckDuckGo's two no-JavaScript result pages, the keyless web index behind+ * Parallel. Both wrap every result link as+ * //duckduckgo.com/l/?uddg=<the page>&rut=…, and both mark ads: the html page+ * with a `result--ad` block linking to duckduckgo.com/y.js, the lite page+ * with a sponsored row. Parsed exactly from their markup:+ *+ * html.duckduckgo.com/html/ <div class="result results_links …">+ * <a class="result__a" href=…>title</a>+ * <a class="result__snippet" href=…>snippet</a>+ * lite.duckduckgo.com/lite/ <a class='result-link' href=…>title</a>+ * <td class='result-snippet'>snippet</td>+ *+ * DuckDuckGo answers some automated requests with a challenge (HTTP 202 or+ * an "anomaly" page). That, and a page with no results and no "No results"+ * notice (the markup changed), count as failures, so the trail says so.*/-export function parseDdg(html: string, max: number): SearchResult[] {++/** The page behind DuckDuckGo's redirect, or null for ads and DuckDuckGo's own links. */+export function ddgTarget(href: string): string | null {+ const raw = unescape(href.trim());+ let u: URL;+ try { u = new URL(raw.startsWith('//') ? `https:${raw}` : raw, 'https://duckduckgo.com'); } catch { return null; }+ const target = /(^|\.)duckduckgo\.com$/.test(u.hostname) ? u.searchParams.get('uddg') : u.href;+ if (!target || !/^https?:\/\//i.test(target)) return null;+ try {+ const t = new URL(target);+ return /(^|\.)duckduckgo\.com$/.test(t.hostname) ? null : t.href;+ } catch {+ return null;+ }+}++export function parseDdgHtml(html: string, max: number): SearchResult[] {const out: SearchResult[] = [];- const anchor = /<a[^>]*class=["'](?:result__a|result-link)["'][^>]*>[\s\S]*?<\/a>|<a[^>]*href="[^"]*"[^>]*class=["'](?:result__a|result-link)["'][^>]*>[\s\S]*?<\/a>/g;- const found = [...html.matchAll(anchor)];- for (let i = 0; i < found.length && out.length < max; i++) {- const tagText = found[i][0];- const href = /href="([^"]+)"/.exec(tagText)?.[1];- if (!href) continue;- let url = unescape(href);- try {- const u = new URL(url.startsWith('//') ? `https:${url}` : url);- url = u.searchParams.get('uddg') ?? u.href;- } catch { continue; }- if (!/^https?:\/\//.test(url) || /^https?:\/\/([a-z]+\.)?duckduckgo\.com\//.test(url)) continue;- const title = decode(tagText.replace(/^<a[^>]*>/, '').replace(/<\/a>$/, ''));- const start = (found[i].index ?? 0) + tagText.length;- const end = i + 1 < found.length ? found[i + 1].index ?? html.length : Math.min(html.length, start + 4000);- const scope = html.slice(start, end);- const snippet = /class=["'](?:result__snippet|result-snippet)["'][^>]*>([\s\S]*?)<\/(?:a|td|div)>/.exec(scope)?.[1] ?? '';- if (title) out.push({ title, url, snippet: decode(snippet).slice(0, 300) });+ const blocks = [...html.matchAll(/<div class="(result\b[^"]*)"/g)].filter((m) => /\b(results_links|web-result)\b/.test(m[1]));+ for (let i = 0; i < blocks.length && out.length < max; i++) {+ if (/\bresult--ad\b/.test(blocks[i][1])) continue;+ const block = html.slice(blocks[i].index ?? 0, blocks[i + 1]?.index ?? html.length);+ const a = /<a\b[^>]*class="result__a"[^>]*>([\s\S]*?)<\/a>/.exec(block);+ const href = a && /href="([^"]*)"/.exec(a[0])?.[1];+ const url = href ? ddgTarget(href) : null;+ const title = a ? decode(a[1]) : '';+ if (!url || !title) continue;+ const snippet = /<(?:a|div)\b[^>]*class="result__snippet"[^>]*>([\s\S]*?)<\/(?:a|div)>/.exec(block)?.[1] ?? '';+ out.push({ title, url, snippet: decode(snippet).slice(0, 300) });}return out;}-/** DuckDuckGo lets roughly every other automated request through: one retry after a short pause. */-const DDG_RETRY_MS = 1500;+export function parseDdgLite(html: string, max: number): SearchResult[] {+ const out: SearchResult[] = [];+ const links = [...html.matchAll(/<a\b[^>]*class=['"]result-link['"][^>]*>([\s\S]*?)<\/a>/g)];+ for (let i = 0; i < links.length && out.length < max; i++) {+ const at = links[i].index ?? 0;+ // A sponsored result sits in a row marked as such.+ if (/sponsored/i.test(html.slice(Math.max(0, html.lastIndexOf('<tr', at)), at))) continue;+ const href = /href=['"]([^'"]*)['"]/.exec(links[i][0])?.[1];+ const url = href ? ddgTarget(href) : null;+ const title = decode(links[i][1]);+ if (!url || !title) continue;+ const scope = html.slice(at, links[i + 1]?.index ?? html.length);+ const snippet = /<td\b[^>]*class=['"]result-snippet['"][^>]*>([\s\S]*?)<\/td>/.exec(scope)?.[1] ?? '';+ out.push({ title, url, snippet: decode(snippet).slice(0, 300) });+ }+ return out;+}++/** Either page: the html page has result__a links, the lite page result-link. */+export function parseDdg(html: string, max: number): SearchResult[] {+ return /class="result__a"/.test(html) ? parseDdgHtml(html, max) : parseDdgLite(html, max);+}++const DDG_CHALLENGE = /anomaly|challenge-form|bots use DuckDuckGo too/i;+const DDG_NO_RESULTS = /No results\.|no-results|No more results/i;async function ddgOnce(kind: 'html' | 'lite', query: string, max: number): Promise<SearchResult[]> {const ac = new AbortController();const timer = setTimeout(() => ac.abort(), TIMEOUT_MS);try {- const res = await guardedFetch(`https://${kind}.duckduckgo.com/${kind}/?q=${encodeURIComponent(query)}`, ac.signal, { 'User-Agent': UA, Accept: 'text/html' });+ const res = await guardedFetch(`https://${kind}.duckduckgo.com/${kind}/?q=${encodeURIComponent(query)}`, ac.signal, {+ 'User-Agent': UA,+ Accept: 'text/html',+ 'Accept-Language': 'en-US,en;q=0.9',+ Referer: `https://${kind}.duckduckgo.com/`,+ });const body = await res.text();- if (res.status === 202 || /anomaly/i.test(body)) throw new Error('bot challenge');+ if (res.status === 202 || DDG_CHALLENGE.test(body)) throw new Error('bot challenge');if (!res.ok) throw new Error(`HTTP ${res.status}`);- return parseDdg(body, max);+ const results = kind === 'html' ? parseDdgHtml(body, max) : parseDdgLite(body, max);+ if (!results.length && !DDG_NO_RESULTS.test(body)) throw new Error('page not recognised');+ return results;} finally {clearTimeout(timer);}}+/** DuckDuckGo lets roughly every other automated request through: one retry after a short pause. */+const DDG_RETRY_MS = 1500;+async function ddg(kind: 'html' | 'lite', query: string, max: number): Promise<SearchResult[]> {if (!query.trim()) return [];try {@@ -285,26 +307,6 @@ async function ddg(kind: 'html' | 'lite', query: string, max: number): Promise<Sconst ddgHtml = (q: string, n: number) => ddg('html', q, n);const ddgLite = (q: string, n: number) => ddg('lite', q, n);-/** Brave rate-limits per IP: after a 429 it is skipped for a minute rather than tried (and waited on) every query. */-const BRAVE_COOLDOWN_MS = 60_000;-let braveBlockedUntil = 0;-export function resetBraveCooldown(): void { braveBlockedUntil = 0; }--async function brave(query: string, max: number): Promise<SearchResult[]> {- if (!query.trim()) return [];- if (Date.now() < braveBlockedUntil) throw new Error('rate-limited, resting for a minute');- const ac = new AbortController();- const timer = setTimeout(() => ac.abort(), TIMEOUT_MS);- try {- const res = await guardedFetch(`https://search.brave.com/search?q=${encodeURIComponent(query)}&source=web`, ac.signal, { 'User-Agent': UA, Accept: 'text/html', 'Accept-Language': 'en-US,en;q=0.9' });- if (res.status === 429) braveBlockedUntil = Date.now() + BRAVE_COOLDOWN_MS;- if (!res.ok) throw new Error(`HTTP ${res.status}`);- return parseBrave(await res.text(), max);- } finally {- clearTimeout(timer);- }-}-async function googleNews(query: string, max: number): Promise<SearchResult[]> {const url = query.trim()? `https://news.google.com/rss/search?q=${encodeURIComponent(query)}&hl=en-US&gl=US&ceid=US:en`@@ -316,21 +318,6 @@ async function fetchJson(url: string): Promise<unknown> {return JSON.parse(await fetchText(url));}-/** Wikipedia's search API: keyless JSON, strong for facts, people, places, concepts. */-export async function wikipedia(query: string, max: number): Promise<SearchResult[]> {- if (!query.trim()) return [];- const data = await fetchJson(`https://en.wikipedia.org/w/api.php?action=query&list=search&srsearch=${encodeURIComponent(query)}&format=json&srlimit=${Math.min(10, max)}&srprop=snippet|timestamp`) as {- query?: { search?: Array<{ title: string; snippet?: string; timestamp?: string }> };- };- return (data.query?.search ?? []).map((hit) => ({- title: hit.title,- url: `https://en.wikipedia.org/wiki/${encodeURIComponent(hit.title.replace(/ /g, '_'))}`,- snippet: decode(hit.snippet ?? '').slice(0, 300),- source: 'Wikipedia',- date: hit.timestamp,- }));-}-/** Hacker News via Algolia: keyless JSON, strong for software, startups and tech news. */export async function hackerNews(query: string, max: number): Promise<SearchResult[]> {if (!query.trim()) return [];@@ -413,9 +400,9 @@ function dedupe(results: SearchResult[]): SearchResult[] {export interface SearchOutcome {results: SearchResult[];- /** The engines whose results are in the answer, e.g. "Brave + DuckDuckGo". */+ /** The engines whose results are in the answer, e.g. "Parallel + DuckDuckGo". */engine: string;- /** What each engine tried did: "Brave: blocked (HTTP 429)", "DuckDuckGo: 6 results". */+ /** What each engine tried did: "Parallel: timed out", "DuckDuckGo: 6 results". */trail: string[];}@@ -432,11 +419,10 @@ function why(err: unknown): string {export async function webSearch(query: string, opts: { kind?: 'web' | 'news'; max?: number } = {}): Promise<SearchOutcome> {const max = Math.max(1, Math.min(opts.max ?? 8, 15));- // Every engine is free and keyless; no library. Brave and DuckDuckGo are- // the real web indexes, and both block some automated requests (by client- // fingerprint and rate), so having two, plus DuckDuckGo's lite page, keeps- // web search answering. Wikipedia, Hacker News and the news feeds follow.- // Bing's web RSS is last: it serves unrelated pages to bots.+ // Every engine is free and keyless; no library. Web: Parallel's search,+ // then DuckDuckGo's html page, then its lite page (DuckDuckGo blocks some+ // automated requests, so each page gets one retry). News: Parallel's+ // article links, then Bing News (direct links), Google News, Hacker News.//// How a search is judged: an engine's answer counts only for the results// that mention the query (relevant()). Engines are asked in order and their@@ -445,7 +431,7 @@ export async function webSearch(query: string, opts: { kind?: 'web' | 'news'; ma// trail, so "nothing found" and "everything was blocked" read differently.const engines: Array<[string, (q: string, n: number) => Promise<SearchResult[]>]> = opts.kind === 'news'? [['Parallel', parallelNews], ['Bing News', bingNews], ['Google News', googleNews], ['Hacker News', hackerNews]]- : [['Parallel', parallel], ['Brave', brave], ['DuckDuckGo', ddgHtml], ['DuckDuckGo Lite', ddgLite], ['Wikipedia', wikipedia], ['Hacker News', hackerNews], ['Bing News', bingNews], ['Google News', googleNews], ['Bing', bing]];+ : [['Parallel', parallel], ['DuckDuckGo', ddgHtml], ['DuckDuckGo Lite', ddgLite]];const want = Math.min(ENOUGH, max);const trail: string[] = [];const used: string[] = [];diff --git a/src/skills/default-skills.ts b/src/skills/default-skills.tsindex efa444ed..9efea155 100644--- a/src/skills/default-skills.ts+++ b/src/skills/default-skills.ts@@ -27,6 +27,59 @@ export const PREVIOUS_SEED_CONTENTS: Record<string, string[]> = {'web-search': [`---name: web-searchdescription: Search the web and the news with the web_search tool, read the best sources, and answer with links.+version: 1.2.0+category: web+categories:+ - web+ - research+intents:+ - search the web+ - look up+ - find online+ - what is+ - current news+ - search for+ - find information about+ - research+ - web search+ - latest news+ - what's happening+tags:+ - search+ - web+ - research+ - news+allowed-tools:+ - web_search+ - fetch_url+---++# Web Search++Use this skill when the user asks for current events, external facts, or web research.++## Workflow++1. Call web_search with a focused query of a few key words (names, product, place, year).+ For news and recent events use kind "news" (an empty query gives today's top stories).+2. Open the most relevant one to three results with fetch_url when the snippets aren't enough.+3. Nothing useful? Rephrase once (synonyms, the official name, fewer words) and search again+ before saying you could not find it.+4. Cross-check key facts across at least 2 sources when possible.+5. Return a concise answer with source links and clear caveats.++## Rules++- Prefer reliable sources (official docs, primary sources, reputable publications).+- If information is uncertain or conflicting, say so explicitly.+- Include source URLs in the response (on phone chats, keep it to the one or two that matter).+- Avoid fabricated citations.+- Never fetch a search engine's results page (DuckDuckGo, Google, Bing) with fetch_url: they block+ automated reads. web_search already tries several free engines (Brave, Wikipedia, Hacker News,+ news feeds) and keeps only results that match the query.+`, `---+name: web-search+description: Search the web and the news with the web_search tool, read the best sources, and answer with links.version: 1.1.0category: webcategories:@@ -128,7 +181,7 @@ export const DEFAULT_SKILL_SEEDS: DefaultSkillSeed[] = [content: `---name: web-searchdescription: Search the web and the news with the web_search tool, read the best sources, and answer with links.-version: 1.2.0+version: 1.3.0category: webcategories:- web@@ -176,8 +229,23 @@ Use this skill when the user asks for current events, external facts, or web res- Include source URLs in the response (on phone chats, keep it to the one or two that matter).- Avoid fabricated citations.- Never fetch a search engine's results page (DuckDuckGo, Google, Bing) with fetch_url: they block- automated reads. web_search already tries several free engines (Brave, Wikipedia, Hacker News,- news feeds) and keeps only results that match the query.+ automated reads. web_search already does it properly.++## Where results come from++web_search needs no key and works with any model. For web searches it asks, in order:++1. Parallel search (search.parallel.ai), a search index built for agents, with excerpts and dates.+2. DuckDuckGo's html page.+3. DuckDuckGo's lite page.++News searches ask Parallel (article links only), then Bing News, Google News and Hacker News.+Only results that mention the query are kept, and sources are combined until there are enough.++Every result ends with a "Searched:" trail, for example+"Parallel: timed out → DuckDuckGo: 8 results". If web_search returns an error, every source failed:+tell the user which ones and why (blocked, timed out, no results) instead of guessing an answer,+then try one shorter or rephrased query.`,},{