Open-source AI browser agent for Chrome and Firefox (monorepo) 🧠
JavaScript ai-agentai-agentsai-toolsbrowser-aibrowser-automationbrowser-extensionchrome-aichrome-devtoolschrome-extensionchrome-extensionsfirefox-addonfirefox-addonsfirefox-extensionfirefox-extensionsgemmagemma4minimaxoffline-aioffline-llmqwen
Hand older readings back to the page, key cut texts alike, and read walls in chunks
Claude0a82fff225fb
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 1d64606dd..bd44e3bf4 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -7,8 +7,8 @@ This changelog was generated from the repository Git history and release tags. V## [Unreleased]### Fixed-- A subscription, registration or sign-in wall in front of a news article is now recognized in any language, by a model reading the wall's text once, instead of only when it used English phrases such as "Subscribe to continue reading". A Turkish "Okumaya devam etmek için abone olun" overlay now keeps WebBrain from reading the article hidden behind it, as an English one did, while an offer to subscribe beside text you can still read is not taken for a wall.-- A dialog asking you to prove you are human now pauses the run's page actions in any language, not only when it says "Security verification", "Verify you are human" or another English phrase. Its title is read by a model once, the decision model first; LinkedIn's Turkish "Güvenlik doğrulaması" now counts, while a cookie notice or a form dialog does not. A dialog naming a CAPTCHA product still counts without any reading.+- A subscription, registration or sign-in wall in front of a news article is now recognized in any language, by a model reading the wall's text once, instead of only when it used English phrases such as "Subscribe to continue reading". A Turkish "Okumaya devam etmek için abone olun" overlay now keeps WebBrain from reading the article hidden behind it, as an English one did, while an offer to subscribe beside text you can still read is not taken for a wall. A wall inside the article is found also after a long preview, and everything after the wall's text stays withheld; the preview before it stays readable, except the short passage just before the wall's text. If a page cannot be read again after its wall was recognized, the read fails rather than return what the wall hides.+- A dialog asking you to prove you are human now pauses the run's page actions in any language, not only when it says "Security verification", "Verify you are human" or another English phrase. Its title is read by a model once, the decision model first; LinkedIn's Turkish "Güvenlik doğrulaması" now counts, while a cookie notice or a form dialog does not. A dialog naming a CAPTCHA product still counts without any reading. A title no model can answer stops nothing and is asked about again only after half a minute, so page actions do not each wait for it.- Finding a verification email: the service the code is for is now matched by its name or web address alone, with no English words such as "code" or "verification" set aside, so the search treats the same request in every language alike. The tool now asks the model for only the service's name or domain.- An AWS WAF puzzle page ("Let's confirm you are human" in English) is now recognized in any language, by AWS's own puzzle script on its interstitial page, instead of by its English heading. Before, the same page in another language was not recognized as a CAPTCHA unless its widget was found, and a page running only AWS's silent check is still not taken for a puzzle.- Editing a file on GitHub in another interface language: the "Commit changes…" button that only opens the commit dialog is now recognized by its closing ellipsis or by the dialog it names, in any language, instead of by the English words, so a Polish "Zatwierdź zmiany…" is treated like the English one. A commit message field beside the button is recognized by GitHub's own field id instead of its English label, so a button next to one keeps the full commit checks in every language.diff --git a/docs/architecture.md b/docs/architecture.mdindex 3403345d3..7a57a2d38 100644--- a/docs/architecture.md+++ b/docs/architecture.md@@ -579,12 +579,41 @@ page prose cannot spoof fallback routing.A `pageGate` is a wall in front of an article: an overlay on an article page, ora block inside the article, whose text a model read as keeping the articlefrom the reader until they subscribe, register or sign in (the `page_blocker`-question, in any language; an offer beside readable text is not a wall). The-page read reports an overlay or block text nobody has read in-`pageTextsNeeded`; the agent has it read (the decision model first, the active-model within four seconds) and reads the page once more with the reading. An-unread text is no wall, so the page reads as it is. A wall withholds the-article, its links, forms and frames; its own text and controls stay readable.+question, in any language; an offer beside readable text is not a wall). An+overlay or block is read by its own name and title and by chunks of its+rendered text: its elements in page order, an element longer than 600+characters with block children split into them, gathered into texts of at+most 600 characters, a heading starting a new one, and a longer stretch of+inline text read in pieces of that length. The chunks are read in page order+up to the first one read as a wall, at most 30 of them. In an article the+wall starts at that chunk, or at a chunk before it nobody has read, since the+wall's text may be there, so only text read as no wall comes before it: a+preview in chunks before the wall stays readable, also inside the wall's own+block, while the part of the wall's own chunk before its text is withheld+with it. Of several walls in an article, the one that starts first decides;+an overlay is the wall as a whole. A wall's label is its name or title, or+the first line of the element its starting chunk begins in, and none when an+unread chunk comes first, since what follows a wall's first line may be the+article. A page read reports at most 40 texts nobody has read, overlays first+and then blocks offering a control a wall would (a button, a field, a form or+a link standing on its own), in `pageTextsNeeded`; the agent has them read+(the decision model first, the active model within four seconds) and reads+the page again with the readings, up to three times while each read reaches+more. When a text reads as a wall and the page cannot be read again, the+read fails rather than return the answer made without that reading. An+unread text is no wall, so the page reads as it is, and a text no model+answered after both had their turn is not asked again for 30 seconds. A wall+withholds the article, its links, forms and frames; its own text and+controls stay readable. The CDP mirror of the page read (`readPage` in+cdp-client.js) applies the same rules.++Checks that run in the page, such as this one, the CAPTCHA scans, the submit+probe and the recipient guard, receive the agent's most recent readings of+their questions (the latest 400, at most 80,000 characters). A text the agent+read earlier that a check still reports unread is not asked again: its reading+becomes one of the most recent and the check runs again with it. Texts are+keyed alike on both sides: spaces collapsed, and a text cut at its limit loses+a trailing space or half of a surrogate pair.Single public-media downloads have a second deterministic guard. If the modelcalls `download_social_media` while an eligible inactive skill ownsdiff --git a/docs/captcha-provider-coverage.md b/docs/captcha-provider-coverage.mdindex 8eba1255f..c51ec283f 100644--- a/docs/captcha-provider-coverage.md+++ b/docs/captcha-provider-coverage.md@@ -111,7 +111,7 @@ After manual completion navigates to another page, the next accessibility read c### Automatic route-A visible dialog stops the run's page actions as a challenge when any of its text names a CAPTCHA product (CAPTCHA, reCAPTCHA, hCaptcha, FunCaptcha), or when a model reads its title as asking the person to prove they are human, in any language (the `page_blocker` question, read once per title, the decision model first). A dialog's title is its accessible name, else its first heading, else its title attribute, else its first line of text when nothing in it is editable. A title nobody could read stops nothing, as a title in a language the former English phrase list lacked never did. Vendor challenge frames (reCAPTCHA, hCaptcha, Arkose) arm the gate without any reading.+A visible dialog stops the run's page actions as a challenge when any of its text names a CAPTCHA product (CAPTCHA, reCAPTCHA, hCaptcha, FunCaptcha), or when a model reads its title as asking the person to prove they are human, in any language (the `page_blocker` question, read once per title, the decision model first). A dialog's title is its accessible name, else its first heading, else its title attribute, else its first line of text when nothing in it is editable. A title nobody could read stops nothing, as a title in a language the former English phrase list lacked never did, and one no model answered is not asked about again for 30 seconds, so page actions do not each wait for it. Vendor challenge frames (reCAPTCHA, hCaptcha, Arkose) arm the gate without any reading.AWS WAF is also routed automatically from a visible challenge. The runtime observes initial `gokuProps` inputs or SDK/resource inputs for [CapSolver's secondary-verification contract](https://docs.capsolver.com/en/guide/captcha/awsWaf/), including the observed SDK API key and existing AWS token. Initial cookie tasks can use CapSolver, CapMonster, and Anti-Captcha. SDK widget tasks now use CapMonster Option 1 and Anti-Captcha's Widget mode as well as CapSolver secondary verification when an existing token is available (otherwise its documented script-only mode). All use the observed SDK API key and script URL; stale interstitial inputs are excluded. Invisible challenge-only pages also support CapMonster Option 3, with its documented empty `iv` and `context`. Enabling a recognition or voucher-returning service does not make it an applicable cookie fallback. `solve_captcha` applies the returned cookie unless `inject:false` was requested, then requires a reload and fresh inspection. A returned answer or successful cookie write alone is not clearance.diff --git a/src/chrome/src/agent/agent.js b/src/chrome/src/agent/agent.jsindex 83ec2f70f..de260ab80 100644--- a/src/chrome/src/agent/agent.js+++ b/src/chrome/src/agent/agent.js@@ -275,6 +275,10 @@ const COMPLETION_READING_TIMEOUT_MS = 6_000;// language, after a click in a run that has to prove a submission; the active// model gets this long when the decision model is not sure.const CONTROL_READING_TIMEOUT_MS = 4_000;+// A dialog title or a wall text no model answered reads as nothing to stop+// for; it is asked again only after this long, so page reads and actions do+// not each wait for it.+const PAGE_BLOCKER_RETRY_MS = 30_000;// The agent's own hover may start a transition on a certified native button.// Its geometry is sampled (read-only) at this interval until it settles, for// at most this long, before the single press.@@ -7461,20 +7465,53 @@ export class Agent extends LoopDetector {// workflow reached its end state) read with one fixed-choice question each,// in any language, and remembered per text for the synchronous checks that// follow. Longer texts are not read; an unanswered text is asked again next- // time.- async _readPageTexts(tabId, kind, texts, context = {}, { maxChars = 300, mainTimeoutMs, mainModel = true } = {}) {+ // time, or, with `retryUnansweredAfterMs`, once that long has passed. A text+ // read before is not read again: its reading becomes the most recent one,+ // so the readings handed to a check in the page (the most recent of each+ // question) include it however long ago it was read.+ async _readPageTexts(tabId, kind, texts, context = {}, { maxChars = 300, mainTimeoutMs, mainModel = true,+ retryUnansweredAfterMs = 0 } = {}) {this._pageTextReadings ??= new Map();+ this._unansweredPageTexts ??= new Map();+ const now = Date.now();const pending = [...new Set((Array.isArray(texts) ? texts : []).map(text => String(text ?? '').replace(/\s+/g, ' ').trim()).filter(text => text && text.length <= maxChars))]- .filter(text => !this._pageTextReadings.has(this._pageTextKey(kind, text, context)));+ .filter(text => {+ const key = this._pageTextKey(kind, text, context);+ if (this._pageTextReadings.has(key)) {+ const choice = this._pageTextReadings.get(key);+ this._pageTextReadings.delete(key);+ this._pageTextReadings.set(key, choice);+ return false;+ }+ const unansweredAt = this._unansweredPageTexts.get(key);+ return !(retryUnansweredAfterMs > 0 && unansweredAt && now - unansweredAt < retryUnansweredAfterMs);+ });if (!pending.length) return;+ const askMain = mainModel && !this._workflowReplayTabs?.has(tabId);const readings = await this._semanticDecisions(tabId, kind, { items: pending.map(text => ({ ...context, text })) },- { ...(mainTimeoutMs ? { mainTimeoutMs } : {}), mainModel: mainModel && !this._workflowReplayTabs?.has(tabId) });+ { ...(mainTimeoutMs ? { mainTimeoutMs } : {}), mainModel: askMain });+ // A text counts as unanswered only once the active model had its turn+ // too: a read by the decision model alone, a stopped run or a provider+ // that cannot answer leaves it for the next read.+ let mainHadTurn = false;+ try {+ mainHadTurn = askMain && typeof this._activeProvider(tabId)?.chat === 'function' && !this._checkAbort(tabId);+ } catch {}+ const answeredAt = Date.now();pending.forEach((text, index) => {- if (readings[index]?.choice) this._pageTextReadings.set(this._pageTextKey(kind, text, context), readings[index].choice);+ const key = this._pageTextKey(kind, text, context);+ if (readings[index]?.choice) {+ this._pageTextReadings.set(key, readings[index].choice);+ this._unansweredPageTexts.delete(key);+ } else if (mainHadTurn) {+ this._unansweredPageTexts.delete(key);+ this._unansweredPageTexts.set(key, answeredAt);+ }});while (this._pageTextReadings.size > 2048) this._pageTextReadings.delete(this._pageTextReadings.keys().next().value);+ while (this._unansweredPageTexts.size > 512) this._unansweredPageTexts.delete(this._unansweredPageTexts.keys().next().value);}_pageTextReading(kind, text, context = {}) {@@ -12116,20 +12153,31 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d_withPageBlockerReadings(name, params) {if (name !== 'read_page' && name !== 'get_accessibility_tree') return params;return { ...(params && typeof params === 'object' ? params : {}),- __wbPageReadings: { page_blocker: this._challengeTitleReadings() } };+ __wbPageReadings: { page_blocker: this._pageBlockerReadings() } };+ }++ // The page_blocker readings the agent holds, for a page read's page-gate+ // check: overlay and article-block texts as well as dialog titles.+ _pageBlockerReadings() {+ return this._pageTextReadingsByCaption('page_blocker');}// Overlay and article-block texts a page read consulted unread, read in any// language (the decision model first, the active model within four- // seconds). True when any of them was read.+ // seconds). A text the agent read before, but that did not reach the page+ // among the most recent readings, becomes one of them again. A text no+ // model answered reads as no wall, and is not asked again for half a+ // minute, so page reads do not each wait for it. True when any of them has+ // a reading.async _readPageBlockerTexts(tabId, entries) {const texts = [...new Set((Array.isArray(entries) ? entries : []).filter(entry => entry?.kind === 'page_blocker').map(entry => String(entry?.text ?? '').replace(/\s+/g, ' ').trim())- .filter(text => text && text.length <= 400 && !this._pageTextReading('page_blocker', text)))].slice(0, 12);+ .filter(text => text && text.length <= 600))].slice(0, 48);if (!texts.length) return false;try {- await this._readPageTexts(tabId, 'page_blocker', texts, {}, { maxChars: 400, mainTimeoutMs: CONTROL_READING_TIMEOUT_MS });+ await this._readPageTexts(tabId, 'page_blocker', texts, {},+ { maxChars: 600, mainTimeoutMs: CONTROL_READING_TIMEOUT_MS, retryUnansweredAfterMs: PAGE_BLOCKER_RETRY_MS });} catch (error) {if (this._checkAbort(tabId)) throw error;}@@ -12137,29 +12185,43 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d}// A page read whose overlay or article-block texts were consulted unread- // has them read, and the page read once more with the readings; a read- // that cannot be repeated keeps the first answer.+ // has them read, and the page read once more with the readings, up to three+ // times while each read reaches texts nobody has read (a page reports at+ // most forty at a time, the likeliest walls first). A read that cannot be+ // repeated keeps the answer before it, unless one of the texts that answer+ // was made without reads as a wall: that answer may hold what the wall+ // withholds, so the read fails instead.async _rereadWithPageBlockerReadings(tabId, name, response, dispatch) {if ((name !== 'read_page' && name !== 'get_accessibility_tree') || !response || typeof response !== 'object') return response;- const needed = response.pageTextsNeeded;- delete response.pageTextsNeeded;- if (!await this._readPageBlockerTexts(tabId, needed)) return response;- let reread = null;- try {- reread = await dispatch();- } catch (error) {- if (error?.name === 'AbortError' || this._checkAbort(tabId)) throw error;- return response;+ let current = response;+ for (let pass = 0; pass < 3; pass += 1) {+ const needed = current.pageTextsNeeded;+ delete current.pageTextsNeeded;+ if (!await this._readPageBlockerTexts(tabId, needed)) return current;+ const wallRead = (Array.isArray(needed) ? needed : []).some(entry => entry?.kind === 'page_blocker'+ && ['subscription', 'registration', 'login'].includes(this._pageTextReading('page_blocker', entry.text)));+ const failed = () => (wallRead ? { success: false, error: 'This page shows a subscription, registration or '+ + 'sign-in wall, and reading the page again with it failed, so nothing behind the wall was returned. '+ + 'Read the page again.' } : current);+ let reread = null;+ try {+ reread = await dispatch();+ } catch (error) {+ if (error?.name === 'AbortError' || this._checkAbort(tabId)) throw error;+ return failed();+ }+ if (!reread || typeof reread !== 'object' || reread.success === false) return failed();+ current = reread;}- if (!reread || typeof reread !== 'object' || reread.success === false) return response;- delete reread.pageTextsNeeded;- return reread;+ delete current.pageTextsNeeded;+ return current;}// The page_blocker readings of dialog titles the agent holds, by title, for- // the CAPTCHA scans that run in the page.+ // the CAPTCHA scans that run in the page: titles are at most 200+ // characters, so the longer overlay and article-block texts stay behind._challengeTitleReadings() {- return this._pageTextReadingsByCaption('page_blocker');+ return this._pageTextReadingsByCaption('page_blocker', { maxLength: 200 });}_challengeTitleReader() {@@ -12168,15 +12230,18 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d// Reads dialog titles, in any language, as asking to prove the person is// human or not: the decision model first, the active model within four- // seconds unless `mainModel` is false. True when any of them was read.+ // seconds unless `mainModel` is false. A title read before becomes one of+ // the most recent readings again, so a scan in the page receives it; one no+ // model answered stops nothing and is not asked again for half a minute,+ // so page actions do not each wait for it. True when any of them has a+ // reading.async _readChallengeTitles(tabId, titles = [], { mainModel = true } = {}) {const pending = [...new Set((Array.isArray(titles) ? titles : [])- .map(title => challengeTitleText(title)).filter(Boolean))]- .filter(title => !this._pageTextReading('page_blocker', title)).slice(0, 12);+ .map(title => challengeTitleText(title)).filter(Boolean))].slice(0, 12);if (!pending.length) return false;try {await this._readPageTexts(tabId, 'page_blocker', pending, {},- { maxChars: 200, mainTimeoutMs: CONTROL_READING_TIMEOUT_MS, mainModel });+ { maxChars: 200, mainTimeoutMs: CONTROL_READING_TIMEOUT_MS, mainModel, retryUnansweredAfterMs: PAGE_BLOCKER_RETRY_MS });} catch (error) {if (this._checkAbort(tabId)) throw error;}@@ -24989,14 +25054,22 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d}// The readings of one page-text question the agent holds, by caption, for a- // check that runs in the page: the 400 most recent.- _pageTextReadingsByCaption(kind) {+ // check that runs in the page: the 400 most recent, of captions up to+ // `maxLength` characters, within `budget` characters in all. A check that+ // consults a text missing here reports it; reading it again makes it one of+ // the most recent (see _readPageTexts).+ _pageTextReadingsByCaption(kind, { maxLength = Infinity, budget = 80_000 } = {}) {const prefix = this._pageTextKey(kind, '', {});const readings = {};const entries = [...(this._pageTextReadings || new Map()).entries()].filter(([key]) => key.startsWith(prefix));- for (const [key, choice] of entries.slice(-400)) {- const caption = key.slice(prefix.length);- if (caption) readings[caption] = choice;+ let count = 0, used = 0;+ for (let index = entries.length - 1; index >= 0 && count < 400; index -= 1) {+ const caption = entries[index][0].slice(prefix.length);+ if (!caption || caption.length > maxLength) continue;+ if (used + caption.length > budget) break;+ readings[caption] = entries[index][1];+ count += 1;+ used += caption.length;}return readings;}@@ -42182,9 +42255,9 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout dawait cdpClient.attach(tabId);// Shadow hosts behind a wall in front of an article stay hidden; the// wall's text is read in any language first when nobody has.- let pageInfo = await cdpClient.readPage(tabId, { pageBlockerReadings: this._challengeTitleReadings() });- if (await this._readPageBlockerTexts(tabId, pageInfo?.pageTextsNeeded)) {- pageInfo = await cdpClient.readPage(tabId, { pageBlockerReadings: this._challengeTitleReadings() });+ let pageInfo = await cdpClient.readPage(tabId, { pageBlockerReadings: this._pageBlockerReadings() });+ for (let pass = 0; pass < 3 && await this._readPageBlockerTexts(tabId, pageInfo?.pageTextsNeeded); pass += 1) {+ pageInfo = await cdpClient.readPage(tabId, { pageBlockerReadings: this._pageBlockerReadings() });}return {success: true,diff --git a/src/chrome/src/agent/captcha-frame-runtime.js b/src/chrome/src/agent/captcha-frame-runtime.jsindex d958de50a..83a8c80eb 100644--- a/src/chrome/src/agent/captcha-frame-runtime.js+++ b/src/chrome/src/agent/captcha-frame-runtime.js@@ -709,7 +709,8 @@ export function detectCaptchaCandidatesInPage(scope = null, matcherOptions = nul// keyed by the title as captcha-gate.js takes it).const givenTitleReadings = matcherOptions?.challengeReadings && typeof matcherOptions.challengeReadings === 'object'? matcherOptions.challengeReadings : {};- const titleText = value => String(value || '').replace(/\s+/g, ' ').trim().slice(0, 200);+ const titleText = value => String(value || '').replace(/\s+/g, ' ').trim().slice(0, 200)+ .replace(/[\ud800-\udbff]$/, '').trim();const challengeDialogs = Array.from(pageDocument.querySelectorAll('dialog, [role="dialog"], [role="alertdialog"]')).filter((element) => {diff --git a/src/chrome/src/agent/captcha-gate.js b/src/chrome/src/agent/captcha-gate.jsindex d01c7c470..76fba7eb7 100644--- a/src/chrome/src/agent/captcha-gate.js+++ b/src/chrome/src/agent/captcha-gate.js@@ -22,9 +22,11 @@ function matchesChallengeLabel(value) {}// A dialog's title as the readings key it: spaces collapsed, at most 200-// characters.+// characters, and neither a space nor half of a surrogate pair at its end+// once cut, so the page and the agent key a long title alike. The scans+// that run in the page and captcha-frame-runtime.js keep the same rule.export function challengeTitleText(value) {- return String(value || '').replace(/\s+/g, ' ').trim().slice(0, 200);+ return String(value || '').replace(/\s+/g, ' ').trim().slice(0, 200).replace(/[\ud800-\udbff]$/, '').trim();}function normalizeChallengeLabel(value) {@@ -156,7 +158,9 @@ export function detectChallengeDialogInPage(options = null) {? options.challengeReadings : {};const titleReading = title => (Object.prototype.hasOwnProperty.call(givenTitleReadings, title)? String(givenTitleReadings[title] || '') : '');- const titleText = value => String(value || '').replace(/\s+/g, ' ').trim().slice(0, 200);+ // Keyed as challengeTitleText keys a title.+ const titleText = value => String(value || '').replace(/\s+/g, ' ').trim().slice(0, 200)+ .replace(/[\ud800-\udbff]$/, '').trim();const unreadTitles = [];const visible = (element) => {try {diff --git a/src/chrome/src/cdp/cdp-client.js b/src/chrome/src/cdp/cdp-client.jsindex bfda5e083..ef7a40459 100644--- a/src/chrome/src/cdp/cdp-client.js+++ b/src/chrome/src/cdp/cdp-client.js@@ -4037,20 +4037,130 @@ export class CDPClient {);} catch (e) { return false; }};+ // The page-gate check of content.js, mirrored: texts keyed as the+ // readings key them, a block read in chunks in page order up to its+ // first wall, a block's wall starting at that chunk or at an unread+ // chunk before it, and the likeliest walls read first.+ const GATE_TEXT_LIMIT = 600;+ const GATE_TYPES = ['subscription', 'registration', 'login'];const unreadGateTexts = [];- const gateText = (el) => String(el?.innerText || '').replace(/[\\u200b-\\u200d\\ufeff]/g, '').replace(/\\s+/g, ' ').trim().slice(0, 400);+ const gateNormalize = (value) => String(value ?? '').replace(/[\\u200b-\\u200d\\ufeff]/g, '').replace(/\\s+/g, ' ').trim();+ const gateSlice = (value) => gateNormalize(gateNormalize(value).slice(0, GATE_TEXT_LIMIT)+ .replace(/^[\\udc00-\\udfff]/, '').replace(/[\\ud800-\\udbff]$/, ''));const gateReading = (text) => {- if (!text) return null;+ if (!text) return 'other';const reading = Object.prototype.hasOwnProperty.call(pageBlockerReadings, text) ? String(pageBlockerReadings[text] || '') : '';- if (!reading) {- if (!unreadGateTexts.includes(text) && unreadGateTexts.length < 12) unreadGateTexts.push(text);- return null;+ if (reading) return GATE_TYPES.includes(reading) ? reading : 'other';+ if (!unreadGateTexts.includes(text)) {+ if (unreadGateTexts.length >= 40) return null;+ unreadGateTexts.push(text);+ }+ return null;+ };+ const gateHasBlockChildren = (el, depth = 0) => {+ for (const child of el.children) {+ let display = '';+ try { display = getComputedStyle(child).display; } catch (e) { continue; }+ if (display === 'contents') {+ if (depth < 20 && gateHasBlockChildren(child, depth + 1)) return true;+ continue;+ }+ if (display && display !== 'none' && !display.startsWith('inline')) return true;+ }+ return false;+ };+ const gateParts = (el) => {+ const parts = [];+ const firstLine = value => gateNormalize(String(value || '').split('\\n').find(line => line.trim()) || '');+ const visit = (node, depth) => {+ for (const child of node.childNodes) {+ if (parts.length >= 600) return;+ if (child.nodeType === Node.TEXT_NODE) {+ const value = gateNormalize(child.data);+ if (value) parts.push({ text: value, line: value, element: node, heading: false });+ continue;+ }+ if (child.nodeType !== Node.ELEMENT_NODE || /^(?:SCRIPT|STYLE|NOSCRIPT|TEMPLATE)$/.test(child.tagName)) continue;+ let display = '';+ try { display = getComputedStyle(child).display; } catch (e) {}+ if (display === 'none') continue;+ if (display === 'contents') {+ if (depth < 40) visit(child, depth + 1);+ continue;+ }+ if (!child.getClientRects?.().length) continue;+ const raw = String(child.innerText || '');+ const value = gateNormalize(raw);+ if (!value) continue;+ if (value.length <= GATE_TEXT_LIMIT || depth >= 40 || !gateHasBlockChildren(child)) {+ parts.push({ text: value, line: firstLine(raw), element: child,+ heading: child.matches?.('h1, h2, h3, h4, h5, h6, [role="heading"]') === true });+ } else {+ visit(child, depth + 1);+ }+ }+ };+ visit(el, 0);+ return parts;+ };+ const gateChunks = (parts) => {+ const chunks = [];+ for (const part of parts) {+ if (part.text.length > GATE_TEXT_LIMIT) {+ for (let offset = 0; offset < part.text.length && chunks.length < 1000; offset += GATE_TEXT_LIMIT) {+ chunks.push({ text: part.text.slice(offset, offset + GATE_TEXT_LIMIT), element: part.element,+ line: offset ? '' : part.line, closed: true });+ }+ continue;+ }+ const last = chunks[chunks.length - 1];+ if (last && !last.closed && !part.heading && last.text.length + 1 + part.text.length <= GATE_TEXT_LIMIT) {+ last.text += ' ' + part.text;+ } else {+ chunks.push({ text: part.text, element: part.element, line: part.line, closed: false });+ }+ }+ return chunks.map(chunk => ({ ...chunk, text: gateSlice(chunk.text) }));+ };+ const gateWall = (el, inline) => {+ const named = [...new Set([el.getAttribute('aria-label'), el.getAttribute('title')].map(gateSlice))].filter(Boolean);+ const namedReadings = named.map(gateReading);+ const namedIndex = namedReadings.findIndex(reading => GATE_TYPES.includes(reading));+ const chunks = gateChunks(gateParts(el)).slice(0, 30);+ let wallIndex = -1, unread = -1, wallType = null;+ for (let index = 0; index < chunks.length; index += 1) {+ const reading = gateReading(chunks[index].text);+ if (GATE_TYPES.includes(reading)) {+ wallIndex = index;+ wallType = reading;+ break;+ }+ if (reading === null && unread < 0) unread = index;+ }+ if (wallIndex < 0 && namedIndex < 0) return null;+ const type = namedIndex >= 0 ? namedReadings[namedIndex] : wallType;+ const label = value => gateSlice(String(value || '').slice(0, 240));+ if (namedIndex >= 0) return { type, label: label(named[namedIndex]), element: el };+ if (!inline) return { type, label: label(chunks[wallIndex].line), element: el };+ const start = unread >= 0 && unread < wallIndex ? unread : wallIndex;+ return {+ type,+ label: start === wallIndex ? label(chunks[wallIndex].line) : '',+ element: start === 0 ? el : chunks[start].element,+ };+ };+ const gateOffersControl = (el) => {+ try {+ if (el.querySelector('button, input, select, textarea, form, [role="button"]')) return true;+ return Array.from(el.querySelectorAll('a[href]')).slice(0, 50)+ .some(link => !link.closest('p, li, blockquote, figcaption, td, dd'));+ } catch (e) {+ return false;}- return ['subscription', 'registration', 'login'].includes(reading) ? reading : null;};const detectPageGate = () => {const seen = new Set();- const candidates = [];+ const found = [];const articleContext = pageHasArticleContext();for (const selector of PAGE_GATE_SELECTORS) {let matches = [];@@ -4072,16 +4182,26 @@ export class CDPClient {} catch (e) {}if (coveringOverlay) surface = 'dialog';if (surface === 'dialog' ? !articleContext : !inArticle) continue;- const label = gateText(el);- const type = gateReading(label);- if (!type) continue;- const namedGate = /paywall|gateway|regiwall|subscription|registration/i.test([el.id, typeof el.className === 'string' ? el.className : '', el.getAttribute('data-testid') || ''].join(' '));- const score = (surface === 'dialog' ? 100 : 0) + (inArticle ? 40 : 0) + (coveringOverlay ? 30 : 0) + (namedGate ? 15 : 0) - Math.min(label.length, 2000) / 2000;- candidates.push({ element: el, type, surface, label: label.slice(0, 240), score });+ found.push({ el, surface, inArticle, coveringOverlay, offersControl: gateOffersControl(el) });}}- candidates.sort((a, b) => b.score - a.score);- return candidates[0] || null;+ found.sort((a, b) => (Number(b.surface === 'dialog') - Number(a.surface === 'dialog'))+ || (Number(b.offersControl) - Number(a.offersControl))+ || (a.el === b.el ? 0 : (a.el.compareDocumentPosition(b.el) & Node.DOCUMENT_POSITION_FOLLOWING ? -1 : 1)));+ const candidates = [];+ for (const { el, surface, inArticle, coveringOverlay } of found) {+ const wall = gateWall(el, surface === 'inline');+ if (!wall) continue;+ const label = wall.label;+ const namedGate = /paywall|gateway|regiwall|subscription|registration/i.test([el.id, typeof el.className === 'string' ? el.className : '', el.getAttribute('data-testid') || ''].join(' '));+ const score = (surface === 'dialog' ? 100 : 0) + (inArticle ? 40 : 0) + (coveringOverlay ? 30 : 0) + (namedGate ? 15 : 0) - Math.min(label.length, 2000) / 2000;+ candidates.push({ element: wall.element, type: wall.type, surface, label, score });+ }+ const overlays = candidates.filter(candidate => candidate.surface === 'dialog').sort((a, b) => b.score - a.score);+ if (overlays.length) return overlays[0];+ const blocks = candidates.filter(candidate => candidate.surface === 'inline');+ blocks.sort((a, b) => (a.element === b.element ? 0 : (a.element.compareDocumentPosition(b.element) & Node.DOCUMENT_POSITION_FOLLOWING ? -1 : 1)));+ return blocks[0] || null;};const textBeforeGate = (root, gateElement) => {if (!root || !gateElement || !root.contains(gateElement)) return '';diff --git a/src/chrome/src/content/content.js b/src/chrome/src/content/content.jsindex dd5d3ee32..077e7e43e 100644--- a/src/chrome/src/content/content.js+++ b/src/chrome/src/content/content.js@@ -3525,15 +3525,167 @@// text nobody has read yet is no wall; it is reported for the agent to// read, and the agent asks again with the reading.const PAGE_GATE_TYPES = new Set(['subscription', 'registration', 'login']);+ // The longest text one wall reading covers; how many of a block's texts are+ // read at most; and how many texts nobody has read one page read reports at+ // most (the agent reads them and reads the page again, up to three times).+ const PAGE_GATE_TEXT_LIMIT = 600;+ const PAGE_GATE_BLOCK_CHUNKS = 30;+ const PAGE_GATE_UNREAD_BUDGET = 40;++ // At most PAGE_GATE_TEXT_LIMIT characters of a text, keyed as the readings+ // key it: spaces collapsed, and no space or half of a surrogate pair at+ // either end.+ function pageGateSlice(value) {+ return _pageReadingText(_pageReadingText(value).slice(0, PAGE_GATE_TEXT_LIMIT)+ .replace(/^[\udc00-\udfff]/, '').replace(/[\ud800-\udbff]$/, ''));+ }++ // What a text reads as: a wall's kind, 'other' when it was read as no wall+ // (or is empty), or null when nobody has read it. Within one page read only+ // the first PAGE_GATE_UNREAD_BUDGET texts nobody has read are reported to+ // the agent, the likeliest walls first; the rest wait for the next read.+ function pageGateReading(text, budget) {+ if (!text) return 'other';+ const known = _pageReading('page_blocker', text, null, { queue: false, maxLength: PAGE_GATE_TEXT_LIMIT });+ if (known) return PAGE_GATE_TYPES.has(known) ? known : 'other';+ if (!budget.seen.has(text)) {+ if (budget.left <= 0) return null;+ budget.left -= 1;+ budget.seen.add(text);+ }+ _pageReading('page_blocker', text, null, { maxLength: PAGE_GATE_TEXT_LIMIT });+ return null;+ }++ // Whether an element lays its children out as blocks, so its text splits+ // into them; an element of inline text is one part however long it is.+ function pageGateHasBlockChildren(el, depth = 0) {+ for (const child of el.children) {+ let display = '';+ try { display = getComputedStyle(child).display; } catch { continue; }+ if (display === 'contents') {+ if (depth < 20 && pageGateHasBlockChildren(child, depth + 1)) return true;+ continue;+ }+ if (display && display !== 'none' && !display.startsWith('inline')) return true;+ }+ return false;+ }- function pageGateText(el) {- return _pageReadingText(el?.innerText || '').slice(0, 400);+ // A block's rendered text in parts, in page order: an element whose text+ // is at most PAGE_GATE_TEXT_LIMIT characters, or that holds only inline+ // text, is one part; a longer one with block children is split into them,+ // and text standing directly in it is a part held by it. Hidden elements+ // are left out. Each part keeps its first line.+ function pageGateParts(el) {+ const parts = [];+ const firstLine = value => _pageReadingText(String(value || '').split('\n').find(line => line.trim()) || '');+ const visit = (node, depth) => {+ for (const child of node.childNodes) {+ if (parts.length >= 600) return;+ if (child.nodeType === Node.TEXT_NODE) {+ const text = _pageReadingText(child.data);+ if (text) parts.push({ text, line: text, element: node, heading: false });+ continue;+ }+ if (child.nodeType !== Node.ELEMENT_NODE || /^(?:SCRIPT|STYLE|NOSCRIPT|TEMPLATE)$/.test(child.tagName)) continue;+ let display = '';+ try { display = getComputedStyle(child).display; } catch { /* treat as rendered */ }+ if (display === 'none') continue;+ if (display === 'contents') {+ if (depth < 40) visit(child, depth + 1);+ continue;+ }+ if (!child.getClientRects?.().length) continue;+ const raw = String(child.innerText || '');+ const text = _pageReadingText(raw);+ if (!text) continue;+ if (text.length <= PAGE_GATE_TEXT_LIMIT || depth >= 40 || !pageGateHasBlockChildren(child)) {+ parts.push({ text, line: firstLine(raw), element: child,+ heading: child.matches?.('h1, h2, h3, h4, h5, h6, [role="heading"]') === true });+ } else {+ visit(child, depth + 1);+ }+ }+ };+ visit(el, 0);+ return parts;+ }++ // A block's parts gathered, in page order, into texts of at most+ // PAGE_GATE_TEXT_LIMIT characters, a heading starting a new one; a longer+ // part is read in pieces of that length. Each is read as a whole: a wall+ // read in it starts no earlier than the element it begins in, and its label+ // is that element's first line, none for a piece after a long part's first.+ function pageGateChunks(parts) {+ const chunks = [];+ for (const part of parts) {+ if (part.text.length > PAGE_GATE_TEXT_LIMIT) {+ for (let offset = 0; offset < part.text.length && chunks.length < 1000; offset += PAGE_GATE_TEXT_LIMIT) {+ chunks.push({ text: part.text.slice(offset, offset + PAGE_GATE_TEXT_LIMIT), element: part.element,+ line: offset ? '' : part.line, closed: true });+ }+ continue;+ }+ const last = chunks[chunks.length - 1];+ if (last && !last.closed && !part.heading && last.text.length + 1 + part.text.length <= PAGE_GATE_TEXT_LIMIT) {+ last.text += ' ' + part.text;+ } else {+ chunks.push({ text: part.text, element: part.element, line: part.line, closed: false });+ }+ }+ return chunks.map(chunk => ({ ...chunk, text: pageGateSlice(chunk.text) }));+ }++ // The wall an overlay or article block shows, or null when none of its+ // texts reads as one (a text nobody has read included). Its texts are its+ // own name and title and its chunks, read in page order up to the first+ // chunk read as a wall, at most PAGE_GATE_BLOCK_CHUNKS of them. In an+ // article the wall starts at that chunk, or at a chunk before it nobody has+ // read, since the wall's text may be there: only text read as no wall comes+ // before it. A block whose name or title reads as a wall starts at its top;+ // an overlay is the wall as a whole. The label is the wall's name or title,+ // or the first line its starting chunk begins with, and none when that+ // chunk was not read as the wall: what follows a wall's first line may be+ // the article it withholds.+ function pageGateWall(el, inline, budget) {+ const named = [...new Set([el.getAttribute('aria-label'), el.getAttribute('title')].map(pageGateSlice))].filter(Boolean);+ const namedReadings = named.map(text => pageGateReading(text, budget));+ const namedIndex = namedReadings.findIndex(reading => PAGE_GATE_TYPES.has(reading));+ const chunks = pageGateChunks(pageGateParts(el)).slice(0, PAGE_GATE_BLOCK_CHUNKS);+ let wallIndex = -1, unread = -1, wallType = null;+ for (let index = 0; index < chunks.length; index += 1) {+ const reading = pageGateReading(chunks[index].text, budget);+ if (PAGE_GATE_TYPES.has(reading)) {+ wallIndex = index;+ wallType = reading;+ break;+ }+ if (reading === null && unread < 0) unread = index;+ }+ if (wallIndex < 0 && namedIndex < 0) return null;+ const type = namedIndex >= 0 ? namedReadings[namedIndex] : wallType;+ const label = text => pageGateSlice(String(text || '').slice(0, 240));+ if (namedIndex >= 0) return { type, label: label(named[namedIndex]), element: el };+ if (!inline) return { type, label: label(chunks[wallIndex].line), element: el };+ const start = unread >= 0 && unread < wallIndex ? unread : wallIndex;+ return {+ type,+ label: start === wallIndex ? label(chunks[wallIndex].line) : '',+ element: start === 0 ? el : chunks[start].element,+ };}- function pageGateReading(text) {- if (!text) return null;- const reading = _pageReading('page_blocker', text, null);- return PAGE_GATE_TYPES.has(reading) ? reading : null;+ // Whether a block offers a control a wall would: a button, a field, a form,+ // or a link standing on its own rather than inside a paragraph.+ function pageGateOffersControl(el) {+ try {+ if (el.querySelector('button, input, select, textarea, form, [role="button"]')) return true;+ return Array.from(el.querySelectorAll('a[href]')).slice(0, 50)+ .some(link => !link.closest('p, li, blockquote, figcaption, td, dd'));+ } catch {+ return false;+ }}function pageHasArticleContext() {@@ -3561,7 +3713,7 @@function detectPageGate() {const seen = new Set();- const candidates = [];+ const found = [];const articleContext = pageHasArticleContext();for (const selector of PAGE_GATE_SELECTORS) {let matches = [];@@ -3587,21 +3739,38 @@// An overlay counts only on an article page, and a block only inside// the article; its text then has to read as a wall.if (surface === 'dialog' ? !articleContext : !inArticle) continue;- const label = pageGateText(el);- const type = pageGateReading(label);- if (!type) continue;- const namedGate = /paywall|gateway|regiwall|subscription|registration/i.test([- el.id,- typeof el.className === 'string' ? el.className : '',- el.getAttribute('data-testid') || '',- ].join(' '));- const score = (surface === 'dialog' ? 100 : 0) + (inArticle ? 40 : 0) +- (coveringOverlay ? 30 : 0) + (namedGate ? 15 : 0) - Math.min(label.length, 2000) / 2000;- candidates.push({ element: el, type, surface, label: label.slice(0, 240), score });+ found.push({ el, surface, inArticle, coveringOverlay, offersControl: pageGateOffersControl(el) });}}- candidates.sort((a, b) => b.score - a.score);- return candidates[0] || null;+ // Overlays are read first, then blocks that offer a control a wall would,+ // then the rest, each in page order, so the texts one page read reports go+ // to the likeliest walls.+ found.sort((a, b) => (Number(b.surface === 'dialog') - Number(a.surface === 'dialog'))+ || (Number(b.offersControl) - Number(a.offersControl))+ || (a.el === b.el ? 0 : (a.el.compareDocumentPosition(b.el) & Node.DOCUMENT_POSITION_FOLLOWING ? -1 : 1)));+ const budget = { left: PAGE_GATE_UNREAD_BUDGET, seen: new Set() };+ const candidates = [];+ for (const { el, surface, inArticle, coveringOverlay } of found) {+ const wall = pageGateWall(el, surface === 'inline', budget);+ if (!wall) continue;+ const label = wall.label;+ const namedGate = /paywall|gateway|regiwall|subscription|registration/i.test([+ el.id,+ typeof el.className === 'string' ? el.className : '',+ el.getAttribute('data-testid') || '',+ ].join(' '));+ const score = (surface === 'dialog' ? 100 : 0) + (inArticle ? 40 : 0) ++ (coveringOverlay ? 30 : 0) + (namedGate ? 15 : 0) - Math.min(label.length, 2000) / 2000;+ candidates.push({ element: wall.element, type: wall.type, surface, label, score });+ }+ // An overlay that is a wall withholds the whole article. Of the walls in+ // an article, the one that starts first decides where reading stops.+ const overlays = candidates.filter(candidate => candidate.surface === 'dialog').sort((a, b) => b.score - a.score);+ if (overlays.length) return overlays[0];+ const blocks = candidates.filter(candidate => candidate.surface === 'inline');+ blocks.sort((a, b) => (a.element === b.element ? 0+ : (a.element.compareDocumentPosition(b.element) & Node.DOCUMENT_POSITION_FOLLOWING ? -1 : 1)));+ return blocks[0] || null;}function renderedArticleTextBeforeGate(root, gateElement) {@@ -6444,9 +6613,11 @@for (const [kind, texts] of Object.entries(readings).slice(0, 8)) {const choices = Object.hasOwn(_PAGE_READING_CHOICES, kind) ? _PAGE_READING_CHOICES[kind] : null;if (!choices || !texts || typeof texts !== 'object' || Array.isArray(texts)) continue;+ // An overlay's or article block's text is read up to 600 characters.+ const limit = kind === 'page_blocker' ? 600 : 400;for (const [value, choice] of Object.entries(texts).slice(0, 800)) {const text = _pageReadingText(value);- if (!text || text.length > 400 || !choices.has(choice)) continue;+ if (!text || text.length > limit || !choices.has(choice)) continue;const key = `${kind}\u0000${text}`;_pageReadings.delete(key);_pageReadings.set(key, choice);@@ -6457,15 +6628,17 @@}// The reading of one text: its choice, '' when there is no text, or null- // when nobody has read it yet (it is then kept for the agent to read). A- // text longer than 400 characters is not read; it reads as `longText`.- function _pageReading(kind, value, longText = null) {+ // when nobody has read it yet (it is then kept for the agent to read,+ // unless `queue` is false). A text longer than `maxLength` characters is+ // not read; it reads as `longText`.+ function _pageReading(kind, value, longText = null, { queue = true, maxLength = 400 } = {}) {const text = _pageReadingText(value);if (!text) return '';- if (text.length > 400) return longText;+ if (text.length > maxLength) return longText;const key = `${kind}\u0000${text}`;const choice = _pageReadings.get(key);if (choice) return choice;+ if (!queue) return null;_pageReadingsUnread.delete(key);_pageReadingsUnread.set(key, { kind, text });while (_pageReadingsUnread.size > 64) _pageReadingsUnread.delete(_pageReadingsUnread.keys().next().value);diff --git a/src/firefox/src/agent/agent.js b/src/firefox/src/agent/agent.jsindex 282b98f69..f57d6286e 100644--- a/src/firefox/src/agent/agent.js+++ b/src/firefox/src/agent/agent.js@@ -208,6 +208,10 @@ const COMPLETION_READING_TIMEOUT_MS = 6_000;// language, after a click in a run that has to prove a submission; the active// model gets this long when the decision model is not sure.const CONTROL_READING_TIMEOUT_MS = 4_000;+// A dialog title or a wall text no model answered reads as nothing to stop+// for; it is asked again only after this long, so page reads and actions do+// not each wait for it.+const PAGE_BLOCKER_RETRY_MS = 30_000;const CONTENT_ACTION_RESPONSE_GRACE_MS = 5_000;const ASK_MODE_HANDOFF_TIMEOUT_MS = 5_000;const CONTENT_ACTION_SIGNAL_DEADLINES = new WeakMap();@@ -7275,20 +7279,53 @@ export class Agent extends LoopDetector {// workflow reached its end state) read with one fixed-choice question each,// in any language, and remembered per text for the synchronous checks that// follow. Longer texts are not read; an unanswered text is asked again next- // time.- async _readPageTexts(tabId, kind, texts, context = {}, { maxChars = 300, mainTimeoutMs, mainModel = true } = {}) {+ // time, or, with `retryUnansweredAfterMs`, once that long has passed. A text+ // read before is not read again: its reading becomes the most recent one,+ // so the readings handed to a check in the page (the most recent of each+ // question) include it however long ago it was read.+ async _readPageTexts(tabId, kind, texts, context = {}, { maxChars = 300, mainTimeoutMs, mainModel = true,+ retryUnansweredAfterMs = 0 } = {}) {this._pageTextReadings ??= new Map();+ this._unansweredPageTexts ??= new Map();+ const now = Date.now();const pending = [...new Set((Array.isArray(texts) ? texts : []).map(text => String(text ?? '').replace(/\s+/g, ' ').trim()).filter(text => text && text.length <= maxChars))]- .filter(text => !this._pageTextReadings.has(this._pageTextKey(kind, text, context)));+ .filter(text => {+ const key = this._pageTextKey(kind, text, context);+ if (this._pageTextReadings.has(key)) {+ const choice = this._pageTextReadings.get(key);+ this._pageTextReadings.delete(key);+ this._pageTextReadings.set(key, choice);+ return false;+ }+ const unansweredAt = this._unansweredPageTexts.get(key);+ return !(retryUnansweredAfterMs > 0 && unansweredAt && now - unansweredAt < retryUnansweredAfterMs);+ });if (!pending.length) return;+ const askMain = mainModel && !this._workflowReplayTabs?.has(tabId);const readings = await this._semanticDecisions(tabId, kind, { items: pending.map(text => ({ ...context, text })) },- { ...(mainTimeoutMs ? { mainTimeoutMs } : {}), mainModel: mainModel && !this._workflowReplayTabs?.has(tabId) });+ { ...(mainTimeoutMs ? { mainTimeoutMs } : {}), mainModel: askMain });+ // A text counts as unanswered only once the active model had its turn+ // too: a read by the decision model alone, a stopped run or a provider+ // that cannot answer leaves it for the next read.+ let mainHadTurn = false;+ try {+ mainHadTurn = askMain && typeof this._activeProvider(tabId)?.chat === 'function' && !this._checkAbort(tabId);+ } catch {}+ const answeredAt = Date.now();pending.forEach((text, index) => {- if (readings[index]?.choice) this._pageTextReadings.set(this._pageTextKey(kind, text, context), readings[index].choice);+ const key = this._pageTextKey(kind, text, context);+ if (readings[index]?.choice) {+ this._pageTextReadings.set(key, readings[index].choice);+ this._unansweredPageTexts.delete(key);+ } else if (mainHadTurn) {+ this._unansweredPageTexts.delete(key);+ this._unansweredPageTexts.set(key, answeredAt);+ }});while (this._pageTextReadings.size > 2048) this._pageTextReadings.delete(this._pageTextReadings.keys().next().value);+ while (this._unansweredPageTexts.size > 512) this._unansweredPageTexts.delete(this._unansweredPageTexts.keys().next().value);}_pageTextReading(kind, text, context = {}) {@@ -10755,20 +10792,31 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d_withPageBlockerReadings(name, params) {if (name !== 'read_page' && name !== 'get_accessibility_tree') return params;return { ...(params && typeof params === 'object' ? params : {}),- __wbPageReadings: { page_blocker: this._challengeTitleReadings() } };+ __wbPageReadings: { page_blocker: this._pageBlockerReadings() } };+ }++ // The page_blocker readings the agent holds, for a page read's page-gate+ // check: overlay and article-block texts as well as dialog titles.+ _pageBlockerReadings() {+ return this._pageTextReadingsByCaption('page_blocker');}// Overlay and article-block texts a page read consulted unread, read in any// language (the decision model first, the active model within four- // seconds). True when any of them was read.+ // seconds). A text the agent read before, but that did not reach the page+ // among the most recent readings, becomes one of them again. A text no+ // model answered reads as no wall, and is not asked again for half a+ // minute, so page reads do not each wait for it. True when any of them has+ // a reading.async _readPageBlockerTexts(tabId, entries) {const texts = [...new Set((Array.isArray(entries) ? entries : []).filter(entry => entry?.kind === 'page_blocker').map(entry => String(entry?.text ?? '').replace(/\s+/g, ' ').trim())- .filter(text => text && text.length <= 400 && !this._pageTextReading('page_blocker', text)))].slice(0, 12);+ .filter(text => text && text.length <= 600))].slice(0, 48);if (!texts.length) return false;try {- await this._readPageTexts(tabId, 'page_blocker', texts, {}, { maxChars: 400, mainTimeoutMs: CONTROL_READING_TIMEOUT_MS });+ await this._readPageTexts(tabId, 'page_blocker', texts, {},+ { maxChars: 600, mainTimeoutMs: CONTROL_READING_TIMEOUT_MS, retryUnansweredAfterMs: PAGE_BLOCKER_RETRY_MS });} catch (error) {if (this._checkAbort(tabId)) throw error;}@@ -10776,29 +10824,43 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d}// A page read whose overlay or article-block texts were consulted unread- // has them read, and the page read once more with the readings; a read- // that cannot be repeated keeps the first answer.+ // has them read, and the page read once more with the readings, up to three+ // times while each read reaches texts nobody has read (a page reports at+ // most forty at a time, the likeliest walls first). A read that cannot be+ // repeated keeps the answer before it, unless one of the texts that answer+ // was made without reads as a wall: that answer may hold what the wall+ // withholds, so the read fails instead.async _rereadWithPageBlockerReadings(tabId, name, response, dispatch) {if ((name !== 'read_page' && name !== 'get_accessibility_tree') || !response || typeof response !== 'object') return response;- const needed = response.pageTextsNeeded;- delete response.pageTextsNeeded;- if (!await this._readPageBlockerTexts(tabId, needed)) return response;- let reread = null;- try {- reread = await dispatch();- } catch (error) {- if (error?.name === 'AbortError' || this._checkAbort(tabId)) throw error;- return response;+ let current = response;+ for (let pass = 0; pass < 3; pass += 1) {+ const needed = current.pageTextsNeeded;+ delete current.pageTextsNeeded;+ if (!await this._readPageBlockerTexts(tabId, needed)) return current;+ const wallRead = (Array.isArray(needed) ? needed : []).some(entry => entry?.kind === 'page_blocker'+ && ['subscription', 'registration', 'login'].includes(this._pageTextReading('page_blocker', entry.text)));+ const failed = () => (wallRead ? { success: false, error: 'This page shows a subscription, registration or '+ + 'sign-in wall, and reading the page again with it failed, so nothing behind the wall was returned. '+ + 'Read the page again.' } : current);+ let reread = null;+ try {+ reread = await dispatch();+ } catch (error) {+ if (error?.name === 'AbortError' || this._checkAbort(tabId)) throw error;+ return failed();+ }+ if (!reread || typeof reread !== 'object' || reread.success === false) return failed();+ current = reread;}- if (!reread || typeof reread !== 'object' || reread.success === false) return response;- delete reread.pageTextsNeeded;- return reread;+ delete current.pageTextsNeeded;+ return current;}// The page_blocker readings of dialog titles the agent holds, by title, for- // the CAPTCHA scans that run in the page.+ // the CAPTCHA scans that run in the page: titles are at most 200+ // characters, so the longer overlay and article-block texts stay behind._challengeTitleReadings() {- return this._pageTextReadingsByCaption('page_blocker');+ return this._pageTextReadingsByCaption('page_blocker', { maxLength: 200 });}_challengeTitleReader() {@@ -10807,15 +10869,18 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d// Reads dialog titles, in any language, as asking to prove the person is// human or not: the decision model first, the active model within four- // seconds unless `mainModel` is false. True when any of them was read.+ // seconds unless `mainModel` is false. A title read before becomes one of+ // the most recent readings again, so a scan in the page receives it; one no+ // model answered stops nothing and is not asked again for half a minute,+ // so page actions do not each wait for it. True when any of them has a+ // reading.async _readChallengeTitles(tabId, titles = [], { mainModel = true } = {}) {const pending = [...new Set((Array.isArray(titles) ? titles : [])- .map(title => challengeTitleText(title)).filter(Boolean))]- .filter(title => !this._pageTextReading('page_blocker', title)).slice(0, 12);+ .map(title => challengeTitleText(title)).filter(Boolean))].slice(0, 12);if (!pending.length) return false;try {await this._readPageTexts(tabId, 'page_blocker', pending, {},- { maxChars: 200, mainTimeoutMs: CONTROL_READING_TIMEOUT_MS, mainModel });+ { maxChars: 200, mainTimeoutMs: CONTROL_READING_TIMEOUT_MS, mainModel, retryUnansweredAfterMs: PAGE_BLOCKER_RETRY_MS });} catch (error) {if (this._checkAbort(tabId)) throw error;}@@ -22421,14 +22486,22 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d}// The readings of one page-text question the agent holds, by caption, for a- // check that runs in the page: the 400 most recent.- _pageTextReadingsByCaption(kind) {+ // check that runs in the page: the 400 most recent, of captions up to+ // `maxLength` characters, within `budget` characters in all. A check that+ // consults a text missing here reports it; reading it again makes it one of+ // the most recent (see _readPageTexts).+ _pageTextReadingsByCaption(kind, { maxLength = Infinity, budget = 80_000 } = {}) {const prefix = this._pageTextKey(kind, '', {});const readings = {};const entries = [...(this._pageTextReadings || new Map()).entries()].filter(([key]) => key.startsWith(prefix));- for (const [key, choice] of entries.slice(-400)) {- const caption = key.slice(prefix.length);- if (caption) readings[caption] = choice;+ let count = 0, used = 0;+ for (let index = entries.length - 1; index >= 0 && count < 400; index -= 1) {+ const caption = entries[index][0].slice(prefix.length);+ if (!caption || caption.length > maxLength) continue;+ if (used + caption.length > budget) break;+ readings[caption] = entries[index][1];+ count += 1;+ used += caption.length;}return readings;}diff --git a/src/firefox/src/agent/captcha-frame-runtime.js b/src/firefox/src/agent/captcha-frame-runtime.jsindex d958de50a..83a8c80eb 100644--- a/src/firefox/src/agent/captcha-frame-runtime.js+++ b/src/firefox/src/agent/captcha-frame-runtime.js@@ -709,7 +709,8 @@ export function detectCaptchaCandidatesInPage(scope = null, matcherOptions = nul// keyed by the title as captcha-gate.js takes it).const givenTitleReadings = matcherOptions?.challengeReadings && typeof matcherOptions.challengeReadings === 'object'? matcherOptions.challengeReadings : {};- const titleText = value => String(value || '').replace(/\s+/g, ' ').trim().slice(0, 200);+ const titleText = value => String(value || '').replace(/\s+/g, ' ').trim().slice(0, 200)+ .replace(/[\ud800-\udbff]$/, '').trim();const challengeDialogs = Array.from(pageDocument.querySelectorAll('dialog, [role="dialog"], [role="alertdialog"]')).filter((element) => {diff --git a/src/firefox/src/agent/captcha-gate.js b/src/firefox/src/agent/captcha-gate.jsindex d01c7c470..76fba7eb7 100644--- a/src/firefox/src/agent/captcha-gate.js+++ b/src/firefox/src/agent/captcha-gate.js@@ -22,9 +22,11 @@ function matchesChallengeLabel(value) {}// A dialog's title as the readings key it: spaces collapsed, at most 200-// characters.+// characters, and neither a space nor half of a surrogate pair at its end+// once cut, so the page and the agent key a long title alike. The scans+// that run in the page and captcha-frame-runtime.js keep the same rule.export function challengeTitleText(value) {- return String(value || '').replace(/\s+/g, ' ').trim().slice(0, 200);+ return String(value || '').replace(/\s+/g, ' ').trim().slice(0, 200).replace(/[\ud800-\udbff]$/, '').trim();}function normalizeChallengeLabel(value) {@@ -156,7 +158,9 @@ export function detectChallengeDialogInPage(options = null) {? options.challengeReadings : {};const titleReading = title => (Object.prototype.hasOwnProperty.call(givenTitleReadings, title)? String(givenTitleReadings[title] || '') : '');- const titleText = value => String(value || '').replace(/\s+/g, ' ').trim().slice(0, 200);+ // Keyed as challengeTitleText keys a title.+ const titleText = value => String(value || '').replace(/\s+/g, ' ').trim().slice(0, 200)+ .replace(/[\ud800-\udbff]$/, '').trim();const unreadTitles = [];const visible = (element) => {try {diff --git a/src/firefox/src/content/content.js b/src/firefox/src/content/content.jsindex 3b0977a54..bddde8c5f 100644--- a/src/firefox/src/content/content.js+++ b/src/firefox/src/content/content.js@@ -44,15 +44,167 @@// text nobody has read yet is no wall; it is reported for the agent to// read, and the agent asks again with the reading.const PAGE_GATE_TYPES = new Set(['subscription', 'registration', 'login']);+ // The longest text one wall reading covers; how many of a block's texts are+ // read at most; and how many texts nobody has read one page read reports at+ // most (the agent reads them and reads the page again, up to three times).+ const PAGE_GATE_TEXT_LIMIT = 600;+ const PAGE_GATE_BLOCK_CHUNKS = 30;+ const PAGE_GATE_UNREAD_BUDGET = 40;++ // At most PAGE_GATE_TEXT_LIMIT characters of a text, keyed as the readings+ // key it: spaces collapsed, and no space or half of a surrogate pair at+ // either end.+ function pageGateSlice(value) {+ return _pageReadingText(_pageReadingText(value).slice(0, PAGE_GATE_TEXT_LIMIT)+ .replace(/^[\udc00-\udfff]/, '').replace(/[\ud800-\udbff]$/, ''));+ }++ // What a text reads as: a wall's kind, 'other' when it was read as no wall+ // (or is empty), or null when nobody has read it. Within one page read only+ // the first PAGE_GATE_UNREAD_BUDGET texts nobody has read are reported to+ // the agent, the likeliest walls first; the rest wait for the next read.+ function pageGateReading(text, budget) {+ if (!text) return 'other';+ const known = _pageReading('page_blocker', text, null, { queue: false, maxLength: PAGE_GATE_TEXT_LIMIT });+ if (known) return PAGE_GATE_TYPES.has(known) ? known : 'other';+ if (!budget.seen.has(text)) {+ if (budget.left <= 0) return null;+ budget.left -= 1;+ budget.seen.add(text);+ }+ _pageReading('page_blocker', text, null, { maxLength: PAGE_GATE_TEXT_LIMIT });+ return null;+ }++ // Whether an element lays its children out as blocks, so its text splits+ // into them; an element of inline text is one part however long it is.+ function pageGateHasBlockChildren(el, depth = 0) {+ for (const child of el.children) {+ let display = '';+ try { display = getComputedStyle(child).display; } catch { continue; }+ if (display === 'contents') {+ if (depth < 20 && pageGateHasBlockChildren(child, depth + 1)) return true;+ continue;+ }+ if (display && display !== 'none' && !display.startsWith('inline')) return true;+ }+ return false;+ }- function pageGateText(el) {- return _pageReadingText(el?.innerText || '').slice(0, 400);+ // A block's rendered text in parts, in page order: an element whose text+ // is at most PAGE_GATE_TEXT_LIMIT characters, or that holds only inline+ // text, is one part; a longer one with block children is split into them,+ // and text standing directly in it is a part held by it. Hidden elements+ // are left out. Each part keeps its first line.+ function pageGateParts(el) {+ const parts = [];+ const firstLine = value => _pageReadingText(String(value || '').split('\n').find(line => line.trim()) || '');+ const visit = (node, depth) => {+ for (const child of node.childNodes) {+ if (parts.length >= 600) return;+ if (child.nodeType === Node.TEXT_NODE) {+ const text = _pageReadingText(child.data);+ if (text) parts.push({ text, line: text, element: node, heading: false });+ continue;+ }+ if (child.nodeType !== Node.ELEMENT_NODE || /^(?:SCRIPT|STYLE|NOSCRIPT|TEMPLATE)$/.test(child.tagName)) continue;+ let display = '';+ try { display = getComputedStyle(child).display; } catch { /* treat as rendered */ }+ if (display === 'none') continue;+ if (display === 'contents') {+ if (depth < 40) visit(child, depth + 1);+ continue;+ }+ if (!child.getClientRects?.().length) continue;+ const raw = String(child.innerText || '');+ const text = _pageReadingText(raw);+ if (!text) continue;+ if (text.length <= PAGE_GATE_TEXT_LIMIT || depth >= 40 || !pageGateHasBlockChildren(child)) {+ parts.push({ text, line: firstLine(raw), element: child,+ heading: child.matches?.('h1, h2, h3, h4, h5, h6, [role="heading"]') === true });+ } else {+ visit(child, depth + 1);+ }+ }+ };+ visit(el, 0);+ return parts;+ }++ // A block's parts gathered, in page order, into texts of at most+ // PAGE_GATE_TEXT_LIMIT characters, a heading starting a new one; a longer+ // part is read in pieces of that length. Each is read as a whole: a wall+ // read in it starts no earlier than the element it begins in, and its label+ // is that element's first line, none for a piece after a long part's first.+ function pageGateChunks(parts) {+ const chunks = [];+ for (const part of parts) {+ if (part.text.length > PAGE_GATE_TEXT_LIMIT) {+ for (let offset = 0; offset < part.text.length && chunks.length < 1000; offset += PAGE_GATE_TEXT_LIMIT) {+ chunks.push({ text: part.text.slice(offset, offset + PAGE_GATE_TEXT_LIMIT), element: part.element,+ line: offset ? '' : part.line, closed: true });+ }+ continue;+ }+ const last = chunks[chunks.length - 1];+ if (last && !last.closed && !part.heading && last.text.length + 1 + part.text.length <= PAGE_GATE_TEXT_LIMIT) {+ last.text += ' ' + part.text;+ } else {+ chunks.push({ text: part.text, element: part.element, line: part.line, closed: false });+ }+ }+ return chunks.map(chunk => ({ ...chunk, text: pageGateSlice(chunk.text) }));+ }++ // The wall an overlay or article block shows, or null when none of its+ // texts reads as one (a text nobody has read included). Its texts are its+ // own name and title and its chunks, read in page order up to the first+ // chunk read as a wall, at most PAGE_GATE_BLOCK_CHUNKS of them. In an+ // article the wall starts at that chunk, or at a chunk before it nobody has+ // read, since the wall's text may be there: only text read as no wall comes+ // before it. A block whose name or title reads as a wall starts at its top;+ // an overlay is the wall as a whole. The label is the wall's name or title,+ // or the first line its starting chunk begins with, and none when that+ // chunk was not read as the wall: what follows a wall's first line may be+ // the article it withholds.+ function pageGateWall(el, inline, budget) {+ const named = [...new Set([el.getAttribute('aria-label'), el.getAttribute('title')].map(pageGateSlice))].filter(Boolean);+ const namedReadings = named.map(text => pageGateReading(text, budget));+ const namedIndex = namedReadings.findIndex(reading => PAGE_GATE_TYPES.has(reading));+ const chunks = pageGateChunks(pageGateParts(el)).slice(0, PAGE_GATE_BLOCK_CHUNKS);+ let wallIndex = -1, unread = -1, wallType = null;+ for (let index = 0; index < chunks.length; index += 1) {+ const reading = pageGateReading(chunks[index].text, budget);+ if (PAGE_GATE_TYPES.has(reading)) {+ wallIndex = index;+ wallType = reading;+ break;+ }+ if (reading === null && unread < 0) unread = index;+ }+ if (wallIndex < 0 && namedIndex < 0) return null;+ const type = namedIndex >= 0 ? namedReadings[namedIndex] : wallType;+ const label = text => pageGateSlice(String(text || '').slice(0, 240));+ if (namedIndex >= 0) return { type, label: label(named[namedIndex]), element: el };+ if (!inline) return { type, label: label(chunks[wallIndex].line), element: el };+ const start = unread >= 0 && unread < wallIndex ? unread : wallIndex;+ return {+ type,+ label: start === wallIndex ? label(chunks[wallIndex].line) : '',+ element: start === 0 ? el : chunks[start].element,+ };}- function pageGateReading(text) {- if (!text) return null;- const reading = _pageReading('page_blocker', text, null);- return PAGE_GATE_TYPES.has(reading) ? reading : null;+ // Whether a block offers a control a wall would: a button, a field, a form,+ // or a link standing on its own rather than inside a paragraph.+ function pageGateOffersControl(el) {+ try {+ if (el.querySelector('button, input, select, textarea, form, [role="button"]')) return true;+ return Array.from(el.querySelectorAll('a[href]')).slice(0, 50)+ .some(link => !link.closest('p, li, blockquote, figcaption, td, dd'));+ } catch {+ return false;+ }}function pageHasArticleContext() {@@ -80,7 +232,7 @@function detectPageGate() {const seen = new Set();- const candidates = [];+ const found = [];const articleContext = pageHasArticleContext();for (const selector of PAGE_GATE_SELECTORS) {let matches = [];@@ -106,21 +258,38 @@// An overlay counts only on an article page, and a block only inside// the article; its text then has to read as a wall.if (surface === 'dialog' ? !articleContext : !inArticle) continue;- const label = pageGateText(el);- const type = pageGateReading(label);- if (!type) continue;- const namedGate = /paywall|gateway|regiwall|subscription|registration/i.test([- el.id,- typeof el.className === 'string' ? el.className : '',- el.getAttribute('data-testid') || '',- ].join(' '));- const score = (surface === 'dialog' ? 100 : 0) + (inArticle ? 40 : 0) +- (coveringOverlay ? 30 : 0) + (namedGate ? 15 : 0) - Math.min(label.length, 2000) / 2000;- candidates.push({ element: el, type, surface, label: label.slice(0, 240), score });+ found.push({ el, surface, inArticle, coveringOverlay, offersControl: pageGateOffersControl(el) });}}- candidates.sort((a, b) => b.score - a.score);- return candidates[0] || null;+ // Overlays are read first, then blocks that offer a control a wall would,+ // then the rest, each in page order, so the texts one page read reports go+ // to the likeliest walls.+ found.sort((a, b) => (Number(b.surface === 'dialog') - Number(a.surface === 'dialog'))+ || (Number(b.offersControl) - Number(a.offersControl))+ || (a.el === b.el ? 0 : (a.el.compareDocumentPosition(b.el) & Node.DOCUMENT_POSITION_FOLLOWING ? -1 : 1)));+ const budget = { left: PAGE_GATE_UNREAD_BUDGET, seen: new Set() };+ const candidates = [];+ for (const { el, surface, inArticle, coveringOverlay } of found) {+ const wall = pageGateWall(el, surface === 'inline', budget);+ if (!wall) continue;+ const label = wall.label;+ const namedGate = /paywall|gateway|regiwall|subscription|registration/i.test([+ el.id,+ typeof el.className === 'string' ? el.className : '',+ el.getAttribute('data-testid') || '',+ ].join(' '));+ const score = (surface === 'dialog' ? 100 : 0) + (inArticle ? 40 : 0) ++ (coveringOverlay ? 30 : 0) + (namedGate ? 15 : 0) - Math.min(label.length, 2000) / 2000;+ candidates.push({ element: wall.element, type: wall.type, surface, label, score });+ }+ // An overlay that is a wall withholds the whole article. Of the walls in+ // an article, the one that starts first decides where reading stops.+ const overlays = candidates.filter(candidate => candidate.surface === 'dialog').sort((a, b) => b.score - a.score);+ if (overlays.length) return overlays[0];+ const blocks = candidates.filter(candidate => candidate.surface === 'inline');+ blocks.sort((a, b) => (a.element === b.element ? 0+ : (a.element.compareDocumentPosition(b.element) & Node.DOCUMENT_POSITION_FOLLOWING ? -1 : 1)));+ return blocks[0] || null;}function renderedArticleTextBeforeGate(root, gateElement) {@@ -5750,9 +5919,11 @@for (const [kind, texts] of Object.entries(readings).slice(0, 8)) {const choices = Object.hasOwn(_PAGE_READING_CHOICES, kind) ? _PAGE_READING_CHOICES[kind] : null;if (!choices || !texts || typeof texts !== 'object' || Array.isArray(texts)) continue;+ // An overlay's or article block's text is read up to 600 characters.+ const limit = kind === 'page_blocker' ? 600 : 400;for (const [value, choice] of Object.entries(texts).slice(0, 800)) {const text = _pageReadingText(value);- if (!text || text.length > 400 || !choices.has(choice)) continue;+ if (!text || text.length > limit || !choices.has(choice)) continue;const key = `${kind}\u0000${text}`;_pageReadings.delete(key);_pageReadings.set(key, choice);@@ -5763,15 +5934,17 @@}// The reading of one text: its choice, '' when there is no text, or null- // when nobody has read it yet (it is then kept for the agent to read). A- // text longer than 400 characters is not read; it reads as `longText`.- function _pageReading(kind, value, longText = null) {+ // when nobody has read it yet (it is then kept for the agent to read,+ // unless `queue` is false). A text longer than `maxLength` characters is+ // not read; it reads as `longText`.+ function _pageReading(kind, value, longText = null, { queue = true, maxLength = 400 } = {}) {const text = _pageReadingText(value);if (!text) return '';- if (text.length > 400) return longText;+ if (text.length > maxLength) return longText;const key = `${kind}\u0000${text}`;const choice = _pageReadings.get(key);if (choice) return choice;+ if (!queue) return null;_pageReadingsUnread.delete(key);_pageReadingsUnread.set(key, { kind, text });while (_pageReadingsUnread.size > 64) _pageReadingsUnread.delete(_pageReadingsUnread.keys().next().value);diff --git a/test/fixtures/dialog-long-intro-wall.html b/test/fixtures/dialog-long-intro-wall.htmlnew file mode 100644index 000000000..a97bd766c--- /dev/null+++ b/test/fixtures/dialog-long-intro-wall.html@@ -0,0 +1,34 @@+<!doctype html>+<html>+<head>+ <meta property="og:type" content="article">+ <title>Long intro dialog wall fixture</title>+ <style>+ body { margin: 0; font: 18px/1.5 sans-serif; }+ article { max-width: 700px; margin: 32px auto; }+ .overlay { position: fixed; inset: 0; z-index: 20; background: #fff; overflow: auto; }+ </style>+</head>+<body>+ <article>+ <h1>Long intro dialog headline</h1>+ <p>SECRET_DIALOG_ARTICLE_BODY is drawn under the overlay and must not be read.</p>+ </article>+ <div class="overlay" role="dialog" aria-modal="true">+ <p>DIALOG_INTRO_01 is a paragraph of about three hundred and thirty characters, written as ordinary reporting about the harbour, the boats that go out before dawn, the market hall where the catch is sold, the council that argues about the breakwater, and the families who have fished here for generations without a day off.</p>+ <p>DIALOG_INTRO_02 is a paragraph of about three hundred and thirty characters, written as ordinary reporting about the harbour, the boats that go out before dawn, the market hall where the catch is sold, the council that argues about the breakwater, and the families who have fished here for generations without a day off.</p>+ <p>DIALOG_INTRO_03 is a paragraph of about three hundred and thirty characters, written as ordinary reporting about the harbour, the boats that go out before dawn, the market hall where the catch is sold, the council that argues about the breakwater, and the families who have fished here for generations without a day off.</p>+ <p>DIALOG_INTRO_04 is a paragraph of about three hundred and thirty characters, written as ordinary reporting about the harbour, the boats that go out before dawn, the market hall where the catch is sold, the council that argues about the breakwater, and the families who have fished here for generations without a day off.</p>+ <p>DIALOG_INTRO_05 is a paragraph of about three hundred and thirty characters, written as ordinary reporting about the harbour, the boats that go out before dawn, the market hall where the catch is sold, the council that argues about the breakwater, and the families who have fished here for generations without a day off.</p>+ <p>DIALOG_INTRO_06 is a paragraph of about three hundred and thirty characters, written as ordinary reporting about the harbour, the boats that go out before dawn, the market hall where the catch is sold, the council that argues about the breakwater, and the families who have fished here for generations without a day off.</p>+ <p>DIALOG_INTRO_07 is a paragraph of about three hundred and thirty characters, written as ordinary reporting about the harbour, the boats that go out before dawn, the market hall where the catch is sold, the council that argues about the breakwater, and the families who have fished here for generations without a day off.</p>+ <h2>Subscribe to continue reading this article</h2>+ <button type="button">Subscribe</button>+ <p>DIALOG_PLANS_01 is a paragraph of about three hundred and thirty characters, written as ordinary reporting about the harbour, the boats that go out before dawn, the market hall where the catch is sold, the council that argues about the breakwater, and the families who have fished here for generations without a day off.</p>+ <p>DIALOG_PLANS_02 is a paragraph of about three hundred and thirty characters, written as ordinary reporting about the harbour, the boats that go out before dawn, the market hall where the catch is sold, the council that argues about the breakwater, and the families who have fished here for generations without a day off.</p>+ <p>DIALOG_PLANS_03 is a paragraph of about three hundred and thirty characters, written as ordinary reporting about the harbour, the boats that go out before dawn, the market hall where the catch is sold, the council that argues about the breakwater, and the families who have fished here for generations without a day off.</p>+ <p>DIALOG_PLANS_04 is a paragraph of about three hundred and thirty characters, written as ordinary reporting about the harbour, the boats that go out before dawn, the market hall where the catch is sold, the council that argues about the breakwater, and the families who have fished here for generations without a day off.</p>+ <p>DIALOG_PLANS_05 is a paragraph of about three hundred and thirty characters, written as ordinary reporting about the harbour, the boats that go out before dawn, the market hall where the catch is sold, the council that argues about the breakwater, and the families who have fished here for generations without a day off.</p>+ </div>+</body>+</html>diff --git a/test/fixtures/inline-bare-prompt-paywall.html b/test/fixtures/inline-bare-prompt-paywall.htmlnew file mode 100644index 000000000..195109985--- /dev/null+++ b/test/fixtures/inline-bare-prompt-paywall.html@@ -0,0 +1,25 @@+<!doctype html>+<html>+<head>+ <meta property="og:type" content="article">+ <title>Bare prompt paywall fixture</title>+ <style>+ body { margin: 0; font: 18px/1.5 sans-serif; }+ article { max-width: 700px; margin: 32px auto; }+ .paywall-wrapper { padding: 24px; border: 1px solid #aaa; }+ </style>+</head>+<body>+ <article>+ <h1>Bare prompt headline</h1>+ <!-- The wall's text stands directly in its block, and comes again after the withheld article. -->+ <div class="paywall-wrapper">+ <p>VISIBLE_BARE_PREVIEW opens the story.</p>+ Subscribe to continue reading this article+ <p>SECRET_BARE_BETWEEN is the withheld article. It runs on about the harbour, the boats, the market hall and the council's vote on the new breakwater, the tide tables, the price of diesel, the families who have fished here for decades, the restaurants that buy the catch each morning, the plans for a new cold store by the quay, the school that teaches net mending on Saturdays, the lifeboat crew and its new engine, the ferry timetable for the winter months, and the dispute over mooring fees that the harbour board has not settled.</p>+ <h3>Subscribe to continue reading this article</h3>+ <p>SECRET_BARE_AFTER must not be returned either.</p>+ </div>+ </article>+</body>+</html>diff --git a/test/fixtures/inline-close-repeat-paywall.html b/test/fixtures/inline-close-repeat-paywall.htmlnew file mode 100644index 000000000..c562b5f8a--- /dev/null+++ b/test/fixtures/inline-close-repeat-paywall.html@@ -0,0 +1,25 @@+<!doctype html>+<html>+<head>+ <meta property="og:type" content="article">+ <title>Close repeat paywall fixture</title>+ <style>+ body { margin: 0; font: 18px/1.5 sans-serif; }+ article { max-width: 700px; margin: 32px auto; }+ .paywall-wrapper { padding: 24px; border: 1px solid #aaa; }+ </style>+</head>+<body>+ <article>+ <h1>Close repeat headline</h1>+ <!-- A line that reads like the wall follows a short withheld sentence. -->+ <div class="paywall-wrapper">+ <p>VISIBLE_CLOSE_PREVIEW opens the story.</p>+ <h3>Subscribe to continue reading this article</h3>+ <p>SECRET_CLOSE_SHORT is a short withheld sentence.</p>+ <p>Already a subscriber? Log in to continue reading.</p>+ <p>SECRET_CLOSE_AFTER must not be returned either.</p>+ </div>+ </article>+</body>+</html>diff --git a/test/fixtures/inline-locked-box-paywall.html b/test/fixtures/inline-locked-box-paywall.htmlnew file mode 100644index 000000000..e591412db--- /dev/null+++ b/test/fixtures/inline-locked-box-paywall.html@@ -0,0 +1,23 @@+<!doctype html>+<html>+<head>+ <meta property="og:type" content="article">+ <title>Locked box paywall fixture</title>+ <style>+ body { margin: 0; font: 18px/1.5 sans-serif; }+ article { max-width: 700px; margin: 32px auto; }+ </style>+</head>+<body>+ <article>+ <h1>Locked box headline</h1>+ <!-- The wall's box holds the locked article right after its heading. -->+ <div class="paywall-container">+ <div class="paywall-box">+ <h2>Subscribe to continue reading</h2>+ <div class="locked-content"><p>SECRET_LOCKED The council voted on the breakwater late last night.</p></div>+ </div>+ </div>+ </article>+</body>+</html>diff --git a/test/fixtures/inline-long-hidden-paywall.html b/test/fixtures/inline-long-hidden-paywall.htmlnew file mode 100644index 000000000..58e167d0e--- /dev/null+++ b/test/fixtures/inline-long-hidden-paywall.html@@ -0,0 +1,45 @@+<!doctype html>+<html>+<head>+ <meta property="og:type" content="article">+ <title>Long hidden paywall fixture</title>+ <style>+ body { margin: 0; font: 18px/1.5 sans-serif; }+ article { max-width: 700px; margin: 32px auto; }+ .paywall-wrapper { padding: 24px; border: 1px solid #aaa; }+ </style>+</head>+<body>+ <article>+ <h1>Long hidden headline</h1>+ <!-- Twelve short preview paragraphs, the wall, then twelve withheld ones still rendered. -->+ <div class="paywall-wrapper">+ <p>PREVIEW_LINE_01 is a short sentence of the free preview.</p>+ <p>PREVIEW_LINE_02 is a short sentence of the free preview.</p>+ <p>PREVIEW_LINE_03 is a short sentence of the free preview.</p>+ <p>PREVIEW_LINE_04 is a short sentence of the free preview.</p>+ <p>PREVIEW_LINE_05 is a short sentence of the free preview.</p>+ <p>PREVIEW_LINE_06 is a short sentence of the free preview.</p>+ <p>PREVIEW_LINE_07 is a short sentence of the free preview.</p>+ <p>PREVIEW_LINE_08 is a short sentence of the free preview.</p>+ <p>PREVIEW_LINE_09 is a short sentence of the free preview.</p>+ <p>PREVIEW_LINE_10 is a short sentence of the free preview.</p>+ <p>PREVIEW_LINE_11 is a short sentence of the free preview.</p>+ <p>PREVIEW_LINE_12 is a short sentence of the free preview.</p>+ <h3>Subscribe to continue reading this article</h3>+ <p>HIDDEN_LINE_01 is a short sentence of the withheld article.</p>+ <p>HIDDEN_LINE_02 is a short sentence of the withheld article.</p>+ <p>HIDDEN_LINE_03 is a short sentence of the withheld article.</p>+ <p>HIDDEN_LINE_04 is a short sentence of the withheld article.</p>+ <p>HIDDEN_LINE_05 is a short sentence of the withheld article.</p>+ <p>HIDDEN_LINE_06 is a short sentence of the withheld article.</p>+ <p>HIDDEN_LINE_07 is a short sentence of the withheld article.</p>+ <p>HIDDEN_LINE_08 is a short sentence of the withheld article.</p>+ <p>HIDDEN_LINE_09 is a short sentence of the withheld article.</p>+ <p>HIDDEN_LINE_10 is a short sentence of the withheld article.</p>+ <p>HIDDEN_LINE_11 is a short sentence of the withheld article.</p>+ <p>HIDDEN_LINE_12 is a short sentence of the withheld article.</p>+ </div>+ </article>+</body>+</html>diff --git a/test/fixtures/inline-long-paragraph-paywall.html b/test/fixtures/inline-long-paragraph-paywall.htmlnew file mode 100644index 000000000..591f198bf--- /dev/null+++ b/test/fixtures/inline-long-paragraph-paywall.html@@ -0,0 +1,24 @@+<!doctype html>+<html>+<head>+ <meta property="og:type" content="article">+ <title>Long paragraph paywall fixture</title>+ <style>+ body { margin: 0; font: 18px/1.5 sans-serif; }+ article { max-width: 700px; margin: 32px auto; }+ .paywall-wrapper { padding: 24px; border: 1px solid #aaa; }+ </style>+</head>+<body>+ <article>+ <h1>Long paragraph headline</h1>+ <!-- Two long preview paragraphs, then a wall paragraph longer than a short line. -->+ <div class="paywall-wrapper">+ <p>LONG_PARAGRAPH_ONE opens the story with a long paragraph about the harbour at dawn, the boats leaving one by one past the lighthouse, the gulls following them out, the smell of diesel and salt, the market hall lights coming on, the first buyers arriving with their vans, and the radio in the harbour office reading the shipping forecast for the day ahead.</p>+ <p>LONG_PARAGRAPH_TWO goes on about the council meeting where the new breakwater was debated late into the evening, the engineers' drawings pinned to the wall, the fishermen standing at the back, the mayor asking for patience, and the vote that was put off again until the funding from the region has been confirmed in writing.</p>+ <p>This article is for subscribers only. To continue reading this article and every other story on the site, subscribe today, or start a free trial and cancel at any time; if you already have a subscription, sign in with the account you subscribed with.</p>+ <p>SECRET_PARAGRAPH_REST must not be returned by read_page.</p>+ </div>+ </article>+</body>+</html>diff --git a/test/fixtures/inline-long-preview-paywall.html b/test/fixtures/inline-long-preview-paywall.htmlnew file mode 100644index 000000000..eb43400b5--- /dev/null+++ b/test/fixtures/inline-long-preview-paywall.html@@ -0,0 +1,28 @@+<!doctype html>+<html>+<head>+ <meta property="og:type" content="article">+ <title>Long preview paywall fixture</title>+ <style>+ body { margin: 0; font: 18px/1.5 sans-serif; }+ article { max-width: 700px; margin: 32px auto; }+ .paywall-wrapper { padding: 24px; border: 1px solid #aaa; }+ </style>+</head>+<body>+ <article>+ <h1>Long preview headline</h1>+ <!-- The wall's text follows more than 400 characters of preview. -->+ <div class="paywall-wrapper">+ <p>LONG_PREVIEW_ONE opens the story with a sentence of ordinary reporting about the harbour and its boats.</p>+ <p>LONG_PREVIEW_TWO adds the morning's weather, the tide tables and the names of the boats that went out.</p>+ <p>LONG_PREVIEW_THREE quotes a fisherman about the price of diesel and the cost of keeping a crew paid.</p>+ <p>LONG_PREVIEW_FOUR describes the market hall where the catch is sold before the restaurants open.</p>+ <p>LONG_PREVIEW_FIVE notes that the council will vote on the new breakwater at its meeting next month.</p>+ <p>LONG_PREVIEW_SIX ends the preview with a line about the families who have fished here for decades.</p>+ <h3>Subscribe to continue reading this article</h3>+ <p>SECRET_LONG_REST_PARAGRAPH must not be returned by read_page.</p>+ </div>+ </article>+</body>+</html>diff --git a/test/fixtures/inline-many-blocks-paywall.html b/test/fixtures/inline-many-blocks-paywall.htmlnew file mode 100644index 000000000..e82381277--- /dev/null+++ b/test/fixtures/inline-many-blocks-paywall.html@@ -0,0 +1,90 @@+<!doctype html>+<html>+<head>+ <meta property="og:type" content="article">+ <title>Many blocks paywall fixture</title>+ <style>+ body { margin: 0; font: 18px/1.5 sans-serif; }+ article { max-width: 700px; margin: 32px auto; }+ .paywall-wrapper { padding: 24px; border: 1px solid #aaa; }+ </style>+</head>+<body>+ <main>+ <article>+ <h1>Many blocks headline</h1>+ <div class="story paywall-content">+ <p>STORY_LINE_01 is a short line of the free story.</p>+ <p>STORY_LINE_02 is a short line of the free story.</p>+ <p>STORY_LINE_03 is a short line of the free story.</p>+ <p>STORY_LINE_04 is a short line of the free story.</p>+ <p>STORY_LINE_05 is a short line of the free story.</p>+ <p>STORY_LINE_06 is a short line of the free story.</p>+ <p>STORY_LINE_07 is a short line of the free story.</p>+ <p>STORY_LINE_08 is a short line of the free story.</p>+ <p>STORY_LINE_09 is a short line of the free story.</p>+ <p>STORY_LINE_10 is a short line of the free story.</p>+ <p>STORY_LINE_11 is a short line of the free story.</p>+ <p>STORY_LINE_12 is a short line of the free story.</p>+ <p>STORY_LINE_13 is a short line of the free story.</p>+ <p>STORY_LINE_14 is a short line of the free story.</p>+ <p>STORY_LINE_15 is a short line of the free story.</p>+ <p>STORY_LINE_16 is a short line of the free story.</p>+ <p>STORY_LINE_17 is a short line of the free story.</p>+ <p>STORY_LINE_18 is a short line of the free story.</p>+ <p>STORY_LINE_19 is a short line of the free story.</p>+ <p>STORY_LINE_20 is a short line of the free story.</p>+ </div>+ <div class="subscription-wall">+ <h3>Subscribe to continue reading this article</h3>+ <button type="button">Subscribe</button>+ </div>+ <p>SECRET_MANY_REST must not be returned by read_page.</p>+ </article>+ <section class="paywall-teasers">+ <p>Teaser A-01: another story from this week.</p>+ <p>Teaser A-02: another story from this week.</p>+ <p>Teaser A-03: another story from this week.</p>+ <p>Teaser A-04: another story from this week.</p>+ <p>Teaser A-05: another story from this week.</p>+ <p>Teaser A-06: another story from this week.</p>+ <p>Teaser A-07: another story from this week.</p>+ <p>Teaser A-08: another story from this week.</p>+ <p>Teaser A-09: another story from this week.</p>+ <p>Teaser A-10: another story from this week.</p>+ <p>Teaser A-11: another story from this week.</p>+ <p>Teaser A-12: another story from this week.</p>+ <p>Teaser A-13: another story from this week.</p>+ <p>Teaser A-14: another story from this week.</p>+ <p>Teaser A-15: another story from this week.</p>+ <p>Teaser A-16: another story from this week.</p>+ <p>Teaser A-17: another story from this week.</p>+ <p>Teaser A-18: another story from this week.</p>+ <p>Teaser A-19: another story from this week.</p>+ <p>Teaser A-20: another story from this week.</p>+ </section>+ <section class="paywall-teasers">+ <p>Teaser B-01: another story from this week.</p>+ <p>Teaser B-02: another story from this week.</p>+ <p>Teaser B-03: another story from this week.</p>+ <p>Teaser B-04: another story from this week.</p>+ <p>Teaser B-05: another story from this week.</p>+ <p>Teaser B-06: another story from this week.</p>+ <p>Teaser B-07: another story from this week.</p>+ <p>Teaser B-08: another story from this week.</p>+ <p>Teaser B-09: another story from this week.</p>+ <p>Teaser B-10: another story from this week.</p>+ <p>Teaser B-11: another story from this week.</p>+ <p>Teaser B-12: another story from this week.</p>+ <p>Teaser B-13: another story from this week.</p>+ <p>Teaser B-14: another story from this week.</p>+ <p>Teaser B-15: another story from this week.</p>+ <p>Teaser B-16: another story from this week.</p>+ <p>Teaser B-17: another story from this week.</p>+ <p>Teaser B-18: another story from this week.</p>+ <p>Teaser B-19: another story from this week.</p>+ <p>Teaser B-20: another story from this week.</p>+ </section>+ </main>+</body>+</html>diff --git a/test/fixtures/inline-middle-wall-paywall.html b/test/fixtures/inline-middle-wall-paywall.htmlnew file mode 100644index 000000000..8493bd1d6--- /dev/null+++ b/test/fixtures/inline-middle-wall-paywall.html@@ -0,0 +1,33 @@+<!doctype html>+<html>+<head>+ <meta property="og:type" content="article">+ <title>Middle wall paywall fixture</title>+ <style>+ body { margin: 0; font: 18px/1.5 sans-serif; }+ article { max-width: 700px; margin: 32px auto; }+ </style>+</head>+<body>+ <article>+ <h1>Middle wall headline</h1>+ <!-- The article body is named for its paywall; the wall sits between seven+ preview paragraphs and five withheld ones that are still rendered. -->+ <div class="article-body paywall-content">+ <p>MIDDLE_PREVIEW_01 is a paragraph of about three hundred and thirty characters, written as ordinary reporting about the harbour, the boats that go out before dawn, the market hall where the catch is sold, the council that argues about the breakwater, and the families who have fished here for generations without a day off.</p>+ <p>MIDDLE_PREVIEW_02 is a paragraph of about three hundred and thirty characters, written as ordinary reporting about the harbour, the boats that go out before dawn, the market hall where the catch is sold, the council that argues about the breakwater, and the families who have fished here for generations without a day off.</p>+ <p>MIDDLE_PREVIEW_03 is a paragraph of about three hundred and thirty characters, written as ordinary reporting about the harbour, the boats that go out before dawn, the market hall where the catch is sold, the council that argues about the breakwater, and the families who have fished here for generations without a day off.</p>+ <p>MIDDLE_PREVIEW_04 is a paragraph of about three hundred and thirty characters, written as ordinary reporting about the harbour, the boats that go out before dawn, the market hall where the catch is sold, the council that argues about the breakwater, and the families who have fished here for generations without a day off.</p>+ <p>MIDDLE_PREVIEW_05 is a paragraph of about three hundred and thirty characters, written as ordinary reporting about the harbour, the boats that go out before dawn, the market hall where the catch is sold, the council that argues about the breakwater, and the families who have fished here for generations without a day off.</p>+ <p>MIDDLE_PREVIEW_06 is a paragraph of about three hundred and thirty characters, written as ordinary reporting about the harbour, the boats that go out before dawn, the market hall where the catch is sold, the council that argues about the breakwater, and the families who have fished here for generations without a day off.</p>+ <p>MIDDLE_PREVIEW_07 is a paragraph of about three hundred and thirty characters, written as ordinary reporting about the harbour, the boats that go out before dawn, the market hall where the catch is sold, the council that argues about the breakwater, and the families who have fished here for generations without a day off.</p>+ <div class="cta"><h3>Subscribe to continue reading this article</h3><button type="button">Subscribe</button></div>+ <p>SECRET_MIDDLE_REST_01 is a paragraph of about three hundred and thirty characters, written as ordinary reporting about the harbour, the boats that go out before dawn, the market hall where the catch is sold, the council that argues about the breakwater, and the families who have fished here for generations without a day off.</p>+ <p>SECRET_MIDDLE_REST_02 is a paragraph of about three hundred and thirty characters, written as ordinary reporting about the harbour, the boats that go out before dawn, the market hall where the catch is sold, the council that argues about the breakwater, and the families who have fished here for generations without a day off.</p>+ <p>SECRET_MIDDLE_REST_03 is a paragraph of about three hundred and thirty characters, written as ordinary reporting about the harbour, the boats that go out before dawn, the market hall where the catch is sold, the council that argues about the breakwater, and the families who have fished here for generations without a day off.</p>+ <p>SECRET_MIDDLE_REST_04 is a paragraph of about three hundred and thirty characters, written as ordinary reporting about the harbour, the boats that go out before dawn, the market hall where the catch is sold, the council that argues about the breakwater, and the families who have fished here for generations without a day off.</p>+ <p>SECRET_MIDDLE_REST_05 is a paragraph of about three hundred and thirty characters, written as ordinary reporting about the harbour, the boats that go out before dawn, the market hall where the catch is sold, the council that argues about the breakwater, and the families who have fished here for generations without a day off.</p>+ </div>+ </article>+</body>+</html>diff --git a/test/fixtures/inline-repeated-prompt-paywall.html b/test/fixtures/inline-repeated-prompt-paywall.htmlnew file mode 100644index 000000000..f435b457f--- /dev/null+++ b/test/fixtures/inline-repeated-prompt-paywall.html@@ -0,0 +1,25 @@+<!doctype html>+<html>+<head>+ <meta property="og:type" content="article">+ <title>Repeated prompt paywall fixture</title>+ <style>+ body { margin: 0; font: 18px/1.5 sans-serif; }+ article { max-width: 700px; margin: 32px auto; }+ .paywall-wrapper { padding: 24px; border: 1px solid #aaa; }+ </style>+</head>+<body>+ <article>+ <h1>Repeated prompt headline</h1>+ <!-- The wall's text comes twice, with the withheld article between. -->+ <div class="paywall-wrapper">+ <p>VISIBLE_REPEAT_PREVIEW is the readable opening of the story.</p>+ <h3>Subscribe to continue reading this article</h3>+ <p>SECRET_BETWEEN_PROMPTS is the withheld article. It runs on about the harbour, the boats, the market hall and the council's vote on the new breakwater, and it goes on long enough that no window starting at the first prompt reaches past it: the tide tables, the price of diesel, the families who have fished here for decades, the restaurants that buy the catch each morning, and the plans for a new cold store by the quay that the council will also discuss next month.</p>+ <h3>Subscribe to continue reading this article</h3>+ <p>SECRET_AFTER_SECOND_PROMPT must not be returned either.</p>+ </div>+ </article>+</body>+</html>diff --git a/test/fixtures/inline-span-words-paywall.html b/test/fixtures/inline-span-words-paywall.htmlnew file mode 100644index 000000000..bdf80cb50--- /dev/null+++ b/test/fixtures/inline-span-words-paywall.html@@ -0,0 +1,27 @@+<!doctype html>+<html>+<head>+ <meta property="og:type" content="article">+ <title>Span words paywall fixture</title>+ <style>+ body { margin: 0; font: 18px/1.5 sans-serif; }+ article { max-width: 700px; margin: 32px auto; }+ </style>+</head>+<body>+ <article>+ <h1>Span words headline</h1>+ <!-- Six long preview paragraphs with every word in its own span, then the wall. -->+ <div class="paywall-wrapper">+ <p><span>SPAN_PREVIEW_1_word1</span> <span>SPAN_PREVIEW_1_word2</span> <span>SPAN_PREVIEW_1_word3</span> <span>SPAN_PREVIEW_1_word4</span> <span>SPAN_PREVIEW_1_word5</span> <span>SPAN_PREVIEW_1_word6</span> <span>SPAN_PREVIEW_1_word7</span> <span>SPAN_PREVIEW_1_word8</span> <span>SPAN_PREVIEW_1_word9</span> <span>SPAN_PREVIEW_1_word10</span> <span>SPAN_PREVIEW_1_word11</span> <span>SPAN_PREVIEW_1_word12</span> <span>SPAN_PREVIEW_1_word13</span> <span>SPAN_PREVIEW_1_word14</span> <span>SPAN_PREVIEW_1_word15</span> <span>SPAN_PREVIEW_1_word16</span> <span>SPAN_PREVIEW_1_word17</span> <span>SPAN_PREVIEW_1_word18</span> <span>SPAN_PREVIEW_1_word19</span> <span>SPAN_PREVIEW_1_word20</span> <span>SPAN_PREVIEW_1_word21</span> <span>SPAN_PREVIEW_1_word22</span> <span>SPAN_PREVIEW_1_word23</span> <span>SPAN_PREVIEW_1_word24</span> <span>SPAN_PREVIEW_1_word25</span> <span>SPAN_PREVIEW_1_word26</span> <span>SPAN_PREVIEW_1_word27</span> <span>SPAN_PREVIEW_1_word28</span> <span>SPAN_PREVIEW_1_word29</span> <span>SPAN_PREVIEW_1_word30</span> <span>SPAN_PREVIEW_1_word31</span> <span>SPAN_PREVIEW_1_word32</span> <span>SPAN_PREVIEW_1_word33</span> <span>SPAN_PREVIEW_1_word34</span> <span>SPAN_PREVIEW_1_word35</span> <span>SPAN_PREVIEW_1_word36</span> <span>SPAN_PREVIEW_1_word37</span> <span>SPAN_PREVIEW_1_word38</span> <span>SPAN_PREVIEW_1_word39</span> <span>SPAN_PREVIEW_1_word40</span> <span>SPAN_PREVIEW_1_word41</span> <span>SPAN_PREVIEW_1_word42</span> <span>SPAN_PREVIEW_1_word43</span> <span>SPAN_PREVIEW_1_word44</span> <span>SPAN_PREVIEW_1_word45</span></p>+ <p><span>SPAN_PREVIEW_2_word1</span> <span>SPAN_PREVIEW_2_word2</span> <span>SPAN_PREVIEW_2_word3</span> <span>SPAN_PREVIEW_2_word4</span> <span>SPAN_PREVIEW_2_word5</span> <span>SPAN_PREVIEW_2_word6</span> <span>SPAN_PREVIEW_2_word7</span> <span>SPAN_PREVIEW_2_word8</span> <span>SPAN_PREVIEW_2_word9</span> <span>SPAN_PREVIEW_2_word10</span> <span>SPAN_PREVIEW_2_word11</span> <span>SPAN_PREVIEW_2_word12</span> <span>SPAN_PREVIEW_2_word13</span> <span>SPAN_PREVIEW_2_word14</span> <span>SPAN_PREVIEW_2_word15</span> <span>SPAN_PREVIEW_2_word16</span> <span>SPAN_PREVIEW_2_word17</span> <span>SPAN_PREVIEW_2_word18</span> <span>SPAN_PREVIEW_2_word19</span> <span>SPAN_PREVIEW_2_word20</span> <span>SPAN_PREVIEW_2_word21</span> <span>SPAN_PREVIEW_2_word22</span> <span>SPAN_PREVIEW_2_word23</span> <span>SPAN_PREVIEW_2_word24</span> <span>SPAN_PREVIEW_2_word25</span> <span>SPAN_PREVIEW_2_word26</span> <span>SPAN_PREVIEW_2_word27</span> <span>SPAN_PREVIEW_2_word28</span> <span>SPAN_PREVIEW_2_word29</span> <span>SPAN_PREVIEW_2_word30</span> <span>SPAN_PREVIEW_2_word31</span> <span>SPAN_PREVIEW_2_word32</span> <span>SPAN_PREVIEW_2_word33</span> <span>SPAN_PREVIEW_2_word34</span> <span>SPAN_PREVIEW_2_word35</span> <span>SPAN_PREVIEW_2_word36</span> <span>SPAN_PREVIEW_2_word37</span> <span>SPAN_PREVIEW_2_word38</span> <span>SPAN_PREVIEW_2_word39</span> <span>SPAN_PREVIEW_2_word40</span> <span>SPAN_PREVIEW_2_word41</span> <span>SPAN_PREVIEW_2_word42</span> <span>SPAN_PREVIEW_2_word43</span> <span>SPAN_PREVIEW_2_word44</span> <span>SPAN_PREVIEW_2_word45</span></p>+ <p><span>SPAN_PREVIEW_3_word1</span> <span>SPAN_PREVIEW_3_word2</span> <span>SPAN_PREVIEW_3_word3</span> <span>SPAN_PREVIEW_3_word4</span> <span>SPAN_PREVIEW_3_word5</span> <span>SPAN_PREVIEW_3_word6</span> <span>SPAN_PREVIEW_3_word7</span> <span>SPAN_PREVIEW_3_word8</span> <span>SPAN_PREVIEW_3_word9</span> <span>SPAN_PREVIEW_3_word10</span> <span>SPAN_PREVIEW_3_word11</span> <span>SPAN_PREVIEW_3_word12</span> <span>SPAN_PREVIEW_3_word13</span> <span>SPAN_PREVIEW_3_word14</span> <span>SPAN_PREVIEW_3_word15</span> <span>SPAN_PREVIEW_3_word16</span> <span>SPAN_PREVIEW_3_word17</span> <span>SPAN_PREVIEW_3_word18</span> <span>SPAN_PREVIEW_3_word19</span> <span>SPAN_PREVIEW_3_word20</span> <span>SPAN_PREVIEW_3_word21</span> <span>SPAN_PREVIEW_3_word22</span> <span>SPAN_PREVIEW_3_word23</span> <span>SPAN_PREVIEW_3_word24</span> <span>SPAN_PREVIEW_3_word25</span> <span>SPAN_PREVIEW_3_word26</span> <span>SPAN_PREVIEW_3_word27</span> <span>SPAN_PREVIEW_3_word28</span> <span>SPAN_PREVIEW_3_word29</span> <span>SPAN_PREVIEW_3_word30</span> <span>SPAN_PREVIEW_3_word31</span> <span>SPAN_PREVIEW_3_word32</span> <span>SPAN_PREVIEW_3_word33</span> <span>SPAN_PREVIEW_3_word34</span> <span>SPAN_PREVIEW_3_word35</span> <span>SPAN_PREVIEW_3_word36</span> <span>SPAN_PREVIEW_3_word37</span> <span>SPAN_PREVIEW_3_word38</span> <span>SPAN_PREVIEW_3_word39</span> <span>SPAN_PREVIEW_3_word40</span> <span>SPAN_PREVIEW_3_word41</span> <span>SPAN_PREVIEW_3_word42</span> <span>SPAN_PREVIEW_3_word43</span> <span>SPAN_PREVIEW_3_word44</span> <span>SPAN_PREVIEW_3_word45</span></p>+ <p><span>SPAN_PREVIEW_4_word1</span> <span>SPAN_PREVIEW_4_word2</span> <span>SPAN_PREVIEW_4_word3</span> <span>SPAN_PREVIEW_4_word4</span> <span>SPAN_PREVIEW_4_word5</span> <span>SPAN_PREVIEW_4_word6</span> <span>SPAN_PREVIEW_4_word7</span> <span>SPAN_PREVIEW_4_word8</span> <span>SPAN_PREVIEW_4_word9</span> <span>SPAN_PREVIEW_4_word10</span> <span>SPAN_PREVIEW_4_word11</span> <span>SPAN_PREVIEW_4_word12</span> <span>SPAN_PREVIEW_4_word13</span> <span>SPAN_PREVIEW_4_word14</span> <span>SPAN_PREVIEW_4_word15</span> <span>SPAN_PREVIEW_4_word16</span> <span>SPAN_PREVIEW_4_word17</span> <span>SPAN_PREVIEW_4_word18</span> <span>SPAN_PREVIEW_4_word19</span> <span>SPAN_PREVIEW_4_word20</span> <span>SPAN_PREVIEW_4_word21</span> <span>SPAN_PREVIEW_4_word22</span> <span>SPAN_PREVIEW_4_word23</span> <span>SPAN_PREVIEW_4_word24</span> <span>SPAN_PREVIEW_4_word25</span> <span>SPAN_PREVIEW_4_word26</span> <span>SPAN_PREVIEW_4_word27</span> <span>SPAN_PREVIEW_4_word28</span> <span>SPAN_PREVIEW_4_word29</span> <span>SPAN_PREVIEW_4_word30</span> <span>SPAN_PREVIEW_4_word31</span> <span>SPAN_PREVIEW_4_word32</span> <span>SPAN_PREVIEW_4_word33</span> <span>SPAN_PREVIEW_4_word34</span> <span>SPAN_PREVIEW_4_word35</span> <span>SPAN_PREVIEW_4_word36</span> <span>SPAN_PREVIEW_4_word37</span> <span>SPAN_PREVIEW_4_word38</span> <span>SPAN_PREVIEW_4_word39</span> <span>SPAN_PREVIEW_4_word40</span> <span>SPAN_PREVIEW_4_word41</span> <span>SPAN_PREVIEW_4_word42</span> <span>SPAN_PREVIEW_4_word43</span> <span>SPAN_PREVIEW_4_word44</span> <span>SPAN_PREVIEW_4_word45</span></p>+ <p><span>SPAN_PREVIEW_5_word1</span> <span>SPAN_PREVIEW_5_word2</span> <span>SPAN_PREVIEW_5_word3</span> <span>SPAN_PREVIEW_5_word4</span> <span>SPAN_PREVIEW_5_word5</span> <span>SPAN_PREVIEW_5_word6</span> <span>SPAN_PREVIEW_5_word7</span> <span>SPAN_PREVIEW_5_word8</span> <span>SPAN_PREVIEW_5_word9</span> <span>SPAN_PREVIEW_5_word10</span> <span>SPAN_PREVIEW_5_word11</span> <span>SPAN_PREVIEW_5_word12</span> <span>SPAN_PREVIEW_5_word13</span> <span>SPAN_PREVIEW_5_word14</span> <span>SPAN_PREVIEW_5_word15</span> <span>SPAN_PREVIEW_5_word16</span> <span>SPAN_PREVIEW_5_word17</span> <span>SPAN_PREVIEW_5_word18</span> <span>SPAN_PREVIEW_5_word19</span> <span>SPAN_PREVIEW_5_word20</span> <span>SPAN_PREVIEW_5_word21</span> <span>SPAN_PREVIEW_5_word22</span> <span>SPAN_PREVIEW_5_word23</span> <span>SPAN_PREVIEW_5_word24</span> <span>SPAN_PREVIEW_5_word25</span> <span>SPAN_PREVIEW_5_word26</span> <span>SPAN_PREVIEW_5_word27</span> <span>SPAN_PREVIEW_5_word28</span> <span>SPAN_PREVIEW_5_word29</span> <span>SPAN_PREVIEW_5_word30</span> <span>SPAN_PREVIEW_5_word31</span> <span>SPAN_PREVIEW_5_word32</span> <span>SPAN_PREVIEW_5_word33</span> <span>SPAN_PREVIEW_5_word34</span> <span>SPAN_PREVIEW_5_word35</span> <span>SPAN_PREVIEW_5_word36</span> <span>SPAN_PREVIEW_5_word37</span> <span>SPAN_PREVIEW_5_word38</span> <span>SPAN_PREVIEW_5_word39</span> <span>SPAN_PREVIEW_5_word40</span> <span>SPAN_PREVIEW_5_word41</span> <span>SPAN_PREVIEW_5_word42</span> <span>SPAN_PREVIEW_5_word43</span> <span>SPAN_PREVIEW_5_word44</span> <span>SPAN_PREVIEW_5_word45</span></p>+ <p><span>SPAN_PREVIEW_6_word1</span> <span>SPAN_PREVIEW_6_word2</span> <span>SPAN_PREVIEW_6_word3</span> <span>SPAN_PREVIEW_6_word4</span> <span>SPAN_PREVIEW_6_word5</span> <span>SPAN_PREVIEW_6_word6</span> <span>SPAN_PREVIEW_6_word7</span> <span>SPAN_PREVIEW_6_word8</span> <span>SPAN_PREVIEW_6_word9</span> <span>SPAN_PREVIEW_6_word10</span> <span>SPAN_PREVIEW_6_word11</span> <span>SPAN_PREVIEW_6_word12</span> <span>SPAN_PREVIEW_6_word13</span> <span>SPAN_PREVIEW_6_word14</span> <span>SPAN_PREVIEW_6_word15</span> <span>SPAN_PREVIEW_6_word16</span> <span>SPAN_PREVIEW_6_word17</span> <span>SPAN_PREVIEW_6_word18</span> <span>SPAN_PREVIEW_6_word19</span> <span>SPAN_PREVIEW_6_word20</span> <span>SPAN_PREVIEW_6_word21</span> <span>SPAN_PREVIEW_6_word22</span> <span>SPAN_PREVIEW_6_word23</span> <span>SPAN_PREVIEW_6_word24</span> <span>SPAN_PREVIEW_6_word25</span> <span>SPAN_PREVIEW_6_word26</span> <span>SPAN_PREVIEW_6_word27</span> <span>SPAN_PREVIEW_6_word28</span> <span>SPAN_PREVIEW_6_word29</span> <span>SPAN_PREVIEW_6_word30</span> <span>SPAN_PREVIEW_6_word31</span> <span>SPAN_PREVIEW_6_word32</span> <span>SPAN_PREVIEW_6_word33</span> <span>SPAN_PREVIEW_6_word34</span> <span>SPAN_PREVIEW_6_word35</span> <span>SPAN_PREVIEW_6_word36</span> <span>SPAN_PREVIEW_6_word37</span> <span>SPAN_PREVIEW_6_word38</span> <span>SPAN_PREVIEW_6_word39</span> <span>SPAN_PREVIEW_6_word40</span> <span>SPAN_PREVIEW_6_word41</span> <span>SPAN_PREVIEW_6_word42</span> <span>SPAN_PREVIEW_6_word43</span> <span>SPAN_PREVIEW_6_word44</span> <span>SPAN_PREVIEW_6_word45</span></p>+ <h3>Subscribe to continue reading this article</h3>+ <p>SECRET_SPAN_REST must not be returned by read_page.</p>+ </div>+ </article>+</body>+</html>diff --git a/test/fixtures/inline-wrapped-preview-paywall.html b/test/fixtures/inline-wrapped-preview-paywall.htmlnew file mode 100644index 000000000..7bd137429--- /dev/null+++ b/test/fixtures/inline-wrapped-preview-paywall.html@@ -0,0 +1,23 @@+<!doctype html>+<html>+<head>+ <meta property="og:type" content="article">+ <title>Wrapped preview paywall fixture</title>+ <style>+ body { margin: 0; font: 18px/1.5 sans-serif; }+ article { max-width: 700px; margin: 32px auto; }+ .paywall-wrapper { padding: 24px; border: 1px solid #aaa; }+ </style>+</head>+<body>+ <article>+ <h1>Wrapped preview headline</h1>+ <!-- The wall's own block begins with the readable preview. -->+ <div class="paywall-wrapper">+ <p>WRAPPED_PREVIEW_PARAGRAPH is the readable preview the wall's own block begins with.</p>+ <h3>Subscribe to continue reading this article</h3>+ <p>SECRET_WRAPPED_REST_PARAGRAPH must not be returned by read_page.</p>+ </div>+ </article>+</body>+</html>diff --git a/test/fixtures/overlay-among-many-blocks.html b/test/fixtures/overlay-among-many-blocks.htmlnew file mode 100644index 000000000..449c52089--- /dev/null+++ b/test/fixtures/overlay-among-many-blocks.html@@ -0,0 +1,222 @@+<!doctype html>+<html>+<head>+ <meta property="og:type" content="article">+ <title>Overlay among many blocks fixture</title>+ <style>+ body { margin: 0; font: 18px/1.5 sans-serif; }+ article { max-width: 700px; margin: 32px auto; }+ .overlay { position: fixed; inset: 0; z-index: 20; background: #fff; }+ </style>+</head>+<body>+ <article>+ <h1>Many notes headline</h1>+ <p>SECRET_NOTES_ARTICLE_BODY is drawn under the overlay and must not be read.</p>+ <p class="subscription-note">Note 001: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 002: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 003: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 004: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 005: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 006: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 007: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 008: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 009: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 010: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 011: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 012: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 013: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 014: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 015: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 016: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 017: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 018: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 019: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 020: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 021: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 022: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 023: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 024: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 025: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 026: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 027: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 028: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 029: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 030: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 031: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 032: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 033: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 034: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 035: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 036: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 037: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 038: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 039: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 040: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 041: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 042: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 043: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 044: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 045: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 046: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 047: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 048: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 049: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 050: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 051: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 052: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 053: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 054: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 055: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 056: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 057: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 058: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 059: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 060: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 061: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 062: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 063: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 064: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 065: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 066: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 067: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 068: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 069: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 070: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 071: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 072: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 073: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 074: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 075: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 076: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 077: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 078: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 079: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 080: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 081: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 082: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 083: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 084: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 085: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 086: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 087: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 088: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 089: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 090: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 091: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 092: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 093: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 094: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 095: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 096: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 097: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 098: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 099: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 100: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 101: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 102: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 103: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 104: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 105: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 106: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 107: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 108: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 109: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 110: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 111: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 112: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 113: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 114: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 115: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 116: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 117: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 118: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 119: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 120: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 121: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 122: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 123: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 124: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 125: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 126: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 127: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 128: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 129: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 130: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 131: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 132: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 133: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 134: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 135: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 136: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 137: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 138: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 139: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 140: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 141: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 142: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 143: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 144: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 145: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 146: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 147: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 148: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 149: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 150: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 151: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 152: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 153: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 154: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 155: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 156: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 157: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 158: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 159: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 160: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 161: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 162: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 163: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 164: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 165: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 166: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 167: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 168: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 169: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 170: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 171: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 172: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 173: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 174: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 175: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 176: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 177: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 178: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 179: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 180: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 181: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 182: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 183: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 184: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 185: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 186: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 187: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 188: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 189: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 190: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 191: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 192: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 193: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 194: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 195: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 196: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 197: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 198: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 199: this story is part of our weekly series.</p>+ <p class="subscription-note">Note 200: this story is part of our weekly series.</p>+ </article>+ <div class="overlay" role="dialog" aria-modal="true">+ <h2>Subscribe to continue reading this article</h2>+ <button type="button">Subscribe</button>+ </div>+</body>+</html>diff --git a/test/fixtures/paywall-class-article-no-wall.html b/test/fixtures/paywall-class-article-no-wall.htmlnew file mode 100644index 000000000..92c8fd32c--- /dev/null+++ b/test/fixtures/paywall-class-article-no-wall.html@@ -0,0 +1,52 @@+<!doctype html>+<html>+<head>+ <meta property="og:type" content="article">+ <title>Paywall class article fixture</title>+ <style>+ body { margin: 0; font: 18px/1.5 sans-serif; }+ article { max-width: 700px; margin: 32px auto; }+ .paywall-wrapper { padding: 24px; border: 1px solid #aaa; }+ </style>+</head>+<body>+ <article>+ <h1>Readable paywall class headline</h1>+ <!-- A signed-in article whose body wrapper is named for the paywall it can show. -->+ <div class="article-body paywall-content">+ <p>READABLE_BODY_01 is a full paragraph of the signed-in article, long enough to fill a line or two with ordinary reporting about the town.</p>+ <p>READABLE_BODY_02 is a full paragraph of the signed-in article, long enough to fill a line or two with ordinary reporting about the town.</p>+ <p>READABLE_BODY_03 is a full paragraph of the signed-in article, long enough to fill a line or two with ordinary reporting about the town.</p>+ <p>READABLE_BODY_04 is a full paragraph of the signed-in article, long enough to fill a line or two with ordinary reporting about the town.</p>+ <p>READABLE_BODY_05 is a full paragraph of the signed-in article, long enough to fill a line or two with ordinary reporting about the town.</p>+ <p>READABLE_BODY_06 is a full paragraph of the signed-in article, long enough to fill a line or two with ordinary reporting about the town.</p>+ <p>READABLE_BODY_07 is a full paragraph of the signed-in article, long enough to fill a line or two with ordinary reporting about the town.</p>+ <p>READABLE_BODY_08 is a full paragraph of the signed-in article, long enough to fill a line or two with ordinary reporting about the town.</p>+ <p>READABLE_BODY_09 is a full paragraph of the signed-in article, long enough to fill a line or two with ordinary reporting about the town.</p>+ <p>READABLE_BODY_10 is a full paragraph of the signed-in article, long enough to fill a line or two with ordinary reporting about the town.</p>+ <p>READABLE_BODY_11 is a full paragraph of the signed-in article, long enough to fill a line or two with ordinary reporting about the town.</p>+ <p>READABLE_BODY_12 is a full paragraph of the signed-in article, long enough to fill a line or two with ordinary reporting about the town.</p>+ <p>READABLE_BODY_13 is a full paragraph of the signed-in article, long enough to fill a line or two with ordinary reporting about the town.</p>+ <p>READABLE_BODY_14 is a full paragraph of the signed-in article, long enough to fill a line or two with ordinary reporting about the town.</p>+ <p>READABLE_BODY_15 is a full paragraph of the signed-in article, long enough to fill a line or two with ordinary reporting about the town.</p>+ <p>READABLE_BODY_16 is a full paragraph of the signed-in article, long enough to fill a line or two with ordinary reporting about the town.</p>+ <p>READABLE_BODY_17 is a full paragraph of the signed-in article, long enough to fill a line or two with ordinary reporting about the town.</p>+ <p>READABLE_BODY_18 is a full paragraph of the signed-in article, long enough to fill a line or two with ordinary reporting about the town.</p>+ <p>READABLE_BODY_19 is a full paragraph of the signed-in article, long enough to fill a line or two with ordinary reporting about the town.</p>+ <p>READABLE_BODY_20 is a full paragraph of the signed-in article, long enough to fill a line or two with ordinary reporting about the town.</p>+ <p>READABLE_BODY_21 is a full paragraph of the signed-in article, long enough to fill a line or two with ordinary reporting about the town.</p>+ <p>READABLE_BODY_22 is a full paragraph of the signed-in article, long enough to fill a line or two with ordinary reporting about the town.</p>+ <p>READABLE_BODY_23 is a full paragraph of the signed-in article, long enough to fill a line or two with ordinary reporting about the town.</p>+ <p>READABLE_BODY_24 is a full paragraph of the signed-in article, long enough to fill a line or two with ordinary reporting about the town.</p>+ <p>READABLE_BODY_25 is a full paragraph of the signed-in article, long enough to fill a line or two with ordinary reporting about the town.</p>+ <p>READABLE_BODY_26 is a full paragraph of the signed-in article, long enough to fill a line or two with ordinary reporting about the town.</p>+ <p>READABLE_BODY_27 is a full paragraph of the signed-in article, long enough to fill a line or two with ordinary reporting about the town.</p>+ <p>READABLE_BODY_28 is a full paragraph of the signed-in article, long enough to fill a line or two with ordinary reporting about the town.</p>+ <p>READABLE_BODY_29 is a full paragraph of the signed-in article, long enough to fill a line or two with ordinary reporting about the town.</p>+ <p>READABLE_BODY_30 is a full paragraph of the signed-in article, long enough to fill a line or two with ordinary reporting about the town.</p>+ </div>+ <aside class="subscription-newsletter"><p>Subscribe to our newsletter for the week's best stories.</p></aside>+ <p>READABLE_AFTER_NEWSLETTER closes the article.</p>+ </article>+</body>+</html>diff --git a/test/fixtures/run.mjs b/test/fixtures/run.mjsindex dc7161a28..22ba7435c 100644--- a/test/fixtures/run.mjs+++ b/test/fixtures/run.mjs@@ -299,37 +299,43 @@ async function call(page, action, params) {};}-async function readThroughCdpMirror(page, opts = {}) {+// A text as the agent keys a reading of it (its _pageTextKey), so a reading+// handed back reaches the page only when the page keyed the text alike.+const agentReadingKey = text => String(text ?? '').replace(/\s+/g, ' ').trim();++// The way the agent reads the overlay and article-block texts a page read+// consulted unread (a model; the fixture oracle stands in): at most 48 of+// them, then the page once more, up to three times while texts get read.+// `stats.needed` records how many texts each read reported.+async function readGateTextsLikeTheAgent(readOnce, readings, stats = null) {+ let result = await readOnce();+ for (let pass = 0; pass < 3; pass += 1) {+ const needed = (result?.pageTextsNeeded || []).filter(entry => entry?.kind === 'page_blocker');+ stats?.needed?.push(needed.length);+ const texts = [...new Set(needed.map(entry => agentReadingKey(entry.text)))].filter(Boolean).slice(0, 48);+ if (!texts.length) break;+ for (const text of texts) readings[text] = fixturePageBlockerReading(text);+ result = await readOnce();+ }+ if (result && typeof result === 'object') delete result.pageTextsNeeded;+ return result;+}++async function readThroughCdpMirror(page, opts = {}, stats = null) {const client = new CDPClient();client.evaluate = async (_tabId, expression) => ({result: { value: await page.evaluate(expression) },});- // The agent reads the overlay and article-block texts the mirror consulted- // unread (a model; the fixture oracle stands in), then reads again.const readings = { ...(opts.pageBlockerReadings || {}) };- let result = await client.readPage(1, { ...opts, pageBlockerReadings: readings });- const needed = (result?.pageTextsNeeded || []).filter(entry => entry?.kind === 'page_blocker');- if (needed.length) {- for (const { text } of needed) readings[text] = fixturePageBlockerReading(text);- result = await client.readPage(1, { ...opts, pageBlockerReadings: readings });- }- if (result && typeof result === 'object') delete result.pageTextsNeeded;- return result;+ return readGateTextsLikeTheAgent(() => client.readPage(1, { ...opts, pageBlockerReadings: readings }), readings, stats);}// A page read the way the agent reads it: overlay and article-block texts the// page-gate check consulted unread are read by a model (the fixture oracle// stands in) and the page is read again with those readings.-async function readWithGateReadings(page, action, params = {}) {+async function readWithGateReadings(page, action, params = {}, stats = null) {const readings = {};- let result = await call(page, action, { ...params, __wbPageReadings: { page_blocker: readings } });- const needed = (result?.pageTextsNeeded || []).filter(entry => entry?.kind === 'page_blocker');- if (needed.length) {- for (const { text } of needed) readings[text] = fixturePageBlockerReading(text);- result = await call(page, action, { ...params, __wbPageReadings: { page_blocker: readings } });- }- if (result && typeof result === 'object') delete result.pageTextsNeeded;- return result;+ return readGateTextsLikeTheAgent(() => call(page, action, { ...params, __wbPageReadings: { page_blocker: readings } }), readings, stats);}async function clickedSentinel(page) {@@ -1098,6 +1104,108 @@ for (const [label, browserKind] of [['Chrome', 'chrome'], ['Firefox', 'firefox']}});+ test(`${label}: a wall inside the block its preview begins leaves that preview readable`, async (page) => {+ await setupContentFixture(page, 'inline-wrapped-preview-paywall.html', browserKind);+ const result = await readWithGateReadings(page, 'get_page_info_cdp', {});+ if (result.pageGate?.type !== 'subscription' || result.pageGate?.surface !== 'inline') {+ throw new Error(`wrapped preview pageGate mismatch: ${JSON.stringify(result.pageGate)}`);+ }+ if (!/WRAPPED_PREVIEW_PARAGRAPH/.test(result.text || '') || /SECRET_WRAPPED_REST_PARAGRAPH/.test(result.text || '')) {+ throw new Error(`wrapped preview boundary mismatch: ${JSON.stringify(result.text)}`);+ }+ if (!/Subscribe to continue reading/.test(result.pageGate.label || '')) {+ throw new Error(`wrapped preview label does not start at the wall: ${JSON.stringify(result.pageGate)}`);+ }+ const tree = await readWithGateReadings(page, 'get_accessibility_tree', { filter: 'visible', maxDepth: 5 });+ if (!/WRAPPED_PREVIEW_PARAGRAPH/.test(tree.pageContent || '') || /SECRET_WRAPPED_REST_PARAGRAPH/.test(tree.pageContent || '')) {+ throw new Error(`wrapped preview accessibility boundary mismatch: ${JSON.stringify(tree)}`);+ }+ });++ test(`${label}: a wall after more than 400 characters of preview still withholds the rest`, async (page) => {+ await setupContentFixture(page, 'inline-long-preview-paywall.html', browserKind);+ const result = await readWithGateReadings(page, 'get_page_info_cdp', {});+ if (result.pageGate?.type !== 'subscription' || result.pageGate?.surface !== 'inline') {+ throw new Error(`long preview pageGate mismatch: ${JSON.stringify(result.pageGate)}`);+ }+ if (/SECRET_LONG_REST_PARAGRAPH/.test(JSON.stringify(result))) {+ throw new Error(`long preview leaked the rest of the article: ${JSON.stringify(result.text)}`);+ }+ for (const marker of ['ONE', 'TWO', 'THREE', 'FOUR', 'FIVE', 'SIX']) {+ if (!new RegExp(`LONG_PREVIEW_${marker}`).test(result.text || '')) {+ throw new Error(`long preview lost LONG_PREVIEW_${marker}: ${JSON.stringify(result.text)}`);+ }+ }+ const tree = await readWithGateReadings(page, 'get_accessibility_tree', { filter: 'visible', maxDepth: 5 });+ if (tree.pageGate?.type !== 'subscription' || /SECRET_LONG_REST_PARAGRAPH/.test(tree.pageContent || '')+ || !/LONG_PREVIEW_SIX/.test(tree.pageContent || '')) {+ throw new Error(`long preview accessibility read mismatch: ${JSON.stringify(tree)}`);+ }+ });++ for (const [fixture, secret, visible, what] of [+ ['inline-bare-prompt-paywall.html', /SECRET_BARE_(?:BETWEEN|AFTER)/, null, 'a wall text standing directly in its block, repeated after the withheld article'],+ ['inline-close-repeat-paywall.html', /SECRET_CLOSE_(?:SHORT|AFTER)/, /VISIBLE_CLOSE_PREVIEW/, 'a line reading like the wall a short sentence after it'],+ ['inline-long-hidden-paywall.html', /HIDDEN_LINE_\d\d/, /PREVIEW_LINE_12/, 'a wall after twelve preview lines with twelve withheld lines rendered after it'],+ ['inline-long-paragraph-paywall.html', /SECRET_PARAGRAPH_REST/, /LONG_PARAGRAPH_ONE/, 'a long wall paragraph after long preview paragraphs'],+ ['inline-many-blocks-paywall.html', /SECRET_MANY_REST/, /STORY_LINE_20/, 'a wall among several blocks named like paywalls'],+ ['inline-middle-wall-paywall.html', /SECRET_MIDDLE_REST/, /MIDDLE_PREVIEW_06/, 'a wall between a long preview and a long withheld article in one block'],+ ['inline-span-words-paywall.html', /SECRET_SPAN_REST/, /SPAN_PREVIEW_5_word1\b/, 'a wall after long preview paragraphs of inline words'],+ ['inline-locked-box-paywall.html', /SECRET_LOCKED/, null, 'a wall box holding the locked article right after its heading'],+ ]) {+ test(`${label}: ${what} withholds the article after it`, async (page) => {+ await setupContentFixture(page, fixture, browserKind);+ for (const [action, field] of [['get_page_info_cdp', 'text'], ['get_accessibility_tree', 'pageContent']]) {+ const result = await readWithGateReadings(page, action, action === 'get_accessibility_tree' ? { filter: 'visible', maxDepth: 6 } : {});+ if (!['subscription', 'registration', 'login'].includes(result.pageGate?.type) || result.pageGate?.surface !== 'inline') {+ throw new Error(`${action}: wall missed: ${JSON.stringify(result.pageGate)}`);+ }+ if (secret.test(JSON.stringify(result))) {+ throw new Error(`${action}: the withheld article was returned: ${JSON.stringify({ pageGate: result.pageGate, text: result[field] })}`);+ }+ if (visible && !visible.test(result[field] || '')) {+ throw new Error(`${action}: the preview before the wall was withheld: ${JSON.stringify(result[field])}`);+ }+ }+ });+ }++ test(`${label}: an overlay wall in the middle of a long text, or among two hundred other blocks, is still read`, async (page) => {+ for (const [fixture, secret] of [['dialog-long-intro-wall.html', /SECRET_DIALOG_ARTICLE_BODY/],+ ['overlay-among-many-blocks.html', /SECRET_NOTES_ARTICLE_BODY/]]) {+ await setupContentFixture(page, fixture, browserKind);+ for (const action of ['get_page_info_cdp', 'get_accessibility_tree']) {+ const result = await readWithGateReadings(page, action, action === 'get_accessibility_tree' ? { filter: 'visible', maxDepth: 6 } : {});+ if (result.pageGate?.type !== 'subscription' || result.pageGate?.surface !== 'dialog' || secret.test(JSON.stringify(result))) {+ throw new Error(`${fixture} ${action}: overlay wall mismatch: ${JSON.stringify({ pageGate: result.pageGate, textSource: result.textSource })}`);+ }+ }+ }+ });++ test(`${label}: an article whose body is named for a paywall reads whole, at a bounded cost`, async (page) => {+ await setupContentFixture(page, 'paywall-class-article-no-wall.html', browserKind);+ const stats = { needed: [] };+ const result = await readWithGateReadings(page, 'get_page_info_cdp', {}, stats);+ if (result.pageGate || !/READABLE_BODY_01/.test(result.text || '') || !/READABLE_BODY_30/.test(result.text || '')) {+ throw new Error(`readable paywall-named article mismatch: ${JSON.stringify({ pageGate: result.pageGate, text: String(result.text || '').slice(-300) })}`);+ }+ if (!(stats.needed[0] > 0 && stats.needed[0] <= 12)) {+ throw new Error(`a readable article asked for ${JSON.stringify(stats.needed)} wall readings; at most 12 expected`);+ }+ });++ test(`${label}: a wall's text repeated after the withheld article does not open what lies between`, async (page) => {+ await setupContentFixture(page, 'inline-repeated-prompt-paywall.html', browserKind);+ for (const [action, field] of [['get_page_info_cdp', 'text'], ['get_accessibility_tree', 'pageContent']]) {+ const result = await readWithGateReadings(page, action, action === 'get_accessibility_tree' ? { filter: 'visible', maxDepth: 5 } : {});+ if (result.pageGate?.type !== 'subscription' || !/VISIBLE_REPEAT_PREVIEW/.test(result[field] || '')+ || /SECRET_BETWEEN_PROMPTS|SECRET_AFTER_SECOND_PROMPT/.test(JSON.stringify(result))) {+ throw new Error(`${action}: repeated prompt boundary mismatch: ${JSON.stringify({ pageGate: result.pageGate, text: result[field] })}`);+ }+ }+ });+test(`${label}: readable article ignores header controls, inline upsells, and hidden gate markup`, async (page) => {await setupContentFixture(page, 'readable-article-no-gate.html', browserKind);const result = await readWithGateReadings(page, 'get_page_info_cdp', {});@@ -1279,6 +1387,72 @@ test('Chrome CDP mirror suppresses a blocking Athletic article body', async (pag}});+test('Chrome CDP mirror keeps a wrapped preview readable and withholds the rest after a long preview', async (page) => {+ await page.goto(fixtureUrl('inline-wrapped-preview-paywall.html'));+ const wrapped = await readThroughCdpMirror(page);+ if (wrapped.pageGate?.type !== 'subscription' || wrapped.pageGate?.surface !== 'inline'+ || !/WRAPPED_PREVIEW_PARAGRAPH/.test(wrapped.text || '') || /SECRET_WRAPPED_REST_PARAGRAPH/.test(JSON.stringify(wrapped))) {+ throw new Error(`CDP wrapped preview mismatch: ${JSON.stringify({ pageGate: wrapped.pageGate, text: wrapped.text })}`);+ }+ await page.goto(fixtureUrl('inline-long-preview-paywall.html'));+ const long = await readThroughCdpMirror(page);+ if (long.pageGate?.type !== 'subscription' || !/LONG_PREVIEW_SIX/.test(long.text || '')+ || /SECRET_LONG_REST_PARAGRAPH/.test(JSON.stringify(long))) {+ throw new Error(`CDP long preview mismatch: ${JSON.stringify({ pageGate: long.pageGate, text: long.text })}`);+ }+});++test('Chrome CDP mirror does not open the article between a wall text and its repeat', async (page) => {+ await page.goto(fixtureUrl('inline-repeated-prompt-paywall.html'));+ const result = await readThroughCdpMirror(page);+ if (result.pageGate?.type !== 'subscription' || !/VISIBLE_REPEAT_PREVIEW/.test(result.text || '')+ || /SECRET_BETWEEN_PROMPTS|SECRET_AFTER_SECOND_PROMPT/.test(JSON.stringify(result))) {+ throw new Error(`CDP repeated prompt mismatch: ${JSON.stringify({ pageGate: result.pageGate, text: result.text })}`);+ }+});++test('Chrome CDP mirror withholds the article after walls the content script finds', async (page) => {+ for (const [fixture, secret] of [+ ['inline-bare-prompt-paywall.html', /SECRET_BARE_(?:BETWEEN|AFTER)/],+ ['inline-close-repeat-paywall.html', /SECRET_CLOSE_(?:SHORT|AFTER)/],+ ['inline-long-hidden-paywall.html', /HIDDEN_LINE_\d\d/],+ ['inline-long-paragraph-paywall.html', /SECRET_PARAGRAPH_REST/],+ ['inline-many-blocks-paywall.html', /SECRET_MANY_REST/],+ ['inline-middle-wall-paywall.html', /SECRET_MIDDLE_REST/],+ ['inline-span-words-paywall.html', /SECRET_SPAN_REST/],+ ['inline-locked-box-paywall.html', /SECRET_LOCKED/],+ ['dialog-long-intro-wall.html', /SECRET_DIALOG_ARTICLE_BODY/],+ ['overlay-among-many-blocks.html', /SECRET_NOTES_ARTICLE_BODY/],+ ]) {+ await page.goto(fixtureUrl(fixture));+ const result = await readThroughCdpMirror(page);+ if (!['subscription', 'registration', 'login'].includes(result.pageGate?.type) || secret.test(JSON.stringify(result))) {+ throw new Error(`CDP ${fixture} mismatch: ${JSON.stringify({ pageGate: result.pageGate, text: result.text })}`);+ }+ }+ await page.goto(fixtureUrl('paywall-class-article-no-wall.html'));+ const stats = { needed: [] };+ const readable = await readThroughCdpMirror(page, {}, stats);+ if (readable.pageGate || !/READABLE_BODY_30/.test(readable.text || '') || !(stats.needed[0] > 0 && stats.needed[0] <= 12)) {+ throw new Error(`CDP readable paywall-named article mismatch: ${JSON.stringify({ pageGate: readable.pageGate, needed: stats.needed })}`);+ }+});++test('Chrome CDP mirror keys a wall text cut at its length limit as the agent does', async (page) => {+ // The overlay's text has a space at its 400th and at its 600th character,+ // where a page read cuts it; the agent keys its reading without that space.+ const opening = 'Okumaya devam etmek için abone olun. ';+ const text = opening + 'x'.repeat(399 - opening.length) + ' ' + 'z'.repeat(199) + ' ' + 'CUT_TAIL ' + 'y'.repeat(80);+ if (text[399] !== ' ' || text[599] !== ' ') throw new Error('the fixture text must have spaces at its 400th and 600th characters');+ await page.setContent(`<!doctype html><meta property="og:type" content="article">+ <article><h1>Kesik duvar</h1><p>SECRET_CUT_ARTICLE_BODY sunucuda çizilmiştir.</p></article>+ <div role="dialog" aria-modal="true" style="position:fixed;inset:0;background:#fff"><p>${text}</p></div>`);+ const result = await readThroughCdpMirror(page);+ if (result.pageGate?.type !== 'subscription' || result.textSource !== 'page-gate' || /SECRET_CUT_ARTICLE_BODY/.test(JSON.stringify(result))) {+ throw new Error(`CDP cut wall text mismatch: ${JSON.stringify({ pageGate: result.pageGate, textSource: result.textSource })}`);+ }+});+test('Chrome CDP mirror preserves a readable article across an inline upsell', async (page) => {await page.goto(fixtureUrl('readable-article-no-gate.html'));const result = await readThroughCdpMirror(page);diff --git a/test/run.js b/test/run.jsindex 2a371aa46..867263750 100644--- a/test/run.js+++ b/test/run.js@@ -18416,6 +18416,19 @@ test('a human-verification dialog in any language arms the CAPTCHA gate once a massert.equal((await none._detectChallengeDialogBeforeMutation(1, { includeStatus: true })).challenge, null,`${build}: an unread title stopped a mutation`);});+ // A title longer than 200 characters whose 200th character is a space is+ // keyed alike in the page and in the agent, so its reading reaches the scan.+ const longTitle = `Güvenlik doğrulaması ${'b'.repeat(178)} devam etmek için robot olmadığınızı doğrulayın`;+ assert.equal(longTitle[199], ' ');+ const longNodes = [captchaEl('div', { role: 'dialog', 'aria-label': longTitle }, [+ captchaEl('button', { textContent: 'Kapat', innerText: 'Kapat' }),+ ])];+ await withCaptchaFakePage(build, longNodes, async () => {+ const preflight = new AgentClass({});+ stubCaptchaTitleReadings(preflight);+ const found = await preflight._detectChallengeDialogBeforeMutation(1, { includeStatus: true });+ assert.equal(found.challenge?.label, longTitle.slice(0, 199), `${build}: a long title read as a human check stopped nothing`);+ });}});@@ -18436,17 +18449,158 @@ test('a page read has its unread wall texts read in any language and reads the passert.deepEqual(asked.map(item => [item.kind, item.text]), [['page_blocker', wall]],`${build}: only the wall text was to be read, and once`);assert.deepEqual(agent._withPageBlockerReadings('read_page', {}).__wbPageReadings.page_blocker, { [wall]: 'subscription' });- // A text already read, or one nobody can read, reads nothing more and the- // first answer stands.+ // A text the agent already read that a page still reports did not reach+ // it: it is not asked again, but the page is read once more with it.const again = await agent._rereadWithPageBlockerReadings(1, 'get_accessibility_tree',- { pageContent: 'tree', pageTextsNeeded: [{ kind: 'page_blocker', text: wall }] }, async () => { dispatches += 1; return {}; });- assert.equal(dispatches, 1);- assert.deepEqual(again, { pageContent: 'tree' });+ { pageContent: 'tree', pageTextsNeeded: [{ kind: 'page_blocker', text: wall }] },+ async () => { dispatches += 1; return { pageContent: 'gate', pageGate: { type: 'subscription' } }; });+ assert.equal(dispatches, 2, `${build}: a held reading the page lacked was not handed over`);+ assert.deepEqual(again, { pageContent: 'gate', pageGate: { type: 'subscription' } });+ assert.equal(asked.length, 1, `${build}: a held reading was asked again`);+ // One nobody can read changes nothing, and the first answer stands.const unread = new AgentClass({});stubPageTextReadings(unread, () => null);const first = await unread._rereadWithPageBlockerReadings(1, 'read_page',{ text: 'SECRET_ARTICLE', pageTextsNeeded: [{ kind: 'page_blocker', text: wall }] }, async () => { throw new Error('read again'); });assert.deepEqual(first, { text: 'SECRET_ARTICLE' }, `${build}: an unread wall changed the read`);+ // When a text reads as a wall but the page cannot be read again, the+ // answer made without that reading is not returned: it may hold what the+ // wall withholds.+ const failing = new AgentClass({});+ stubCaptchaTitleReadings(failing);+ for (const dispatch of [async () => { throw new Error('read again'); }, async () => ({ success: false, error: 'gone' })]) {+ const refused = await failing._rereadWithPageBlockerReadings(1, 'read_page',+ { text: 'SECRET_ARTICLE', pageTextsNeeded: [{ kind: 'page_blocker', text: wall }] }, dispatch);+ assert.equal(refused.success, false, `${build}: a read made without its wall reading was returned`);+ assert.doesNotMatch(JSON.stringify(refused), /SECRET_ARTICLE|\[|\]/);+ }+ // A text read as no wall keeps the first answer when the page cannot be+ // read again.+ const other = await failing._rereadWithPageBlockerReadings(1, 'read_page',+ { text: 'ARTICLE', pageTextsNeeded: [{ kind: 'page_blocker', text: 'Bültenimize abone olun' }] },+ async () => { throw new Error('read again'); });+ assert.deepEqual(other, { text: 'ARTICLE' });+ }+});++test('a reading older than the ones handed to the page still reaches it when the page asks for it', async () => {+ for (const [build, AgentClass] of [['chrome', AgentCh], ['firefox', AgentFx]]) {+ const wall = 'Bu içerik abonelere özeldir Okumaya devam etmek için abone olun. Abone ol';+ const agent = new AgentClass({});+ const asked = stubPageTextReadings(agent, (kind, item) => (kind === 'page_blocker'+ ? (item.text === wall ? 'subscription' : 'other') : kind === 'message_control' ? 'other' : null));+ await agent._readPageTexts(1, 'page_blocker', [wall], {}, { maxChars: 400 });+ // Many texts read since push it out of the readings a page read carries.+ await agent._readPageTexts(1, 'page_blocker', Array.from({ length: 450 }, (_, index) => `Bülten kutusu ${index}`), {}, { maxChars: 400 });+ assert.equal(agent._withPageBlockerReadings('read_page', {}).__wbPageReadings.page_blocker[wall], undefined,+ `${build}: the fixture did not push the old reading out`);+ asked.length = 0;+ let handed = null;+ const reread = await agent._rereadWithPageBlockerReadings(1, 'read_page',+ { text: 'SECRET_ARTICLE', pageTextsNeeded: [{ kind: 'page_blocker', text: wall }] },+ async () => {+ handed = agent._withPageBlockerReadings('read_page', {}).__wbPageReadings.page_blocker;+ return { text: wall, textSource: 'page-gate', pageGate: { type: 'subscription', blocking: true, surface: 'dialog', label: wall } };+ });+ assert.equal(handed?.[wall], 'subscription', `${build}: the page was read again without the reading it lacked`);+ assert.equal(reread.pageGate?.type, 'subscription');+ assert.deepEqual(asked, [], `${build}: a held reading was asked of a model again`);+ // The same holds for any question a check in the page reads, such as a+ // messaging page's controls: a held reading becomes one of the most recent.+ await agent._readPageTexts(1, 'message_control', ['Gönder'], {}, { maxChars: 400 });+ await agent._readPageTexts(1, 'message_control', Array.from({ length: 450 }, (_, index) => `Düğme ${index}`), {}, { maxChars: 400 });+ assert.equal(agent._messagingPageReadings().message_control['Gönder'], undefined);+ assert.equal(await agent._readUnreadPageTexts(1, [{ kind: 'message_control', text: 'Gönder' }]), true);+ assert.equal(agent._messagingPageReadings().message_control['Gönder'], 'other', `${build}: a held control reading did not reach the page`);+ // Overlay and article-block texts stay out of the dialog-title readings+ // the CAPTCHA scans carry, which are at most 200 characters.+ const window = `${'Önizleme paragrafı. '.repeat(12)}Okumaya devam etmek için abone olun.`;+ await agent._readPageTexts(1, 'page_blocker', [window], {}, { maxChars: 400 });+ assert.equal(agent._pageBlockerReadings()[window], 'other');+ assert.equal(agent._challengeTitleReadings()[window], undefined, `${build}: a long wall text was handed to the CAPTCHA scans`);+ }+});++test('a page read reaching more unread wall texts after a reread is read again, at most three times', async () => {+ for (const [build, AgentClass] of [['chrome', AgentCh], ['firefox', AgentFx]]) {+ const agent = new AgentClass({});+ const asked = stubPageTextReadings(agent, (kind, item) => (kind === 'page_blocker'+ ? (/abone olun/.test(item.text) ? 'subscription' : 'other') : null));+ const batch = prefix => Array.from({ length: 40 }, (_, index) => ({ kind: 'page_blocker', text: `${prefix} ${index}` }));+ // The page reports forty texts at a time; the wall's text comes with the+ // second read.+ const responses = [+ { text: 'SECRET_SECOND', pageTextsNeeded: batch('Bülten B') },+ { text: 'SECRET_THIRD', pageTextsNeeded: [{ kind: 'page_blocker', text: 'Okumaya devam etmek için abone olun.' }] },+ { text: 'gate', pageGate: { type: 'subscription' } },+ { text: 'never read' },+ ];+ let dispatches = 0;+ const result = await agent._rereadWithPageBlockerReadings(1, 'read_page',+ { text: 'SECRET_FIRST', pageTextsNeeded: batch('Bülten A') }, async () => responses[dispatches++]);+ assert.equal(dispatches, 3, `${build}: the page was not read again for the texts each read reached`);+ assert.deepEqual(result, { text: 'gate', pageGate: { type: 'subscription' } });+ assert.equal(asked.length, 81);+ // It stops after three reads even when every read reaches more.+ const endless = new AgentClass({});+ stubPageTextReadings(endless, () => 'other');+ let reads = 0;+ const last = await endless._rereadWithPageBlockerReadings(1, 'get_accessibility_tree',+ { pageContent: 'tree', pageTextsNeeded: batch('Kutu 0') },+ async () => { reads += 1; return { pageContent: `tree ${reads}`, pageTextsNeeded: batch(`Kutu ${reads}`) }; });+ assert.equal(reads, 3);+ assert.deepEqual(last, { pageContent: 'tree 3' });+ }+});++test('a wall or dialog title no model answered is not asked again on every read for a while', async () => {+ for (const [build, AgentClass] of [['chrome', AgentCh], ['firefox', AgentFx]]) {+ const realNow = Date.now;+ let now = 1_000_000;+ Date.now = () => now;+ try {+ const agent = new AgentClass({});+ // An active model that answers nothing, as one that times out.+ agent._activeProvider = () => ({ chat: async () => ({}) });+ const asked = stubPageTextReadings(agent, () => null);+ const wall = 'Okumaya devam etmek için abone olun.';+ assert.equal(await agent._readPageBlockerTexts(1, [{ kind: 'page_blocker', text: wall }]), false);+ assert.equal(await agent._readChallengeTitles(1, ['Güvenlik doğrulaması']), false);+ assert.equal(asked.length, 2);+ // Within half a minute neither is asked again, so reads and actions do+ // not each wait for a model that cannot answer.+ now += 29_000;+ assert.equal(await agent._readPageBlockerTexts(1, [{ kind: 'page_blocker', text: wall }]), false);+ assert.equal(await agent._readChallengeTitles(1, ['Güvenlik doğrulaması']), false);+ assert.equal(asked.length, 2, `${build}: an unanswered text was asked again at once`);+ now += 2_000;+ await agent._readPageBlockerTexts(1, [{ kind: 'page_blocker', text: wall }]);+ await agent._readChallengeTitles(1, ['Güvenlik doğrulaması']);+ assert.equal(asked.length, 4, `${build}: an unanswered text was never asked again`);+ // A read by the decision model alone leaves a title for the next full+ // read, which may ask the active model at once; so does a read the+ // active model could not take part in.+ const decisionOnly = new AgentClass({});+ decisionOnly._activeProvider = () => ({ chat: async () => ({}) });+ const askedOnce = stubPageTextReadings(decisionOnly, () => null);+ await decisionOnly._readChallengeTitles(1, ['Doğrulama gerekli'], { mainModel: false });+ await decisionOnly._readChallengeTitles(1, ['Doğrulama gerekli']);+ assert.equal(askedOnce.length, 2, `${build}: a decision-model-only miss held back the active model`);+ const noModel = new AgentClass({});+ noModel._activeProvider = () => null;+ const askedNoModel = stubPageTextReadings(noModel, () => null);+ await noModel._readPageBlockerTexts(1, [{ kind: 'page_blocker', text: wall }]);+ await noModel._readPageBlockerTexts(1, [{ kind: 'page_blocker', text: wall }]);+ assert.equal(askedNoModel.length, 2, `${build}: a read no active model took part in held the text back`);+ // Other questions are asked again at once, as before.+ const other = new AgentClass({});+ const askedOther = stubPageTextReadings(other, () => null);+ await other._readControlEffects(1, ['Sil']);+ await other._readControlEffects(1, ['Sil']);+ assert.equal(askedOther.length, 2, `${build}: a control caption was held back`);+ } finally {+ Date.now = realNow;+ }}});