Best headless browser for AI agents. Lite, Fast, High-Compatibility. Built in Rust
Rust ai-agentsai-toolsbrowserbrowser-automationcloud-browserheadlessheadless-browserkitesurfplaywrightpuppeteerrustservoweb-crawlerweb-crawlingweb-scraperweb-scraping
refactor(benchmark): unify public web comparisons
ldm0c461cb1799ee
diff --git a/moli-benchmark/README.md b/moli-benchmark/README.mdindex 45a54a3c8..dc4e87b4b 100644--- a/moli-benchmark/README.md+++ b/moli-benchmark/README.md@@ -84,20 +84,19 @@ canvas dependencies. Failed reftests retain `test.png`, `reference-N.png`, andAn unfiltered full `default` or `all` run refreshes the unified status listsdirectly under `wpt-cross-current/`.-Public-web suites read a Markdown seed list. A minimal `sites.md` looks like:+Public-web suites read a `rank,target` CSV seed list. A minimal `sites.csv` looks like:-```markdown-## Top 2--1. `https://example.com/`-2. `https://www.rust-lang.org/`+```csv+rank,target+1,https://example.com/+2,https://www.rust-lang.org/```Compare Moli and Chrome on that sample:```bashuv run moli-benchmark top-sites \- --list-path sites.md \+ --list-path sites.csv \--profile quick \--target moli \--target chrome \@@ -108,7 +107,7 @@ Compare visible content with Chrome as the baseline:```bashuv run moli-benchmark render-compare \- --list-path sites.md \+ --list-path sites.csv \--profile quick \--target moli \--baseline-target chrome \@@ -196,9 +195,58 @@ The harness distinguishes the browser engine from the way it is driven:Not every suite accepts every target. Its `--help` output lists the validchoices.-The predefined public-web sources expect curated seed documents under the-repository's `docs/` directory. Those files are not present in every checkout;-use `--list-path` with `top-sites` or `render-compare` when they are unavailable.+The predefined public-web CSV sources ship under `fixtures/top-sites/`, including+the WebFetch longtail corpus used by `webfetch-mix`. Markdown remains supported+for custom `--list-path` inputs.++Public-web results distinguish an individual attempt from a stable per-site+outcome. `raw-runs.csv` and `runs.json` retain every attempt, while+`site-outcomes.json` groups repeated attempts into `all-pass`, `all-fail`, and+`flaky` sites. Pairwise engine rows are marked `SINGLE SAMPLE` unless the suite+was run with at least three attempts per site; use `--runs 3` or more before+treating an engine-only result as repeat-validated.++Public-web summaries keep three different populations explicit. `raw_*`+metrics include reachable and unreachable observations before cross-engine site+exclusions; the normal pass rate uses counted, comparable attempts; and+`successful_*` latency/memory metrics contain successful attempts only. The+`common_success` cohort contains the exact run/site attempts that succeeded on+every selected target and is the source for cross-engine speed and memory+claims in the HTML report.++Multi-engine public-web runs are scheduled in site-paired groups. Each group+runs the selected targets sequentially, rotates which target goes first, and+allows multiple site groups to run concurrently. Raw rows record the schedule+and target-order indexes, UTC start/finish times, output hashes and samples,+response MIME/body-capture evidence, and the final URL when CDP exposes it.+This makes order effects and response changes auditable without retaining every+successful response body.++When the driver exposes the main-document HTTP status, that status is+authoritative: a rendered 4xx or 5xx error document is a failure even if its+body exceeds the content-size threshold. The report includes HTTP-status+coverage, evidence source, and classification basis for each target. Moli CLI+failures that name a terminal main-document status retain it as+`cli-diagnostic` evidence. CLI fetch drivers that expose neither protocol nor+diagnostic status fall back to conservative error-document markers; for a+protocol-aligned comparison with status coverage, select the `moli-cdp` and+`lightpanda-cdp` targets alongside `chrome`.++A CDP browser reports a binary main-document MIME type before it exposes a+download body. That is retained as `binary-response-headers`, not fabricated+into a successful PDF or archive. Header-only observations are neutral and do+not enter body-success or latency denominators. A CLI transfer that times out+after receiving only part of a binary response remains a transfer failure. For+an explicit `.pdf` main-resource URL, the Moli CLI adapter omits DOM-only page+wait options and counts the result only when the real PDF body is returned.++The built-in `wild-web` targets use the same DOMContentLoaded snapshot boundary+for Moli, Lightpanda, and Chrome: after the adapter observes DOMContentLoaded,+the page event loop must advance for at least 50 ms before the DOM is dumped.+The total readiness deadline still applies to that settle period. Seed+extraction checks require the expected site identity in the title plus a+non-trivial body; the brand name does not need to be repeated in the first+body-text sample.Moli is discovered from `MOLI_BIN`, `../target/release/moli`, or `PATH`, inthat order. Comparison browsers are optional and can be selected through@@ -295,8 +343,8 @@ Run the core CLI tests from `moli-benchmark/`:uv run python -m unittest discover -s tests -p 'test_cli.py'```-The complete test suite uses `uv run python -m unittest discover -s tests` and-also expects the curated public-web seed documents described above.+The complete test suite uses `uv run python -m unittest discover -s tests`; its+curated public-web seed CSV files are versioned under `fixtures/top-sites/`.Keep benchmark claims tied to archived raw data, exact binary versions, andthe readiness checks. When adding a suite or target, update its CLI help anddiff --git a/moli-benchmark/fixtures/top-sites/chinese-community-top100-websites.csv b/moli-benchmark/fixtures/top-sites/chinese-community-top100-websites.csvnew file mode 100644index 000000000..b3287fcbf--- /dev/null+++ b/moli-benchmark/fixtures/top-sites/chinese-community-top100-websites.csv@@ -0,0 +1,101 @@+rank,target+1,zhihu.com+2,tieba.baidu.com+3,douban.com+4,bilibili.com+5,weibo.com+6,xiaohongshu.com+7,douyin.com+8,kuaishou.com+9,hupu.com+10,guokr.com+11,lofter.com+12,www.acfun.cn+13,6park.com+14,wenxuecity.com+15,backchina.com+16,juejin.cn+17,csdn.net+18,cnblogs.com+19,oschina.net+20,segmentfault.com+21,v2ex.com+22,hostloc.com+23,linux.do+24,ruby-china.org+25,learnku.com+26,right.com.cn+27,testerhome.com+28,kanxue.com+29,chiphell.com+30,52pojie.cn+31,coolapk.com+32,expreview.com+33,mydigit.cn+34,pconline.com.cn+35,ithome.com+36,cnbeta.com+37,www.huxiu.com+38,zol.com.cn+39,mobile01.com+40,dcard.tw+41,ptt.cc+42,disp.cc+43,gamer.com.tw+44,lihkg.com+45,uwants.com+46,hkepc.com+47,discuss.com.hk+48,ck101.com+49,eyny.com+50,babyhome.com.tw+51,hk01.com+52,nga.cn+53,gcores.com+54,keylol.com+55,stage1st.com+56,a9vg.com+57,ali213.net+58,3dmgame.com+59,gameres.com+60,indienova.com+61,bangumi.tv+62,dmhy.org+63,tgbus.com+64,gamensky.com+65,douyu.com+66,huya.com+67,autohome.com.cn+68,dongchedi.com+69,xcar.com.cn+70,qyer.com+71,mafengwo.cn+72,fengniao.com+73,xitek.com+74,8264.com+75,babytree.com+76,mama.cn+77,pcbaby.com.cn+78,dxy.cn+79,haodf.com+80,zcool.com.cn+81,huaban.com+82,goumin.com+83,zhidao.baidu.com+84,smzdm.com+85,xueqiu.com+86,eastmoney.com+87,taoguba.com.cn+88,futunn.com+89,manmanbuy.com+90,xiachufang.com+91,kdslife.com+92,hi-pda.com+93,qidian.com+94,zongheng.com+95,jjwxc.net+96,17k.com+97,ximalaya.com+98,zsxq.com+99,anquanke.com+100,muchong.comdiff --git a/moli-benchmark/fixtures/top-sites/global-top-websites-seed-list.csv b/moli-benchmark/fixtures/top-sites/global-top-websites-seed-list.csvnew file mode 100644index 000000000..3eb81b3d6--- /dev/null+++ b/moli-benchmark/fixtures/top-sites/global-top-websites-seed-list.csv@@ -0,0 +1,101 @@+rank,target+1,google.com+2,facebook.com+3,microsoft.com+4,youtube.com+5,apple.com+6,instagram.com+7,linkedin.com+8,amazon.com+9,wikipedia.org+10,github.com+11,bing.com+12,netflix.com+13,chatgpt.com+14,pinterest.com+15,yahoo.com+16,x.com+17,tiktok.com+18,spotify.com+19,reddit.com+20,dropbox.com+21,nytimes.com+22,tumblr.com+23,paypal.com+24,discord.com+25,medium.com+26,cnn.com+27,theguardian.com+28,bbc.com+29,ebay.com+30,imdb.com+31,openai.com+32,booking.com+33,reuters.com+34,forbes.com+35,slack.com+36,wsj.com+37,nasa.gov+38,duckduckgo.com+39,brave.com+40,gitlab.com+41,quora.com+42,goodreads.com+43,stackoverflow.com+44,wired.com+45,techcrunch.com+46,theverge.com+47,anthropic.com+48,airbnb.com+49,whitehouse.gov+50,economist.com+51,perplexity.ai+52,huggingface.co+53,npmjs.com+54,ycombinator.com+55,kaggle.com+56,wolframalpha.com+57,vimeo.com+58,roblox.com+59,wordpress.com+60,bbc.co.uk+61,t.me+62,samsung.com+63,zoom.us+64,whatsapp.com+65,live.com+66,bit.ly+67,discord.gg+68,unity3d.com+69,intuit.com+70,snapchat.com+71,cnbc.com+72,bloomberg.com+73,npr.org+74,washingtonpost.com+75,apnews.com+76,usatoday.com+77,theatlantic.com+78,vox.com+79,arstechnica.com+80,news.ycombinator.com+81,gov.uk+82,usa.gov+83,mit.edu+84,stanford.edu+85,cmu.edu+86,harvard.edu+87,nature.com+88,science.org+89,coursera.org+90,edx.org+91,khanacademy.org+92,udemy.com+93,figma.com+94,canva.com+95,notion.so+96,airtable.com+97,linear.app+98,vercel.com+99,cloudflare.com+100,npmjs.orgdiff --git a/moli-benchmark/fixtures/top-sites/legacy-encoding-websites-seed-list.csv b/moli-benchmark/fixtures/top-sites/legacy-encoding-websites-seed-list.csvnew file mode 100644index 000000000..ed5890db0--- /dev/null+++ b/moli-benchmark/fixtures/top-sites/legacy-encoding-websites-seed-list.csv@@ -0,0 +1,7 @@+rank,target+1,https://www.aozora.gr.jp/cards/000081/files/456_15050.html+2,https://www.debian.or.jp/+3,http://www.hiking.com.hk/camping/laichi.htm+4,http://lib.ru/+5,https://www.w3.org/TR/1998/REC-html40-19980424/+6,https://www.yes24.com/Main/default.aspxdiff --git a/moli-benchmark/fixtures/top-sites/render-quality-seed-list.csv b/moli-benchmark/fixtures/top-sites/render-quality-seed-list.csvnew file mode 100644index 000000000..108b4d0bc--- /dev/null+++ b/moli-benchmark/fixtures/top-sites/render-quality-seed-list.csv@@ -0,0 +1,13 @@+rank,target+1,https://www.feishu.cn/hc/zh-CN/articles/485770964672-管理者查看企业-ai-功能使用数据+2,https://docs.python.org/3/tutorial/index.html+3,https://nextjs.org/docs+4,https://react.dev/learn+5,https://vite.dev/guide/+6,https://nodejs.org/en/learn/getting-started/introduction-to-nodejs+7,https://axios-http.com/docs/intro+8,https://www.liaoxuefeng.com/wiki/1016959663602400/1017063413904832+9,https://pnpm.io/installation+10,https://www.rust-lang.org/learn+11,https://www.apple.com/iphone/+12,https://learn.microsoft.com/en-us/windows/win32/apiindex/windows-api-listdiff --git a/moli-benchmark/fixtures/top-sites/webfetch-longtail-seed-list.csv b/moli-benchmark/fixtures/top-sites/webfetch-longtail-seed-list.csvnew file mode 100644index 000000000..5afd6c0ac--- /dev/null+++ b/moli-benchmark/fixtures/top-sites/webfetch-longtail-seed-list.csv@@ -0,0 +1,160 @@+rank,target+1,https://www.cell.com/cell/fulltext/S0092-8674(24)00500-9+2,https://www.science.org/doi/10.1126/science.abj0890+3,https://www.investopedia.com/articles/personal-finance/082316/how-much-money-do-you-need-live-newyork-city.asp+4,https://seekingalpha.com/article/4666666-tesla-q1-2024+5,https://www.metacritic.com/movie/oppenheimer+6,https://www.reddit.com/r/MachineLearning/comments/1aqxhol/d_thoughts_on_llama_3/+7,https://www.reddit.com/r/learnpython/comments/1arpz0k/how_do_i_start_learning_python/+8,https://www.reddit.com/r/programming/comments/1ar0khc/openai_sora_text_to_video_model/+9,https://dl.acm.org/doi/10.1145/3597503.3623298+10,https://www.adidas.com/us/samba-og-shoes/B75806.html+11,https://www.allrecipes.com/recipe/231170/cuban-mojo-marinated-pork/+12,https://www.seriouseats.com/foolproof-pan-pizza-recipe+13,https://www.travelandleisure.com/best-tokyo-restaurants+14,https://www.tripadvisor.com/Attraction_Review-g60750-d136919-Reviews-Empire_State_Building-New_York_City_New_York.html+15,https://stackoverflow.com/questions/2081586/web-scraping-with-python+16,https://stackoverflow.com/questions/3437059/does-anyone-have-a-good-overview-of-the-pros-cons-of-using-x-frame-options+17,https://www.douban.com/note/856000000/+18,https://sourceforge.net/projects/sevenzip/+19,https://www.adidas.com/us/men-shoes+20,https://martinfowler.com/articles/+21,https://community.cloudflare.com/+22,https://discuss.python.org/+23,https://www2.census.gov/programs-surveys/decennial/2020/data/apportionment/apportionment-2020-table02.pdf+24,https://askubuntu.com/questions/3973/how-to-find-out-which-package-a-file-belongs-to+25,https://stackoverflow.com/questions/89228/how-do-i-execute-a-program-or-call-a-system-command+26,https://www.pro-football-reference.com/players/M/McDoRi20.htm+27,https://www.northjersey.com/obituaries/ber117653+28,https://stackoverflow.com/questions/231767/what-does-the-yield-keyword-do-in-python+29,https://www.espn.com/tennis/story/_/id/18401434/tennis-why-12-year-old-cori-gauff-thinks-greatest-all+30,https://archaeologydataservice.ac.uk/catalogue/adsdata/arch-481-1/dissemination/pdf/archaeol7-93506_1.pdf+31,https://www.nea.org/sites/default/files/2024-04/2023-rankings-and-estimates-report-1.pdf+32,https://www.sports-reference.com/cbb/schools/connecticut/women/2023.html+33,https://www.sports-reference.com/cbb/schools/south-carolina/women/2023.html+34,https://www.sports-reference.com/cbb/schools/washington/women/2023.html+35,https://www.europeana.eu/ro/collections/person/141236-leon-levy-brunswick+36,https://serverfault.com/questions/2670/how-can-i-detect-which-machine-on-a-network-changed-an-active-directory-objec+37,https://www.legacy.com/obituaries/name/lawrence-kramer-obituary?pid=205175579+38,https://www.pro-football-reference.com/players/M/McDoDe20.htm+39,https://catalog.hathitrust.org/Record/001042525+40,https://getmusicbee.com/help/release-note/+41,https://newjerseyglobe.com/in-memoriam/pat-kramer-four-term-paterson-mayor-and-gop-gubernatorial-frontrunner-dies-at-90/+42,https://stats.bls.gov/lau/lastrk20.htm+43,https://www.researchgate.net/publication/266379416_Identifying_semantic_role_clusters_and_alignment_types_via_microrole_coexpression_tendencies+44,https://electionresults.govt.nz/electionresults_2017/+45,https://tennispredict.com/coco-gauff/+46,https://press.vub.ac.be/vub-honorary-doctorates-for-science-communicator-ann-druyan-artist-william-kentridge-politician-pierre-kompany-and-immunotherapy-experts-antoni-ribas-and-thierry-boon+47,https://www.ask-oracle.com/birthday/1956/04/12/+48,https://electionresults.govt.nz/electionresults_2023/electorate-details-60.html+49,https://netflixtechblog.com/scaling-time-series-data-storage-part-i-ec2b6d44ba39+50,https://netflixtechblog.com/+51,https://lobste.rs/+52,https://fsi.nic.in/isfr2019/isfr-fsi-vol2.pdf+53,https://wikinewzealand.org/table/1ZZwmJ2Hoc3tHDqc+54,http://www.wfsq.gov.cn/91/1993483352635740160.html+55,https://www.titan24.com/publish/app/data/2017/02/04/38517/os_news.html+56,https://beta.boost.org/doc/libs/1_68_0/libs/numeric/ublas/index.html+57,http://202.106.184.141:89/gate/big5/www.chinanews.com.cn/2002-10-12/26/231332.html+58,http://big5.chinanews.com.cn:89/gate/big5/www.chinanews.com.cn/2003-09-20/26/348867.html+59,https://manage.eblcu.cn/specls/cls/405213/course/03/03_11_02_02.html+60,https://tc.canada.ca/en/road-transportation/statistics-data/canadian-motor-vehicle-traffic-collision-statistics/2022/canadian-motor-vehicle-traffic-collision-statistics-2022#fn5+61,https://tc.canada.ca/en/road-transportation/statistics-data/canadian-motor-vehicle-traffic-collision-statistics/2022/canadian-motor-vehicle-traffic-collision-statistics-2022?wbdisable=true+62,https://api.worldbank.org/v2/country/IRL/indicator/EN.GHG.CO2.PC.CE.AR5?date=2023&format=json+63,https://api.worldbank.org/v2/country/ISL/indicator/EN.GHG.CO2.PC.CE.AR5?date=2023&format=json+64,https://api.worldbank.org/v2/country/NZL/indicator/EN.GHG.CO2.PC.CE.AR5?date=2023&format=json+65,https://api.worldbank.org/v2/country/all/indicator/EG.FEC.RNEW.ZS?mrv=1&format=json&per_page=100&page=2+66,http://epaper.bingtuannet.com/pc/cont/202010/29/c134120.html+67,http://gxq.qingdao.gov.cn/xxgk_113/zwzx_113/gxdt_113/202203/t20220322_4955636.shtml+68,http://hndfxj.txhn.net/xjwx/xjyj/wz/201611/t20161128_523882.html+69,http://house.enorth.com.cn/system/2017/06/09/033161887.shtml+70,http://huaxia.com/c/2023/01/31/1600892.shtml+71,https://japan.people.com.cn/BIG5/n1/2026/0211/c35421-40663642.html+72,http://ly.jiangxi.gov.cn/jxslyj/wzs/pc/content/content_1962334293316018176.html+73,http://mee.gov.cn/gkml/sthjbgw/stbgth/201806/W020180613394538476583.pdf+74,https://revues.imist.ma/index.php/JALCS/article/view/37261+75,http://wlt.shanxi.gov.cn/zh/sourcefiles/html/attractions/1749.shtml+76,https://www.cqmetro.cn/tpxw/2025/8/14/2170123.shtml+77,http://www.dg.gov.cn/zjdz/dzgk/zrdl/content/post_2824920.html+78,https://www.foshan.gov.cn/gzjg/fswenhua/wzfs/content/post_7145707.html+79,https://www.govinfo.gov/content/pkg/GPO-CRECB-1939-pt1-v84/pdf/GPO-CRECB-1939-pt1-v84-3-1.pdf+80,https://www.irya.unam.mx/web/en/news/archive/464-remembering-four-unam-women-astronomers+81,http://www.polygon.com/2012/10/22/3539692/dark-souls-update-106-weakens-weapons-adds-warp-zones-dlc+82,https://www.polygon.com/2012/10/22/3539692/dark-souls-update-106-weakens-weapons-adds-warp-zones-dlc/+83,https://www.polygon.com/2013/4/18/4238514/pokemon-mini-trailer-confirms-new-legendarys-connection-to-mewtwo/+84,https://www.scottishmountainrescue.org/wp-content/uploads/2025/04/SMR_Statistics_Report_2023.pdf+85,https://www.shanxi.gov.cn/zjsx/sxfg/cysx/202007/t20200724_6047084.shtml+86,https://www.zhangzhou.gov.cn/cms/html/zzshyyyyj/2025-11-28/1082544542.html+87,http://www.zuxun100.com/index/article/mingren/info/52174+88,https://yjglt.jiangxi.gov.cn/jxsyjglt/col/col37571/content/content_1766359793244237824.html+89,http://dgcg.dg.gov.cn/zwy/yqhx/201008/aafc2e43b7cd437ab757e5d3fcdbe69a.shtml+90,https://download.militaryonesource.mil/12038/MOS/Reports/2023-demographics-report.pdf+91,https://forests.gujarat.gov.in/writereaddata/images/pdf/GFS-2019-20.pdf+92,https://jtys.foshan.gov.cn/jtdt/jtdt/content/post_6957426.html+93,https://de.wikipedia.org/wiki/Generative_K%C3%BCnstliche_Intelligenz+94,https://en.wikipedia.org/wiki/Brihaspati_VidyaSadan+95,"https://en.wikipedia.org/wiki/Brihaspati_VidyaSadan#:~:text=Brihaspati%20Vidyasadan%20was%20established%20in,the%20school%20was%20Maurice%20Banerjee."+96,"https://en.wikipedia.org/wiki/Homey,_Don't_Ya_Know_Me%3F"+97,https://en.wikipedia.org/wiki/Mary_Letitia_Caldwella+98,"https://ghoapi.azureedge.net/api/NCD_BMI_30A?$filter=TimeDim%20eq%202021%20and%20Dim1%20eq%20'SEX_BTSX'%20and%20Dim2%20eq%20'AGEGROUP_YEARS18-PLUS'%20and%20NumericValue%20gt%2010&$select=SpatialDim,NumericValue&$top=5000"+99,https://ghoapi.azureedge.net/api/NCD_BMI_30A?$filter=TimeDim%20eq%202021%20and%20Dim1%20eq%20'SEX_BTSX'%20and%20Dim2%20eq%20'AGEGROUP_YEARS18-PLUS'%20and%20NumericValue%20gt%2010&$top=5000+100,https://zh.wikipedia.org/wiki/%E6%9F%AC%E5%9F%AC%E5%AF%A8%E7%8E%8B%E5%AE%B6%E7%B7%A8%E5%B9%B4%E5%8F%B2+101,https://en.wikipedia.org/wiki/Where_Is_the_Love%3F_(The_Black_Eyed_Peas_song)+102,https://web.archive.org/web/20200418034340/https://www.sports-reference.com/olympics/athletes/br/anne-brusletto-1.html+103,https://web.archive.org/web/20030830023637/http://www.abchsn.com/users/aus_wildlife/broad_faced_potoroo.html+104,https://web.archive.org/web/20110604164452/http://portal.unesco.org/ci/en/ev.php-URL_ID%3D21749%26URL_DO%3DDO_TOPIC%26URL_SECTION%3D201.html+105,https://web.archive.org/web/20121013165655/http://www.palmspringswalkofstars.com/web-storage/Stars/Stars%20dedicated%20by%20date.pdf+106,https://www.metmuseum.org/research-centers/leonard-a-lauder-research-center/research-resources/modern-art-index-project/cone+107,https://thefourwallmag.wordpress.com/2015/12/07/daks-in-the-city-interview-with-bonang-matheba+108,https://www.nfsa.gov.au/stories/articles/colour-tv-part-1+109,https://alistapart.com/article/css-positioning-101/+110,https://borthcat.york.ac.uk/index.php/gale-george-william-1900-1976-professor-of-medicine+111,https://finance.yahoo.com/news/4ever-young-revolutionizing-wellness-aesthetics-190000784.html+112,https://www.gbif.org/species/7301494+113,https://nz.news.yahoo.com/history-behind-met-gala-215735843.html+114,https://www.alamy.com/statue-of-alexander-mackenzie-1822-1892-pm-of-canada-1873-1878-the-statue-was-carved-by-louis-philippe-hbert-in-1900-and-placed-on-parliament-hill-in-1901-image185562270.html+115,https://finance.yahoo.com/+116,https://www.engadget.com/+117,https://www.expedia.com/+118,https://www.reuters.com/technology/artificial-intelligence/openai-launches-sora-create-video-prompt-2024-02-15/+119,https://www.reuters.com/technology/+120,https://www.historyofparliamentonline.org/volume/1820-1832/member/kerr-john-1794-1841+121,https://www.ranker.com/list/famous-politicians-from-italy/reference?page=3+122,https://owid.netlify.app/grapher/co2-per-capita.csv?time=2022+123,https://owid.netlify.app/grapher/per-capita-energy-use.csv?time=2022+124,https://owid.netlify.app/grapher/share-electricity-renewables.csv?time=2022+125,https://www.bankrate.com/banking/savings/rates/+126,https://www.bilibili.com/video/BV1uJ4m1H7Mh+127,https://www.taptap.cn/moment/795408535407037831+128,https://ecommons.cornell.edu/bitstream/handle/1813/18102/Wang_Hsien-Chung_1978.pdf+129,http://www.gamefaqs.com/mac/707665-sid-meiers-civilization-v-brave-new-world/credit+130,https://webarchive.nationalarchives.gov.uk/ukgwa/20240404230934/https://www.gov.uk/government/statistics/fly-tipping-in-england/local-authority-fly-tipping-enforcement-league-tables-for-england-202122+131,https://epaper.fsonline.com.cn/fsrb/html/2026-04/19/content_71070_325266.htm+132,https://www.gmw.cn/01gmrb/1999-01/05/GB/17928^GM5-0507.HTM+133,https://max.book118.com/html/2018/0607/171282284.shtm+134,https://www.transfermarkt.com/fc-barcelona_rcd-espanyol-barcelona/index/spielbericht/58777+135,https://www.everand.com/author/366145856/Helen-Schreider+136,https://www.scribd.com/document/402397780/PROS-SAGSSAGO-NOTES-IN-CRIMINAL-LAW-2017-EDITION-converted-docx+137,"https://ghoapi.azureedge.net/api/M_Est_tob_curr_std?$filter=TimeDim%20eq%202021%20and%20Dim1%20eq%20'SEX_BTSX'%20and%20NumericValue%20gt%2030&$select=SpatialDim,NumericValue&$top=5000"+138,https://ghoapi.azureedge.net/api/M_Est_tob_curr_std?$filter=TimeDim%20eq%202021%20and%20Dim1%20eq%20'SEX_BTSX'%20and%20NumericValue%20gt%2030&$top=5000+139,https://m.chem960.com/msds/92875+140,https://www.fzfzjx.com/tech_news/detail-6754.html+141,https://www.state.gov/press-releases/+142,https://www.bankrate.com/banking/savings/+143,"https://ghoapi.azureedge.net/api/NCDMORT3070?$filter=TimeDim%20eq%202021%20and%20Dim1%20eq%20'SEX_BTSX'%20and%20Dim2%20eq%20'AGEGROUP_YEARS30-69'%20and%20NumericValue%20gt%2025&$select=SpatialDim,NumericValue&$top=5000"+144,https://jssdfz.jiangsu.gov.cn/n101/20240612/i34078.html+145,https://m.liuxue114.com/japan/jpmx/1358996225139.html+146,https://data.worldbank.org/indicator/SL.AGR.0714.ZS?locations=EC+147,https://staging.europepmc.org/backend/articlerender.fcgi?accid=PMC11156239+148,https://www.qiuwenbaike.cn/wiki/%E4%BA%91%E5%AD%98%E5%82%A8+149,https://www.qiuwenbaike.cn/wiki/%E7%94%B0%E5%83%96%E5%AD%90+150,https://www.qiuwenbaike.cn/wiki/%E8%90%A5%E4%B8%9A%E7%A8%8E+151,https://bloodstainedritualofthenight.wiki.fextralife.com/Partisan+152,https://darksouls2.wiki.fextralife.com/Puzzling+Stone+Sword+153,https://demonssouls.wiki.fextralife.com/Spears+154,http://mp.weixin.qq.com/s?__biz=MjM5MDI1MDQ2Mw==&mid=400754239&idx=2&sn=ee6a9f8f5ac304c477f64fa1f66cb2b5&chksm=35aace048703251ed2d2c6c09ea90c1d7d9a2e477146e3a8bd649edccaafc7430849a72bfceb#rd+155,http://mp.weixin.qq.com/s?__biz=MjM5ODgzODk2Mw==&mid=2652134873&idx=1&sn=e54b222dac2ee123809c91e103cc20ad&chksm=bc581578fe2e77a4166c4b880bef6337b724807a2cacf7efb9ca1bf06597fda29c721543dc2b#rd+156,http://mp.weixin.qq.com/s?__biz=MzA4MDIxMjcyMA==&mid=2653599762&idx=3&sn=417d0c4afa2eacc6cc2f059c3ca8269e&chksm=8589040c58b452402abbdb5dd36178206036c4724ae2232e74517c749c22cc3961b7c197bc76#rd+157,http://mp.weixin.qq.com/s?__biz=MzA5MzI3NjkyNA==&mid=2654050551&idx=1&sn=3620410479393ed9eb72e096b93b6a7f&chksm=8a6fcaebf0edeaf4e232f873051fc60f73984ba33078017849765bee30dcba96e65442852e10#rd+158,https://sekiroshadowsdietwice.wiki.fextralife.com/Lady+Tomoe+159,https://www.linkedin.com/in/williamhgatesdiff --git a/moli-benchmark/moli_benchmark/chrome_dcl.py b/moli-benchmark/moli_benchmark/chrome_dcl.pyindex 7bdafec1a..5525703d4 100644--- a/moli-benchmark/moli_benchmark/chrome_dcl.py+++ b/moli-benchmark/moli_benchmark/chrome_dcl.py@@ -6,11 +6,13 @@ import signalimport subprocessimport tempfileimport time+from dataclasses import dataclassfrom pathlib import Pathfrom typing import Anyfrom .config import REPO_ROOT, reserve_portfrom .process import ProcessResult+from .public_web import POST_DCL_SETTLE_MILLISECONDSfrom .raw_cdp import RawCdpClient, RawCdpError, connect_raw_cdpfrom .sampling import ResourceSamplerfrom .target_serve import start_target_serve, stop_target_serve@@ -26,6 +28,14 @@ DEFAULT_CHROME_DCL_USER_AGENT = (# timeout. Keep this small so failed navigations do not distort timing reports.CDP_LATE_ERROR_GRACE_SECONDS = 2.0+_POST_DCL_SETTLE_EXPRESSION = (+ "new Promise(resolve => "+ f"setTimeout(resolve, {POST_DCL_SETTLE_MILLISECONDS}))"+)+_POST_DCL_OUTER_HTML_EXPRESSION = (+ "document.documentElement ? document.documentElement.outerHTML : ''"+)+class CdpDclDumpTimeoutError(TimeoutError):def __init__(self, stage: str, error: BaseException) -> None:@@ -40,6 +50,15 @@ class CdpDclDumpTimeoutError(TimeoutError):return detail if detail else self.stage+@dataclass(frozen=True)+class CdpDclDumpResult:+ body: str+ response_status: int | None+ response_mime_type: str | None+ main_document_body_capture: str+ final_url: str | None++def _chrome_command(binary: Path, port: int, profile_dir: Path) -> list[str]:return [str(binary),@@ -126,19 +145,29 @@ async def _raise_late_command_error_or_timeout(raise CdpDclDumpTimeoutError(stage, timeout_error) from timeout_error-def _is_dcl_event(message: dict[str, Any], session_id: str, frame_id: str | None) -> bool:+def _is_dcl_event(+ message: dict[str, Any],+ session_id: str,+ frame_id: str | None,+ expected_loader_id: str | None,+) -> bool:if message.get("sessionId") != session_id:return False- method = message.get("method")- if method == "Page.domContentEventFired":- return True- if method != "Page.lifecycleEvent":+ # Page.domContentEventFired has no loader identity. Lifecycle events are+ # enabled for this runner and can be matched to the loader returned by+ # Page.navigate below.+ if message.get("method") != "Page.lifecycleEvent":return Falseparams = message.get("params")if not isinstance(params, dict):return Falseif frame_id is not None and params.get("frameId") != frame_id:return False+ if (+ expected_loader_id is not None+ and params.get("loaderId") != expected_loader_id+ ):+ return Falsereturn params.get("name") in {"DOMContentLoaded", "domContentLoaded"}@@ -170,40 +199,112 @@ def _is_binary_document_mime_type(mime_type: str) -> bool:)-def _binary_main_resource_body(mime_type: str) -> str:- normalized = mime_type.split(";", 1)[0].strip().lower()- if normalized == "application/pdf":- return "%PDF-1.7\n% moli benchmark CDP binary main resource\n" + ("\0" * 512)- return f"moli benchmark CDP binary main resource: {normalized}\n" + ("\0" * 512)---def _binary_main_resource_body_from_message(+def _main_document_response_from_message(message: dict[str, Any],*,session_id: str,frame_id: str | None,-) -> str | None:+ expected_loader_id: str | None = None,+ main_document_request_ids: set[str] | None = None,+) -> dict[str, Any] | None:if message.get("sessionId") != session_id:return None- if message.get("method") != "Network.responseReceived":- return Noneparams = message.get("params")if not isinstance(params, dict):return None- if params.get("type") != "Document":+ if (+ expected_loader_id is not None+ and params.get("loaderId") != expected_loader_id+ ):+ return None++ method = message.get("method")+ if method == "Network.requestWillBeSent":+ if params.get("type") != "Document":+ return None+ if frame_id is not None and params.get("frameId") != frame_id:+ return None+ request_id = params.get("requestId")+ if main_document_request_ids is not None and isinstance(request_id, str):+ main_document_request_ids.clear()+ main_document_request_ids.add(request_id)+ return None++ if method != "Network.responseReceived":return Noneif frame_id is not None and params.get("frameId") != frame_id:return None+ request_id = params.get("requestId")+ request_was_document = (+ isinstance(request_id, str)+ and main_document_request_ids is not None+ and request_id in main_document_request_ids+ )+ if (+ isinstance(request_id, str)+ and main_document_request_ids+ and not request_was_document+ ):+ return None+ if params.get("type") != "Document" and not request_was_document:+ return None+ if main_document_request_ids is not None and isinstance(request_id, str):+ main_document_request_ids.clear()+ main_document_request_ids.add(request_id)response = params.get("response")if not isinstance(response, dict):return None+ return response+++def _response_status(response: dict[str, Any]) -> int | None:status = response.get("status")- if isinstance(status, (int, float)) and not 200 <= status < 400:+ if isinstance(status, bool) or not isinstance(status, (int, float)):+ return None+ return int(status)+++def _response_mime_type(response: dict[str, Any]) -> str | None:+ mime_type = response.get("mimeType")+ if not isinstance(mime_type, str):+ return None+ normalized = mime_type.strip().lower()+ return normalized or None+++def _response_url(response: dict[str, Any]) -> str | None:+ url = response.get("url")+ return url if isinstance(url, str) and url else None+++def _binary_main_resource_mime_type_from_message(+ message: dict[str, Any],+ *,+ session_id: str,+ frame_id: str | None,+ expected_loader_id: str | None = None,+) -> str | None:+ response = _main_document_response_from_message(+ message,+ session_id=session_id,+ frame_id=frame_id,+ expected_loader_id=expected_loader_id,+ )+ return _binary_main_resource_mime_type_from_response(response)+++def _binary_main_resource_mime_type_from_response(+ response: dict[str, Any] | None,+) -> str | None:+ if response is None:+ return None+ status = _response_status(response)+ if status is not None and not 200 <= status < 400:return Nonemime_type = response.get("mimeType")if not isinstance(mime_type, str) or not _is_binary_document_mime_type(mime_type):return None- return _binary_main_resource_body(mime_type)+ return _response_mime_type(response)async def _recv_until_dcl_or_binary_main_resource(@@ -211,36 +312,91 @@ async def _recv_until_dcl_or_binary_main_resource(*,session_id: str,frame_id: str | None,+ expected_loader_id: str | None,deadline: float,seen: list[dict[str, Any]],-) -> str | None:+) -> tuple[str | None, int | None, str | None, str | None]:+ response_status: int | None = None+ response_mime_type: str | None = None+ response_url: str | None = None+ main_document_request_ids: set[str] = set()for message in seen:- binary_body = _binary_main_resource_body_from_message(+ response = _main_document_response_from_message(message,session_id=session_id,frame_id=frame_id,+ expected_loader_id=expected_loader_id,+ main_document_request_ids=main_document_request_ids,)- if binary_body is not None:- return binary_body- if any(_is_dcl_event(message, session_id, frame_id) for message in seen):- return None+ if response is not None:+ response_status = _response_status(response)+ response_mime_type = _response_mime_type(response)+ response_url = _response_url(response)+ binary_mime_type = _binary_main_resource_mime_type_from_response(response)+ if binary_mime_type is not None:+ return (+ binary_mime_type,+ response_status,+ response_mime_type,+ response_url,+ )+ if any(+ _is_dcl_event(message, session_id, frame_id, expected_loader_id)+ for message in seen+ ):+ return None, response_status, response_mime_type, response_urlwhile True:remaining = deadline - time.perf_counter()if remaining <= 0:raise TimeoutError("timed out waiting for Page.domContentEventFired")message = await asyncio.wait_for(client.recv(), timeout=remaining)- binary_body = _binary_main_resource_body_from_message(+ response = _main_document_response_from_message(message,session_id=session_id,frame_id=frame_id,+ expected_loader_id=expected_loader_id,+ main_document_request_ids=main_document_request_ids,)- if binary_body is not None:- return binary_body- if _is_dcl_event(message, session_id, frame_id):- return None+ if response is not None:+ response_status = _response_status(response)+ response_mime_type = _response_mime_type(response)+ response_url = _response_url(response)+ binary_mime_type = _binary_main_resource_mime_type_from_response(response)+ if binary_mime_type is not None:+ return (+ binary_mime_type,+ response_status,+ response_mime_type,+ response_url,+ )+ if _is_dcl_event(message, session_id, frame_id, expected_loader_id):+ return None, response_status, response_mime_type, response_url+++def _navigation_frame_and_loader(+ navigate_response: dict[str, Any],+ url: str,+) -> tuple[str | None, str | None]:+ result = navigate_response.get("result")+ if not isinstance(result, dict):+ result = {}+ error_text = result.get("errorText")+ if isinstance(error_text, str) and error_text:+ raise RawCdpError(f"Page.navigate failed for `{url}`: {error_text}")+ frame_id = result.get("frameId")+ loader_id = result.get("loaderId")+ return (+ str(frame_id) if frame_id is not None else None,+ str(loader_id) if loader_id is not None else None,+ )-async def _dump_dcl_html(endpoint: str, process: subprocess.Popen[bytes], url: str, timeout_seconds: float) -> str:+async def _dump_dcl_html(+ endpoint: str,+ process: subprocess.Popen[bytes],+ url: str,+ timeout_seconds: float,+) -> CdpDclDumpResult:deadline = time.perf_counter() + timeout_secondstry:client = await _wait_for_cdp(endpoint, process, min(5.0, max(0.1, timeout_seconds)))@@ -290,26 +446,56 @@ async def _dump_dcl_html(endpoint: str, process: subprocess.Popen[bytes], url: sstage="Page.navigate",late_error_grace_seconds=CDP_LATE_ERROR_GRACE_SECONDS,)- frame_id = navigate_response.get("result", {}).get("frameId")- if frame_id is not None:- frame_id = str(frame_id)+ frame_id, expected_loader_id = _navigation_frame_and_loader(+ navigate_response,+ url,+ )try:- binary_body = await _recv_until_dcl_or_binary_main_resource(+ main_document = await _recv_until_dcl_or_binary_main_resource(client,session_id=session_id,frame_id=frame_id,+ expected_loader_id=expected_loader_id,deadline=deadline,seen=seen,)+ (+ binary_mime_type,+ response_status,+ response_mime_type,+ final_url,+ ) = main_documentexcept TimeoutError as error:raise CdpDclDumpTimeoutError("DCL", error) from error- if binary_body is not None:- return binary_body+ if binary_mime_type is not None:+ return CdpDclDumpResult(+ body="",+ response_status=response_status,+ response_mime_type=binary_mime_type,+ main_document_body_capture="response-headers-only",+ final_url=final_url,+ )++ settle_id = await client.send(+ "Runtime.evaluate",+ {+ "expression": _POST_DCL_SETTLE_EXPRESSION,+ "awaitPromise": True,+ "returnByValue": True,+ },+ session_id=session_id,+ )+ await _recv_command_response(+ client,+ settle_id,+ deadline=deadline,+ stage="post-DCL settle",+ )evaluate_id = await client.send("Runtime.evaluate",{- "expression": "document.documentElement ? document.documentElement.outerHTML : ''",+ "expression": _POST_DCL_OUTER_HTML_EXPRESSION,"returnByValue": True,},session_id=session_id,@@ -322,7 +508,13 @@ async def _dump_dcl_html(endpoint: str, process: subprocess.Popen[bytes], url: s)result = evaluate_response.get("result", {}).get("result", {})value = result.get("value", "")- return value if isinstance(value, str) else ""+ return CdpDclDumpResult(+ body=value if isinstance(value, str) else "",+ response_status=response_status,+ response_mime_type=response_mime_type,+ main_document_body_capture="dom-snapshot",+ final_url=final_url,+ )finally:if target_id is not None:try:@@ -383,6 +575,10 @@ def run_chrome_dcl_dump(error_suffix = b""timed_out = Falsereturncode: int | None = None+ response_status: int | None = None+ response_mime_type: str | None = None+ main_document_body_capture: str | None = None+ final_url: str | None = Nonewith tempfile.TemporaryDirectory(prefix="moli-benchmark-chrome-") as temp_dir:profile_dir = Path(temp_dir) / "profile"profile_dir.mkdir()@@ -409,8 +605,14 @@ def run_chrome_dcl_dump(sampler.start()try:try:- html = asyncio.run(_dump_dcl_html(endpoint, process, url, timeout_seconds))- stdout = html.encode("utf-8", errors="replace")+ dump = asyncio.run(+ _dump_dcl_html(endpoint, process, url, timeout_seconds)+ )+ stdout = dump.body.encode("utf-8", errors="replace")+ response_status = dump.response_status+ response_mime_type = dump.response_mime_type+ main_document_body_capture = dump.main_document_body_capture+ final_url = dump.final_urlreturncode = 0except CdpDclDumpTimeoutError as error:timed_out = True@@ -435,7 +637,11 @@ def run_chrome_dcl_dump(finally:elapsed_ms = (time.perf_counter() - started) * 1000.0resources = sampler.stop() if sampler is not None else {}- if browser_stdout.strip() and not stdout:+ if (+ browser_stdout.strip()+ and not stdout+ and main_document_body_capture is None+ ):stdout = browser_stdoutreturn ProcessResult(command=command,@@ -445,6 +651,10 @@ def run_chrome_dcl_dump(stderr=stderr,timed_out=timed_out,resources=resources,+ response_status=response_status,+ response_mime_type=response_mime_type,+ main_document_body_capture=main_document_body_capture,+ final_url=final_url,)@@ -466,13 +676,23 @@ def run_served_cdp_dcl_dump(timed_out = Falsereturncode: int | None = Noneresources: dict[str, Any] = {}+ response_status: int | None = None+ response_mime_type: str | None = None+ main_document_body_capture: str | None = None+ final_url: str | None = Noneserve = Nonetry:serve = start_target_serve(target, binary, timeout_seconds)command = serve.commandtry:- html = asyncio.run(_dump_dcl_html(serve.endpoint, serve.process, url, timeout_seconds))- stdout = html.encode("utf-8", errors="replace")+ dump = asyncio.run(+ _dump_dcl_html(serve.endpoint, serve.process, url, timeout_seconds)+ )+ stdout = dump.body.encode("utf-8", errors="replace")+ response_status = dump.response_status+ response_mime_type = dump.response_mime_type+ main_document_body_capture = dump.main_document_body_capture+ final_url = dump.final_urlreturncode = 0except CdpDclDumpTimeoutError as error:timed_out = True@@ -521,4 +741,8 @@ def run_served_cdp_dcl_dump(stderr=stderr,timed_out=timed_out,resources=resources,+ response_status=response_status,+ response_mime_type=response_mime_type,+ main_document_body_capture=main_document_body_capture,+ final_url=final_url,)diff --git a/moli-benchmark/moli_benchmark/html_report.py b/moli-benchmark/moli_benchmark/html_report.pyindex 419fb0d8b..ce1464bd0 100644--- a/moli-benchmark/moli_benchmark/html_report.py+++ b/moli-benchmark/moli_benchmark/html_report.py@@ -250,7 +250,11 @@ def _artifact_paths_by_suite(summaries: list[dict[str, Any]]) -> dict[str, tuple"crawler": ("crawler/summary.json", "crawler/raw-runs.csv"),"amiibo-crawler": ("amiibo-crawler/summary.json", "amiibo-crawler/raw-runs.csv"),"wild-web": ("wild-web/summary.json", "wild-web/raw-runs.csv"),- "top-sites": ("top-sites/summary.json", "top-sites/raw-runs.csv"),+ "top-sites": (+ "top-sites/summary.json",+ "top-sites/raw-runs.csv",+ "top-sites/site-outcomes.json",+ ),"render-compare": ("render-compare/summary.json","render-compare/raw-runs.csv",@@ -997,11 +1001,25 @@ function targetLabel(targets, target) {function targetColor(targets, target) {return colors[targetEngine(targets, target)] || '#64748b';}-function firstHorizontalSummary() {+function horizontalSummaries() {const comparisons = report.horizontal_comparisons || [];- if (comparisons.length) return comparisons[0];const summaries = report.summaries || [];- return summaries.find(s => s.suite === 'synthetic-compare') || summaries.find(s => s.suite === 'cdp-session') || summaries.find(s => s.targets);+ const result = [...comparisons];+ const seen = new Set(result.map(summary => summary.suite));+ for (const summary of summaries) {+ if (!summary?.targets || seen.has(summary.suite)) continue;+ result.push(summary);+ seen.add(summary.suite);+ }+ return result;+}+function firstHorizontalSummary() {+ const summaries = horizontalSummaries();+ for (const suite of ['web-scraping-variants', 'synthetic-compare', 'top-sites', 'wild-web', 'cdp-session']) {+ const summary = summaries.find(candidate => candidate.suite === suite);+ if (summary) return summary;+ }+ return summaries[0];}function summaryCases(summary) {const cases = summary?.cases || [];@@ -1014,8 +1032,20 @@ function resultFor(summary, target, caseName) {if (isAggregateSummary(summary)) return summary?.targets?.[target] || {};return summary?.targets?.[target]?.cases?.[caseName] || {};}-function metricFor(summary, target, caseName, metric) {+function performanceResultFor(summary, target, caseName) {+ if (!isAggregateSummary(summary)) return resultFor(summary, target, caseName);+ const common = summary?.common_success?.targets?.[target];+ if (common && Number(common?.elapsed_ms?.count || 0) > 0) return common;const result = resultFor(summary, target, caseName);+ return {+ ...result,+ elapsed_ms: result?.successful_elapsed_ms || result?.elapsed_ms,+ peak_pss_bytes: result?.successful_peak_pss_bytes || result?.peak_pss_bytes,+ peak_rss_bytes: result?.successful_peak_rss_bytes || result?.peak_rss_bytes,+ };+}+function metricFor(summary, target, caseName, metric) {+ const result = performanceResultFor(summary, target, caseName);return result?.[metric]?.p50 ?? null;}function targetFailures(summary, target) {@@ -1033,7 +1063,7 @@ function horizontalCells(summary) {const targets = targetNames(summary?.targets || {});const cases = summaryCases(summary);return cases.flatMap(caseName => targets.map(target => {- const result = resultFor(summary, target, caseName);+ const result = performanceResultFor(summary, target, caseName);return {caseName, target, result};}));}@@ -1055,6 +1085,30 @@ function passRate(result) {if (!denominator) return '';return `${passes}/${denominator} (${(passes / denominator * 100).toFixed(1)}%)`;}+function rawPassRate(result) {+ const passes = Number(result?.raw_passes);+ const attempts = Number(result?.raw_comparable_attempts);+ if (!Number.isFinite(passes) || !Number.isFinite(attempts) || attempts <= 0) return '';+ return `${passes}/${attempts} (${(passes / attempts * 100).toFixed(1)}%)`;+}+function siteStability(result) {+ const sites = Number(result?.unique_sites);+ if (!Number.isFinite(sites) || sites <= 0) return '';+ const runs = Number(result?.runs || 0);+ if (!result?.repeat_validated) return `${sites} sites; ${runs || 1} run(s), not repeat-validated`;+ return `${Number(result?.all_pass_sites || 0)} all-pass / ${Number(result?.flaky_sites || 0)} flaky / ${Number(result?.all_fail_sites || 0)} all-fail`;+}+function statusCoverage(result) {+ const observed = Number(result?.status_observed_attempts);+ const attempts = Number(result?.observed_sites ?? result?.attempts);+ const denominator = Number.isFinite(attempts) && attempts > 0 ? attempts : Number(result?.seeds);+ if (!Number.isFinite(observed) || !Number.isFinite(denominator) || denominator <= 0) return '';+ const sources = Object.entries(result?.response_status_sources || {})+ .map(([source, count]) => `${source}: ${count}`)+ .join(', ');+ const coverage = `${observed}/${denominator} (${(observed / denominator * 100).toFixed(1)}%)`;+ return sources ? `${coverage}; ${sources}` : coverage;+}function breakdown(items) {if (!items || typeof items !== 'object') return '';return Object.entries(items).map(([key, value]) => `${htmlEscape(key)}: ${htmlEscape(value)}`).join(', ');@@ -1108,12 +1162,14 @@ function renderExecutive() {if (horizontal) {const fastest = fastestCell(horizontal);const memory = lowestMemoryCell(horizontal);+ const commonAttempts = Number(horizontal?.common_success?.attempts || 0);+ const cohortNote = commonAttempts > 0 ? ` over ${commonAttempts} common-success attempt(s)` : '';const failureTargets = targetNames(horizontal.targets || {}).filter(target => targetFailures(horizontal, target) > 0);document.getElementById('executiveSummary').innerHTML = `<div class="finding-list"><div class="finding">Horizontal source: <strong>${htmlEscape(horizontal.suite)}</strong>. Gate target: <strong>${htmlEscape(horizontal.gate_target || 'moli')}</strong>.</div>- <div class="finding">${fastest ? `Fastest p50 cell: <strong>${htmlEscape(targetLabel(horizontal.targets, fastest.target))}</strong> / ${htmlEscape(fastest.caseName)} at ${number(fastest.result.elapsed_ms.p50)} ms.` : 'No successful latency cell is available.'}</div>- <div class="finding">${memory ? `Lowest PSS cell: <strong>${htmlEscape(targetLabel(horizontal.targets, memory.target))}</strong> / ${htmlEscape(memory.caseName)} at ${mib(memory.result.peak_pss_bytes.p50)}.` : 'No successful memory cell is available.'}</div>+ <div class="finding">${fastest ? `Fastest comparable-success p50: <strong>${htmlEscape(targetLabel(horizontal.targets, fastest.target))}</strong> / ${htmlEscape(fastest.caseName)} at ${number(fastest.result.elapsed_ms.p50)} ms${cohortNote}.` : 'No comparable successful latency cell is available.'}</div>+ <div class="finding">${memory ? `Lowest comparable-success PSS: <strong>${htmlEscape(targetLabel(horizontal.targets, memory.target))}</strong> / ${htmlEscape(memory.caseName)} at ${mib(memory.result.peak_pss_bytes.p50)}${cohortNote}.` : 'No comparable successful memory cell is available.'}</div><div class="finding ${failureTargets.length ? 'warn' : ''}">${failureTargets.length ? `Targets with comparison failures: <strong>${htmlEscape(failureTargets.map(target => targetLabel(horizontal.targets, target)).join(', '))}</strong>.` : 'No comparison target failures in this suite.'}</div></div>`;@@ -1170,30 +1226,58 @@ function renderSuites() {</tr>`);document.getElementById('suiteTable').innerHTML = table(['suite', 'profile', 'gate target', 'gate failures', 'all failures', 'runs', 'timeout'], rows);}-function renderComparisonTable() {- const summary = firstHorizontalSummary();- if (!summary || !summary.targets) {- document.getElementById('comparisonTable').innerHTML = '<p class="muted">No horizontal comparison data.</p>';- return;- }+function comparisonSummaryHtml(summary) {const targets = targetNames(summary.targets);if (isAggregateSummary(summary)) {const rows = targets.map(target => {const result = summary.targets[target] || {};+ const successfulElapsed = result.successful_elapsed_ms || result.elapsed_ms || {};+ const successfulPss = result.successful_peak_pss_bytes || result.peak_pss_bytes || {};+ const common = summary?.common_success?.targets?.[target] || {};+ const commonAttempts = Number(summary?.common_success?.attempts || 0);+ const exclusions = [+ Number(result.excluded_runs || 0) ? `excluded: ${result.excluded_runs}` : '',+ Number(result.non_comparable_attempts || 0) ? `neutral: ${result.non_comparable_attempts}` : '',+ ].filter(Boolean).join('; ');return `<tr><th>${htmlEscape(targetLabel(summary.targets, target))}</th><td>${passRate(result)}</td>+ <td>${rawPassRate(result)}</td>+ <td>${htmlEscape(exclusions)}</td>+ <td>${htmlEscape(siteStability(result))}</td>+ <td>${htmlEscape(statusCoverage(result))}</td><td>${htmlEscape(result.successes ?? result.categories?.success ?? result.passes ?? '')}</td>- <td>${htmlEscape(result.categories?.challenge ?? '')}</td>- <td>${htmlEscape(result.categories?.thin ?? '')}</td><td>${htmlEscape(result.failures ?? '')}</td>- <td>${number(result.elapsed_ms?.p50)} / ${number(result.elapsed_ms?.p90)} / ${number(result.elapsed_ms?.p95)}</td>- <td>${mib(result.peak_pss_bytes?.p50)} / ${mib(result.peak_pss_bytes?.p95)}</td>+ <td>${number(successfulElapsed.p50)} / ${number(successfulElapsed.p90)} / ${number(successfulElapsed.p95)}</td>+ <td>${mib(successfulPss.p50)} / ${mib(successfulPss.p95)}</td>+ <td>${commonAttempts || ''}</td>+ <td>${number(common.elapsed_ms?.p50)} / ${number(common.elapsed_ms?.p90)} / ${number(common.elapsed_ms?.p95)}</td>+ <td>${mib(common.peak_pss_bytes?.p50)} / ${mib(common.peak_pss_bytes?.p95)}</td><td>${breakdown(result.failure_kinds)}</td></tr>`;});- document.getElementById('comparisonTable').innerHTML = table(['target', 'pass rate', 'success', 'challenge', 'thin', 'failures', 'p50 / p90 / p95 ms', 'PSS p50 / p95', 'failure kinds'], rows);- return;+ const pairwise = Array.isArray(summary.pairwise) ? summary.pairwise : [];+ const pairRows = pairwise.map(item => `<tr>+ <th>${htmlEscape(targetLabel(summary.targets, item.left))} vs ${htmlEscape(targetLabel(summary.targets, item.right))}</th>+ <td>${badge(item.repeat_validated ? 'REPEAT VALIDATED' : 'SINGLE SAMPLE')}</td>+ <td>${htmlEscape(item.both_all_pass ?? '')}</td>+ <td>${htmlEscape(item.left_all_pass_right_all_fail ?? '')}</td>+ <td>${htmlEscape(item.right_all_pass_left_all_fail ?? '')}</td>+ <td>${htmlEscape(item.both_all_fail ?? '')}</td>+ <td>${htmlEscape(item.flaky_or_incomplete ?? '')}</td>+ <td>${htmlEscape(item.not_comparable ?? '')}</td>+ <td>${htmlEscape(item.common_success?.attempts ?? '')}</td>+ </tr>`);+ const pairTable = pairRows.length+ ? `<h4>Pairwise site outcomes</h4>${table(['targets', 'validation', 'both pass', 'left only', 'right only', 'both fail', 'flaky / incomplete', 'not comparable', 'common-success attempts'], pairRows)}`+ : '';+ const contract = summary.snapshot_contract+ ? `<p class="muted">Snapshot contract: ${htmlEscape(summary.snapshot_contract)}</p>`+ : '';+ return contract + table(+ ['target', 'counted pass rate', 'raw pass rate', 'excluded / neutral', 'site stability', 'HTTP status observed', 'success', 'failures', 'successful p50 / p90 / p95 ms', 'successful PSS p50 / p95', 'common n', 'common p50 / p90 / p95 ms', 'common PSS p50 / p95', 'failure kinds'],+ rows,+ ) + pairTable;}const cases = summaryCases(summary);const rows = cases.map(caseName => {@@ -1207,7 +1291,20 @@ function renderComparisonTable() {}).join('');return `<tr><th>${htmlEscape(caseName)}</th>${cells}</tr>`;});- document.getElementById('comparisonTable').innerHTML = table(['case', ...targets.map(target => targetLabel(summary.targets, target))], rows);+ return table(['case', ...targets.map(target => targetLabel(summary.targets, target))], rows);+}+function renderComparisonTable() {+ const summaries = horizontalSummaries();+ if (!summaries.length) {+ document.getElementById('comparisonTable').innerHTML = '<p class="muted">No horizontal comparison data.</p>';+ return;+ }+ document.getElementById('comparisonTable').innerHTML = summaries.map(summary => `+ <section class="comparison-summary">+ <h3>${htmlEscape(summary.suite || 'comparison')}</h3>+ ${comparisonSummaryHtml(summary)}+ </section>+ `).join('');}function renderWebPages() {const section = document.getElementById('web-pages');diff --git a/moli-benchmark/moli_benchmark/process.py b/moli-benchmark/moli_benchmark/process.pyindex 59c4b9f5f..b26156112 100644--- a/moli-benchmark/moli_benchmark/process.py+++ b/moli-benchmark/moli_benchmark/process.py@@ -22,6 +22,10 @@ class ProcessResult:resources: dict[str, Any]time_verbose: dict[str, Any] | None = Nonecgroup: dict[str, Any] | None = None+ response_status: int | None = None+ response_mime_type: str | None = None+ main_document_body_capture: str | None = None+ final_url: str | None = Nonedef output_digest_material(self) -> bytes:return self.stdout + b"\n--- stderr ---\n" + self.stderr@@ -40,6 +44,14 @@ class ProcessResult:summary["time_verbose"] = self.time_verboseif self.cgroup is not None:summary["cgroup"] = self.cgroup+ if self.response_status is not None:+ summary["response_status"] = self.response_status+ if self.response_mime_type is not None:+ summary["response_mime_type"] = self.response_mime_type+ if self.main_document_body_capture is not None:+ summary["main_document_body_capture"] = self.main_document_body_capture+ if self.final_url is not None:+ summary["final_url"] = self.final_urlif include_output:summary["stdout_tail"] = self.stdout[-4096:].decode("utf-8", errors="replace")summary["stderr_tail"] = self.stderr[-4096:].decode("utf-8", errors="replace")diff --git a/moli-benchmark/moli_benchmark/public_web.py b/moli-benchmark/moli_benchmark/public_web.pynew file mode 100644index 000000000..1569ff29b--- /dev/null+++ b/moli-benchmark/moli_benchmark/public_web.py@@ -0,0 +1,1306 @@+from __future__ import annotations++import html.parser+import re+from collections.abc import Callable, Mapping, Sequence+from concurrent.futures import ThreadPoolExecutor, as_completed+from dataclasses import dataclass+from datetime import datetime, timezone+from hashlib import sha256+from pathlib import Path+from threading import BoundedSemaphore+from typing import Any, Generic, TypeVar++from .artifacts import write_json, write_text+from .config import REPO_ROOT+from .process import ProcessResult+from .stats import summarize+from .synthetic_compare import (+ target_enables_all_resource_fetch,+ target_is_cdp,+ target_metadata,+)+++POST_DCL_SETTLE_MILLISECONDS = 50+PUBLIC_WEB_SNAPSHOT_CONTRACT = (+ "at least "+ f"{POST_DCL_SETTLE_MILLISECONDS} ms of page event-loop time after "+ "the adapter observes DOMContentLoaded"+)+POST_DCL_WAIT_SCRIPT = (+ "globalThis.__moliBenchmarkPostDclReady === true || "+ "(setTimeout(() => { globalThis.__moliBenchmarkPostDclReady = true; }, "+ f"{POST_DCL_SETTLE_MILLISECONDS}), false)"+)++_MAIN_DOCUMENT_STATUS_DIAGNOSTIC = re.compile(+ rb"(?:lifecycle\s+target\s+document|http\s+request)\s+"+ rb"`[^`\r\n]+`\s+returned\s+([45]\d{2})\b",+ re.IGNORECASE,+)+_LIGHTPANDA_FETCH_TIMEOUT_DIAGNOSTIC = re.compile(+ rb'\$msg="fetch error"[^\r\n]*\berr=timeout\b',+ re.IGNORECASE,+)+_SAFE_ARTIFACT_TOKEN = re.compile(r"[^A-Za-z0-9._-]+")+_TITLE_RE = re.compile(r"<title[^>]*>(.*?)</title>", re.IGNORECASE | re.DOTALL)+_TOP_SITES_BLOCKED_TITLE_403 = re.compile(r"^\s*(?:http\s*)?403\b")+_TOP_SITES_HTTP_ERROR_TITLE_STATUS = re.compile(+ r"^\s*(?:"+ r"error\s*:\s*([45]\d{2})\b"+ r"|(?:http\s*)?([45]\d{2})\s+(?:"+ r"bad\s+gateway|gateway\s+timeout|internal\s+server\s+error|"+ r"service\s+unavailable|forbidden|not\s+found|unauthorized|"+ r"too\s+many\s+requests|request\s+timeout|error"+ r")\b"+ r")",+ re.IGNORECASE,+)+_WILD_WEB_HTTP_ERROR_TITLE_STATUS = re.compile(+ r"^\s*(?:"+ r"error\s*:\s*([45]\d{2})\b"+ r"|(?:http\s*)?([45]\d{2})\s+(?:"+ r"bad\s+gateway|gateway\s+timeout|internal\s+server\s+error|"+ r"service\s+unavailable|forbidden|not\s+found|unauthorized|error"+ r")\b"+ r")",+ re.IGNORECASE,+)++_TOP_SITES_NETWORK_ERROR_MARKERS = (+ "privacy error",+ "your connection is not private",+ "net::err_cert",+ "this site can't be reached",+ "this site can’t be reached",+ "err_name_not_resolved",+ "err_connection",+ "err_timed_out",+ "dns_probe_finished",+ "could not resolve",+ "could not resolve host",+ "could not resolve hostname",+ "failed to resolve",+ "failed to lookup address information",+ "could not connect to server",+ "name resolution",+ "connection refused",+ "connection reset",+ "connection timed out",+ "no route to host",+ "network is unreachable",+ "ssl handshake",+ "ssl error",+ "ssl connect error",+ "tls handshake",+ "peerfailedverification",+ "peer failed verification",+ "timed out connecting",+ "timeout was reached",+ "request timeout",+ "i/o error",+ "broken pipe",+ "operation timed out",+ "failed to connect",+ "dns lookup",+ "curl request failed",+ "recv failure",+)+_WILD_WEB_NETWORK_ERROR_MARKERS = (+ "peerfailedverification",+ "peer failed verification",+ "privacy error",+ "your connection is not private",+ "this site can't be reached",+ "this site can’t be reached",+ "net::err_",+ "could not resolve",+ "failed to resolve",+ "failed to lookup address information",+ "connection refused",+ "curl request failed",+ "couldntresolvehost",+ "http2stream",+)+_NAVIGATION_NETWORK_FAILURE_MARKERS = (+ "couldntresolvehost",+ "http2stream",+ "reason: http2",+)+_CLI_DEADLINE_COMMAND_MARKERS = (+ "fetch_document_allow_http_error_with_wait_until",+ "fetch document allow-http-error",+ "fetch allow-http-error wait_until",+ "fetch wait_until",+)+_CLI_DEADLINE_TIMEOUT_MARKERS = ("timed out after",)+_CLI_FETCH_READINESS_TIMEOUT_MARKER = "fetch readiness timed out after"+_ELAPSED_FAILURE_TIMEOUT_GRACE_SECONDS = 1.0+_CAPTCHA_MARKERS = (+ "captcha",+ "安全验证",+ "验证你是真人",+ "人机验证",+ "human verification",+ "verification",+ "verify you are human",+ "verify that you're not a robot",+ "verify that you are not a robot",+ "are you a robot",+ "not a robot",+ "bot or not",+ "checking your browser",+ "perform security check",+ "security verification",+ "performing security verification",+ "请完成验证",+ "完成验证后",+ "当前环境异常",+ "select the 2 matching tiles",+ "powered and protected by privacy",+)+_LOGIN_MARKERS = ("sign in", "log in", "login", "登录", "注册", "账号")+_LOGIN_FORM_MARKERS = (+ "login to your account",+ "email/username",+ "your password is a required field",+ "forgot password",+ "create a free account",+)+_LOGIN_CONTEXT_MARKERS = (+ "password",+ "验证码",+ "短信验证码",+ "语音验证码",+ "手机号",+ "手机验证",+ "邮箱",+ "email",+)+_JS_CHALLENGE_MARKERS = (+ "__cf_chl_",+ "c2wf946j0/probe",+ "cf-challenge",+ "__tencent_chaos_vm",+ "__eo_jschallenge_vm",+ "teojschallengesdk.js",+ "eojschallengesdk",+ "window.solvechallenge(",+ "jsl_clearance",+ "acw_sc__v2",+ "window._phantom",+ "_$jsvmprt",+ "awswaf",+ "aliyun_waf",+ "aliyun waf",+ "_waf_",+ "just a moment...",+ "vercel security checkpoint",+ "we're verifying your browser",+ "we’re verifying your browser",+ "enable javascript and cookies to continue",+ "javascript is needed to access this site",+)+_BLOCKED_BODY_MARKERS = (+ "unable to give you access to our site",+ "access denied",+ "access to this page has been denied",+ "sorry, you have been blocked",+ "security issue was automatically identified",+ "access restricted",+ "exception: forbidden",+ "waf拦截",+ "被waf拦截",+ "security detection powered by safeline waf",+)+_NOT_FOUND_MARKERS = (+ "404 not found",+ "file not found",+ "page not found",+ "this page cannot be found",+ "page you requested is missing",+ "page you requested could not be found",+ "页面不存在",+ "视频不见了",+)+_FORBIDDEN_ERROR_MARKERS = ("403 forbidden", "returned 403", "http 403")+_NOT_FOUND_ERROR_MARKERS = ("returned 404", "http 404")+++def reports_lightpanda_fetch_timeout(stderr: bytes) -> bool:+ """Return whether Lightpanda reported its terminal fetch-timeout record."""++ return _LIGHTPANDA_FETCH_TIMEOUT_DIAGNOSTIC.search(stderr) is not None+++def reports_cli_fetch_timeout(stderr: bytes, returncode: int | None) -> bool:+ """Recognize stable typed and legacy Moli fetch-deadline diagnostics."""++ if returncode in (None, 0):+ return False+ diagnostic = stderr.decode("utf-8", errors="replace").lower()+ if _CLI_FETCH_READINESS_TIMEOUT_MARKER in diagnostic:+ return True+ return any(marker in diagnostic for marker in _CLI_DEADLINE_COMMAND_MARKERS) and any(+ marker in diagnostic for marker in _CLI_DEADLINE_TIMEOUT_MARKERS+ )+++def resolve_main_document_status(+ protocol_status: int | None,+ stderr: bytes,+ returncode: int | None,+) -> tuple[int | None, str | None]:+ """Return the main-document status and the evidence that supplied it.++ CDP is the preferred source. Moli's fetch CLI also includes the terminal+ main-document status in its non-zero-exit diagnostic, so preserve that+ structured evidence instead of reducing it to a generic process error.+ The diagnostic matcher is deliberately narrow to avoid treating a logged+ subresource failure as the main navigation status.+ """++ if protocol_status is not None:+ return protocol_status, "protocol"+ if returncode in {None, 0}:+ return None, None+ matches = _MAIN_DOCUMENT_STATUS_DIAGNOSTIC.findall(stderr)+ if not matches:+ return None, None+ return int(matches[-1]), "cli-diagnostic"+++def utc_timestamp() -> str:+ return datetime.now(timezone.utc).isoformat().replace("+00:00", "Z")+++def rotated_target_order(+ targets: tuple[str, ...],+ schedule_index: int,+) -> tuple[str, ...]:+ if not targets:+ return ()+ offset = schedule_index % len(targets)+ return (*targets[offset:], *targets[:offset])+++@dataclass(frozen=True)+class PublicWebSnapshotProfile:+ hidden_tags: frozenset[str]+ preserve_title_entities: bool = False+ sample_chars: int = 500+++TOP_SITES_SNAPSHOT_PROFILE = PublicWebSnapshotProfile(+ hidden_tags=frozenset(+ {"head", "script", "style", "noscript", "svg", "template"}+ ),+ preserve_title_entities=True,+)+WILD_WEB_SNAPSHOT_PROFILE = PublicWebSnapshotProfile(+ hidden_tags=frozenset({"script", "style", "noscript"}),+)+++@dataclass(frozen=True)+class PublicWebArtifactPolicy:+ output_sample_bytes: int+ failure_stdout_limit_bytes: int+ failure_stderr_limit_bytes: int+ write_empty_failure_stdout: bool+++TOP_SITES_ARTIFACT_POLICY = PublicWebArtifactPolicy(+ output_sample_bytes=512,+ failure_stdout_limit_bytes=32 * 1024,+ failure_stderr_limit_bytes=16 * 1024,+ write_empty_failure_stdout=False,+)+WILD_WEB_ARTIFACT_POLICY = PublicWebArtifactPolicy(+ output_sample_bytes=1024,+ failure_stdout_limit_bytes=128 * 1024,+ failure_stderr_limit_bytes=32 * 1024,+ write_empty_failure_stdout=True,+)+++class _PublicWebTextExtractor(html.parser.HTMLParser):+ def __init__(self, hidden_tags: frozenset[str]) -> None:+ super().__init__(convert_charrefs=True)+ self._hidden_tags = hidden_tags+ self._hidden_depth = 0+ self._in_title = False+ self.title_parts: list[str] = []+ self.text_parts: list[str] = []++ def handle_starttag(+ self,+ tag: str,+ attrs: list[tuple[str, str | None]],+ ) -> None:+ del attrs+ normalized = tag.lower()+ if normalized in self._hidden_tags:+ self._hidden_depth += 1+ if normalized == "title":+ self._in_title = True++ def handle_endtag(self, tag: str) -> None:+ normalized = tag.lower()+ if normalized == "title":+ self._in_title = False+ if normalized in self._hidden_tags and self._hidden_depth:+ self._hidden_depth -= 1++ def handle_data(self, data: str) -> None:+ text = " ".join(data.split())+ if not text:+ return+ if self._in_title:+ self.title_parts.append(text)+ if self._hidden_depth == 0 and not self._in_title:+ self.text_parts.append(text)+++def extract_public_web_snapshot(+ stdout: bytes,+ *,+ profile: PublicWebSnapshotProfile,+) -> dict[str, Any]:+ html_text = stdout.decode("utf-8", errors="replace")+ parser = _PublicWebTextExtractor(profile.hidden_tags)+ try:+ parser.feed(html_text)+ except html.parser.HTMLParseError:+ pass+ title = " ".join(parser.title_parts).strip()+ if profile.preserve_title_entities:+ match = _TITLE_RE.search(html_text)+ title = re.sub(r"\s+", " ", match.group(1)).strip() if match else ""+ visible_text = re.sub(r"\s+", " ", " ".join(parser.text_parts)).strip()+ return {+ "title": title,+ "text_length": len(visible_text),+ "text_sample": visible_text[: profile.sample_chars],+ }+++def _looks_like_binary_content(body: bytes) -> bool:+ sample = body[:4096]+ if sample.startswith(b"%PDF-") or b"\x00" in sample:+ return True+ decoded = sample.decode("utf-8", errors="replace")+ return decoded.count("\ufffd") > max(8, len(decoded) // 20)+++def _looks_like_navigation_network_failure(text: str) -> bool:+ return (+ "navigate failed" in text or "navigation failed" in text+ ) and any(marker in text for marker in _NAVIGATION_NETWORK_FAILURE_MARKERS)+++def _looks_like_login_wall(+ title: str,+ sample: str,+ text_length: int,+) -> bool:+ page_text = f"{title}\n{sample}"+ if sum(marker in page_text for marker in _LOGIN_FORM_MARKERS) >= 2:+ return True+ if text_length >= 800:+ return False+ login_hits = {marker for marker in _LOGIN_MARKERS if marker in page_text}+ if not login_hits:+ return False+ title_has_login = any(marker in title for marker in _LOGIN_MARKERS)+ if title_has_login and len(login_hits) >= 2:+ return True+ return len(login_hits) >= 2 and any(+ marker in page_text for marker in _LOGIN_CONTEXT_MARKERS+ )+++def _top_sites_http_error_category(status: int) -> str:+ if status in {401, 403}:+ return "blocked-or-forbidden"+ if status == 404:+ return "not-found"+ return "http-error"+++def _wild_web_http_error_category(status: int) -> str:+ if status in {401, 403}:+ return "blocked"+ if status == 404:+ return "not-found"+ return "http-error"+++@dataclass(frozen=True)+class PublicWebClassifier:+ """Classify normalized public-web evidence with a suite policy.++ Transport/status handling and evidence precedence live here once. The+ two policies retain their established category vocabulary because those+ values are part of each suite's artifact schema.+ """++ policy: str+ snapshot_profile: PublicWebSnapshotProfile+ ok_categories: frozenset[str]+ non_comparable_categories: frozenset[str]+ failure_kind_aliases: Mapping[str, str]+ use_category_as_failure_kind: bool++ def classification_ok(self, category: str) -> bool:+ return category in self.ok_categories++ def comparable(self, category: str) -> bool:+ return category not in self.non_comparable_categories++ def failure_kind(+ self,+ category: str,+ *,+ error: str | None = None,+ extraction_failed: bool = False,+ ) -> str | None:+ if error == "target binary unavailable":+ return "target-unavailable"+ alias = self.failure_kind_aliases.get(category)+ if alias is not None:+ return alias+ if self.use_category_as_failure_kind and not self.classification_ok(+ category+ ):+ return category+ if extraction_failed:+ return "extraction-failure"+ return None++ def snapshot(self, stdout: bytes) -> dict[str, Any]:+ return extract_public_web_snapshot(+ stdout,+ profile=self.snapshot_profile,+ )++ def classify_output(+ self,+ *,+ stdout: bytes,+ stderr: bytes,+ returncode: int | None,+ timed_out: bool,+ min_body_bytes: int = 1,+ snapshot: Mapping[str, Any] | None = None,+ response_status: int | None = None,+ main_document_body_capture: str | None = None,+ ) -> str:+ if timed_out:+ return "timeout"+ response_status, _ = resolve_main_document_status(+ response_status,+ stderr,+ returncode,+ )+ if response_status is not None and response_status >= 400:+ return self._http_error_category(response_status)+ if (+ self.policy == "top-sites"+ and main_document_body_capture == "response-headers-only"+ ):+ return "binary-response-headers"+ if reports_lightpanda_fetch_timeout(stderr):+ return "timeout"+ if reports_cli_fetch_timeout(stderr, returncode):+ return "timeout"+ if self.policy == "top-sites":+ return self._classify_top_sites_document(+ stdout=stdout,+ stderr=stderr,+ returncode=returncode,+ min_body_bytes=min_body_bytes,+ snapshot=snapshot,+ )+ if self.policy == "wild-web":+ return self._classify_wild_web_document(+ stdout=stdout,+ stderr=stderr,+ returncode=returncode,+ snapshot=snapshot,+ )+ raise ValueError(f"unknown public-web classifier policy: {self.policy}")++ def classify_result(+ self,+ result: PublicWebResult,+ *,+ min_body_bytes: int = 1,+ ) -> str:+ process = result.process+ return self.classify_output(+ stdout=process.stdout,+ stderr=process.stderr,+ returncode=process.returncode,+ timed_out=process.timed_out,+ min_body_bytes=min_body_bytes,+ snapshot=result.snapshot,+ response_status=result.response_status,+ main_document_body_capture=process.main_document_body_capture,+ )++ def classification_basis(+ self,+ result: PublicWebResult,+ category: str,+ ) -> str:+ return self.classification_basis_for_process(+ result.process,+ category,+ response_status=result.response_status,+ response_status_source=result.response_status_source,+ )++ def classification_basis_for_process(+ self,+ process: ProcessResult,+ category: str,+ *,+ response_status: int | None,+ response_status_source: str | None,+ ) -> str:+ if process.timed_out or category == "timeout":+ return "timeout"+ if response_status is not None and response_status >= 400:+ source = response_status_source or "unknown"+ return f"{source}-http-status"+ if category == "binary-response-headers":+ return "protocol-binary-response-headers"+ if process.returncode not in {None, 0}:+ return "process-exit"+ if category == "success-binary-content":+ return "binary-signature"+ visible_categories = (+ {"success-content", "app-shell-only", "empty-response"}+ if self.policy == "top-sites"+ else {"success", "empty"}+ )+ if category in visible_categories:+ return "visible-content"+ return "document-marker"++ def _http_error_category(self, status: int) -> str:+ if self.policy == "top-sites":+ return _top_sites_http_error_category(status)+ if self.policy == "wild-web":+ return _wild_web_http_error_category(status)+ raise ValueError(f"unknown public-web classifier policy: {self.policy}")++ def _classify_top_sites_document(+ self,+ *,+ stdout: bytes,+ stderr: bytes,+ returncode: int | None,+ min_body_bytes: int,+ snapshot: Mapping[str, Any] | None,+ ) -> str:+ combined_text = (stdout + b"\n" + stderr).decode(+ "utf-8", errors="replace"+ ).lower()+ if "operationtimedout" in combined_text and (+ "navigate failed" in combined_text+ or "navigation failed" in combined_text+ ):+ return "timeout"+ if _looks_like_navigation_network_failure(combined_text):+ return "network-error"+ if returncode != 0:+ if any(+ marker in combined_text+ for marker in _FORBIDDEN_ERROR_MARKERS+ ):+ return "blocked-or-forbidden"+ if any(+ marker in combined_text for marker in _NOT_FOUND_ERROR_MARKERS+ ):+ return "not-found"+ if any(+ marker in combined_text+ for marker in _TOP_SITES_NETWORK_ERROR_MARKERS+ ):+ return "network-error"+ return "process-error"++ body = stdout.strip()+ if not body:+ if any(+ marker in combined_text+ for marker in _TOP_SITES_NETWORK_ERROR_MARKERS+ ):+ return "network-error"+ return "empty-response"+ if len(body) >= min_body_bytes and _looks_like_binary_content(body):+ return "success-binary-content"++ page_snapshot = snapshot or self.snapshot(stdout)+ text_length = int(page_snapshot.get("text_length") or 0)+ title = str(page_snapshot.get("title") or "").lower()+ sample = str(page_snapshot.get("text_sample") or "").lower()+ page_text = f"{title}\n{sample}"+ if any(+ marker in page_text for marker in _TOP_SITES_NETWORK_ERROR_MARKERS+ ):+ return "network-error"+ inferred_error = self._inferred_top_sites_http_error(title, sample)+ if inferred_error is not None:+ return inferred_error+ if (+ "blocked" in title+ or "forbidden" in title+ or _TOP_SITES_BLOCKED_TITLE_403.search(title)+ or "access restricted" in title+ ):+ return "blocked-or-forbidden"+ if any(marker in page_text for marker in _NOT_FOUND_MARKERS):+ return "not-found"+ if any(marker in page_text for marker in _BLOCKED_BODY_MARKERS):+ return "blocked-or-forbidden"+ if any(marker in page_text for marker in _CAPTCHA_MARKERS):+ return "captcha-or-verification"+ if any(marker in combined_text for marker in _JS_CHALLENGE_MARKERS):+ return "js-challenge"+ if _looks_like_login_wall(title, sample, text_length):+ return "login-wall"+ if len(body) < min_body_bytes or text_length < min_body_bytes:+ return "app-shell-only"+ return "success-content"++ def _classify_wild_web_document(+ self,+ *,+ stdout: bytes,+ stderr: bytes,+ returncode: int | None,+ snapshot: Mapping[str, Any] | None,+ ) -> str:+ diagnostic_text = stderr.decode("utf-8", errors="replace").lower()+ if returncode != 0:+ if any(+ marker in diagnostic_text+ for marker in _WILD_WEB_NETWORK_ERROR_MARKERS+ ):+ return "network-error"+ return "error"++ page_snapshot = snapshot or self.snapshot(stdout)+ visible_text = " ".join(+ str(page_snapshot.get(key) or "")+ for key in ("title", "text_sample")+ ).lower()+ text = f"{visible_text}\n{diagnostic_text}"+ if any(marker in text for marker in _WILD_WEB_NETWORK_ERROR_MARKERS):+ return "network-error"+ title = str(page_snapshot.get("title") or "")+ status_match = _WILD_WEB_HTTP_ERROR_TITLE_STATUS.match(title)+ if status_match is not None:+ status = int(status_match.group(1) or status_match.group(2))+ return _wild_web_http_error_category(status)+ sample = str(page_snapshot.get("text_sample") or "")+ if sample.lstrip().startswith(('{"error":{', '{"errors":[')):+ return "http-error"+ if "页面不存在" in text or "视频不见了" in text:+ return "not-found"+ if "security detection powered by safeline waf" in text:+ return "blocked"+ if (+ "captcha" in text+ or "verify" in text+ or "challenge" in text+ or "安全验证" in text+ or "完成验证后" in text+ or "当前环境异常" in text+ or "select the 2 matching tiles" in text+ or "powered and protected by privacy" in text+ ):+ return "challenge"+ if "login" in text or "登录" in text:+ return "login"+ if (+ "blocked" in text+ or "forbidden" in text+ or "403" in text+ or "access restricted" in text+ ):+ return "blocked"+ return "success" if stdout.strip() else "empty"++ @staticmethod+ def _inferred_top_sites_http_error(+ title: str,+ sample: str,+ ) -> str | None:+ match = _TOP_SITES_HTTP_ERROR_TITLE_STATUS.match(title)+ if match is not None:+ status_text = match.group(1) or match.group(2)+ if status_text is not None:+ return _top_sites_http_error_category(int(status_text))+ if sample.lstrip().startswith(('{"error":{', '{"errors":[')):+ return "http-error"+ return None+++TOP_SITES_CLASSIFIER = PublicWebClassifier(+ policy="top-sites",+ snapshot_profile=TOP_SITES_SNAPSHOT_PROFILE,+ ok_categories=frozenset({"success-content", "success-binary-content"}),+ non_comparable_categories=frozenset({"binary-response-headers"}),+ failure_kind_aliases={},+ use_category_as_failure_kind=True,+)+WILD_WEB_CLASSIFIER = PublicWebClassifier(+ policy="wild-web",+ snapshot_profile=WILD_WEB_SNAPSHOT_PROFILE,+ ok_categories=frozenset({"success", "login", "challenge"}),+ non_comparable_categories=frozenset(),+ failure_kind_aliases={+ "timeout": "timeout",+ "blocked": "blocked",+ "challenge": "challenge",+ "login": "login",+ "empty": "empty-response",+ "error": "process-error",+ "network-error": "network-error",+ "not-found": "not-found",+ "http-error": "http-error",+ },+ use_category_as_failure_kind=False,+)+++def elapsed_failure_reached_timeout(+ elapsed_ms: float | None,+ timeout_seconds: float,+) -> bool:+ if elapsed_ms is None or timeout_seconds <= 0:+ return False+ grace_seconds = min(+ _ELAPSED_FAILURE_TIMEOUT_GRACE_SECONDS,+ timeout_seconds * 0.05,+ )+ timeout_floor_ms = max(+ 0.0,+ (timeout_seconds - grace_seconds) * 1000.0,+ )+ return elapsed_ms >= timeout_floor_ms+++@dataclass(frozen=True)+class PublicWebAttempt:+ """One target's execution of one scheduled public-web case."""++ target: str+ metadata: Mapping[str, str]+ target_info: Mapping[str, Any]+ run: int+ case_fields: Mapping[str, Any]+ url: str+ schedule_index: int+ target_order_index: int+ artifact_stem: str+ started_at: str++ @classmethod+ def start(+ cls,+ *,+ target: str,+ metadata: Mapping[str, str],+ target_info: Mapping[str, Any],+ run: int,+ case_fields: Mapping[str, Any],+ url: str,+ schedule_index: int,+ target_order_index: int,+ artifact_stem: str,+ ) -> PublicWebAttempt:+ return cls(+ target=target,+ metadata=dict(metadata),+ target_info=dict(target_info),+ run=run,+ case_fields=dict(case_fields),+ url=url,+ schedule_index=schedule_index,+ target_order_index=target_order_index,+ artifact_stem=artifact_stem,+ started_at=utc_timestamp(),+ )++ def binary_path(self) -> Path | None:+ path = self.target_info.get("path")+ if not self.target_info.get("available") or not path:+ return None+ return Path(str(path))++ def base_row(self, *, finished_at: str | None = None) -> dict[str, Any]:+ return {+ "target": self.target,+ **dict(self.metadata),+ "run": self.run,+ **dict(self.case_fields),+ "url": self.url,+ "schedule_index": self.schedule_index,+ "target_order_index": self.target_order_index,+ "started_at": self.started_at,+ "finished_at": finished_at or utc_timestamp(),+ }+++@dataclass(frozen=True)+class PublicWebResult:+ """Normalized process and response evidence for a public-web attempt."""++ attempt: PublicWebAttempt+ process: ProcessResult+ response_status: int | None+ response_status_source: str | None+ snapshot: dict[str, Any]+ finished_at: str++ @classmethod+ def capture(+ cls,+ attempt: PublicWebAttempt,+ process: ProcessResult,+ *,+ classifier: PublicWebClassifier,+ ) -> PublicWebResult:+ response_status, response_status_source = resolve_main_document_status(+ process.response_status,+ process.stderr,+ process.returncode,+ )+ return cls(+ attempt=attempt,+ process=process,+ response_status=response_status,+ response_status_source=response_status_source,+ snapshot=classifier.snapshot(process.stdout),+ finished_at=utc_timestamp(),+ )++ def base_row(self) -> dict[str, Any]:+ return self.attempt.base_row(finished_at=self.finished_at)++ def evidence_fields(+ self,+ *,+ policy: PublicWebArtifactPolicy,+ ) -> dict[str, Any]:+ process = self.process+ sample_bytes = policy.output_sample_bytes+ return {+ "final_url": process.final_url,+ "elapsed_ms": process.elapsed_ms,+ "returncode": process.returncode,+ "timed_out": process.timed_out,+ "response_status": self.response_status,+ "response_status_source": self.response_status_source,+ "response_mime_type": process.response_mime_type,+ "main_document_body_capture": process.main_document_body_capture,+ "stdout_bytes": len(process.stdout),+ "stderr_bytes": len(process.stderr),+ "stdout_sha256": sha256(process.stdout).hexdigest(),+ "stderr_sha256": sha256(process.stderr).hexdigest(),+ "title": self.snapshot["title"],+ "text_length": self.snapshot["text_length"],+ "text_sample": self.snapshot["text_sample"],+ "stdout_head": process.stdout[:sample_bytes].decode(+ "utf-8", errors="replace"+ ),+ "stdout_tail": process.stdout[-sample_bytes:].decode(+ "utf-8", errors="replace"+ ),+ "stderr_tail": process.stderr[-sample_bytes:].decode(+ "utf-8", errors="replace"+ ),+ "peak_pss_bytes": process.resources.get("peak_pss_bytes"),+ "peak_rss_bytes": process.resources.get("peak_rss_bytes"),+ }+++def unavailable_public_web_row(+ attempt: PublicWebAttempt,+ *,+ category: str,+ extra_fields: Mapping[str, Any] | None = None,+) -> dict[str, Any]:+ return {+ **attempt.base_row(),+ "category": category,+ "ok": False,+ "failure_kind": "target-unavailable",+ "error": "target binary unavailable",+ **dict(extra_fields or {}),+ }+++def build_public_web_fetch_command(+ target: str,+ binary: Path,+ url: str,+ timeout_seconds: float,+ *,+ suite_name: str,+ omit_moli_page_wait: bool = False,+) -> list[str]:+ if target_is_cdp(target):+ raise RuntimeError(f"{target} is a CDP target; use the cdp-session suite")+ if target == "chrome":+ raise RuntimeError(f"chrome {suite_name} uses the CDP DCL runner")+ timeout_ms = str(int(timeout_seconds * 1000))+ if target in {"moli", "moli-full"}:+ compatibility_args = (+ ["--layout", "--resource"]+ if target_enables_all_resource_fetch(target)+ else []+ )+ page_wait_args = (+ []+ if omit_moli_page_wait+ else [+ "--wait-until",+ "domcontentloaded",+ "--wait-script",+ POST_DCL_WAIT_SCRIPT,+ ]+ )+ return [+ str(binary),+ "fetch",+ *compatibility_args,+ "--dump",+ "html",+ *page_wait_args,+ "--timeout",+ timeout_ms,+ "--http-timeout",+ timeout_ms,+ url,+ ]+ if target == "lightpanda":+ return [+ str(binary),+ "fetch",+ "--dump",+ "html",+ "--wait-until",+ "domcontentloaded",+ "--wait-script",+ POST_DCL_WAIT_SCRIPT,+ "--wait-ms",+ timeout_ms,+ "--http-timeout",+ timeout_ms,+ "--terminate-ms",+ timeout_ms,+ url,+ ]+ if target == "obscura":+ return [+ str(binary),+ "fetch",+ "--dump",+ "html",+ "--wait-until",+ "load",+ "--wait",+ "0",+ "--timeout",+ str(max(1, int(timeout_seconds))),+ url,+ ]+ raise RuntimeError(f"unknown target: {target}")+++def public_web_target_metadata(target: str) -> dict[str, str]:+ metadata = dict(target_metadata(target))+ if target == "chrome" or target_is_cdp(target):+ metadata["driver"] = "cdp-dcl"+ prefix = "moli full" if target == "moli-full-cdp" else metadata["engine"]+ metadata["label"] = f"{prefix} / cdp-dcl"+ return metadata+++def run_public_web_target(+ *,+ target: str,+ binary: Path,+ url: str,+ timeout_seconds: float,+ proc_env: dict[str, str],+ command_builder: Callable[[str, Path, str, float], list[str]],+ chrome_runner: Callable[..., ProcessResult],+ served_cdp_runner: Callable[..., ProcessResult],+ process_runner: Callable[..., ProcessResult],+) -> ProcessResult:+ if target == "chrome":+ return chrome_runner(+ binary,+ url,+ cwd=REPO_ROOT,+ timeout_seconds=timeout_seconds,+ env=proc_env,+ )+ if target_is_cdp(target):+ return served_cdp_runner(+ target,+ binary,+ url,+ cwd=REPO_ROOT,+ timeout_seconds=timeout_seconds,+ env=proc_env,+ )+ return process_runner(+ command_builder(target, binary, url, timeout_seconds),+ cwd=REPO_ROOT,+ timeout_seconds=timeout_seconds + 2,+ env=proc_env,+ )+++CaseT = TypeVar("CaseT")+ExecutionT = TypeVar("ExecutionT")+++@dataclass(frozen=True)+class PublicWebScheduledCase(Generic[CaseT]):+ schedule_index: int+ case_index: int+ run: int+ case: CaseT++ @property+ def target_rotation_index(self) -> int:+ return self.case_index + self.run - 1+++def schedule_public_web_cases(+ cases: Sequence[CaseT],+ *,+ runs: int,+) -> list[PublicWebScheduledCase[CaseT]]:+ return [+ PublicWebScheduledCase(+ schedule_index=schedule_index,+ case_index=case_index,+ run=run,+ case=case,+ )+ for schedule_index, (run, case_index, case) in enumerate(+ (run, case_index, case)+ for run in range(1, runs + 1)+ for case_index, case in enumerate(cases)+ )+ ]+++class PublicWebScheduler(Generic[CaseT, ExecutionT]):+ """Run target attempts sequentially per case and cases concurrently."""++ def __init__(+ self,+ targets: tuple[str, ...],+ *,+ parallelism: int = 1,+ target_parallelism: Mapping[str, int] | None = None,+ ) -> None:+ if parallelism <= 0:+ raise ValueError("public-web parallelism must be positive")+ limits = dict(target_parallelism or {})+ if any(limit <= 0 for limit in limits.values()):+ raise ValueError("public-web target parallelism must be positive")+ self._targets = targets+ self._parallelism = parallelism+ self._target_slots = {+ target: BoundedSemaphore(limit) for target, limit in limits.items()+ }++ def run(+ self,+ scheduled_cases: Sequence[PublicWebScheduledCase[CaseT]],+ execute: Callable[[PublicWebScheduledCase[CaseT], str, int], ExecutionT],+ ) -> list[ExecutionT]:+ def execute_group(+ scheduled: PublicWebScheduledCase[CaseT],+ ) -> list[ExecutionT]:+ results: list[ExecutionT] = []+ target_order = rotated_target_order(+ self._targets,+ scheduled.target_rotation_index,+ )+ for target_order_index, target in enumerate(target_order, start=1):+ slots = self._target_slots.get(target)+ if slots is None:+ results.append(execute(scheduled, target, target_order_index))+ continue+ with slots:+ results.append(execute(scheduled, target, target_order_index))+ return results++ if self._parallelism == 1:+ return [+ result+ for scheduled in scheduled_cases+ for result in execute_group(scheduled)+ ]++ results_by_index: dict[int, list[ExecutionT]] = {}+ with ThreadPoolExecutor(max_workers=self._parallelism) as executor:+ future_to_case = {+ executor.submit(execute_group, scheduled): scheduled+ for scheduled in scheduled_cases+ }+ for future in as_completed(future_to_case):+ scheduled = future_to_case[future]+ results_by_index[scheduled.schedule_index] = future.result()+ return [+ result+ for scheduled in scheduled_cases+ for result in results_by_index[scheduled.schedule_index]+ ]+++def safe_artifact_filename_token(value: str, *, max_length: int = 80) -> str:+ token = _SAFE_ARTIFACT_TOKEN.sub("_", value)+ while ".." in token:+ token = token.replace("..", "_")+ token = token.strip("._-") or "site"+ if token == value and len(token) <= max_length:+ return token+ digest = sha256(value.encode("utf-8", errors="surrogatepass")).hexdigest()[:12]+ token = token[:max_length].rstrip("._-") or "site"+ return f"{token}-{digest}"+++def write_public_web_failure_artifacts(+ *,+ suite_dir: Path,+ attempt: PublicWebAttempt,+ row: Mapping[str, Any],+ stdout: bytes,+ stderr: bytes,+ policy: PublicWebArtifactPolicy,+) -> str:+ failures_dir = suite_dir / "failures"+ json_path = failures_dir / f"{attempt.artifact_stem}.json"+ stdout_path = failures_dir / f"{attempt.artifact_stem}.stdout.html"+ stderr_path = failures_dir / f"{attempt.artifact_stem}.stderr.txt"+ write_json(json_path, dict(row))+ write_text(+ stderr_path,+ stderr[-policy.failure_stderr_limit_bytes :].decode(+ "utf-8", errors="replace"+ ),+ )+ if policy.write_empty_failure_stdout or stdout.strip():+ write_text(+ stdout_path,+ stdout[-policy.failure_stdout_limit_bytes :].decode(+ "utf-8", errors="replace"+ ),+ )+ return str(json_path.relative_to(suite_dir))+++def write_public_web_replay_artifact(+ *,+ suite_dir: Path,+ attempt: PublicWebAttempt,+ stdout: bytes,+) -> str:+ replay_path = suite_dir / "replay" / f"{attempt.artifact_stem}.html"+ write_text(replay_path, stdout.decode("utf-8", errors="replace"))+ return str(replay_path.relative_to(suite_dir))+++def successful_public_web_attempt_cohort(+ rows: list[dict[str, Any]],+ targets: tuple[str, ...],+ *,+ attempt_key_fields: tuple[str, ...],+ unique_key_fields: tuple[str, ...],+ unique_count_name: str,+ metric_fields: tuple[str, ...],+ require_comparable: bool = False,+) -> dict[str, Any]:+ by_attempt: dict[tuple[Any, ...], dict[str, dict[str, Any]]] = {}+ for row in rows:+ key = tuple(row[field] for field in attempt_key_fields)+ by_attempt.setdefault(key, {})[str(row["target"])] = row+ selected_keys = [+ key+ for key, target_rows in by_attempt.items()+ if all(+ target in target_rows+ and (+ not require_comparable+ or target_rows[target].get("comparable", True)+ )+ and target_rows[target].get("ok")+ for target in targets+ )+ ]+ unique_indexes = tuple(+ attempt_key_fields.index(field) for field in unique_key_fields+ )+ cohort: dict[str, Any] = {+ "attempts": len(selected_keys),+ unique_count_name: len(+ {+ tuple(key[index] for index in unique_indexes)+ for key in selected_keys+ }+ ),+ "targets": {},+ }+ for target in targets:+ target_rows = [by_attempt[key][target] for key in selected_keys]+ cohort["targets"][target] = {+ field: summarize(+ row[field]+ for row in target_rows+ if row.get(field) is not None+ )+ for field in metric_fields+ }+ return cohort+++def count_public_web_row_values(+ rows: Sequence[Mapping[str, Any]],+) -> dict[str, dict[str, int]]:+ counts: dict[str, dict[str, int]] = {+ "categories": {},+ "failure_kinds": {},+ "classification_bases": {},+ "response_status_sources": {},+ }+ field_names = {+ "category": "categories",+ "failure_kind": "failure_kinds",+ "classification_basis": "classification_bases",+ "response_status_source": "response_status_sources",+ }+ for row in rows:+ for field, count_name in field_names.items():+ value = row.get(field)+ if field == "classification_basis" and not value:+ value = "unknown"+ if not isinstance(value, str) or not value:+ continue+ field_counts = counts[count_name]+ field_counts[value] = field_counts.get(value, 0) + 1+ return countsdiff --git a/moli-benchmark/moli_benchmark/sequential_navigate.py b/moli-benchmark/moli_benchmark/sequential_navigate.pyindex 2288135fa..abe7fbee0 100644--- a/moli-benchmark/moli_benchmark/sequential_navigate.py+++ b/moli-benchmark/moli_benchmark/sequential_navigate.py@@ -25,12 +25,13 @@ from .raw_cdp import ()from .sampling import snapshot_resourcesfrom .target_serve import start_target_serve, stop_target_serve+from .top_sites import TOP_SITES_LIST_PATH, parse_top_sites_listDEFAULT_SEED = 20260809DEFAULT_COUNT = 50DEFAULT_PINNED_DOMAINS = ("csdn.net", "zol.com.cn")-DEFAULT_SEED_FILE = REPO_ROOT / "docs" / "chinese-community-top100-websites.md"+DEFAULT_SEED_FILE = TOP_SITES_LIST_PATHTRACE_METHODS = {"Page.frameStartedNavigating","Page.frameStartedLoading",@@ -1310,7 +1311,7 @@ def _selected_urls(args: argparse.Namespace) -> list[str]:if args.url:base = [normalize_public_url(url) for url in args.url]else:- domains = parse_top_100_domains(args.seed_file.read_text(encoding="utf-8"))+ domains = [target for _, target in parse_top_sites_list(args.seed_file)]base = select_seed_urls(domains,seed=args.seed,diff --git a/moli-benchmark/moli_benchmark/targets.py b/moli-benchmark/moli_benchmark/targets.pyindex e381a490e..c2c626dbd 100644--- a/moli-benchmark/moli_benchmark/targets.py+++ b/moli-benchmark/moli_benchmark/targets.py@@ -5,7 +5,7 @@ from pathlib import Pathfrom typing import Anyfrom .config import chrome_binary, moli_binary, lightpanda_binary, obscura_binary-from .versions import sha256_file+from .versions import MOLI_VERSION_ARGS, sha256_filedef _version(command: list[str]) -> str | None:@@ -39,7 +39,7 @@ def collect_target_binaries() -> dict[str, Any]:moli_path = moli_binary(moli_override)return {- "moli": _binary_info(moli_path, ("version",)),+ "moli": _binary_info(moli_path, MOLI_VERSION_ARGS),"lightpanda": _binary_info(lightpanda_binary(lightpanda_override), ("version",)),"chrome": _binary_info(chrome_binary(chrome_override), ("--version",)),"obscura": _binary_info(obscura_binary(obscura_override), ("--help",)),diff --git a/moli-benchmark/moli_benchmark/top_sites.py b/moli-benchmark/moli_benchmark/top_sites.pyindex d21606c10..ba5a8ea10 100644--- a/moli-benchmark/moli_benchmark/top_sites.py+++ b/moli-benchmark/moli_benchmark/top_sites.py@@ -1,31 +1,47 @@from __future__ import annotations+import csvimport osimport re-import html.parser-from hashlib import sha256-from concurrent.futures import ThreadPoolExecutor, as_completed+from itertools import combinationsfrom pathlib import Pathfrom typing import Any+from urllib.parse import urlsplit-from .artifacts import write_csv, write_json, write_text+from .artifacts import write_csv, write_jsonfrom .chrome_dcl import run_chrome_dcl_dump, run_served_cdp_dcl_dump-from .config import REPO_ROOT, clear_proxy_env+from .config import PROJECT_ROOT, REPO_ROOT, clear_proxy_envfrom .process import ProcessResult, run_process-from .stats import summarize-from .synthetic_compare import (- WEBFETCH_TARGETS,- target_enables_all_resource_fetch,- target_is_cdp,- target_metadata,+from .public_web import (+ PUBLIC_WEB_SNAPSHOT_CONTRACT,+ TOP_SITES_ARTIFACT_POLICY,+ TOP_SITES_CLASSIFIER,+ PublicWebAttempt,+ PublicWebResult,+ PublicWebScheduledCase,+ PublicWebScheduler,+ build_public_web_fetch_command,+ count_public_web_row_values,+ elapsed_failure_reached_timeout as _elapsed_failure_reached_timeout,+ public_web_target_metadata as _top_sites_target_metadata,+ rotated_target_order as _rotated_target_order,+ run_public_web_target,+ safe_artifact_filename_token as _artifact_filename_token,+ schedule_public_web_cases,+ successful_public_web_attempt_cohort,+ unavailable_public_web_row,+ write_public_web_failure_artifacts,)+from .stats import summarize+from .synthetic_compare import WEBFETCH_TARGETS-TOP_SITES_LIST_PATH = REPO_ROOT / "docs" / "chinese-community-top100-websites.md"-GLOBAL_TOP_SITES_LIST_PATH = REPO_ROOT / "docs" / "global-top-websites-seed-list.md"-WEBFETCH_LONGTAIL_LIST_PATH = REPO_ROOT / "docs" / "webfetch-longtail-seed-list.md"-RENDER_QUALITY_LIST_PATH = REPO_ROOT / "docs" / "render-quality-seed-list.md"-LEGACY_ENCODING_LIST_PATH = REPO_ROOT / "docs" / "legacy-encoding-websites-seed-list.md"+TOP_SITES_FIXTURE_ROOT = PROJECT_ROOT / "fixtures" / "top-sites"+TOP_SITES_LIST_PATH = TOP_SITES_FIXTURE_ROOT / "chinese-community-top100-websites.csv"+GLOBAL_TOP_SITES_LIST_PATH = TOP_SITES_FIXTURE_ROOT / "global-top-websites-seed-list.csv"+WEBFETCH_LONGTAIL_LIST_PATH = TOP_SITES_FIXTURE_ROOT / "webfetch-longtail-seed-list.csv"+RENDER_QUALITY_LIST_PATH = TOP_SITES_FIXTURE_ROOT / "render-quality-seed-list.csv"+LEGACY_ENCODING_LIST_PATH = TOP_SITES_FIXTURE_ROOT / "legacy-encoding-websites-seed-list.csv"TOP_SITES_SOURCES: dict[str, dict[str, Any]] = {"chinese-community": {@@ -75,232 +91,11 @@ def _default_top_sites_parallelism() -> int:DEFAULT_TOP_SITES_PARALLELISM = _default_top_sites_parallelism()-DEFAULT_TOP_SITES_TEXT_SAMPLE_CHARS = 500_TOP_LIST_HEADING = re.compile(r"^##\s+Top\s+\d+\b", re.IGNORECASE)_NEXT_SECTION_HEADING = re.compile(r"^##\s+")_LIST_ENTRY = re.compile(r"^\s*(\d+)\.\s+`([^`]+)`")-_BLOCKED_TITLE_403 = re.compile(r"^\s*(?:http\s*)?403\b")-_SAFE_ARTIFACT_TOKEN = re.compile(r"[^A-Za-z0-9._-]+")-_TITLE_RE = re.compile(r"<title[^>]*>(.*?)</title>", re.IGNORECASE | re.DOTALL)-_NETWORK_ERROR_MARKERS = (- "privacy error",- "your connection is not private",- "net::err_cert",- "this site can't be reached",- "this site can\u2019t be reached",- "err_name_not_resolved",- "err_connection",- "err_timed_out",- "dns_probe_finished",- "could not resolve",- "could not resolve host",- "could not resolve hostname",- "could not connect to server",- "name resolution",- "connection refused",- "connection reset",- "connection timed out",- "no route to host",- "network is unreachable",- "ssl handshake",- "ssl error",- "ssl connect error",- "tls handshake",- "timed out connecting",- "timeout was reached",- "request timeout",- "i/o error",- "broken pipe",- "operation timed out",- "failed to connect",- "dns lookup",- "curl request failed",- "recv failure",-)-_CLI_DEADLINE_COMMAND_MARKERS = (- "fetch_document_allow_http_error_with_wait_until",- "fetch document allow-http-error",- "fetch allow-http-error wait_until",- "fetch wait_until",-)-_CLI_DEADLINE_TIMEOUT_MARKERS = (- "timed out after",-)-_RAW_DOCUMENT_BODY_PROGRESS_RE = re.compile(- r"with\s+(\d+)\s+out\s+of\s+(\d+)\s+bytes\s+received",- re.IGNORECASE,-)-_ELAPSED_FAILURE_TIMEOUT_GRACE_SECONDS = 1.0-_CAPTCHA_MARKERS = (- "captcha",- "安全验证",- "验证你是真人",- "人机验证",- "human verification",- "verification",- "verify you are human",- "verify that you're not a robot",- "verify that you are not a robot",- "are you a robot",- "not a robot",- "bot or not",- "checking your browser",- "perform security check",- "security verification",- "performing security verification",- "请完成验证",-)-_LOGIN_MARKERS = (- "sign in",- "log in",- "login",- "登录",- "注册",- "账号",-)-_LOGIN_FORM_MARKERS = (- "login to your account",- "email/username",- "your password is a required field",- "forgot password",- "create a free account",-)-_LOGIN_CONTEXT_MARKERS = (- "password",- "验证码",- "短信验证码",- "语音验证码",- "手机号",- "手机验证",- "邮箱",- "email",-)-_JS_CHALLENGE_MARKERS = (- "__cf_chl_",- "c2wf946j0/probe",- "cf-challenge",- "__tencent_chaos_vm",- "__eo_jschallenge_vm",- "teojschallengesdk.js",- "eojschallengesdk",- "window.solvechallenge(",- "jsl_clearance",- "acw_sc__v2",- "window._phantom",- "_$jsvmprt",- "awswaf",- "aliyun_waf",- "aliyun waf",- "_waf_",- "just a moment...",- "vercel security checkpoint",- "we're verifying your browser",- "we\u2019re verifying your browser",- "enable javascript and cookies to continue",- "javascript is needed to access this site",-)-_BLOCKED_BODY_MARKERS = (- "unable to give you access to our site",- "access denied",- "access to this page has been denied",- "sorry, you have been blocked",- "security issue was automatically identified",- "waf拦截",- "被waf拦截",-)-_NOT_FOUND_MARKERS = (- "404 not found",- "file not found",- "page not found",- "this page cannot be found",- "page you requested is missing",- "page you requested could not be found",-)-_FORBIDDEN_ERROR_MARKERS = (- "403 forbidden",- "returned 403",- "http 403",-)-_NOT_FOUND_ERROR_MARKERS = (- "returned 404",- "http 404",-)-_SHELL_MARKERS = (- 'id="root"',- 'id="app"',- 'id="__next"',- 'data-reactroot',- "webpackJsonp",- "__NUXT__",- "__NEXT_DATA__",-)---class _VisibleTextExtractor(html.parser.HTMLParser):- def __init__(self) -> None:- super().__init__(convert_charrefs=True)- self._skip_depth = 0- self.parts: list[str] = []-- def handle_starttag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None:- if tag.lower() in {"script", "style", "noscript", "svg", "template"}:- self._skip_depth += 1-- def handle_endtag(self, tag: str) -> None:- if tag.lower() in {"script", "style", "noscript", "svg", "template"} and self._skip_depth:- self._skip_depth -= 1-- def handle_data(self, data: str) -> None:- if self._skip_depth:- return- text = " ".join(data.split())- if text:- self.parts.append(text)---def _extract_title_and_text(stdout: bytes) -> dict[str, Any]:- html_text = stdout.decode("utf-8", errors="replace")- title = ""- match = _TITLE_RE.search(html_text)- if match:- title = re.sub(r"\s+", " ", match.group(1)).strip()- parser = _VisibleTextExtractor()- try:- parser.feed(html_text)- except html.parser.HTMLParseError:- pass- visible_text = " ".join(parser.parts)- visible_text = re.sub(r"\s+", " ", visible_text).strip()- return {- "title": title,- "text_length": len(visible_text),- "text_sample": visible_text[:DEFAULT_TOP_SITES_TEXT_SAMPLE_CHARS],- }---def _looks_like_binary_content(body: bytes) -> bool:- sample = body[:4096]- if sample.startswith(b"%PDF-"):- return True- if b"\x00" in sample:- return True- decoded = sample.decode("utf-8", errors="replace")- replacement_count = decoded.count("\ufffd")- return replacement_count > max(8, len(decoded) // 20)---def _looks_like_raw_binary_main_resource_timeout(text: str, min_body_bytes: int) -> bool:- if "failed to read raw document body" not in text:- return False- if "timeout was reached" not in text and "operation timed out" not in text:- return False- match = _RAW_DOCUMENT_BODY_PROGRESS_RE.search(text)- if match is None:- return False- received = int(match.group(1))- total = int(match.group(2))- return received >= min_body_bytes and total >= received+_extract_title_and_text = TOP_SITES_CLASSIFIER.snapshotdef _parse_top_sites_sections(path: Path) -> list[tuple[str, list[tuple[int, str]]]]:@@ -343,11 +138,47 @@ def _parse_top_sites_sections(path: Path) -> list[tuple[str, list[tuple[int, strreturn sections+def _parse_top_sites_csv(path: Path) -> list[tuple[int, str]]:+ if not path.exists():+ raise RuntimeError(f"top sites list not found: {path}")+ try:+ with path.open(encoding="utf-8-sig", newline="") as handle:+ reader = csv.DictReader(handle)+ if reader.fieldnames != ["rank", "target"]:+ raise RuntimeError(f"expected CSV columns `rank,target` in {path}")+ entries: list[tuple[int, str]] = []+ seen_targets: set[str] = set()+ for row_number, row in enumerate(reader, start=2):+ rank_raw = (row.get("rank") or "").strip()+ target = (row.get("target") or "").strip()+ try:+ rank = int(rank_raw)+ except ValueError as error:+ raise RuntimeError(f"invalid rank on CSV row {row_number} in {path}") from error+ if rank <= 0 or not target:+ raise RuntimeError(f"invalid entry on CSV row {row_number} in {path}")+ if target not in seen_targets:+ seen_targets.add(target)+ entries.append((rank, target))+ except csv.Error as error:+ raise RuntimeError(f"invalid CSV seed list {path}: {error}") from error+ if not entries:+ raise RuntimeError(f"no entries parsed from {path}")+ return entries++def parse_top_sites_list(path: Path) -> list[tuple[int, str]]:+ if path.suffix.lower() == ".csv":+ return _parse_top_sites_csv(path)return _parse_top_sites_sections(path)[0][1]def _parse_top_sites_list_by_count(path: Path, count: int) -> list[tuple[int, str]]:+ if path.suffix.lower() == ".csv":+ entries = _parse_top_sites_csv(path)+ if len(entries) != count:+ raise RuntimeError(f"expected {count} entries in {path}, found {len(entries)}")+ return entriesheading = re.compile(rf"^##\s+Top\s+{count}\b", re.IGNORECASE)for section_heading, entries in _parse_top_sites_sections(path):if heading.match(section_heading):@@ -360,9 +191,9 @@ def resolve_top_sites_source(source: str, list_path: Path | None) -> tuple[str,return ("custom", list_path)if source not in TOP_SITES_SOURCES:if source == "mixed":- return (source, REPO_ROOT / "docs" / "mixed-top-websites")+ return (source, TOP_SITES_FIXTURE_ROOT / "mixed-top-websites")if source == "webfetch-mix":- return (source, REPO_ROOT / "docs" / "webfetch-mix-websites")+ return (source, TOP_SITES_FIXTURE_ROOT / "webfetch-mix-websites")raise RuntimeError(f"unknown top-sites source `{source}`; expected one of "f"{sorted(TOP_SITES_SOURCES) + list(COMPOSITE_TOP_SITES_SOURCES)} or a --list-path override"@@ -450,73 +281,24 @@ def load_top_sites_entries(source: str, list_path: Path | None) -> tuple[list[tureturn entries, [f"{source}:{info['path'].name}"]-def _top_command_for_target(target: str, binary: Path, url: str, timeout_seconds: float) -> list[str]:- metadata = target_metadata(target)- if target_is_cdp(target):- raise RuntimeError(f"{target} is a CDP target; use the cdp-session suite")- if target == "chrome":- raise RuntimeError("chrome top-sites uses the CDP DCL runner")- timeout_ms = str(int(timeout_seconds * 1000))- if target in {"moli", "moli-full"}:- compatibility_args = (- ["--layout", "--resource"]- if target_enables_all_resource_fetch(target)- else []- )- return [- str(binary),- "fetch",- *compatibility_args,- "--dump",- "html",- "--wait-until",- "domcontentloaded",- "--timeout",- timeout_ms,- "--http-timeout",- timeout_ms,- url,- ]- if target == "lightpanda":- return [- str(binary),- "fetch",- "--dump",- "html",- "--wait-until",- "domcontentloaded",- "--wait-ms",- timeout_ms,- "--http-timeout",- timeout_ms,- "--terminate-ms",- timeout_ms,- url,- ]- if target == "obscura":- return [- str(binary),- "fetch",- "--dump",- "html",- "--wait-until",- "load",- "--wait",- "0",- "--timeout",- str(max(1, int(timeout_seconds))),- url,- ]- raise RuntimeError(f"unknown target: {target}")+def _is_explicit_pdf_url(url: str) -> bool:+ return urlsplit(url).path.lower().endswith(".pdf")-def _top_sites_target_metadata(target: str) -> dict[str, str]:- metadata = dict(target_metadata(target))- if target == "chrome" or target_is_cdp(target):- metadata["driver"] = "cdp-dcl"- prefix = "moli full" if target == "moli-full-cdp" else metadata["engine"]- metadata["label"] = f"{prefix} / cdp-dcl"- return metadata+def _top_command_for_target(+ target: str,+ binary: Path,+ url: str,+ timeout_seconds: float,+) -> list[str]:+ return build_public_web_fetch_command(+ target,+ binary,+ url,+ timeout_seconds,+ suite_name="top-sites",+ omit_moli_page_wait=_is_explicit_pdf_url(url),+ )def _run_top_sites_target(@@ -527,29 +309,16 @@ def _run_top_sites_target(timeout_seconds: float,proc_env: dict[str, str],) -> ProcessResult:- if target == "chrome":- return run_chrome_dcl_dump(- binary,- url,- cwd=REPO_ROOT,- timeout_seconds=timeout_seconds,- env=proc_env,- )- if target_is_cdp(target):- return run_served_cdp_dcl_dump(- target,- binary,- url,- cwd=REPO_ROOT,- timeout_seconds=timeout_seconds,- env=proc_env,- )- command = _top_command_for_target(target, binary, url, timeout_seconds)- return run_process(- command,- cwd=REPO_ROOT,- timeout_seconds=timeout_seconds + 2,- env=proc_env,+ return run_public_web_target(+ target=target,+ binary=binary,+ url=url,+ timeout_seconds=timeout_seconds,+ proc_env=proc_env,+ command_builder=_top_command_for_target,+ chrome_runner=run_chrome_dcl_dump,+ served_cdp_runner=run_served_cdp_dcl_dump,+ process_runner=run_process,)@@ -560,118 +329,32 @@ def _classify(timed_out: bool,min_body_bytes: int,snapshot: dict[str, Any] | None = None,+ response_status: int | None = None,+ main_document_body_capture: str | None = None,) -> str:- if timed_out:- return "timeout"- combined_text = (stdout + b"\n" + stderr).decode("utf-8", errors="replace").lower()- if "operationtimedout" in combined_text and (- "navigate failed" in combined_text or "navigation failed" in combined_text- ):- return "timeout"- if (- returncode != 0- and any(marker in combined_text for marker in _CLI_DEADLINE_COMMAND_MARKERS)- and any(marker in combined_text for marker in _CLI_DEADLINE_TIMEOUT_MARKERS)- ):- return "timeout"- if returncode != 0 and _looks_like_raw_binary_main_resource_timeout(- combined_text,- min_body_bytes,- ):- return "success-binary-main-resource"- if returncode != 0:- if any(marker in combined_text for marker in _FORBIDDEN_ERROR_MARKERS):- return "blocked-or-forbidden"- if any(marker in combined_text for marker in _NOT_FOUND_ERROR_MARKERS):- return "not-found"- if any(marker in combined_text for marker in _NETWORK_ERROR_MARKERS):- return "network-error"- return "process-error"- body = stdout.strip()- if not body:- if any(marker in combined_text for marker in _NETWORK_ERROR_MARKERS):- return "network-error"- return "empty-response"- if len(body) >= min_body_bytes and _looks_like_binary_content(body):- return "success-binary-content"- if snapshot is None:- snapshot = _extract_title_and_text(stdout)- text_length = int(snapshot.get("text_length") or 0)- title = str(snapshot.get("title") or "").lower()- sample = str(snapshot.get("text_sample") or "").lower()- page_text = f"{title}\n{sample}"- if any(marker in page_text for marker in _NETWORK_ERROR_MARKERS):- return "network-error"- if (- "blocked" in title- or "forbidden" in title- or _BLOCKED_TITLE_403.search(title)- or "access restricted" in title- ):- return "blocked-or-forbidden"- if any(marker in page_text for marker in _NOT_FOUND_MARKERS):- return "not-found"- if any(marker in page_text for marker in _BLOCKED_BODY_MARKERS):- return "blocked-or-forbidden"- if any(marker in page_text for marker in _CAPTCHA_MARKERS):- return "captcha-or-verification"- if any(marker in combined_text for marker in _JS_CHALLENGE_MARKERS):- return "js-challenge"- if _looks_like_login_wall(title, sample, text_length):- return "login-wall"- if len(body) < min_body_bytes:- return "app-shell-only"- if text_length < min_body_bytes:- return "app-shell-only"- return "success-content"---def _elapsed_failure_reached_timeout(elapsed_ms: float | None, timeout_seconds: float) -> bool:- if elapsed_ms is None or timeout_seconds <= 0:- return False- grace_seconds = min(_ELAPSED_FAILURE_TIMEOUT_GRACE_SECONDS, timeout_seconds * 0.05)- timeout_floor_ms = max(0.0, (timeout_seconds - grace_seconds) * 1000.0)- return elapsed_ms >= timeout_floor_ms---def _looks_like_login_wall(title: str, sample: str, text_length: int) -> bool:- page_text = f"{title}\n{sample}"- if sum(1 for marker in _LOGIN_FORM_MARKERS if marker in page_text) >= 2:- return True- if text_length >= 800:- return False- login_hits = {marker for marker in _LOGIN_MARKERS if marker in page_text}- if not login_hits:- return False- title_has_login = any(marker in title for marker in _LOGIN_MARKERS)- if title_has_login and len(login_hits) >= 2:- return True- if len(login_hits) >= 2 and any(marker in page_text for marker in _LOGIN_CONTEXT_MARKERS):- return True- return False+ return TOP_SITES_CLASSIFIER.classify_output(+ stdout=stdout,+ stderr=stderr,+ returncode=returncode,+ timed_out=timed_out,+ min_body_bytes=min_body_bytes,+ snapshot=snapshot,+ response_status=response_status,+ main_document_body_capture=main_document_body_capture,+ )def _ok_categories() -> set[str]:- return {"success-content", "success-binary-content", "success-binary-main-resource"}+ return set(TOP_SITES_CLASSIFIER.ok_categories)-def _failure_kind(category: str, error: str | None) -> str | None:- if error == "target binary unavailable":- return "target-unavailable"- if category not in _ok_categories():- return category- return None+_SITE_UNREACHABLE_FAILURE_KINDS = {"network-error", "timeout"}-_SITE_UNREACHABLE_FAILURE_KINDS = {- "network-error",- "timeout",- "empty-response",- "process-error",-}---def _site_unreachable_exclusions(rows: list[dict[str, Any]]) -> dict[str, str]:+def _site_unreachable_exclusions(+ rows: list[dict[str, Any]],+ targets: tuple[str, ...],+) -> dict[str, str]:rows_by_domain: dict[str, list[dict[str, Any]]] = {}for row in rows:rows_by_domain.setdefault(str(row["domain"]), []).append(row)@@ -680,50 +363,171 @@ def _site_unreachable_exclusions(rows: list[dict[str, Any]]) -> dict[str, str]:for domain, domain_rows in rows_by_domain.items():if any(row.get("ok") for row in domain_rows):continue+ if {str(row.get("target")) for row in domain_rows} != set(targets):+ continuefailure_kinds = {- str(row.get("failure_kind") or _failure_kind(str(row.get("category")), None) or "")+ str(+ row.get("failure_kind")+ or TOP_SITES_CLASSIFIER.failure_kind(+ str(row.get("category"))+ )+ or ""+ )for row in domain_rows}failure_kinds.discard("")if (failure_kindsand failure_kinds <= _SITE_UNREACHABLE_FAILURE_KINDS- and (failure_kinds - {"process-error"})):excluded[domain] = "site-unreachable"return excluded-def _artifact_filename_token(value: str, *, max_length: int = 80) -> str:- token = _SAFE_ARTIFACT_TOKEN.sub("_", value)- while ".." in token:- token = token.replace("..", "_")- token = token.strip("._-") or "site"- if token == value and len(token) <= max_length:- return token- digest = sha256(value.encode("utf-8", errors="surrogatepass")).hexdigest()[:12]- token = token[:max_length].rstrip("._-") or "site"- return f"{token}-{digest}"+def _build_site_outcomes(rows: list[dict[str, Any]]) -> list[dict[str, Any]]:+ grouped: dict[tuple[str, int, str], list[dict[str, Any]]] = {}+ for row in rows:+ key = (str(row["target"]), int(row["rank"]), str(row["domain"]))+ grouped.setdefault(key, []).append(row)++ outcomes: list[dict[str, Any]] = []+ for (target, rank, domain), attempts in grouped.items():+ comparable_attempts = [+ row for row in attempts if row.get("comparable", True)+ ]+ passes = sum(1 for row in comparable_attempts if row.get("ok"))+ attempt_count = len(comparable_attempts)+ if attempt_count == 0:+ outcome = "not-comparable"+ elif passes == attempt_count:+ outcome = "all-pass"+ elif passes == 0:+ outcome = "all-fail"+ else:+ outcome = "flaky"+ categories: dict[str, int] = {}+ for row in attempts:+ category = str(row.get("category") or "unknown")+ categories[category] = categories.get(category, 0) + 1+ response_statuses: set[int] = set()+ response_status_sources: dict[str, int] = {}+ for row in attempts:+ status = row.get("response_status")+ if isinstance(status, int) and not isinstance(status, bool):+ response_statuses.add(status)+ source = row.get("response_status_source")+ if isinstance(source, str) and source:+ response_status_sources[source] = (+ response_status_sources.get(source, 0) + 1+ )+ outcomes.append(+ {+ "target": target,+ "rank": rank,+ "domain": domain,+ "url": str(attempts[0].get("url") or domain),+ "attempts": attempt_count,+ "raw_attempts": len(attempts),+ "non_comparable_attempts": len(attempts) - attempt_count,+ "passes": passes,+ "failures": attempt_count - passes,+ "pass_rate_percent": (+ (passes / attempt_count) * 100.0 if attempt_count else None+ ),+ "outcome": outcome,+ "categories": categories,+ "response_statuses": sorted(response_statuses),+ "response_status_sources": response_status_sources,+ "status_observed_attempts": sum(+ 1 for row in attempts if row.get("response_status") is not None+ ),+ }+ )+ outcomes.sort(+ key=lambda row: (+ str(row["target"]),+ int(row["rank"]),+ str(row["domain"]),+ )+ )+ return outcomes-def _write_failure_artifact(+def _pairwise_site_comparisons(+ outcomes: list[dict[str, Any]],+ targets: tuple[str, ...],*,- suite_dir: Path,- row: dict[str, Any],- stdout: bytes,- stderr: bytes,-) -> str:- domain = _artifact_filename_token(str(row["domain"]))- name = f"{row['target']}-run-{row['run']}-rank{row['rank']:03d}-{domain}"- failures_dir = suite_dir / "failures"- json_path = failures_dir / f"{name}.json"- stderr_path = failures_dir / f"{name}.stderr.txt"- write_json(json_path, row)- write_text(stderr_path, stderr[-16 * 1024 :].decode("utf-8", errors="replace"))- if stdout.strip():- stdout_path = failures_dir / f"{name}.stdout.html"- write_text(stdout_path, stdout[-32 * 1024 :].decode("utf-8", errors="replace"))- return str(json_path.relative_to(suite_dir))+ runs: int,+) -> list[dict[str, Any]]:+ by_target_site = {+ (str(row["target"]), int(row["rank"]), str(row["domain"])): row+ for row in outcomes+ }+ site_keys = sorted({(int(row["rank"]), str(row["domain"])) for row in outcomes})+ comparisons: list[dict[str, Any]] = []+ for left, right in combinations(targets, 2):+ counts = {+ "both_all_pass": 0,+ "left_all_pass_right_all_fail": 0,+ "right_all_pass_left_all_fail": 0,+ "both_all_fail": 0,+ "flaky_or_incomplete": 0,+ "not_comparable": 0,+ }+ left_only_sites: list[dict[str, Any]] = []+ right_only_sites: list[dict[str, Any]] = []+ evaluated_sites = 0+ for rank, domain in site_keys:+ left_row = by_target_site.get((left, rank, domain))+ right_row = by_target_site.get((right, rank, domain))+ if left_row is None or right_row is None:+ continue+ left_outcome = left_row["outcome"]+ right_outcome = right_row["outcome"]+ if "not-comparable" in {left_outcome, right_outcome}:+ counts["not_comparable"] += 1+ continue+ evaluated_sites += 1+ if left_outcome == "all-pass" and right_outcome == "all-pass":+ counts["both_all_pass"] += 1+ elif left_outcome == "all-pass" and right_outcome == "all-fail":+ counts["left_all_pass_right_all_fail"] += 1+ left_only_sites.append({"rank": rank, "domain": domain})+ elif right_outcome == "all-pass" and left_outcome == "all-fail":+ counts["right_all_pass_left_all_fail"] += 1+ right_only_sites.append({"rank": rank, "domain": domain})+ elif left_outcome == "all-fail" and right_outcome == "all-fail":+ counts["both_all_fail"] += 1+ else:+ counts["flaky_or_incomplete"] += 1+ comparisons.append(+ {+ "left": left,+ "right": right,+ "runs_per_site": runs,+ "repeat_validated": runs >= 3,+ "evaluated_sites": evaluated_sites,+ **counts,+ "left_only_sites": left_only_sites,+ "right_only_sites": right_only_sites,+ }+ )+ return comparisons+++def _successful_attempt_cohort(+ rows: list[dict[str, Any]],+ targets: tuple[str, ...],+) -> dict[str, Any]:+ return successful_public_web_attempt_cohort(+ rows,+ targets,+ attempt_key_fields=("run", "rank", "domain"),+ unique_key_fields=("rank", "domain"),+ unique_count_name="unique_sites",+ metric_fields=("elapsed_ms", "peak_pss_bytes", "peak_rss_bytes"),+ require_comparable=True,+ )def _domain_to_url(domain: str) -> str:@@ -744,69 +548,88 @@ def _execute_one(timeout_seconds: float,min_body_bytes: int,proc_env: dict[str, str],+ schedule_index: int,+ target_order_index: int,) -> dict[str, Any]:url = _domain_to_url(domain)- if not info.get("available") or not info.get("path"):- row = {- "target": target,- **metadata,- "run": run_id,- "rank": rank,- "domain": domain,- "url": url,- "category": "error",- "ok": False,- "failure_kind": "target-unavailable",- "error": "target binary unavailable",- }- row["failure_artifact"] = _write_failure_artifact(suite_dir=suite_dir, row=row, stdout=b"", stderr=b"")+ attempt = PublicWebAttempt.start(+ target=target,+ metadata=metadata,+ target_info=info,+ run=run_id,+ case_fields={"rank": rank, "domain": domain},+ url=url,+ schedule_index=schedule_index,+ target_order_index=target_order_index,+ artifact_stem=(+ f"{target}-run-{run_id}-rank{rank:03d}-"+ f"{_artifact_filename_token(domain)}"+ ),+ )+ binary = attempt.binary_path()+ if binary is None:+ row = unavailable_public_web_row(attempt, category="error")+ row["failure_artifact"] = write_public_web_failure_artifacts(+ suite_dir=suite_dir,+ attempt=attempt,+ row=row,+ stdout=b"",+ stderr=b"",+ policy=TOP_SITES_ARTIFACT_POLICY,+ )return row- result = _run_top_sites_target(++ process = _run_top_sites_target(target=target,- binary=Path(info["path"]),+ binary=binary,url=url,timeout_seconds=timeout_seconds,proc_env=proc_env,)- snapshot = _extract_title_and_text(result.stdout)- category = _classify(- result.stdout,- result.stderr,- result.returncode,- result.timed_out,- min_body_bytes,- snapshot,+ result = PublicWebResult.capture(+ attempt,+ process,+ classifier=TOP_SITES_CLASSIFIER,+ )+ category = TOP_SITES_CLASSIFIER.classify_result(+ result,+ min_body_bytes=min_body_bytes,)- if category not in _ok_categories() and _elapsed_failure_reached_timeout(result.elapsed_ms, timeout_seconds):+ if (+ result.response_status is None+ and not TOP_SITES_CLASSIFIER.classification_ok(category)+ and _elapsed_failure_reached_timeout(process.elapsed_ms, timeout_seconds)+ ):category = "timeout"- ok = category in _ok_categories()+ comparable = TOP_SITES_CLASSIFIER.comparable(category)+ ok = TOP_SITES_CLASSIFIER.classification_ok(category)row: dict[str, Any] = {- "target": target,- **metadata,- "run": run_id,- "rank": rank,- "domain": domain,- "url": url,- "command": result.command,+ **result.base_row(),+ **result.evidence_fields(policy=TOP_SITES_ARTIFACT_POLICY),+ "command": process.command,"category": category,+ "classification_basis": TOP_SITES_CLASSIFIER.classification_basis(+ result, category+ ),"ok": ok,- "failure_kind": None if ok else _failure_kind(category, None),- "elapsed_ms": result.elapsed_ms,- "returncode": result.returncode,- "timed_out": result.timed_out,- "stdout_bytes": len(result.stdout),- "stderr_bytes": len(result.stderr),- "title": snapshot["title"],- "text_length": snapshot["text_length"],- "text_sample": snapshot["text_sample"],- "stdout_tail": result.stdout[-512:].decode("utf-8", errors="replace"),- "stderr_tail": result.stderr[-512:].decode("utf-8", errors="replace"),- "peak_pss_bytes": result.resources.get("peak_pss_bytes"),- "peak_rss_bytes": result.resources.get("peak_rss_bytes"),+ "comparable": comparable,+ "non_comparable_reason": (+ None if comparable else "main-document-body-not-captured"+ ),+ "failure_kind": (+ None+ if ok or not comparable+ else TOP_SITES_CLASSIFIER.failure_kind(category)+ ),}- if not ok:- row["failure_artifact"] = _write_failure_artifact(- suite_dir=suite_dir, row=row, stdout=result.stdout, stderr=result.stderr+ if comparable and not ok:+ row["failure_artifact"] = write_public_web_failure_artifacts(+ suite_dir=suite_dir,+ attempt=attempt,+ row=row,+ stdout=process.stdout,+ stderr=process.stderr,+ policy=TOP_SITES_ARTIFACT_POLICY,)return row@@ -853,52 +676,89 @@ def run_top_sites_suite(list_source = primary_pathproc_env = clear_proxy_env(os.environ)- rows: list[dict[str, Any]] = []- for target in targets:- metadata = _top_sites_target_metadata(target)- info = target_matrix.get(metadata["binary_key"], {})- job_specs = [- (run_id, rank, domain)- for run_id in range(1, runs_count + 1)- for rank, domain in entries- ]- target_workers = chrome_parallelism if target == "chrome" else parallelism- with ThreadPoolExecutor(max_workers=target_workers) as executor:- future_to_spec = {- executor.submit(- _execute_one,- suite_dir=suite_dir,- target=target,- metadata=metadata,- info=info,- run_id=run_id,- rank=rank,- domain=domain,- timeout_seconds=timeout_seconds,- min_body_bytes=min_body_bytes,- proc_env=proc_env,- ): (target, run_id, rank, domain)- for run_id, rank, domain in job_specs- }- for future in as_completed(future_to_spec):- rows.append(future.result())+ metadata_by_target = {+ target: _top_sites_target_metadata(target) for target in targets+ }+ info_by_target = {+ target: target_matrix.get(metadata_by_target[target]["binary_key"], {})+ for target in targets+ }+ scheduled_cases = schedule_public_web_cases(entries, runs=runs_count)++ def execute_attempt(+ scheduled: PublicWebScheduledCase[tuple[int, str]],+ target: str,+ target_order_index: int,+ ) -> dict[str, Any]:+ rank, domain = scheduled.case+ return _execute_one(+ suite_dir=suite_dir,+ target=target,+ metadata=metadata_by_target[target],+ info=info_by_target[target],+ run_id=scheduled.run,+ rank=rank,+ domain=domain,+ timeout_seconds=timeout_seconds,+ min_body_bytes=min_body_bytes,+ proc_env=proc_env,+ schedule_index=scheduled.schedule_index,+ target_order_index=target_order_index,+ )- rows.sort(key=lambda row: (row["target"], row["run"], row["rank"]))+ rows = PublicWebScheduler[tuple[int, str], dict[str, Any]](+ targets,+ parallelism=parallelism,+ target_parallelism={"chrome": chrome_parallelism},+ ).run(+ scheduled_cases,+ execute_attempt,+ )- excluded_domains = _site_unreachable_exclusions(rows) if len(targets) > 1 else {}+ excluded_domains = (+ _site_unreachable_exclusions(rows, targets) if len(targets) > 1 else {}+ )for row in rows:exclusion_reason = excluded_domains.get(str(row["domain"]))row["excluded"] = exclusion_reason is not Nonerow["exclusion_reason"] = exclusion_reasoncounted_rows = [row for row in rows if not row.get("excluded")]- gate_failures = sum(1 for row in counted_rows if row["target"] == gate_target and not row.get("ok"))+ comparable_counted_rows = [+ row for row in counted_rows if row.get("comparable", True)+ ]+ site_outcomes = _build_site_outcomes(counted_rows)+ pairwise_comparisons = _pairwise_site_comparisons(+ site_outcomes,+ targets,+ runs=runs_count,+ )+ for comparison in pairwise_comparisons:+ comparison["common_success"] = _successful_attempt_cohort(+ counted_rows,+ (str(comparison["left"]), str(comparison["right"])),+ )+ common_success = _successful_attempt_cohort(counted_rows, targets)+ gate_failures = sum(+ 1+ for row in comparable_counted_rows+ if row["target"] == gate_target and not row.get("ok")+ )+ gate_site_outcomes = [+ row+ for row in site_outcomes+ if row["target"] == gate_target and row["outcome"] != "not-comparable"+ ]summary: dict[str, Any] = {"suite": "top-sites","profile": profile,"limit": limit,"source": resolved_source,- "list_source": str(list_source.relative_to(REPO_ROOT)) if list_source.is_relative_to(REPO_ROOT) else str(list_source),+ "list_source": (+ str(list_source.relative_to(REPO_ROOT))+ if list_source.is_relative_to(REPO_ROOT)+ else str(list_source)+ ),"list_sources": list_source_labels,"site_count": len(entries),"counted_site_count": len(entries) - len(excluded_domains),@@ -913,40 +773,158 @@ def run_top_sites_suite("runs": runs_count,"timeout_seconds": timeout_seconds,"min_body_bytes": min_body_bytes,+ "snapshot_contract": PUBLIC_WEB_SNAPSHOT_CONTRACT,+ "schedule": "site-paired-rotating-target-order",+ "scheduled_site_groups": len(scheduled_cases),"parallelism": parallelism,"chrome_parallelism": chrome_parallelism,"gate_target": gate_target,"gate_failures": gate_failures,- "total_failures": sum(1 for row in counted_rows if not row.get("ok")),+ "gate_site_failures": sum(+ 1 for row in gate_site_outcomes if row["outcome"] != "all-pass"+ ),+ "gate_flaky_sites": sum(+ 1 for row in gate_site_outcomes if row["outcome"] == "flaky"+ ),+ "total_failures": sum(+ 1 for row in comparable_counted_rows if not row.get("ok")+ ),+ "total_non_comparable_runs": len(counted_rows)+ - len(comparable_counted_rows),"total_excluded_runs": sum(1 for row in rows if row.get("excluded")),+ "repeat_validated": runs_count >= 3,+ "pairwise": pairwise_comparisons,+ "common_success": common_success,"targets": {},}for target in targets:all_target_rows = [row for row in rows if row["target"] == target]- target_rows = [row for row in all_target_rows if not row.get("excluded")]- categories: dict[str, int] = {}- failure_kinds: dict[str, int] = {}- for row in target_rows:- categories[row["category"]] = categories.get(row["category"], 0) + 1- failure_kind = row.get("failure_kind")- if isinstance(failure_kind, str) and failure_kind:- failure_kinds[failure_kind] = failure_kinds.get(failure_kind, 0) + 1+ target_observations = [+ row for row in all_target_rows if not row.get("excluded")+ ]+ target_rows = [+ row for row in target_observations if row.get("comparable", True)+ ]+ raw_comparable_rows = [+ row for row in all_target_rows if row.get("comparable", True)+ ]+ successful_rows = [row for row in target_rows if row.get("ok")]+ all_target_site_outcomes = [+ row for row in site_outcomes if row["target"] == target+ ]+ target_site_outcomes = [+ row+ for row in all_target_site_outcomes+ if row["outcome"] != "not-comparable"+ ]+ value_counts = count_public_web_row_values(target_observations)+ status_observed_attempts = sum(+ 1+ for row in target_observations+ if row.get("response_status") is not None+ )summary["targets"][target] = {**_top_sites_target_metadata(target),"sites": len(target_rows),"raw_sites": len(all_target_rows),- "excluded_runs": len(all_target_rows) - len(target_rows),+ "observed_sites": len(target_observations),+ "unique_sites": len(target_site_outcomes),+ "observed_unique_sites": len(all_target_site_outcomes),+ "attempts": len(target_rows),+ "raw_observations": len(all_target_rows),+ "raw_comparable_attempts": len(raw_comparable_rows),+ "raw_non_comparable_attempts": len(all_target_rows)+ - len(raw_comparable_rows),+ "raw_passes": sum(1 for row in raw_comparable_rows if row.get("ok")),+ "raw_failures": sum(+ 1 for row in raw_comparable_rows if not row.get("ok")+ ),+ "raw_pass_rate_percent": (+ (+ sum(1 for row in raw_comparable_rows if row.get("ok"))+ / len(raw_comparable_rows)+ )+ * 100.0+ if raw_comparable_rows+ else None+ ),+ "non_comparable_attempts": len(target_observations) - len(target_rows),+ "excluded_runs": len(all_target_rows) - len(target_observations),"runs": runs_count,+ "repeat_validated": runs_count >= 3,"passes": sum(1 for row in target_rows if row.get("ok")),"failures": sum(1 for row in target_rows if not row.get("ok")),- "categories": categories,- "failure_kinds": failure_kinds,- "elapsed_ms": summarize(row["elapsed_ms"] for row in target_rows if row.get("elapsed_ms") is not None),- "peak_pss_bytes": summarize(row["peak_pss_bytes"] for row in target_rows if row.get("peak_pss_bytes") is not None),- "peak_rss_bytes": summarize(row["peak_rss_bytes"] for row in target_rows if row.get("peak_rss_bytes") is not None),+ "pass_rate_percent": (+ (sum(1 for row in target_rows if row.get("ok")) / len(target_rows))+ * 100.0+ if target_rows+ else None+ ),+ "all_pass_sites": sum(+ 1 for row in target_site_outcomes if row["outcome"] == "all-pass"+ ),+ "all_fail_sites": sum(+ 1 for row in target_site_outcomes if row["outcome"] == "all-fail"+ ),+ "flaky_sites": sum(+ 1 for row in target_site_outcomes if row["outcome"] == "flaky"+ ),+ "status_observed_attempts": status_observed_attempts,+ "status_unobserved_attempts": len(target_observations)+ - status_observed_attempts,+ "response_status_sources": value_counts[+ "response_status_sources"+ ],+ "status_coverage_percent": (+ (status_observed_attempts / len(target_observations)) * 100.0+ if target_observations+ else None+ ),+ "content_inferred_http_failures": sum(+ 1+ for row in target_rows+ if row.get("response_status") is None+ and row.get("returncode") == 0+ and row.get("category")+ in {"blocked-or-forbidden", "not-found", "http-error"}+ ),+ "categories": value_counts["categories"],+ "failure_kinds": value_counts["failure_kinds"],+ "classification_bases": value_counts["classification_bases"],+ "elapsed_ms": summarize(+ row["elapsed_ms"]+ for row in target_rows+ if row.get("elapsed_ms") is not None+ ),+ "successful_elapsed_ms": summarize(+ row["elapsed_ms"]+ for row in successful_rows+ if row.get("elapsed_ms") is not None+ ),+ "peak_pss_bytes": summarize(+ row["peak_pss_bytes"]+ for row in target_rows+ if row.get("peak_pss_bytes") is not None+ ),+ "successful_peak_pss_bytes": summarize(+ row["peak_pss_bytes"]+ for row in successful_rows+ if row.get("peak_pss_bytes") is not None+ ),+ "peak_rss_bytes": summarize(+ row["peak_rss_bytes"]+ for row in target_rows+ if row.get("peak_rss_bytes") is not None+ ),+ "successful_peak_rss_bytes": summarize(+ row["peak_rss_bytes"]+ for row in successful_rows+ if row.get("peak_rss_bytes") is not None+ ),}write_csv(suite_dir / "raw-runs.csv", rows)write_json(suite_dir / "runs.json", rows)+ write_json(suite_dir / "site-outcomes.json", site_outcomes)write_json(suite_dir / "summary.json", summary)return summarydiff --git a/moli-benchmark/moli_benchmark/versions.py b/moli-benchmark/moli_benchmark/versions.pyindex 48724a581..bf1c9e192 100644--- a/moli-benchmark/moli_benchmark/versions.py+++ b/moli-benchmark/moli_benchmark/versions.py@@ -9,6 +9,9 @@ from . import __version__from .config import REPO_ROOT+MOLI_VERSION_ARGS = ("--version",)++def sha256_file(path: Path) -> str:digest = hashlib.sha256()with path.open("rb") as file:@@ -33,7 +36,7 @@ def collect_versions(moli_bin: Path | None = None, targets: dict[str, Any] | Non"path": str(moli_bin),"size_bytes": moli_bin.stat().st_size,"sha256": sha256_file(moli_bin),- "version": _run([str(moli_bin), "version"]),+ "version": _run([str(moli_bin), *MOLI_VERSION_ARGS]),}elif targets and isinstance(targets.get("moli"), dict):moli = targets["moli"]diff --git a/moli-benchmark/moli_benchmark/wild_web.py b/moli-benchmark/moli_benchmark/wild_web.pyindex d4a4dad10..df968efaa 100644--- a/moli-benchmark/moli_benchmark/wild_web.py+++ b/moli-benchmark/moli_benchmark/wild_web.py@@ -1,21 +1,34 @@from __future__ import annotations-import html.parserimport osfrom pathlib import Pathfrom typing import Any-from .artifacts import write_csv, write_json, write_text+from .artifacts import write_csv, write_jsonfrom .chrome_dcl import run_chrome_dcl_dump, run_served_cdp_dcl_dump-from .config import REPO_ROOT, clear_proxy_env-from .process import run_process-from .stats import summarize-from .synthetic_compare import (- WEBFETCH_TARGETS,- target_enables_all_resource_fetch,- target_is_cdp,- target_metadata,+from .config import clear_proxy_env+from .process import ProcessResult, run_process+from .public_web import (+ PUBLIC_WEB_SNAPSHOT_CONTRACT,+ WILD_WEB_ARTIFACT_POLICY,+ WILD_WEB_CLASSIFIER,+ PublicWebAttempt,+ PublicWebResult,+ PublicWebScheduledCase,+ PublicWebScheduler,+ build_public_web_fetch_command,+ count_public_web_row_values,+ public_web_target_metadata as _wild_web_target_metadata,+ rotated_target_order as _rotated_target_order,+ run_public_web_target,+ schedule_public_web_cases,+ successful_public_web_attempt_cohort,+ unavailable_public_web_row,+ write_public_web_failure_artifacts,+ write_public_web_replay_artifact,)+from .stats import summarize+from .synthetic_compare import WEBFETCH_TARGETSWILD_WEB_SEEDS: dict[str, str] = {@@ -26,54 +39,13 @@ WILD_WEB_SEEDS: dict[str, str] = {}WILD_WEB_SEED_ASSERTIONS: dict[str, dict[str, Any]] = {- "baidu-home": {"title_any": ("百度", "baidu"), "text_any": ("百度", "baidu"), "min_text_length": 20},- "bilibili-home": {"title_any": ("哔哩", "bilibili"), "text_any": ("哔哩", "bilibili"), "min_text_length": 20},- "zhihu-home": {"title_any": ("知乎", "zhihu"), "text_any": ("知乎", "zhihu"), "min_text_length": 20},- "toutiao-home": {"title_any": ("头条", "toutiao"), "text_any": ("头条", "toutiao"), "min_text_length": 20},+ "baidu-home": {"title_any": ("百度", "baidu"), "min_text_length": 20},+ "bilibili-home": {"title_any": ("哔哩", "bilibili"), "min_text_length": 20},+ "zhihu-home": {"title_any": ("知乎", "zhihu"), "min_text_length": 20},+ "toutiao-home": {"title_any": ("头条", "toutiao"), "min_text_length": 20},}--class _TextExtractor(html.parser.HTMLParser):- def __init__(self) -> None:- super().__init__(convert_charrefs=True)- self._hidden_depth = 0- self._in_title = False- self.title_parts: list[str] = []- self.text_parts: list[str] = []-- def handle_starttag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None:- if tag in {"script", "style", "noscript"}:- self._hidden_depth += 1- if tag == "title":- self._in_title = True-- def handle_endtag(self, tag: str) -> None:- if tag in {"script", "style", "noscript"} and self._hidden_depth:- self._hidden_depth -= 1- if tag == "title":- self._in_title = False-- def handle_data(self, data: str) -> None:- text = " ".join(data.split())- if not text:- return- if self._in_title:- self.title_parts.append(text)- if self._hidden_depth == 0:- self.text_parts.append(text)---def _extract_page_snapshot(stdout: bytes) -> dict[str, Any]:- text = stdout.decode("utf-8", errors="replace")- parser = _TextExtractor()- parser.feed(text)- title = " ".join(parser.title_parts).strip()- body_text = " ".join(parser.text_parts).strip()- return {- "title": title,- "text_length": len(body_text),- "text_sample": body_text[:500],- }+_extract_page_snapshot = WILD_WEB_CLASSIFIER.snapshotdef _wild_web_extraction_failures(seed: str, snapshot: dict[str, Any]) -> list[str]:@@ -98,142 +70,191 @@ def _wild_web_extraction_failures(seed: str, snapshot: dict[str, Any]) -> list[sreturn failures-def _wild_command_for_target(target: str, binary: Path, url: str, timeout_seconds: float) -> list[str]:- metadata = target_metadata(target)- if target_is_cdp(target):- raise RuntimeError(f"{target} is a CDP target; use the cdp-session suite")- if target == "chrome":- raise RuntimeError("chrome wild-web uses the CDP DCL runner")- timeout_ms = str(int(timeout_seconds * 1000))- if target in {"moli", "moli-full"}:- compatibility_args = (- ["--layout", "--resource"]- if target_enables_all_resource_fetch(target)- else []- )- return [- str(binary),- "fetch",- *compatibility_args,- "--dump",- "html",- "--wait-until",- "done",- "--timeout",- timeout_ms,- url,- ]- if target == "lightpanda":- return [- str(binary),- "fetch",- "--dump",- "html",- "--wait-until",- "done",- "--wait-ms",- timeout_ms,- "--http-timeout",- timeout_ms,- "--terminate-ms",- timeout_ms,- url,- ]- if target == "obscura":- return [- str(binary),- "fetch",- "--dump",- "html",- "--wait-until",- "load",- "--wait",- "0",- "--timeout",- str(max(1, int(timeout_seconds))),- url,- ]- raise RuntimeError(f"unknown target: {target}")+def _wild_command_for_target(+ target: str,+ binary: Path,+ url: str,+ timeout_seconds: float,+) -> list[str]:+ return build_public_web_fetch_command(+ target,+ binary,+ url,+ timeout_seconds,+ suite_name="wild-web",+ )-def _wild_web_target_metadata(target: str) -> dict[str, str]:- metadata = dict(target_metadata(target))- if target == "chrome" or target_is_cdp(target):- metadata["driver"] = "cdp-dcl"- prefix = "moli full" if target == "moli-full-cdp" else metadata["engine"]- metadata["label"] = f"{prefix} / cdp-dcl"- return metadata+def _run_wild_web_target(+ *,+ target: str,+ binary: Path,+ url: str,+ timeout_seconds: float,+ proc_env: dict[str, str],+) -> ProcessResult:+ return run_public_web_target(+ target=target,+ binary=binary,+ url=url,+ timeout_seconds=timeout_seconds,+ proc_env=proc_env,+ command_builder=_wild_command_for_target,+ chrome_runner=run_chrome_dcl_dump,+ served_cdp_runner=run_served_cdp_dcl_dump,+ process_runner=run_process,+ )-def _classify(stdout: bytes, stderr: bytes, returncode: int | None, timed_out: bool) -> str:- if timed_out:- return "timeout"- if returncode != 0:- return "error"- snapshot = _extract_page_snapshot(stdout)- visible_text = " ".join(- str(snapshot.get(key) or "") for key in ("title", "text_sample")- ).lower()- diagnostic_text = stderr.decode("utf-8", errors="replace").lower()- text = f"{visible_text}\n{diagnostic_text}"- if "captcha" in text or "verify" in text or "challenge" in text or "安全验证" in text:- return "challenge"- if "login" in text or "登录" in text:- return "login"- if "blocked" in text or "forbidden" in text or "403" in text:- return "blocked"- return "success" if stdout.strip() else "empty"+def _classify(+ stdout: bytes,+ stderr: bytes,+ returncode: int | None,+ timed_out: bool,+ response_status: int | None = None,+) -> str:+ return WILD_WEB_CLASSIFIER.classify_output(+ stdout=stdout,+ stderr=stderr,+ returncode=returncode,+ timed_out=timed_out,+ response_status=response_status,+ )def _failure_kind(category: str, extraction_failures: list[str], error: str | None = None) -> str | None:- if error == "target binary unavailable":- return "target-unavailable"- if category == "timeout":- return "timeout"- if category == "blocked":- return "blocked"- if category == "challenge":- return "challenge"- if category == "login":- return "login"- if category == "empty":- return "empty-response"- if category == "error":- return "process-error"- if extraction_failures:- return "extraction-failure"- return None+ return WILD_WEB_CLASSIFIER.failure_kind(+ category,+ error=error,+ extraction_failed=bool(extraction_failures),+ )-def _write_failure_artifacts(- *,- suite_dir: Path,- row: dict[str, Any],- stdout: bytes,- stderr: bytes,-) -> str:- name = f"{row['target']}-run-{row['run']}-{row['seed']}"- failures_dir = suite_dir / "failures"- json_path = failures_dir / f"{name}.json"- stdout_path = failures_dir / f"{name}.stdout.html"- stderr_path = failures_dir / f"{name}.stderr.txt"- write_json(json_path, row)- write_text(stdout_path, stdout[-128 * 1024 :].decode("utf-8", errors="replace"))- write_text(stderr_path, stderr[-32 * 1024 :].decode("utf-8", errors="replace"))- return str(json_path.relative_to(suite_dir))+def _successful_attempt_cohort(+ rows: list[dict[str, Any]],+ targets: tuple[str, ...],+) -> dict[str, Any]:+ return successful_public_web_attempt_cohort(+ rows,+ targets,+ attempt_key_fields=("run", "seed"),+ unique_key_fields=("seed",),+ unique_count_name="unique_seeds",+ metric_fields=("elapsed_ms", "peak_pss_bytes"),+ )-def _write_replay_artifact(+def _execute_wild_web_attempt(*,suite_dir: Path,- row: dict[str, Any],- stdout: bytes,-) -> str:- name = f"{row['target']}-run-{row['run']}-{row['seed']}.html"- replay_dir = suite_dir / "replay"- replay_path = replay_dir / name- write_text(replay_path, stdout.decode("utf-8", errors="replace"))- return str(replay_path.relative_to(suite_dir))+ target: str,+ metadata: dict[str, str],+ info: dict[str, Any],+ run_id: int,+ seed: str,+ timeout_seconds: float,+ proc_env: dict[str, str],+ schedule_index: int,+ target_order_index: int,+ capture_replay: bool,+) -> tuple[dict[str, Any], dict[str, Any] | None]:+ url = WILD_WEB_SEEDS[seed]+ attempt = PublicWebAttempt.start(+ target=target,+ metadata=metadata,+ target_info=info,+ run=run_id,+ case_fields={"seed": seed},+ url=url,+ schedule_index=schedule_index,+ target_order_index=target_order_index,+ artifact_stem=f"{target}-run-{run_id}-{seed}",+ )+ binary = attempt.binary_path()+ if binary is None:+ row = unavailable_public_web_row(+ attempt,+ category="error",+ extra_fields={+ "classification_ok": False,+ "extraction_ok": False,+ "extraction_failures": ["target-unavailable"],+ "extraction_failure_count": 1,+ },+ )+ row["failure_artifact"] = write_public_web_failure_artifacts(+ suite_dir=suite_dir,+ attempt=attempt,+ row=row,+ stdout=b"",+ stderr=b"",+ policy=WILD_WEB_ARTIFACT_POLICY,+ )+ return row, None++ process = _run_wild_web_target(+ target=target,+ binary=binary,+ url=url,+ timeout_seconds=timeout_seconds,+ proc_env=proc_env,+ )+ result = PublicWebResult.capture(+ attempt,+ process,+ classifier=WILD_WEB_CLASSIFIER,+ )+ category = WILD_WEB_CLASSIFIER.classify_result(result)+ snapshot = result.snapshot+ extraction_failures = _wild_web_extraction_failures(seed, snapshot)+ classification_ok = WILD_WEB_CLASSIFIER.classification_ok(category)+ extraction_ok = not extraction_failures+ ok = classification_ok and extraction_ok+ row = {+ **result.base_row(),+ **result.evidence_fields(policy=WILD_WEB_ARTIFACT_POLICY),+ "category": category,+ "classification_basis": WILD_WEB_CLASSIFIER.classification_basis(+ result, category+ ),+ "classification_ok": classification_ok,+ "extraction_ok": extraction_ok,+ "ok": ok,+ "title": snapshot["title"],+ "text_length": snapshot["text_length"],+ "text_sample": snapshot["text_sample"],+ "extraction_failures": extraction_failures,+ "extraction_failure_count": len(extraction_failures),+ "failure_kind": None if ok else _failure_kind(category, extraction_failures),+ }+ if not ok:+ row["failure_artifact"] = write_public_web_failure_artifacts(+ suite_dir=suite_dir,+ attempt=attempt,+ row=row,+ stdout=process.stdout,+ stderr=process.stderr,+ policy=WILD_WEB_ARTIFACT_POLICY,+ )+ return row, None+ if not capture_replay:+ return row, None+ row["replay_artifact"] = write_public_web_replay_artifact(+ suite_dir=suite_dir,+ attempt=attempt,+ stdout=process.stdout,+ )+ return row, {+ "target": target,+ **metadata,+ "run": run_id,+ "seed": seed,+ "url": url,+ "category": category,+ "title": snapshot["title"],+ "text_length": snapshot["text_length"],+ "artifact": row["replay_artifact"],+ }def run_wild_web_suite(@@ -254,121 +275,49 @@ def run_wild_web_suite(raise RuntimeError(f"unknown gate target: {gate_target}")suite_dir = output_dir / "wild-web"- rows: list[dict[str, Any]] = []- replay_manifest: list[dict[str, Any]] = []selected = seeds or tuple(WILD_WEB_SEEDS.keys())for seed in selected:if seed not in WILD_WEB_SEEDS:raise RuntimeError(f"unknown wild web seed: {seed}")- for target in targets:- metadata = _wild_web_target_metadata(target)- info = target_matrix.get(metadata["binary_key"], {})- path = info.get("path")- for run_id in range(1, runs + 1):- for seed in selected:- url = WILD_WEB_SEEDS[seed]- if not info.get("available") or not path:- row = {- "target": target,- **metadata,- "run": run_id,- "seed": seed,- "url": url,- "category": "error",- "classification_ok": False,- "extraction_ok": False,- "ok": False,- "extraction_failures": ["target-unavailable"],- "extraction_failure_count": 1,- "failure_kind": "target-unavailable",- "error": "target binary unavailable",- }- row["failure_artifact"] = _write_failure_artifacts(suite_dir=suite_dir, row=row, stdout=b"", stderr=b"")- rows.append(row)- continue- proc_env = clear_proxy_env(os.environ)- if target == "chrome":- result = run_chrome_dcl_dump(- Path(path),- url,- cwd=REPO_ROOT,- timeout_seconds=timeout_seconds,- env=proc_env,- )- elif target_is_cdp(target):- result = run_served_cdp_dcl_dump(- target,- Path(path),- url,- cwd=REPO_ROOT,- timeout_seconds=timeout_seconds,- env=proc_env,- )- else:- result = run_process(- _wild_command_for_target(target, Path(path), url, timeout_seconds),- cwd=REPO_ROOT,- timeout_seconds=timeout_seconds + 2,- env=proc_env,- )- category = _classify(result.stdout, result.stderr, result.returncode, result.timed_out)- snapshot = _extract_page_snapshot(result.stdout)- extraction_failures = _wild_web_extraction_failures(seed, snapshot)- classification_ok = category in {"success", "login", "challenge"}- extraction_ok = not extraction_failures- ok = classification_ok and extraction_ok- row = {- "target": target,- **metadata,- "run": run_id,- "seed": seed,- "url": url,- "category": category,- "classification_ok": classification_ok,- "extraction_ok": extraction_ok,- "ok": ok,- "title": snapshot["title"],- "text_length": snapshot["text_length"],- "text_sample": snapshot["text_sample"],- "extraction_failures": extraction_failures,- "extraction_failure_count": len(extraction_failures),- "failure_kind": None if ok else _failure_kind(category, extraction_failures),- "elapsed_ms": result.elapsed_ms,- "returncode": result.returncode,- "timed_out": result.timed_out,- "stdout_bytes": len(result.stdout),- "stderr_bytes": len(result.stderr),- "stdout_tail": result.stdout[-1024:].decode("utf-8", errors="replace"),- "stderr_tail": result.stderr[-1024:].decode("utf-8", errors="replace"),- "peak_pss_bytes": result.resources.get("peak_pss_bytes"),- }- if not ok:- row["failure_artifact"] = _write_failure_artifacts(- suite_dir=suite_dir,- row=row,- stdout=result.stdout,- stderr=result.stderr,- )- elif capture_replay:- row["replay_artifact"] = _write_replay_artifact(- suite_dir=suite_dir,- row=row,- stdout=result.stdout,- )- replay_manifest.append(- {- "target": target,- **metadata,- "run": run_id,- "seed": seed,- "url": url,- "category": category,- "title": snapshot["title"],- "text_length": snapshot["text_length"],- "artifact": row["replay_artifact"],- }- )- rows.append(row)+ proc_env = clear_proxy_env(os.environ)+ metadata_by_target = {+ target: _wild_web_target_metadata(target) for target in targets+ }+ info_by_target = {+ target: target_matrix.get(metadata_by_target[target]["binary_key"], {})+ for target in targets+ }+ scheduled_cases = schedule_public_web_cases(selected, runs=runs)++ def execute_attempt(+ scheduled: PublicWebScheduledCase[str],+ target: str,+ target_order_index: int,+ ) -> tuple[dict[str, Any], dict[str, Any] | None]:+ return _execute_wild_web_attempt(+ suite_dir=suite_dir,+ target=target,+ metadata=metadata_by_target[target],+ info=info_by_target[target],+ run_id=scheduled.run,+ seed=scheduled.case,+ timeout_seconds=timeout_seconds,+ proc_env=proc_env,+ schedule_index=scheduled.schedule_index,+ target_order_index=target_order_index,+ capture_replay=capture_replay,+ )++ executions = PublicWebScheduler[+ str,+ tuple[dict[str, Any], dict[str, Any] | None],+ ](targets).run(scheduled_cases, execute_attempt)+ rows = [row for row, _ in executions]+ replay_manifest = [+ replay_entry+ for _, replay_entry in executions+ if replay_entry is not None+ ]gate_failures = sum(1 for row in rows if row["target"] == gate_target and not row.get("ok"))summary: dict[str, Any] = {@@ -376,33 +325,52 @@ def run_wild_web_suite("runs": runs,"seeds": list(selected),"timeout_seconds": timeout_seconds,+ "snapshot_contract": PUBLIC_WEB_SNAPSHOT_CONTRACT,+ "schedule": "site-paired-rotating-target-order",+ "scheduled_site_groups": len(scheduled_cases),"gate_target": gate_target,"gate_failures": gate_failures,"targets": {},"total_failures": sum(1 for row in rows if not row.get("ok")),"replay_capture": bool(capture_replay),"replay_artifacts": len(replay_manifest),+ "common_success": _successful_attempt_cohort(rows, targets),}for target in targets:target_rows = [row for row in rows if row["target"] == target]- categories: dict[str, int] = {}- failure_kinds: dict[str, int] = {}- for row in target_rows:- categories[row["category"]] = categories.get(row["category"], 0) + 1- failure_kind = row.get("failure_kind")- if isinstance(failure_kind, str) and failure_kind:- failure_kinds[failure_kind] = failure_kinds.get(failure_kind, 0) + 1+ successful_rows = [row for row in target_rows if row.get("ok")]+ value_counts = count_public_web_row_values(target_rows)summary["targets"][target] = {**_wild_web_target_metadata(target),"seeds": len(target_rows),"runs": runs,"passes": sum(1 for row in target_rows if row.get("ok")),"failures": sum(1 for row in target_rows if not row.get("ok")),+ "status_observed_attempts": sum(+ 1 for row in target_rows if row.get("response_status") is not None+ ),+ "status_unobserved_attempts": sum(+ 1 for row in target_rows if row.get("response_status") is None+ ),+ "response_status_sources": value_counts[+ "response_status_sources"+ ],"extraction_failures": sum(int(row.get("extraction_failure_count", 0) or 0) for row in target_rows),- "categories": categories,- "failure_kinds": failure_kinds,+ "categories": value_counts["categories"],+ "failure_kinds": value_counts["failure_kinds"],+ "classification_bases": value_counts["classification_bases"],"elapsed_ms": summarize(row["elapsed_ms"] for row in target_rows if row.get("elapsed_ms") is not None),+ "successful_elapsed_ms": summarize(+ row["elapsed_ms"]+ for row in successful_rows+ if row.get("elapsed_ms") is not None+ ),"peak_pss_bytes": summarize(row["peak_pss_bytes"] for row in target_rows if row.get("peak_pss_bytes") is not None),+ "successful_peak_pss_bytes": summarize(+ row["peak_pss_bytes"]+ for row in successful_rows+ if row.get("peak_pss_bytes") is not None+ ),}write_csv(suite_dir / "raw-runs.csv", rows)write_json(suite_dir / "runs.json", rows)diff --git a/moli-benchmark/tests/test_chrome_dcl.py b/moli-benchmark/tests/test_chrome_dcl.pyindex a7d92b9fc..8dde4898c 100644--- a/moli-benchmark/tests/test_chrome_dcl.py+++ b/moli-benchmark/tests/test_chrome_dcl.py@@ -9,14 +9,20 @@ from pathlib import Pathfrom unittest import mockfrom moli_benchmark.chrome_dcl import (+ CdpDclDumpResult,CdpDclDumpTimeoutError,DEFAULT_CHROME_DCL_USER_AGENT,- _binary_main_resource_body_from_message,+ POST_DCL_SETTLE_MILLISECONDS,+ _POST_DCL_OUTER_HTML_EXPRESSION,+ _POST_DCL_SETTLE_EXPRESSION,+ _binary_main_resource_mime_type_from_message,_chrome_command,+ _navigation_frame_and_loader,_recv_command_response,_recv_until_dcl_or_binary_main_resource,run_chrome_dcl_dump,)+from moli_benchmark.public_web import TOP_SITES_CLASSIFIER, WILD_WEB_CLASSIFIERfrom moli_benchmark.raw_cdp import RawCdpClient, RawCdpError, RawCdpTimeoutError@@ -70,11 +76,23 @@ class _LateCommandResponseClient:return self.late_message+class _QueuedMessageClient:+ def __init__(self, messages: list[dict[str, object]]) -> None:+ self.messages = messages++ async def recv(self) -> dict[str, object]:+ if not self.messages:+ raise AssertionError("CDP receiver exhausted before matching navigation evidence")+ return self.messages.pop(0)++def _document_response_message(mime_type: str,*,- resource_type: str = "Document",+ resource_type: str | None = "Document",+ request_id: str = "REQUEST-1",status: int = 200,+ loader_id: str = "LOADER-1",) -> dict[str, object]:return {"sessionId": "SID-1",@@ -82,12 +100,55 @@ def _document_response_message("params": {"type": resource_type,"frameId": "FRAME-1",- "response": {"mimeType": mime_type, "status": status},+ "loaderId": loader_id,+ "requestId": request_id,+ "response": {+ "mimeType": mime_type,+ "status": status,+ "url": "https://example.test/final",+ },+ },+ }+++def _document_request_message(+ *,+ request_id: str = "REQUEST-1",+ loader_id: str = "LOADER-1",+) -> dict[str, object]:+ return {+ "sessionId": "SID-1",+ "method": "Network.requestWillBeSent",+ "params": {+ "type": "Document",+ "frameId": "FRAME-1",+ "loaderId": loader_id,+ "requestId": request_id,+ "request": {"url": "https://example.test/"},+ },+ }+++def _dcl_lifecycle_message(*, loader_id: str = "LOADER-1") -> dict[str, object]:+ return {+ "sessionId": "SID-1",+ "method": "Page.lifecycleEvent",+ "params": {+ "frameId": "FRAME-1",+ "loaderId": loader_id,+ "name": "DOMContentLoaded",},}class ChromeDclTests(unittest.TestCase):+ def test_post_dcl_snapshot_uses_page_event_loop_settle(self) -> None:+ self.assertEqual(POST_DCL_SETTLE_MILLISECONDS, 50)+ self.assertIn("new Promise", _POST_DCL_SETTLE_EXPRESSION)+ self.assertIn("setTimeout", _POST_DCL_SETTLE_EXPRESSION)+ self.assertIn("50", _POST_DCL_SETTLE_EXPRESSION)+ self.assertIn("document.documentElement.outerHTML", _POST_DCL_OUTER_HTML_EXPRESSION)+def test_chrome_command_uses_non_headless_desktop_user_agent(self) -> None:with tempfile.TemporaryDirectory() as temp_dir:command = _chrome_command(Path("/bin/chromium"), 12345, Path(temp_dir))@@ -163,81 +224,370 @@ class ChromeDclTests(unittest.TestCase):asyncio.run(run())- def test_binary_main_document_response_returns_benchmark_binary_body(self) -> None:- body = _binary_main_resource_body_from_message(+ def test_navigation_result_surfaces_error_text_before_dcl_wait(self) -> None:+ with self.assertRaises(RawCdpError) as raised:+ _navigation_frame_and_loader(+ {+ "result": {+ "frameId": "FRAME-1",+ "errorText": "net::ERR_NAME_NOT_RESOLVED",+ }+ },+ "https://missing.example.test/",+ )++ self.assertEqual(+ str(raised.exception),+ "Page.navigate failed for `https://missing.example.test/`: "+ "net::ERR_NAME_NOT_RESOLVED",+ )++ def test_navigation_result_exposes_frame_and_loader_identity(self) -> None:+ self.assertEqual(+ _navigation_frame_and_loader(+ {"result": {"frameId": "FRAME-1", "loaderId": "LOADER-1"}},+ "https://example.test/",+ ),+ ("FRAME-1", "LOADER-1"),+ )++ def test_binary_main_document_response_returns_mime_evidence(self) -> None:+ mime_type = _binary_main_resource_mime_type_from_message(_document_response_message("application/pdf"),session_id="SID-1",frame_id="FRAME-1",)- self.assertIsNotNone(body)- self.assertTrue(body.startswith("%PDF-"))- self.assertIn("\0", body or "")+ self.assertEqual(mime_type, "application/pdf")def test_binary_main_document_detection_ignores_html_and_subresources(self) -> None:- html_body = _binary_main_resource_body_from_message(+ html_mime_type = _binary_main_resource_mime_type_from_message(_document_response_message("text/html; charset=utf-8"),session_id="SID-1",frame_id="FRAME-1",)- script_pdf_body = _binary_main_resource_body_from_message(+ script_pdf_mime_type = _binary_main_resource_mime_type_from_message(_document_response_message("application/pdf", resource_type="Script"),session_id="SID-1",frame_id="FRAME-1",)- self.assertIsNone(html_body)- self.assertIsNone(script_pdf_body)+ self.assertIsNone(html_mime_type)+ self.assertIsNone(script_pdf_mime_type)def test_binary_main_document_detection_ignores_error_status(self) -> None:- body = _binary_main_resource_body_from_message(+ mime_type = _binary_main_resource_mime_type_from_message(_document_response_message("application/pdf", status=404),session_id="SID-1",frame_id="FRAME-1",)- self.assertIsNone(body)+ self.assertIsNone(mime_type)def test_recv_until_dcl_short_circuits_binary_document_seen_before_dcl(self) -> None:async def run() -> None:- body = await _recv_until_dcl_or_binary_main_resource(+ observation = await _recv_until_dcl_or_binary_main_resource(mock.Mock(),session_id="SID-1",frame_id="FRAME-1",+ expected_loader_id="LOADER-1",deadline=time.perf_counter() + 1.0,seen=[_document_response_message("application/pdf"),+ _dcl_lifecycle_message(),+ ],+ )+ binary_mime_type, response_status, response_mime_type, final_url = observation+ self.assertEqual(binary_mime_type, "application/pdf")+ self.assertEqual(response_status, 200)+ self.assertEqual(response_mime_type, "application/pdf")+ self.assertEqual(final_url, "https://example.test/final")++ asyncio.run(run())++ def test_recv_until_dcl_accepts_main_frame_event_seen_before_command_response(self) -> None:+ async def run() -> None:+ observation = await _recv_until_dcl_or_binary_main_resource(+ mock.Mock(),+ session_id="SID-1",+ frame_id="FRAME-1",+ expected_loader_id="LOADER-1",+ deadline=time.perf_counter() + 1.0,+ seen=[_dcl_lifecycle_message()],+ )+ binary_mime_type, response_status, response_mime_type, final_url = observation+ self.assertIsNone(binary_mime_type)+ self.assertIsNone(response_status)+ self.assertIsNone(response_mime_type)+ self.assertIsNone(final_url)++ asyncio.run(run())++ def test_recv_until_dcl_ignores_events_from_previous_loader(self) -> None:+ async def run() -> None:+ client = _QueuedMessageClient(+ [+ _document_response_message(+ "text/html",+ status=200,+ loader_id="LOADER-NEW",+ ),{"sessionId": "SID-1",- "method": "Page.lifecycleEvent",- "params": {"frameId": "FRAME-1", "name": "DOMContentLoaded"},+ "method": "Page.domContentEventFired",+ "params": {},},+ _dcl_lifecycle_message(loader_id="LOADER-NEW"),+ ]+ )+ observation = await _recv_until_dcl_or_binary_main_resource( # type: ignore[arg-type]+ client,+ session_id="SID-1",+ frame_id="FRAME-1",+ expected_loader_id="LOADER-NEW",+ deadline=time.perf_counter() + 1.0,+ seen=[+ _document_response_message(+ "text/html",+ status=502,+ loader_id="LOADER-OLD",+ ),+ _dcl_lifecycle_message(loader_id="LOADER-OLD"),],)- self.assertIsNotNone(body)- self.assertTrue((body or "").startswith("%PDF-"))++ self.assertEqual(+ observation,+ (None, 200, "text/html", "https://example.test/final"),+ )+ self.assertEqual(client.messages, [])asyncio.run(run())- def test_recv_until_dcl_accepts_main_frame_event_seen_before_command_response(self) -> None:+ def test_recv_until_dcl_retains_html_main_document_status(self) -> None:async def run() -> None:- body = await _recv_until_dcl_or_binary_main_resource(+ observation = await _recv_until_dcl_or_binary_main_resource(mock.Mock(),session_id="SID-1",frame_id="FRAME-1",+ expected_loader_id="LOADER-1",deadline=time.perf_counter() + 1.0,seen=[- {- "sessionId": "SID-1",- "method": "Page.lifecycleEvent",- "params": {"frameId": "FRAME-1", "name": "DOMContentLoaded"},- }+ _document_response_message("text/html", status=502),+ _dcl_lifecycle_message(),+ ],+ )+ binary_mime_type, response_status, response_mime_type, final_url = observation+ self.assertIsNone(binary_mime_type)+ self.assertEqual(response_status, 502)+ self.assertEqual(response_mime_type, "text/html")+ self.assertEqual(final_url, "https://example.test/final")++ asyncio.run(run())++ def test_recv_until_dcl_correlates_response_without_resource_type(self) -> None:+ async def run() -> None:+ observation = await _recv_until_dcl_or_binary_main_resource(+ mock.Mock(),+ session_id="SID-1",+ frame_id="FRAME-1",+ expected_loader_id="LOADER-1",+ deadline=time.perf_counter() + 1.0,+ seen=[+ _document_request_message(),+ _document_response_message(+ "text/html",+ resource_type=None,+ status=400,+ ),+ _dcl_lifecycle_message(),+ ],+ )+ binary_mime_type, response_status, response_mime_type, final_url = observation+ self.assertIsNone(binary_mime_type)+ self.assertEqual(response_status, 400)+ self.assertEqual(response_mime_type, "text/html")+ self.assertEqual(final_url, "https://example.test/final")++ asyncio.run(run())++ def test_recv_until_dcl_ignores_untracked_response_without_resource_type(self) -> None:+ async def run() -> None:+ observation = await _recv_until_dcl_or_binary_main_resource(+ mock.Mock(),+ session_id="SID-1",+ frame_id="FRAME-1",+ expected_loader_id="LOADER-1",+ deadline=time.perf_counter() + 1.0,+ seen=[+ _document_request_message(),+ _document_response_message(+ "application/json",+ resource_type=None,+ request_id="SUBRESOURCE-1",+ status=503,+ ),+ _dcl_lifecycle_message(),],)- self.assertIsNone(body)+ binary_mime_type, response_status, response_mime_type, final_url = observation+ self.assertIsNone(binary_mime_type)+ self.assertIsNone(response_status)+ self.assertIsNone(response_mime_type)+ self.assertIsNone(final_url)asyncio.run(run())+ def test_recv_until_dcl_keeps_status_for_latest_document_request(self) -> None:+ async def run() -> None:+ observation = await _recv_until_dcl_or_binary_main_resource(+ mock.Mock(),+ session_id="SID-1",+ frame_id="FRAME-1",+ expected_loader_id="LOADER-1",+ deadline=time.perf_counter() + 1.0,+ seen=[+ _document_request_message(request_id="OLD"),+ _document_request_message(request_id="NEW"),+ _document_response_message(+ "text/html",+ request_id="OLD",+ status=500,+ ),+ _document_response_message(+ "text/html",+ resource_type=None,+ request_id="NEW",+ status=200,+ ),+ _dcl_lifecycle_message(),+ ],+ )+ binary_mime_type, response_status, response_mime_type, final_url = observation+ self.assertIsNone(binary_mime_type)+ self.assertEqual(response_status, 200)+ self.assertEqual(response_mime_type, "text/html")+ self.assertEqual(final_url, "https://example.test/final")++ asyncio.run(run())++ def test_chrome_runner_exposes_main_document_status(self) -> None:+ process = _FakeProcess()++ def terminate(fake_process: _FakeProcess) -> None:+ fake_process.returncode = -15++ with (+ mock.patch("moli_benchmark.chrome_dcl.subprocess.Popen", return_value=process),+ mock.patch(+ "moli_benchmark.chrome_dcl._dump_dcl_html",+ return_value=CdpDclDumpResult(+ body="<html><body>gateway error</body></html>",+ response_status=502,+ response_mime_type="text/html",+ main_document_body_capture="dom-snapshot",+ final_url="https://example.test/final",+ ),+ ),+ mock.patch(+ "moli_benchmark.chrome_dcl._terminate_process_group",+ side_effect=terminate,+ ),+ ):+ result = run_chrome_dcl_dump(+ Path("/bin/chromium"),+ "https://example.test/",+ timeout_seconds=1.0,+ sample_resources=False,+ )++ self.assertEqual(result.returncode, 0)+ self.assertEqual(result.response_status, 502)+ self.assertEqual(result.response_mime_type, "text/html")+ self.assertEqual(result.main_document_body_capture, "dom-snapshot")+ self.assertEqual(result.final_url, "https://example.test/final")+ self.assertIn(b"gateway error", result.stdout)++ def test_chrome_runner_classifies_navigation_error_without_timeout(self) -> None:+ process = _FakeProcess()++ def terminate(fake_process: _FakeProcess) -> None:+ fake_process.returncode = -15++ with (+ mock.patch("moli_benchmark.chrome_dcl.subprocess.Popen", return_value=process),+ mock.patch(+ "moli_benchmark.chrome_dcl._dump_dcl_html",+ side_effect=RawCdpError(+ "Page.navigate failed for `https://missing.example.test/`: "+ "net::ERR_NAME_NOT_RESOLVED"+ ),+ ),+ mock.patch(+ "moli_benchmark.chrome_dcl._terminate_process_group",+ side_effect=terminate,+ ),+ ):+ result = run_chrome_dcl_dump(+ Path("/bin/chromium"),+ "https://missing.example.test/",+ timeout_seconds=1.0,+ sample_resources=False,+ )++ self.assertFalse(result.timed_out)+ self.assertEqual(result.returncode, 1)+ self.assertIn(b"net::ERR_NAME_NOT_RESOLVED", result.stderr)+ for classifier in (TOP_SITES_CLASSIFIER, WILD_WEB_CLASSIFIER):+ with self.subTest(policy=classifier.policy):+ self.assertEqual(+ classifier.classify_output(+ stdout=result.stdout,+ stderr=result.stderr,+ returncode=result.returncode,+ timed_out=result.timed_out,+ ),+ "network-error",+ )++ def test_chrome_runner_does_not_fabricate_binary_response_body(self) -> None:+ process = _FakeProcess()++ def terminate(fake_process: _FakeProcess) -> None:+ fake_process.returncode = -15++ with (+ mock.patch("moli_benchmark.chrome_dcl.subprocess.Popen", return_value=process),+ mock.patch(+ "moli_benchmark.chrome_dcl._dump_dcl_html",+ return_value=CdpDclDumpResult(+ body="",+ response_status=200,+ response_mime_type="application/pdf",+ main_document_body_capture="response-headers-only",+ final_url="https://example.test/document.pdf",+ ),+ ),+ mock.patch(+ "moli_benchmark.chrome_dcl._terminate_process_group",+ side_effect=terminate,+ ),+ ):+ result = run_chrome_dcl_dump(+ Path("/bin/chromium"),+ "https://example.test/document.pdf",+ timeout_seconds=1.0,+ sample_resources=False,+ )++ self.assertEqual(result.returncode, 0)+ self.assertEqual(result.stdout, b"")+ self.assertEqual(result.response_mime_type, "application/pdf")+ self.assertEqual(+ result.main_document_body_capture,+ "response-headers-only",+ )+def test_chrome_runner_records_raw_cdp_deadline_as_timeout(self) -> None:process = _FakeProcess()diff --git a/moli-benchmark/tests/test_html_report.py b/moli-benchmark/tests/test_html_report.pyindex b375987ef..6958e4f67 100644--- a/moli-benchmark/tests/test_html_report.py+++ b/moli-benchmark/tests/test_html_report.py@@ -49,6 +49,13 @@ class HtmlReportTests(unittest.TestCase):self.assertIn("render-compare/runs.json", paths["render-compare"])self.assertIn("render-compare/baseline-sites.csv", paths["render-compare"])+ def test_artifact_index_includes_top_sites_site_outcomes(self) -> None:+ paths = _artifact_paths_by_suite(+ [{"suite": "top-sites", "total_failures": 0}]+ )++ self.assertIn("top-sites/site-outcomes.json", paths["top-sites"])+def test_cdp_session_trace_link_uses_nested_summary_fallback(self) -> None:paths = _artifact_paths_by_suite([@@ -225,6 +232,36 @@ class HtmlReportTests(unittest.TestCase):self.assertIn("const attempts = passes + failures;", document)self.assertIn("result.pages ?? result.seeds ?? result.sites ?? result.runs", document)self.assertIn("result.successes ?? result.categories?.success ?? result.passes", document)+ self.assertIn("counted pass rate", document)+ self.assertIn("raw pass rate", document)+ self.assertIn("successful p50 / p90 / p95 ms", document)+ self.assertIn("common p50 / p90 / p95 ms", document)+ self.assertIn("result?.successful_elapsed_ms || result?.elapsed_ms", document)+ self.assertIn("summary?.common_success?.targets?.[target]", document)+ self.assertIn("site stability", document)+ self.assertIn("HTTP status observed", document)+ self.assertIn("not repeat-validated", document)+ self.assertIn("Pairwise site outcomes", document)+ self.assertIn("SINGLE SAMPLE", document)++ def test_chartjs_report_renders_every_aggregate_summary_and_prioritizes_top_sites(+ self,+ ) -> None:+ document = _chartjs_document(+ {+ "output_dir": "/tmp/report",+ "versions": {},+ "summaries": [+ {"suite": "wild-web", "targets": {"moli": {}}},+ {"suite": "top-sites", "targets": {"moli": {}}},+ ],+ }+ )++ self.assertIn("function horizontalSummaries()", document)+ self.assertIn("'top-sites', 'wild-web'", document)+ self.assertIn("summaries.map(summary =>", document)+ self.assertIn('class="comparison-summary"', document)if __name__ == "__main__":diff --git a/moli-benchmark/tests/test_public_web.py b/moli-benchmark/tests/test_public_web.pynew file mode 100644index 000000000..987662e44--- /dev/null+++ b/moli-benchmark/tests/test_public_web.py@@ -0,0 +1,188 @@+from __future__ import annotations++import unittest++from moli_benchmark.process import ProcessResult+from moli_benchmark.public_web import (+ POST_DCL_SETTLE_MILLISECONDS,+ POST_DCL_WAIT_SCRIPT,+ PUBLIC_WEB_SNAPSHOT_CONTRACT,+ TOP_SITES_ARTIFACT_POLICY,+ TOP_SITES_CLASSIFIER,+ WILD_WEB_CLASSIFIER,+ PublicWebAttempt,+ PublicWebResult,+ PublicWebScheduler,+ resolve_main_document_status,+ schedule_public_web_cases,+)+++class PublicWebTests(unittest.TestCase):+ def test_post_dcl_wait_script_encodes_the_reported_snapshot_contract(self) -> None:+ self.assertEqual(POST_DCL_SETTLE_MILLISECONDS, 50)+ self.assertIn("setTimeout", POST_DCL_WAIT_SCRIPT)+ self.assertIn("}, 50)", POST_DCL_WAIT_SCRIPT)+ self.assertIn("PostDclReady", POST_DCL_WAIT_SCRIPT)+ self.assertIn("50 ms", PUBLIC_WEB_SNAPSHOT_CONTRACT)++ def test_typed_cli_fetch_readiness_timeout_is_shared_by_both_suites(self) -> None:+ stderr = b"""+Error: failed to fetch `https://example.test/`+Reason: fetch readiness timed out after 30000 ms while waiting for DOMContentLoaded+"""++ for classifier in (TOP_SITES_CLASSIFIER, WILD_WEB_CLASSIFIER):+ with self.subTest(policy=classifier.policy):+ self.assertEqual(+ classifier.classify_output(+ stdout=b"",+ stderr=stderr,+ returncode=1,+ timed_out=False,+ ),+ "timeout",+ )++ def test_protocol_status_is_preferred(self) -> None:+ self.assertEqual(+ resolve_main_document_status(+ 200,+ b"lifecycle target document `https://example.test/` returned 502 Bad Gateway",+ 1,+ ),+ (200, "protocol"),+ )++ def test_nonzero_cli_diagnostic_supplies_main_document_status(self) -> None:+ self.assertEqual(+ resolve_main_document_status(+ None,+ b"Reason: lifecycle target document `https://example.test/` "+ b"returned 429 Too Many Requests",+ 1,+ ),+ (429, "cli-diagnostic"),+ )+ self.assertEqual(+ resolve_main_document_status(+ None,+ b"HTTP request `https://example.test/` returned 404 Not Found",+ 1,+ ),+ (404, "cli-diagnostic"),+ )++ def test_success_and_subresource_diagnostics_are_not_promoted(self) -> None:+ diagnostic = b"script subresource `https://cdn.example.test/a.js` returned 500"+ self.assertEqual(resolve_main_document_status(None, diagnostic, 1), (None, None))+ self.assertEqual(+ resolve_main_document_status(+ None,+ b"lifecycle target document `https://example.test/` returned 500",+ 0,+ ),+ (None, None),+ )++ def test_attempt_result_owns_common_evidence_fields(self) -> None:+ attempt = PublicWebAttempt.start(+ target="chrome",+ metadata={"engine": "chrome", "driver": "cdp-dcl"},+ target_info={"available": True, "path": "/bin/chromium"},+ run=2,+ case_fields={"seed": "example"},+ url="https://example.test/",+ schedule_index=3,+ target_order_index=2,+ artifact_stem="chrome-run-2-example",+ )+ process = ProcessResult(+ command=["chromium"],+ returncode=0,+ elapsed_ms=12.5,+ stdout=b"<title>Example</title><body>Hello world</body>",+ stderr=b"",+ timed_out=False,+ resources={"peak_pss_bytes": 123},+ response_status=200,+ final_url="https://example.test/final",+ )++ result = PublicWebResult.capture(+ attempt,+ process,+ classifier=TOP_SITES_CLASSIFIER,+ )+ row = {+ **result.base_row(),+ **result.evidence_fields(policy=TOP_SITES_ARTIFACT_POLICY),+ }++ self.assertEqual(row["seed"], "example")+ self.assertEqual(row["response_status_source"], "protocol")+ self.assertEqual(row["final_url"], "https://example.test/final")+ self.assertEqual(row["title"], "Example")+ self.assertEqual(row["peak_pss_bytes"], 123)+ self.assertEqual(len(row["stdout_sha256"]), 64)++ def test_scheduler_rotates_targets_and_restores_schedule_order(self) -> None:+ scheduled = schedule_public_web_cases(("first", "second"), runs=1)+ scheduler = PublicWebScheduler[str, tuple[int, str, int]](+ ("moli", "lightpanda", "chrome"),+ parallelism=2,+ )++ results = scheduler.run(+ scheduled,+ lambda case, target, target_order_index: (+ case.schedule_index,+ target,+ target_order_index,+ ),+ )++ self.assertEqual(+ results,+ [+ (0, "moli", 1),+ (0, "lightpanda", 2),+ (0, "chrome", 3),+ (1, "lightpanda", 1),+ (1, "chrome", 2),+ (1, "moli", 3),+ ],+ )++ def test_scheduler_changes_each_site_first_target_across_runs(self) -> None:+ cases = tuple(f"site-{index}" for index in range(20))+ scheduled = schedule_public_web_cases(cases, runs=3)+ scheduler = PublicWebScheduler[str, tuple[str, int, str, int]](+ ("moli", "chrome"),+ )++ results = scheduler.run(+ scheduled,+ lambda case, target, target_order_index: (+ case.case,+ case.run,+ target,+ target_order_index,+ ),+ )+ first_targets = {+ (case, run): target+ for case, run, target, target_order_index in results+ if target_order_index == 1+ }++ for case_index, case in enumerate(cases):+ expected = (+ ("moli", "chrome", "moli")+ if case_index % 2 == 0+ else ("chrome", "moli", "chrome")+ )+ self.assertEqual(+ tuple(first_targets[(case, run)] for run in range(1, 4)),+ expected,+ )diff --git a/moli-benchmark/tests/test_top_sites.py b/moli-benchmark/tests/test_top_sites.pyindex 1080412f6..a87fb8ab1 100644--- a/moli-benchmark/tests/test_top_sites.py+++ b/moli-benchmark/tests/test_top_sites.py@@ -7,15 +7,22 @@ from pathlib import Pathfrom unittest.mock import patchfrom moli_benchmark.process import ProcessResult+from moli_benchmark.public_web import POST_DCL_WAIT_SCRIPTfrom moli_benchmark.top_sites import (COMPOSITE_TOP_SITES_SOURCES,DEFAULT_TOP_SITES_SOURCE,+ TOP_SITES_FIXTURE_ROOT,TOP_SITES_PROFILES,TOP_SITES_SOURCES,+ _build_site_outcomes,_classify,_default_top_sites_parallelism,_elapsed_failure_reached_timeout,+ _extract_title_and_text,_ok_categories,+ _pairwise_site_comparisons,+ _rotated_target_order,+ _site_unreachable_exclusions,_top_command_for_target,_top_sites_target_metadata,load_top_sites_entries,@@ -26,6 +33,163 @@ from moli_benchmark.top_sites import (class TopSitesTests(unittest.TestCase):+ def test_target_order_rotates_per_site_group(self) -> None:+ targets = ("moli", "lightpanda", "chrome")++ self.assertEqual(_rotated_target_order(targets, 0), targets)+ self.assertEqual(+ _rotated_target_order(targets, 1),+ ("lightpanda", "chrome", "moli"),+ )+ self.assertEqual(+ _rotated_target_order(targets, 2),+ ("chrome", "moli", "lightpanda"),+ )++ def test_site_unreachable_exclusion_requires_transport_failure_from_every_target(+ self,+ ) -> None:+ rows = [+ {+ "target": "moli",+ "domain": "dead.example",+ "ok": False,+ "category": "network-error",+ "failure_kind": "network-error",+ },+ {+ "target": "chrome",+ "domain": "dead.example",+ "ok": False,+ "category": "process-error",+ "failure_kind": "process-error",+ },+ ]++ self.assertEqual(+ _site_unreachable_exclusions(rows, ("moli", "chrome")),+ {},+ )+ rows[1]["category"] = "timeout"+ rows[1]["failure_kind"] = "timeout"+ self.assertEqual(+ _site_unreachable_exclusions(rows, ("moli", "chrome")),+ {"dead.example": "site-unreachable"},+ )++ def test_site_outcomes_keep_header_only_binary_observations_neutral(self) -> None:+ outcomes = _build_site_outcomes(+ [+ {+ "target": "chrome",+ "rank": 1,+ "domain": "document.example",+ "url": "https://document.example/file.pdf",+ "run": 1,+ "ok": False,+ "comparable": False,+ "category": "binary-response-headers",+ "response_status": 200,+ }+ ]+ )++ self.assertEqual(outcomes[0]["outcome"], "not-comparable")+ self.assertEqual(outcomes[0]["attempts"], 0)+ self.assertEqual(outcomes[0]["raw_attempts"], 1)+ self.assertEqual(outcomes[0]["non_comparable_attempts"], 1)+ self.assertIsNone(outcomes[0]["pass_rate_percent"])++ def test_site_outcomes_separate_repeat_stability_from_attempt_passes(self) -> None:+ rows: list[dict[str, object]] = []+ for run, left_ok, right_ok in (+ (1, True, False),+ (2, True, False),+ (3, True, False),+ ):+ rows.extend(+ [+ {+ "target": "left",+ "rank": 1,+ "domain": "stable.example",+ "url": "https://stable.example/",+ "run": run,+ "ok": left_ok,+ "category": "success-content",+ "response_status": 200,+ },+ {+ "target": "right",+ "rank": 1,+ "domain": "stable.example",+ "url": "https://stable.example/",+ "run": run,+ "ok": right_ok,+ "category": "http-error",+ "response_status": 500,+ },+ {+ "target": "left",+ "rank": 2,+ "domain": "flaky.example",+ "url": "https://flaky.example/",+ "run": run,+ "ok": run != 2,+ "category": "success-content" if run != 2 else "timeout",+ "response_status": 200 if run != 2 else None,+ },+ {+ "target": "right",+ "rank": 2,+ "domain": "flaky.example",+ "url": "https://flaky.example/",+ "run": run,+ "ok": True,+ "category": "success-content",+ "response_status": 200,+ },+ ]+ )++ outcomes = _build_site_outcomes(rows) # type: ignore[arg-type]+ comparisons = _pairwise_site_comparisons(+ outcomes,+ ("left", "right"),+ runs=3,+ )+ by_target_rank = {+ (row["target"], row["rank"]): row for row in outcomes+ }++ self.assertEqual(by_target_rank[("left", 1)]["outcome"], "all-pass")+ self.assertEqual(by_target_rank[("right", 1)]["outcome"], "all-fail")+ self.assertEqual(by_target_rank[("left", 2)]["outcome"], "flaky")+ self.assertAlmostEqual(+ by_target_rank[("left", 2)]["pass_rate_percent"],+ 200 / 3,+ )+ self.assertTrue(comparisons[0]["repeat_validated"])+ self.assertEqual(comparisons[0]["left_all_pass_right_all_fail"], 1)+ self.assertEqual(comparisons[0]["flaky_or_incomplete"], 1)+ self.assertEqual(+ comparisons[0]["left_only_sites"],+ [{"rank": 1, "domain": "stable.example"}],+ )++ def test_builtin_seed_lists_are_benchmark_local(self) -> None:+ for source in TOP_SITES_SOURCES:+ resolved, path = resolve_top_sites_source(source, None)+ self.assertEqual(resolved, source)+ self.assertTrue(path.is_relative_to(TOP_SITES_FIXTURE_ROOT))+ self.assertEqual(path.suffix, ".csv")+ self.assertTrue(path.is_file(), f"missing benchmark seed fixture: {path}")++ for source in COMPOSITE_TOP_SITES_SOURCES:+ resolved, path = resolve_top_sites_source(source, None)+ self.assertEqual(resolved, source)+ self.assertEqual(path.parent, TOP_SITES_FIXTURE_ROOT)+def test_parse_top_sites_list_reads_top_100_section(self) -> None:with tempfile.TemporaryDirectory() as temp_dir:path = Path(temp_dir) / "list.md"@@ -56,6 +220,21 @@ class TopSitesTests(unittest.TestCase):entries = parse_top_sites_list(path)self.assertEqual(entries, [(1, "first.example")])+ def test_parse_top_sites_list_reads_csv(self) -> None:+ with tempfile.TemporaryDirectory() as temp_dir:+ path = Path(temp_dir) / "list.csv"+ path.write_text(+ "rank,target\n"+ "1,example.com\n"+ '2,"https://example.com/search?q=one,two"\n',+ encoding="utf-8",+ )+ entries = parse_top_sites_list(path)+ self.assertEqual(+ entries,+ [(1, "example.com"), (2, "https://example.com/search?q=one,two")],+ )+def test_legacy_encoding_source_is_registered_and_parses(self) -> None:source, path = resolve_top_sites_source("legacy-encoding", None)entries = parse_top_sites_list(path)@@ -75,6 +254,7 @@ class TopSitesTests(unittest.TestCase):self.assertNotIn("--resource", command)self.assertIn("--wait-until", command)self.assertEqual(command[command.index("--wait-until") + 1], "domcontentloaded")+ self.assertEqual(command[command.index("--wait-script") + 1], POST_DCL_WAIT_SCRIPT)self.assertEqual(command[command.index("--timeout") + 1], "12000")self.assertEqual(command[command.index("--http-timeout") + 1], "12000")self.assertEqual(command[-1], "https://example.test/")@@ -87,13 +267,29 @@ class TopSitesTests(unittest.TestCase):self.assertLess(command.index("--layout"), command.index("--dump"))self.assertLess(command.index("--resource"), command.index("--dump"))self.assertEqual(command[command.index("--wait-until") + 1], "domcontentloaded")+ self.assertEqual(command[command.index("--wait-script") + 1], POST_DCL_WAIT_SCRIPT)self.assertEqual(command[-1], "https://example.test/")+ def test_top_command_for_moli_omits_page_waits_for_explicit_pdf(self) -> None:+ command = _top_command_for_target(+ "moli",+ Path("/bin/moli"),+ "https://example.test/report.PDF?download=1",+ 12.0,+ )++ self.assertNotIn("--wait-until", command)+ self.assertNotIn("--wait-script", command)+ self.assertEqual(command[command.index("--timeout") + 1], "12000")+ self.assertEqual(command[command.index("--http-timeout") + 1], "12000")+ self.assertEqual(command[-1], "https://example.test/report.PDF?download=1")+def test_top_command_for_lightpanda_aligns_wait_and_http_timeouts(self) -> None:command = _top_command_for_target("lightpanda", Path("/bin/lightpanda"), "https://example.test/", 12.0)self.assertEqual(command[0], "/bin/lightpanda")self.assertIn("--wait-until", command)self.assertEqual(command[command.index("--wait-until") + 1], "domcontentloaded")+ self.assertEqual(command[command.index("--wait-script") + 1], POST_DCL_WAIT_SCRIPT)self.assertEqual(command[command.index("--wait-ms") + 1], "12000")self.assertEqual(command[command.index("--http-timeout") + 1], "12000")self.assertEqual(command[command.index("--terminate-ms") + 1], "12000")@@ -244,6 +440,154 @@ class TopSitesTests(unittest.TestCase):self.assertEqual(_classify(b"", b"HTTP request `https://example.test/` returned 404 Not Found", 1, False, 256), "not-found")self.assertEqual(_classify(b"", b"", None, True, 256), "timeout")+ def test_classify_recognizes_lightpanda_fatal_fetch_timeout(self) -> None:+ stderr = (+ b'$scope=app $level=fatal $msg="fetch error" err=Timeout '+ b"url=https://example.test/data.json"+ )++ self.assertEqual(_classify(b"", stderr, 0, False, 256), "timeout")++ def test_classify_preserves_cli_main_document_status_and_dns_reason(self) -> None:+ self.assertEqual(+ _classify(+ b"",+ b"Reason: lifecycle target document `https://example.test/` "+ b"returned 429 Too Many Requests",+ 1,+ False,+ 256,+ ),+ "http-error",+ )+ self.assertEqual(+ _classify(+ b"",+ b"Reason: lifecycle target document `https://example.test/` "+ b"returned 401 Unauthorized",+ 1,+ False,+ 256,+ ),+ "blocked-or-forbidden",+ )+ self.assertEqual(+ _classify(+ b"",+ b"Reason: failed to resolve example.test:443: failed to lookup "+ b"address information",+ 1,+ False,+ 256,+ ),+ "network-error",+ )++ def test_classify_uses_main_document_status_before_body_size(self) -> None:+ large_error_body = (+ b"<html><head><title>upstream response</title></head><body>"+ + b"error details " * 100+ + b"</body></html>"+ )++ for status in (400, 500, 502):+ with self.subTest(status=status):+ self.assertEqual(+ _classify(+ large_error_body,+ b"",+ 0,+ False,+ 256,+ response_status=status,+ ),+ "http-error",+ )+ self.assertEqual(+ _classify(+ large_error_body,+ b"",+ 0,+ False,+ 256,+ response_status=403,+ ),+ "blocked-or-forbidden",+ )+ self.assertEqual(+ _classify(+ large_error_body,+ b"",+ 0,+ False,+ 256,+ response_status=404,+ ),+ "not-found",+ )++ def test_classify_infers_http_error_documents_when_cli_hides_status(self) -> None:+ cases = [+ (+ b"<html><head><title>Error: 500 | Service</title></head>"+ b"<body>There was a technical error.</body></html>",+ "http-error",+ ),+ (+ b"<html><head><title>502 Bad Gateway</title></head>"+ b"<body>Origin unavailable.</body></html>",+ "http-error",+ ),+ (+ b"<html><head><title></title></head><body><pre>"+ b'{"error":{"message":"invalid query"}}'+ b"</pre></body></html>",+ "http-error",+ ),+ (+ b"<html><head><title>ResearchGate - Temporarily Unavailable</title></head>"+ b"<body>Access restricted due to unusual activity.</body></html>",+ "blocked-or-forbidden",+ ),+ ]++ for html, expected in cases:+ with self.subTest(expected=expected, html=html):+ self.assertEqual(_classify(html, b"", 0, False, 256), expected)++ def test_classify_distinguishes_tls_verification_from_human_verification(self) -> None:+ tls_error = (+ b"<html><body><h1>Navigation failed</h1>"+ b"<p>Reason: PeerFailedVerification</p></body></html>"+ )+ human_verification = (+ "<html><body>环境异常 当前环境异常,完成验证后即可继续访问。去验证</body></html>".encode()+ )++ self.assertEqual(_classify(tls_error, b"", 0, False, 256), "network-error")+ self.assertEqual(+ _classify(human_verification, b"", 0, False, 256),+ "captcha-or-verification",+ )++ def test_classify_recognizes_chinese_not_found_document(self) -> None:+ html = (+ "<html><head><title>出错了!</title></head>"+ "<body>404 很抱歉,你访问的页面不存在或暂时无法访问!</body></html>"+ ).encode()++ self.assertEqual(_classify(html, b"", 0, False, 256), "not-found")++ def test_visible_text_excludes_document_head(self) -> None:+ snapshot = _extract_title_and_text(+ b"<html><head><title>A very long title</title></head>"+ b"<body>short body</body></html>"+ )++ self.assertEqual(snapshot["title"], "A very long title")+ self.assertEqual(snapshot["text_sample"], "short body")+ self.assertEqual(snapshot["text_length"], len("short body"))+def test_classify_treats_browser_network_error_pages_as_failures(self) -> None:html = (b"<html><head><title>www.example.test</title></head><body>"@@ -316,6 +660,21 @@ class TopSitesTests(unittest.TestCase):stderr = b'$scope=frame $level=error $msg="navigate failed" err=OperationTimedout type=root'self.assertEqual(_classify(html, stderr, 0, False, 256), "timeout")+ def test_classify_treats_lightpanda_navigation_transport_pages_as_network_errors(+ self,+ ) -> None:+ for reason in ("CouldntResolveHost", "Http2Stream"):+ with self.subTest(reason=reason):+ html = (+ "<!doctype html><body><h1>Navigation failed</h1>"+ f"<p>Reason: {reason}</p></body>"+ ).encode()+ stderr = f'$msg="navigate failed" err={reason} type=root'.encode()+ self.assertEqual(+ _classify(html, stderr, 0, False, 256),+ "network-error",+ )+def test_classify_ignores_lightpanda_subresource_operation_timeout_when_main_content_loaded(self,) -> None:@@ -398,6 +757,28 @@ class TopSitesTests(unittest.TestCase):self.assertEqual(_classify(vercel_checkpoint, b"", 0, False, 256), "js-challenge")self.assertEqual(_classify(bot_or_not, b"", 0, False, 256), "captcha-or-verification")+ def test_classify_recognizes_safeline_and_privacy_tile_challenges(self) -> None:+ safeline = (+ b"<html><body>Security Detection Powered By SafeLine WAF "+ + b"Confirm " * 100+ + b"</body></html>"+ )+ privacy_tiles = (+ b"<html><body>Select the 2 matching tiles A B C D Submit "+ b"Powered and protected by Privacy "+ + b"challenge " * 100+ + b"</body></html>"+ )++ self.assertEqual(+ _classify(safeline, b"", 0, False, 256),+ "blocked-or-forbidden",+ )+ self.assertEqual(+ _classify(privacy_tiles, b"", 0, False, 256),+ "captcha-or-verification",+ )+def test_classify_treats_c2wf_probe_shell_as_js_challenge(self) -> None:probe_v1 = b"""<!DOCTYPE html><html><head><meta charset="UTF-8">@@ -490,7 +871,7 @@ if(window.EOJsChallengeSDK){new window.EOJsChallengeSDK({callback:function(){}})self.assertEqual(_classify(pdf, b"", 0, False, 256), "success-binary-content")self.assertIn("success-binary-content", _ok_categories())- def test_classify_treats_raw_binary_body_timeout_as_binary_main_resource(+ def test_classify_keeps_partial_raw_binary_body_timeout_as_network_error(self,) -> None:stderr = (@@ -503,9 +884,24 @@ if(window.EOJsChallengeSDK){new window.EOJsChallengeSDK({callback:function(){}}))self.assertEqual(_classify(b"", stderr, 1, False, 256),- "success-binary-main-resource",+ "network-error",+ )+ self.assertNotIn("success-binary-main-resource", _ok_categories())++ def test_classify_marks_cdp_binary_headers_as_not_a_body_success(self) -> None:+ self.assertEqual(+ _classify(+ b"",+ b"",+ 0,+ False,+ 256,+ response_status=200,+ main_document_body_capture="response-headers-only",+ ),+ "binary-response-headers",)- self.assertIn("success-binary-main-resource", _ok_categories())+ self.assertNotIn("binary-response-headers", _ok_categories())def test_classify_keeps_raw_binary_header_timeout_as_network_error(self) -> None:stderr = (@@ -565,20 +961,230 @@ if(window.EOJsChallengeSDK){new window.EOJsChallengeSDK({callback:function(){}})failure_files = sorted(p.name for p in (output_dir / "top-sites" / "failures").iterdir())summary_payload = (output_dir / "top-sites" / "summary.json").read_text(encoding="utf-8")run_rows = json.loads((output_dir / "top-sites" / "runs.json").read_text(encoding="utf-8"))+ site_outcomes = json.loads(+ (output_dir / "top-sites" / "site-outcomes.json").read_text(+ encoding="utf-8"+ )+ )self.assertEqual(summary["site_count"], 2)self.assertEqual(summary["gate_failures"], 1)self.assertEqual(summary["targets"]["moli"]["passes"], 1)self.assertEqual(summary["targets"]["moli"]["failures"], 1)+ self.assertEqual(summary["targets"]["moli"]["raw_observations"], 2)+ self.assertEqual(summary["targets"]["moli"]["raw_comparable_attempts"], 2)+ self.assertEqual(summary["targets"]["moli"]["raw_passes"], 1)+ self.assertEqual(summary["targets"]["moli"]["raw_failures"], 1)+ self.assertEqual(summary["targets"]["moli"]["raw_pass_rate_percent"], 50.0)+ self.assertEqual(summary["targets"]["moli"]["pass_rate_percent"], 50.0)+ self.assertEqual(summary["targets"]["moli"]["elapsed_ms"]["count"], 2)+ self.assertEqual(+ summary["targets"]["moli"]["successful_elapsed_ms"]["count"],+ 1,+ )+ self.assertEqual(summary["common_success"]["attempts"], 1)+ self.assertEqual(+ summary["common_success"]["targets"]["moli"]["elapsed_ms"]["count"],+ 1,+ )+ self.assertEqual(summary["targets"]["moli"]["unique_sites"], 2)+ self.assertEqual(summary["targets"]["moli"]["all_pass_sites"], 1)+ self.assertEqual(summary["targets"]["moli"]["all_fail_sites"], 1)+ self.assertEqual(summary["targets"]["moli"]["flaky_sites"], 0)+ self.assertFalse(summary["repeat_validated"])self.assertEqual(summary["targets"]["moli"]["failure_kinds"], {"process-error": 1})self.assertEqual(summary["targets"]["moli"]["categories"]["success-content"], 1)self.assertEqual(summary["targets"]["moli"]["peak_rss_bytes"]["max"], 2048.0)self.assertEqual(summary["chrome_parallelism"], 2)+ self.assertEqual(summary["schedule"], "site-paired-rotating-target-order")+ self.assertEqual(summary["scheduled_site_groups"], 2)self.assertEqual(run_rows[0]["command"][0], "/bin/echo")self.assertEqual(run_rows[0]["peak_rss_bytes"], 2048)+ self.assertEqual(run_rows[0]["schedule_index"], 0)+ self.assertEqual(run_rows[0]["target_order_index"], 1)+ self.assertTrue(run_rows[0]["started_at"].endswith("Z"))+ self.assertTrue(run_rows[0]["finished_at"].endswith("Z"))+ self.assertEqual(len(run_rows[0]["stdout_sha256"]), 64)+ self.assertEqual(len(run_rows[0]["stderr_sha256"]), 64)+ self.assertTrue(run_rows[0]["stdout_head"].startswith("<html>"))+ self.assertEqual(len(site_outcomes), 2)+ self.assertEqual(+ {row["outcome"] for row in site_outcomes},+ {"all-pass", "all-fail"},+ )self.assertIn("moli-run-1-rank002-fail.example.json", failure_files)self.assertIn("\"profile\": \"quick\"", summary_payload)+ def test_run_top_sites_suite_records_chrome_http_error_status(self) -> None:+ list_md = "## Top 100\n\n1. `error.example` — synthetic HTTP error row\n"++ def fake_chrome_run(*args: object, **kwargs: object) -> ProcessResult:+ return ProcessResult(+ command=["/bin/chromium", "https://error.example/"],+ returncode=0,+ elapsed_ms=995.0,+ stdout=(+ b"<html><head><title>upstream response</title></head><body>"+ + b"error details " * 100+ + b"</body></html>"+ ),+ stderr=b"",+ timed_out=False,+ resources={},+ response_status=502,+ )++ with tempfile.TemporaryDirectory() as temp_dir:+ output_dir = Path(temp_dir)+ list_path = output_dir / "top.md"+ list_path.write_text(list_md, encoding="utf-8")+ with patch(+ "moli_benchmark.top_sites.run_chrome_dcl_dump",+ side_effect=fake_chrome_run,+ ):+ summary = run_top_sites_suite(+ output_dir=output_dir,+ target_matrix={"chrome": {"available": True, "path": "/bin/chromium"}},+ targets=("chrome",),+ profile="quick",+ list_path=list_path,+ runs=1,+ timeout_seconds=1.0,+ gate_target="chrome",+ parallelism=1,+ chrome_parallelism=1,+ limit_override=1,+ )+ rows = json.loads(+ (output_dir / "top-sites" / "runs.json").read_text(encoding="utf-8")+ )++ self.assertEqual(rows[0]["response_status"], 502)+ self.assertEqual(rows[0]["category"], "http-error")+ self.assertEqual(rows[0]["response_status_source"], "protocol")+ self.assertEqual(rows[0]["classification_basis"], "protocol-http-status")+ self.assertFalse(rows[0]["ok"])+ self.assertEqual(summary["targets"]["chrome"]["failure_kinds"], {"http-error": 1})++ def test_run_top_sites_suite_excludes_header_only_binary_from_denominator(+ self,+ ) -> None:+ list_md = (+ "## Top 100\n\n"+ "1. `https://document.example/file.pdf` — synthetic PDF row\n"+ )++ def fake_chrome_run(*args: object, **kwargs: object) -> ProcessResult:+ return ProcessResult(+ command=["/bin/chromium", "https://document.example/file.pdf"],+ returncode=0,+ elapsed_ms=50.0,+ stdout=b"",+ stderr=b"",+ timed_out=False,+ resources={},+ response_status=200,+ response_mime_type="application/pdf",+ main_document_body_capture="response-headers-only",+ )++ with tempfile.TemporaryDirectory() as temp_dir:+ output_dir = Path(temp_dir)+ list_path = output_dir / "top.md"+ list_path.write_text(list_md, encoding="utf-8")+ with patch(+ "moli_benchmark.top_sites.run_chrome_dcl_dump",+ side_effect=fake_chrome_run,+ ):+ summary = run_top_sites_suite(+ output_dir=output_dir,+ target_matrix={+ "chrome": {"available": True, "path": "/bin/chromium"}+ },+ targets=("chrome",),+ profile="quick",+ list_path=list_path,+ runs=1,+ timeout_seconds=1.0,+ gate_target="chrome",+ parallelism=1,+ chrome_parallelism=1,+ limit_override=1,+ )+ rows = json.loads(+ (output_dir / "top-sites" / "runs.json").read_text(+ encoding="utf-8"+ )+ )++ self.assertEqual(rows[0]["category"], "binary-response-headers")+ self.assertFalse(rows[0]["comparable"])+ self.assertNotIn("failure_artifact", rows[0])+ self.assertEqual(summary["gate_failures"], 0)+ self.assertEqual(summary["targets"]["chrome"]["attempts"], 0)+ self.assertEqual(summary["targets"]["chrome"]["raw_comparable_attempts"], 0)+ self.assertEqual(summary["targets"]["chrome"]["raw_non_comparable_attempts"], 1)+ self.assertEqual(+ summary["targets"]["chrome"]["non_comparable_attempts"],+ 1,+ )+ self.assertEqual(summary["targets"]["chrome"]["failures"], 0)+ self.assertEqual(summary["common_success"]["attempts"], 0)++ def test_run_top_sites_suite_records_cli_diagnostic_http_status(self) -> None:+ list_md = "## Top 100\n\n1. `error.example` — synthetic CLI HTTP error row\n"++ def fake_run(command: list[str], **kwargs: object) -> ProcessResult:+ return ProcessResult(+ command=command,+ returncode=1,+ elapsed_ms=12.0,+ stdout=b"",+ stderr=(+ b"Error: failed to fetch `https://error.example/`\n"+ b"Reason: lifecycle target document `https://error.example/` "+ b"returned 502 Bad Gateway"+ ),+ timed_out=False,+ resources={},+ )++ with tempfile.TemporaryDirectory() as temp_dir:+ output_dir = Path(temp_dir)+ list_path = output_dir / "top.md"+ list_path.write_text(list_md, encoding="utf-8")+ with patch("moli_benchmark.top_sites.run_process", fake_run):+ summary = run_top_sites_suite(+ output_dir=output_dir,+ target_matrix={"moli": {"available": True, "path": "/bin/moli"}},+ targets=("moli",),+ profile="quick",+ list_path=list_path,+ runs=1,+ timeout_seconds=1.0,+ gate_target="moli",+ parallelism=1,+ limit_override=1,+ )+ rows = json.loads(+ (output_dir / "top-sites" / "runs.json").read_text(+ encoding="utf-8"+ )+ )++ self.assertEqual(rows[0]["response_status"], 502)+ self.assertEqual(rows[0]["response_status_source"], "cli-diagnostic")+ self.assertEqual(rows[0]["category"], "http-error")+ self.assertEqual(+ rows[0]["classification_basis"],+ "cli-diagnostic-http-status",+ )+ self.assertEqual(summary["targets"]["moli"]["status_observed_attempts"], 1)+ self.assertEqual(+ summary["targets"]["moli"]["response_status_sources"],+ {"cli-diagnostic": 1},+ )+def test_run_top_sites_suite_promotes_deadline_adjacent_failures_to_timeout(self,) -> None:@@ -686,6 +1292,11 @@ if(window.EOJsChallengeSDK){new window.EOJsChallengeSDK({callback:function(){}})limit_override=3,)raw_runs = (output_dir / "top-sites" / "raw-runs.csv").read_text(encoding="utf-8")+ run_rows = json.loads(+ (output_dir / "top-sites" / "runs.json").read_text(+ encoding="utf-8"+ )+ )self.assertEqual(summary["site_count"], 3)self.assertEqual(summary["counted_site_count"], 2)@@ -698,6 +1309,14 @@ if(window.EOJsChallengeSDK){new window.EOJsChallengeSDK({callback:function(){}})self.assertEqual(summary["targets"]["moli"]["passes"], 1)self.assertEqual(summary["targets"]["moli"]["failures"], 1)self.assertIn("site-unreachable", raw_runs)+ order_by_target_rank = {+ (row["target"], row["rank"]): row["target_order_index"]+ for row in run_rows+ }+ self.assertEqual(order_by_target_rank[("moli", 1)], 1)+ self.assertEqual(order_by_target_rank[("chrome", 1)], 2)+ self.assertEqual(order_by_target_rank[("chrome", 2)], 1)+ self.assertEqual(order_by_target_rank[("moli", 2)], 2)def test_run_top_sites_suite_keeps_single_target_unreachable_sites_counted(self) -> None:list_md = "## Top 100\n\n1. `dead.example` — synthetic unreachable row\n"@@ -812,7 +1431,7 @@ if(window.EOJsChallengeSDK){new window.EOJsChallengeSDK({callback:function(){}})self.assertEqual(entries[19][0], 20)self.assertEqual(entries[-1][0], 100)self.assertIn("www.huxiu.com", domains)- self.assertEqual(labels, ["chinese-community:chinese-community-top100-websites.md"])+ self.assertEqual(labels, ["chinese-community:chinese-community-top100-websites.csv"])def test_global_source_returns_english_world_sites(self) -> None:resolved, _ = resolve_top_sites_source("global", None)@@ -823,7 +1442,7 @@ if(window.EOJsChallengeSDK){new window.EOJsChallengeSDK({callback:function(){}})self.assertIn("github.com", domains)self.assertIn("wikipedia.org", domains)self.assertNotIn("baidu.com", domains, "global list must not contain Chinese top sites")- self.assertEqual(labels, ["global:global-top-websites-seed-list.md"])+ self.assertEqual(labels, ["global:global-top-websites-seed-list.csv"])def test_mixed_source_interleaves_cn_and_global(self) -> None:resolved, list_source = resolve_top_sites_source("mixed", None)@@ -848,7 +1467,7 @@ if(window.EOJsChallengeSDK){new window.EOJsChallengeSDK({callback:function(){}})urls,)self.assertNotIn("https://github.com/golang/go/blob/master/doc/effective_go.md", urls)- self.assertEqual(labels, ["webfetch-longtail:webfetch-longtail-seed-list.md"])+ self.assertEqual(labels, ["webfetch-longtail:webfetch-longtail-seed-list.csv"])def test_render_quality_source_returns_curated_url_paths(self) -> None:resolved, _ = resolve_top_sites_source("render-quality", None)@@ -860,7 +1479,7 @@ if(window.EOJsChallengeSDK){new window.EOJsChallengeSDK({callback:function(){}})"https://www.feishu.cn/hc/zh-CN/articles/485770964672-管理者查看企业-ai-功能使用数据",urls,)- self.assertEqual(labels, ["render-quality:render-quality-seed-list.md"])+ self.assertEqual(labels, ["render-quality:render-quality-seed-list.csv"])def test_webfetch_mix_keeps_top_site_mix_then_appends_longtail(self) -> None:resolved, list_source = resolve_top_sites_source("webfetch-mix", None)@@ -883,9 +1502,9 @@ if(window.EOJsChallengeSDK){new window.EOJsChallengeSDK({callback:function(){}})self.assertEqual(labels,[- "chinese-community:chinese-community-top100-websites.md",- "global:global-top-websites-seed-list.md",- "webfetch-longtail:webfetch-longtail-seed-list.md",+ "chinese-community:chinese-community-top100-websites.csv",+ "global:global-top-websites-seed-list.csv",+ "webfetch-longtail:webfetch-longtail-seed-list.csv",],)diff --git a/moli-benchmark/tests/test_versions.py b/moli-benchmark/tests/test_versions.pynew file mode 100644index 000000000..ef2df7010--- /dev/null+++ b/moli-benchmark/tests/test_versions.py@@ -0,0 +1,51 @@+from __future__ import annotations++import tempfile+import unittest+from pathlib import Path+from unittest.mock import patch++from moli_benchmark.targets import collect_target_binaries+from moli_benchmark.versions import collect_versions+++class VersionsTests(unittest.TestCase):+ def test_collect_versions_uses_moli_dash_version(self) -> None:+ commands: list[list[str]] = []++ def run(command: list[str], timeout: float = 5.0) -> str:+ del timeout+ commands.append(command)+ return "moli 1.0.2"++ with tempfile.TemporaryDirectory() as temp_dir:+ binary = Path(temp_dir) / "moli"+ binary.write_bytes(b"test binary")+ with patch("moli_benchmark.versions._run", side_effect=run):+ versions = collect_versions(binary)++ self.assertIn([str(binary), "--version"], commands)+ self.assertNotIn([str(binary), "version"], commands)+ self.assertEqual(versions["moli"]["version"], "moli 1.0.2")++ def test_target_metadata_uses_moli_dash_version(self) -> None:+ moli = Path("/tmp/test-moli")++ with (+ patch("moli_benchmark.targets.moli_binary", return_value=moli),+ patch("moli_benchmark.targets.lightpanda_binary", return_value=None),+ patch("moli_benchmark.targets.chrome_binary", return_value=None),+ patch("moli_benchmark.targets.obscura_binary", return_value=None),+ patch(+ "moli_benchmark.targets._binary_info",+ side_effect=lambda path, args: {"path": path, "version_args": args},+ ) as binary_info,+ ):+ targets = collect_target_binaries()++ self.assertEqual(binary_info.call_args_list[0].args, (moli, ("--version",)))+ self.assertEqual(targets["moli"]["version_args"], ("--version",))+++if __name__ == "__main__":+ unittest.main()diff --git a/moli-benchmark/tests/test_wild_web.py b/moli-benchmark/tests/test_wild_web.pyindex 6fb8b12ea..3e520078b 100644--- a/moli-benchmark/tests/test_wild_web.py+++ b/moli-benchmark/tests/test_wild_web.py@@ -1,15 +1,18 @@from __future__ import annotations+import jsonimport tempfileimport unittestfrom pathlib import Pathfrom unittest.mock import patchfrom moli_benchmark.process import ProcessResult+from moli_benchmark.public_web import POST_DCL_WAIT_SCRIPTfrom moli_benchmark.wild_web import (_classify,_extract_page_snapshot,_failure_kind,+ _rotated_target_order,_wild_command_for_target,_wild_web_target_metadata,_wild_web_extraction_failures,@@ -18,6 +21,15 @@ from moli_benchmark.wild_web import (class WildWebTests(unittest.TestCase):+ def test_target_order_rotates_per_seed(self) -> None:+ targets = ("moli", "lightpanda", "chrome")++ self.assertEqual(_rotated_target_order(targets, 0), targets)+ self.assertEqual(+ _rotated_target_order(targets, 1),+ ("lightpanda", "chrome", "moli"),+ )+def test_obscura_wild_web_command_uses_fetch_timeout_seconds(self) -> None:command = _wild_command_for_target("obscura", Path("/bin/obscura"), "https://example.test/", 7.9)@@ -32,7 +44,14 @@ class WildWebTests(unittest.TestCase):command = _wild_command_for_target("lightpanda", Path("/bin/lightpanda"), "https://example.test/", 30.0)self.assertEqual(command[:4], ["/bin/lightpanda", "fetch", "--dump", "html"])- self.assertEqual(command[command.index("--wait-until") + 1], "done")+ self.assertEqual(+ command[command.index("--wait-until") + 1],+ "domcontentloaded",+ )+ self.assertEqual(+ command[command.index("--wait-script") + 1],+ POST_DCL_WAIT_SCRIPT,+ )self.assertEqual(command[command.index("--wait-ms") + 1], "30000")self.assertEqual(command[command.index("--http-timeout") + 1], "30000")self.assertEqual(command[command.index("--terminate-ms") + 1], "30000")@@ -41,6 +60,25 @@ class WildWebTests(unittest.TestCase):self.assertNotIn("--http_timeout", command)self.assertEqual(command[-1], "https://example.test/")+ def test_moli_wild_web_command_uses_the_same_dcl_boundary(self) -> None:+ command = _wild_command_for_target(+ "moli",+ Path("/bin/moli"),+ "https://example.test/",+ 30.0,+ )++ self.assertEqual(+ command[command.index("--wait-until") + 1],+ "domcontentloaded",+ )+ self.assertEqual(+ command[command.index("--wait-script") + 1],+ POST_DCL_WAIT_SCRIPT,+ )+ self.assertEqual(command[command.index("--timeout") + 1], "30000")+ self.assertEqual(command[command.index("--http-timeout") + 1], "30000")+def test_wild_web_chrome_uses_cdp_dcl_metadata(self) -> None:self.assertEqual(_wild_web_target_metadata("chrome")["driver"], "cdp-dcl")self.assertEqual(_wild_web_target_metadata("chrome")["label"], "chrome / cdp-dcl")@@ -55,7 +93,7 @@ class WildWebTests(unittest.TestCase):command=["/bin/obscura", "serve", "--port", "12345"],returncode=0,elapsed_ms=12.0,- stdout="<!doctype html><title>知乎</title><body>知乎 首页 内容 推荐 回答 专栏</body>".encode(),+ stdout="<!doctype html><title>知乎</title><body>知乎 首页 内容 推荐 回答 专栏 热榜 发现 等你来答</body>".encode(),stderr=b"",timed_out=False,resources={"peak_pss_bytes": 123},@@ -86,7 +124,7 @@ class WildWebTests(unittest.TestCase):"""<!doctype html><title>知乎 - 有问题,就会有答案</title>- <body><main>知乎 首页 内容 推荐 回答 专栏</main><script>hidden()</script></body>+ <body><main>知乎 首页 内容 推荐 回答 专栏 热榜 发现 等你来答</main><script>hidden()</script></body>""".encode())@@ -94,6 +132,15 @@ class WildWebTests(unittest.TestCase):self.assertIn("知乎 首页", snapshot["text_sample"])self.assertEqual(_wild_web_extraction_failures("zhihu-home", snapshot), [])+ def test_seed_brand_is_required_in_title_but_not_repeated_in_body(self) -> None:+ snapshot = _extract_page_snapshot(+ b"<!doctype html><title>Baidu</title><body>"+ + b"useful navigation content " * 20+ + b"</body>"+ )++ self.assertEqual(_wild_web_extraction_failures("baidu-home", snapshot), [])+def test_extraction_assertions_report_missing_business_fields(self) -> None:snapshot = _extract_page_snapshot(b"<!doctype html><title>Example</title><body>tiny</body>")@@ -103,6 +150,15 @@ class WildWebTests(unittest.TestCase):)self.assertEqual(_failure_kind("success", ["title-keyword-mismatch"]), "extraction-failure")+ def test_snapshot_does_not_count_title_as_body_text(self) -> None:+ snapshot = _extract_page_snapshot(+ b"<!doctype html><title>A long title</title><body>short body</body>"+ )++ self.assertEqual(snapshot["title"], "A long title")+ self.assertEqual(snapshot["text_sample"], "short body")+ self.assertEqual(snapshot["text_length"], len("short body"))+def test_classify_ignores_blocked_markers_inside_scripts_when_visible_content_exists(self,) -> None:@@ -120,6 +176,101 @@ class WildWebTests(unittest.TestCase):self.assertEqual(_classify(html, b"", 0, False), "blocked")+ def test_classify_rejects_contentful_http_error_response(self) -> None:+ html = (+ b"<!doctype html><title>upstream response</title><body>"+ + b"error details " * 100+ + b"</body>"+ )++ self.assertEqual(+ _classify(html, b"", 0, False, response_status=500),+ "http-error",+ )+ self.assertEqual(_failure_kind("http-error", []), "http-error")++ def test_classify_infers_cli_error_documents_and_network_failures(self) -> None:+ gateway = b"<!doctype html><title>502 Bad Gateway</title><body>origin unavailable</body>"+ api_error = (+ b"<!doctype html><title></title><body><pre>"+ b'{"error":{"message":"invalid query"}}'+ b"</pre></body>"+ )+ tls_error = b"<!doctype html><body>Navigation failed Reason: PeerFailedVerification</body>"+ chinese_verification = "<!doctype html><body>环境异常,完成验证后即可继续访问</body>".encode()++ self.assertEqual(_classify(gateway, b"", 0, False), "http-error")+ self.assertEqual(_classify(api_error, b"", 0, False), "http-error")+ self.assertEqual(_classify(tls_error, b"", 0, False), "network-error")+ self.assertEqual(_failure_kind("network-error", []), "network-error")+ self.assertEqual(+ _classify(chinese_verification, b"", 0, False),+ "challenge",+ )++ def test_classify_recognizes_lightpanda_transport_and_waf_documents(self) -> None:+ dns = (+ b"<!doctype html><body><h1>Navigation failed</h1>"+ b"<p>Reason: CouldntResolveHost</p></body>"+ )+ http2 = (+ b"<!doctype html><body><h1>Navigation failed</h1>"+ b"<p>Reason: Http2Stream</p></body>"+ )+ safeline = (+ b"<!doctype html><body>Security Detection Powered By SafeLine WAF"+ b"</body>"+ )+ privacy = (+ b"<!doctype html><body>Select the 2 matching tiles Submit "+ b"Powered and protected by Privacy</body>"+ )++ self.assertEqual(_classify(dns, b"", 0, False), "network-error")+ self.assertEqual(_classify(http2, b"", 0, False), "network-error")+ self.assertEqual(_classify(safeline, b"", 0, False), "blocked")+ self.assertEqual(_classify(privacy, b"", 0, False), "challenge")++ def test_classify_recognizes_lightpanda_fatal_fetch_timeout(self) -> None:+ stderr = (+ b'$scope=app $level=fatal $msg="fetch error" err=Timeout '+ b"url=https://example.test/"+ )++ self.assertEqual(_classify(b"", stderr, 0, False), "timeout")++ def test_classify_preserves_cli_main_document_status_and_dns_reason(self) -> None:+ self.assertEqual(+ _classify(+ b"",+ b"Reason: lifecycle target document `https://example.test/` "+ b"returned 500 Internal Server Error",+ 1,+ False,+ ),+ "http-error",+ )+ self.assertEqual(+ _classify(+ b"",+ b"Reason: lifecycle target document `https://example.test/` "+ b"returned 401 Unauthorized",+ 1,+ False,+ ),+ "blocked",+ )+ self.assertEqual(+ _classify(+ b"",+ b"Reason: failed to resolve example.test:443: failed to lookup "+ b"address information",+ 1,+ False,+ ),+ "network-error",+ )+def test_run_wild_web_suite_writes_failure_snapshots(self) -> None:def run_process(*args: object, **kwargs: object) -> ProcessResult:return ProcessResult(@@ -147,10 +298,22 @@ class WildWebTests(unittest.TestCase):failures_dir = output_dir / "wild-web" / "failures"failure_files = sorted(path.name for path in failures_dir.iterdir())+ rows = json.loads(+ (output_dir / "wild-web" / "runs.json").read_text(+ encoding="utf-8"+ )+ )self.assertEqual(summary["gate_failures"], 1)self.assertEqual(summary["targets"]["moli"]["failure_kinds"], {"extraction-failure": 1})self.assertEqual(summary["targets"]["moli"]["extraction_failures"], 2)+ self.assertEqual(summary["schedule"], "site-paired-rotating-target-order")+ self.assertEqual(summary["scheduled_site_groups"], 1)+ self.assertEqual(rows[0]["schedule_index"], 0)+ self.assertEqual(rows[0]["target_order_index"], 1)+ self.assertTrue(rows[0]["started_at"].endswith("Z"))+ self.assertEqual(len(rows[0]["stdout_sha256"]), 64)+ self.assertEqual(len(rows[0]["stderr_sha256"]), 64)self.assertIn("moli-run-1-toutiao-home.json", failure_files)self.assertIn("moli-run-1-toutiao-home.stdout.html", failure_files)@@ -182,6 +345,12 @@ class WildWebTests(unittest.TestCase):self.assertEqual(summary["gate_failures"], 0)self.assertEqual(summary["targets"]["moli"]["categories"], {"login": 1})self.assertEqual(summary["targets"]["moli"]["failure_kinds"], {})+ self.assertEqual(summary["targets"]["moli"]["elapsed_ms"]["count"], 1)+ self.assertEqual(+ summary["targets"]["moli"]["successful_elapsed_ms"]["count"],+ 1,+ )+ self.assertEqual(summary["common_success"]["attempts"], 1)def test_capture_replay_writes_manifest_for_successful_rows(self) -> None:def run_process(*args: object, **kwargs: object) -> ProcessResult:@@ -189,7 +358,7 @@ class WildWebTests(unittest.TestCase):command=["fake"],returncode=0,elapsed_ms=12.0,- stdout="<!doctype html><title>知乎</title><body>知乎 首页 内容 推荐 回答 专栏</body>".encode(),+ stdout="<!doctype html><title>知乎</title><body>知乎 首页 内容 推荐 回答 专栏 热榜 发现 等你来答</body>".encode(),stderr=b"",timed_out=False,resources={"peak_pss_bytes": 123},