Skip to content

armgit / moli

Public Imported from GitHub
0

Best headless browser for AI agents. Lite, Fast, High-Compatibility. Built in Rust

refactor(benchmark): unify public web comparisons

ldm0c461cb1799ee
diff --git a/moli-benchmark/README.md b/moli-benchmark/README.md
index 45a54a3c8..dc4e87b4b 100644
--- a/moli-benchmark/README.md
+++ b/moli-benchmark/README.md
@@ -84,20 +84,19 @@ canvas dependencies. Failed reftests retain `test.png`, `reference-N.png`, and
An unfiltered full `default` or `all` run refreshes the unified status lists
directly under `wpt-cross-current/`.
-Public-web suites read a Markdown seed list. A minimal `sites.md` looks like:
+Public-web suites read a `rank,target` CSV seed list. A minimal `sites.csv` looks like:
-```markdown
-## Top 2
-
-1. `https://example.com/`
-2. `https://www.rust-lang.org/`
+```csv
+rank,target
+1,https://example.com/
+2,https://www.rust-lang.org/
```
Compare Moli and Chrome on that sample:
```bash
uv run moli-benchmark top-sites \
- --list-path sites.md \
+ --list-path sites.csv \
--profile quick \
--target moli \
--target chrome \
@@ -108,7 +107,7 @@ Compare visible content with Chrome as the baseline:
```bash
uv run moli-benchmark render-compare \
- --list-path sites.md \
+ --list-path sites.csv \
--profile quick \
--target moli \
--baseline-target chrome \
@@ -196,9 +195,58 @@ The harness distinguishes the browser engine from the way it is driven:
Not every suite accepts every target. Its `--help` output lists the valid
choices.
-The predefined public-web sources expect curated seed documents under the
-repository's `docs/` directory. Those files are not present in every checkout;
-use `--list-path` with `top-sites` or `render-compare` when they are unavailable.
+The predefined public-web CSV sources ship under `fixtures/top-sites/`, including
+the WebFetch longtail corpus used by `webfetch-mix`. Markdown remains supported
+for custom `--list-path` inputs.
+
+Public-web results distinguish an individual attempt from a stable per-site
+outcome. `raw-runs.csv` and `runs.json` retain every attempt, while
+`site-outcomes.json` groups repeated attempts into `all-pass`, `all-fail`, and
+`flaky` sites. Pairwise engine rows are marked `SINGLE SAMPLE` unless the suite
+was run with at least three attempts per site; use `--runs 3` or more before
+treating an engine-only result as repeat-validated.
+
+Public-web summaries keep three different populations explicit. `raw_*`
+metrics include reachable and unreachable observations before cross-engine site
+exclusions; the normal pass rate uses counted, comparable attempts; and
+`successful_*` latency/memory metrics contain successful attempts only. The
+`common_success` cohort contains the exact run/site attempts that succeeded on
+every selected target and is the source for cross-engine speed and memory
+claims in the HTML report.
+
+Multi-engine public-web runs are scheduled in site-paired groups. Each group
+runs the selected targets sequentially, rotates which target goes first, and
+allows multiple site groups to run concurrently. Raw rows record the schedule
+and target-order indexes, UTC start/finish times, output hashes and samples,
+response MIME/body-capture evidence, and the final URL when CDP exposes it.
+This makes order effects and response changes auditable without retaining every
+successful response body.
+
+When the driver exposes the main-document HTTP status, that status is
+authoritative: a rendered 4xx or 5xx error document is a failure even if its
+body exceeds the content-size threshold. The report includes HTTP-status
+coverage, evidence source, and classification basis for each target. Moli CLI
+failures that name a terminal main-document status retain it as
+`cli-diagnostic` evidence. CLI fetch drivers that expose neither protocol nor
+diagnostic status fall back to conservative error-document markers; for a
+protocol-aligned comparison with status coverage, select the `moli-cdp` and
+`lightpanda-cdp` targets alongside `chrome`.
+
+A CDP browser reports a binary main-document MIME type before it exposes a
+download body. That is retained as `binary-response-headers`, not fabricated
+into a successful PDF or archive. Header-only observations are neutral and do
+not enter body-success or latency denominators. A CLI transfer that times out
+after receiving only part of a binary response remains a transfer failure. For
+an explicit `.pdf` main-resource URL, the Moli CLI adapter omits DOM-only page
+wait options and counts the result only when the real PDF body is returned.
+
+The built-in `wild-web` targets use the same DOMContentLoaded snapshot boundary
+for Moli, Lightpanda, and Chrome: after the adapter observes DOMContentLoaded,
+the page event loop must advance for at least 50 ms before the DOM is dumped.
+The total readiness deadline still applies to that settle period. Seed
+extraction checks require the expected site identity in the title plus a
+non-trivial body; the brand name does not need to be repeated in the first
+body-text sample.
Moli is discovered from `MOLI_BIN`, `../target/release/moli`, or `PATH`, in
that order. Comparison browsers are optional and can be selected through
@@ -295,8 +343,8 @@ Run the core CLI tests from `moli-benchmark/`:
uv run python -m unittest discover -s tests -p 'test_cli.py'
```
-The complete test suite uses `uv run python -m unittest discover -s tests` and
-also expects the curated public-web seed documents described above.
+The complete test suite uses `uv run python -m unittest discover -s tests`; its
+curated public-web seed CSV files are versioned under `fixtures/top-sites/`.
Keep benchmark claims tied to archived raw data, exact binary versions, and
the readiness checks. When adding a suite or target, update its CLI help and
diff --git a/moli-benchmark/fixtures/top-sites/chinese-community-top100-websites.csv b/moli-benchmark/fixtures/top-sites/chinese-community-top100-websites.csv
new file mode 100644
index 000000000..b3287fcbf
--- /dev/null
+++ b/moli-benchmark/fixtures/top-sites/chinese-community-top100-websites.csv
@@ -0,0 +1,101 @@
+rank,target
+1,zhihu.com
+2,tieba.baidu.com
+3,douban.com
+4,bilibili.com
+5,weibo.com
+6,xiaohongshu.com
+7,douyin.com
+8,kuaishou.com
+9,hupu.com
+10,guokr.com
+11,lofter.com
+12,www.acfun.cn
+13,6park.com
+14,wenxuecity.com
+15,backchina.com
+16,juejin.cn
+17,csdn.net
+18,cnblogs.com
+19,oschina.net
+20,segmentfault.com
+21,v2ex.com
+22,hostloc.com
+23,linux.do
+24,ruby-china.org
+25,learnku.com
+26,right.com.cn
+27,testerhome.com
+28,kanxue.com
+29,chiphell.com
+30,52pojie.cn
+31,coolapk.com
+32,expreview.com
+33,mydigit.cn
+34,pconline.com.cn
+35,ithome.com
+36,cnbeta.com
+37,www.huxiu.com
+38,zol.com.cn
+39,mobile01.com
+40,dcard.tw
+41,ptt.cc
+42,disp.cc
+43,gamer.com.tw
+44,lihkg.com
+45,uwants.com
+46,hkepc.com
+47,discuss.com.hk
+48,ck101.com
+49,eyny.com
+50,babyhome.com.tw
+51,hk01.com
+52,nga.cn
+53,gcores.com
+54,keylol.com
+55,stage1st.com
+56,a9vg.com
+57,ali213.net
+58,3dmgame.com
+59,gameres.com
+60,indienova.com
+61,bangumi.tv
+62,dmhy.org
+63,tgbus.com
+64,gamensky.com
+65,douyu.com
+66,huya.com
+67,autohome.com.cn
+68,dongchedi.com
+69,xcar.com.cn
+70,qyer.com
+71,mafengwo.cn
+72,fengniao.com
+73,xitek.com
+74,8264.com
+75,babytree.com
+76,mama.cn
+77,pcbaby.com.cn
+78,dxy.cn
+79,haodf.com
+80,zcool.com.cn
+81,huaban.com
+82,goumin.com
+83,zhidao.baidu.com
+84,smzdm.com
+85,xueqiu.com
+86,eastmoney.com
+87,taoguba.com.cn
+88,futunn.com
+89,manmanbuy.com
+90,xiachufang.com
+91,kdslife.com
+92,hi-pda.com
+93,qidian.com
+94,zongheng.com
+95,jjwxc.net
+96,17k.com
+97,ximalaya.com
+98,zsxq.com
+99,anquanke.com
+100,muchong.com
diff --git a/moli-benchmark/fixtures/top-sites/global-top-websites-seed-list.csv b/moli-benchmark/fixtures/top-sites/global-top-websites-seed-list.csv
new file mode 100644
index 000000000..3eb81b3d6
--- /dev/null
+++ b/moli-benchmark/fixtures/top-sites/global-top-websites-seed-list.csv
@@ -0,0 +1,101 @@
+rank,target
+1,google.com
+2,facebook.com
+3,microsoft.com
+4,youtube.com
+5,apple.com
+6,instagram.com
+7,linkedin.com
+8,amazon.com
+9,wikipedia.org
+10,github.com
+11,bing.com
+12,netflix.com
+13,chatgpt.com
+14,pinterest.com
+15,yahoo.com
+16,x.com
+17,tiktok.com
+18,spotify.com
+19,reddit.com
+20,dropbox.com
+21,nytimes.com
+22,tumblr.com
+23,paypal.com
+24,discord.com
+25,medium.com
+26,cnn.com
+27,theguardian.com
+28,bbc.com
+29,ebay.com
+30,imdb.com
+31,openai.com
+32,booking.com
+33,reuters.com
+34,forbes.com
+35,slack.com
+36,wsj.com
+37,nasa.gov
+38,duckduckgo.com
+39,brave.com
+40,gitlab.com
+41,quora.com
+42,goodreads.com
+43,stackoverflow.com
+44,wired.com
+45,techcrunch.com
+46,theverge.com
+47,anthropic.com
+48,airbnb.com
+49,whitehouse.gov
+50,economist.com
+51,perplexity.ai
+52,huggingface.co
+53,npmjs.com
+54,ycombinator.com
+55,kaggle.com
+56,wolframalpha.com
+57,vimeo.com
+58,roblox.com
+59,wordpress.com
+60,bbc.co.uk
+61,t.me
+62,samsung.com
+63,zoom.us
+64,whatsapp.com
+65,live.com
+66,bit.ly
+67,discord.gg
+68,unity3d.com
+69,intuit.com
+70,snapchat.com
+71,cnbc.com
+72,bloomberg.com
+73,npr.org
+74,washingtonpost.com
+75,apnews.com
+76,usatoday.com
+77,theatlantic.com
+78,vox.com
+79,arstechnica.com
+80,news.ycombinator.com
+81,gov.uk
+82,usa.gov
+83,mit.edu
+84,stanford.edu
+85,cmu.edu
+86,harvard.edu
+87,nature.com
+88,science.org
+89,coursera.org
+90,edx.org
+91,khanacademy.org
+92,udemy.com
+93,figma.com
+94,canva.com
+95,notion.so
+96,airtable.com
+97,linear.app
+98,vercel.com
+99,cloudflare.com
+100,npmjs.org
diff --git a/moli-benchmark/fixtures/top-sites/legacy-encoding-websites-seed-list.csv b/moli-benchmark/fixtures/top-sites/legacy-encoding-websites-seed-list.csv
new file mode 100644
index 000000000..ed5890db0
--- /dev/null
+++ b/moli-benchmark/fixtures/top-sites/legacy-encoding-websites-seed-list.csv
@@ -0,0 +1,7 @@
+rank,target
+1,https://www.aozora.gr.jp/cards/000081/files/456_15050.html
+2,https://www.debian.or.jp/
+3,http://www.hiking.com.hk/camping/laichi.htm
+4,http://lib.ru/
+5,https://www.w3.org/TR/1998/REC-html40-19980424/
+6,https://www.yes24.com/Main/default.aspx
diff --git a/moli-benchmark/fixtures/top-sites/render-quality-seed-list.csv b/moli-benchmark/fixtures/top-sites/render-quality-seed-list.csv
new file mode 100644
index 000000000..108b4d0bc
--- /dev/null
+++ b/moli-benchmark/fixtures/top-sites/render-quality-seed-list.csv
@@ -0,0 +1,13 @@
+rank,target
+1,https://www.feishu.cn/hc/zh-CN/articles/485770964672-管理者查看企业-ai-功能使用数据
+2,https://docs.python.org/3/tutorial/index.html
+3,https://nextjs.org/docs
+4,https://react.dev/learn
+5,https://vite.dev/guide/
+6,https://nodejs.org/en/learn/getting-started/introduction-to-nodejs
+7,https://axios-http.com/docs/intro
+8,https://www.liaoxuefeng.com/wiki/1016959663602400/1017063413904832
+9,https://pnpm.io/installation
+10,https://www.rust-lang.org/learn
+11,https://www.apple.com/iphone/
+12,https://learn.microsoft.com/en-us/windows/win32/apiindex/windows-api-list
diff --git a/moli-benchmark/fixtures/top-sites/webfetch-longtail-seed-list.csv b/moli-benchmark/fixtures/top-sites/webfetch-longtail-seed-list.csv
new file mode 100644
index 000000000..5afd6c0ac
--- /dev/null
+++ b/moli-benchmark/fixtures/top-sites/webfetch-longtail-seed-list.csv
@@ -0,0 +1,160 @@
+rank,target
+1,https://www.cell.com/cell/fulltext/S0092-8674(24)00500-9
+2,https://www.science.org/doi/10.1126/science.abj0890
+3,https://www.investopedia.com/articles/personal-finance/082316/how-much-money-do-you-need-live-newyork-city.asp
+4,https://seekingalpha.com/article/4666666-tesla-q1-2024
+5,https://www.metacritic.com/movie/oppenheimer
+6,https://www.reddit.com/r/MachineLearning/comments/1aqxhol/d_thoughts_on_llama_3/
+7,https://www.reddit.com/r/learnpython/comments/1arpz0k/how_do_i_start_learning_python/
+8,https://www.reddit.com/r/programming/comments/1ar0khc/openai_sora_text_to_video_model/
+9,https://dl.acm.org/doi/10.1145/3597503.3623298
+10,https://www.adidas.com/us/samba-og-shoes/B75806.html
+11,https://www.allrecipes.com/recipe/231170/cuban-mojo-marinated-pork/
+12,https://www.seriouseats.com/foolproof-pan-pizza-recipe
+13,https://www.travelandleisure.com/best-tokyo-restaurants
+14,https://www.tripadvisor.com/Attraction_Review-g60750-d136919-Reviews-Empire_State_Building-New_York_City_New_York.html
+15,https://stackoverflow.com/questions/2081586/web-scraping-with-python
+16,https://stackoverflow.com/questions/3437059/does-anyone-have-a-good-overview-of-the-pros-cons-of-using-x-frame-options
+17,https://www.douban.com/note/856000000/
+18,https://sourceforge.net/projects/sevenzip/
+19,https://www.adidas.com/us/men-shoes
+20,https://martinfowler.com/articles/
+21,https://community.cloudflare.com/
+22,https://discuss.python.org/
+23,https://www2.census.gov/programs-surveys/decennial/2020/data/apportionment/apportionment-2020-table02.pdf
+24,https://askubuntu.com/questions/3973/how-to-find-out-which-package-a-file-belongs-to
+25,https://stackoverflow.com/questions/89228/how-do-i-execute-a-program-or-call-a-system-command
+26,https://www.pro-football-reference.com/players/M/McDoRi20.htm
+27,https://www.northjersey.com/obituaries/ber117653
+28,https://stackoverflow.com/questions/231767/what-does-the-yield-keyword-do-in-python
+29,https://www.espn.com/tennis/story/_/id/18401434/tennis-why-12-year-old-cori-gauff-thinks-greatest-all
+30,https://archaeologydataservice.ac.uk/catalogue/adsdata/arch-481-1/dissemination/pdf/archaeol7-93506_1.pdf
+31,https://www.nea.org/sites/default/files/2024-04/2023-rankings-and-estimates-report-1.pdf
+32,https://www.sports-reference.com/cbb/schools/connecticut/women/2023.html
+33,https://www.sports-reference.com/cbb/schools/south-carolina/women/2023.html
+34,https://www.sports-reference.com/cbb/schools/washington/women/2023.html
+35,https://www.europeana.eu/ro/collections/person/141236-leon-levy-brunswick
+36,https://serverfault.com/questions/2670/how-can-i-detect-which-machine-on-a-network-changed-an-active-directory-objec
+37,https://www.legacy.com/obituaries/name/lawrence-kramer-obituary?pid=205175579
+38,https://www.pro-football-reference.com/players/M/McDoDe20.htm
+39,https://catalog.hathitrust.org/Record/001042525
+40,https://getmusicbee.com/help/release-note/
+41,https://newjerseyglobe.com/in-memoriam/pat-kramer-four-term-paterson-mayor-and-gop-gubernatorial-frontrunner-dies-at-90/
+42,https://stats.bls.gov/lau/lastrk20.htm
+43,https://www.researchgate.net/publication/266379416_Identifying_semantic_role_clusters_and_alignment_types_via_microrole_coexpression_tendencies
+44,https://electionresults.govt.nz/electionresults_2017/
+45,https://tennispredict.com/coco-gauff/
+46,https://press.vub.ac.be/vub-honorary-doctorates-for-science-communicator-ann-druyan-artist-william-kentridge-politician-pierre-kompany-and-immunotherapy-experts-antoni-ribas-and-thierry-boon
+47,https://www.ask-oracle.com/birthday/1956/04/12/
+48,https://electionresults.govt.nz/electionresults_2023/electorate-details-60.html
+49,https://netflixtechblog.com/scaling-time-series-data-storage-part-i-ec2b6d44ba39
+50,https://netflixtechblog.com/
+51,https://lobste.rs/
+52,https://fsi.nic.in/isfr2019/isfr-fsi-vol2.pdf
+53,https://wikinewzealand.org/table/1ZZwmJ2Hoc3tHDqc
+54,http://www.wfsq.gov.cn/91/1993483352635740160.html
+55,https://www.titan24.com/publish/app/data/2017/02/04/38517/os_news.html
+56,https://beta.boost.org/doc/libs/1_68_0/libs/numeric/ublas/index.html
+57,http://202.106.184.141:89/gate/big5/www.chinanews.com.cn/2002-10-12/26/231332.html
+58,http://big5.chinanews.com.cn:89/gate/big5/www.chinanews.com.cn/2003-09-20/26/348867.html
+59,https://manage.eblcu.cn/specls/cls/405213/course/03/03_11_02_02.html
+60,https://tc.canada.ca/en/road-transportation/statistics-data/canadian-motor-vehicle-traffic-collision-statistics/2022/canadian-motor-vehicle-traffic-collision-statistics-2022#fn5
+61,https://tc.canada.ca/en/road-transportation/statistics-data/canadian-motor-vehicle-traffic-collision-statistics/2022/canadian-motor-vehicle-traffic-collision-statistics-2022?wbdisable=true
+62,https://api.worldbank.org/v2/country/IRL/indicator/EN.GHG.CO2.PC.CE.AR5?date=2023&format=json
+63,https://api.worldbank.org/v2/country/ISL/indicator/EN.GHG.CO2.PC.CE.AR5?date=2023&format=json
+64,https://api.worldbank.org/v2/country/NZL/indicator/EN.GHG.CO2.PC.CE.AR5?date=2023&format=json
+65,https://api.worldbank.org/v2/country/all/indicator/EG.FEC.RNEW.ZS?mrv=1&format=json&per_page=100&page=2
+66,http://epaper.bingtuannet.com/pc/cont/202010/29/c134120.html
+67,http://gxq.qingdao.gov.cn/xxgk_113/zwzx_113/gxdt_113/202203/t20220322_4955636.shtml
+68,http://hndfxj.txhn.net/xjwx/xjyj/wz/201611/t20161128_523882.html
+69,http://house.enorth.com.cn/system/2017/06/09/033161887.shtml
+70,http://huaxia.com/c/2023/01/31/1600892.shtml
+71,https://japan.people.com.cn/BIG5/n1/2026/0211/c35421-40663642.html
+72,http://ly.jiangxi.gov.cn/jxslyj/wzs/pc/content/content_1962334293316018176.html
+73,http://mee.gov.cn/gkml/sthjbgw/stbgth/201806/W020180613394538476583.pdf
+74,https://revues.imist.ma/index.php/JALCS/article/view/37261
+75,http://wlt.shanxi.gov.cn/zh/sourcefiles/html/attractions/1749.shtml
+76,https://www.cqmetro.cn/tpxw/2025/8/14/2170123.shtml
+77,http://www.dg.gov.cn/zjdz/dzgk/zrdl/content/post_2824920.html
+78,https://www.foshan.gov.cn/gzjg/fswenhua/wzfs/content/post_7145707.html
+79,https://www.govinfo.gov/content/pkg/GPO-CRECB-1939-pt1-v84/pdf/GPO-CRECB-1939-pt1-v84-3-1.pdf
+80,https://www.irya.unam.mx/web/en/news/archive/464-remembering-four-unam-women-astronomers
+81,http://www.polygon.com/2012/10/22/3539692/dark-souls-update-106-weakens-weapons-adds-warp-zones-dlc
+82,https://www.polygon.com/2012/10/22/3539692/dark-souls-update-106-weakens-weapons-adds-warp-zones-dlc/
+83,https://www.polygon.com/2013/4/18/4238514/pokemon-mini-trailer-confirms-new-legendarys-connection-to-mewtwo/
+84,https://www.scottishmountainrescue.org/wp-content/uploads/2025/04/SMR_Statistics_Report_2023.pdf
+85,https://www.shanxi.gov.cn/zjsx/sxfg/cysx/202007/t20200724_6047084.shtml
+86,https://www.zhangzhou.gov.cn/cms/html/zzshyyyyj/2025-11-28/1082544542.html
+87,http://www.zuxun100.com/index/article/mingren/info/52174
+88,https://yjglt.jiangxi.gov.cn/jxsyjglt/col/col37571/content/content_1766359793244237824.html
+89,http://dgcg.dg.gov.cn/zwy/yqhx/201008/aafc2e43b7cd437ab757e5d3fcdbe69a.shtml
+90,https://download.militaryonesource.mil/12038/MOS/Reports/2023-demographics-report.pdf
+91,https://forests.gujarat.gov.in/writereaddata/images/pdf/GFS-2019-20.pdf
+92,https://jtys.foshan.gov.cn/jtdt/jtdt/content/post_6957426.html
+93,https://de.wikipedia.org/wiki/Generative_K%C3%BCnstliche_Intelligenz
+94,https://en.wikipedia.org/wiki/Brihaspati_VidyaSadan
+95,"https://en.wikipedia.org/wiki/Brihaspati_VidyaSadan#:~:text=Brihaspati%20Vidyasadan%20was%20established%20in,the%20school%20was%20Maurice%20Banerjee."
+96,"https://en.wikipedia.org/wiki/Homey,_Don't_Ya_Know_Me%3F"
+97,https://en.wikipedia.org/wiki/Mary_Letitia_Caldwella
+98,"https://ghoapi.azureedge.net/api/NCD_BMI_30A?$filter=TimeDim%20eq%202021%20and%20Dim1%20eq%20'SEX_BTSX'%20and%20Dim2%20eq%20'AGEGROUP_YEARS18-PLUS'%20and%20NumericValue%20gt%2010&$select=SpatialDim,NumericValue&$top=5000"
+99,https://ghoapi.azureedge.net/api/NCD_BMI_30A?$filter=TimeDim%20eq%202021%20and%20Dim1%20eq%20'SEX_BTSX'%20and%20Dim2%20eq%20'AGEGROUP_YEARS18-PLUS'%20and%20NumericValue%20gt%2010&$top=5000
+100,https://zh.wikipedia.org/wiki/%E6%9F%AC%E5%9F%AC%E5%AF%A8%E7%8E%8B%E5%AE%B6%E7%B7%A8%E5%B9%B4%E5%8F%B2
+101,https://en.wikipedia.org/wiki/Where_Is_the_Love%3F_(The_Black_Eyed_Peas_song)
+102,https://web.archive.org/web/20200418034340/https://www.sports-reference.com/olympics/athletes/br/anne-brusletto-1.html
+103,https://web.archive.org/web/20030830023637/http://www.abchsn.com/users/aus_wildlife/broad_faced_potoroo.html
+104,https://web.archive.org/web/20110604164452/http://portal.unesco.org/ci/en/ev.php-URL_ID%3D21749%26URL_DO%3DDO_TOPIC%26URL_SECTION%3D201.html
+105,https://web.archive.org/web/20121013165655/http://www.palmspringswalkofstars.com/web-storage/Stars/Stars%20dedicated%20by%20date.pdf
+106,https://www.metmuseum.org/research-centers/leonard-a-lauder-research-center/research-resources/modern-art-index-project/cone
+107,https://thefourwallmag.wordpress.com/2015/12/07/daks-in-the-city-interview-with-bonang-matheba
+108,https://www.nfsa.gov.au/stories/articles/colour-tv-part-1
+109,https://alistapart.com/article/css-positioning-101/
+110,https://borthcat.york.ac.uk/index.php/gale-george-william-1900-1976-professor-of-medicine
+111,https://finance.yahoo.com/news/4ever-young-revolutionizing-wellness-aesthetics-190000784.html
+112,https://www.gbif.org/species/7301494
+113,https://nz.news.yahoo.com/history-behind-met-gala-215735843.html
+114,https://www.alamy.com/statue-of-alexander-mackenzie-1822-1892-pm-of-canada-1873-1878-the-statue-was-carved-by-louis-philippe-hbert-in-1900-and-placed-on-parliament-hill-in-1901-image185562270.html
+115,https://finance.yahoo.com/
+116,https://www.engadget.com/
+117,https://www.expedia.com/
+118,https://www.reuters.com/technology/artificial-intelligence/openai-launches-sora-create-video-prompt-2024-02-15/
+119,https://www.reuters.com/technology/
+120,https://www.historyofparliamentonline.org/volume/1820-1832/member/kerr-john-1794-1841
+121,https://www.ranker.com/list/famous-politicians-from-italy/reference?page=3
+122,https://owid.netlify.app/grapher/co2-per-capita.csv?time=2022
+123,https://owid.netlify.app/grapher/per-capita-energy-use.csv?time=2022
+124,https://owid.netlify.app/grapher/share-electricity-renewables.csv?time=2022
+125,https://www.bankrate.com/banking/savings/rates/
+126,https://www.bilibili.com/video/BV1uJ4m1H7Mh
+127,https://www.taptap.cn/moment/795408535407037831
+128,https://ecommons.cornell.edu/bitstream/handle/1813/18102/Wang_Hsien-Chung_1978.pdf
+129,http://www.gamefaqs.com/mac/707665-sid-meiers-civilization-v-brave-new-world/credit
+130,https://webarchive.nationalarchives.gov.uk/ukgwa/20240404230934/https://www.gov.uk/government/statistics/fly-tipping-in-england/local-authority-fly-tipping-enforcement-league-tables-for-england-202122
+131,https://epaper.fsonline.com.cn/fsrb/html/2026-04/19/content_71070_325266.htm
+132,https://www.gmw.cn/01gmrb/1999-01/05/GB/17928^GM5-0507.HTM
+133,https://max.book118.com/html/2018/0607/171282284.shtm
+134,https://www.transfermarkt.com/fc-barcelona_rcd-espanyol-barcelona/index/spielbericht/58777
+135,https://www.everand.com/author/366145856/Helen-Schreider
+136,https://www.scribd.com/document/402397780/PROS-SAGSSAGO-NOTES-IN-CRIMINAL-LAW-2017-EDITION-converted-docx
+137,"https://ghoapi.azureedge.net/api/M_Est_tob_curr_std?$filter=TimeDim%20eq%202021%20and%20Dim1%20eq%20'SEX_BTSX'%20and%20NumericValue%20gt%2030&$select=SpatialDim,NumericValue&$top=5000"
+138,https://ghoapi.azureedge.net/api/M_Est_tob_curr_std?$filter=TimeDim%20eq%202021%20and%20Dim1%20eq%20'SEX_BTSX'%20and%20NumericValue%20gt%2030&$top=5000
+139,https://m.chem960.com/msds/92875
+140,https://www.fzfzjx.com/tech_news/detail-6754.html
+141,https://www.state.gov/press-releases/
+142,https://www.bankrate.com/banking/savings/
+143,"https://ghoapi.azureedge.net/api/NCDMORT3070?$filter=TimeDim%20eq%202021%20and%20Dim1%20eq%20'SEX_BTSX'%20and%20Dim2%20eq%20'AGEGROUP_YEARS30-69'%20and%20NumericValue%20gt%2025&$select=SpatialDim,NumericValue&$top=5000"
+144,https://jssdfz.jiangsu.gov.cn/n101/20240612/i34078.html
+145,https://m.liuxue114.com/japan/jpmx/1358996225139.html
+146,https://data.worldbank.org/indicator/SL.AGR.0714.ZS?locations=EC
+147,https://staging.europepmc.org/backend/articlerender.fcgi?accid=PMC11156239
+148,https://www.qiuwenbaike.cn/wiki/%E4%BA%91%E5%AD%98%E5%82%A8
+149,https://www.qiuwenbaike.cn/wiki/%E7%94%B0%E5%83%96%E5%AD%90
+150,https://www.qiuwenbaike.cn/wiki/%E8%90%A5%E4%B8%9A%E7%A8%8E
+151,https://bloodstainedritualofthenight.wiki.fextralife.com/Partisan
+152,https://darksouls2.wiki.fextralife.com/Puzzling+Stone+Sword
+153,https://demonssouls.wiki.fextralife.com/Spears
+154,http://mp.weixin.qq.com/s?__biz=MjM5MDI1MDQ2Mw==&mid=400754239&idx=2&sn=ee6a9f8f5ac304c477f64fa1f66cb2b5&chksm=35aace048703251ed2d2c6c09ea90c1d7d9a2e477146e3a8bd649edccaafc7430849a72bfceb#rd
+155,http://mp.weixin.qq.com/s?__biz=MjM5ODgzODk2Mw==&mid=2652134873&idx=1&sn=e54b222dac2ee123809c91e103cc20ad&chksm=bc581578fe2e77a4166c4b880bef6337b724807a2cacf7efb9ca1bf06597fda29c721543dc2b#rd
+156,http://mp.weixin.qq.com/s?__biz=MzA4MDIxMjcyMA==&mid=2653599762&idx=3&sn=417d0c4afa2eacc6cc2f059c3ca8269e&chksm=8589040c58b452402abbdb5dd36178206036c4724ae2232e74517c749c22cc3961b7c197bc76#rd
+157,http://mp.weixin.qq.com/s?__biz=MzA5MzI3NjkyNA==&mid=2654050551&idx=1&sn=3620410479393ed9eb72e096b93b6a7f&chksm=8a6fcaebf0edeaf4e232f873051fc60f73984ba33078017849765bee30dcba96e65442852e10#rd
+158,https://sekiroshadowsdietwice.wiki.fextralife.com/Lady+Tomoe
+159,https://www.linkedin.com/in/williamhgates
diff --git a/moli-benchmark/moli_benchmark/chrome_dcl.py b/moli-benchmark/moli_benchmark/chrome_dcl.py
index 7bdafec1a..5525703d4 100644
--- a/moli-benchmark/moli_benchmark/chrome_dcl.py
+++ b/moli-benchmark/moli_benchmark/chrome_dcl.py
@@ -6,11 +6,13 @@ import signal
import subprocess
import tempfile
import time
+from dataclasses import dataclass
from pathlib import Path
from typing import Any
from .config import REPO_ROOT, reserve_port
from .process import ProcessResult
+from .public_web import POST_DCL_SETTLE_MILLISECONDS
from .raw_cdp import RawCdpClient, RawCdpError, connect_raw_cdp
from .sampling import ResourceSampler
from .target_serve import start_target_serve, stop_target_serve
@@ -26,6 +28,14 @@ DEFAULT_CHROME_DCL_USER_AGENT = (
# timeout. Keep this small so failed navigations do not distort timing reports.
CDP_LATE_ERROR_GRACE_SECONDS = 2.0
+_POST_DCL_SETTLE_EXPRESSION = (
+ "new Promise(resolve => "
+ f"setTimeout(resolve, {POST_DCL_SETTLE_MILLISECONDS}))"
+)
+_POST_DCL_OUTER_HTML_EXPRESSION = (
+ "document.documentElement ? document.documentElement.outerHTML : ''"
+)
+
class CdpDclDumpTimeoutError(TimeoutError):
def __init__(self, stage: str, error: BaseException) -> None:
@@ -40,6 +50,15 @@ class CdpDclDumpTimeoutError(TimeoutError):
return detail if detail else self.stage
+@dataclass(frozen=True)
+class CdpDclDumpResult:
+ body: str
+ response_status: int | None
+ response_mime_type: str | None
+ main_document_body_capture: str
+ final_url: str | None
+
+
def _chrome_command(binary: Path, port: int, profile_dir: Path) -> list[str]:
return [
str(binary),
@@ -126,19 +145,29 @@ async def _raise_late_command_error_or_timeout(
raise CdpDclDumpTimeoutError(stage, timeout_error) from timeout_error
-def _is_dcl_event(message: dict[str, Any], session_id: str, frame_id: str | None) -> bool:
+def _is_dcl_event(
+ message: dict[str, Any],
+ session_id: str,
+ frame_id: str | None,
+ expected_loader_id: str | None,
+) -> bool:
if message.get("sessionId") != session_id:
return False
- method = message.get("method")
- if method == "Page.domContentEventFired":
- return True
- if method != "Page.lifecycleEvent":
+ # Page.domContentEventFired has no loader identity. Lifecycle events are
+ # enabled for this runner and can be matched to the loader returned by
+ # Page.navigate below.
+ if message.get("method") != "Page.lifecycleEvent":
return False
params = message.get("params")
if not isinstance(params, dict):
return False
if frame_id is not None and params.get("frameId") != frame_id:
return False
+ if (
+ expected_loader_id is not None
+ and params.get("loaderId") != expected_loader_id
+ ):
+ return False
return params.get("name") in {"DOMContentLoaded", "domContentLoaded"}
@@ -170,40 +199,112 @@ def _is_binary_document_mime_type(mime_type: str) -> bool:
)
-def _binary_main_resource_body(mime_type: str) -> str:
- normalized = mime_type.split(";", 1)[0].strip().lower()
- if normalized == "application/pdf":
- return "%PDF-1.7\n% moli benchmark CDP binary main resource\n" + ("\0" * 512)
- return f"moli benchmark CDP binary main resource: {normalized}\n" + ("\0" * 512)
-
-
-def _binary_main_resource_body_from_message(
+def _main_document_response_from_message(
message: dict[str, Any],
*,
session_id: str,
frame_id: str | None,
-) -> str | None:
+ expected_loader_id: str | None = None,
+ main_document_request_ids: set[str] | None = None,
+) -> dict[str, Any] | None:
if message.get("sessionId") != session_id:
return None
- if message.get("method") != "Network.responseReceived":
- return None
params = message.get("params")
if not isinstance(params, dict):
return None
- if params.get("type") != "Document":
+ if (
+ expected_loader_id is not None
+ and params.get("loaderId") != expected_loader_id
+ ):
+ return None
+
+ method = message.get("method")
+ if method == "Network.requestWillBeSent":
+ if params.get("type") != "Document":
+ return None
+ if frame_id is not None and params.get("frameId") != frame_id:
+ return None
+ request_id = params.get("requestId")
+ if main_document_request_ids is not None and isinstance(request_id, str):
+ main_document_request_ids.clear()
+ main_document_request_ids.add(request_id)
+ return None
+
+ if method != "Network.responseReceived":
return None
if frame_id is not None and params.get("frameId") != frame_id:
return None
+ request_id = params.get("requestId")
+ request_was_document = (
+ isinstance(request_id, str)
+ and main_document_request_ids is not None
+ and request_id in main_document_request_ids
+ )
+ if (
+ isinstance(request_id, str)
+ and main_document_request_ids
+ and not request_was_document
+ ):
+ return None
+ if params.get("type") != "Document" and not request_was_document:
+ return None
+ if main_document_request_ids is not None and isinstance(request_id, str):
+ main_document_request_ids.clear()
+ main_document_request_ids.add(request_id)
response = params.get("response")
if not isinstance(response, dict):
return None
+ return response
+
+
+def _response_status(response: dict[str, Any]) -> int | None:
status = response.get("status")
- if isinstance(status, (int, float)) and not 200 <= status < 400:
+ if isinstance(status, bool) or not isinstance(status, (int, float)):
+ return None
+ return int(status)
+
+
+def _response_mime_type(response: dict[str, Any]) -> str | None:
+ mime_type = response.get("mimeType")
+ if not isinstance(mime_type, str):
+ return None
+ normalized = mime_type.strip().lower()
+ return normalized or None
+
+
+def _response_url(response: dict[str, Any]) -> str | None:
+ url = response.get("url")
+ return url if isinstance(url, str) and url else None
+
+
+def _binary_main_resource_mime_type_from_message(
+ message: dict[str, Any],
+ *,
+ session_id: str,
+ frame_id: str | None,
+ expected_loader_id: str | None = None,
+) -> str | None:
+ response = _main_document_response_from_message(
+ message,
+ session_id=session_id,
+ frame_id=frame_id,
+ expected_loader_id=expected_loader_id,
+ )
+ return _binary_main_resource_mime_type_from_response(response)
+
+
+def _binary_main_resource_mime_type_from_response(
+ response: dict[str, Any] | None,
+) -> str | None:
+ if response is None:
+ return None
+ status = _response_status(response)
+ if status is not None and not 200 <= status < 400:
return None
mime_type = response.get("mimeType")
if not isinstance(mime_type, str) or not _is_binary_document_mime_type(mime_type):
return None
- return _binary_main_resource_body(mime_type)
+ return _response_mime_type(response)
async def _recv_until_dcl_or_binary_main_resource(
@@ -211,36 +312,91 @@ async def _recv_until_dcl_or_binary_main_resource(
*,
session_id: str,
frame_id: str | None,
+ expected_loader_id: str | None,
deadline: float,
seen: list[dict[str, Any]],
-) -> str | None:
+) -> tuple[str | None, int | None, str | None, str | None]:
+ response_status: int | None = None
+ response_mime_type: str | None = None
+ response_url: str | None = None
+ main_document_request_ids: set[str] = set()
for message in seen:
- binary_body = _binary_main_resource_body_from_message(
+ response = _main_document_response_from_message(
message,
session_id=session_id,
frame_id=frame_id,
+ expected_loader_id=expected_loader_id,
+ main_document_request_ids=main_document_request_ids,
)
- if binary_body is not None:
- return binary_body
- if any(_is_dcl_event(message, session_id, frame_id) for message in seen):
- return None
+ if response is not None:
+ response_status = _response_status(response)
+ response_mime_type = _response_mime_type(response)
+ response_url = _response_url(response)
+ binary_mime_type = _binary_main_resource_mime_type_from_response(response)
+ if binary_mime_type is not None:
+ return (
+ binary_mime_type,
+ response_status,
+ response_mime_type,
+ response_url,
+ )
+ if any(
+ _is_dcl_event(message, session_id, frame_id, expected_loader_id)
+ for message in seen
+ ):
+ return None, response_status, response_mime_type, response_url
while True:
remaining = deadline - time.perf_counter()
if remaining <= 0:
raise TimeoutError("timed out waiting for Page.domContentEventFired")
message = await asyncio.wait_for(client.recv(), timeout=remaining)
- binary_body = _binary_main_resource_body_from_message(
+ response = _main_document_response_from_message(
message,
session_id=session_id,
frame_id=frame_id,
+ expected_loader_id=expected_loader_id,
+ main_document_request_ids=main_document_request_ids,
)
- if binary_body is not None:
- return binary_body
- if _is_dcl_event(message, session_id, frame_id):
- return None
+ if response is not None:
+ response_status = _response_status(response)
+ response_mime_type = _response_mime_type(response)
+ response_url = _response_url(response)
+ binary_mime_type = _binary_main_resource_mime_type_from_response(response)
+ if binary_mime_type is not None:
+ return (
+ binary_mime_type,
+ response_status,
+ response_mime_type,
+ response_url,
+ )
+ if _is_dcl_event(message, session_id, frame_id, expected_loader_id):
+ return None, response_status, response_mime_type, response_url
+
+
+def _navigation_frame_and_loader(
+ navigate_response: dict[str, Any],
+ url: str,
+) -> tuple[str | None, str | None]:
+ result = navigate_response.get("result")
+ if not isinstance(result, dict):
+ result = {}
+ error_text = result.get("errorText")
+ if isinstance(error_text, str) and error_text:
+ raise RawCdpError(f"Page.navigate failed for `{url}`: {error_text}")
+ frame_id = result.get("frameId")
+ loader_id = result.get("loaderId")
+ return (
+ str(frame_id) if frame_id is not None else None,
+ str(loader_id) if loader_id is not None else None,
+ )
-async def _dump_dcl_html(endpoint: str, process: subprocess.Popen[bytes], url: str, timeout_seconds: float) -> str:
+async def _dump_dcl_html(
+ endpoint: str,
+ process: subprocess.Popen[bytes],
+ url: str,
+ timeout_seconds: float,
+) -> CdpDclDumpResult:
deadline = time.perf_counter() + timeout_seconds
try:
client = await _wait_for_cdp(endpoint, process, min(5.0, max(0.1, timeout_seconds)))
@@ -290,26 +446,56 @@ async def _dump_dcl_html(endpoint: str, process: subprocess.Popen[bytes], url: s
stage="Page.navigate",
late_error_grace_seconds=CDP_LATE_ERROR_GRACE_SECONDS,
)
- frame_id = navigate_response.get("result", {}).get("frameId")
- if frame_id is not None:
- frame_id = str(frame_id)
+ frame_id, expected_loader_id = _navigation_frame_and_loader(
+ navigate_response,
+ url,
+ )
try:
- binary_body = await _recv_until_dcl_or_binary_main_resource(
+ main_document = await _recv_until_dcl_or_binary_main_resource(
client,
session_id=session_id,
frame_id=frame_id,
+ expected_loader_id=expected_loader_id,
deadline=deadline,
seen=seen,
)
+ (
+ binary_mime_type,
+ response_status,
+ response_mime_type,
+ final_url,
+ ) = main_document
except TimeoutError as error:
raise CdpDclDumpTimeoutError("DCL", error) from error
- if binary_body is not None:
- return binary_body
+ if binary_mime_type is not None:
+ return CdpDclDumpResult(
+ body="",
+ response_status=response_status,
+ response_mime_type=binary_mime_type,
+ main_document_body_capture="response-headers-only",
+ final_url=final_url,
+ )
+
+ settle_id = await client.send(
+ "Runtime.evaluate",
+ {
+ "expression": _POST_DCL_SETTLE_EXPRESSION,
+ "awaitPromise": True,
+ "returnByValue": True,
+ },
+ session_id=session_id,
+ )
+ await _recv_command_response(
+ client,
+ settle_id,
+ deadline=deadline,
+ stage="post-DCL settle",
+ )
evaluate_id = await client.send(
"Runtime.evaluate",
{
- "expression": "document.documentElement ? document.documentElement.outerHTML : ''",
+ "expression": _POST_DCL_OUTER_HTML_EXPRESSION,
"returnByValue": True,
},
session_id=session_id,
@@ -322,7 +508,13 @@ async def _dump_dcl_html(endpoint: str, process: subprocess.Popen[bytes], url: s
)
result = evaluate_response.get("result", {}).get("result", {})
value = result.get("value", "")
- return value if isinstance(value, str) else ""
+ return CdpDclDumpResult(
+ body=value if isinstance(value, str) else "",
+ response_status=response_status,
+ response_mime_type=response_mime_type,
+ main_document_body_capture="dom-snapshot",
+ final_url=final_url,
+ )
finally:
if target_id is not None:
try:
@@ -383,6 +575,10 @@ def run_chrome_dcl_dump(
error_suffix = b""
timed_out = False
returncode: int | None = None
+ response_status: int | None = None
+ response_mime_type: str | None = None
+ main_document_body_capture: str | None = None
+ final_url: str | None = None
with tempfile.TemporaryDirectory(prefix="moli-benchmark-chrome-") as temp_dir:
profile_dir = Path(temp_dir) / "profile"
profile_dir.mkdir()
@@ -409,8 +605,14 @@ def run_chrome_dcl_dump(
sampler.start()
try:
try:
- html = asyncio.run(_dump_dcl_html(endpoint, process, url, timeout_seconds))
- stdout = html.encode("utf-8", errors="replace")
+ dump = asyncio.run(
+ _dump_dcl_html(endpoint, process, url, timeout_seconds)
+ )
+ stdout = dump.body.encode("utf-8", errors="replace")
+ response_status = dump.response_status
+ response_mime_type = dump.response_mime_type
+ main_document_body_capture = dump.main_document_body_capture
+ final_url = dump.final_url
returncode = 0
except CdpDclDumpTimeoutError as error:
timed_out = True
@@ -435,7 +637,11 @@ def run_chrome_dcl_dump(
finally:
elapsed_ms = (time.perf_counter() - started) * 1000.0
resources = sampler.stop() if sampler is not None else {}
- if browser_stdout.strip() and not stdout:
+ if (
+ browser_stdout.strip()
+ and not stdout
+ and main_document_body_capture is None
+ ):
stdout = browser_stdout
return ProcessResult(
command=command,
@@ -445,6 +651,10 @@ def run_chrome_dcl_dump(
stderr=stderr,
timed_out=timed_out,
resources=resources,
+ response_status=response_status,
+ response_mime_type=response_mime_type,
+ main_document_body_capture=main_document_body_capture,
+ final_url=final_url,
)
@@ -466,13 +676,23 @@ def run_served_cdp_dcl_dump(
timed_out = False
returncode: int | None = None
resources: dict[str, Any] = {}
+ response_status: int | None = None
+ response_mime_type: str | None = None
+ main_document_body_capture: str | None = None
+ final_url: str | None = None
serve = None
try:
serve = start_target_serve(target, binary, timeout_seconds)
command = serve.command
try:
- html = asyncio.run(_dump_dcl_html(serve.endpoint, serve.process, url, timeout_seconds))
- stdout = html.encode("utf-8", errors="replace")
+ dump = asyncio.run(
+ _dump_dcl_html(serve.endpoint, serve.process, url, timeout_seconds)
+ )
+ stdout = dump.body.encode("utf-8", errors="replace")
+ response_status = dump.response_status
+ response_mime_type = dump.response_mime_type
+ main_document_body_capture = dump.main_document_body_capture
+ final_url = dump.final_url
returncode = 0
except CdpDclDumpTimeoutError as error:
timed_out = True
@@ -521,4 +741,8 @@ def run_served_cdp_dcl_dump(
stderr=stderr,
timed_out=timed_out,
resources=resources,
+ response_status=response_status,
+ response_mime_type=response_mime_type,
+ main_document_body_capture=main_document_body_capture,
+ final_url=final_url,
)
diff --git a/moli-benchmark/moli_benchmark/html_report.py b/moli-benchmark/moli_benchmark/html_report.py
index 419fb0d8b..ce1464bd0 100644
--- a/moli-benchmark/moli_benchmark/html_report.py
+++ b/moli-benchmark/moli_benchmark/html_report.py
@@ -250,7 +250,11 @@ def _artifact_paths_by_suite(summaries: list[dict[str, Any]]) -> dict[str, tuple
"crawler": ("crawler/summary.json", "crawler/raw-runs.csv"),
"amiibo-crawler": ("amiibo-crawler/summary.json", "amiibo-crawler/raw-runs.csv"),
"wild-web": ("wild-web/summary.json", "wild-web/raw-runs.csv"),
- "top-sites": ("top-sites/summary.json", "top-sites/raw-runs.csv"),
+ "top-sites": (
+ "top-sites/summary.json",
+ "top-sites/raw-runs.csv",
+ "top-sites/site-outcomes.json",
+ ),
"render-compare": (
"render-compare/summary.json",
"render-compare/raw-runs.csv",
@@ -997,11 +1001,25 @@ function targetLabel(targets, target) {
function targetColor(targets, target) {
return colors[targetEngine(targets, target)] || '#64748b';
}
-function firstHorizontalSummary() {
+function horizontalSummaries() {
const comparisons = report.horizontal_comparisons || [];
- if (comparisons.length) return comparisons[0];
const summaries = report.summaries || [];
- return summaries.find(s => s.suite === 'synthetic-compare') || summaries.find(s => s.suite === 'cdp-session') || summaries.find(s => s.targets);
+ const result = [...comparisons];
+ const seen = new Set(result.map(summary => summary.suite));
+ for (const summary of summaries) {
+ if (!summary?.targets || seen.has(summary.suite)) continue;
+ result.push(summary);
+ seen.add(summary.suite);
+ }
+ return result;
+}
+function firstHorizontalSummary() {
+ const summaries = horizontalSummaries();
+ for (const suite of ['web-scraping-variants', 'synthetic-compare', 'top-sites', 'wild-web', 'cdp-session']) {
+ const summary = summaries.find(candidate => candidate.suite === suite);
+ if (summary) return summary;
+ }
+ return summaries[0];
}
function summaryCases(summary) {
const cases = summary?.cases || [];
@@ -1014,8 +1032,20 @@ function resultFor(summary, target, caseName) {
if (isAggregateSummary(summary)) return summary?.targets?.[target] || {};
return summary?.targets?.[target]?.cases?.[caseName] || {};
}
-function metricFor(summary, target, caseName, metric) {
+function performanceResultFor(summary, target, caseName) {
+ if (!isAggregateSummary(summary)) return resultFor(summary, target, caseName);
+ const common = summary?.common_success?.targets?.[target];
+ if (common && Number(common?.elapsed_ms?.count || 0) > 0) return common;
const result = resultFor(summary, target, caseName);
+ return {
+ ...result,
+ elapsed_ms: result?.successful_elapsed_ms || result?.elapsed_ms,
+ peak_pss_bytes: result?.successful_peak_pss_bytes || result?.peak_pss_bytes,
+ peak_rss_bytes: result?.successful_peak_rss_bytes || result?.peak_rss_bytes,
+ };
+}
+function metricFor(summary, target, caseName, metric) {
+ const result = performanceResultFor(summary, target, caseName);
return result?.[metric]?.p50 ?? null;
}
function targetFailures(summary, target) {
@@ -1033,7 +1063,7 @@ function horizontalCells(summary) {
const targets = targetNames(summary?.targets || {});
const cases = summaryCases(summary);
return cases.flatMap(caseName => targets.map(target => {
- const result = resultFor(summary, target, caseName);
+ const result = performanceResultFor(summary, target, caseName);
return {caseName, target, result};
}));
}
@@ -1055,6 +1085,30 @@ function passRate(result) {
if (!denominator) return '';
return `${passes}/${denominator} (${(passes / denominator * 100).toFixed(1)}%)`;
}
+function rawPassRate(result) {
+ const passes = Number(result?.raw_passes);
+ const attempts = Number(result?.raw_comparable_attempts);
+ if (!Number.isFinite(passes) || !Number.isFinite(attempts) || attempts <= 0) return '';
+ return `${passes}/${attempts} (${(passes / attempts * 100).toFixed(1)}%)`;
+}
+function siteStability(result) {
+ const sites = Number(result?.unique_sites);
+ if (!Number.isFinite(sites) || sites <= 0) return '';
+ const runs = Number(result?.runs || 0);
+ if (!result?.repeat_validated) return `${sites} sites; ${runs || 1} run(s), not repeat-validated`;
+ return `${Number(result?.all_pass_sites || 0)} all-pass / ${Number(result?.flaky_sites || 0)} flaky / ${Number(result?.all_fail_sites || 0)} all-fail`;
+}
+function statusCoverage(result) {
+ const observed = Number(result?.status_observed_attempts);
+ const attempts = Number(result?.observed_sites ?? result?.attempts);
+ const denominator = Number.isFinite(attempts) && attempts > 0 ? attempts : Number(result?.seeds);
+ if (!Number.isFinite(observed) || !Number.isFinite(denominator) || denominator <= 0) return '';
+ const sources = Object.entries(result?.response_status_sources || {})
+ .map(([source, count]) => `${source}: ${count}`)
+ .join(', ');
+ const coverage = `${observed}/${denominator} (${(observed / denominator * 100).toFixed(1)}%)`;
+ return sources ? `${coverage}; ${sources}` : coverage;
+}
function breakdown(items) {
if (!items || typeof items !== 'object') return '';
return Object.entries(items).map(([key, value]) => `${htmlEscape(key)}: ${htmlEscape(value)}`).join(', ');
@@ -1108,12 +1162,14 @@ function renderExecutive() {
if (horizontal) {
const fastest = fastestCell(horizontal);
const memory = lowestMemoryCell(horizontal);
+ const commonAttempts = Number(horizontal?.common_success?.attempts || 0);
+ const cohortNote = commonAttempts > 0 ? ` over ${commonAttempts} common-success attempt(s)` : '';
const failureTargets = targetNames(horizontal.targets || {}).filter(target => targetFailures(horizontal, target) > 0);
document.getElementById('executiveSummary').innerHTML = `
<div class="finding-list">
<div class="finding">Horizontal source: <strong>${htmlEscape(horizontal.suite)}</strong>. Gate target: <strong>${htmlEscape(horizontal.gate_target || 'moli')}</strong>.</div>
- <div class="finding">${fastest ? `Fastest p50 cell: <strong>${htmlEscape(targetLabel(horizontal.targets, fastest.target))}</strong> / ${htmlEscape(fastest.caseName)} at ${number(fastest.result.elapsed_ms.p50)} ms.` : 'No successful latency cell is available.'}</div>
- <div class="finding">${memory ? `Lowest PSS cell: <strong>${htmlEscape(targetLabel(horizontal.targets, memory.target))}</strong> / ${htmlEscape(memory.caseName)} at ${mib(memory.result.peak_pss_bytes.p50)}.` : 'No successful memory cell is available.'}</div>
+ <div class="finding">${fastest ? `Fastest comparable-success p50: <strong>${htmlEscape(targetLabel(horizontal.targets, fastest.target))}</strong> / ${htmlEscape(fastest.caseName)} at ${number(fastest.result.elapsed_ms.p50)} ms${cohortNote}.` : 'No comparable successful latency cell is available.'}</div>
+ <div class="finding">${memory ? `Lowest comparable-success PSS: <strong>${htmlEscape(targetLabel(horizontal.targets, memory.target))}</strong> / ${htmlEscape(memory.caseName)} at ${mib(memory.result.peak_pss_bytes.p50)}${cohortNote}.` : 'No comparable successful memory cell is available.'}</div>
<div class="finding ${failureTargets.length ? 'warn' : ''}">${failureTargets.length ? `Targets with comparison failures: <strong>${htmlEscape(failureTargets.map(target => targetLabel(horizontal.targets, target)).join(', '))}</strong>.` : 'No comparison target failures in this suite.'}</div>
</div>
`;
@@ -1170,30 +1226,58 @@ function renderSuites() {
</tr>`);
document.getElementById('suiteTable').innerHTML = table(['suite', 'profile', 'gate target', 'gate failures', 'all failures', 'runs', 'timeout'], rows);
}
-function renderComparisonTable() {
- const summary = firstHorizontalSummary();
- if (!summary || !summary.targets) {
- document.getElementById('comparisonTable').innerHTML = '<p class="muted">No horizontal comparison data.</p>';
- return;
- }
+function comparisonSummaryHtml(summary) {
const targets = targetNames(summary.targets);
if (isAggregateSummary(summary)) {
const rows = targets.map(target => {
const result = summary.targets[target] || {};
+ const successfulElapsed = result.successful_elapsed_ms || result.elapsed_ms || {};
+ const successfulPss = result.successful_peak_pss_bytes || result.peak_pss_bytes || {};
+ const common = summary?.common_success?.targets?.[target] || {};
+ const commonAttempts = Number(summary?.common_success?.attempts || 0);
+ const exclusions = [
+ Number(result.excluded_runs || 0) ? `excluded: ${result.excluded_runs}` : '',
+ Number(result.non_comparable_attempts || 0) ? `neutral: ${result.non_comparable_attempts}` : '',
+ ].filter(Boolean).join('; ');
return `<tr>
<th>${htmlEscape(targetLabel(summary.targets, target))}</th>
<td>${passRate(result)}</td>
+ <td>${rawPassRate(result)}</td>
+ <td>${htmlEscape(exclusions)}</td>
+ <td>${htmlEscape(siteStability(result))}</td>
+ <td>${htmlEscape(statusCoverage(result))}</td>
<td>${htmlEscape(result.successes ?? result.categories?.success ?? result.passes ?? '')}</td>
- <td>${htmlEscape(result.categories?.challenge ?? '')}</td>
- <td>${htmlEscape(result.categories?.thin ?? '')}</td>
<td>${htmlEscape(result.failures ?? '')}</td>
- <td>${number(result.elapsed_ms?.p50)} / ${number(result.elapsed_ms?.p90)} / ${number(result.elapsed_ms?.p95)}</td>
- <td>${mib(result.peak_pss_bytes?.p50)} / ${mib(result.peak_pss_bytes?.p95)}</td>
+ <td>${number(successfulElapsed.p50)} / ${number(successfulElapsed.p90)} / ${number(successfulElapsed.p95)}</td>
+ <td>${mib(successfulPss.p50)} / ${mib(successfulPss.p95)}</td>
+ <td>${commonAttempts || ''}</td>
+ <td>${number(common.elapsed_ms?.p50)} / ${number(common.elapsed_ms?.p90)} / ${number(common.elapsed_ms?.p95)}</td>
+ <td>${mib(common.peak_pss_bytes?.p50)} / ${mib(common.peak_pss_bytes?.p95)}</td>
<td>${breakdown(result.failure_kinds)}</td>
</tr>`;
});
- document.getElementById('comparisonTable').innerHTML = table(['target', 'pass rate', 'success', 'challenge', 'thin', 'failures', 'p50 / p90 / p95 ms', 'PSS p50 / p95', 'failure kinds'], rows);
- return;
+ const pairwise = Array.isArray(summary.pairwise) ? summary.pairwise : [];
+ const pairRows = pairwise.map(item => `<tr>
+ <th>${htmlEscape(targetLabel(summary.targets, item.left))} vs ${htmlEscape(targetLabel(summary.targets, item.right))}</th>
+ <td>${badge(item.repeat_validated ? 'REPEAT VALIDATED' : 'SINGLE SAMPLE')}</td>
+ <td>${htmlEscape(item.both_all_pass ?? '')}</td>
+ <td>${htmlEscape(item.left_all_pass_right_all_fail ?? '')}</td>
+ <td>${htmlEscape(item.right_all_pass_left_all_fail ?? '')}</td>
+ <td>${htmlEscape(item.both_all_fail ?? '')}</td>
+ <td>${htmlEscape(item.flaky_or_incomplete ?? '')}</td>
+ <td>${htmlEscape(item.not_comparable ?? '')}</td>
+ <td>${htmlEscape(item.common_success?.attempts ?? '')}</td>
+ </tr>`);
+ const pairTable = pairRows.length
+ ? `<h4>Pairwise site outcomes</h4>${table(['targets', 'validation', 'both pass', 'left only', 'right only', 'both fail', 'flaky / incomplete', 'not comparable', 'common-success attempts'], pairRows)}`
+ : '';
+ const contract = summary.snapshot_contract
+ ? `<p class="muted">Snapshot contract: ${htmlEscape(summary.snapshot_contract)}</p>`
+ : '';
+ return contract + table(
+ ['target', 'counted pass rate', 'raw pass rate', 'excluded / neutral', 'site stability', 'HTTP status observed', 'success', 'failures', 'successful p50 / p90 / p95 ms', 'successful PSS p50 / p95', 'common n', 'common p50 / p90 / p95 ms', 'common PSS p50 / p95', 'failure kinds'],
+ rows,
+ ) + pairTable;
}
const cases = summaryCases(summary);
const rows = cases.map(caseName => {
@@ -1207,7 +1291,20 @@ function renderComparisonTable() {
}).join('');
return `<tr><th>${htmlEscape(caseName)}</th>${cells}</tr>`;
});
- document.getElementById('comparisonTable').innerHTML = table(['case', ...targets.map(target => targetLabel(summary.targets, target))], rows);
+ return table(['case', ...targets.map(target => targetLabel(summary.targets, target))], rows);
+}
+function renderComparisonTable() {
+ const summaries = horizontalSummaries();
+ if (!summaries.length) {
+ document.getElementById('comparisonTable').innerHTML = '<p class="muted">No horizontal comparison data.</p>';
+ return;
+ }
+ document.getElementById('comparisonTable').innerHTML = summaries.map(summary => `
+ <section class="comparison-summary">
+ <h3>${htmlEscape(summary.suite || 'comparison')}</h3>
+ ${comparisonSummaryHtml(summary)}
+ </section>
+ `).join('');
}
function renderWebPages() {
const section = document.getElementById('web-pages');
diff --git a/moli-benchmark/moli_benchmark/process.py b/moli-benchmark/moli_benchmark/process.py
index 59c4b9f5f..b26156112 100644
--- a/moli-benchmark/moli_benchmark/process.py
+++ b/moli-benchmark/moli_benchmark/process.py
@@ -22,6 +22,10 @@ class ProcessResult:
resources: dict[str, Any]
time_verbose: dict[str, Any] | None = None
cgroup: dict[str, Any] | None = None
+ response_status: int | None = None
+ response_mime_type: str | None = None
+ main_document_body_capture: str | None = None
+ final_url: str | None = None
def output_digest_material(self) -> bytes:
return self.stdout + b"\n--- stderr ---\n" + self.stderr
@@ -40,6 +44,14 @@ class ProcessResult:
summary["time_verbose"] = self.time_verbose
if self.cgroup is not None:
summary["cgroup"] = self.cgroup
+ if self.response_status is not None:
+ summary["response_status"] = self.response_status
+ if self.response_mime_type is not None:
+ summary["response_mime_type"] = self.response_mime_type
+ if self.main_document_body_capture is not None:
+ summary["main_document_body_capture"] = self.main_document_body_capture
+ if self.final_url is not None:
+ summary["final_url"] = self.final_url
if include_output:
summary["stdout_tail"] = self.stdout[-4096:].decode("utf-8", errors="replace")
summary["stderr_tail"] = self.stderr[-4096:].decode("utf-8", errors="replace")
diff --git a/moli-benchmark/moli_benchmark/public_web.py b/moli-benchmark/moli_benchmark/public_web.py
new file mode 100644
index 000000000..1569ff29b
--- /dev/null
+++ b/moli-benchmark/moli_benchmark/public_web.py
@@ -0,0 +1,1306 @@
+from __future__ import annotations
+
+import html.parser
+import re
+from collections.abc import Callable, Mapping, Sequence
+from concurrent.futures import ThreadPoolExecutor, as_completed
+from dataclasses import dataclass
+from datetime import datetime, timezone
+from hashlib import sha256
+from pathlib import Path
+from threading import BoundedSemaphore
+from typing import Any, Generic, TypeVar
+
+from .artifacts import write_json, write_text
+from .config import REPO_ROOT
+from .process import ProcessResult
+from .stats import summarize
+from .synthetic_compare import (
+ target_enables_all_resource_fetch,
+ target_is_cdp,
+ target_metadata,
+)
+
+
+POST_DCL_SETTLE_MILLISECONDS = 50
+PUBLIC_WEB_SNAPSHOT_CONTRACT = (
+ "at least "
+ f"{POST_DCL_SETTLE_MILLISECONDS} ms of page event-loop time after "
+ "the adapter observes DOMContentLoaded"
+)
+POST_DCL_WAIT_SCRIPT = (
+ "globalThis.__moliBenchmarkPostDclReady === true || "
+ "(setTimeout(() => { globalThis.__moliBenchmarkPostDclReady = true; }, "
+ f"{POST_DCL_SETTLE_MILLISECONDS}), false)"
+)
+
+_MAIN_DOCUMENT_STATUS_DIAGNOSTIC = re.compile(
+ rb"(?:lifecycle\s+target\s+document|http\s+request)\s+"
+ rb"`[^`\r\n]+`\s+returned\s+([45]\d{2})\b",
+ re.IGNORECASE,
+)
+_LIGHTPANDA_FETCH_TIMEOUT_DIAGNOSTIC = re.compile(
+ rb'\$msg="fetch error"[^\r\n]*\berr=timeout\b',
+ re.IGNORECASE,
+)
+_SAFE_ARTIFACT_TOKEN = re.compile(r"[^A-Za-z0-9._-]+")
+_TITLE_RE = re.compile(r"<title[^>]*>(.*?)</title>", re.IGNORECASE | re.DOTALL)
+_TOP_SITES_BLOCKED_TITLE_403 = re.compile(r"^\s*(?:http\s*)?403\b")
+_TOP_SITES_HTTP_ERROR_TITLE_STATUS = re.compile(
+ r"^\s*(?:"
+ r"error\s*:\s*([45]\d{2})\b"
+ r"|(?:http\s*)?([45]\d{2})\s+(?:"
+ r"bad\s+gateway|gateway\s+timeout|internal\s+server\s+error|"
+ r"service\s+unavailable|forbidden|not\s+found|unauthorized|"
+ r"too\s+many\s+requests|request\s+timeout|error"
+ r")\b"
+ r")",
+ re.IGNORECASE,
+)
+_WILD_WEB_HTTP_ERROR_TITLE_STATUS = re.compile(
+ r"^\s*(?:"
+ r"error\s*:\s*([45]\d{2})\b"
+ r"|(?:http\s*)?([45]\d{2})\s+(?:"
+ r"bad\s+gateway|gateway\s+timeout|internal\s+server\s+error|"
+ r"service\s+unavailable|forbidden|not\s+found|unauthorized|error"
+ r")\b"
+ r")",
+ re.IGNORECASE,
+)
+
+_TOP_SITES_NETWORK_ERROR_MARKERS = (
+ "privacy error",
+ "your connection is not private",
+ "net::err_cert",
+ "this site can't be reached",
+ "this site can’t be reached",
+ "err_name_not_resolved",
+ "err_connection",
+ "err_timed_out",
+ "dns_probe_finished",
+ "could not resolve",
+ "could not resolve host",
+ "could not resolve hostname",
+ "failed to resolve",
+ "failed to lookup address information",
+ "could not connect to server",
+ "name resolution",
+ "connection refused",
+ "connection reset",
+ "connection timed out",
+ "no route to host",
+ "network is unreachable",
+ "ssl handshake",
+ "ssl error",
+ "ssl connect error",
+ "tls handshake",
+ "peerfailedverification",
+ "peer failed verification",
+ "timed out connecting",
+ "timeout was reached",
+ "request timeout",
+ "i/o error",
+ "broken pipe",
+ "operation timed out",
+ "failed to connect",
+ "dns lookup",
+ "curl request failed",
+ "recv failure",
+)
+_WILD_WEB_NETWORK_ERROR_MARKERS = (
+ "peerfailedverification",
+ "peer failed verification",
+ "privacy error",
+ "your connection is not private",
+ "this site can't be reached",
+ "this site can’t be reached",
+ "net::err_",
+ "could not resolve",
+ "failed to resolve",
+ "failed to lookup address information",
+ "connection refused",
+ "curl request failed",
+ "couldntresolvehost",
+ "http2stream",
+)
+_NAVIGATION_NETWORK_FAILURE_MARKERS = (
+ "couldntresolvehost",
+ "http2stream",
+ "reason: http2",
+)
+_CLI_DEADLINE_COMMAND_MARKERS = (
+ "fetch_document_allow_http_error_with_wait_until",
+ "fetch document allow-http-error",
+ "fetch allow-http-error wait_until",
+ "fetch wait_until",
+)
+_CLI_DEADLINE_TIMEOUT_MARKERS = ("timed out after",)
+_CLI_FETCH_READINESS_TIMEOUT_MARKER = "fetch readiness timed out after"
+_ELAPSED_FAILURE_TIMEOUT_GRACE_SECONDS = 1.0
+_CAPTCHA_MARKERS = (
+ "captcha",
+ "安全验证",
+ "验证你是真人",
+ "人机验证",
+ "human verification",
+ "verification",
+ "verify you are human",
+ "verify that you're not a robot",
+ "verify that you are not a robot",
+ "are you a robot",
+ "not a robot",
+ "bot or not",
+ "checking your browser",
+ "perform security check",
+ "security verification",
+ "performing security verification",
+ "请完成验证",
+ "完成验证后",
+ "当前环境异常",
+ "select the 2 matching tiles",
+ "powered and protected by privacy",
+)
+_LOGIN_MARKERS = ("sign in", "log in", "login", "登录", "注册", "账号")
+_LOGIN_FORM_MARKERS = (
+ "login to your account",
+ "email/username",
+ "your password is a required field",
+ "forgot password",
+ "create a free account",
+)
+_LOGIN_CONTEXT_MARKERS = (
+ "password",
+ "验证码",
+ "短信验证码",
+ "语音验证码",
+ "手机号",
+ "手机验证",
+ "邮箱",
+ "email",
+)
+_JS_CHALLENGE_MARKERS = (
+ "__cf_chl_",
+ "c2wf946j0/probe",
+ "cf-challenge",
+ "__tencent_chaos_vm",
+ "__eo_jschallenge_vm",
+ "teojschallengesdk.js",
+ "eojschallengesdk",
+ "window.solvechallenge(",
+ "jsl_clearance",
+ "acw_sc__v2",
+ "window._phantom",
+ "_$jsvmprt",
+ "awswaf",
+ "aliyun_waf",
+ "aliyun waf",
+ "_waf_",
+ "just a moment...",
+ "vercel security checkpoint",
+ "we're verifying your browser",
+ "we’re verifying your browser",
+ "enable javascript and cookies to continue",
+ "javascript is needed to access this site",
+)
+_BLOCKED_BODY_MARKERS = (
+ "unable to give you access to our site",
+ "access denied",
+ "access to this page has been denied",
+ "sorry, you have been blocked",
+ "security issue was automatically identified",
+ "access restricted",
+ "exception: forbidden",
+ "waf拦截",
+ "被waf拦截",
+ "security detection powered by safeline waf",
+)
+_NOT_FOUND_MARKERS = (
+ "404 not found",
+ "file not found",
+ "page not found",
+ "this page cannot be found",
+ "page you requested is missing",
+ "page you requested could not be found",
+ "页面不存在",
+ "视频不见了",
+)
+_FORBIDDEN_ERROR_MARKERS = ("403 forbidden", "returned 403", "http 403")
+_NOT_FOUND_ERROR_MARKERS = ("returned 404", "http 404")
+
+
+def reports_lightpanda_fetch_timeout(stderr: bytes) -> bool:
+ """Return whether Lightpanda reported its terminal fetch-timeout record."""
+
+ return _LIGHTPANDA_FETCH_TIMEOUT_DIAGNOSTIC.search(stderr) is not None
+
+
+def reports_cli_fetch_timeout(stderr: bytes, returncode: int | None) -> bool:
+ """Recognize stable typed and legacy Moli fetch-deadline diagnostics."""
+
+ if returncode in (None, 0):
+ return False
+ diagnostic = stderr.decode("utf-8", errors="replace").lower()
+ if _CLI_FETCH_READINESS_TIMEOUT_MARKER in diagnostic:
+ return True
+ return any(marker in diagnostic for marker in _CLI_DEADLINE_COMMAND_MARKERS) and any(
+ marker in diagnostic for marker in _CLI_DEADLINE_TIMEOUT_MARKERS
+ )
+
+
+def resolve_main_document_status(
+ protocol_status: int | None,
+ stderr: bytes,
+ returncode: int | None,
+) -> tuple[int | None, str | None]:
+ """Return the main-document status and the evidence that supplied it.
+
+ CDP is the preferred source. Moli's fetch CLI also includes the terminal
+ main-document status in its non-zero-exit diagnostic, so preserve that
+ structured evidence instead of reducing it to a generic process error.
+ The diagnostic matcher is deliberately narrow to avoid treating a logged
+ subresource failure as the main navigation status.
+ """
+
+ if protocol_status is not None:
+ return protocol_status, "protocol"
+ if returncode in {None, 0}:
+ return None, None
+ matches = _MAIN_DOCUMENT_STATUS_DIAGNOSTIC.findall(stderr)
+ if not matches:
+ return None, None
+ return int(matches[-1]), "cli-diagnostic"
+
+
+def utc_timestamp() -> str:
+ return datetime.now(timezone.utc).isoformat().replace("+00:00", "Z")
+
+
+def rotated_target_order(
+ targets: tuple[str, ...],
+ schedule_index: int,
+) -> tuple[str, ...]:
+ if not targets:
+ return ()
+ offset = schedule_index % len(targets)
+ return (*targets[offset:], *targets[:offset])
+
+
+@dataclass(frozen=True)
+class PublicWebSnapshotProfile:
+ hidden_tags: frozenset[str]
+ preserve_title_entities: bool = False
+ sample_chars: int = 500
+
+
+TOP_SITES_SNAPSHOT_PROFILE = PublicWebSnapshotProfile(
+ hidden_tags=frozenset(
+ {"head", "script", "style", "noscript", "svg", "template"}
+ ),
+ preserve_title_entities=True,
+)
+WILD_WEB_SNAPSHOT_PROFILE = PublicWebSnapshotProfile(
+ hidden_tags=frozenset({"script", "style", "noscript"}),
+)
+
+
+@dataclass(frozen=True)
+class PublicWebArtifactPolicy:
+ output_sample_bytes: int
+ failure_stdout_limit_bytes: int
+ failure_stderr_limit_bytes: int
+ write_empty_failure_stdout: bool
+
+
+TOP_SITES_ARTIFACT_POLICY = PublicWebArtifactPolicy(
+ output_sample_bytes=512,
+ failure_stdout_limit_bytes=32 * 1024,
+ failure_stderr_limit_bytes=16 * 1024,
+ write_empty_failure_stdout=False,
+)
+WILD_WEB_ARTIFACT_POLICY = PublicWebArtifactPolicy(
+ output_sample_bytes=1024,
+ failure_stdout_limit_bytes=128 * 1024,
+ failure_stderr_limit_bytes=32 * 1024,
+ write_empty_failure_stdout=True,
+)
+
+
+class _PublicWebTextExtractor(html.parser.HTMLParser):
+ def __init__(self, hidden_tags: frozenset[str]) -> None:
+ super().__init__(convert_charrefs=True)
+ self._hidden_tags = hidden_tags
+ self._hidden_depth = 0
+ self._in_title = False
+ self.title_parts: list[str] = []
+ self.text_parts: list[str] = []
+
+ def handle_starttag(
+ self,
+ tag: str,
+ attrs: list[tuple[str, str | None]],
+ ) -> None:
+ del attrs
+ normalized = tag.lower()
+ if normalized in self._hidden_tags:
+ self._hidden_depth += 1
+ if normalized == "title":
+ self._in_title = True
+
+ def handle_endtag(self, tag: str) -> None:
+ normalized = tag.lower()
+ if normalized == "title":
+ self._in_title = False
+ if normalized in self._hidden_tags and self._hidden_depth:
+ self._hidden_depth -= 1
+
+ def handle_data(self, data: str) -> None:
+ text = " ".join(data.split())
+ if not text:
+ return
+ if self._in_title:
+ self.title_parts.append(text)
+ if self._hidden_depth == 0 and not self._in_title:
+ self.text_parts.append(text)
+
+
+def extract_public_web_snapshot(
+ stdout: bytes,
+ *,
+ profile: PublicWebSnapshotProfile,
+) -> dict[str, Any]:
+ html_text = stdout.decode("utf-8", errors="replace")
+ parser = _PublicWebTextExtractor(profile.hidden_tags)
+ try:
+ parser.feed(html_text)
+ except html.parser.HTMLParseError:
+ pass
+ title = " ".join(parser.title_parts).strip()
+ if profile.preserve_title_entities:
+ match = _TITLE_RE.search(html_text)
+ title = re.sub(r"\s+", " ", match.group(1)).strip() if match else ""
+ visible_text = re.sub(r"\s+", " ", " ".join(parser.text_parts)).strip()
+ return {
+ "title": title,
+ "text_length": len(visible_text),
+ "text_sample": visible_text[: profile.sample_chars],
+ }
+
+
+def _looks_like_binary_content(body: bytes) -> bool:
+ sample = body[:4096]
+ if sample.startswith(b"%PDF-") or b"\x00" in sample:
+ return True
+ decoded = sample.decode("utf-8", errors="replace")
+ return decoded.count("\ufffd") > max(8, len(decoded) // 20)
+
+
+def _looks_like_navigation_network_failure(text: str) -> bool:
+ return (
+ "navigate failed" in text or "navigation failed" in text
+ ) and any(marker in text for marker in _NAVIGATION_NETWORK_FAILURE_MARKERS)
+
+
+def _looks_like_login_wall(
+ title: str,
+ sample: str,
+ text_length: int,
+) -> bool:
+ page_text = f"{title}\n{sample}"
+ if sum(marker in page_text for marker in _LOGIN_FORM_MARKERS) >= 2:
+ return True
+ if text_length >= 800:
+ return False
+ login_hits = {marker for marker in _LOGIN_MARKERS if marker in page_text}
+ if not login_hits:
+ return False
+ title_has_login = any(marker in title for marker in _LOGIN_MARKERS)
+ if title_has_login and len(login_hits) >= 2:
+ return True
+ return len(login_hits) >= 2 and any(
+ marker in page_text for marker in _LOGIN_CONTEXT_MARKERS
+ )
+
+
+def _top_sites_http_error_category(status: int) -> str:
+ if status in {401, 403}:
+ return "blocked-or-forbidden"
+ if status == 404:
+ return "not-found"
+ return "http-error"
+
+
+def _wild_web_http_error_category(status: int) -> str:
+ if status in {401, 403}:
+ return "blocked"
+ if status == 404:
+ return "not-found"
+ return "http-error"
+
+
+@dataclass(frozen=True)
+class PublicWebClassifier:
+ """Classify normalized public-web evidence with a suite policy.
+
+ Transport/status handling and evidence precedence live here once. The
+ two policies retain their established category vocabulary because those
+ values are part of each suite's artifact schema.
+ """
+
+ policy: str
+ snapshot_profile: PublicWebSnapshotProfile
+ ok_categories: frozenset[str]
+ non_comparable_categories: frozenset[str]
+ failure_kind_aliases: Mapping[str, str]
+ use_category_as_failure_kind: bool
+
+ def classification_ok(self, category: str) -> bool:
+ return category in self.ok_categories
+
+ def comparable(self, category: str) -> bool:
+ return category not in self.non_comparable_categories
+
+ def failure_kind(
+ self,
+ category: str,
+ *,
+ error: str | None = None,
+ extraction_failed: bool = False,
+ ) -> str | None:
+ if error == "target binary unavailable":
+ return "target-unavailable"
+ alias = self.failure_kind_aliases.get(category)
+ if alias is not None:
+ return alias
+ if self.use_category_as_failure_kind and not self.classification_ok(
+ category
+ ):
+ return category
+ if extraction_failed:
+ return "extraction-failure"
+ return None
+
+ def snapshot(self, stdout: bytes) -> dict[str, Any]:
+ return extract_public_web_snapshot(
+ stdout,
+ profile=self.snapshot_profile,
+ )
+
+ def classify_output(
+ self,
+ *,
+ stdout: bytes,
+ stderr: bytes,
+ returncode: int | None,
+ timed_out: bool,
+ min_body_bytes: int = 1,
+ snapshot: Mapping[str, Any] | None = None,
+ response_status: int | None = None,
+ main_document_body_capture: str | None = None,
+ ) -> str:
+ if timed_out:
+ return "timeout"
+ response_status, _ = resolve_main_document_status(
+ response_status,
+ stderr,
+ returncode,
+ )
+ if response_status is not None and response_status >= 400:
+ return self._http_error_category(response_status)
+ if (
+ self.policy == "top-sites"
+ and main_document_body_capture == "response-headers-only"
+ ):
+ return "binary-response-headers"
+ if reports_lightpanda_fetch_timeout(stderr):
+ return "timeout"
+ if reports_cli_fetch_timeout(stderr, returncode):
+ return "timeout"
+ if self.policy == "top-sites":
+ return self._classify_top_sites_document(
+ stdout=stdout,
+ stderr=stderr,
+ returncode=returncode,
+ min_body_bytes=min_body_bytes,
+ snapshot=snapshot,
+ )
+ if self.policy == "wild-web":
+ return self._classify_wild_web_document(
+ stdout=stdout,
+ stderr=stderr,
+ returncode=returncode,
+ snapshot=snapshot,
+ )
+ raise ValueError(f"unknown public-web classifier policy: {self.policy}")
+
+ def classify_result(
+ self,
+ result: PublicWebResult,
+ *,
+ min_body_bytes: int = 1,
+ ) -> str:
+ process = result.process
+ return self.classify_output(
+ stdout=process.stdout,
+ stderr=process.stderr,
+ returncode=process.returncode,
+ timed_out=process.timed_out,
+ min_body_bytes=min_body_bytes,
+ snapshot=result.snapshot,
+ response_status=result.response_status,
+ main_document_body_capture=process.main_document_body_capture,
+ )
+
+ def classification_basis(
+ self,
+ result: PublicWebResult,
+ category: str,
+ ) -> str:
+ return self.classification_basis_for_process(
+ result.process,
+ category,
+ response_status=result.response_status,
+ response_status_source=result.response_status_source,
+ )
+
+ def classification_basis_for_process(
+ self,
+ process: ProcessResult,
+ category: str,
+ *,
+ response_status: int | None,
+ response_status_source: str | None,
+ ) -> str:
+ if process.timed_out or category == "timeout":
+ return "timeout"
+ if response_status is not None and response_status >= 400:
+ source = response_status_source or "unknown"
+ return f"{source}-http-status"
+ if category == "binary-response-headers":
+ return "protocol-binary-response-headers"
+ if process.returncode not in {None, 0}:
+ return "process-exit"
+ if category == "success-binary-content":
+ return "binary-signature"
+ visible_categories = (
+ {"success-content", "app-shell-only", "empty-response"}
+ if self.policy == "top-sites"
+ else {"success", "empty"}
+ )
+ if category in visible_categories:
+ return "visible-content"
+ return "document-marker"
+
+ def _http_error_category(self, status: int) -> str:
+ if self.policy == "top-sites":
+ return _top_sites_http_error_category(status)
+ if self.policy == "wild-web":
+ return _wild_web_http_error_category(status)
+ raise ValueError(f"unknown public-web classifier policy: {self.policy}")
+
+ def _classify_top_sites_document(
+ self,
+ *,
+ stdout: bytes,
+ stderr: bytes,
+ returncode: int | None,
+ min_body_bytes: int,
+ snapshot: Mapping[str, Any] | None,
+ ) -> str:
+ combined_text = (stdout + b"\n" + stderr).decode(
+ "utf-8", errors="replace"
+ ).lower()
+ if "operationtimedout" in combined_text and (
+ "navigate failed" in combined_text
+ or "navigation failed" in combined_text
+ ):
+ return "timeout"
+ if _looks_like_navigation_network_failure(combined_text):
+ return "network-error"
+ if returncode != 0:
+ if any(
+ marker in combined_text
+ for marker in _FORBIDDEN_ERROR_MARKERS
+ ):
+ return "blocked-or-forbidden"
+ if any(
+ marker in combined_text for marker in _NOT_FOUND_ERROR_MARKERS
+ ):
+ return "not-found"
+ if any(
+ marker in combined_text
+ for marker in _TOP_SITES_NETWORK_ERROR_MARKERS
+ ):
+ return "network-error"
+ return "process-error"
+
+ body = stdout.strip()
+ if not body:
+ if any(
+ marker in combined_text
+ for marker in _TOP_SITES_NETWORK_ERROR_MARKERS
+ ):
+ return "network-error"
+ return "empty-response"
+ if len(body) >= min_body_bytes and _looks_like_binary_content(body):
+ return "success-binary-content"
+
+ page_snapshot = snapshot or self.snapshot(stdout)
+ text_length = int(page_snapshot.get("text_length") or 0)
+ title = str(page_snapshot.get("title") or "").lower()
+ sample = str(page_snapshot.get("text_sample") or "").lower()
+ page_text = f"{title}\n{sample}"
+ if any(
+ marker in page_text for marker in _TOP_SITES_NETWORK_ERROR_MARKERS
+ ):
+ return "network-error"
+ inferred_error = self._inferred_top_sites_http_error(title, sample)
+ if inferred_error is not None:
+ return inferred_error
+ if (
+ "blocked" in title
+ or "forbidden" in title
+ or _TOP_SITES_BLOCKED_TITLE_403.search(title)
+ or "access restricted" in title
+ ):
+ return "blocked-or-forbidden"
+ if any(marker in page_text for marker in _NOT_FOUND_MARKERS):
+ return "not-found"
+ if any(marker in page_text for marker in _BLOCKED_BODY_MARKERS):
+ return "blocked-or-forbidden"
+ if any(marker in page_text for marker in _CAPTCHA_MARKERS):
+ return "captcha-or-verification"
+ if any(marker in combined_text for marker in _JS_CHALLENGE_MARKERS):
+ return "js-challenge"
+ if _looks_like_login_wall(title, sample, text_length):
+ return "login-wall"
+ if len(body) < min_body_bytes or text_length < min_body_bytes:
+ return "app-shell-only"
+ return "success-content"
+
+ def _classify_wild_web_document(
+ self,
+ *,
+ stdout: bytes,
+ stderr: bytes,
+ returncode: int | None,
+ snapshot: Mapping[str, Any] | None,
+ ) -> str:
+ diagnostic_text = stderr.decode("utf-8", errors="replace").lower()
+ if returncode != 0:
+ if any(
+ marker in diagnostic_text
+ for marker in _WILD_WEB_NETWORK_ERROR_MARKERS
+ ):
+ return "network-error"
+ return "error"
+
+ page_snapshot = snapshot or self.snapshot(stdout)
+ visible_text = " ".join(
+ str(page_snapshot.get(key) or "")
+ for key in ("title", "text_sample")
+ ).lower()
+ text = f"{visible_text}\n{diagnostic_text}"
+ if any(marker in text for marker in _WILD_WEB_NETWORK_ERROR_MARKERS):
+ return "network-error"
+ title = str(page_snapshot.get("title") or "")
+ status_match = _WILD_WEB_HTTP_ERROR_TITLE_STATUS.match(title)
+ if status_match is not None:
+ status = int(status_match.group(1) or status_match.group(2))
+ return _wild_web_http_error_category(status)
+ sample = str(page_snapshot.get("text_sample") or "")
+ if sample.lstrip().startswith(('{"error":{', '{"errors":[')):
+ return "http-error"
+ if "页面不存在" in text or "视频不见了" in text:
+ return "not-found"
+ if "security detection powered by safeline waf" in text:
+ return "blocked"
+ if (
+ "captcha" in text
+ or "verify" in text
+ or "challenge" in text
+ or "安全验证" in text
+ or "完成验证后" in text
+ or "当前环境异常" in text
+ or "select the 2 matching tiles" in text
+ or "powered and protected by privacy" in text
+ ):
+ return "challenge"
+ if "login" in text or "登录" in text:
+ return "login"
+ if (
+ "blocked" in text
+ or "forbidden" in text
+ or "403" in text
+ or "access restricted" in text
+ ):
+ return "blocked"
+ return "success" if stdout.strip() else "empty"
+
+ @staticmethod
+ def _inferred_top_sites_http_error(
+ title: str,
+ sample: str,
+ ) -> str | None:
+ match = _TOP_SITES_HTTP_ERROR_TITLE_STATUS.match(title)
+ if match is not None:
+ status_text = match.group(1) or match.group(2)
+ if status_text is not None:
+ return _top_sites_http_error_category(int(status_text))
+ if sample.lstrip().startswith(('{"error":{', '{"errors":[')):
+ return "http-error"
+ return None
+
+
+TOP_SITES_CLASSIFIER = PublicWebClassifier(
+ policy="top-sites",
+ snapshot_profile=TOP_SITES_SNAPSHOT_PROFILE,
+ ok_categories=frozenset({"success-content", "success-binary-content"}),
+ non_comparable_categories=frozenset({"binary-response-headers"}),
+ failure_kind_aliases={},
+ use_category_as_failure_kind=True,
+)
+WILD_WEB_CLASSIFIER = PublicWebClassifier(
+ policy="wild-web",
+ snapshot_profile=WILD_WEB_SNAPSHOT_PROFILE,
+ ok_categories=frozenset({"success", "login", "challenge"}),
+ non_comparable_categories=frozenset(),
+ failure_kind_aliases={
+ "timeout": "timeout",
+ "blocked": "blocked",
+ "challenge": "challenge",
+ "login": "login",
+ "empty": "empty-response",
+ "error": "process-error",
+ "network-error": "network-error",
+ "not-found": "not-found",
+ "http-error": "http-error",
+ },
+ use_category_as_failure_kind=False,
+)
+
+
+def elapsed_failure_reached_timeout(
+ elapsed_ms: float | None,
+ timeout_seconds: float,
+) -> bool:
+ if elapsed_ms is None or timeout_seconds <= 0:
+ return False
+ grace_seconds = min(
+ _ELAPSED_FAILURE_TIMEOUT_GRACE_SECONDS,
+ timeout_seconds * 0.05,
+ )
+ timeout_floor_ms = max(
+ 0.0,
+ (timeout_seconds - grace_seconds) * 1000.0,
+ )
+ return elapsed_ms >= timeout_floor_ms
+
+
+@dataclass(frozen=True)
+class PublicWebAttempt:
+ """One target's execution of one scheduled public-web case."""
+
+ target: str
+ metadata: Mapping[str, str]
+ target_info: Mapping[str, Any]
+ run: int
+ case_fields: Mapping[str, Any]
+ url: str
+ schedule_index: int
+ target_order_index: int
+ artifact_stem: str
+ started_at: str
+
+ @classmethod
+ def start(
+ cls,
+ *,
+ target: str,
+ metadata: Mapping[str, str],
+ target_info: Mapping[str, Any],
+ run: int,
+ case_fields: Mapping[str, Any],
+ url: str,
+ schedule_index: int,
+ target_order_index: int,
+ artifact_stem: str,
+ ) -> PublicWebAttempt:
+ return cls(
+ target=target,
+ metadata=dict(metadata),
+ target_info=dict(target_info),
+ run=run,
+ case_fields=dict(case_fields),
+ url=url,
+ schedule_index=schedule_index,
+ target_order_index=target_order_index,
+ artifact_stem=artifact_stem,
+ started_at=utc_timestamp(),
+ )
+
+ def binary_path(self) -> Path | None:
+ path = self.target_info.get("path")
+ if not self.target_info.get("available") or not path:
+ return None
+ return Path(str(path))
+
+ def base_row(self, *, finished_at: str | None = None) -> dict[str, Any]:
+ return {
+ "target": self.target,
+ **dict(self.metadata),
+ "run": self.run,
+ **dict(self.case_fields),
+ "url": self.url,
+ "schedule_index": self.schedule_index,
+ "target_order_index": self.target_order_index,
+ "started_at": self.started_at,
+ "finished_at": finished_at or utc_timestamp(),
+ }
+
+
+@dataclass(frozen=True)
+class PublicWebResult:
+ """Normalized process and response evidence for a public-web attempt."""
+
+ attempt: PublicWebAttempt
+ process: ProcessResult
+ response_status: int | None
+ response_status_source: str | None
+ snapshot: dict[str, Any]
+ finished_at: str
+
+ @classmethod
+ def capture(
+ cls,
+ attempt: PublicWebAttempt,
+ process: ProcessResult,
+ *,
+ classifier: PublicWebClassifier,
+ ) -> PublicWebResult:
+ response_status, response_status_source = resolve_main_document_status(
+ process.response_status,
+ process.stderr,
+ process.returncode,
+ )
+ return cls(
+ attempt=attempt,
+ process=process,
+ response_status=response_status,
+ response_status_source=response_status_source,
+ snapshot=classifier.snapshot(process.stdout),
+ finished_at=utc_timestamp(),
+ )
+
+ def base_row(self) -> dict[str, Any]:
+ return self.attempt.base_row(finished_at=self.finished_at)
+
+ def evidence_fields(
+ self,
+ *,
+ policy: PublicWebArtifactPolicy,
+ ) -> dict[str, Any]:
+ process = self.process
+ sample_bytes = policy.output_sample_bytes
+ return {
+ "final_url": process.final_url,
+ "elapsed_ms": process.elapsed_ms,
+ "returncode": process.returncode,
+ "timed_out": process.timed_out,
+ "response_status": self.response_status,
+ "response_status_source": self.response_status_source,
+ "response_mime_type": process.response_mime_type,
+ "main_document_body_capture": process.main_document_body_capture,
+ "stdout_bytes": len(process.stdout),
+ "stderr_bytes": len(process.stderr),
+ "stdout_sha256": sha256(process.stdout).hexdigest(),
+ "stderr_sha256": sha256(process.stderr).hexdigest(),
+ "title": self.snapshot["title"],
+ "text_length": self.snapshot["text_length"],
+ "text_sample": self.snapshot["text_sample"],
+ "stdout_head": process.stdout[:sample_bytes].decode(
+ "utf-8", errors="replace"
+ ),
+ "stdout_tail": process.stdout[-sample_bytes:].decode(
+ "utf-8", errors="replace"
+ ),
+ "stderr_tail": process.stderr[-sample_bytes:].decode(
+ "utf-8", errors="replace"
+ ),
+ "peak_pss_bytes": process.resources.get("peak_pss_bytes"),
+ "peak_rss_bytes": process.resources.get("peak_rss_bytes"),
+ }
+
+
+def unavailable_public_web_row(
+ attempt: PublicWebAttempt,
+ *,
+ category: str,
+ extra_fields: Mapping[str, Any] | None = None,
+) -> dict[str, Any]:
+ return {
+ **attempt.base_row(),
+ "category": category,
+ "ok": False,
+ "failure_kind": "target-unavailable",
+ "error": "target binary unavailable",
+ **dict(extra_fields or {}),
+ }
+
+
+def build_public_web_fetch_command(
+ target: str,
+ binary: Path,
+ url: str,
+ timeout_seconds: float,
+ *,
+ suite_name: str,
+ omit_moli_page_wait: bool = False,
+) -> list[str]:
+ if target_is_cdp(target):
+ raise RuntimeError(f"{target} is a CDP target; use the cdp-session suite")
+ if target == "chrome":
+ raise RuntimeError(f"chrome {suite_name} uses the CDP DCL runner")
+ timeout_ms = str(int(timeout_seconds * 1000))
+ if target in {"moli", "moli-full"}:
+ compatibility_args = (
+ ["--layout", "--resource"]
+ if target_enables_all_resource_fetch(target)
+ else []
+ )
+ page_wait_args = (
+ []
+ if omit_moli_page_wait
+ else [
+ "--wait-until",
+ "domcontentloaded",
+ "--wait-script",
+ POST_DCL_WAIT_SCRIPT,
+ ]
+ )
+ return [
+ str(binary),
+ "fetch",
+ *compatibility_args,
+ "--dump",
+ "html",
+ *page_wait_args,
+ "--timeout",
+ timeout_ms,
+ "--http-timeout",
+ timeout_ms,
+ url,
+ ]
+ if target == "lightpanda":
+ return [
+ str(binary),
+ "fetch",
+ "--dump",
+ "html",
+ "--wait-until",
+ "domcontentloaded",
+ "--wait-script",
+ POST_DCL_WAIT_SCRIPT,
+ "--wait-ms",
+ timeout_ms,
+ "--http-timeout",
+ timeout_ms,
+ "--terminate-ms",
+ timeout_ms,
+ url,
+ ]
+ if target == "obscura":
+ return [
+ str(binary),
+ "fetch",
+ "--dump",
+ "html",
+ "--wait-until",
+ "load",
+ "--wait",
+ "0",
+ "--timeout",
+ str(max(1, int(timeout_seconds))),
+ url,
+ ]
+ raise RuntimeError(f"unknown target: {target}")
+
+
+def public_web_target_metadata(target: str) -> dict[str, str]:
+ metadata = dict(target_metadata(target))
+ if target == "chrome" or target_is_cdp(target):
+ metadata["driver"] = "cdp-dcl"
+ prefix = "moli full" if target == "moli-full-cdp" else metadata["engine"]
+ metadata["label"] = f"{prefix} / cdp-dcl"
+ return metadata
+
+
+def run_public_web_target(
+ *,
+ target: str,
+ binary: Path,
+ url: str,
+ timeout_seconds: float,
+ proc_env: dict[str, str],
+ command_builder: Callable[[str, Path, str, float], list[str]],
+ chrome_runner: Callable[..., ProcessResult],
+ served_cdp_runner: Callable[..., ProcessResult],
+ process_runner: Callable[..., ProcessResult],
+) -> ProcessResult:
+ if target == "chrome":
+ return chrome_runner(
+ binary,
+ url,
+ cwd=REPO_ROOT,
+ timeout_seconds=timeout_seconds,
+ env=proc_env,
+ )
+ if target_is_cdp(target):
+ return served_cdp_runner(
+ target,
+ binary,
+ url,
+ cwd=REPO_ROOT,
+ timeout_seconds=timeout_seconds,
+ env=proc_env,
+ )
+ return process_runner(
+ command_builder(target, binary, url, timeout_seconds),
+ cwd=REPO_ROOT,
+ timeout_seconds=timeout_seconds + 2,
+ env=proc_env,
+ )
+
+
+CaseT = TypeVar("CaseT")
+ExecutionT = TypeVar("ExecutionT")
+
+
+@dataclass(frozen=True)
+class PublicWebScheduledCase(Generic[CaseT]):
+ schedule_index: int
+ case_index: int
+ run: int
+ case: CaseT
+
+ @property
+ def target_rotation_index(self) -> int:
+ return self.case_index + self.run - 1
+
+
+def schedule_public_web_cases(
+ cases: Sequence[CaseT],
+ *,
+ runs: int,
+) -> list[PublicWebScheduledCase[CaseT]]:
+ return [
+ PublicWebScheduledCase(
+ schedule_index=schedule_index,
+ case_index=case_index,
+ run=run,
+ case=case,
+ )
+ for schedule_index, (run, case_index, case) in enumerate(
+ (run, case_index, case)
+ for run in range(1, runs + 1)
+ for case_index, case in enumerate(cases)
+ )
+ ]
+
+
+class PublicWebScheduler(Generic[CaseT, ExecutionT]):
+ """Run target attempts sequentially per case and cases concurrently."""
+
+ def __init__(
+ self,
+ targets: tuple[str, ...],
+ *,
+ parallelism: int = 1,
+ target_parallelism: Mapping[str, int] | None = None,
+ ) -> None:
+ if parallelism <= 0:
+ raise ValueError("public-web parallelism must be positive")
+ limits = dict(target_parallelism or {})
+ if any(limit <= 0 for limit in limits.values()):
+ raise ValueError("public-web target parallelism must be positive")
+ self._targets = targets
+ self._parallelism = parallelism
+ self._target_slots = {
+ target: BoundedSemaphore(limit) for target, limit in limits.items()
+ }
+
+ def run(
+ self,
+ scheduled_cases: Sequence[PublicWebScheduledCase[CaseT]],
+ execute: Callable[[PublicWebScheduledCase[CaseT], str, int], ExecutionT],
+ ) -> list[ExecutionT]:
+ def execute_group(
+ scheduled: PublicWebScheduledCase[CaseT],
+ ) -> list[ExecutionT]:
+ results: list[ExecutionT] = []
+ target_order = rotated_target_order(
+ self._targets,
+ scheduled.target_rotation_index,
+ )
+ for target_order_index, target in enumerate(target_order, start=1):
+ slots = self._target_slots.get(target)
+ if slots is None:
+ results.append(execute(scheduled, target, target_order_index))
+ continue
+ with slots:
+ results.append(execute(scheduled, target, target_order_index))
+ return results
+
+ if self._parallelism == 1:
+ return [
+ result
+ for scheduled in scheduled_cases
+ for result in execute_group(scheduled)
+ ]
+
+ results_by_index: dict[int, list[ExecutionT]] = {}
+ with ThreadPoolExecutor(max_workers=self._parallelism) as executor:
+ future_to_case = {
+ executor.submit(execute_group, scheduled): scheduled
+ for scheduled in scheduled_cases
+ }
+ for future in as_completed(future_to_case):
+ scheduled = future_to_case[future]
+ results_by_index[scheduled.schedule_index] = future.result()
+ return [
+ result
+ for scheduled in scheduled_cases
+ for result in results_by_index[scheduled.schedule_index]
+ ]
+
+
+def safe_artifact_filename_token(value: str, *, max_length: int = 80) -> str:
+ token = _SAFE_ARTIFACT_TOKEN.sub("_", value)
+ while ".." in token:
+ token = token.replace("..", "_")
+ token = token.strip("._-") or "site"
+ if token == value and len(token) <= max_length:
+ return token
+ digest = sha256(value.encode("utf-8", errors="surrogatepass")).hexdigest()[:12]
+ token = token[:max_length].rstrip("._-") or "site"
+ return f"{token}-{digest}"
+
+
+def write_public_web_failure_artifacts(
+ *,
+ suite_dir: Path,
+ attempt: PublicWebAttempt,
+ row: Mapping[str, Any],
+ stdout: bytes,
+ stderr: bytes,
+ policy: PublicWebArtifactPolicy,
+) -> str:
+ failures_dir = suite_dir / "failures"
+ json_path = failures_dir / f"{attempt.artifact_stem}.json"
+ stdout_path = failures_dir / f"{attempt.artifact_stem}.stdout.html"
+ stderr_path = failures_dir / f"{attempt.artifact_stem}.stderr.txt"
+ write_json(json_path, dict(row))
+ write_text(
+ stderr_path,
+ stderr[-policy.failure_stderr_limit_bytes :].decode(
+ "utf-8", errors="replace"
+ ),
+ )
+ if policy.write_empty_failure_stdout or stdout.strip():
+ write_text(
+ stdout_path,
+ stdout[-policy.failure_stdout_limit_bytes :].decode(
+ "utf-8", errors="replace"
+ ),
+ )
+ return str(json_path.relative_to(suite_dir))
+
+
+def write_public_web_replay_artifact(
+ *,
+ suite_dir: Path,
+ attempt: PublicWebAttempt,
+ stdout: bytes,
+) -> str:
+ replay_path = suite_dir / "replay" / f"{attempt.artifact_stem}.html"
+ write_text(replay_path, stdout.decode("utf-8", errors="replace"))
+ return str(replay_path.relative_to(suite_dir))
+
+
+def successful_public_web_attempt_cohort(
+ rows: list[dict[str, Any]],
+ targets: tuple[str, ...],
+ *,
+ attempt_key_fields: tuple[str, ...],
+ unique_key_fields: tuple[str, ...],
+ unique_count_name: str,
+ metric_fields: tuple[str, ...],
+ require_comparable: bool = False,
+) -> dict[str, Any]:
+ by_attempt: dict[tuple[Any, ...], dict[str, dict[str, Any]]] = {}
+ for row in rows:
+ key = tuple(row[field] for field in attempt_key_fields)
+ by_attempt.setdefault(key, {})[str(row["target"])] = row
+ selected_keys = [
+ key
+ for key, target_rows in by_attempt.items()
+ if all(
+ target in target_rows
+ and (
+ not require_comparable
+ or target_rows[target].get("comparable", True)
+ )
+ and target_rows[target].get("ok")
+ for target in targets
+ )
+ ]
+ unique_indexes = tuple(
+ attempt_key_fields.index(field) for field in unique_key_fields
+ )
+ cohort: dict[str, Any] = {
+ "attempts": len(selected_keys),
+ unique_count_name: len(
+ {
+ tuple(key[index] for index in unique_indexes)
+ for key in selected_keys
+ }
+ ),
+ "targets": {},
+ }
+ for target in targets:
+ target_rows = [by_attempt[key][target] for key in selected_keys]
+ cohort["targets"][target] = {
+ field: summarize(
+ row[field]
+ for row in target_rows
+ if row.get(field) is not None
+ )
+ for field in metric_fields
+ }
+ return cohort
+
+
+def count_public_web_row_values(
+ rows: Sequence[Mapping[str, Any]],
+) -> dict[str, dict[str, int]]:
+ counts: dict[str, dict[str, int]] = {
+ "categories": {},
+ "failure_kinds": {},
+ "classification_bases": {},
+ "response_status_sources": {},
+ }
+ field_names = {
+ "category": "categories",
+ "failure_kind": "failure_kinds",
+ "classification_basis": "classification_bases",
+ "response_status_source": "response_status_sources",
+ }
+ for row in rows:
+ for field, count_name in field_names.items():
+ value = row.get(field)
+ if field == "classification_basis" and not value:
+ value = "unknown"
+ if not isinstance(value, str) or not value:
+ continue
+ field_counts = counts[count_name]
+ field_counts[value] = field_counts.get(value, 0) + 1
+ return counts
diff --git a/moli-benchmark/moli_benchmark/sequential_navigate.py b/moli-benchmark/moli_benchmark/sequential_navigate.py
index 2288135fa..abe7fbee0 100644
--- a/moli-benchmark/moli_benchmark/sequential_navigate.py
+++ b/moli-benchmark/moli_benchmark/sequential_navigate.py
@@ -25,12 +25,13 @@ from .raw_cdp import (
)
from .sampling import snapshot_resources
from .target_serve import start_target_serve, stop_target_serve
+from .top_sites import TOP_SITES_LIST_PATH, parse_top_sites_list
DEFAULT_SEED = 20260809
DEFAULT_COUNT = 50
DEFAULT_PINNED_DOMAINS = ("csdn.net", "zol.com.cn")
-DEFAULT_SEED_FILE = REPO_ROOT / "docs" / "chinese-community-top100-websites.md"
+DEFAULT_SEED_FILE = TOP_SITES_LIST_PATH
TRACE_METHODS = {
"Page.frameStartedNavigating",
"Page.frameStartedLoading",
@@ -1310,7 +1311,7 @@ def _selected_urls(args: argparse.Namespace) -> list[str]:
if args.url:
base = [normalize_public_url(url) for url in args.url]
else:
- domains = parse_top_100_domains(args.seed_file.read_text(encoding="utf-8"))
+ domains = [target for _, target in parse_top_sites_list(args.seed_file)]
base = select_seed_urls(
domains,
seed=args.seed,
diff --git a/moli-benchmark/moli_benchmark/targets.py b/moli-benchmark/moli_benchmark/targets.py
index e381a490e..c2c626dbd 100644
--- a/moli-benchmark/moli_benchmark/targets.py
+++ b/moli-benchmark/moli_benchmark/targets.py
@@ -5,7 +5,7 @@ from pathlib import Path
from typing import Any
from .config import chrome_binary, moli_binary, lightpanda_binary, obscura_binary
-from .versions import sha256_file
+from .versions import MOLI_VERSION_ARGS, sha256_file
def _version(command: list[str]) -> str | None:
@@ -39,7 +39,7 @@ def collect_target_binaries(
) -> dict[str, Any]:
moli_path = moli_binary(moli_override)
return {
- "moli": _binary_info(moli_path, ("version",)),
+ "moli": _binary_info(moli_path, MOLI_VERSION_ARGS),
"lightpanda": _binary_info(lightpanda_binary(lightpanda_override), ("version",)),
"chrome": _binary_info(chrome_binary(chrome_override), ("--version",)),
"obscura": _binary_info(obscura_binary(obscura_override), ("--help",)),
diff --git a/moli-benchmark/moli_benchmark/top_sites.py b/moli-benchmark/moli_benchmark/top_sites.py
index d21606c10..ba5a8ea10 100644
--- a/moli-benchmark/moli_benchmark/top_sites.py
+++ b/moli-benchmark/moli_benchmark/top_sites.py
@@ -1,31 +1,47 @@
from __future__ import annotations
+import csv
import os
import re
-import html.parser
-from hashlib import sha256
-from concurrent.futures import ThreadPoolExecutor, as_completed
+from itertools import combinations
from pathlib import Path
from typing import Any
+from urllib.parse import urlsplit
-from .artifacts import write_csv, write_json, write_text
+from .artifacts import write_csv, write_json
from .chrome_dcl import run_chrome_dcl_dump, run_served_cdp_dcl_dump
-from .config import REPO_ROOT, clear_proxy_env
+from .config import PROJECT_ROOT, REPO_ROOT, clear_proxy_env
from .process import ProcessResult, run_process
-from .stats import summarize
-from .synthetic_compare import (
- WEBFETCH_TARGETS,
- target_enables_all_resource_fetch,
- target_is_cdp,
- target_metadata,
+from .public_web import (
+ PUBLIC_WEB_SNAPSHOT_CONTRACT,
+ TOP_SITES_ARTIFACT_POLICY,
+ TOP_SITES_CLASSIFIER,
+ PublicWebAttempt,
+ PublicWebResult,
+ PublicWebScheduledCase,
+ PublicWebScheduler,
+ build_public_web_fetch_command,
+ count_public_web_row_values,
+ elapsed_failure_reached_timeout as _elapsed_failure_reached_timeout,
+ public_web_target_metadata as _top_sites_target_metadata,
+ rotated_target_order as _rotated_target_order,
+ run_public_web_target,
+ safe_artifact_filename_token as _artifact_filename_token,
+ schedule_public_web_cases,
+ successful_public_web_attempt_cohort,
+ unavailable_public_web_row,
+ write_public_web_failure_artifacts,
)
+from .stats import summarize
+from .synthetic_compare import WEBFETCH_TARGETS
-TOP_SITES_LIST_PATH = REPO_ROOT / "docs" / "chinese-community-top100-websites.md"
-GLOBAL_TOP_SITES_LIST_PATH = REPO_ROOT / "docs" / "global-top-websites-seed-list.md"
-WEBFETCH_LONGTAIL_LIST_PATH = REPO_ROOT / "docs" / "webfetch-longtail-seed-list.md"
-RENDER_QUALITY_LIST_PATH = REPO_ROOT / "docs" / "render-quality-seed-list.md"
-LEGACY_ENCODING_LIST_PATH = REPO_ROOT / "docs" / "legacy-encoding-websites-seed-list.md"
+TOP_SITES_FIXTURE_ROOT = PROJECT_ROOT / "fixtures" / "top-sites"
+TOP_SITES_LIST_PATH = TOP_SITES_FIXTURE_ROOT / "chinese-community-top100-websites.csv"
+GLOBAL_TOP_SITES_LIST_PATH = TOP_SITES_FIXTURE_ROOT / "global-top-websites-seed-list.csv"
+WEBFETCH_LONGTAIL_LIST_PATH = TOP_SITES_FIXTURE_ROOT / "webfetch-longtail-seed-list.csv"
+RENDER_QUALITY_LIST_PATH = TOP_SITES_FIXTURE_ROOT / "render-quality-seed-list.csv"
+LEGACY_ENCODING_LIST_PATH = TOP_SITES_FIXTURE_ROOT / "legacy-encoding-websites-seed-list.csv"
TOP_SITES_SOURCES: dict[str, dict[str, Any]] = {
"chinese-community": {
@@ -75,232 +91,11 @@ def _default_top_sites_parallelism() -> int:
DEFAULT_TOP_SITES_PARALLELISM = _default_top_sites_parallelism()
-DEFAULT_TOP_SITES_TEXT_SAMPLE_CHARS = 500
_TOP_LIST_HEADING = re.compile(r"^##\s+Top\s+\d+\b", re.IGNORECASE)
_NEXT_SECTION_HEADING = re.compile(r"^##\s+")
_LIST_ENTRY = re.compile(r"^\s*(\d+)\.\s+`([^`]+)`")
-_BLOCKED_TITLE_403 = re.compile(r"^\s*(?:http\s*)?403\b")
-_SAFE_ARTIFACT_TOKEN = re.compile(r"[^A-Za-z0-9._-]+")
-_TITLE_RE = re.compile(r"<title[^>]*>(.*?)</title>", re.IGNORECASE | re.DOTALL)
-_NETWORK_ERROR_MARKERS = (
- "privacy error",
- "your connection is not private",
- "net::err_cert",
- "this site can't be reached",
- "this site can\u2019t be reached",
- "err_name_not_resolved",
- "err_connection",
- "err_timed_out",
- "dns_probe_finished",
- "could not resolve",
- "could not resolve host",
- "could not resolve hostname",
- "could not connect to server",
- "name resolution",
- "connection refused",
- "connection reset",
- "connection timed out",
- "no route to host",
- "network is unreachable",
- "ssl handshake",
- "ssl error",
- "ssl connect error",
- "tls handshake",
- "timed out connecting",
- "timeout was reached",
- "request timeout",
- "i/o error",
- "broken pipe",
- "operation timed out",
- "failed to connect",
- "dns lookup",
- "curl request failed",
- "recv failure",
-)
-_CLI_DEADLINE_COMMAND_MARKERS = (
- "fetch_document_allow_http_error_with_wait_until",
- "fetch document allow-http-error",
- "fetch allow-http-error wait_until",
- "fetch wait_until",
-)
-_CLI_DEADLINE_TIMEOUT_MARKERS = (
- "timed out after",
-)
-_RAW_DOCUMENT_BODY_PROGRESS_RE = re.compile(
- r"with\s+(\d+)\s+out\s+of\s+(\d+)\s+bytes\s+received",
- re.IGNORECASE,
-)
-_ELAPSED_FAILURE_TIMEOUT_GRACE_SECONDS = 1.0
-_CAPTCHA_MARKERS = (
- "captcha",
- "安全验证",
- "验证你是真人",
- "人机验证",
- "human verification",
- "verification",
- "verify you are human",
- "verify that you're not a robot",
- "verify that you are not a robot",
- "are you a robot",
- "not a robot",
- "bot or not",
- "checking your browser",
- "perform security check",
- "security verification",
- "performing security verification",
- "请完成验证",
-)
-_LOGIN_MARKERS = (
- "sign in",
- "log in",
- "login",
- "登录",
- "注册",
- "账号",
-)
-_LOGIN_FORM_MARKERS = (
- "login to your account",
- "email/username",
- "your password is a required field",
- "forgot password",
- "create a free account",
-)
-_LOGIN_CONTEXT_MARKERS = (
- "password",
- "验证码",
- "短信验证码",
- "语音验证码",
- "手机号",
- "手机验证",
- "邮箱",
- "email",
-)
-_JS_CHALLENGE_MARKERS = (
- "__cf_chl_",
- "c2wf946j0/probe",
- "cf-challenge",
- "__tencent_chaos_vm",
- "__eo_jschallenge_vm",
- "teojschallengesdk.js",
- "eojschallengesdk",
- "window.solvechallenge(",
- "jsl_clearance",
- "acw_sc__v2",
- "window._phantom",
- "_$jsvmprt",
- "awswaf",
- "aliyun_waf",
- "aliyun waf",
- "_waf_",
- "just a moment...",
- "vercel security checkpoint",
- "we're verifying your browser",
- "we\u2019re verifying your browser",
- "enable javascript and cookies to continue",
- "javascript is needed to access this site",
-)
-_BLOCKED_BODY_MARKERS = (
- "unable to give you access to our site",
- "access denied",
- "access to this page has been denied",
- "sorry, you have been blocked",
- "security issue was automatically identified",
- "waf拦截",
- "被waf拦截",
-)
-_NOT_FOUND_MARKERS = (
- "404 not found",
- "file not found",
- "page not found",
- "this page cannot be found",
- "page you requested is missing",
- "page you requested could not be found",
-)
-_FORBIDDEN_ERROR_MARKERS = (
- "403 forbidden",
- "returned 403",
- "http 403",
-)
-_NOT_FOUND_ERROR_MARKERS = (
- "returned 404",
- "http 404",
-)
-_SHELL_MARKERS = (
- 'id="root"',
- 'id="app"',
- 'id="__next"',
- 'data-reactroot',
- "webpackJsonp",
- "__NUXT__",
- "__NEXT_DATA__",
-)
-
-
-class _VisibleTextExtractor(html.parser.HTMLParser):
- def __init__(self) -> None:
- super().__init__(convert_charrefs=True)
- self._skip_depth = 0
- self.parts: list[str] = []
-
- def handle_starttag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None:
- if tag.lower() in {"script", "style", "noscript", "svg", "template"}:
- self._skip_depth += 1
-
- def handle_endtag(self, tag: str) -> None:
- if tag.lower() in {"script", "style", "noscript", "svg", "template"} and self._skip_depth:
- self._skip_depth -= 1
-
- def handle_data(self, data: str) -> None:
- if self._skip_depth:
- return
- text = " ".join(data.split())
- if text:
- self.parts.append(text)
-
-
-def _extract_title_and_text(stdout: bytes) -> dict[str, Any]:
- html_text = stdout.decode("utf-8", errors="replace")
- title = ""
- match = _TITLE_RE.search(html_text)
- if match:
- title = re.sub(r"\s+", " ", match.group(1)).strip()
- parser = _VisibleTextExtractor()
- try:
- parser.feed(html_text)
- except html.parser.HTMLParseError:
- pass
- visible_text = " ".join(parser.parts)
- visible_text = re.sub(r"\s+", " ", visible_text).strip()
- return {
- "title": title,
- "text_length": len(visible_text),
- "text_sample": visible_text[:DEFAULT_TOP_SITES_TEXT_SAMPLE_CHARS],
- }
-
-
-def _looks_like_binary_content(body: bytes) -> bool:
- sample = body[:4096]
- if sample.startswith(b"%PDF-"):
- return True
- if b"\x00" in sample:
- return True
- decoded = sample.decode("utf-8", errors="replace")
- replacement_count = decoded.count("\ufffd")
- return replacement_count > max(8, len(decoded) // 20)
-
-
-def _looks_like_raw_binary_main_resource_timeout(text: str, min_body_bytes: int) -> bool:
- if "failed to read raw document body" not in text:
- return False
- if "timeout was reached" not in text and "operation timed out" not in text:
- return False
- match = _RAW_DOCUMENT_BODY_PROGRESS_RE.search(text)
- if match is None:
- return False
- received = int(match.group(1))
- total = int(match.group(2))
- return received >= min_body_bytes and total >= received
+_extract_title_and_text = TOP_SITES_CLASSIFIER.snapshot
def _parse_top_sites_sections(path: Path) -> list[tuple[str, list[tuple[int, str]]]]:
@@ -343,11 +138,47 @@ def _parse_top_sites_sections(path: Path) -> list[tuple[str, list[tuple[int, str
return sections
+def _parse_top_sites_csv(path: Path) -> list[tuple[int, str]]:
+ if not path.exists():
+ raise RuntimeError(f"top sites list not found: {path}")
+ try:
+ with path.open(encoding="utf-8-sig", newline="") as handle:
+ reader = csv.DictReader(handle)
+ if reader.fieldnames != ["rank", "target"]:
+ raise RuntimeError(f"expected CSV columns `rank,target` in {path}")
+ entries: list[tuple[int, str]] = []
+ seen_targets: set[str] = set()
+ for row_number, row in enumerate(reader, start=2):
+ rank_raw = (row.get("rank") or "").strip()
+ target = (row.get("target") or "").strip()
+ try:
+ rank = int(rank_raw)
+ except ValueError as error:
+ raise RuntimeError(f"invalid rank on CSV row {row_number} in {path}") from error
+ if rank <= 0 or not target:
+ raise RuntimeError(f"invalid entry on CSV row {row_number} in {path}")
+ if target not in seen_targets:
+ seen_targets.add(target)
+ entries.append((rank, target))
+ except csv.Error as error:
+ raise RuntimeError(f"invalid CSV seed list {path}: {error}") from error
+ if not entries:
+ raise RuntimeError(f"no entries parsed from {path}")
+ return entries
+
+
def parse_top_sites_list(path: Path) -> list[tuple[int, str]]:
+ if path.suffix.lower() == ".csv":
+ return _parse_top_sites_csv(path)
return _parse_top_sites_sections(path)[0][1]
def _parse_top_sites_list_by_count(path: Path, count: int) -> list[tuple[int, str]]:
+ if path.suffix.lower() == ".csv":
+ entries = _parse_top_sites_csv(path)
+ if len(entries) != count:
+ raise RuntimeError(f"expected {count} entries in {path}, found {len(entries)}")
+ return entries
heading = re.compile(rf"^##\s+Top\s+{count}\b", re.IGNORECASE)
for section_heading, entries in _parse_top_sites_sections(path):
if heading.match(section_heading):
@@ -360,9 +191,9 @@ def resolve_top_sites_source(source: str, list_path: Path | None) -> tuple[str,
return ("custom", list_path)
if source not in TOP_SITES_SOURCES:
if source == "mixed":
- return (source, REPO_ROOT / "docs" / "mixed-top-websites")
+ return (source, TOP_SITES_FIXTURE_ROOT / "mixed-top-websites")
if source == "webfetch-mix":
- return (source, REPO_ROOT / "docs" / "webfetch-mix-websites")
+ return (source, TOP_SITES_FIXTURE_ROOT / "webfetch-mix-websites")
raise RuntimeError(
f"unknown top-sites source `{source}`; expected one of "
f"{sorted(TOP_SITES_SOURCES) + list(COMPOSITE_TOP_SITES_SOURCES)} or a --list-path override"
@@ -450,73 +281,24 @@ def load_top_sites_entries(source: str, list_path: Path | None) -> tuple[list[tu
return entries, [f"{source}:{info['path'].name}"]
-def _top_command_for_target(target: str, binary: Path, url: str, timeout_seconds: float) -> list[str]:
- metadata = target_metadata(target)
- if target_is_cdp(target):
- raise RuntimeError(f"{target} is a CDP target; use the cdp-session suite")
- if target == "chrome":
- raise RuntimeError("chrome top-sites uses the CDP DCL runner")
- timeout_ms = str(int(timeout_seconds * 1000))
- if target in {"moli", "moli-full"}:
- compatibility_args = (
- ["--layout", "--resource"]
- if target_enables_all_resource_fetch(target)
- else []
- )
- return [
- str(binary),
- "fetch",
- *compatibility_args,
- "--dump",
- "html",
- "--wait-until",
- "domcontentloaded",
- "--timeout",
- timeout_ms,
- "--http-timeout",
- timeout_ms,
- url,
- ]
- if target == "lightpanda":
- return [
- str(binary),
- "fetch",
- "--dump",
- "html",
- "--wait-until",
- "domcontentloaded",
- "--wait-ms",
- timeout_ms,
- "--http-timeout",
- timeout_ms,
- "--terminate-ms",
- timeout_ms,
- url,
- ]
- if target == "obscura":
- return [
- str(binary),
- "fetch",
- "--dump",
- "html",
- "--wait-until",
- "load",
- "--wait",
- "0",
- "--timeout",
- str(max(1, int(timeout_seconds))),
- url,
- ]
- raise RuntimeError(f"unknown target: {target}")
+def _is_explicit_pdf_url(url: str) -> bool:
+ return urlsplit(url).path.lower().endswith(".pdf")
-def _top_sites_target_metadata(target: str) -> dict[str, str]:
- metadata = dict(target_metadata(target))
- if target == "chrome" or target_is_cdp(target):
- metadata["driver"] = "cdp-dcl"
- prefix = "moli full" if target == "moli-full-cdp" else metadata["engine"]
- metadata["label"] = f"{prefix} / cdp-dcl"
- return metadata
+def _top_command_for_target(
+ target: str,
+ binary: Path,
+ url: str,
+ timeout_seconds: float,
+) -> list[str]:
+ return build_public_web_fetch_command(
+ target,
+ binary,
+ url,
+ timeout_seconds,
+ suite_name="top-sites",
+ omit_moli_page_wait=_is_explicit_pdf_url(url),
+ )
def _run_top_sites_target(
@@ -527,29 +309,16 @@ def _run_top_sites_target(
timeout_seconds: float,
proc_env: dict[str, str],
) -> ProcessResult:
- if target == "chrome":
- return run_chrome_dcl_dump(
- binary,
- url,
- cwd=REPO_ROOT,
- timeout_seconds=timeout_seconds,
- env=proc_env,
- )
- if target_is_cdp(target):
- return run_served_cdp_dcl_dump(
- target,
- binary,
- url,
- cwd=REPO_ROOT,
- timeout_seconds=timeout_seconds,
- env=proc_env,
- )
- command = _top_command_for_target(target, binary, url, timeout_seconds)
- return run_process(
- command,
- cwd=REPO_ROOT,
- timeout_seconds=timeout_seconds + 2,
- env=proc_env,
+ return run_public_web_target(
+ target=target,
+ binary=binary,
+ url=url,
+ timeout_seconds=timeout_seconds,
+ proc_env=proc_env,
+ command_builder=_top_command_for_target,
+ chrome_runner=run_chrome_dcl_dump,
+ served_cdp_runner=run_served_cdp_dcl_dump,
+ process_runner=run_process,
)
@@ -560,118 +329,32 @@ def _classify(
timed_out: bool,
min_body_bytes: int,
snapshot: dict[str, Any] | None = None,
+ response_status: int | None = None,
+ main_document_body_capture: str | None = None,
) -> str:
- if timed_out:
- return "timeout"
- combined_text = (stdout + b"\n" + stderr).decode("utf-8", errors="replace").lower()
- if "operationtimedout" in combined_text and (
- "navigate failed" in combined_text or "navigation failed" in combined_text
- ):
- return "timeout"
- if (
- returncode != 0
- and any(marker in combined_text for marker in _CLI_DEADLINE_COMMAND_MARKERS)
- and any(marker in combined_text for marker in _CLI_DEADLINE_TIMEOUT_MARKERS)
- ):
- return "timeout"
- if returncode != 0 and _looks_like_raw_binary_main_resource_timeout(
- combined_text,
- min_body_bytes,
- ):
- return "success-binary-main-resource"
- if returncode != 0:
- if any(marker in combined_text for marker in _FORBIDDEN_ERROR_MARKERS):
- return "blocked-or-forbidden"
- if any(marker in combined_text for marker in _NOT_FOUND_ERROR_MARKERS):
- return "not-found"
- if any(marker in combined_text for marker in _NETWORK_ERROR_MARKERS):
- return "network-error"
- return "process-error"
- body = stdout.strip()
- if not body:
- if any(marker in combined_text for marker in _NETWORK_ERROR_MARKERS):
- return "network-error"
- return "empty-response"
- if len(body) >= min_body_bytes and _looks_like_binary_content(body):
- return "success-binary-content"
- if snapshot is None:
- snapshot = _extract_title_and_text(stdout)
- text_length = int(snapshot.get("text_length") or 0)
- title = str(snapshot.get("title") or "").lower()
- sample = str(snapshot.get("text_sample") or "").lower()
- page_text = f"{title}\n{sample}"
- if any(marker in page_text for marker in _NETWORK_ERROR_MARKERS):
- return "network-error"
- if (
- "blocked" in title
- or "forbidden" in title
- or _BLOCKED_TITLE_403.search(title)
- or "access restricted" in title
- ):
- return "blocked-or-forbidden"
- if any(marker in page_text for marker in _NOT_FOUND_MARKERS):
- return "not-found"
- if any(marker in page_text for marker in _BLOCKED_BODY_MARKERS):
- return "blocked-or-forbidden"
- if any(marker in page_text for marker in _CAPTCHA_MARKERS):
- return "captcha-or-verification"
- if any(marker in combined_text for marker in _JS_CHALLENGE_MARKERS):
- return "js-challenge"
- if _looks_like_login_wall(title, sample, text_length):
- return "login-wall"
- if len(body) < min_body_bytes:
- return "app-shell-only"
- if text_length < min_body_bytes:
- return "app-shell-only"
- return "success-content"
-
-
-def _elapsed_failure_reached_timeout(elapsed_ms: float | None, timeout_seconds: float) -> bool:
- if elapsed_ms is None or timeout_seconds <= 0:
- return False
- grace_seconds = min(_ELAPSED_FAILURE_TIMEOUT_GRACE_SECONDS, timeout_seconds * 0.05)
- timeout_floor_ms = max(0.0, (timeout_seconds - grace_seconds) * 1000.0)
- return elapsed_ms >= timeout_floor_ms
-
-
-def _looks_like_login_wall(title: str, sample: str, text_length: int) -> bool:
- page_text = f"{title}\n{sample}"
- if sum(1 for marker in _LOGIN_FORM_MARKERS if marker in page_text) >= 2:
- return True
- if text_length >= 800:
- return False
- login_hits = {marker for marker in _LOGIN_MARKERS if marker in page_text}
- if not login_hits:
- return False
- title_has_login = any(marker in title for marker in _LOGIN_MARKERS)
- if title_has_login and len(login_hits) >= 2:
- return True
- if len(login_hits) >= 2 and any(marker in page_text for marker in _LOGIN_CONTEXT_MARKERS):
- return True
- return False
+ return TOP_SITES_CLASSIFIER.classify_output(
+ stdout=stdout,
+ stderr=stderr,
+ returncode=returncode,
+ timed_out=timed_out,
+ min_body_bytes=min_body_bytes,
+ snapshot=snapshot,
+ response_status=response_status,
+ main_document_body_capture=main_document_body_capture,
+ )
def _ok_categories() -> set[str]:
- return {"success-content", "success-binary-content", "success-binary-main-resource"}
+ return set(TOP_SITES_CLASSIFIER.ok_categories)
-def _failure_kind(category: str, error: str | None) -> str | None:
- if error == "target binary unavailable":
- return "target-unavailable"
- if category not in _ok_categories():
- return category
- return None
+_SITE_UNREACHABLE_FAILURE_KINDS = {"network-error", "timeout"}
-_SITE_UNREACHABLE_FAILURE_KINDS = {
- "network-error",
- "timeout",
- "empty-response",
- "process-error",
-}
-
-
-def _site_unreachable_exclusions(rows: list[dict[str, Any]]) -> dict[str, str]:
+def _site_unreachable_exclusions(
+ rows: list[dict[str, Any]],
+ targets: tuple[str, ...],
+) -> dict[str, str]:
rows_by_domain: dict[str, list[dict[str, Any]]] = {}
for row in rows:
rows_by_domain.setdefault(str(row["domain"]), []).append(row)
@@ -680,50 +363,171 @@ def _site_unreachable_exclusions(rows: list[dict[str, Any]]) -> dict[str, str]:
for domain, domain_rows in rows_by_domain.items():
if any(row.get("ok") for row in domain_rows):
continue
+ if {str(row.get("target")) for row in domain_rows} != set(targets):
+ continue
failure_kinds = {
- str(row.get("failure_kind") or _failure_kind(str(row.get("category")), None) or "")
+ str(
+ row.get("failure_kind")
+ or TOP_SITES_CLASSIFIER.failure_kind(
+ str(row.get("category"))
+ )
+ or ""
+ )
for row in domain_rows
}
failure_kinds.discard("")
if (
failure_kinds
and failure_kinds <= _SITE_UNREACHABLE_FAILURE_KINDS
- and (failure_kinds - {"process-error"})
):
excluded[domain] = "site-unreachable"
return excluded
-def _artifact_filename_token(value: str, *, max_length: int = 80) -> str:
- token = _SAFE_ARTIFACT_TOKEN.sub("_", value)
- while ".." in token:
- token = token.replace("..", "_")
- token = token.strip("._-") or "site"
- if token == value and len(token) <= max_length:
- return token
- digest = sha256(value.encode("utf-8", errors="surrogatepass")).hexdigest()[:12]
- token = token[:max_length].rstrip("._-") or "site"
- return f"{token}-{digest}"
+def _build_site_outcomes(rows: list[dict[str, Any]]) -> list[dict[str, Any]]:
+ grouped: dict[tuple[str, int, str], list[dict[str, Any]]] = {}
+ for row in rows:
+ key = (str(row["target"]), int(row["rank"]), str(row["domain"]))
+ grouped.setdefault(key, []).append(row)
+
+ outcomes: list[dict[str, Any]] = []
+ for (target, rank, domain), attempts in grouped.items():
+ comparable_attempts = [
+ row for row in attempts if row.get("comparable", True)
+ ]
+ passes = sum(1 for row in comparable_attempts if row.get("ok"))
+ attempt_count = len(comparable_attempts)
+ if attempt_count == 0:
+ outcome = "not-comparable"
+ elif passes == attempt_count:
+ outcome = "all-pass"
+ elif passes == 0:
+ outcome = "all-fail"
+ else:
+ outcome = "flaky"
+ categories: dict[str, int] = {}
+ for row in attempts:
+ category = str(row.get("category") or "unknown")
+ categories[category] = categories.get(category, 0) + 1
+ response_statuses: set[int] = set()
+ response_status_sources: dict[str, int] = {}
+ for row in attempts:
+ status = row.get("response_status")
+ if isinstance(status, int) and not isinstance(status, bool):
+ response_statuses.add(status)
+ source = row.get("response_status_source")
+ if isinstance(source, str) and source:
+ response_status_sources[source] = (
+ response_status_sources.get(source, 0) + 1
+ )
+ outcomes.append(
+ {
+ "target": target,
+ "rank": rank,
+ "domain": domain,
+ "url": str(attempts[0].get("url") or domain),
+ "attempts": attempt_count,
+ "raw_attempts": len(attempts),
+ "non_comparable_attempts": len(attempts) - attempt_count,
+ "passes": passes,
+ "failures": attempt_count - passes,
+ "pass_rate_percent": (
+ (passes / attempt_count) * 100.0 if attempt_count else None
+ ),
+ "outcome": outcome,
+ "categories": categories,
+ "response_statuses": sorted(response_statuses),
+ "response_status_sources": response_status_sources,
+ "status_observed_attempts": sum(
+ 1 for row in attempts if row.get("response_status") is not None
+ ),
+ }
+ )
+ outcomes.sort(
+ key=lambda row: (
+ str(row["target"]),
+ int(row["rank"]),
+ str(row["domain"]),
+ )
+ )
+ return outcomes
-def _write_failure_artifact(
+def _pairwise_site_comparisons(
+ outcomes: list[dict[str, Any]],
+ targets: tuple[str, ...],
*,
- suite_dir: Path,
- row: dict[str, Any],
- stdout: bytes,
- stderr: bytes,
-) -> str:
- domain = _artifact_filename_token(str(row["domain"]))
- name = f"{row['target']}-run-{row['run']}-rank{row['rank']:03d}-{domain}"
- failures_dir = suite_dir / "failures"
- json_path = failures_dir / f"{name}.json"
- stderr_path = failures_dir / f"{name}.stderr.txt"
- write_json(json_path, row)
- write_text(stderr_path, stderr[-16 * 1024 :].decode("utf-8", errors="replace"))
- if stdout.strip():
- stdout_path = failures_dir / f"{name}.stdout.html"
- write_text(stdout_path, stdout[-32 * 1024 :].decode("utf-8", errors="replace"))
- return str(json_path.relative_to(suite_dir))
+ runs: int,
+) -> list[dict[str, Any]]:
+ by_target_site = {
+ (str(row["target"]), int(row["rank"]), str(row["domain"])): row
+ for row in outcomes
+ }
+ site_keys = sorted({(int(row["rank"]), str(row["domain"])) for row in outcomes})
+ comparisons: list[dict[str, Any]] = []
+ for left, right in combinations(targets, 2):
+ counts = {
+ "both_all_pass": 0,
+ "left_all_pass_right_all_fail": 0,
+ "right_all_pass_left_all_fail": 0,
+ "both_all_fail": 0,
+ "flaky_or_incomplete": 0,
+ "not_comparable": 0,
+ }
+ left_only_sites: list[dict[str, Any]] = []
+ right_only_sites: list[dict[str, Any]] = []
+ evaluated_sites = 0
+ for rank, domain in site_keys:
+ left_row = by_target_site.get((left, rank, domain))
+ right_row = by_target_site.get((right, rank, domain))
+ if left_row is None or right_row is None:
+ continue
+ left_outcome = left_row["outcome"]
+ right_outcome = right_row["outcome"]
+ if "not-comparable" in {left_outcome, right_outcome}:
+ counts["not_comparable"] += 1
+ continue
+ evaluated_sites += 1
+ if left_outcome == "all-pass" and right_outcome == "all-pass":
+ counts["both_all_pass"] += 1
+ elif left_outcome == "all-pass" and right_outcome == "all-fail":
+ counts["left_all_pass_right_all_fail"] += 1
+ left_only_sites.append({"rank": rank, "domain": domain})
+ elif right_outcome == "all-pass" and left_outcome == "all-fail":
+ counts["right_all_pass_left_all_fail"] += 1
+ right_only_sites.append({"rank": rank, "domain": domain})
+ elif left_outcome == "all-fail" and right_outcome == "all-fail":
+ counts["both_all_fail"] += 1
+ else:
+ counts["flaky_or_incomplete"] += 1
+ comparisons.append(
+ {
+ "left": left,
+ "right": right,
+ "runs_per_site": runs,
+ "repeat_validated": runs >= 3,
+ "evaluated_sites": evaluated_sites,
+ **counts,
+ "left_only_sites": left_only_sites,
+ "right_only_sites": right_only_sites,
+ }
+ )
+ return comparisons
+
+
+def _successful_attempt_cohort(
+ rows: list[dict[str, Any]],
+ targets: tuple[str, ...],
+) -> dict[str, Any]:
+ return successful_public_web_attempt_cohort(
+ rows,
+ targets,
+ attempt_key_fields=("run", "rank", "domain"),
+ unique_key_fields=("rank", "domain"),
+ unique_count_name="unique_sites",
+ metric_fields=("elapsed_ms", "peak_pss_bytes", "peak_rss_bytes"),
+ require_comparable=True,
+ )
def _domain_to_url(domain: str) -> str:
@@ -744,69 +548,88 @@ def _execute_one(
timeout_seconds: float,
min_body_bytes: int,
proc_env: dict[str, str],
+ schedule_index: int,
+ target_order_index: int,
) -> dict[str, Any]:
url = _domain_to_url(domain)
- if not info.get("available") or not info.get("path"):
- row = {
- "target": target,
- **metadata,
- "run": run_id,
- "rank": rank,
- "domain": domain,
- "url": url,
- "category": "error",
- "ok": False,
- "failure_kind": "target-unavailable",
- "error": "target binary unavailable",
- }
- row["failure_artifact"] = _write_failure_artifact(suite_dir=suite_dir, row=row, stdout=b"", stderr=b"")
+ attempt = PublicWebAttempt.start(
+ target=target,
+ metadata=metadata,
+ target_info=info,
+ run=run_id,
+ case_fields={"rank": rank, "domain": domain},
+ url=url,
+ schedule_index=schedule_index,
+ target_order_index=target_order_index,
+ artifact_stem=(
+ f"{target}-run-{run_id}-rank{rank:03d}-"
+ f"{_artifact_filename_token(domain)}"
+ ),
+ )
+ binary = attempt.binary_path()
+ if binary is None:
+ row = unavailable_public_web_row(attempt, category="error")
+ row["failure_artifact"] = write_public_web_failure_artifacts(
+ suite_dir=suite_dir,
+ attempt=attempt,
+ row=row,
+ stdout=b"",
+ stderr=b"",
+ policy=TOP_SITES_ARTIFACT_POLICY,
+ )
return row
- result = _run_top_sites_target(
+
+ process = _run_top_sites_target(
target=target,
- binary=Path(info["path"]),
+ binary=binary,
url=url,
timeout_seconds=timeout_seconds,
proc_env=proc_env,
)
- snapshot = _extract_title_and_text(result.stdout)
- category = _classify(
- result.stdout,
- result.stderr,
- result.returncode,
- result.timed_out,
- min_body_bytes,
- snapshot,
+ result = PublicWebResult.capture(
+ attempt,
+ process,
+ classifier=TOP_SITES_CLASSIFIER,
+ )
+ category = TOP_SITES_CLASSIFIER.classify_result(
+ result,
+ min_body_bytes=min_body_bytes,
)
- if category not in _ok_categories() and _elapsed_failure_reached_timeout(result.elapsed_ms, timeout_seconds):
+ if (
+ result.response_status is None
+ and not TOP_SITES_CLASSIFIER.classification_ok(category)
+ and _elapsed_failure_reached_timeout(process.elapsed_ms, timeout_seconds)
+ ):
category = "timeout"
- ok = category in _ok_categories()
+ comparable = TOP_SITES_CLASSIFIER.comparable(category)
+ ok = TOP_SITES_CLASSIFIER.classification_ok(category)
row: dict[str, Any] = {
- "target": target,
- **metadata,
- "run": run_id,
- "rank": rank,
- "domain": domain,
- "url": url,
- "command": result.command,
+ **result.base_row(),
+ **result.evidence_fields(policy=TOP_SITES_ARTIFACT_POLICY),
+ "command": process.command,
"category": category,
+ "classification_basis": TOP_SITES_CLASSIFIER.classification_basis(
+ result, category
+ ),
"ok": ok,
- "failure_kind": None if ok else _failure_kind(category, None),
- "elapsed_ms": result.elapsed_ms,
- "returncode": result.returncode,
- "timed_out": result.timed_out,
- "stdout_bytes": len(result.stdout),
- "stderr_bytes": len(result.stderr),
- "title": snapshot["title"],
- "text_length": snapshot["text_length"],
- "text_sample": snapshot["text_sample"],
- "stdout_tail": result.stdout[-512:].decode("utf-8", errors="replace"),
- "stderr_tail": result.stderr[-512:].decode("utf-8", errors="replace"),
- "peak_pss_bytes": result.resources.get("peak_pss_bytes"),
- "peak_rss_bytes": result.resources.get("peak_rss_bytes"),
+ "comparable": comparable,
+ "non_comparable_reason": (
+ None if comparable else "main-document-body-not-captured"
+ ),
+ "failure_kind": (
+ None
+ if ok or not comparable
+ else TOP_SITES_CLASSIFIER.failure_kind(category)
+ ),
}
- if not ok:
- row["failure_artifact"] = _write_failure_artifact(
- suite_dir=suite_dir, row=row, stdout=result.stdout, stderr=result.stderr
+ if comparable and not ok:
+ row["failure_artifact"] = write_public_web_failure_artifacts(
+ suite_dir=suite_dir,
+ attempt=attempt,
+ row=row,
+ stdout=process.stdout,
+ stderr=process.stderr,
+ policy=TOP_SITES_ARTIFACT_POLICY,
)
return row
@@ -853,52 +676,89 @@ def run_top_sites_suite(
list_source = primary_path
proc_env = clear_proxy_env(os.environ)
- rows: list[dict[str, Any]] = []
- for target in targets:
- metadata = _top_sites_target_metadata(target)
- info = target_matrix.get(metadata["binary_key"], {})
- job_specs = [
- (run_id, rank, domain)
- for run_id in range(1, runs_count + 1)
- for rank, domain in entries
- ]
- target_workers = chrome_parallelism if target == "chrome" else parallelism
- with ThreadPoolExecutor(max_workers=target_workers) as executor:
- future_to_spec = {
- executor.submit(
- _execute_one,
- suite_dir=suite_dir,
- target=target,
- metadata=metadata,
- info=info,
- run_id=run_id,
- rank=rank,
- domain=domain,
- timeout_seconds=timeout_seconds,
- min_body_bytes=min_body_bytes,
- proc_env=proc_env,
- ): (target, run_id, rank, domain)
- for run_id, rank, domain in job_specs
- }
- for future in as_completed(future_to_spec):
- rows.append(future.result())
+ metadata_by_target = {
+ target: _top_sites_target_metadata(target) for target in targets
+ }
+ info_by_target = {
+ target: target_matrix.get(metadata_by_target[target]["binary_key"], {})
+ for target in targets
+ }
+ scheduled_cases = schedule_public_web_cases(entries, runs=runs_count)
+
+ def execute_attempt(
+ scheduled: PublicWebScheduledCase[tuple[int, str]],
+ target: str,
+ target_order_index: int,
+ ) -> dict[str, Any]:
+ rank, domain = scheduled.case
+ return _execute_one(
+ suite_dir=suite_dir,
+ target=target,
+ metadata=metadata_by_target[target],
+ info=info_by_target[target],
+ run_id=scheduled.run,
+ rank=rank,
+ domain=domain,
+ timeout_seconds=timeout_seconds,
+ min_body_bytes=min_body_bytes,
+ proc_env=proc_env,
+ schedule_index=scheduled.schedule_index,
+ target_order_index=target_order_index,
+ )
- rows.sort(key=lambda row: (row["target"], row["run"], row["rank"]))
+ rows = PublicWebScheduler[tuple[int, str], dict[str, Any]](
+ targets,
+ parallelism=parallelism,
+ target_parallelism={"chrome": chrome_parallelism},
+ ).run(
+ scheduled_cases,
+ execute_attempt,
+ )
- excluded_domains = _site_unreachable_exclusions(rows) if len(targets) > 1 else {}
+ excluded_domains = (
+ _site_unreachable_exclusions(rows, targets) if len(targets) > 1 else {}
+ )
for row in rows:
exclusion_reason = excluded_domains.get(str(row["domain"]))
row["excluded"] = exclusion_reason is not None
row["exclusion_reason"] = exclusion_reason
counted_rows = [row for row in rows if not row.get("excluded")]
- gate_failures = sum(1 for row in counted_rows if row["target"] == gate_target and not row.get("ok"))
+ comparable_counted_rows = [
+ row for row in counted_rows if row.get("comparable", True)
+ ]
+ site_outcomes = _build_site_outcomes(counted_rows)
+ pairwise_comparisons = _pairwise_site_comparisons(
+ site_outcomes,
+ targets,
+ runs=runs_count,
+ )
+ for comparison in pairwise_comparisons:
+ comparison["common_success"] = _successful_attempt_cohort(
+ counted_rows,
+ (str(comparison["left"]), str(comparison["right"])),
+ )
+ common_success = _successful_attempt_cohort(counted_rows, targets)
+ gate_failures = sum(
+ 1
+ for row in comparable_counted_rows
+ if row["target"] == gate_target and not row.get("ok")
+ )
+ gate_site_outcomes = [
+ row
+ for row in site_outcomes
+ if row["target"] == gate_target and row["outcome"] != "not-comparable"
+ ]
summary: dict[str, Any] = {
"suite": "top-sites",
"profile": profile,
"limit": limit,
"source": resolved_source,
- "list_source": str(list_source.relative_to(REPO_ROOT)) if list_source.is_relative_to(REPO_ROOT) else str(list_source),
+ "list_source": (
+ str(list_source.relative_to(REPO_ROOT))
+ if list_source.is_relative_to(REPO_ROOT)
+ else str(list_source)
+ ),
"list_sources": list_source_labels,
"site_count": len(entries),
"counted_site_count": len(entries) - len(excluded_domains),
@@ -913,40 +773,158 @@ def run_top_sites_suite(
"runs": runs_count,
"timeout_seconds": timeout_seconds,
"min_body_bytes": min_body_bytes,
+ "snapshot_contract": PUBLIC_WEB_SNAPSHOT_CONTRACT,
+ "schedule": "site-paired-rotating-target-order",
+ "scheduled_site_groups": len(scheduled_cases),
"parallelism": parallelism,
"chrome_parallelism": chrome_parallelism,
"gate_target": gate_target,
"gate_failures": gate_failures,
- "total_failures": sum(1 for row in counted_rows if not row.get("ok")),
+ "gate_site_failures": sum(
+ 1 for row in gate_site_outcomes if row["outcome"] != "all-pass"
+ ),
+ "gate_flaky_sites": sum(
+ 1 for row in gate_site_outcomes if row["outcome"] == "flaky"
+ ),
+ "total_failures": sum(
+ 1 for row in comparable_counted_rows if not row.get("ok")
+ ),
+ "total_non_comparable_runs": len(counted_rows)
+ - len(comparable_counted_rows),
"total_excluded_runs": sum(1 for row in rows if row.get("excluded")),
+ "repeat_validated": runs_count >= 3,
+ "pairwise": pairwise_comparisons,
+ "common_success": common_success,
"targets": {},
}
for target in targets:
all_target_rows = [row for row in rows if row["target"] == target]
- target_rows = [row for row in all_target_rows if not row.get("excluded")]
- categories: dict[str, int] = {}
- failure_kinds: dict[str, int] = {}
- for row in target_rows:
- categories[row["category"]] = categories.get(row["category"], 0) + 1
- failure_kind = row.get("failure_kind")
- if isinstance(failure_kind, str) and failure_kind:
- failure_kinds[failure_kind] = failure_kinds.get(failure_kind, 0) + 1
+ target_observations = [
+ row for row in all_target_rows if not row.get("excluded")
+ ]
+ target_rows = [
+ row for row in target_observations if row.get("comparable", True)
+ ]
+ raw_comparable_rows = [
+ row for row in all_target_rows if row.get("comparable", True)
+ ]
+ successful_rows = [row for row in target_rows if row.get("ok")]
+ all_target_site_outcomes = [
+ row for row in site_outcomes if row["target"] == target
+ ]
+ target_site_outcomes = [
+ row
+ for row in all_target_site_outcomes
+ if row["outcome"] != "not-comparable"
+ ]
+ value_counts = count_public_web_row_values(target_observations)
+ status_observed_attempts = sum(
+ 1
+ for row in target_observations
+ if row.get("response_status") is not None
+ )
summary["targets"][target] = {
**_top_sites_target_metadata(target),
"sites": len(target_rows),
"raw_sites": len(all_target_rows),
- "excluded_runs": len(all_target_rows) - len(target_rows),
+ "observed_sites": len(target_observations),
+ "unique_sites": len(target_site_outcomes),
+ "observed_unique_sites": len(all_target_site_outcomes),
+ "attempts": len(target_rows),
+ "raw_observations": len(all_target_rows),
+ "raw_comparable_attempts": len(raw_comparable_rows),
+ "raw_non_comparable_attempts": len(all_target_rows)
+ - len(raw_comparable_rows),
+ "raw_passes": sum(1 for row in raw_comparable_rows if row.get("ok")),
+ "raw_failures": sum(
+ 1 for row in raw_comparable_rows if not row.get("ok")
+ ),
+ "raw_pass_rate_percent": (
+ (
+ sum(1 for row in raw_comparable_rows if row.get("ok"))
+ / len(raw_comparable_rows)
+ )
+ * 100.0
+ if raw_comparable_rows
+ else None
+ ),
+ "non_comparable_attempts": len(target_observations) - len(target_rows),
+ "excluded_runs": len(all_target_rows) - len(target_observations),
"runs": runs_count,
+ "repeat_validated": runs_count >= 3,
"passes": sum(1 for row in target_rows if row.get("ok")),
"failures": sum(1 for row in target_rows if not row.get("ok")),
- "categories": categories,
- "failure_kinds": failure_kinds,
- "elapsed_ms": summarize(row["elapsed_ms"] for row in target_rows if row.get("elapsed_ms") is not None),
- "peak_pss_bytes": summarize(row["peak_pss_bytes"] for row in target_rows if row.get("peak_pss_bytes") is not None),
- "peak_rss_bytes": summarize(row["peak_rss_bytes"] for row in target_rows if row.get("peak_rss_bytes") is not None),
+ "pass_rate_percent": (
+ (sum(1 for row in target_rows if row.get("ok")) / len(target_rows))
+ * 100.0
+ if target_rows
+ else None
+ ),
+ "all_pass_sites": sum(
+ 1 for row in target_site_outcomes if row["outcome"] == "all-pass"
+ ),
+ "all_fail_sites": sum(
+ 1 for row in target_site_outcomes if row["outcome"] == "all-fail"
+ ),
+ "flaky_sites": sum(
+ 1 for row in target_site_outcomes if row["outcome"] == "flaky"
+ ),
+ "status_observed_attempts": status_observed_attempts,
+ "status_unobserved_attempts": len(target_observations)
+ - status_observed_attempts,
+ "response_status_sources": value_counts[
+ "response_status_sources"
+ ],
+ "status_coverage_percent": (
+ (status_observed_attempts / len(target_observations)) * 100.0
+ if target_observations
+ else None
+ ),
+ "content_inferred_http_failures": sum(
+ 1
+ for row in target_rows
+ if row.get("response_status") is None
+ and row.get("returncode") == 0
+ and row.get("category")
+ in {"blocked-or-forbidden", "not-found", "http-error"}
+ ),
+ "categories": value_counts["categories"],
+ "failure_kinds": value_counts["failure_kinds"],
+ "classification_bases": value_counts["classification_bases"],
+ "elapsed_ms": summarize(
+ row["elapsed_ms"]
+ for row in target_rows
+ if row.get("elapsed_ms") is not None
+ ),
+ "successful_elapsed_ms": summarize(
+ row["elapsed_ms"]
+ for row in successful_rows
+ if row.get("elapsed_ms") is not None
+ ),
+ "peak_pss_bytes": summarize(
+ row["peak_pss_bytes"]
+ for row in target_rows
+ if row.get("peak_pss_bytes") is not None
+ ),
+ "successful_peak_pss_bytes": summarize(
+ row["peak_pss_bytes"]
+ for row in successful_rows
+ if row.get("peak_pss_bytes") is not None
+ ),
+ "peak_rss_bytes": summarize(
+ row["peak_rss_bytes"]
+ for row in target_rows
+ if row.get("peak_rss_bytes") is not None
+ ),
+ "successful_peak_rss_bytes": summarize(
+ row["peak_rss_bytes"]
+ for row in successful_rows
+ if row.get("peak_rss_bytes") is not None
+ ),
}
write_csv(suite_dir / "raw-runs.csv", rows)
write_json(suite_dir / "runs.json", rows)
+ write_json(suite_dir / "site-outcomes.json", site_outcomes)
write_json(suite_dir / "summary.json", summary)
return summary
diff --git a/moli-benchmark/moli_benchmark/versions.py b/moli-benchmark/moli_benchmark/versions.py
index 48724a581..bf1c9e192 100644
--- a/moli-benchmark/moli_benchmark/versions.py
+++ b/moli-benchmark/moli_benchmark/versions.py
@@ -9,6 +9,9 @@ from . import __version__
from .config import REPO_ROOT
+MOLI_VERSION_ARGS = ("--version",)
+
+
def sha256_file(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as file:
@@ -33,7 +36,7 @@ def collect_versions(moli_bin: Path | None = None, targets: dict[str, Any] | Non
"path": str(moli_bin),
"size_bytes": moli_bin.stat().st_size,
"sha256": sha256_file(moli_bin),
- "version": _run([str(moli_bin), "version"]),
+ "version": _run([str(moli_bin), *MOLI_VERSION_ARGS]),
}
elif targets and isinstance(targets.get("moli"), dict):
moli = targets["moli"]
diff --git a/moli-benchmark/moli_benchmark/wild_web.py b/moli-benchmark/moli_benchmark/wild_web.py
index d4a4dad10..df968efaa 100644
--- a/moli-benchmark/moli_benchmark/wild_web.py
+++ b/moli-benchmark/moli_benchmark/wild_web.py
@@ -1,21 +1,34 @@
from __future__ import annotations
-import html.parser
import os
from pathlib import Path
from typing import Any
-from .artifacts import write_csv, write_json, write_text
+from .artifacts import write_csv, write_json
from .chrome_dcl import run_chrome_dcl_dump, run_served_cdp_dcl_dump
-from .config import REPO_ROOT, clear_proxy_env
-from .process import run_process
-from .stats import summarize
-from .synthetic_compare import (
- WEBFETCH_TARGETS,
- target_enables_all_resource_fetch,
- target_is_cdp,
- target_metadata,
+from .config import clear_proxy_env
+from .process import ProcessResult, run_process
+from .public_web import (
+ PUBLIC_WEB_SNAPSHOT_CONTRACT,
+ WILD_WEB_ARTIFACT_POLICY,
+ WILD_WEB_CLASSIFIER,
+ PublicWebAttempt,
+ PublicWebResult,
+ PublicWebScheduledCase,
+ PublicWebScheduler,
+ build_public_web_fetch_command,
+ count_public_web_row_values,
+ public_web_target_metadata as _wild_web_target_metadata,
+ rotated_target_order as _rotated_target_order,
+ run_public_web_target,
+ schedule_public_web_cases,
+ successful_public_web_attempt_cohort,
+ unavailable_public_web_row,
+ write_public_web_failure_artifacts,
+ write_public_web_replay_artifact,
)
+from .stats import summarize
+from .synthetic_compare import WEBFETCH_TARGETS
WILD_WEB_SEEDS: dict[str, str] = {
@@ -26,54 +39,13 @@ WILD_WEB_SEEDS: dict[str, str] = {
}
WILD_WEB_SEED_ASSERTIONS: dict[str, dict[str, Any]] = {
- "baidu-home": {"title_any": ("百度", "baidu"), "text_any": ("百度", "baidu"), "min_text_length": 20},
- "bilibili-home": {"title_any": ("哔哩", "bilibili"), "text_any": ("哔哩", "bilibili"), "min_text_length": 20},
- "zhihu-home": {"title_any": ("知乎", "zhihu"), "text_any": ("知乎", "zhihu"), "min_text_length": 20},
- "toutiao-home": {"title_any": ("头条", "toutiao"), "text_any": ("头条", "toutiao"), "min_text_length": 20},
+ "baidu-home": {"title_any": ("百度", "baidu"), "min_text_length": 20},
+ "bilibili-home": {"title_any": ("哔哩", "bilibili"), "min_text_length": 20},
+ "zhihu-home": {"title_any": ("知乎", "zhihu"), "min_text_length": 20},
+ "toutiao-home": {"title_any": ("头条", "toutiao"), "min_text_length": 20},
}
-
-class _TextExtractor(html.parser.HTMLParser):
- def __init__(self) -> None:
- super().__init__(convert_charrefs=True)
- self._hidden_depth = 0
- self._in_title = False
- self.title_parts: list[str] = []
- self.text_parts: list[str] = []
-
- def handle_starttag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None:
- if tag in {"script", "style", "noscript"}:
- self._hidden_depth += 1
- if tag == "title":
- self._in_title = True
-
- def handle_endtag(self, tag: str) -> None:
- if tag in {"script", "style", "noscript"} and self._hidden_depth:
- self._hidden_depth -= 1
- if tag == "title":
- self._in_title = False
-
- def handle_data(self, data: str) -> None:
- text = " ".join(data.split())
- if not text:
- return
- if self._in_title:
- self.title_parts.append(text)
- if self._hidden_depth == 0:
- self.text_parts.append(text)
-
-
-def _extract_page_snapshot(stdout: bytes) -> dict[str, Any]:
- text = stdout.decode("utf-8", errors="replace")
- parser = _TextExtractor()
- parser.feed(text)
- title = " ".join(parser.title_parts).strip()
- body_text = " ".join(parser.text_parts).strip()
- return {
- "title": title,
- "text_length": len(body_text),
- "text_sample": body_text[:500],
- }
+_extract_page_snapshot = WILD_WEB_CLASSIFIER.snapshot
def _wild_web_extraction_failures(seed: str, snapshot: dict[str, Any]) -> list[str]:
@@ -98,142 +70,191 @@ def _wild_web_extraction_failures(seed: str, snapshot: dict[str, Any]) -> list[s
return failures
-def _wild_command_for_target(target: str, binary: Path, url: str, timeout_seconds: float) -> list[str]:
- metadata = target_metadata(target)
- if target_is_cdp(target):
- raise RuntimeError(f"{target} is a CDP target; use the cdp-session suite")
- if target == "chrome":
- raise RuntimeError("chrome wild-web uses the CDP DCL runner")
- timeout_ms = str(int(timeout_seconds * 1000))
- if target in {"moli", "moli-full"}:
- compatibility_args = (
- ["--layout", "--resource"]
- if target_enables_all_resource_fetch(target)
- else []
- )
- return [
- str(binary),
- "fetch",
- *compatibility_args,
- "--dump",
- "html",
- "--wait-until",
- "done",
- "--timeout",
- timeout_ms,
- url,
- ]
- if target == "lightpanda":
- return [
- str(binary),
- "fetch",
- "--dump",
- "html",
- "--wait-until",
- "done",
- "--wait-ms",
- timeout_ms,
- "--http-timeout",
- timeout_ms,
- "--terminate-ms",
- timeout_ms,
- url,
- ]
- if target == "obscura":
- return [
- str(binary),
- "fetch",
- "--dump",
- "html",
- "--wait-until",
- "load",
- "--wait",
- "0",
- "--timeout",
- str(max(1, int(timeout_seconds))),
- url,
- ]
- raise RuntimeError(f"unknown target: {target}")
+def _wild_command_for_target(
+ target: str,
+ binary: Path,
+ url: str,
+ timeout_seconds: float,
+) -> list[str]:
+ return build_public_web_fetch_command(
+ target,
+ binary,
+ url,
+ timeout_seconds,
+ suite_name="wild-web",
+ )
-def _wild_web_target_metadata(target: str) -> dict[str, str]:
- metadata = dict(target_metadata(target))
- if target == "chrome" or target_is_cdp(target):
- metadata["driver"] = "cdp-dcl"
- prefix = "moli full" if target == "moli-full-cdp" else metadata["engine"]
- metadata["label"] = f"{prefix} / cdp-dcl"
- return metadata
+def _run_wild_web_target(
+ *,
+ target: str,
+ binary: Path,
+ url: str,
+ timeout_seconds: float,
+ proc_env: dict[str, str],
+) -> ProcessResult:
+ return run_public_web_target(
+ target=target,
+ binary=binary,
+ url=url,
+ timeout_seconds=timeout_seconds,
+ proc_env=proc_env,
+ command_builder=_wild_command_for_target,
+ chrome_runner=run_chrome_dcl_dump,
+ served_cdp_runner=run_served_cdp_dcl_dump,
+ process_runner=run_process,
+ )
-def _classify(stdout: bytes, stderr: bytes, returncode: int | None, timed_out: bool) -> str:
- if timed_out:
- return "timeout"
- if returncode != 0:
- return "error"
- snapshot = _extract_page_snapshot(stdout)
- visible_text = " ".join(
- str(snapshot.get(key) or "") for key in ("title", "text_sample")
- ).lower()
- diagnostic_text = stderr.decode("utf-8", errors="replace").lower()
- text = f"{visible_text}\n{diagnostic_text}"
- if "captcha" in text or "verify" in text or "challenge" in text or "安全验证" in text:
- return "challenge"
- if "login" in text or "登录" in text:
- return "login"
- if "blocked" in text or "forbidden" in text or "403" in text:
- return "blocked"
- return "success" if stdout.strip() else "empty"
+def _classify(
+ stdout: bytes,
+ stderr: bytes,
+ returncode: int | None,
+ timed_out: bool,
+ response_status: int | None = None,
+) -> str:
+ return WILD_WEB_CLASSIFIER.classify_output(
+ stdout=stdout,
+ stderr=stderr,
+ returncode=returncode,
+ timed_out=timed_out,
+ response_status=response_status,
+ )
def _failure_kind(category: str, extraction_failures: list[str], error: str | None = None) -> str | None:
- if error == "target binary unavailable":
- return "target-unavailable"
- if category == "timeout":
- return "timeout"
- if category == "blocked":
- return "blocked"
- if category == "challenge":
- return "challenge"
- if category == "login":
- return "login"
- if category == "empty":
- return "empty-response"
- if category == "error":
- return "process-error"
- if extraction_failures:
- return "extraction-failure"
- return None
+ return WILD_WEB_CLASSIFIER.failure_kind(
+ category,
+ error=error,
+ extraction_failed=bool(extraction_failures),
+ )
-def _write_failure_artifacts(
- *,
- suite_dir: Path,
- row: dict[str, Any],
- stdout: bytes,
- stderr: bytes,
-) -> str:
- name = f"{row['target']}-run-{row['run']}-{row['seed']}"
- failures_dir = suite_dir / "failures"
- json_path = failures_dir / f"{name}.json"
- stdout_path = failures_dir / f"{name}.stdout.html"
- stderr_path = failures_dir / f"{name}.stderr.txt"
- write_json(json_path, row)
- write_text(stdout_path, stdout[-128 * 1024 :].decode("utf-8", errors="replace"))
- write_text(stderr_path, stderr[-32 * 1024 :].decode("utf-8", errors="replace"))
- return str(json_path.relative_to(suite_dir))
+def _successful_attempt_cohort(
+ rows: list[dict[str, Any]],
+ targets: tuple[str, ...],
+) -> dict[str, Any]:
+ return successful_public_web_attempt_cohort(
+ rows,
+ targets,
+ attempt_key_fields=("run", "seed"),
+ unique_key_fields=("seed",),
+ unique_count_name="unique_seeds",
+ metric_fields=("elapsed_ms", "peak_pss_bytes"),
+ )
-def _write_replay_artifact(
+def _execute_wild_web_attempt(
*,
suite_dir: Path,
- row: dict[str, Any],
- stdout: bytes,
-) -> str:
- name = f"{row['target']}-run-{row['run']}-{row['seed']}.html"
- replay_dir = suite_dir / "replay"
- replay_path = replay_dir / name
- write_text(replay_path, stdout.decode("utf-8", errors="replace"))
- return str(replay_path.relative_to(suite_dir))
+ target: str,
+ metadata: dict[str, str],
+ info: dict[str, Any],
+ run_id: int,
+ seed: str,
+ timeout_seconds: float,
+ proc_env: dict[str, str],
+ schedule_index: int,
+ target_order_index: int,
+ capture_replay: bool,
+) -> tuple[dict[str, Any], dict[str, Any] | None]:
+ url = WILD_WEB_SEEDS[seed]
+ attempt = PublicWebAttempt.start(
+ target=target,
+ metadata=metadata,
+ target_info=info,
+ run=run_id,
+ case_fields={"seed": seed},
+ url=url,
+ schedule_index=schedule_index,
+ target_order_index=target_order_index,
+ artifact_stem=f"{target}-run-{run_id}-{seed}",
+ )
+ binary = attempt.binary_path()
+ if binary is None:
+ row = unavailable_public_web_row(
+ attempt,
+ category="error",
+ extra_fields={
+ "classification_ok": False,
+ "extraction_ok": False,
+ "extraction_failures": ["target-unavailable"],
+ "extraction_failure_count": 1,
+ },
+ )
+ row["failure_artifact"] = write_public_web_failure_artifacts(
+ suite_dir=suite_dir,
+ attempt=attempt,
+ row=row,
+ stdout=b"",
+ stderr=b"",
+ policy=WILD_WEB_ARTIFACT_POLICY,
+ )
+ return row, None
+
+ process = _run_wild_web_target(
+ target=target,
+ binary=binary,
+ url=url,
+ timeout_seconds=timeout_seconds,
+ proc_env=proc_env,
+ )
+ result = PublicWebResult.capture(
+ attempt,
+ process,
+ classifier=WILD_WEB_CLASSIFIER,
+ )
+ category = WILD_WEB_CLASSIFIER.classify_result(result)
+ snapshot = result.snapshot
+ extraction_failures = _wild_web_extraction_failures(seed, snapshot)
+ classification_ok = WILD_WEB_CLASSIFIER.classification_ok(category)
+ extraction_ok = not extraction_failures
+ ok = classification_ok and extraction_ok
+ row = {
+ **result.base_row(),
+ **result.evidence_fields(policy=WILD_WEB_ARTIFACT_POLICY),
+ "category": category,
+ "classification_basis": WILD_WEB_CLASSIFIER.classification_basis(
+ result, category
+ ),
+ "classification_ok": classification_ok,
+ "extraction_ok": extraction_ok,
+ "ok": ok,
+ "title": snapshot["title"],
+ "text_length": snapshot["text_length"],
+ "text_sample": snapshot["text_sample"],
+ "extraction_failures": extraction_failures,
+ "extraction_failure_count": len(extraction_failures),
+ "failure_kind": None if ok else _failure_kind(category, extraction_failures),
+ }
+ if not ok:
+ row["failure_artifact"] = write_public_web_failure_artifacts(
+ suite_dir=suite_dir,
+ attempt=attempt,
+ row=row,
+ stdout=process.stdout,
+ stderr=process.stderr,
+ policy=WILD_WEB_ARTIFACT_POLICY,
+ )
+ return row, None
+ if not capture_replay:
+ return row, None
+ row["replay_artifact"] = write_public_web_replay_artifact(
+ suite_dir=suite_dir,
+ attempt=attempt,
+ stdout=process.stdout,
+ )
+ return row, {
+ "target": target,
+ **metadata,
+ "run": run_id,
+ "seed": seed,
+ "url": url,
+ "category": category,
+ "title": snapshot["title"],
+ "text_length": snapshot["text_length"],
+ "artifact": row["replay_artifact"],
+ }
def run_wild_web_suite(
@@ -254,121 +275,49 @@ def run_wild_web_suite(
raise RuntimeError(f"unknown gate target: {gate_target}")
suite_dir = output_dir / "wild-web"
- rows: list[dict[str, Any]] = []
- replay_manifest: list[dict[str, Any]] = []
selected = seeds or tuple(WILD_WEB_SEEDS.keys())
for seed in selected:
if seed not in WILD_WEB_SEEDS:
raise RuntimeError(f"unknown wild web seed: {seed}")
- for target in targets:
- metadata = _wild_web_target_metadata(target)
- info = target_matrix.get(metadata["binary_key"], {})
- path = info.get("path")
- for run_id in range(1, runs + 1):
- for seed in selected:
- url = WILD_WEB_SEEDS[seed]
- if not info.get("available") or not path:
- row = {
- "target": target,
- **metadata,
- "run": run_id,
- "seed": seed,
- "url": url,
- "category": "error",
- "classification_ok": False,
- "extraction_ok": False,
- "ok": False,
- "extraction_failures": ["target-unavailable"],
- "extraction_failure_count": 1,
- "failure_kind": "target-unavailable",
- "error": "target binary unavailable",
- }
- row["failure_artifact"] = _write_failure_artifacts(suite_dir=suite_dir, row=row, stdout=b"", stderr=b"")
- rows.append(row)
- continue
- proc_env = clear_proxy_env(os.environ)
- if target == "chrome":
- result = run_chrome_dcl_dump(
- Path(path),
- url,
- cwd=REPO_ROOT,
- timeout_seconds=timeout_seconds,
- env=proc_env,
- )
- elif target_is_cdp(target):
- result = run_served_cdp_dcl_dump(
- target,
- Path(path),
- url,
- cwd=REPO_ROOT,
- timeout_seconds=timeout_seconds,
- env=proc_env,
- )
- else:
- result = run_process(
- _wild_command_for_target(target, Path(path), url, timeout_seconds),
- cwd=REPO_ROOT,
- timeout_seconds=timeout_seconds + 2,
- env=proc_env,
- )
- category = _classify(result.stdout, result.stderr, result.returncode, result.timed_out)
- snapshot = _extract_page_snapshot(result.stdout)
- extraction_failures = _wild_web_extraction_failures(seed, snapshot)
- classification_ok = category in {"success", "login", "challenge"}
- extraction_ok = not extraction_failures
- ok = classification_ok and extraction_ok
- row = {
- "target": target,
- **metadata,
- "run": run_id,
- "seed": seed,
- "url": url,
- "category": category,
- "classification_ok": classification_ok,
- "extraction_ok": extraction_ok,
- "ok": ok,
- "title": snapshot["title"],
- "text_length": snapshot["text_length"],
- "text_sample": snapshot["text_sample"],
- "extraction_failures": extraction_failures,
- "extraction_failure_count": len(extraction_failures),
- "failure_kind": None if ok else _failure_kind(category, extraction_failures),
- "elapsed_ms": result.elapsed_ms,
- "returncode": result.returncode,
- "timed_out": result.timed_out,
- "stdout_bytes": len(result.stdout),
- "stderr_bytes": len(result.stderr),
- "stdout_tail": result.stdout[-1024:].decode("utf-8", errors="replace"),
- "stderr_tail": result.stderr[-1024:].decode("utf-8", errors="replace"),
- "peak_pss_bytes": result.resources.get("peak_pss_bytes"),
- }
- if not ok:
- row["failure_artifact"] = _write_failure_artifacts(
- suite_dir=suite_dir,
- row=row,
- stdout=result.stdout,
- stderr=result.stderr,
- )
- elif capture_replay:
- row["replay_artifact"] = _write_replay_artifact(
- suite_dir=suite_dir,
- row=row,
- stdout=result.stdout,
- )
- replay_manifest.append(
- {
- "target": target,
- **metadata,
- "run": run_id,
- "seed": seed,
- "url": url,
- "category": category,
- "title": snapshot["title"],
- "text_length": snapshot["text_length"],
- "artifact": row["replay_artifact"],
- }
- )
- rows.append(row)
+ proc_env = clear_proxy_env(os.environ)
+ metadata_by_target = {
+ target: _wild_web_target_metadata(target) for target in targets
+ }
+ info_by_target = {
+ target: target_matrix.get(metadata_by_target[target]["binary_key"], {})
+ for target in targets
+ }
+ scheduled_cases = schedule_public_web_cases(selected, runs=runs)
+
+ def execute_attempt(
+ scheduled: PublicWebScheduledCase[str],
+ target: str,
+ target_order_index: int,
+ ) -> tuple[dict[str, Any], dict[str, Any] | None]:
+ return _execute_wild_web_attempt(
+ suite_dir=suite_dir,
+ target=target,
+ metadata=metadata_by_target[target],
+ info=info_by_target[target],
+ run_id=scheduled.run,
+ seed=scheduled.case,
+ timeout_seconds=timeout_seconds,
+ proc_env=proc_env,
+ schedule_index=scheduled.schedule_index,
+ target_order_index=target_order_index,
+ capture_replay=capture_replay,
+ )
+
+ executions = PublicWebScheduler[
+ str,
+ tuple[dict[str, Any], dict[str, Any] | None],
+ ](targets).run(scheduled_cases, execute_attempt)
+ rows = [row for row, _ in executions]
+ replay_manifest = [
+ replay_entry
+ for _, replay_entry in executions
+ if replay_entry is not None
+ ]
gate_failures = sum(1 for row in rows if row["target"] == gate_target and not row.get("ok"))
summary: dict[str, Any] = {
@@ -376,33 +325,52 @@ def run_wild_web_suite(
"runs": runs,
"seeds": list(selected),
"timeout_seconds": timeout_seconds,
+ "snapshot_contract": PUBLIC_WEB_SNAPSHOT_CONTRACT,
+ "schedule": "site-paired-rotating-target-order",
+ "scheduled_site_groups": len(scheduled_cases),
"gate_target": gate_target,
"gate_failures": gate_failures,
"targets": {},
"total_failures": sum(1 for row in rows if not row.get("ok")),
"replay_capture": bool(capture_replay),
"replay_artifacts": len(replay_manifest),
+ "common_success": _successful_attempt_cohort(rows, targets),
}
for target in targets:
target_rows = [row for row in rows if row["target"] == target]
- categories: dict[str, int] = {}
- failure_kinds: dict[str, int] = {}
- for row in target_rows:
- categories[row["category"]] = categories.get(row["category"], 0) + 1
- failure_kind = row.get("failure_kind")
- if isinstance(failure_kind, str) and failure_kind:
- failure_kinds[failure_kind] = failure_kinds.get(failure_kind, 0) + 1
+ successful_rows = [row for row in target_rows if row.get("ok")]
+ value_counts = count_public_web_row_values(target_rows)
summary["targets"][target] = {
**_wild_web_target_metadata(target),
"seeds": len(target_rows),
"runs": runs,
"passes": sum(1 for row in target_rows if row.get("ok")),
"failures": sum(1 for row in target_rows if not row.get("ok")),
+ "status_observed_attempts": sum(
+ 1 for row in target_rows if row.get("response_status") is not None
+ ),
+ "status_unobserved_attempts": sum(
+ 1 for row in target_rows if row.get("response_status") is None
+ ),
+ "response_status_sources": value_counts[
+ "response_status_sources"
+ ],
"extraction_failures": sum(int(row.get("extraction_failure_count", 0) or 0) for row in target_rows),
- "categories": categories,
- "failure_kinds": failure_kinds,
+ "categories": value_counts["categories"],
+ "failure_kinds": value_counts["failure_kinds"],
+ "classification_bases": value_counts["classification_bases"],
"elapsed_ms": summarize(row["elapsed_ms"] for row in target_rows if row.get("elapsed_ms") is not None),
+ "successful_elapsed_ms": summarize(
+ row["elapsed_ms"]
+ for row in successful_rows
+ if row.get("elapsed_ms") is not None
+ ),
"peak_pss_bytes": summarize(row["peak_pss_bytes"] for row in target_rows if row.get("peak_pss_bytes") is not None),
+ "successful_peak_pss_bytes": summarize(
+ row["peak_pss_bytes"]
+ for row in successful_rows
+ if row.get("peak_pss_bytes") is not None
+ ),
}
write_csv(suite_dir / "raw-runs.csv", rows)
write_json(suite_dir / "runs.json", rows)
diff --git a/moli-benchmark/tests/test_chrome_dcl.py b/moli-benchmark/tests/test_chrome_dcl.py
index a7d92b9fc..8dde4898c 100644
--- a/moli-benchmark/tests/test_chrome_dcl.py
+++ b/moli-benchmark/tests/test_chrome_dcl.py
@@ -9,14 +9,20 @@ from pathlib import Path
from unittest import mock
from moli_benchmark.chrome_dcl import (
+ CdpDclDumpResult,
CdpDclDumpTimeoutError,
DEFAULT_CHROME_DCL_USER_AGENT,
- _binary_main_resource_body_from_message,
+ POST_DCL_SETTLE_MILLISECONDS,
+ _POST_DCL_OUTER_HTML_EXPRESSION,
+ _POST_DCL_SETTLE_EXPRESSION,
+ _binary_main_resource_mime_type_from_message,
_chrome_command,
+ _navigation_frame_and_loader,
_recv_command_response,
_recv_until_dcl_or_binary_main_resource,
run_chrome_dcl_dump,
)
+from moli_benchmark.public_web import TOP_SITES_CLASSIFIER, WILD_WEB_CLASSIFIER
from moli_benchmark.raw_cdp import RawCdpClient, RawCdpError, RawCdpTimeoutError
@@ -70,11 +76,23 @@ class _LateCommandResponseClient:
return self.late_message
+class _QueuedMessageClient:
+ def __init__(self, messages: list[dict[str, object]]) -> None:
+ self.messages = messages
+
+ async def recv(self) -> dict[str, object]:
+ if not self.messages:
+ raise AssertionError("CDP receiver exhausted before matching navigation evidence")
+ return self.messages.pop(0)
+
+
def _document_response_message(
mime_type: str,
*,
- resource_type: str = "Document",
+ resource_type: str | None = "Document",
+ request_id: str = "REQUEST-1",
status: int = 200,
+ loader_id: str = "LOADER-1",
) -> dict[str, object]:
return {
"sessionId": "SID-1",
@@ -82,12 +100,55 @@ def _document_response_message(
"params": {
"type": resource_type,
"frameId": "FRAME-1",
- "response": {"mimeType": mime_type, "status": status},
+ "loaderId": loader_id,
+ "requestId": request_id,
+ "response": {
+ "mimeType": mime_type,
+ "status": status,
+ "url": "https://example.test/final",
+ },
+ },
+ }
+
+
+def _document_request_message(
+ *,
+ request_id: str = "REQUEST-1",
+ loader_id: str = "LOADER-1",
+) -> dict[str, object]:
+ return {
+ "sessionId": "SID-1",
+ "method": "Network.requestWillBeSent",
+ "params": {
+ "type": "Document",
+ "frameId": "FRAME-1",
+ "loaderId": loader_id,
+ "requestId": request_id,
+ "request": {"url": "https://example.test/"},
+ },
+ }
+
+
+def _dcl_lifecycle_message(*, loader_id: str = "LOADER-1") -> dict[str, object]:
+ return {
+ "sessionId": "SID-1",
+ "method": "Page.lifecycleEvent",
+ "params": {
+ "frameId": "FRAME-1",
+ "loaderId": loader_id,
+ "name": "DOMContentLoaded",
},
}
class ChromeDclTests(unittest.TestCase):
+ def test_post_dcl_snapshot_uses_page_event_loop_settle(self) -> None:
+ self.assertEqual(POST_DCL_SETTLE_MILLISECONDS, 50)
+ self.assertIn("new Promise", _POST_DCL_SETTLE_EXPRESSION)
+ self.assertIn("setTimeout", _POST_DCL_SETTLE_EXPRESSION)
+ self.assertIn("50", _POST_DCL_SETTLE_EXPRESSION)
+ self.assertIn("document.documentElement.outerHTML", _POST_DCL_OUTER_HTML_EXPRESSION)
+
def test_chrome_command_uses_non_headless_desktop_user_agent(self) -> None:
with tempfile.TemporaryDirectory() as temp_dir:
command = _chrome_command(Path("/bin/chromium"), 12345, Path(temp_dir))
@@ -163,81 +224,370 @@ class ChromeDclTests(unittest.TestCase):
asyncio.run(run())
- def test_binary_main_document_response_returns_benchmark_binary_body(self) -> None:
- body = _binary_main_resource_body_from_message(
+ def test_navigation_result_surfaces_error_text_before_dcl_wait(self) -> None:
+ with self.assertRaises(RawCdpError) as raised:
+ _navigation_frame_and_loader(
+ {
+ "result": {
+ "frameId": "FRAME-1",
+ "errorText": "net::ERR_NAME_NOT_RESOLVED",
+ }
+ },
+ "https://missing.example.test/",
+ )
+
+ self.assertEqual(
+ str(raised.exception),
+ "Page.navigate failed for `https://missing.example.test/`: "
+ "net::ERR_NAME_NOT_RESOLVED",
+ )
+
+ def test_navigation_result_exposes_frame_and_loader_identity(self) -> None:
+ self.assertEqual(
+ _navigation_frame_and_loader(
+ {"result": {"frameId": "FRAME-1", "loaderId": "LOADER-1"}},
+ "https://example.test/",
+ ),
+ ("FRAME-1", "LOADER-1"),
+ )
+
+ def test_binary_main_document_response_returns_mime_evidence(self) -> None:
+ mime_type = _binary_main_resource_mime_type_from_message(
_document_response_message("application/pdf"),
session_id="SID-1",
frame_id="FRAME-1",
)
- self.assertIsNotNone(body)
- self.assertTrue(body.startswith("%PDF-"))
- self.assertIn("\0", body or "")
+ self.assertEqual(mime_type, "application/pdf")
def test_binary_main_document_detection_ignores_html_and_subresources(self) -> None:
- html_body = _binary_main_resource_body_from_message(
+ html_mime_type = _binary_main_resource_mime_type_from_message(
_document_response_message("text/html; charset=utf-8"),
session_id="SID-1",
frame_id="FRAME-1",
)
- script_pdf_body = _binary_main_resource_body_from_message(
+ script_pdf_mime_type = _binary_main_resource_mime_type_from_message(
_document_response_message("application/pdf", resource_type="Script"),
session_id="SID-1",
frame_id="FRAME-1",
)
- self.assertIsNone(html_body)
- self.assertIsNone(script_pdf_body)
+ self.assertIsNone(html_mime_type)
+ self.assertIsNone(script_pdf_mime_type)
def test_binary_main_document_detection_ignores_error_status(self) -> None:
- body = _binary_main_resource_body_from_message(
+ mime_type = _binary_main_resource_mime_type_from_message(
_document_response_message("application/pdf", status=404),
session_id="SID-1",
frame_id="FRAME-1",
)
- self.assertIsNone(body)
+ self.assertIsNone(mime_type)
def test_recv_until_dcl_short_circuits_binary_document_seen_before_dcl(self) -> None:
async def run() -> None:
- body = await _recv_until_dcl_or_binary_main_resource(
+ observation = await _recv_until_dcl_or_binary_main_resource(
mock.Mock(),
session_id="SID-1",
frame_id="FRAME-1",
+ expected_loader_id="LOADER-1",
deadline=time.perf_counter() + 1.0,
seen=[
_document_response_message("application/pdf"),
+ _dcl_lifecycle_message(),
+ ],
+ )
+ binary_mime_type, response_status, response_mime_type, final_url = observation
+ self.assertEqual(binary_mime_type, "application/pdf")
+ self.assertEqual(response_status, 200)
+ self.assertEqual(response_mime_type, "application/pdf")
+ self.assertEqual(final_url, "https://example.test/final")
+
+ asyncio.run(run())
+
+ def test_recv_until_dcl_accepts_main_frame_event_seen_before_command_response(self) -> None:
+ async def run() -> None:
+ observation = await _recv_until_dcl_or_binary_main_resource(
+ mock.Mock(),
+ session_id="SID-1",
+ frame_id="FRAME-1",
+ expected_loader_id="LOADER-1",
+ deadline=time.perf_counter() + 1.0,
+ seen=[_dcl_lifecycle_message()],
+ )
+ binary_mime_type, response_status, response_mime_type, final_url = observation
+ self.assertIsNone(binary_mime_type)
+ self.assertIsNone(response_status)
+ self.assertIsNone(response_mime_type)
+ self.assertIsNone(final_url)
+
+ asyncio.run(run())
+
+ def test_recv_until_dcl_ignores_events_from_previous_loader(self) -> None:
+ async def run() -> None:
+ client = _QueuedMessageClient(
+ [
+ _document_response_message(
+ "text/html",
+ status=200,
+ loader_id="LOADER-NEW",
+ ),
{
"sessionId": "SID-1",
- "method": "Page.lifecycleEvent",
- "params": {"frameId": "FRAME-1", "name": "DOMContentLoaded"},
+ "method": "Page.domContentEventFired",
+ "params": {},
},
+ _dcl_lifecycle_message(loader_id="LOADER-NEW"),
+ ]
+ )
+ observation = await _recv_until_dcl_or_binary_main_resource( # type: ignore[arg-type]
+ client,
+ session_id="SID-1",
+ frame_id="FRAME-1",
+ expected_loader_id="LOADER-NEW",
+ deadline=time.perf_counter() + 1.0,
+ seen=[
+ _document_response_message(
+ "text/html",
+ status=502,
+ loader_id="LOADER-OLD",
+ ),
+ _dcl_lifecycle_message(loader_id="LOADER-OLD"),
],
)
- self.assertIsNotNone(body)
- self.assertTrue((body or "").startswith("%PDF-"))
+
+ self.assertEqual(
+ observation,
+ (None, 200, "text/html", "https://example.test/final"),
+ )
+ self.assertEqual(client.messages, [])
asyncio.run(run())
- def test_recv_until_dcl_accepts_main_frame_event_seen_before_command_response(self) -> None:
+ def test_recv_until_dcl_retains_html_main_document_status(self) -> None:
async def run() -> None:
- body = await _recv_until_dcl_or_binary_main_resource(
+ observation = await _recv_until_dcl_or_binary_main_resource(
mock.Mock(),
session_id="SID-1",
frame_id="FRAME-1",
+ expected_loader_id="LOADER-1",
deadline=time.perf_counter() + 1.0,
seen=[
- {
- "sessionId": "SID-1",
- "method": "Page.lifecycleEvent",
- "params": {"frameId": "FRAME-1", "name": "DOMContentLoaded"},
- }
+ _document_response_message("text/html", status=502),
+ _dcl_lifecycle_message(),
+ ],
+ )
+ binary_mime_type, response_status, response_mime_type, final_url = observation
+ self.assertIsNone(binary_mime_type)
+ self.assertEqual(response_status, 502)
+ self.assertEqual(response_mime_type, "text/html")
+ self.assertEqual(final_url, "https://example.test/final")
+
+ asyncio.run(run())
+
+ def test_recv_until_dcl_correlates_response_without_resource_type(self) -> None:
+ async def run() -> None:
+ observation = await _recv_until_dcl_or_binary_main_resource(
+ mock.Mock(),
+ session_id="SID-1",
+ frame_id="FRAME-1",
+ expected_loader_id="LOADER-1",
+ deadline=time.perf_counter() + 1.0,
+ seen=[
+ _document_request_message(),
+ _document_response_message(
+ "text/html",
+ resource_type=None,
+ status=400,
+ ),
+ _dcl_lifecycle_message(),
+ ],
+ )
+ binary_mime_type, response_status, response_mime_type, final_url = observation
+ self.assertIsNone(binary_mime_type)
+ self.assertEqual(response_status, 400)
+ self.assertEqual(response_mime_type, "text/html")
+ self.assertEqual(final_url, "https://example.test/final")
+
+ asyncio.run(run())
+
+ def test_recv_until_dcl_ignores_untracked_response_without_resource_type(self) -> None:
+ async def run() -> None:
+ observation = await _recv_until_dcl_or_binary_main_resource(
+ mock.Mock(),
+ session_id="SID-1",
+ frame_id="FRAME-1",
+ expected_loader_id="LOADER-1",
+ deadline=time.perf_counter() + 1.0,
+ seen=[
+ _document_request_message(),
+ _document_response_message(
+ "application/json",
+ resource_type=None,
+ request_id="SUBRESOURCE-1",
+ status=503,
+ ),
+ _dcl_lifecycle_message(),
],
)
- self.assertIsNone(body)
+ binary_mime_type, response_status, response_mime_type, final_url = observation
+ self.assertIsNone(binary_mime_type)
+ self.assertIsNone(response_status)
+ self.assertIsNone(response_mime_type)
+ self.assertIsNone(final_url)
asyncio.run(run())
+ def test_recv_until_dcl_keeps_status_for_latest_document_request(self) -> None:
+ async def run() -> None:
+ observation = await _recv_until_dcl_or_binary_main_resource(
+ mock.Mock(),
+ session_id="SID-1",
+ frame_id="FRAME-1",
+ expected_loader_id="LOADER-1",
+ deadline=time.perf_counter() + 1.0,
+ seen=[
+ _document_request_message(request_id="OLD"),
+ _document_request_message(request_id="NEW"),
+ _document_response_message(
+ "text/html",
+ request_id="OLD",
+ status=500,
+ ),
+ _document_response_message(
+ "text/html",
+ resource_type=None,
+ request_id="NEW",
+ status=200,
+ ),
+ _dcl_lifecycle_message(),
+ ],
+ )
+ binary_mime_type, response_status, response_mime_type, final_url = observation
+ self.assertIsNone(binary_mime_type)
+ self.assertEqual(response_status, 200)
+ self.assertEqual(response_mime_type, "text/html")
+ self.assertEqual(final_url, "https://example.test/final")
+
+ asyncio.run(run())
+
+ def test_chrome_runner_exposes_main_document_status(self) -> None:
+ process = _FakeProcess()
+
+ def terminate(fake_process: _FakeProcess) -> None:
+ fake_process.returncode = -15
+
+ with (
+ mock.patch("moli_benchmark.chrome_dcl.subprocess.Popen", return_value=process),
+ mock.patch(
+ "moli_benchmark.chrome_dcl._dump_dcl_html",
+ return_value=CdpDclDumpResult(
+ body="<html><body>gateway error</body></html>",
+ response_status=502,
+ response_mime_type="text/html",
+ main_document_body_capture="dom-snapshot",
+ final_url="https://example.test/final",
+ ),
+ ),
+ mock.patch(
+ "moli_benchmark.chrome_dcl._terminate_process_group",
+ side_effect=terminate,
+ ),
+ ):
+ result = run_chrome_dcl_dump(
+ Path("/bin/chromium"),
+ "https://example.test/",
+ timeout_seconds=1.0,
+ sample_resources=False,
+ )
+
+ self.assertEqual(result.returncode, 0)
+ self.assertEqual(result.response_status, 502)
+ self.assertEqual(result.response_mime_type, "text/html")
+ self.assertEqual(result.main_document_body_capture, "dom-snapshot")
+ self.assertEqual(result.final_url, "https://example.test/final")
+ self.assertIn(b"gateway error", result.stdout)
+
+ def test_chrome_runner_classifies_navigation_error_without_timeout(self) -> None:
+ process = _FakeProcess()
+
+ def terminate(fake_process: _FakeProcess) -> None:
+ fake_process.returncode = -15
+
+ with (
+ mock.patch("moli_benchmark.chrome_dcl.subprocess.Popen", return_value=process),
+ mock.patch(
+ "moli_benchmark.chrome_dcl._dump_dcl_html",
+ side_effect=RawCdpError(
+ "Page.navigate failed for `https://missing.example.test/`: "
+ "net::ERR_NAME_NOT_RESOLVED"
+ ),
+ ),
+ mock.patch(
+ "moli_benchmark.chrome_dcl._terminate_process_group",
+ side_effect=terminate,
+ ),
+ ):
+ result = run_chrome_dcl_dump(
+ Path("/bin/chromium"),
+ "https://missing.example.test/",
+ timeout_seconds=1.0,
+ sample_resources=False,
+ )
+
+ self.assertFalse(result.timed_out)
+ self.assertEqual(result.returncode, 1)
+ self.assertIn(b"net::ERR_NAME_NOT_RESOLVED", result.stderr)
+ for classifier in (TOP_SITES_CLASSIFIER, WILD_WEB_CLASSIFIER):
+ with self.subTest(policy=classifier.policy):
+ self.assertEqual(
+ classifier.classify_output(
+ stdout=result.stdout,
+ stderr=result.stderr,
+ returncode=result.returncode,
+ timed_out=result.timed_out,
+ ),
+ "network-error",
+ )
+
+ def test_chrome_runner_does_not_fabricate_binary_response_body(self) -> None:
+ process = _FakeProcess()
+
+ def terminate(fake_process: _FakeProcess) -> None:
+ fake_process.returncode = -15
+
+ with (
+ mock.patch("moli_benchmark.chrome_dcl.subprocess.Popen", return_value=process),
+ mock.patch(
+ "moli_benchmark.chrome_dcl._dump_dcl_html",
+ return_value=CdpDclDumpResult(
+ body="",
+ response_status=200,
+ response_mime_type="application/pdf",
+ main_document_body_capture="response-headers-only",
+ final_url="https://example.test/document.pdf",
+ ),
+ ),
+ mock.patch(
+ "moli_benchmark.chrome_dcl._terminate_process_group",
+ side_effect=terminate,
+ ),
+ ):
+ result = run_chrome_dcl_dump(
+ Path("/bin/chromium"),
+ "https://example.test/document.pdf",
+ timeout_seconds=1.0,
+ sample_resources=False,
+ )
+
+ self.assertEqual(result.returncode, 0)
+ self.assertEqual(result.stdout, b"")
+ self.assertEqual(result.response_mime_type, "application/pdf")
+ self.assertEqual(
+ result.main_document_body_capture,
+ "response-headers-only",
+ )
+
def test_chrome_runner_records_raw_cdp_deadline_as_timeout(self) -> None:
process = _FakeProcess()
diff --git a/moli-benchmark/tests/test_html_report.py b/moli-benchmark/tests/test_html_report.py
index b375987ef..6958e4f67 100644
--- a/moli-benchmark/tests/test_html_report.py
+++ b/moli-benchmark/tests/test_html_report.py
@@ -49,6 +49,13 @@ class HtmlReportTests(unittest.TestCase):
self.assertIn("render-compare/runs.json", paths["render-compare"])
self.assertIn("render-compare/baseline-sites.csv", paths["render-compare"])
+ def test_artifact_index_includes_top_sites_site_outcomes(self) -> None:
+ paths = _artifact_paths_by_suite(
+ [{"suite": "top-sites", "total_failures": 0}]
+ )
+
+ self.assertIn("top-sites/site-outcomes.json", paths["top-sites"])
+
def test_cdp_session_trace_link_uses_nested_summary_fallback(self) -> None:
paths = _artifact_paths_by_suite(
[
@@ -225,6 +232,36 @@ class HtmlReportTests(unittest.TestCase):
self.assertIn("const attempts = passes + failures;", document)
self.assertIn("result.pages ?? result.seeds ?? result.sites ?? result.runs", document)
self.assertIn("result.successes ?? result.categories?.success ?? result.passes", document)
+ self.assertIn("counted pass rate", document)
+ self.assertIn("raw pass rate", document)
+ self.assertIn("successful p50 / p90 / p95 ms", document)
+ self.assertIn("common p50 / p90 / p95 ms", document)
+ self.assertIn("result?.successful_elapsed_ms || result?.elapsed_ms", document)
+ self.assertIn("summary?.common_success?.targets?.[target]", document)
+ self.assertIn("site stability", document)
+ self.assertIn("HTTP status observed", document)
+ self.assertIn("not repeat-validated", document)
+ self.assertIn("Pairwise site outcomes", document)
+ self.assertIn("SINGLE SAMPLE", document)
+
+ def test_chartjs_report_renders_every_aggregate_summary_and_prioritizes_top_sites(
+ self,
+ ) -> None:
+ document = _chartjs_document(
+ {
+ "output_dir": "/tmp/report",
+ "versions": {},
+ "summaries": [
+ {"suite": "wild-web", "targets": {"moli": {}}},
+ {"suite": "top-sites", "targets": {"moli": {}}},
+ ],
+ }
+ )
+
+ self.assertIn("function horizontalSummaries()", document)
+ self.assertIn("'top-sites', 'wild-web'", document)
+ self.assertIn("summaries.map(summary =>", document)
+ self.assertIn('class="comparison-summary"', document)
if __name__ == "__main__":
diff --git a/moli-benchmark/tests/test_public_web.py b/moli-benchmark/tests/test_public_web.py
new file mode 100644
index 000000000..987662e44
--- /dev/null
+++ b/moli-benchmark/tests/test_public_web.py
@@ -0,0 +1,188 @@
+from __future__ import annotations
+
+import unittest
+
+from moli_benchmark.process import ProcessResult
+from moli_benchmark.public_web import (
+ POST_DCL_SETTLE_MILLISECONDS,
+ POST_DCL_WAIT_SCRIPT,
+ PUBLIC_WEB_SNAPSHOT_CONTRACT,
+ TOP_SITES_ARTIFACT_POLICY,
+ TOP_SITES_CLASSIFIER,
+ WILD_WEB_CLASSIFIER,
+ PublicWebAttempt,
+ PublicWebResult,
+ PublicWebScheduler,
+ resolve_main_document_status,
+ schedule_public_web_cases,
+)
+
+
+class PublicWebTests(unittest.TestCase):
+ def test_post_dcl_wait_script_encodes_the_reported_snapshot_contract(self) -> None:
+ self.assertEqual(POST_DCL_SETTLE_MILLISECONDS, 50)
+ self.assertIn("setTimeout", POST_DCL_WAIT_SCRIPT)
+ self.assertIn("}, 50)", POST_DCL_WAIT_SCRIPT)
+ self.assertIn("PostDclReady", POST_DCL_WAIT_SCRIPT)
+ self.assertIn("50 ms", PUBLIC_WEB_SNAPSHOT_CONTRACT)
+
+ def test_typed_cli_fetch_readiness_timeout_is_shared_by_both_suites(self) -> None:
+ stderr = b"""
+Error: failed to fetch `https://example.test/`
+Reason: fetch readiness timed out after 30000 ms while waiting for DOMContentLoaded
+"""
+
+ for classifier in (TOP_SITES_CLASSIFIER, WILD_WEB_CLASSIFIER):
+ with self.subTest(policy=classifier.policy):
+ self.assertEqual(
+ classifier.classify_output(
+ stdout=b"",
+ stderr=stderr,
+ returncode=1,
+ timed_out=False,
+ ),
+ "timeout",
+ )
+
+ def test_protocol_status_is_preferred(self) -> None:
+ self.assertEqual(
+ resolve_main_document_status(
+ 200,
+ b"lifecycle target document `https://example.test/` returned 502 Bad Gateway",
+ 1,
+ ),
+ (200, "protocol"),
+ )
+
+ def test_nonzero_cli_diagnostic_supplies_main_document_status(self) -> None:
+ self.assertEqual(
+ resolve_main_document_status(
+ None,
+ b"Reason: lifecycle target document `https://example.test/` "
+ b"returned 429 Too Many Requests",
+ 1,
+ ),
+ (429, "cli-diagnostic"),
+ )
+ self.assertEqual(
+ resolve_main_document_status(
+ None,
+ b"HTTP request `https://example.test/` returned 404 Not Found",
+ 1,
+ ),
+ (404, "cli-diagnostic"),
+ )
+
+ def test_success_and_subresource_diagnostics_are_not_promoted(self) -> None:
+ diagnostic = b"script subresource `https://cdn.example.test/a.js` returned 500"
+ self.assertEqual(resolve_main_document_status(None, diagnostic, 1), (None, None))
+ self.assertEqual(
+ resolve_main_document_status(
+ None,
+ b"lifecycle target document `https://example.test/` returned 500",
+ 0,
+ ),
+ (None, None),
+ )
+
+ def test_attempt_result_owns_common_evidence_fields(self) -> None:
+ attempt = PublicWebAttempt.start(
+ target="chrome",
+ metadata={"engine": "chrome", "driver": "cdp-dcl"},
+ target_info={"available": True, "path": "/bin/chromium"},
+ run=2,
+ case_fields={"seed": "example"},
+ url="https://example.test/",
+ schedule_index=3,
+ target_order_index=2,
+ artifact_stem="chrome-run-2-example",
+ )
+ process = ProcessResult(
+ command=["chromium"],
+ returncode=0,
+ elapsed_ms=12.5,
+ stdout=b"<title>Example</title><body>Hello world</body>",
+ stderr=b"",
+ timed_out=False,
+ resources={"peak_pss_bytes": 123},
+ response_status=200,
+ final_url="https://example.test/final",
+ )
+
+ result = PublicWebResult.capture(
+ attempt,
+ process,
+ classifier=TOP_SITES_CLASSIFIER,
+ )
+ row = {
+ **result.base_row(),
+ **result.evidence_fields(policy=TOP_SITES_ARTIFACT_POLICY),
+ }
+
+ self.assertEqual(row["seed"], "example")
+ self.assertEqual(row["response_status_source"], "protocol")
+ self.assertEqual(row["final_url"], "https://example.test/final")
+ self.assertEqual(row["title"], "Example")
+ self.assertEqual(row["peak_pss_bytes"], 123)
+ self.assertEqual(len(row["stdout_sha256"]), 64)
+
+ def test_scheduler_rotates_targets_and_restores_schedule_order(self) -> None:
+ scheduled = schedule_public_web_cases(("first", "second"), runs=1)
+ scheduler = PublicWebScheduler[str, tuple[int, str, int]](
+ ("moli", "lightpanda", "chrome"),
+ parallelism=2,
+ )
+
+ results = scheduler.run(
+ scheduled,
+ lambda case, target, target_order_index: (
+ case.schedule_index,
+ target,
+ target_order_index,
+ ),
+ )
+
+ self.assertEqual(
+ results,
+ [
+ (0, "moli", 1),
+ (0, "lightpanda", 2),
+ (0, "chrome", 3),
+ (1, "lightpanda", 1),
+ (1, "chrome", 2),
+ (1, "moli", 3),
+ ],
+ )
+
+ def test_scheduler_changes_each_site_first_target_across_runs(self) -> None:
+ cases = tuple(f"site-{index}" for index in range(20))
+ scheduled = schedule_public_web_cases(cases, runs=3)
+ scheduler = PublicWebScheduler[str, tuple[str, int, str, int]](
+ ("moli", "chrome"),
+ )
+
+ results = scheduler.run(
+ scheduled,
+ lambda case, target, target_order_index: (
+ case.case,
+ case.run,
+ target,
+ target_order_index,
+ ),
+ )
+ first_targets = {
+ (case, run): target
+ for case, run, target, target_order_index in results
+ if target_order_index == 1
+ }
+
+ for case_index, case in enumerate(cases):
+ expected = (
+ ("moli", "chrome", "moli")
+ if case_index % 2 == 0
+ else ("chrome", "moli", "chrome")
+ )
+ self.assertEqual(
+ tuple(first_targets[(case, run)] for run in range(1, 4)),
+ expected,
+ )
diff --git a/moli-benchmark/tests/test_top_sites.py b/moli-benchmark/tests/test_top_sites.py
index 1080412f6..a87fb8ab1 100644
--- a/moli-benchmark/tests/test_top_sites.py
+++ b/moli-benchmark/tests/test_top_sites.py
@@ -7,15 +7,22 @@ from pathlib import Path
from unittest.mock import patch
from moli_benchmark.process import ProcessResult
+from moli_benchmark.public_web import POST_DCL_WAIT_SCRIPT
from moli_benchmark.top_sites import (
COMPOSITE_TOP_SITES_SOURCES,
DEFAULT_TOP_SITES_SOURCE,
+ TOP_SITES_FIXTURE_ROOT,
TOP_SITES_PROFILES,
TOP_SITES_SOURCES,
+ _build_site_outcomes,
_classify,
_default_top_sites_parallelism,
_elapsed_failure_reached_timeout,
+ _extract_title_and_text,
_ok_categories,
+ _pairwise_site_comparisons,
+ _rotated_target_order,
+ _site_unreachable_exclusions,
_top_command_for_target,
_top_sites_target_metadata,
load_top_sites_entries,
@@ -26,6 +33,163 @@ from moli_benchmark.top_sites import (
class TopSitesTests(unittest.TestCase):
+ def test_target_order_rotates_per_site_group(self) -> None:
+ targets = ("moli", "lightpanda", "chrome")
+
+ self.assertEqual(_rotated_target_order(targets, 0), targets)
+ self.assertEqual(
+ _rotated_target_order(targets, 1),
+ ("lightpanda", "chrome", "moli"),
+ )
+ self.assertEqual(
+ _rotated_target_order(targets, 2),
+ ("chrome", "moli", "lightpanda"),
+ )
+
+ def test_site_unreachable_exclusion_requires_transport_failure_from_every_target(
+ self,
+ ) -> None:
+ rows = [
+ {
+ "target": "moli",
+ "domain": "dead.example",
+ "ok": False,
+ "category": "network-error",
+ "failure_kind": "network-error",
+ },
+ {
+ "target": "chrome",
+ "domain": "dead.example",
+ "ok": False,
+ "category": "process-error",
+ "failure_kind": "process-error",
+ },
+ ]
+
+ self.assertEqual(
+ _site_unreachable_exclusions(rows, ("moli", "chrome")),
+ {},
+ )
+ rows[1]["category"] = "timeout"
+ rows[1]["failure_kind"] = "timeout"
+ self.assertEqual(
+ _site_unreachable_exclusions(rows, ("moli", "chrome")),
+ {"dead.example": "site-unreachable"},
+ )
+
+ def test_site_outcomes_keep_header_only_binary_observations_neutral(self) -> None:
+ outcomes = _build_site_outcomes(
+ [
+ {
+ "target": "chrome",
+ "rank": 1,
+ "domain": "document.example",
+ "url": "https://document.example/file.pdf",
+ "run": 1,
+ "ok": False,
+ "comparable": False,
+ "category": "binary-response-headers",
+ "response_status": 200,
+ }
+ ]
+ )
+
+ self.assertEqual(outcomes[0]["outcome"], "not-comparable")
+ self.assertEqual(outcomes[0]["attempts"], 0)
+ self.assertEqual(outcomes[0]["raw_attempts"], 1)
+ self.assertEqual(outcomes[0]["non_comparable_attempts"], 1)
+ self.assertIsNone(outcomes[0]["pass_rate_percent"])
+
+ def test_site_outcomes_separate_repeat_stability_from_attempt_passes(self) -> None:
+ rows: list[dict[str, object]] = []
+ for run, left_ok, right_ok in (
+ (1, True, False),
+ (2, True, False),
+ (3, True, False),
+ ):
+ rows.extend(
+ [
+ {
+ "target": "left",
+ "rank": 1,
+ "domain": "stable.example",
+ "url": "https://stable.example/",
+ "run": run,
+ "ok": left_ok,
+ "category": "success-content",
+ "response_status": 200,
+ },
+ {
+ "target": "right",
+ "rank": 1,
+ "domain": "stable.example",
+ "url": "https://stable.example/",
+ "run": run,
+ "ok": right_ok,
+ "category": "http-error",
+ "response_status": 500,
+ },
+ {
+ "target": "left",
+ "rank": 2,
+ "domain": "flaky.example",
+ "url": "https://flaky.example/",
+ "run": run,
+ "ok": run != 2,
+ "category": "success-content" if run != 2 else "timeout",
+ "response_status": 200 if run != 2 else None,
+ },
+ {
+ "target": "right",
+ "rank": 2,
+ "domain": "flaky.example",
+ "url": "https://flaky.example/",
+ "run": run,
+ "ok": True,
+ "category": "success-content",
+ "response_status": 200,
+ },
+ ]
+ )
+
+ outcomes = _build_site_outcomes(rows) # type: ignore[arg-type]
+ comparisons = _pairwise_site_comparisons(
+ outcomes,
+ ("left", "right"),
+ runs=3,
+ )
+ by_target_rank = {
+ (row["target"], row["rank"]): row for row in outcomes
+ }
+
+ self.assertEqual(by_target_rank[("left", 1)]["outcome"], "all-pass")
+ self.assertEqual(by_target_rank[("right", 1)]["outcome"], "all-fail")
+ self.assertEqual(by_target_rank[("left", 2)]["outcome"], "flaky")
+ self.assertAlmostEqual(
+ by_target_rank[("left", 2)]["pass_rate_percent"],
+ 200 / 3,
+ )
+ self.assertTrue(comparisons[0]["repeat_validated"])
+ self.assertEqual(comparisons[0]["left_all_pass_right_all_fail"], 1)
+ self.assertEqual(comparisons[0]["flaky_or_incomplete"], 1)
+ self.assertEqual(
+ comparisons[0]["left_only_sites"],
+ [{"rank": 1, "domain": "stable.example"}],
+ )
+
+ def test_builtin_seed_lists_are_benchmark_local(self) -> None:
+ for source in TOP_SITES_SOURCES:
+ resolved, path = resolve_top_sites_source(source, None)
+ self.assertEqual(resolved, source)
+ self.assertTrue(path.is_relative_to(TOP_SITES_FIXTURE_ROOT))
+ self.assertEqual(path.suffix, ".csv")
+ self.assertTrue(path.is_file(), f"missing benchmark seed fixture: {path}")
+
+ for source in COMPOSITE_TOP_SITES_SOURCES:
+ resolved, path = resolve_top_sites_source(source, None)
+ self.assertEqual(resolved, source)
+ self.assertEqual(path.parent, TOP_SITES_FIXTURE_ROOT)
+
def test_parse_top_sites_list_reads_top_100_section(self) -> None:
with tempfile.TemporaryDirectory() as temp_dir:
path = Path(temp_dir) / "list.md"
@@ -56,6 +220,21 @@ class TopSitesTests(unittest.TestCase):
entries = parse_top_sites_list(path)
self.assertEqual(entries, [(1, "first.example")])
+ def test_parse_top_sites_list_reads_csv(self) -> None:
+ with tempfile.TemporaryDirectory() as temp_dir:
+ path = Path(temp_dir) / "list.csv"
+ path.write_text(
+ "rank,target\n"
+ "1,example.com\n"
+ '2,"https://example.com/search?q=one,two"\n',
+ encoding="utf-8",
+ )
+ entries = parse_top_sites_list(path)
+ self.assertEqual(
+ entries,
+ [(1, "example.com"), (2, "https://example.com/search?q=one,two")],
+ )
+
def test_legacy_encoding_source_is_registered_and_parses(self) -> None:
source, path = resolve_top_sites_source("legacy-encoding", None)
entries = parse_top_sites_list(path)
@@ -75,6 +254,7 @@ class TopSitesTests(unittest.TestCase):
self.assertNotIn("--resource", command)
self.assertIn("--wait-until", command)
self.assertEqual(command[command.index("--wait-until") + 1], "domcontentloaded")
+ self.assertEqual(command[command.index("--wait-script") + 1], POST_DCL_WAIT_SCRIPT)
self.assertEqual(command[command.index("--timeout") + 1], "12000")
self.assertEqual(command[command.index("--http-timeout") + 1], "12000")
self.assertEqual(command[-1], "https://example.test/")
@@ -87,13 +267,29 @@ class TopSitesTests(unittest.TestCase):
self.assertLess(command.index("--layout"), command.index("--dump"))
self.assertLess(command.index("--resource"), command.index("--dump"))
self.assertEqual(command[command.index("--wait-until") + 1], "domcontentloaded")
+ self.assertEqual(command[command.index("--wait-script") + 1], POST_DCL_WAIT_SCRIPT)
self.assertEqual(command[-1], "https://example.test/")
+ def test_top_command_for_moli_omits_page_waits_for_explicit_pdf(self) -> None:
+ command = _top_command_for_target(
+ "moli",
+ Path("/bin/moli"),
+ "https://example.test/report.PDF?download=1",
+ 12.0,
+ )
+
+ self.assertNotIn("--wait-until", command)
+ self.assertNotIn("--wait-script", command)
+ self.assertEqual(command[command.index("--timeout") + 1], "12000")
+ self.assertEqual(command[command.index("--http-timeout") + 1], "12000")
+ self.assertEqual(command[-1], "https://example.test/report.PDF?download=1")
+
def test_top_command_for_lightpanda_aligns_wait_and_http_timeouts(self) -> None:
command = _top_command_for_target("lightpanda", Path("/bin/lightpanda"), "https://example.test/", 12.0)
self.assertEqual(command[0], "/bin/lightpanda")
self.assertIn("--wait-until", command)
self.assertEqual(command[command.index("--wait-until") + 1], "domcontentloaded")
+ self.assertEqual(command[command.index("--wait-script") + 1], POST_DCL_WAIT_SCRIPT)
self.assertEqual(command[command.index("--wait-ms") + 1], "12000")
self.assertEqual(command[command.index("--http-timeout") + 1], "12000")
self.assertEqual(command[command.index("--terminate-ms") + 1], "12000")
@@ -244,6 +440,154 @@ class TopSitesTests(unittest.TestCase):
self.assertEqual(_classify(b"", b"HTTP request `https://example.test/` returned 404 Not Found", 1, False, 256), "not-found")
self.assertEqual(_classify(b"", b"", None, True, 256), "timeout")
+ def test_classify_recognizes_lightpanda_fatal_fetch_timeout(self) -> None:
+ stderr = (
+ b'$scope=app $level=fatal $msg="fetch error" err=Timeout '
+ b"url=https://example.test/data.json"
+ )
+
+ self.assertEqual(_classify(b"", stderr, 0, False, 256), "timeout")
+
+ def test_classify_preserves_cli_main_document_status_and_dns_reason(self) -> None:
+ self.assertEqual(
+ _classify(
+ b"",
+ b"Reason: lifecycle target document `https://example.test/` "
+ b"returned 429 Too Many Requests",
+ 1,
+ False,
+ 256,
+ ),
+ "http-error",
+ )
+ self.assertEqual(
+ _classify(
+ b"",
+ b"Reason: lifecycle target document `https://example.test/` "
+ b"returned 401 Unauthorized",
+ 1,
+ False,
+ 256,
+ ),
+ "blocked-or-forbidden",
+ )
+ self.assertEqual(
+ _classify(
+ b"",
+ b"Reason: failed to resolve example.test:443: failed to lookup "
+ b"address information",
+ 1,
+ False,
+ 256,
+ ),
+ "network-error",
+ )
+
+ def test_classify_uses_main_document_status_before_body_size(self) -> None:
+ large_error_body = (
+ b"<html><head><title>upstream response</title></head><body>"
+ + b"error details " * 100
+ + b"</body></html>"
+ )
+
+ for status in (400, 500, 502):
+ with self.subTest(status=status):
+ self.assertEqual(
+ _classify(
+ large_error_body,
+ b"",
+ 0,
+ False,
+ 256,
+ response_status=status,
+ ),
+ "http-error",
+ )
+ self.assertEqual(
+ _classify(
+ large_error_body,
+ b"",
+ 0,
+ False,
+ 256,
+ response_status=403,
+ ),
+ "blocked-or-forbidden",
+ )
+ self.assertEqual(
+ _classify(
+ large_error_body,
+ b"",
+ 0,
+ False,
+ 256,
+ response_status=404,
+ ),
+ "not-found",
+ )
+
+ def test_classify_infers_http_error_documents_when_cli_hides_status(self) -> None:
+ cases = [
+ (
+ b"<html><head><title>Error: 500 | Service</title></head>"
+ b"<body>There was a technical error.</body></html>",
+ "http-error",
+ ),
+ (
+ b"<html><head><title>502 Bad Gateway</title></head>"
+ b"<body>Origin unavailable.</body></html>",
+ "http-error",
+ ),
+ (
+ b"<html><head><title></title></head><body><pre>"
+ b'{"error":{"message":"invalid query"}}'
+ b"</pre></body></html>",
+ "http-error",
+ ),
+ (
+ b"<html><head><title>ResearchGate - Temporarily Unavailable</title></head>"
+ b"<body>Access restricted due to unusual activity.</body></html>",
+ "blocked-or-forbidden",
+ ),
+ ]
+
+ for html, expected in cases:
+ with self.subTest(expected=expected, html=html):
+ self.assertEqual(_classify(html, b"", 0, False, 256), expected)
+
+ def test_classify_distinguishes_tls_verification_from_human_verification(self) -> None:
+ tls_error = (
+ b"<html><body><h1>Navigation failed</h1>"
+ b"<p>Reason: PeerFailedVerification</p></body></html>"
+ )
+ human_verification = (
+ "<html><body>环境异常 当前环境异常,完成验证后即可继续访问。去验证</body></html>".encode()
+ )
+
+ self.assertEqual(_classify(tls_error, b"", 0, False, 256), "network-error")
+ self.assertEqual(
+ _classify(human_verification, b"", 0, False, 256),
+ "captcha-or-verification",
+ )
+
+ def test_classify_recognizes_chinese_not_found_document(self) -> None:
+ html = (
+ "<html><head><title>出错了!</title></head>"
+ "<body>404 很抱歉,你访问的页面不存在或暂时无法访问!</body></html>"
+ ).encode()
+
+ self.assertEqual(_classify(html, b"", 0, False, 256), "not-found")
+
+ def test_visible_text_excludes_document_head(self) -> None:
+ snapshot = _extract_title_and_text(
+ b"<html><head><title>A very long title</title></head>"
+ b"<body>short body</body></html>"
+ )
+
+ self.assertEqual(snapshot["title"], "A very long title")
+ self.assertEqual(snapshot["text_sample"], "short body")
+ self.assertEqual(snapshot["text_length"], len("short body"))
+
def test_classify_treats_browser_network_error_pages_as_failures(self) -> None:
html = (
b"<html><head><title>www.example.test</title></head><body>"
@@ -316,6 +660,21 @@ class TopSitesTests(unittest.TestCase):
stderr = b'$scope=frame $level=error $msg="navigate failed" err=OperationTimedout type=root'
self.assertEqual(_classify(html, stderr, 0, False, 256), "timeout")
+ def test_classify_treats_lightpanda_navigation_transport_pages_as_network_errors(
+ self,
+ ) -> None:
+ for reason in ("CouldntResolveHost", "Http2Stream"):
+ with self.subTest(reason=reason):
+ html = (
+ "<!doctype html><body><h1>Navigation failed</h1>"
+ f"<p>Reason: {reason}</p></body>"
+ ).encode()
+ stderr = f'$msg="navigate failed" err={reason} type=root'.encode()
+ self.assertEqual(
+ _classify(html, stderr, 0, False, 256),
+ "network-error",
+ )
+
def test_classify_ignores_lightpanda_subresource_operation_timeout_when_main_content_loaded(
self,
) -> None:
@@ -398,6 +757,28 @@ class TopSitesTests(unittest.TestCase):
self.assertEqual(_classify(vercel_checkpoint, b"", 0, False, 256), "js-challenge")
self.assertEqual(_classify(bot_or_not, b"", 0, False, 256), "captcha-or-verification")
+ def test_classify_recognizes_safeline_and_privacy_tile_challenges(self) -> None:
+ safeline = (
+ b"<html><body>Security Detection Powered By SafeLine WAF "
+ + b"Confirm " * 100
+ + b"</body></html>"
+ )
+ privacy_tiles = (
+ b"<html><body>Select the 2 matching tiles A B C D Submit "
+ b"Powered and protected by Privacy "
+ + b"challenge " * 100
+ + b"</body></html>"
+ )
+
+ self.assertEqual(
+ _classify(safeline, b"", 0, False, 256),
+ "blocked-or-forbidden",
+ )
+ self.assertEqual(
+ _classify(privacy_tiles, b"", 0, False, 256),
+ "captcha-or-verification",
+ )
+
def test_classify_treats_c2wf_probe_shell_as_js_challenge(self) -> None:
probe_v1 = b"""<!DOCTYPE html><html><head>
<meta charset="UTF-8">
@@ -490,7 +871,7 @@ if(window.EOJsChallengeSDK){new window.EOJsChallengeSDK({callback:function(){}})
self.assertEqual(_classify(pdf, b"", 0, False, 256), "success-binary-content")
self.assertIn("success-binary-content", _ok_categories())
- def test_classify_treats_raw_binary_body_timeout_as_binary_main_resource(
+ def test_classify_keeps_partial_raw_binary_body_timeout_as_network_error(
self,
) -> None:
stderr = (
@@ -503,9 +884,24 @@ if(window.EOJsChallengeSDK){new window.EOJsChallengeSDK({callback:function(){}})
)
self.assertEqual(
_classify(b"", stderr, 1, False, 256),
- "success-binary-main-resource",
+ "network-error",
+ )
+ self.assertNotIn("success-binary-main-resource", _ok_categories())
+
+ def test_classify_marks_cdp_binary_headers_as_not_a_body_success(self) -> None:
+ self.assertEqual(
+ _classify(
+ b"",
+ b"",
+ 0,
+ False,
+ 256,
+ response_status=200,
+ main_document_body_capture="response-headers-only",
+ ),
+ "binary-response-headers",
)
- self.assertIn("success-binary-main-resource", _ok_categories())
+ self.assertNotIn("binary-response-headers", _ok_categories())
def test_classify_keeps_raw_binary_header_timeout_as_network_error(self) -> None:
stderr = (
@@ -565,20 +961,230 @@ if(window.EOJsChallengeSDK){new window.EOJsChallengeSDK({callback:function(){}})
failure_files = sorted(p.name for p in (output_dir / "top-sites" / "failures").iterdir())
summary_payload = (output_dir / "top-sites" / "summary.json").read_text(encoding="utf-8")
run_rows = json.loads((output_dir / "top-sites" / "runs.json").read_text(encoding="utf-8"))
+ site_outcomes = json.loads(
+ (output_dir / "top-sites" / "site-outcomes.json").read_text(
+ encoding="utf-8"
+ )
+ )
self.assertEqual(summary["site_count"], 2)
self.assertEqual(summary["gate_failures"], 1)
self.assertEqual(summary["targets"]["moli"]["passes"], 1)
self.assertEqual(summary["targets"]["moli"]["failures"], 1)
+ self.assertEqual(summary["targets"]["moli"]["raw_observations"], 2)
+ self.assertEqual(summary["targets"]["moli"]["raw_comparable_attempts"], 2)
+ self.assertEqual(summary["targets"]["moli"]["raw_passes"], 1)
+ self.assertEqual(summary["targets"]["moli"]["raw_failures"], 1)
+ self.assertEqual(summary["targets"]["moli"]["raw_pass_rate_percent"], 50.0)
+ self.assertEqual(summary["targets"]["moli"]["pass_rate_percent"], 50.0)
+ self.assertEqual(summary["targets"]["moli"]["elapsed_ms"]["count"], 2)
+ self.assertEqual(
+ summary["targets"]["moli"]["successful_elapsed_ms"]["count"],
+ 1,
+ )
+ self.assertEqual(summary["common_success"]["attempts"], 1)
+ self.assertEqual(
+ summary["common_success"]["targets"]["moli"]["elapsed_ms"]["count"],
+ 1,
+ )
+ self.assertEqual(summary["targets"]["moli"]["unique_sites"], 2)
+ self.assertEqual(summary["targets"]["moli"]["all_pass_sites"], 1)
+ self.assertEqual(summary["targets"]["moli"]["all_fail_sites"], 1)
+ self.assertEqual(summary["targets"]["moli"]["flaky_sites"], 0)
+ self.assertFalse(summary["repeat_validated"])
self.assertEqual(summary["targets"]["moli"]["failure_kinds"], {"process-error": 1})
self.assertEqual(summary["targets"]["moli"]["categories"]["success-content"], 1)
self.assertEqual(summary["targets"]["moli"]["peak_rss_bytes"]["max"], 2048.0)
self.assertEqual(summary["chrome_parallelism"], 2)
+ self.assertEqual(summary["schedule"], "site-paired-rotating-target-order")
+ self.assertEqual(summary["scheduled_site_groups"], 2)
self.assertEqual(run_rows[0]["command"][0], "/bin/echo")
self.assertEqual(run_rows[0]["peak_rss_bytes"], 2048)
+ self.assertEqual(run_rows[0]["schedule_index"], 0)
+ self.assertEqual(run_rows[0]["target_order_index"], 1)
+ self.assertTrue(run_rows[0]["started_at"].endswith("Z"))
+ self.assertTrue(run_rows[0]["finished_at"].endswith("Z"))
+ self.assertEqual(len(run_rows[0]["stdout_sha256"]), 64)
+ self.assertEqual(len(run_rows[0]["stderr_sha256"]), 64)
+ self.assertTrue(run_rows[0]["stdout_head"].startswith("<html>"))
+ self.assertEqual(len(site_outcomes), 2)
+ self.assertEqual(
+ {row["outcome"] for row in site_outcomes},
+ {"all-pass", "all-fail"},
+ )
self.assertIn("moli-run-1-rank002-fail.example.json", failure_files)
self.assertIn("\"profile\": \"quick\"", summary_payload)
+ def test_run_top_sites_suite_records_chrome_http_error_status(self) -> None:
+ list_md = "## Top 100\n\n1. `error.example` — synthetic HTTP error row\n"
+
+ def fake_chrome_run(*args: object, **kwargs: object) -> ProcessResult:
+ return ProcessResult(
+ command=["/bin/chromium", "https://error.example/"],
+ returncode=0,
+ elapsed_ms=995.0,
+ stdout=(
+ b"<html><head><title>upstream response</title></head><body>"
+ + b"error details " * 100
+ + b"</body></html>"
+ ),
+ stderr=b"",
+ timed_out=False,
+ resources={},
+ response_status=502,
+ )
+
+ with tempfile.TemporaryDirectory() as temp_dir:
+ output_dir = Path(temp_dir)
+ list_path = output_dir / "top.md"
+ list_path.write_text(list_md, encoding="utf-8")
+ with patch(
+ "moli_benchmark.top_sites.run_chrome_dcl_dump",
+ side_effect=fake_chrome_run,
+ ):
+ summary = run_top_sites_suite(
+ output_dir=output_dir,
+ target_matrix={"chrome": {"available": True, "path": "/bin/chromium"}},
+ targets=("chrome",),
+ profile="quick",
+ list_path=list_path,
+ runs=1,
+ timeout_seconds=1.0,
+ gate_target="chrome",
+ parallelism=1,
+ chrome_parallelism=1,
+ limit_override=1,
+ )
+ rows = json.loads(
+ (output_dir / "top-sites" / "runs.json").read_text(encoding="utf-8")
+ )
+
+ self.assertEqual(rows[0]["response_status"], 502)
+ self.assertEqual(rows[0]["category"], "http-error")
+ self.assertEqual(rows[0]["response_status_source"], "protocol")
+ self.assertEqual(rows[0]["classification_basis"], "protocol-http-status")
+ self.assertFalse(rows[0]["ok"])
+ self.assertEqual(summary["targets"]["chrome"]["failure_kinds"], {"http-error": 1})
+
+ def test_run_top_sites_suite_excludes_header_only_binary_from_denominator(
+ self,
+ ) -> None:
+ list_md = (
+ "## Top 100\n\n"
+ "1. `https://document.example/file.pdf` — synthetic PDF row\n"
+ )
+
+ def fake_chrome_run(*args: object, **kwargs: object) -> ProcessResult:
+ return ProcessResult(
+ command=["/bin/chromium", "https://document.example/file.pdf"],
+ returncode=0,
+ elapsed_ms=50.0,
+ stdout=b"",
+ stderr=b"",
+ timed_out=False,
+ resources={},
+ response_status=200,
+ response_mime_type="application/pdf",
+ main_document_body_capture="response-headers-only",
+ )
+
+ with tempfile.TemporaryDirectory() as temp_dir:
+ output_dir = Path(temp_dir)
+ list_path = output_dir / "top.md"
+ list_path.write_text(list_md, encoding="utf-8")
+ with patch(
+ "moli_benchmark.top_sites.run_chrome_dcl_dump",
+ side_effect=fake_chrome_run,
+ ):
+ summary = run_top_sites_suite(
+ output_dir=output_dir,
+ target_matrix={
+ "chrome": {"available": True, "path": "/bin/chromium"}
+ },
+ targets=("chrome",),
+ profile="quick",
+ list_path=list_path,
+ runs=1,
+ timeout_seconds=1.0,
+ gate_target="chrome",
+ parallelism=1,
+ chrome_parallelism=1,
+ limit_override=1,
+ )
+ rows = json.loads(
+ (output_dir / "top-sites" / "runs.json").read_text(
+ encoding="utf-8"
+ )
+ )
+
+ self.assertEqual(rows[0]["category"], "binary-response-headers")
+ self.assertFalse(rows[0]["comparable"])
+ self.assertNotIn("failure_artifact", rows[0])
+ self.assertEqual(summary["gate_failures"], 0)
+ self.assertEqual(summary["targets"]["chrome"]["attempts"], 0)
+ self.assertEqual(summary["targets"]["chrome"]["raw_comparable_attempts"], 0)
+ self.assertEqual(summary["targets"]["chrome"]["raw_non_comparable_attempts"], 1)
+ self.assertEqual(
+ summary["targets"]["chrome"]["non_comparable_attempts"],
+ 1,
+ )
+ self.assertEqual(summary["targets"]["chrome"]["failures"], 0)
+ self.assertEqual(summary["common_success"]["attempts"], 0)
+
+ def test_run_top_sites_suite_records_cli_diagnostic_http_status(self) -> None:
+ list_md = "## Top 100\n\n1. `error.example` — synthetic CLI HTTP error row\n"
+
+ def fake_run(command: list[str], **kwargs: object) -> ProcessResult:
+ return ProcessResult(
+ command=command,
+ returncode=1,
+ elapsed_ms=12.0,
+ stdout=b"",
+ stderr=(
+ b"Error: failed to fetch `https://error.example/`\n"
+ b"Reason: lifecycle target document `https://error.example/` "
+ b"returned 502 Bad Gateway"
+ ),
+ timed_out=False,
+ resources={},
+ )
+
+ with tempfile.TemporaryDirectory() as temp_dir:
+ output_dir = Path(temp_dir)
+ list_path = output_dir / "top.md"
+ list_path.write_text(list_md, encoding="utf-8")
+ with patch("moli_benchmark.top_sites.run_process", fake_run):
+ summary = run_top_sites_suite(
+ output_dir=output_dir,
+ target_matrix={"moli": {"available": True, "path": "/bin/moli"}},
+ targets=("moli",),
+ profile="quick",
+ list_path=list_path,
+ runs=1,
+ timeout_seconds=1.0,
+ gate_target="moli",
+ parallelism=1,
+ limit_override=1,
+ )
+ rows = json.loads(
+ (output_dir / "top-sites" / "runs.json").read_text(
+ encoding="utf-8"
+ )
+ )
+
+ self.assertEqual(rows[0]["response_status"], 502)
+ self.assertEqual(rows[0]["response_status_source"], "cli-diagnostic")
+ self.assertEqual(rows[0]["category"], "http-error")
+ self.assertEqual(
+ rows[0]["classification_basis"],
+ "cli-diagnostic-http-status",
+ )
+ self.assertEqual(summary["targets"]["moli"]["status_observed_attempts"], 1)
+ self.assertEqual(
+ summary["targets"]["moli"]["response_status_sources"],
+ {"cli-diagnostic": 1},
+ )
+
def test_run_top_sites_suite_promotes_deadline_adjacent_failures_to_timeout(
self,
) -> None:
@@ -686,6 +1292,11 @@ if(window.EOJsChallengeSDK){new window.EOJsChallengeSDK({callback:function(){}})
limit_override=3,
)
raw_runs = (output_dir / "top-sites" / "raw-runs.csv").read_text(encoding="utf-8")
+ run_rows = json.loads(
+ (output_dir / "top-sites" / "runs.json").read_text(
+ encoding="utf-8"
+ )
+ )
self.assertEqual(summary["site_count"], 3)
self.assertEqual(summary["counted_site_count"], 2)
@@ -698,6 +1309,14 @@ if(window.EOJsChallengeSDK){new window.EOJsChallengeSDK({callback:function(){}})
self.assertEqual(summary["targets"]["moli"]["passes"], 1)
self.assertEqual(summary["targets"]["moli"]["failures"], 1)
self.assertIn("site-unreachable", raw_runs)
+ order_by_target_rank = {
+ (row["target"], row["rank"]): row["target_order_index"]
+ for row in run_rows
+ }
+ self.assertEqual(order_by_target_rank[("moli", 1)], 1)
+ self.assertEqual(order_by_target_rank[("chrome", 1)], 2)
+ self.assertEqual(order_by_target_rank[("chrome", 2)], 1)
+ self.assertEqual(order_by_target_rank[("moli", 2)], 2)
def test_run_top_sites_suite_keeps_single_target_unreachable_sites_counted(self) -> None:
list_md = "## Top 100\n\n1. `dead.example` — synthetic unreachable row\n"
@@ -812,7 +1431,7 @@ if(window.EOJsChallengeSDK){new window.EOJsChallengeSDK({callback:function(){}})
self.assertEqual(entries[19][0], 20)
self.assertEqual(entries[-1][0], 100)
self.assertIn("www.huxiu.com", domains)
- self.assertEqual(labels, ["chinese-community:chinese-community-top100-websites.md"])
+ self.assertEqual(labels, ["chinese-community:chinese-community-top100-websites.csv"])
def test_global_source_returns_english_world_sites(self) -> None:
resolved, _ = resolve_top_sites_source("global", None)
@@ -823,7 +1442,7 @@ if(window.EOJsChallengeSDK){new window.EOJsChallengeSDK({callback:function(){}})
self.assertIn("github.com", domains)
self.assertIn("wikipedia.org", domains)
self.assertNotIn("baidu.com", domains, "global list must not contain Chinese top sites")
- self.assertEqual(labels, ["global:global-top-websites-seed-list.md"])
+ self.assertEqual(labels, ["global:global-top-websites-seed-list.csv"])
def test_mixed_source_interleaves_cn_and_global(self) -> None:
resolved, list_source = resolve_top_sites_source("mixed", None)
@@ -848,7 +1467,7 @@ if(window.EOJsChallengeSDK){new window.EOJsChallengeSDK({callback:function(){}})
urls,
)
self.assertNotIn("https://github.com/golang/go/blob/master/doc/effective_go.md", urls)
- self.assertEqual(labels, ["webfetch-longtail:webfetch-longtail-seed-list.md"])
+ self.assertEqual(labels, ["webfetch-longtail:webfetch-longtail-seed-list.csv"])
def test_render_quality_source_returns_curated_url_paths(self) -> None:
resolved, _ = resolve_top_sites_source("render-quality", None)
@@ -860,7 +1479,7 @@ if(window.EOJsChallengeSDK){new window.EOJsChallengeSDK({callback:function(){}})
"https://www.feishu.cn/hc/zh-CN/articles/485770964672-管理者查看企业-ai-功能使用数据",
urls,
)
- self.assertEqual(labels, ["render-quality:render-quality-seed-list.md"])
+ self.assertEqual(labels, ["render-quality:render-quality-seed-list.csv"])
def test_webfetch_mix_keeps_top_site_mix_then_appends_longtail(self) -> None:
resolved, list_source = resolve_top_sites_source("webfetch-mix", None)
@@ -883,9 +1502,9 @@ if(window.EOJsChallengeSDK){new window.EOJsChallengeSDK({callback:function(){}})
self.assertEqual(
labels,
[
- "chinese-community:chinese-community-top100-websites.md",
- "global:global-top-websites-seed-list.md",
- "webfetch-longtail:webfetch-longtail-seed-list.md",
+ "chinese-community:chinese-community-top100-websites.csv",
+ "global:global-top-websites-seed-list.csv",
+ "webfetch-longtail:webfetch-longtail-seed-list.csv",
],
)
diff --git a/moli-benchmark/tests/test_versions.py b/moli-benchmark/tests/test_versions.py
new file mode 100644
index 000000000..ef2df7010
--- /dev/null
+++ b/moli-benchmark/tests/test_versions.py
@@ -0,0 +1,51 @@
+from __future__ import annotations
+
+import tempfile
+import unittest
+from pathlib import Path
+from unittest.mock import patch
+
+from moli_benchmark.targets import collect_target_binaries
+from moli_benchmark.versions import collect_versions
+
+
+class VersionsTests(unittest.TestCase):
+ def test_collect_versions_uses_moli_dash_version(self) -> None:
+ commands: list[list[str]] = []
+
+ def run(command: list[str], timeout: float = 5.0) -> str:
+ del timeout
+ commands.append(command)
+ return "moli 1.0.2"
+
+ with tempfile.TemporaryDirectory() as temp_dir:
+ binary = Path(temp_dir) / "moli"
+ binary.write_bytes(b"test binary")
+ with patch("moli_benchmark.versions._run", side_effect=run):
+ versions = collect_versions(binary)
+
+ self.assertIn([str(binary), "--version"], commands)
+ self.assertNotIn([str(binary), "version"], commands)
+ self.assertEqual(versions["moli"]["version"], "moli 1.0.2")
+
+ def test_target_metadata_uses_moli_dash_version(self) -> None:
+ moli = Path("/tmp/test-moli")
+
+ with (
+ patch("moli_benchmark.targets.moli_binary", return_value=moli),
+ patch("moli_benchmark.targets.lightpanda_binary", return_value=None),
+ patch("moli_benchmark.targets.chrome_binary", return_value=None),
+ patch("moli_benchmark.targets.obscura_binary", return_value=None),
+ patch(
+ "moli_benchmark.targets._binary_info",
+ side_effect=lambda path, args: {"path": path, "version_args": args},
+ ) as binary_info,
+ ):
+ targets = collect_target_binaries()
+
+ self.assertEqual(binary_info.call_args_list[0].args, (moli, ("--version",)))
+ self.assertEqual(targets["moli"]["version_args"], ("--version",))
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/moli-benchmark/tests/test_wild_web.py b/moli-benchmark/tests/test_wild_web.py
index 6fb8b12ea..3e520078b 100644
--- a/moli-benchmark/tests/test_wild_web.py
+++ b/moli-benchmark/tests/test_wild_web.py
@@ -1,15 +1,18 @@
from __future__ import annotations
+import json
import tempfile
import unittest
from pathlib import Path
from unittest.mock import patch
from moli_benchmark.process import ProcessResult
+from moli_benchmark.public_web import POST_DCL_WAIT_SCRIPT
from moli_benchmark.wild_web import (
_classify,
_extract_page_snapshot,
_failure_kind,
+ _rotated_target_order,
_wild_command_for_target,
_wild_web_target_metadata,
_wild_web_extraction_failures,
@@ -18,6 +21,15 @@ from moli_benchmark.wild_web import (
class WildWebTests(unittest.TestCase):
+ def test_target_order_rotates_per_seed(self) -> None:
+ targets = ("moli", "lightpanda", "chrome")
+
+ self.assertEqual(_rotated_target_order(targets, 0), targets)
+ self.assertEqual(
+ _rotated_target_order(targets, 1),
+ ("lightpanda", "chrome", "moli"),
+ )
+
def test_obscura_wild_web_command_uses_fetch_timeout_seconds(self) -> None:
command = _wild_command_for_target("obscura", Path("/bin/obscura"), "https://example.test/", 7.9)
@@ -32,7 +44,14 @@ class WildWebTests(unittest.TestCase):
command = _wild_command_for_target("lightpanda", Path("/bin/lightpanda"), "https://example.test/", 30.0)
self.assertEqual(command[:4], ["/bin/lightpanda", "fetch", "--dump", "html"])
- self.assertEqual(command[command.index("--wait-until") + 1], "done")
+ self.assertEqual(
+ command[command.index("--wait-until") + 1],
+ "domcontentloaded",
+ )
+ self.assertEqual(
+ command[command.index("--wait-script") + 1],
+ POST_DCL_WAIT_SCRIPT,
+ )
self.assertEqual(command[command.index("--wait-ms") + 1], "30000")
self.assertEqual(command[command.index("--http-timeout") + 1], "30000")
self.assertEqual(command[command.index("--terminate-ms") + 1], "30000")
@@ -41,6 +60,25 @@ class WildWebTests(unittest.TestCase):
self.assertNotIn("--http_timeout", command)
self.assertEqual(command[-1], "https://example.test/")
+ def test_moli_wild_web_command_uses_the_same_dcl_boundary(self) -> None:
+ command = _wild_command_for_target(
+ "moli",
+ Path("/bin/moli"),
+ "https://example.test/",
+ 30.0,
+ )
+
+ self.assertEqual(
+ command[command.index("--wait-until") + 1],
+ "domcontentloaded",
+ )
+ self.assertEqual(
+ command[command.index("--wait-script") + 1],
+ POST_DCL_WAIT_SCRIPT,
+ )
+ self.assertEqual(command[command.index("--timeout") + 1], "30000")
+ self.assertEqual(command[command.index("--http-timeout") + 1], "30000")
+
def test_wild_web_chrome_uses_cdp_dcl_metadata(self) -> None:
self.assertEqual(_wild_web_target_metadata("chrome")["driver"], "cdp-dcl")
self.assertEqual(_wild_web_target_metadata("chrome")["label"], "chrome / cdp-dcl")
@@ -55,7 +93,7 @@ class WildWebTests(unittest.TestCase):
command=["/bin/obscura", "serve", "--port", "12345"],
returncode=0,
elapsed_ms=12.0,
- stdout="<!doctype html><title>知乎</title><body>知乎 首页 内容 推荐 回答 专栏</body>".encode(),
+ stdout="<!doctype html><title>知乎</title><body>知乎 首页 内容 推荐 回答 专栏 热榜 发现 等你来答</body>".encode(),
stderr=b"",
timed_out=False,
resources={"peak_pss_bytes": 123},
@@ -86,7 +124,7 @@ class WildWebTests(unittest.TestCase):
"""
<!doctype html>
<title>知乎 - 有问题,就会有答案</title>
- <body><main>知乎 首页 内容 推荐 回答 专栏</main><script>hidden()</script></body>
+ <body><main>知乎 首页 内容 推荐 回答 专栏 热榜 发现 等你来答</main><script>hidden()</script></body>
""".encode()
)
@@ -94,6 +132,15 @@ class WildWebTests(unittest.TestCase):
self.assertIn("知乎 首页", snapshot["text_sample"])
self.assertEqual(_wild_web_extraction_failures("zhihu-home", snapshot), [])
+ def test_seed_brand_is_required_in_title_but_not_repeated_in_body(self) -> None:
+ snapshot = _extract_page_snapshot(
+ b"<!doctype html><title>Baidu</title><body>"
+ + b"useful navigation content " * 20
+ + b"</body>"
+ )
+
+ self.assertEqual(_wild_web_extraction_failures("baidu-home", snapshot), [])
+
def test_extraction_assertions_report_missing_business_fields(self) -> None:
snapshot = _extract_page_snapshot(b"<!doctype html><title>Example</title><body>tiny</body>")
@@ -103,6 +150,15 @@ class WildWebTests(unittest.TestCase):
)
self.assertEqual(_failure_kind("success", ["title-keyword-mismatch"]), "extraction-failure")
+ def test_snapshot_does_not_count_title_as_body_text(self) -> None:
+ snapshot = _extract_page_snapshot(
+ b"<!doctype html><title>A long title</title><body>short body</body>"
+ )
+
+ self.assertEqual(snapshot["title"], "A long title")
+ self.assertEqual(snapshot["text_sample"], "short body")
+ self.assertEqual(snapshot["text_length"], len("short body"))
+
def test_classify_ignores_blocked_markers_inside_scripts_when_visible_content_exists(
self,
) -> None:
@@ -120,6 +176,101 @@ class WildWebTests(unittest.TestCase):
self.assertEqual(_classify(html, b"", 0, False), "blocked")
+ def test_classify_rejects_contentful_http_error_response(self) -> None:
+ html = (
+ b"<!doctype html><title>upstream response</title><body>"
+ + b"error details " * 100
+ + b"</body>"
+ )
+
+ self.assertEqual(
+ _classify(html, b"", 0, False, response_status=500),
+ "http-error",
+ )
+ self.assertEqual(_failure_kind("http-error", []), "http-error")
+
+ def test_classify_infers_cli_error_documents_and_network_failures(self) -> None:
+ gateway = b"<!doctype html><title>502 Bad Gateway</title><body>origin unavailable</body>"
+ api_error = (
+ b"<!doctype html><title></title><body><pre>"
+ b'{"error":{"message":"invalid query"}}'
+ b"</pre></body>"
+ )
+ tls_error = b"<!doctype html><body>Navigation failed Reason: PeerFailedVerification</body>"
+ chinese_verification = "<!doctype html><body>环境异常,完成验证后即可继续访问</body>".encode()
+
+ self.assertEqual(_classify(gateway, b"", 0, False), "http-error")
+ self.assertEqual(_classify(api_error, b"", 0, False), "http-error")
+ self.assertEqual(_classify(tls_error, b"", 0, False), "network-error")
+ self.assertEqual(_failure_kind("network-error", []), "network-error")
+ self.assertEqual(
+ _classify(chinese_verification, b"", 0, False),
+ "challenge",
+ )
+
+ def test_classify_recognizes_lightpanda_transport_and_waf_documents(self) -> None:
+ dns = (
+ b"<!doctype html><body><h1>Navigation failed</h1>"
+ b"<p>Reason: CouldntResolveHost</p></body>"
+ )
+ http2 = (
+ b"<!doctype html><body><h1>Navigation failed</h1>"
+ b"<p>Reason: Http2Stream</p></body>"
+ )
+ safeline = (
+ b"<!doctype html><body>Security Detection Powered By SafeLine WAF"
+ b"</body>"
+ )
+ privacy = (
+ b"<!doctype html><body>Select the 2 matching tiles Submit "
+ b"Powered and protected by Privacy</body>"
+ )
+
+ self.assertEqual(_classify(dns, b"", 0, False), "network-error")
+ self.assertEqual(_classify(http2, b"", 0, False), "network-error")
+ self.assertEqual(_classify(safeline, b"", 0, False), "blocked")
+ self.assertEqual(_classify(privacy, b"", 0, False), "challenge")
+
+ def test_classify_recognizes_lightpanda_fatal_fetch_timeout(self) -> None:
+ stderr = (
+ b'$scope=app $level=fatal $msg="fetch error" err=Timeout '
+ b"url=https://example.test/"
+ )
+
+ self.assertEqual(_classify(b"", stderr, 0, False), "timeout")
+
+ def test_classify_preserves_cli_main_document_status_and_dns_reason(self) -> None:
+ self.assertEqual(
+ _classify(
+ b"",
+ b"Reason: lifecycle target document `https://example.test/` "
+ b"returned 500 Internal Server Error",
+ 1,
+ False,
+ ),
+ "http-error",
+ )
+ self.assertEqual(
+ _classify(
+ b"",
+ b"Reason: lifecycle target document `https://example.test/` "
+ b"returned 401 Unauthorized",
+ 1,
+ False,
+ ),
+ "blocked",
+ )
+ self.assertEqual(
+ _classify(
+ b"",
+ b"Reason: failed to resolve example.test:443: failed to lookup "
+ b"address information",
+ 1,
+ False,
+ ),
+ "network-error",
+ )
+
def test_run_wild_web_suite_writes_failure_snapshots(self) -> None:
def run_process(*args: object, **kwargs: object) -> ProcessResult:
return ProcessResult(
@@ -147,10 +298,22 @@ class WildWebTests(unittest.TestCase):
failures_dir = output_dir / "wild-web" / "failures"
failure_files = sorted(path.name for path in failures_dir.iterdir())
+ rows = json.loads(
+ (output_dir / "wild-web" / "runs.json").read_text(
+ encoding="utf-8"
+ )
+ )
self.assertEqual(summary["gate_failures"], 1)
self.assertEqual(summary["targets"]["moli"]["failure_kinds"], {"extraction-failure": 1})
self.assertEqual(summary["targets"]["moli"]["extraction_failures"], 2)
+ self.assertEqual(summary["schedule"], "site-paired-rotating-target-order")
+ self.assertEqual(summary["scheduled_site_groups"], 1)
+ self.assertEqual(rows[0]["schedule_index"], 0)
+ self.assertEqual(rows[0]["target_order_index"], 1)
+ self.assertTrue(rows[0]["started_at"].endswith("Z"))
+ self.assertEqual(len(rows[0]["stdout_sha256"]), 64)
+ self.assertEqual(len(rows[0]["stderr_sha256"]), 64)
self.assertIn("moli-run-1-toutiao-home.json", failure_files)
self.assertIn("moli-run-1-toutiao-home.stdout.html", failure_files)
@@ -182,6 +345,12 @@ class WildWebTests(unittest.TestCase):
self.assertEqual(summary["gate_failures"], 0)
self.assertEqual(summary["targets"]["moli"]["categories"], {"login": 1})
self.assertEqual(summary["targets"]["moli"]["failure_kinds"], {})
+ self.assertEqual(summary["targets"]["moli"]["elapsed_ms"]["count"], 1)
+ self.assertEqual(
+ summary["targets"]["moli"]["successful_elapsed_ms"]["count"],
+ 1,
+ )
+ self.assertEqual(summary["common_success"]["attempts"], 1)
def test_capture_replay_writes_manifest_for_successful_rows(self) -> None:
def run_process(*args: object, **kwargs: object) -> ProcessResult:
@@ -189,7 +358,7 @@ class WildWebTests(unittest.TestCase):
command=["fake"],
returncode=0,
elapsed_ms=12.0,
- stdout="<!doctype html><title>知乎</title><body>知乎 首页 内容 推荐 回答 专栏</body>".encode(),
+ stdout="<!doctype html><title>知乎</title><body>知乎 首页 内容 推荐 回答 专栏 热榜 发现 等你来答</body>".encode(),
stderr=b"",
timed_out=False,
resources={"peak_pss_bytes": 123},