The ultimate harness for coding agents and beyond. All your agents. All your machines. One command center. Start with code, then follow your curiosity and build across disciplines: CAD, circuits, robots, games and music.
TypeScript 3d-modelingagent-frameworkagent-harnessai-agentscadclaude-codecodexcoding-agentsdeveloper-toolsesp32game-developmentmcp-servermulti-agentopen-hardwareopencodeopenharnesspcbroboticsterminaltmux
fix(jev-browser): catch a robot-policy page, and say which sites really block (#144)
Dee2dd4b2b86b93
diff --git a/store/agents/jev-browser/AGENTS.md b/store/agents/jev-browser/AGENTS.mdindex c438a5238..26986a4be 100644--- a/store/agents/jev-browser/AGENTS.md+++ b/store/agents/jev-browser/AGENTS.md@@ -160,11 +160,13 @@ Never tell a person you can work around these, and never ask them for a password- A site may say in its terms that it does not want to be read this way, and some sites charge foran API that gives the same data. Say so once, and let the person decide. Do not go around ablock, a login wall, a rate limit or a robots rule.-- **Big shops and social sites usually block an automated browser.** The verdict says so plainly- (`run.walled`), the pane says so, and that is the end of that site. Do not retry it, do not try- another address on it, and do not pretend it half worked. Offer a site that does allow reading,- and say what it will give them instead. A specialist shop often has better columns than a- marketplace anyway.+- **Many sites block an automated browser, and you cannot tell which by looking.** Measured on+ 2026-09-20: Hacker News, arXiv, gov.uk, data.gov.uk, GitHub, We Work Remotely and shop.bbc.com+ read; Amazon, Wikipedia, Rightmove and three specialist fencing shops blocked. So do not promise+ a person a site will work. Write the job, let it run, and read what comes back: it takes ten+ seconds. When the verdict carries `run.walled`, that is the end of that site. Do not retry it,+ do not try another address on it, and do not pretend it half worked. Offer a different source+ and say what that one will give them instead.- Take what is asked for and no more. `maxItems` is a page load each, so it costs the site morethan it costs you: set it to what the person actually needs, not to the maximum.- The rows are what the page said on the day it was read. If that matters, say when it was read.diff --git a/store/agents/jev-browser/README.md b/store/agents/jev-browser/README.mdindex af604e063..d25124148 100644--- a/store/agents/jev-browser/README.md+++ b/store/agents/jev-browser/README.md@@ -112,14 +112,26 @@ about being a stand-in, but its rows are not worth acting on. Paste an OpenRouteinto the **Jev · live mind** panel in the pane. It is saved on your machine in`~/.config/typesafe/credentials` and checked with one tiny call.-## When a site says no+## Which sites let it read-Big shops, marketplaces and social sites mostly block an automated browser. What comes back is a-wall: "Sorry, something went wrong", a captcha, or a near-empty page. The harness recognises that,-says which site did it, and stops. It will not work around a block, and neither should the agent.+Checked on 2026-09-20, one page each, with the headless browser the tests use.-A specialist shop, a maker's own site or a public dataset usually reads perfectly, and its pages-often print the fields a marketplace hides.+| Reads | Blocks |+|---|---|+| news.ycombinator.com, arxiv.org, gov.uk, data.gov.uk, github.com, weworkremotely.com, shop.bbc.com, books.toscrape.com | amazon.com, en.wikipedia.org, rightmove.co.uk, and three fencing retailers (absolutefencinggear.com, blue-gauntlet.com, leonpaul.com) |++The pattern is not "big versus small". It is whether the site runs bot protection. Cloudflare's+"Just a moment…", Amazon's "Sorry! Something went wrong", Rightmove's "Client Challenge" and+Wikipedia's robot-policy page are all the same answer: no. Plenty of small specialist shops sit+behind Cloudflare, and a large shop like shop.bbc.com does not.++Public, reference, government, open-data, code, papers, forums and job boards mostly read. Consumer+retail and property mostly do not. There is no way to know but to try, which takes about ten+seconds: point it at the page and the pane tells you straight away.++When a site says no, that is the end of it. The harness names the site and stops. It will not+change what it looks like to get past a challenge, and neither should the agent. Where a site sells+an API for the same data, that is the route they want you to use.## The honest limitdiff --git a/store/agents/jev-browser/skills/browser/SKILL.md b/store/agents/jev-browser/skills/browser/SKILL.mdindex 126e7e9fa..782100b11 100644--- a/store/agents/jev-browser/skills/browser/SKILL.md+++ b/store/agents/jev-browser/skills/browser/SKILL.md@@ -85,13 +85,18 @@ cell empty than write something the page never said.## When a site says no-Big shops, marketplaces and social sites mostly block an automated browser. What comes back is a-wall page: "Sorry, something went wrong", a captcha, or a near-empty page. The harness names it and-stops, and the verdict carries `run.walled`. That is not a bug to work around, and trying another-address on the same site is the same wall.--Say it plainly and move: a specialist shop, a maker's own site, a public dataset or an official API.-Those pages are usually better anyway, because they print the fields a marketplace hides.+A site that runs bot protection answers with a wall: "Just a moment…", "Sorry, something went+wrong", a captcha, a 403, or a robot-policy page. The harness names it, stops, and puts it in+`run.walled`. Trying another address on the same site is the same wall. Do not work around it.++**Do not guess which sites will block.** Measured on 2026-09-20: Hacker News, arXiv, gov.uk,+data.gov.uk, GitHub, We Work Remotely and shop.bbc.com all read. Amazon, Wikipedia, Rightmove and+three specialist fencing retailers all blocked. Small does not mean open, and big does not mean+shut: it is whether the site runs a challenge. Point the job at the page and let the pane tell you+in ten seconds, rather than promising a person it will work.++When it walls, say so plainly and offer a different source: a public dataset, an official API, a+government or reference site, or the maker's own pages. Do not re-run it hoping.## Neverdiff --git a/store/agents/jev-browser/viewer/page.mjs b/store/agents/jev-browser/viewer/page.mjsindex fb64031a6..9c3e814ad 100644--- a/store/agents/jev-browser/viewer/page.mjs+++ b/store/agents/jev-browser/viewer/page.mjs@@ -157,7 +157,7 @@ export function blockText(page, pick) {return m[2] === undefined ? block.text : block.parts?.[Number(m[2])] ?? block.text}-const WALL = /sorry[!,. ]{0,3}something went wrong|are you a (robot|human)|unusual traffic|access denied|permission denied|request blocked|captcha|verify (you are|your) human|enable javascript|checking your browser|rate limit|too many requests|403 forbidden|pardon our interruption/i+const WALL = /sorry[!,. ]{0,3}something went wrong|are you a (robot|human)|unusual traffic|access denied|permission denied|request blocked|captcha|verify (you are|your) human|enable javascript|checking your browser|rate limit|too many requests|40[139] (forbidden|error)|pardon our interruption|robots?\.txt|robot policy|automated (access|traffic|requests?)|bot (detection|protection)|client challenge|just a moment|attention required|ddos protection|please enable cookies|unsupported browser|your (request|activity) (has been|was) (blocked|flagged)/i/*** Did the site serve its page, or a wall? A bot wall has almost nothing on it, or says so outright.* Worth naming: a run that quietly collects nothing looks like a broken tool, and it is not.@@ -165,7 +165,12 @@ const WALL = /sorry[!,. ]{0,3}something went wrong|are you a (robot|human)|unusu*/export function wallReason(page) {const hit = `${page.title} ${page.text.slice(0, 600)}`.match(WALL)- if (hit) return `the site answered with a block page ("${hit[0]}") instead of its own`+ if (hit) return `the site answered with a block page ("${hit[0].trim()}") instead of its own`+ // An error page carries plenty of text, so the emptiness test below never catches it, and its+ // words would otherwise be collected as if they were values.+ if (/^(error|wikimedia error|access denied|forbidden|blocked|not acceptable|service unavailable|too many requests)\b/i.test(page.title.trim())) {+ return `the site answered with an error page ("${page.title.trim().slice(0, 40)}") instead of its own`+ }if (page.links.length < 3 && page.blocks.length < 4) return 'the page came back nearly empty, which usually means the site refused an automated browser or needs a sign-in'return null}