Skip to content

The ultimate harness for coding agents and beyond. All your agents. All your machines. One command center. Start with code, then follow your curiosity and build across disciplines: CAD, circuits, robots, games and music.

feat(jev-browser): Jev works out the columns, so an address and a sentence is the job (#147)

Dee04973cfc0da8
diff --git a/store/agents/jev-browser/AGENTS.md b/store/agents/jev-browser/AGENTS.md
index 26986a4be..7d423bbbd 100644
--- a/store/agents/jev-browser/AGENTS.md
+++ b/store/agents/jev-browser/AGENTS.md
@@ -26,10 +26,12 @@ in front of the person, and a run starts by itself the moment `browse.json` chan
do with Chrome yourself is slower, invisible to them, and doubles the work. Write the file first,
let it run, and look at a page only if something comes back wrong.
-1. **Take what they said and turn it into a job.** A start address, what one "thing" is, and the
- columns. Guess sensibly from their words: "the best fencing gloves on a shop" means a product
- listing page, one thing is a glove for sale, and the columns are name, price, rating, reviews.
- Do not interview them. Write something, watch it, then fix it.
+1. **Take what they said and turn it into a job.** Often all you need is `start` and `want`:
+ put their own sentence in `want` and leave `fields` out, and Jev reads the page and works the
+ columns out in about a second. That is faster than you deciding, and it names the columns after
+ the page's own labels. Write the columns yourself only when they asked for something specific
+ that a page would not volunteer, such as a judgement ("is this remote?") or a score.
+ Do not interview them. Write something, let it run, then fix it.
2. **Write `browse.json`** and validate with `node "$JEV_DSH/toolchain/check.mjs"`. Saving it starts
the run: the browser opens in the pane and rows begin landing within a few seconds.
3. **Watch `.harness/verdict.json`** (see "While it runs"). Say one short line about what is
@@ -51,6 +53,7 @@ let it run, and look at a page only if something comes back wrong.
"task": "Every flat for rent in the search results, with rent and address",
"start": "https://example.com/search?area=leeds", // the page to begin on. "demo" is the made-up job board this harness serves itself
"search": "", // optional: words to type into the site's own search box first
+ "want": "what each one costs and whether it is in stock", // the person's own words. Leave "fields" out and Jev works the columns out from the page
"item": "a flat for rent", // one of the things. Used in every question, so make it concrete
"fields": [
{ "id": "address", "name": "Address", "ask": "the street address" },
@@ -70,9 +73,13 @@ let it run, and look at a page only if something comes back wrong.
}
```
-- **`fields`**: up to 12. `pick` (the default) takes the exact text off the page. `yesno` is Jev's
- judgement about the thing. `score` puts it on your named scale. Every field gets a confidence
- column in the spreadsheet.
+- **`fields`**: up to 12, and optional. Leave it out and Jev proposes the columns from one of the
+ pages, writing them back into `browse.json` so the recipe stays complete. `pick` (the default)
+ takes the exact text off the page. `yesno` is Jev's judgement about the thing. `score` puts it on
+ your named scale. Every field gets a confidence column in the spreadsheet.
+- **`want`**: the person's own sentence. It is never parsed. It is shown to Jev as context while it
+ decides which values are worth a column, so "what each one costs" pulls the proposal towards
+ price and away from the site's boilerplate.
- **`item`** goes into the question asked about every link, so "a flat for rent" works and "an item"
does not.
diff --git a/store/agents/jev-browser/README.md b/store/agents/jev-browser/README.md
index d25124148..f7b7ab31e 100644
--- a/store/agents/jev-browser/README.md
+++ b/store/agents/jev-browser/README.md
@@ -6,18 +6,25 @@ watch. No code, no selectors, no copy and paste.
## How you use it
-The pane asks you three questions and then goes. There is nothing to press afterwards.
+An address and a sentence is the whole job.
| | |
|---|---|
| **Start on** | `books.toscrape.com/catalogue/category/books/travel_2/index.html` |
-| **Search for** | leave it empty, or put words to type into the site's own search box |
-| **Each one is** | `a book for sale` |
-| **Columns** | `the title` · `the price` · `how many copies are available` · `Is it in stock?` |
+| **What you want** | `what each book costs and whether it is in stock` |
-Press **Go**. The browser opens, the rows land, and `results.csv` is in your project folder. A line
-that reads like a question ("Is it in stock?") becomes a yes-or-no column; everything else is taken
-off the page word for word.
+Press **Go**. Jev reads the page, works out what it lists and what the columns should be in about
+a second, and starts collecting. There is nothing else to press.
+
+**Jev picks the columns, it does not write them.** It is shown every piece of text on one of the
+pages and asked, for each, whether that is a fact about this thing worth a column. The column's
+name comes from the page's own label where there is one, so a spreadsheet of books comes out with
+Price, Availability, UPC and Number of reviews on it. Your sentence is never parsed into a config;
+it is given to Jev as context while it decides what matters.
+
+Name the columns yourself if you would rather: put them in the **Columns** box, one per line. A
+line that reads like a question ("Is it in stock?") becomes a yes-or-no column, and everything else
+is taken off the page word for word.
The agent on the right is for the rest: a site that fights back, a column that comes back empty,
and reading what you collected. You do not need it to run a job. Ask it in plain words and it
@@ -102,6 +109,11 @@ On 2026-09-20 with live Jev (`typesafe/jev-1.13`) through OpenRouter.
| The built-in job board, 36 roles | 24 | 33 s | $0.0023 | 144 of 144 cells exactly matched the site's own data, no repeats, nothing skipped |
| `books.toscrape.com`, a public sandbox | 8 | 21 s | $0.0018 | every title, price, stock count and UPC code right |
+Working out the job from scratch, on the same site: two Jev calls, 61 questions then 33, **1.2
+seconds** to decide the page lists products and to propose columns named Price, Availability, UPC
+and Number of reviews. No language model is involved at any point, in setting the job up or in
+running it.
+
The product codes matter: `a22124811bfa8350` is not something a model could write from memory. It
came off the page, which is the whole point. Small runs are a sanity check, not a benchmark.
diff --git a/store/agents/jev-browser/skills/browser/SKILL.md b/store/agents/jev-browser/skills/browser/SKILL.md
index 782100b11..3f103dec2 100644
--- a/store/agents/jev-browser/skills/browser/SKILL.md
+++ b/store/agents/jev-browser/skills/browser/SKILL.md
@@ -26,6 +26,17 @@ Point `start` at the list. If you point it at one thing's page, that one row is
run ends. If the things have no page of their own (everything is on the list), say so to the person:
this harness collects one row per page, so a list-only site gives one row for the whole list.
+## Letting Jev write the fields
+
+Put the person's sentence in `want`, leave `fields` out, and two Jev calls decide it: one asks what
+the page lists and which links are the things, the other asks of every piece of text on one of them
+"is that a fact about this one, worth a column?" and "what kind of value is it?". About a second,
+and the names come off the page's own labels.
+
+Do that first. Read what it proposed, then add or reword. What it cannot propose is a judgement:
+"is this remote?", "how urgent is this". Those are yours to write, and they are where the tool earns
+its keep, because no page prints them.
+
## Writing the fields
A field is what to look for, in the words a person would use.
diff --git a/store/agents/jev-browser/store.json b/store/agents/jev-browser/store.json
index cc4bd8c0e..426536f20 100644
--- a/store/agents/jev-browser/store.json
+++ b/store/agents/jev-browser/store.json
@@ -2,17 +2,17 @@
"homepage": "https://typesafe.ai",
"upstream": "https://typesafe.ai",
"license": "MIT",
- "tagline": "Say what you want off a web page, and get a spreadsheet",
+ "tagline": "An address and a sentence. Jev works out the columns and fills the sheet",
"examples": [
{
"prompt": "Here is the search results page for two-bedroom flats in my area. Get me every listing with its address, monthly rent, how many bedrooms, and whether it says pets are allowed. Then tell me which five are the best value.",
- "image": "https://raw.githubusercontent.com/autonomous-ai/openharness/main/store/showcase/jev-browser/reads-real-pages.jpg",
- "caption": "One call judged all 16 links: the twelve real roles at 76-88%, the menu at 2%"
+ "image": "https://raw.githubusercontent.com/autonomous-ai/openharness/main/store/showcase/jev-browser/jev-picks-the-columns.jpg",
+ "caption": "No columns were typed. Jev read one page and chose them in about a second"
},
{
"prompt": "Read every product in this category and put the name, price, stock count and product code in a spreadsheet. Stop after 40. Then tell me which ones are out of stock.",
- "image": "https://raw.githubusercontent.com/autonomous-ai/openharness/main/store/showcase/jev-browser/say-what-you-want.jpg",
- "caption": "Three boxes and a list of columns. No code, no selectors, and nothing to press after Go"
+ "image": "https://raw.githubusercontent.com/autonomous-ai/openharness/main/store/showcase/jev-browser/reads-real-pages.jpg",
+ "caption": "One call judged all 16 links: the real items at 76-88%, the site menu at 2%"
},
{
"prompt": "I have to check these 60 supplier pages every month for their lead times and minimum order. Write me a job I can run again, then run it and tell me what changed.",
diff --git a/store/agents/jev-browser/test/viewer.test.mjs b/store/agents/jev-browser/test/viewer.test.mjs
index 080db2bf0..3e4ab366f 100644
--- a/store/agents/jev-browser/test/viewer.test.mjs
+++ b/store/agents/jev-browser/test/viewer.test.mjs
@@ -7,7 +7,7 @@ import { tmpdir } from 'node:os'
import { connect } from 'node:net'
import { join, dirname } from 'node:path'
import { fileURLToPath } from 'node:url'
-import { normalizeJob, listQuestions, itemQuestions, rowFrom, pickLinks, LIMITS } from '../viewer/crawl.mjs'
+import { normalizeJob, listQuestions, itemQuestions, rowFrom, pickLinks, proposeJob, THINGS, VALUE_KINDS, LIMITS } from '../viewer/crawl.mjs'
import { readPage, blockOptions, blockText, canonical, pageState, wallReason, findSearchBox } from '../viewer/page.mjs'
import { openChrome, findChrome, checkUrl, Refused } from '../toolchain/chrome.mjs'
import { startDemoSite } from '../viewer/demosite.mjs'
@@ -39,7 +39,11 @@ test('the job file: what it fills in, and what it says is wrong', () => {
const bare = normalizeJob({})
assert.match(bare.errors.join(' '), /no "start"/)
- assert.match(bare.errors.join(' '), /no "fields"/)
+ // No columns is not a mistake: Jev reads the page and proposes them.
+ const noColumns = normalizeJob({ start: 'https://example.com/all', want: 'what each one costs' })
+ assert.deepEqual(noColumns.errors, [])
+ assert.deepEqual(noColumns.job.fields, [])
+ assert.equal(noColumns.job.want, 'what each one costs')
const messy = normalizeJob({
start: 'https://shop.example.com/all', item: 'a product', alsoVisit: ['https://cdn.example.org/x'],
@@ -210,11 +214,11 @@ test('a whole run: rows land in results.csv, the verdict says what happened', {
test('a broken job file keeps the pane alive and says what to fix', async () => {
const ws = mkdtempSync(join(tmpdir(), 'jev-browser-bad-'))
- writeFileSync(join(ws, 'browse.json'), '{ "start": "https://example.com", "fields": [] }')
+ writeFileSync(join(ws, 'browse.json'), '{ "fields": ["the price"] }') // no "start"
const v = await startBrowserViewer({ workspace: ws, port: 0 })
try {
const s = await (await fetch(`${v.url}/state`)).json()
- assert.match(s.error, /no "fields"/)
+ assert.match(s.error, /no "start"/)
const start = await (await fetch(`${v.url}/control`, { method: 'POST', headers: { 'content-type': 'application/json' }, body: '{"cmd":"start"}' })).json()
assert.equal(start.ok, false, 'it will not run a job it cannot read')
const verdict = JSON.parse(readFileSync(join(ws, '.harness/verdict.json'), 'utf8'))
@@ -365,3 +369,50 @@ test('a site that serves a wall is named, not silently collected from', { skip:
assert.match(verdict.run.walled, /block page/)
} finally { await v.close(); blocker.close(); delete process.env.JEV_OFFLINE }
})
+
+test('Jev proposes the whole job from the page, and it then runs with no columns given', { skip: noChrome }, async () => {
+ process.env.JEV_OFFLINE = '1'
+ const site = await startDemoSite()
+ const ws = mkdtempSync(join(tmpdir(), 'jev-browser-prop-'))
+ // An address and a sentence, and no columns at all.
+ writeFileSync(join(ws, 'browse.json'), JSON.stringify({ start: site.url, want: 'what each role pays and where it is', maxItems: 2, maxPages: 1, autoStart: false }))
+ const v = await startBrowserViewer({ workspace: ws, port: 0 })
+ const state = async () => (await fetch(`${v.url}/state`)).json()
+ try {
+ const before = await state()
+ assert.deepEqual(before.fields, [], 'it starts with no columns')
+ assert.equal(before.want, 'what each role pays and where it is')
+ assert.equal(before.error, null, 'no columns is not an error any more')
+
+ await (await fetch(`${v.url}/control`, { method: 'POST', headers: { 'content-type': 'application/json' }, body: '{"cmd":"run"}' })).json()
+ const s = await state()
+ assert.ok(s.fields.length >= 1, `Jev proposed columns: ${JSON.stringify(s.fields.map((f) => f.name))}`)
+ // What it worked out is written back, because browse.json is the recipe.
+ const written = JSON.parse(readFileSync(join(ws, 'browse.json'), 'utf8'))
+ assert.equal(written.fields.length, s.fields.length)
+ assert.ok(written.fields.every((f) => f.id && f.name && f.ask))
+ assert.equal(s.rows.length, 2, 'and then it collected with them')
+ const csv = readFileSync(join(ws, 'results.csv'), 'utf8').trim().split('\n')
+ assert.equal(csv.length, 3)
+ assert.ok(csv[0].includes(s.fields[0].name))
+ } finally { await v.close(); await site.close(); delete process.env.JEV_OFFLINE }
+})
+
+test("a proposed column is named off the page, and none of it is written", { skip: noChrome }, async () => {
+ process.env.JEV_OFFLINE = '1'
+ const site = await startDemoSite()
+ const chrome = await browser()
+ try {
+ const { evaluate } = await import('../toolchain/jev.mjs')
+ const got = await proposeJob({ chrome, ask: (x) => evaluate({ ...x, salt: 3 }), start: site.url, want: 'the pay' })
+ assert.ok(Object.keys(THINGS).includes(got.kind))
+ assert.ok(got.from.includes('/job/'), 'it opened one of the things to look at')
+ assert.equal(new Set(got.columns.map((x) => x.id)).size, got.columns.length, 'no two columns share an id')
+ for (const c of got.columns) {
+ assert.ok(c.name && c.id && c.ask, JSON.stringify(c))
+ // Every ask is either the page's own label or one of the fixed phrases. Nothing is invented.
+ assert.ok(c.ask.startsWith('the ') || Object.values(VALUE_KINDS).includes(c.ask), c.ask)
+ assert.ok(c.p >= 0.6 && c.p <= 1)
+ }
+ } finally { await chrome.close(); await site.close(); delete process.env.JEV_OFFLINE }
+})
diff --git a/store/agents/jev-browser/viewer/crawl.mjs b/store/agents/jev-browser/viewer/crawl.mjs
index 56b1059a9..04320b52f 100644
--- a/store/agents/jev-browser/viewer/crawl.mjs
+++ b/store/agents/jev-browser/viewer/crawl.mjs
@@ -36,7 +36,8 @@ export function normalizeJob(raw) {
if (type === 'score' && levels.length < 2) { errors.push(`field "${id}" is a score, so it needs at least two levels`); continue }
fields.push({ id, ask, type, levels, name: short(typeof f === 'object' ? f?.name : '', 40) || id })
}
- if (!fields.length) errors.push('no "fields": say what to collect for each item, in plain words')
+ // No fields is not a mistake any more: the pane reads the page and proposes them.
+ const wants = short(j.want, 300)
let hosts = []
try { hosts = [new URL(start).hostname.toLowerCase()] } catch { /* the start address is already reported */ }
for (const h of Array.isArray(j.alsoVisit) ? j.alsoVisit : []) { const s = short(h, 200).toLowerCase().replace(/^https?:\/\//, '').split('/')[0]; if (s) hosts.push(s) }
@@ -45,6 +46,7 @@ export function normalizeJob(raw) {
task: short(j.task, 200) || `Collect ${item}`,
start, item, fields,
search: short(j.search, 120),
+ want: wants,
keep: short(j.keep, 300),
maxPages: clamp(j.maxPages, 1, LIMITS.maxPages, 25),
maxItems: clamp(j.maxItems, 1, LIMITS.maxItems, 60),
@@ -83,7 +85,8 @@ export function itemQuestions(page, job) {
for (const f of job.fields) {
if (f.type === 'yesno') q[`f_${f.id}`] = jev.noul(`About ${job.item} on this page: ${f.ask}`)
else if (f.type === 'score') q[`f_${f.id}`] = jev.score(f.levels, `About ${job.item} on this page: ${f.ask}`)
- else q[`f_${f.id}`] = jev.choice(options, `On this page, which piece of text is ${f.ask}? Pick the exact words from the page.`)
+ // "Number of reviews: 0" and "0" are both offered; a column wants the 0.
+ else q[`f_${f.id}`] = jev.choice(options, `On this page, which piece of text is ${f.ask}? Pick the exact words from the page, and where the page prints a label next to its value, pick the value on its own rather than the whole line.`)
}
if (job.keep) q.keep = jev.noul(`The person wants only the ones where: ${job.keep}. Does ${job.item} on this page fit that?`)
return { questions: q, blocks: page.blocks.length }
@@ -117,6 +120,123 @@ export function pickLinks(links, answers, { threshold = 0.5 } = {}) {
return { items, next, judged: links.length }
}
+
+// ---- proposing a job ----------------------------------------------------------------------------
+/** What a page can be listing. A fixed list, so Jev only has to point at one. */
+export const THINGS = {
+ product: 'things for sale, with prices', job: 'job openings or roles', article: 'articles, posts or news stories',
+ issue: 'bug reports, tickets or issues', property: 'homes or places to rent or buy', person: 'people or profiles',
+ paper: 'research papers or publications', repo: 'code projects or repositories', event: 'events with dates',
+ listing: 'listings of some other kind', other: 'this page does not list many of one thing',
+}
+const ITEM_WORDS = {
+ product: 'a product for sale', job: 'a job opening', article: 'an article', issue: 'an issue or bug report',
+ property: 'a property', person: 'a person', paper: 'a research paper', repo: 'a code project',
+ event: 'an event', listing: 'one of the listings', other: 'one of the things on this page',
+}
+/** What kind of value a piece of text is, and what to ask for it on every page. */
+export const VALUE_KINDS = {
+ name: 'the name or title of it', price: 'the price or cost', date: 'the date', quantity: 'how many there are',
+ code: 'the reference, product code or number', rating: 'the rating or score', place: 'where it is',
+ status: 'its current status', who: 'the person or company behind it', kind: 'what type it is',
+ detail: 'a longer description of it', other: 'none of these',
+}
+
+/**
+ * Read one list page and one of its things, and propose the whole job.
+ * @param {object} o
+ * @param {object} o.chrome
+ * @param {function} o.ask
+ * @param {string} o.start the page to read
+ * @param {string} [o.want] what the person said they want, in their own words. Never parsed: it is
+ * shown to Jev as context while it decides what is worth a column.
+ * @param {string} [o.search]
+ * @returns {Promise<{ kind, item, columns, sample, from, walled }>}
+ */
+export async function proposeJob({ chrome, ask, start, want = '', search = '', onEvent }) {
+ const say = (type, data) => { try { onEvent?.({ type, at: Date.now(), ...data }) } catch { /* the pane may be gone */ } }
+ say('going', { url: start, what: 'the page you gave' })
+ await chrome.go(start)
+ let list = await readPage(chrome)
+ if (search) {
+ const box = findSearchBox(list)
+ if (box) { say('searching', { words: search }); await chrome.search(box.css, search); list = await readPage(chrome) }
+ }
+ const wall = wallReason(list)
+ if (wall) return { walled: `${new URL(list.url).hostname}: ${wall}`, columns: [] }
+
+ // CALL 1 — what does this page list, and which of its links are the things?
+ const wanted = want ? ` The person asked for: "${want}".` : ''
+ const links = list.links.slice(0, LIMITS.linksPerCall)
+ const q1 = { kind: jev.choice(THINGS, `This page may list many of one kind of thing.${wanted} What is it listing?`) }
+ for (const l of links) {
+ q1[`l${l.n}`] = jev.noul(`"${l.label}" is the text of a link on this page, pointing at ${l.path}. Is "${l.label}" the title of one of the things this page lists, rather than part of the site's own menu?`)
+ }
+ const r1 = await ask({ state: pageState(list, want ? { what_they_want: want } : {}) , questions: q1 })
+ const kind = r1.answers.kind?.choice ?? 'listing'
+ const items = links.map((l) => ({ ...l, p: r1.answers[`l${l.n}`]?.noul ?? 0 })).filter((l) => l.p >= 0.5)
+ // To learn what a thing's page looks like, open the one Jev is surest about, not the first on the
+ // page: the first is often a menu link that only just scraped over the line.
+ const surest = [...items].sort((a, b) => b.p - a.p)[0]
+ const item = ITEM_WORDS[kind] ?? ITEM_WORDS.other
+ say('judged', { url: list.url, kind, judged: links.length, found: items.length, latencyMs: r1.latencyMs, links: links.map((l) => ({ n: l.n, label: l.label, p: r1.answers[`l${l.n}`]?.noul ?? 0 })).sort((a, b) => b.p - a.p).slice(0, 40) })
+ if (!items.length) return { kind, item, columns: [], from: list.url, none: 'no link on this page looks like one of the things' }
+
+ // CALL 2 — on one of them, which pieces of text deserve a column, and what is each one?
+ say('going', { url: surest.url, what: `one ${item}, to see what it says` })
+ await chrome.go(surest.url)
+ const one = await readPage(chrome)
+ const itemWall = wallReason(one)
+ if (itemWall) return { kind, item, columns: [], from: one.url, walled: `${new URL(one.url).hostname}: ${itemWall}` }
+ const q2 = {}
+ for (const b of one.blocks) {
+ const text = b.text.slice(0, 140)
+ // "A fact about this one" has to exclude the other things a page shows alongside: a page of one
+ // product carries a "recently viewed" strip full of other products' names and prices, and those
+ // are facts about something, just not about this.
+ q2[`w${b.n}`] = jev.noul(`This page is about ONE ${item}: "${one.title.slice(0, 80)}".${wanted} The page also shows the text "${text}". Is that a fact about THAT ONE thing, worth its own column in a spreadsheet?`,
+ { true: 'it is a fact about the one this page is about, such as its name, price, date, code or status', false: 'it is a heading, a button, a menu, boilerplate, the same words on every page, or a fact about a DIFFERENT thing shown alongside such as a related or recently viewed one' })
+ q2[`k${b.n}`] = jev.choice(VALUE_KINDS, `On the page of ${item}, the text "${text}" is what kind of value?`)
+ }
+ const r2 = await ask({ state: pageState(one, want ? { what_they_want: want } : {}), questions: q2 })
+
+ // A page prints the same value more than once: "£45.17", "Price (excl. tax): £45.17",
+ // "Price (incl. tax): £45.17". Three columns of the same number is not a spreadsheet, and it
+ // makes Jev choose between identical texts on every page, which is where the low confidences
+ // come from. Group by the value each column would actually pull, and keep one of each.
+ const groups = new Map()
+ for (const b of one.blocks) {
+ const worth = r2.answers[`w${b.n}`]?.noul ?? 0
+ if (worth < 0.6) continue
+ const label = b.text.includes(': ') ? b.text.slice(0, b.text.indexOf(': ')).trim() : ''
+ const vk = r2.answers[`k${b.n}`]?.choice ?? 'other'
+ if (!label && vk === 'other') continue // nothing to call it, and nothing it is
+ const value = (label ? b.text.slice(label.length + 2) : b.text).trim()
+ const key = value.toLowerCase().replace(/[^a-z0-9]+/g, '')
+ if (!key) continue
+ const entry = groups.get(key) ?? { best: null, label: '', kind: vk }
+ if (label && !entry.label) entry.label = label
+ if (!entry.best || worth > entry.best.worth) entry.best = { text: b.text, worth, label, vk, value }
+ groups.set(key, entry)
+ }
+ const seen = new Set()
+ const columns = []
+ for (const { best, label } of groups.values()) {
+ // The name is the page's own label where there is one, so the spreadsheet reads like the site.
+ const name = (best.label || label || best.vk).slice(0, 40)
+ const id = name.toLowerCase().replace(/[^a-z0-9]+/g, '_').replace(/^_+|_+$/g, '').slice(0, 32)
+ if (!id || seen.has(id)) continue
+ seen.add(id)
+ // Ask for the bare value, not the label-and-value line: a column should hold £45.17, not
+ // "Price (excl. tax): £45.17".
+ const ask = best.label ? `the ${best.label}` : VALUE_KINDS[best.vk]
+ columns.push({ id, name, ask, sample: best.value.slice(0, 80), p: Math.round(best.worth * 100) / 100, kind: best.vk })
+ }
+ columns.sort((a, b) => b.p - a.p)
+ say('proposed', { columns: columns.length, latencyMs: r1.latencyMs + r2.latencyMs })
+ return { kind, item, columns: columns.slice(0, LIMITS.fields), from: one.url, ms: Math.round(r1.latencyMs + r2.latencyMs) }
+}
+
/**
* Run a job. Every step is reported through `onEvent` so the pane can show it live.
* @param {object} o
@@ -191,6 +311,15 @@ export async function runJob({ chrome, job, ask, onEvent, stopped = () => false
} catch (e) { out.errors.push({ url: link.url, message: String(e.message ?? e) }); say('trouble', { url: link.url, message: String(e.message ?? e) }); continue }
out.pagesRead++
say('read', { url: itemPage.url, title: itemPage.title, links: itemPage.links.length, blocks: itemPage.blocks.length })
+ // A site can serve its list happily and then challenge every page under it. Without this the
+ // challenge page's words become a blank row and the run looks like it worked.
+ const itemWall = wallReason(itemPage)
+ if (itemWall) {
+ out.walled = `${new URL(itemPage.url).hostname}: ${itemWall}`
+ out.errors.push({ url: itemPage.url, message: out.walled })
+ say('walled', { url: itemPage.url, message: out.walled })
+ break
+ }
const row = await collect(itemPage, link.label)
if (row) say('row', { row, of: out.rows.length })
}
diff --git a/store/agents/jev-browser/viewer/index.html b/store/agents/jev-browser/viewer/index.html
index 63121a9b3..ef3990d5b 100644
--- a/store/agents/jev-browser/viewer/index.html
+++ b/store/agents/jev-browser/viewer/index.html
@@ -45,18 +45,21 @@
<img id="screen" alt="">
<div id="screenIdle" class="screen-idle">
<form id="askForm" class="ask">
- <div class="ask-head"><b>What do you want off the web?</b><span>Fill this in and it goes. A real Chrome opens and reads the pages for you.</span></div>
+ <div class="ask-head"><b>What do you want off the web?</b><span>An address and a sentence is enough. Jev reads the page, works out the columns in about a second, and goes.</span></div>
<label class="ask-row"><span>Start on</span>
<input id="askStart" type="text" spellcheck="false" autocomplete="off" placeholder="books.toscrape.com/catalogue/category/books/travel_2/index.html">
</label>
<label class="ask-row"><span>Search for <i>optional</i></span>
<input id="askSearch" type="text" spellcheck="false" autocomplete="off" placeholder="leave empty if the address is already the list you want">
</label>
- <label class="ask-row"><span>Each one is</span>
- <input id="askItem" type="text" spellcheck="false" autocomplete="off" placeholder="a book for sale">
+ <label class="ask-row"><span>What you want</span>
+ <input id="askWant" type="text" spellcheck="false" autocomplete="off" placeholder="what each book costs and whether it is in stock">
</label>
- <label class="ask-row tall"><span>Columns<i>one per line</i></span>
- <textarea id="askColumns" spellcheck="false" placeholder="the title&#10;the price&#10;how many copies are available&#10;Is it in stock?"></textarea>
+ <label class="ask-row"><span>Each one is <i>optional</i></span>
+ <input id="askItem" type="text" spellcheck="false" autocomplete="off" placeholder="Jev works this out from the page">
+ </label>
+ <label class="ask-row tall"><span>Columns<i>optional, one per line</i></span>
+ <textarea id="askColumns" spellcheck="false" placeholder="Leave this empty and Jev picks the columns off the page.&#10;&#10;Or name them yourself, one per line:&#10;the title&#10;the price&#10;Is it in stock?"></textarea>
</label>
<label class="ask-row"><span>Only keep <i>optional</i></span>
<input id="askKeep" type="text" spellcheck="false" autocomplete="off" placeholder="only the ones under £30">
diff --git a/store/agents/jev-browser/viewer/page.mjs b/store/agents/jev-browser/viewer/page.mjs
index 9c3e814ad..dc1236609 100644
--- a/store/agents/jev-browser/viewer/page.mjs
+++ b/store/agents/jev-browser/viewer/page.mjs
@@ -73,26 +73,38 @@ export const READER = `(() => {
const FURNITURE = 'header, footer, nav, aside, [role="banner"], [role="contentinfo"], [role="navigation"], [role="search"]'
const LABELLED = /^(dt|th)$/i // a label whose value sits in the element next to it
const SPLIT = /\\s+[·|•\\u2014\\u2013\\u2022]\\s+/
- const blocks = [], sawText = new Set(), consumed = new Set()
+ const MAIN = 'main, article, [role="main"], #main, #content, .content'
+ const found = [], sawText = new Set(), consumed = new Set()
const ownText = (e) => tidy(Array.from(e.childNodes).filter((c) => c.nodeType === 3).map((c) => c.textContent).join(' '), MAX_BLOCK)
+ const allText = (e) => tidy(e.innerText || e.textContent, MAX_BLOCK + 1)
+ /** Does a child element carry text of its own? Then that child is the block, not this one. */
+ const hasTextChild = (e) => { for (const kid of e.children) if (allText(kid).length >= 2) return true; return false }
// Numbers a person would want on their own: "In stock (19 available)" also offers "19".
const numbersIn = (t) => (t.match(/\\d[\\d,.]*/g) || []).map((x) => x.replace(/[.,]$/, '')).filter((x) => x.length && x !== t).slice(0, 2)
+ const main = document.querySelector(MAIN)
+ let scanned = 0
const walker = document.createTreeWalker(document.body || document.documentElement, NodeFilter.SHOW_ELEMENT)
- for (let e = walker.currentNode; e && blocks.length < MAX_BLOCKS; e = walker.nextNode()) {
- if (/^(script|style|noscript|svg|head|meta|link)$/i.test(e.tagName)) continue
+ for (let e = walker.currentNode; e && scanned < 6000; e = walker.nextNode()) {
+ scanned++
+ if (/^(script|style|noscript|svg|head|meta|link|template)$/i.test(e.tagName)) continue
if (consumed.has(e)) continue
- let text = ownText(e)
- let label = ''
- // A label and its value belong together, or the value gets dropped as a repeat of one further up
- // and the label is left looking like an empty promise.
+ if (e.closest(FURNITURE)) continue
+ let text = '', label = ''
+ // A label and its value belong together, or the value gets dropped as a repeat of one further
+ // up and the label is left looking like an empty promise.
if (LABELLED.test(e.tagName)) {
const next = e.nextElementSibling
- const value = next && /^(dd|td)$/i.test(next.tagName) ? ownText(next) : ''
- if (text && value) { consumed.add(next); label = text; text = tidy(text.replace(/:\\s*$/, '') + ': ' + value, MAX_BLOCK) }
+ const mine = allText(e), value = next && /^(dd|td)$/i.test(next.tagName) ? allText(next) : ''
+ if (mine && value) { consumed.add(next); label = mine; text = tidy(mine.replace(/:\\s*$/, '') + ': ' + value, MAX_BLOCK) }
+ }
+ // Otherwise: the innermost element that carries text. A modern page wraps its words several
+ // elements deep, so this is the only place the value really is.
+ if (!text) {
+ if (hasTextChild(e)) continue
+ text = ownText(e) || allText(e)
}
- if (text.length < 2) continue
+ if (text.length < 2 || text.length > MAX_BLOCK) continue
if (!vis(e)) continue
- if (e.closest(FURNITURE)) continue
const key = text.toLowerCase()
if (sawText.has(key)) continue
sawText.add(key)
@@ -100,8 +112,13 @@ export const READER = `(() => {
if (label) parts = [text.slice(label.replace(/:\\s*$/, '').length + 2)]
parts = parts.map((p) => p.trim()).filter((p) => p.length > 1 && p.length < 80)
for (const p of [...parts, text]) for (const num of numbersIn(p)) if (!parts.includes(num)) parts.push(num)
- blocks.push({ n: mark(e), text, tag: e.tagName.toLowerCase(), parts: parts.slice(0, 5) })
+ found.push({ e, text, tag: e.tagName.toLowerCase(), parts: parts.slice(0, 5), inMain: !!main && main.contains(e) })
}
+ // What the page is about comes before the furniture the site puts round it.
+ found.sort((a, b) => (b.inMain ? 1 : 0) - (a.inMain ? 1 : 0))
+ const blocks = found.slice(0, MAX_BLOCKS).map((b) => ({ n: mark(b.e), text: b.text, tag: b.tag, parts: b.parts }))
+ blocks.sort((a, b) => a.n - b.n)
+
const heads = Array.from(document.querySelectorAll('h1, h2')).map((h) => tidy(h.innerText, 120)).filter(Boolean).slice(0, 6)
return {
@@ -112,10 +129,19 @@ export const READER = `(() => {
}
})()`
-/** Read the live page. Returns the shape above, with `path` selectors ready for a click. */
-export async function readPage(chrome) {
- const page = await chrome.evaluate(READER)
+/**
+ * Read the live page. A page that builds itself in the browser can look finished while it is still
+ * filling in, and a read a moment too early comes back nearly empty. So a thin first read is given
+ * one more second and tried again, which is the difference between a column and a blank column.
+ */
+export async function readPage(chrome, { retry = true } = {}) {
+ let page = await chrome.evaluate(READER)
if (!page) throw new Error('the page could not be read')
+ if (retry && page.blocks.length < 8 && page.links.length < 8) {
+ await chrome.settle(4000)
+ const again = await chrome.evaluate(READER)
+ if (again && again.blocks.length + again.links.length > page.blocks.length + page.links.length) page = again
+ }
for (const list of [page.links, page.controls, page.blocks]) for (const row of list) row.css = `[data-jev-n="${row.n}"]`
return page
}
diff --git a/store/agents/jev-browser/viewer/studio.js b/store/agents/jev-browser/viewer/studio.js
index 6ec9efc01..f68625de1 100644
--- a/store/agents/jev-browser/viewer/studio.js
+++ b/store/agents/jev-browser/viewer/studio.js
@@ -41,6 +41,7 @@ function renderJob() {
chip.title = f.ask
fields.append(chip)
}
+ if (!S.fields.length && S.proposing) fields.append(h('span', 'job-line dim', 'Jev is reading the page to work them out…'))
box.append(fields)
if (S.keep) box.append(h('div', 'job-line keep', `keeps only: ${S.keep}`))
box.append(h('div', 'job-line dim', `at most ${fmtN(S.maxItems)} things from ${fmtN(S.maxPages)} pages`))
@@ -120,7 +121,7 @@ function renderFeed() {
function renderTop() {
const p = S.progress
tw.rows.t = p.rows; tw.pages.t = p.pages; tw.links.t = p.links; tw.rate.t = p.perSec; tw.cost.t = p.costUsd
- $('phasePill').textContent = S.phase === 'running' ? 'browsing' : S.phase
+ $('phasePill').textContent = S.proposing ? 'reading the page' : S.phase === 'running' ? 'browsing' : S.phase
$('phasePill').className = 'pill ' + (S.phase === 'running' ? 'ok' : S.phase === 'done' ? 'ok' : S.phase === 'stopped' ? 'warn' : '')
$('startBtn').classList.toggle('hidden', S.phase === 'running')
$('stopBtn').classList.toggle('hidden', S.phase !== 'running')
@@ -176,6 +177,7 @@ function showAsk(on) {
$('askStart').value = S?.demoUrl && S.start === S.demoUrl ? '' : (S?.start ?? '')
$('askSearch').value = S?.search ?? ''
$('askItem').value = S?.item && S.item !== 'one of the things to collect' ? S.item : ''
+ $('askWant').value = S?.want ?? ''
$('askColumns').value = (S?.fields ?? []).map((f) => f.ask).join('\n')
$('askKeep').value = S?.keep ?? ''
$('askMax').value = S?.fields.length ? S.maxItems : 25
@@ -185,12 +187,12 @@ function showAsk(on) {
}
async function sendJob(over = {}) {
const body = {
- start: $('askStart').value.trim(), search: $('askSearch').value.trim(), item: $('askItem').value.trim(),
+ start: $('askStart').value.trim(), search: $('askSearch').value.trim(), item: $('askItem').value.trim(), want: $('askWant').value.trim(),
columns: $('askColumns').value, keep: $('askKeep').value.trim(), maxItems: Number($('askMax').value) || 25,
...over,
}
$('askMsg').className = 'ask-msg'
- $('askMsg').textContent = 'Setting the job…'
+ $('askMsg').textContent = body.columns.trim() ? 'Setting the job…' : 'Reading the page to work out the columns…'
$('askSubmit').disabled = true
const r = await post('setJob', body)
$('askSubmit').disabled = false
@@ -200,9 +202,7 @@ async function sendJob(over = {}) {
}
$('askForm').addEventListener('submit', (e) => { e.preventDefault(); sendJob() })
$('askDemo').addEventListener('click', () => sendJob({
- start: 'demo', search: '', item: 'a job posting',
- columns: 'the job title\nthe name of the company hiring\nthe pay or salary range\nwhere the job is based\nCan this job be done fully remotely?',
- keep: '',
+ start: 'demo', search: '', item: '', want: 'what each role pays and where it is', columns: '', keep: '',
}))
$('editBtn').addEventListener('click', () => showAsk(!askOpen))
diff --git a/store/agents/jev-browser/viewer/viewer.mjs b/store/agents/jev-browser/viewer/viewer.mjs
index fb2743afb..9287c578d 100644
--- a/store/agents/jev-browser/viewer/viewer.mjs
+++ b/store/agents/jev-browser/viewer/viewer.mjs
@@ -9,7 +9,7 @@ import { writeFileSync, renameSync, mkdirSync } from 'node:fs'
import { evaluate, PRICE_PER_MTOK, resolveCredentials, describeCredentials } from '../toolchain/jev.mjs'
import { serveViewer, writeVerdict, watchConfig, clean } from './kit.mjs'
import { openChrome, findChrome, Refused } from '../toolchain/chrome.mjs'
-import { normalizeJob, runJob, LIMITS } from './crawl.mjs'
+import { normalizeJob, runJob, proposeJob, LIMITS } from './crawl.mjs'
import { browserMock } from './mock.mjs'
import { startDemoSite } from './demosite.mjs'
@@ -23,7 +23,7 @@ export async function startBrowserViewer({ workspace, port = 0, show = null } =
let job = normalizeJob({}).job, configError = null, jevError = null
let rows = [], feed = [], links = [], here = { url: '', title: '' }
let counters = { pages: 0, calls: 0, questions: 0, tokens: 0, costUsd: 0, links: 0, errors: 0 }
- let phase = 'idle', startedAt = 0, finishedAt = 0, lastRun = null
+ let phase = 'idle', startedAt = 0, finishedAt = 0, lastRun = null, proposing = false
let shotTimer = null, shotBusy = false
let expecting = { expecting: 'list', item: '' }
const mock = browserMock(() => expecting)
@@ -33,7 +33,7 @@ export async function startBrowserViewer({ workspace, port = 0, show = null } =
// ---- what the pane sees ---------------------------------------------------------------------
const view = () => ({
task: job.task, item: job.item, start: job.start, search: job.search, fields: job.fields.map((f) => ({ id: f.id, name: f.name, ask: f.ask, type: f.type })),
- walled: lastRun?.walled ?? null,
+ walled: lastRun?.walled ?? null, want: job.want, proposing,
keep: job.keep, maxItems: job.maxItems, maxPages: job.maxPages, limits: LIMITS,
error: configError, jevError, client: liveRoute() ?? 'mock', jevSays: describeCredentials(),
chrome: { found: !!chromePath, open: !!chrome?.alive },
@@ -170,6 +170,34 @@ export async function startBrowserViewer({ workspace, port = 0, show = null } =
const opened = await openBrowser()
if (opened.ok === false) return opened
rows = []; links = []; feed = []
+ // Nobody should have to name the columns. Jev reads one page and proposes them, in about a
+ // second, and the answer is written back into browse.json so the job stays the recipe.
+ if (!job.fields.length) {
+ proposing = true
+ say('Reading the page to work out the columns…', 'start')
+ counters = { pages: 0, calls: 0, questions: 0, tokens: 0, costUsd: 0, links: 0, errors: 0 }
+ startedAt = Date.now(); finishedAt = 0; phase = 'running'
+ push()
+ let got
+ try {
+ got = await proposeJob({ chrome, ask, start: job.start, want: job.want, search: job.search, onEvent: (e) => { if (e.type === 'judged') links = e.links ?? []; if (e.type === 'going') here = { url: e.url, title: '' }; push('view') } })
+ } catch (e) { got = { columns: [], error: String(e.message ?? e) } }
+ proposing = false
+ if (got.walled) { lastRun = { walled: got.walled, errors: [] }; say(got.walled, 'bad'); phase = 'done'; finishedAt = Date.now(); push(); saveVerdict(); return { ok: false, error: got.walled } }
+ if (!got.columns.length) {
+ phase = 'idle'; finishedAt = Date.now()
+ const why = got.error || got.none || 'nothing on that page looked like a value worth a column'
+ say(`No columns could be worked out: ${why}`, 'bad')
+ push(); saveVerdict()
+ return { ok: false, error: `${why}. Write the columns yourself, one per line.` }
+ }
+ const saidItem = String(watcher.get()?.item ?? '').trim()
+ const next = { ...watcher.get(), item: saidItem || got.item, fields: got.columns.map((c) => ({ id: c.id, name: c.name, ask: c.ask })) }
+ writeMarker(next)
+ applyJobQuiet(next)
+ say(`Jev found ${got.columns.length} columns in ${got.ms} ms: ${got.columns.map((c) => c.name).slice(0, 6).join(', ')}`, 'judge')
+ push()
+ }
counters = { pages: 0, calls: 0, questions: 0, tokens: 0, costUsd: 0, links: 0, errors: 0 }
stopFlag = false; phase = 'running'; startedAt = Date.now(); finishedAt = 0
say(`Starting: ${job.task}`, 'start')
@@ -223,19 +251,26 @@ export async function startBrowserViewer({ workspace, port = 0, show = null } =
case 'export': { saveResults(); return { file: RESULTS, rows: rows.length } }
// The pane's own form. It writes browse.json exactly as the agent would, so the file stays
// the one place a job lives, and the recipe is the same however it was made.
+ case 'propose': {
+ // Just the proposal, without running: the pane can show it and let a person untick.
+ if (!chrome?.alive) { const o = await openBrowser(); if (o.ok === false) return o }
+ proposing = true; push()
+ try { return await proposeJob({ chrome, ask, start: String(body.start ?? job.start), want: String(body.want ?? job.want ?? ''), search: String(body.search ?? '') }) }
+ catch (e) { return { ok: false, error: String(e.message ?? e) } } finally { proposing = false; push() }
+ }
case 'setJob': {
const columns = String(body.columns ?? '').split('\n').map((l) => l.trim()).filter(Boolean).slice(0, LIMITS.fields)
- if (!columns.length) return { ok: false, error: 'Say what you want for each one, one per line.' }
const start = String(body.start ?? '').trim()
if (!start) return { ok: false, error: 'Put in the address of the page to start on.' }
const next = {
- task: String(body.task ?? '').trim() || `${columns[0]} for every ${String(body.item ?? 'thing').trim()}`,
+ task: String(body.task ?? '').trim() || String(body.want ?? '').trim() || (columns[0] ? `${columns[0]} for every ${String(body.item ?? 'thing').trim()}` : `Read ${start.replace(/^https?:\/\//, '').split('/')[0]}`),
start: /^[a-z]+:\/\//i.test(start) || /^demo$/i.test(start) ? start : `https://${start}`,
search: String(body.search ?? '').trim(),
- item: String(body.item ?? '').trim() || 'one of the things on this page',
+ item: String(body.item ?? '').trim(),
fields: columns.map((ask) => (/^(is|are|does|do|can|has|have|will|should|was|were)\b/i.test(ask) || ask.endsWith('?')
? { ask, type: 'yesno', name: ask.replace(/\?$/, '').slice(0, 40) }
: { ask, name: ask.slice(0, 40) })),
+ want: String(body.want ?? '').trim(),
keep: String(body.keep ?? '').trim(),
maxItems: Math.max(1, Math.min(LIMITS.maxItems, Math.floor(Number(body.maxItems) || 25))),
maxPages: Math.max(1, Math.min(LIMITS.maxPages, Math.floor(Number(body.maxPages) || 5))),
@@ -243,8 +278,7 @@ export async function startBrowserViewer({ workspace, port = 0, show = null } =
}
const { errors } = normalizeJob(next)
if (errors.length) return { ok: false, error: errors.join('; ') }
- const f = join(workspace, MARKER)
- writeFileSync(f + '.tmp', JSON.stringify(next, null, 2) + '\n'); renameSync(f + '.tmp', f)
+ writeMarker(next)
applyJob(next) // this starts it
return { ok: true, start: next.start, fields: next.fields.length }
}
@@ -263,6 +297,20 @@ export async function startBrowserViewer({ workspace, port = 0, show = null } =
const jobPrint = (j) => JSON.stringify([j.start, j.search, j.item, j.keep, j.fields.map((f) => [f.id, f.ask, f.type, f.levels])])
let lastPrint = null, autoTimer = null
+ /** browse.json is the recipe, so anything the pane works out is written back into it. */
+ function writeMarker(obj) {
+ const f = join(workspace, MARKER)
+ writeFileSync(f + '.tmp', JSON.stringify(obj, null, 2) + '\n'); renameSync(f + '.tmp', f)
+ }
+ /** Take a job on without setting it running: used from inside a run that is already going. */
+ function applyJobQuiet(raw) {
+ const { job: next, errors } = normalizeJob(raw)
+ configError = errors.length ? `${MARKER}: ${errors.join('; ')}` : null
+ job = demo && /^demo$/i.test(next.start) ? { ...next, start: demo.url, hosts: [...new Set([...next.hosts, '127.0.0.1', 'localhost'])] } : next
+ if (chrome) chrome.allowedHosts = job.sameSiteOnly ? [...job.hosts, ...(demo ? ['127.0.0.1', 'localhost'] : [])] : []
+ lastPrint = jobPrint(job)
+ }
+
function applyJob(raw, fresh = false) {
const { job: next, errors } = normalizeJob(raw)
configError = errors.length ? `${MARKER}: ${errors.join('; ')}` : null
diff --git a/store/showcase/jev-browser/jev-picks-the-columns.jpg b/store/showcase/jev-browser/jev-picks-the-columns.jpg
new file mode 100644
index 000000000..fd6085b85
Binary files /dev/null and b/store/showcase/jev-browser/jev-picks-the-columns.jpg differ