Lightpanda: the headless browser designed for AI and automation
removed in favor of https://lightpanda.io/docs/usage/pandascript
Pierre Tachoiref2af841f1fb4
diff --git a/docs/agent-script.md b/docs/agent-script.mddeleted file mode 100644index 1afa501c4..000000000--- a/docs/agent-script.md+++ /dev/null@@ -1,443 +0,0 @@-# Agent JavaScript scripts--`lightpanda agent <script.js>` runs a JavaScript file that drives a-Lightpanda browser session through a small object-oriented API. The format is-intentionally plain JavaScript: use normal variables, functions, loops,-objects, arrays, `JSON.parse`, `JSON.stringify`, and other standard ECMAScript-built-ins.--```console-./lightpanda agent session.js-```--The interactive REPL still uses slash commands. `/save` writes the recorded-browser actions as JavaScript calls so the saved file can be replayed without-an LLM.--## Runtime Environment--Agent scripts run in their own V8 context. That context is separate from the-web page's JavaScript context.--- `Page` is the only global. `new Page()` makes a page object and- `await page.goto(url)` navigates it; every other primitive is a **method on- that page**: `const page = new Page(); await page.goto(url);- page.extract({...}); page.click(sel);`.-- It is not the browser page environment. There is no `window`, `document`,- DOM, `localStorage`, `navigator`, or page global state in the agent script.- Read page data with `page.extract(...)`, or explicitly run page JavaScript- with `page.evaluate(...)` when that is the right tool.-- It is not Node.js. There is no `require`, `process`, `fs`, `path`, npm- package loading, command-line argument API, or Node network/filesystem API.-- Page scripts cannot see agent variables or Lightpanda primitives. Agent- scripts cannot directly see page variables.-- The `page.evaluate(...)` method runs JavaScript in the page context,- distinct from the agent context's own native `eval`.-- Agent variables persist for the lifetime of one script run, across- navigations and primitive calls. A later `lightpanda agent script.js` run- starts with a fresh agent context.-- `page.goto(...)` is **async — always `await` it**. Every other page method- is **synchronous**: write `const data = page.extract({...})`, never- `await page.extract(...)`. The script body runs as an async function, so- top-level `await` is allowed.-- **Re-navigating reuses the same page.** `await page.goto(url2)` keeps `page`- valid and points it at the new URL. Work through one page at a time and read- it before navigating away.-- Tool failures throw JavaScript `Error` exceptions and stop execution unless- you catch them.-- **`return <value>` is the script's output**, printed automatically (objects- and arrays as JSON; other values coerced). End a script with the value you- want as output, e.g. `return page.extract({ ... });` or `return results;`. A- bare trailing expression is NOT printed, and neither is- `console.log(JSON.stringify(...))`.--The agent context includes a small `console` object:--```js-console.log("printed to stdout");-console.info("printed to stdout");-console.debug("printed to stdout");-console.warn("printed to stderr");-console.error("printed to stderr");-```--## Values And Return Types--Most page methods return the browser tool's result text as a JavaScript-string. `page.extract(...)` is the exception: it returns extracted data as a-normal JavaScript value, so local script logic can use it directly. The result-mirrors your schema — an object schema returns an object keyed by your fields-(even with a single field), and a bare array schema returns an array:--```js-const page = new Page();-await page.goto("https://news.ycombinator.com/");--const data = page.extract({- title: "title",- stories: [{- selector: "tr.athing",- limit: 5,- fields: {- id: { attr: "id" },- title: ".titleline > a"- }- }]-});--return data; // printed automatically as JSON-```--Destructure when a single field is all you need:--```js-const { stories } = page.extract({- stories: [{- selector: "tr.athing",- limit: 5,- fields: {- id: { attr: "id" },- title: ".titleline > a"- }- }]-});--for (const story of stories) {- console.log(story.title);-}-```--`page.evaluate(...)` returns the page evaluate tool result text. When page-`evaluate(...)` returns an object or array, that text is JSON.--Primitive arguments must be JSON-serializable. Strings, numbers, booleans,-arrays, plain objects, and `null` work. `undefined`, functions, symbols, and-cyclic objects do not.--## The Page object--`Page` is the only global. `new Page()` makes a page object; everything else is-a method on it. Call `Page(...)` without `new` and you get-`Page must be called with new`.--| Method | Arguments | Runs in |-|--------|-----------|---------|-| `new Page()` | — | Makes a page object. No navigation yet — call `page.goto(url)` before any other method. |-| `page.goto` | `goto(url[, { timeout }])` | **Async — must be `await`ed.** Navigates the page (re-navigating reuses the same object). |-| `page.close` | `close()` | Marks the page done; later method calls on it error. The page is otherwise reclaimed at script end. |-| `page.extract` | `extract(schema)` or `extract({ schema })` | Browser page via extractor; returns a JS object or array |-| `page.evaluate` | `evaluate(script[, { url, timeout, save }])` | Browser page JS context |-| `page.click` | `click(selector)` or `click({ selector })` | Browser page |-| `page.fill` | `fill(selector, value)` or `fill({ selector, value })` | Browser page |-| `page.scroll` | `scroll()` or `scroll({ x, y })` | Browser page |-| `page.waitForSelector` | `waitForSelector(selector[, { timeout }])` | Browser page |-| `page.waitForScript` | `waitForScript(script[, { timeout }])` | Browser page JS context |-| `page.waitForState` | `waitForState(state[, { timeout }])` | Browser page |-| `page.hover` | `hover(selector)` or `hover({ selector })` | Browser page |-| `page.press` | `press(selector, key)` or `press({ key[, selector] })` | Browser page |-| `page.selectOption` | `selectOption(selector, value)` or `selectOption({ selector, value })` | Browser page |-| `page.setChecked` | `setChecked(selector[, checked])` or `setChecked({ selector, checked })` | Browser page |--`page.goto` returns at the `load` event (a fast snapshot). When a page's-content is still loading (rendered by post-load JS), call-`page.waitForState("networkidle")` before reading. `waitForState`'s `state`-accepts `"load"`, `"domcontentloaded"`, `"networkalmostidle"`,-`"networkidle"`, `"done"`. `goto`'s `timeout` defaults to 10000 ms; the-`waitFor*` timeouts default to 5000 ms.--The `[, { … }]` is an optional trailing options object: leading arguments are-positional (`page.waitForSelector("#row", { timeout: 2000 })`), and the-options ride in a final object. Passing a single object with everything-(`page.waitForSelector({ selector: "#row", timeout: 2000 })`) is equivalent —-that's the shape `/save` records into saved scripts. An option can't be a bare-positional, though: `page.waitForSelector("#row", 2000)` is an error. A `null`-positional omits that field (`page.press(null, "Enter")` presses on the-focused element), and setting the same field positionally and in the options-object (`page.goto(url, { url: ... })`) is an `invalid arguments` error.--Page methods address elements by CSS selector only. The tools that hand out-`backendNodeId`s (`tree`, `findElement`, `nodeDetails`) aren't installed in-the script context, and a raw node ID wouldn't survive replay anyway. When-you're exploring in the REPL and have a `backendNodeId` — e.g. the leading-number on a `/tree` line, or a `/findElement` hit — run `/nodeDetails-backendNodeId=<id>` to get a durable CSS `selector`, then paste that into your-script.--## Navigation--Use `page.goto(...)` to open a page. It is the only async method, so always-`await` it:--```js-const page = new Page();-await page.goto("https://example.com");--await page.goto({- url: "https://example.com/app",- timeout: 15000-});-```--Re-navigating reuses the same page object: a later `await page.goto(url2)`-keeps `page` valid and points it at the new URL. Read a page before navigating-away from it.--The call returns a status string and throws if navigation fails. A timeout-does **not** throw: the call returns `"Navigation started but the page did not-finish loading before the timeout."` and the page stays in whatever state it-reached. Check the return value — or follow with `page.waitForState(...)` /-`page.waitForSelector(...)` — when completeness matters.--## Structured Extraction--Use `page.extract(...)` to read data from the current page without writing-page-side JavaScript. This is the preferred bridge from page content into-local agent logic.--```js-const result = page.extract({- heading: "h1",- links: [{- selector: "a",- limit: 10,- fields: {- text: "",- href: { attr: "href" }- }- }]-});-```--The schema forms are:--| Schema value | Meaning |-|--------------|---------|-| `"<selector>"` | Text of the first matching element, or `null` |-| `""` | Text of the current matched element inside a `fields` block |-| `["<selector>"]` | Text of all matching elements |-| `{ selector: "<selector>", attr: "<name>" }` | Attribute from the first match |-| `[{ selector: "<selector>", attr: "<name>" }]` | Attribute from all matches |-| `[{ selector: "<selector>", fields: { ... } }]` | Array of records, with fields resolved relative to each matched element |-| `limit: N` | Cap array extraction to `N` matches |--Return shape follows the top-level schema:--- `page.extract({ title: "h1" })` returns `{ title: "..." }`.-- `page.extract({ title: "h1", links: [{ selector: "a" }] })` returns an- object with both fields.-- `page.extract({ links: [{ selector: "a" }] })` returns `{ links: [...] }` —- an object schema always returns an object, even with a single field.-- `page.extract([{ selector: "a" }])` is shorthand for a single anonymous- array extraction and returns the array directly.--Every value is a string (trimmed text or a raw attribute) or `null` — parse-numbers in script logic. An array field that matches nothing yields `[]`-without complaint (a page with zero comments is a valid result), but if-*every* field in the schema misses, `page.extract(...)` throws-`no schema selector matched any element` — treat that as "my selectors are-wrong", not "the page is empty".--`page.extract(...)` reads only the current page. For list-to-detail scraping —-capture a list, then visit each row for more — capture the list, then loop in-the script: `await page.goto` each row's URL and `page.extract` the detail,-reading each detail page before navigating to the next. The local agent-context keeps the data across navigations, so the assembly happens in plain-JavaScript. See the [complete example](#complete-example) below.--When passing an object directly to `page.extract(...)`, the runtime serializes-it as the extractor schema. These forms are equivalent:--```js-page.extract({ title: "h1" });-page.extract({ schema: { title: "h1" } });-page.extract('{ "title": "h1" }');-```--The wrapped form accepts only `schema`: the REPL's `save=` option does not-exist in scripts (`page.extract({ schema: ..., save: ... })` is rejected).-Keep results in local variables instead.--Use local variables to keep extracted data available to later script logic:--```js-const head = page.extract({ title: "title" });-```--## Page JavaScript--`page.evaluate(...)` is the explicit escape hatch into the current page's-JavaScript context. Its script string runs where `window` and `document`-exist.--```js-const page = new Page();-await page.goto("https://example.com");--const title = page.evaluate("document.title");-console.log(title);-```--Keep the boundary clear:--```js-const selector = "h1";--// Good: local agent logic builds an extract schema.-const data = page.extract({ heading: selector });--// Bad: page evaluate cannot see local agent variables.-page.evaluate("document.querySelector(selector).textContent");-```--`page.evaluate(...)` cannot call `goto`, `extract`, or other agent primitives.-Agent scripts cannot access `document` directly. If you need page DOM data,-prefer `page.extract(...)`; use `page.evaluate(...)` only for page behavior-that extraction cannot express, and remember its state dies on every-navigation/reload, while script variables persist.--`page.waitForScript(...)` also evaluates in the page context, repeatedly,-until the expression is truthy or the timeout expires:--```js-page.waitForScript("document.querySelectorAll('.row').length >= 5");-```--## Interaction Primitives--The action methods operate on the current page. Most take one object whose-fields match the browser tool schema:--```js-page.click({ selector: "a.login" });-page.fill({ selector: "input[name='acct']", value: "$LP_HN_USERNAME" });-page.fill({ selector: "input[name='pw']", value: "$LP_HN_PASSWORD" });-page.press({ key: "Enter" });--page.waitForSelector("#logout");--page.hover({ selector: "#menu" });-page.selectOption({ selector: "select[name='country']", value: "FR" });-page.setChecked({ selector: "input[name='terms']", checked: true });-page.setChecked({ selector: "input[name='newsletter']", checked: false });--page.scroll({ y: 600 });-page.scroll();-```--`page.setChecked` defaults `checked` to `true` when the field is omitted-(`page.setChecked("#chk")` checks the box). `page.press`'s leading positional-is the optional `selector`, not `key`: a bare `page.press("Enter")` binds-`"Enter"` to `selector` and fails. Press on the focused element with-`page.press({ key: "Enter" })` or `page.press(null, "Enter")`; target an-element with `page.press("#search", "Enter")` or-`page.press({ key: "Enter", selector: "#search" })`.--`$LP_*` placeholders in string arguments are resolved inside the Lightpanda-process. This keeps credentials out of recorded scripts and LLM prompts. In-recordings, resolved `LP_*` values are scrubbed back to placeholders.--## Recording Format--The REPL remains slash-command based:--```text-> /goto https://example.com-> /click selector='a.login'-> /save-```--`/save` writes JavaScript by default:--```js-const page = new Page();-await page.goto("https://example.com");-page.click({ selector: "a.login" });-```--Only replayable browser actions are recorded:--- Recorded: `goto`, `click`, `fill`, `scroll`, `hover`, `press`,- `selectOption`, `setChecked`, `waitForSelector`, `waitForScript`,- `waitForState`, `evaluate`, and `extract` — the same set installed as page- methods.-- Not recorded: read-only exploration tools such as `tree`, `markdown`, `html`,- `links`, `findElement`, `consoleLogs`, `getUrl`, `getCookies`, and `getEnv`.-- Natural-language prompts are written as `//` comments above the actions they- produced.--This is the deterministic `--no-llm` transcription. When `/save` runs with an-LLM it instead synthesizes an idiomatic script from the session and is asked to-emit JavaScript only, so those `//` comments generally won't appear.--## Error Handling--Primitive failures throw JavaScript exceptions:--```js-try {- page.waitForSelector({ selector: "#dashboard", timeout: 1000 });-} catch (err) {- console.error("dashboard did not appear:", err.message);- throw err;-}-```--Common failures:--| Error | Meaning |-|-------|---------|-| `extract is not defined` (or `click`/`fill`/…) | These are methods on the page object, not globals. Use `const page = new Page(); await page.goto(url); page.extract(...)`. |-| `Page must be called with new` | `Page(...)` was called without `new`. Use `const page = new Page();`. |-| `page is not navigated or has been closed; call page.goto(url) first` | A method ran on a fresh `new Page()` (or a closed page). `await page.goto(url)` first. |-| `page handle is no longer valid` | A page was used after a later `goto` replaced it. Read a page before navigating away. |-| `ReferenceError: document is not defined` | You tried to use browser DOM APIs in the agent context. Use `page.extract(...)` or `page.evaluate(...)`. |-| `ReferenceError: require is not defined` | Agent scripts are not Node.js scripts. |-| `no page loaded - run page.goto(url) first` | A page method ran before navigation. |-| `invalid arguments` | A method received the wrong number or shape of arguments, or a non-JSON-serializable value. |-| `extract: no schema selector matched any element` | Every field in the schema missed. Fix the selectors; an empty page section yields `null`/`[]` per field, not this error. |--## Complete Example--This script opens Hacker News, extracts five stories, visits each comments-page, and prints one JSON object. A single reusable `page` walks the list-sequentially — read each detail page before navigating to the next. The-looping and object assembly happen in the local agent script, not in the page.--```js-const HN = "https://news.ycombinator.com";--const page = new Page();-await page.goto(HN);--const { stories } = page.extract({- stories: [{- selector: "tr.athing",- limit: 5,- fields: {- id: { attr: "id" },- title: ".titleline > a",- url: { selector: ".titleline > a", attr: "href" }- }- }]-});--for (const story of stories) {- story.comments = [];- if (!story.id) continue;-- await page.goto(`${HN}/item?id=${story.id}`);- const { comments } = page.extract({- comments: [{- selector: "tr.athing.comtr:has(.commtext)",- limit: 3,- fields: {- author: ".hnuser",- text: ".commtext"- }- }]- });- story.comments = comments;-}--return stories; // printed automatically as JSON-```