Skip to content

Automate browser based workflows with AI

Let Task V3 keep perception reads by size and stop rewriting old messages every turn (#8686)

pedrohsdb4621842c5fb9
diff --git a/skyvern/forge/taskv3/engine.py b/skyvern/forge/taskv3/engine.py
index b9128932c..e6b4edf66 100644
--- a/skyvern/forge/taskv3/engine.py
+++ b/skyvern/forge/taskv3/engine.py
@@ -147,7 +147,7 @@ How to work:
- Be efficient — this is the whole point of the engine. After observing a form once, fill every field you can before doing anything that reloads the page. Minimize tool calls and turns.
- Batch aggressively: in ONE turn you can `type` into many fields AND `click` many radio/checkbox options AND `select_option` on several dropdowns. Answer a whole form section in a single turn — never spend a separate turn on each click.
- Autocomplete / typeahead / combobox fields (location, school, employer lookups) render suggestions only AFTER you type, and the raw text you type is NOT accepted until you pick a suggestion. Use the `select_combobox` tool (selector + value) for these — it types, waits for the suggestions to render, selects the best-matching one, and verifies the field committed. Do NOT `type` into them or press keys on your own initiative. If `select_combobox` returns an error, the field is genuinely unfilled — never treat it as done. Act on what that error tells you rather than substituting a value of your own: this field commits only the suggestions the page itself offers, and those are often coarser than the value you hold.
-- `observe` already gives you everything you need to fill a field (ref, label, type, current value, options, and the surrounding question text) — act on it directly. `get_html` markup is a rare last resort for ONE specific element `observe` failed to describe: NEVER read a whole page/form/section's markup, NEVER re-read the same element, and NEVER inspect more than once before acting. Its text format (the page's visible text) is the one whole-page read that is cheap and honest — use it when the goal is about what the page shows, not where a control is. A read that reports being cut is the one case where calling `get_html` again is right: it names the total size and the `offset` that continues it, so read on until you have the part you need — that is finishing ONE read, not inspecting twice. Only your last couple of reads stay in this conversation, so as you go, write down in your own words what each part told you — that is what you will still have when the earlier part is gone.
+- `observe` already gives you everything you need to fill a field (ref, label, type, current value, options, and the surrounding question text) — act on it directly. `get_html` markup is a rare last resort for ONE specific element `observe` failed to describe: NEVER read a whole page/form/section's markup, NEVER re-read the same element, and NEVER inspect more than once before acting. Its text format (the page's visible text) is the one whole-page read that is cheap and honest — use it when the goal is about what the page shows, not where a control is. A read that reports being cut is the one case where calling `get_html` again is right: it names the total size and the `offset` that continues it, so read on until you have the part you need — that is finishing ONE read, not inspecting twice. Your reads stay in this conversation until they add up to about two full-size reads; past that the oldest are replaced by a one-line note naming the read, so as you go, write down in your own words what each part told you — that is what you will still have when the earlier part is gone. A read that returns exactly the bytes of a read still shown above is replaced by a short note saying so: look at that earlier read.
- `look` is a separate last resort for when the TEXT tools are not enough: you can't tell what the page looks like, a control you expect isn't in `observe` (custom or shadow-DOM widgets), or an action isn't taking and you can't tell why. It returns ONE screenshot with every visible control boxed and numbered; then act on a number with `click(mark=N)` or `type(mark=N, text=...)`. Do NOT call `look` to double-check what `observe` already told you, and do not call it every turn — it is for when you are genuinely stuck on something visual.
- Inspecting the page does NOT progress the task — only `type`/`select_option`/`click` do. If your recent turns were mostly `observe`/`get_html` with little typing or clicking, you are stuck inspecting: stop, and fill every field you can from the latest `observe` snapshot using its refs before doing anything else.
- Before calling finish with status=completed, re-check with `observe` that the goal's effect is present in the page's SETTLED, loaded content (no loading indicators or empty panels standing in for it), that every required field holds its intended value, and that the only remaining step is the final submit; fix anything missing first. Call `finish(status, reason, extracted_output)` when the goal is achieved, or when you have established that it cannot be achieved; the finish tool's own description says which status each outcome takes.
diff --git a/skyvern/forge/taskv3/loop.py b/skyvern/forge/taskv3/loop.py
index 86830ba99..ed089f495 100644
--- a/skyvern/forge/taskv3/loop.py
+++ b/skyvern/forge/taskv3/loop.py
@@ -602,21 +602,17 @@ NO_TOOL_CALL_NUDGE = (
PERCEPTION_STALL_NUDGE_AFTER = 6
PERCEPTION_STALL_TERMINATE_AFTER = 15
-# SKY-16330. How many DISTINCT reads of one compactable tool survive compaction. Supersession means a
-# snapshot went stale, and a read of region B does not make a read of region A stale — they answer
-# different questions — so eliding A on B's arrival leaves a document larger than one result
-# impossible to assemble: broad reads are cut at HTML_MAX_CHARS and narrow ones are erased.
-#
-# 2, and the ceiling is what sets it. `st.total_tokens` accumulates the re-sent transcript EVERY turn
-# against DEFAULT_MAX_TOKENS, which exists for exactly this spiral, and the measured failing runs are
-# dying on it — so every retained snapshot costs its size times the turns that follow it. At 2 this
-# holds ~2x HTML_MAX_CHARS of get_html; each further unit is another HTML_MAX_CHARS re-sent ~100
-# times. RAISE ONLY ON MEASURED EVIDENCE that two windows are not enough, not on the intuition that
-# more context helps. It cannot widen a tool that issues handles: observe and look declare no arguments
-# and their key carries no content, so all of their calls share one key and exactly one of each
-# survives — whatever a non-strict provider adds to the call. Known limit: a run cycling one read over
-# THREE views (tabs A, B, C in turn) evicts the view it needs next on every read.
-PERCEPTION_SNAPSHOT_RETAIN = 2
+# SKY-17222. Read bodies (tools without handles) the model has already seen are retained by SIZE, not
+# count: once they sum past HIGH, the oldest are elided down to LOW in one rewrite (an epoch). HIGH is
+# the previous ceiling (two get_html windows, `tools.HTML_MAX_CHARS` each), so the worst case does not
+# move. Between epochs the transcript only grows at the end, which is what a provider prefix cache
+# needs: it hits only when the whole previous request is a byte prefix of the next one.
+PERCEPTION_RETAIN_CHARS_LOW = 20_000
+PERCEPTION_RETAIN_CHARS_HIGH = 2 * PERCEPTION_RETAIN_CHARS_LOW
+# However large they are, the newest reads the model has seen that an epoch keeps: two paged windows.
+PERCEPTION_RETAIN_MIN_READS = 2
+# A same-bytes marker replaces a body only when it saves most of it; on a short body it costs more.
+PERCEPTION_MARKER_MIN_RATIO = 4
# The stall verdict's machine class. Stable and facetable — telemetry keys on it to measure how often
# the policy fires — and it travels beside the verdict (`LoopOutcome.guard`, the guard's log line, the
@@ -1636,6 +1632,7 @@ class TerminalTelemetry:
peak_page_state_stall_rounds: int | None = None
peak_probe_revisits: int | None = None
semantic_commit: SemanticCommitStats | None = None
+ perception_reads: dict[str, int] | None = None
def log_fields(self) -> dict[str, Any]:
fields: dict[str, Any] = {"form_ever_armed": self.form_ever_armed, **self.survival}
@@ -1647,6 +1644,8 @@ class TerminalTelemetry:
fields["peak_probe_revisits"] = self.peak_probe_revisits
if self.semantic_commit is not None:
fields.update(self.semantic_commit.log_fields())
+ if self.perception_reads is not None:
+ fields.update(self.perception_reads)
return fields
@@ -1740,7 +1739,7 @@ def _refresh_nudge_text() -> str:
def _budget_extended_observation(cap: str, recency: ActivityRecency | None) -> str:
"""The retraction of a `_budget_exhausted_observation` whose cap has since been raised.
- APPENDED, never popped: `snapshot_keys` is keyed by absolute message index, so deleting the
+ APPENDED, never popped: `LoopState.reads` is keyed by absolute message index, so deleting the
stale message would silently re-anchor compaction onto the wrong ones. Without this the model
keeps reading "this is the final turn" for the rest of the run and wraps up early — which spends
the extension the release exists to preserve, through the prompt instead of through a counter."""
@@ -2542,16 +2541,11 @@ def make_finish_tool(
_READ_LABEL_MAX_CHARS = 120
-class _ReadKey(NamedTuple):
- args: str # `_declared_args_key`; the only half an elision placeholder may name
- content: str # digest of what the read returned; empty for a tool that issues handles
-
-
def _declared_args_key(spec: ToolSpec, args: dict[str, Any]) -> str:
- """The argument half of a read's supersession identity: only the arguments the tool DECLARES.
+ """The argument part of a read's key (`_PerceptionEntry.key`): only the arguments the tool DECLARES.
An argument the tool does not declare cannot change what it reads, so it cannot make two calls different
- reads (what the page shows can, which is the content half). The specs are not emitted strict, so a
+ reads (the page it reads can, which is the page part of the key). The specs are not emitted strict, so a
provider is free to add one — and for an argumentless tool that would split the key and retain two
snapshots where the tool only ever describes the page as it is NOW.
@@ -2559,7 +2553,7 @@ def _declared_args_key(spec: ToolSpec, args: dict[str, Any]) -> str:
`_look_manifest` and renumbers the marks (`tools.py`), so a retained older legend describes numbers
that now address different controls, and `click(mark=N)` following it acts on the wrong one. It
fails open: the stale legend looks perfectly valid. `look` and `observe` declare none and carry no
- content half (`issues_handles`), so all their calls collapse to one key and exactly one survives.
+ page part (`issues_handles`), so all their calls collapse to one key and exactly one survives.
"""
declared = (spec.parameters or {}).get("properties") or {}
return json.dumps({k: v for k, v in args.items() if k in declared}, sort_keys=True, default=str)
@@ -2587,110 +2581,187 @@ def _read_label(tool_name: str, args_key: str) -> str:
_COMPACTED_PREFIX = "[superseded "
-_EARLIER_READ_PREFIX = "[earlier read: a later "
-def _compact_transcript(
- messages: list[dict[str, Any]],
- snapshot_keys: dict[int, _ReadKey],
-) -> None:
- """Bound the persistent conversation by eliding stale perception snapshots.
-
- The full transcript is re-sent every turn, so large perception outputs (an `observe` snapshot the
- agent has already acted past, or a 20k-char `get_html` dump) otherwise pile up until the token
- backstop trips on perception-heavy pages. `snapshot_keys` maps the message index of each
- *successful* perception result (recorded as it is appended) to its supersession key; keep at most
- `PERCEPTION_SNAPSHOT_RETAIN` reads per tool, and replace the rest with a short placeholder.
-
- The key is the READ, not the tool. A second `observe` is a fresh view of the same thing and
- genuinely supersedes the first; a `get_html` of one region does not supersede a read of another,
- and eliding it there is what makes a document larger than one result impossible to assemble. Nor
- does one taken with the same arguments on a page that has since changed — another tab, another
- view — so a read without handles is superseded only by one whose canonical content is the same.
- The window is filled in two passes. The first keeps what an arguments-only key would: the newest read
- of each of the most recent distinct argument sets. Only slots it leaves free go to other versions of
- those reads, so a region re-read while its bytes keep changing can never push a different region out.
- A kept read whose content differs from the newest kept read of the same arguments is labelled as
- earlier, so the model can tell it predates its later actions. Three things are deliberately protected:
-
- - The most-recent round (results after the last assistant message) is never elided — a single turn
- can batch several perception calls, and compaction runs *before* the model has seen that round's
- results, so eliding any of them would drop data the model requested but never read.
- - Only a successful snapshot is ever a candidate: a skip/error result is never recorded in
- `snapshot_keys`, so it can neither be elided nor shadow the real snapshot and leave the agent
- with no usable page view — regardless of content length (a verbose provider error included).
- - The placeholder NAMES the read it dropped. An erasure the model cannot see is one it cannot
- plan around: it re-reads by accident instead of by decision, which is the loop this bounds.
-
- Only a `tool` message's content is shrunk, never removed, so every tool_call keeps a matching result
- and the transcript stays valid. Eliding also drops the index, so re-running is a no-op and an elided
- placeholder can never re-anchor as the live snapshot."""
- if not snapshot_keys:
- return
- last_assistant_idx = -1
- for i in range(len(messages) - 1, -1, -1):
- if messages[i].get("role") == "assistant":
- last_assistant_idx = i
- break
+@dataclass(slots=True)
+class _PerceptionEntry:
+ tool: str
+ args: str # `_declared_args_key`; the only part of the key a placeholder or marker may name
+ page: str # the page probe sampled before the call; "" for a tool that issues handles
+ # sha256 of the model-facing bytes before `delta`, not the canonical digest: a re-minted marker is new bytes.
+ raw: str
+ turn: int
+ handles: bool
+ # The tool-reported tail at `delta_at` (a text delta of what an action changed). Outside `raw`, so it never
+ # makes an unchanged read look changed, and carried verbatim through every placeholder and marker.
+ delta: str = ""
+ shown: bool = True
- # The unread round is protected but does NOT spend retention slots. It cannot: one turn may batch
- # several reads (the prompt asks for batching), and counting them against the window would let a
- # single batched turn evict every earlier read and make accumulation a no-op on exactly the
- # behaviour the prompt trains. Its size is bounded by the per-turn tool-call budget, and the old
- # rule protected the whole round the same way, so nothing here widens that.
- unread: dict[str, set[_ReadKey]] = {}
- read: dict[str, list[int]] = {}
- for i in sorted(snapshot_keys, reverse=True):
- cls = messages[i]["name"]
- if i > last_assistant_idx:
- unread.setdefault(cls, set()).add(snapshot_keys[i])
- else:
- read.setdefault(cls, []).append(i)
-
- for cls, indices in read.items():
- fresh = unread.get(cls, set())
- fresh_args = {key.args for key in fresh}
- keep: list[int] = []
- # Pass 1: the newest read of each recent argument set the unread round has not re-taken.
- window_args: list[str] = []
- for i in indices:
- args = snapshot_keys[i].args
- if args in fresh_args or args in window_args or len(window_args) >= PERCEPTION_SNAPSHOT_RETAIN:
- continue
- window_args.append(args)
- keep.append(i)
- # Pass 2: free slots go to other versions, newest first. A key seen newer returned the same content
- # (or, for a tool that issues handles, had its handles disposed), so it is superseded.
- kept_keys = {snapshot_keys[i] for i in keep}
- for i in indices:
- if len(keep) >= PERCEPTION_SNAPSHOT_RETAIN:
+ @property
+ def key(self) -> tuple[str, str, str]:
+ return (self.tool, self.page, self.args)
+
+
+@dataclass
+class _PerceptionStore:
+ """Every successful perception read, by message index, and the rewrites that bound them.
+
+ Read bodies are budgeted by size (`PERCEPTION_RETAIN_CHARS_HIGH`/`_LOW`) and rewritten only in
+ epochs, so between epochs each request is a byte prefix of the next. Tools that issue handles
+ (`observe`, `look`) keep exactly one body and elide the older one at once: the next call renumbers
+ refs/marks, so a stale legend addresses the wrong control and fails open.
+
+ A re-read whose bytes equal a body still shown becomes a marker citing it. Nothing is predicted: the
+ read ran, and this compares what it returned. A marker is not an entry, and an epoch moves a cited
+ body into the newest citing marker's slot before it elides anything, so a marker never outlives its body.
+ """
+
+ entries: dict[int, _PerceptionEntry] = field(default_factory=dict)
+ # marker message index -> (cited body index, turn the marker's read was taken, the marker's own delta)
+ markers: dict[int, tuple[int, int, str]] = field(default_factory=dict)
+ unchanged_marks: int = 0
+ epoch_rewrites: int = 0
+ retained_chars_peak: int = 0
+
+ def admit(
+ self,
+ index: int,
+ *,
+ tool: str,
+ args: str,
+ page: str,
+ handles: bool,
+ content: str,
+ turn: int,
+ delta_at: int | None = None,
+ ) -> str:
+ """Record the read about to be appended at `index`; returns what the model is shown for it."""
+ cut = delta_at if delta_at is not None and 0 <= delta_at <= len(content) else len(content)
+ head, delta = content[:cut], content[cut:]
+ entry = _PerceptionEntry(
+ tool=tool,
+ args=args,
+ page="" if handles else page,
+ raw=hashlib.sha256(head.encode()).hexdigest(),
+ turn=turn,
+ handles=handles,
+ delta=delta,
+ )
+ if not handles:
+ prior_index = max((i for i, e in self.entries.items() if e.key == entry.key), default=None)
+ prior = None if prior_index is None else self.entries[prior_index]
+ if prior_index is not None and prior is not None and prior.shown:
+ label = _read_label(tool, args)
+ if prior.raw == entry.raw:
+ # The marker locates its body only by label (the turn number is not in the transcript), so a
+ # newer body with the same label from another page would be the one the model looks at.
+ latest_same_label = max(
+ i for i, e in self.entries.items() if e.shown and e.tool == tool and e.args == args
+ )
+ marker = (
+ f"[{label} returned the same {len(head)} chars as your read at turn {prior.turn}, "
+ "still shown above]"
+ )
+ if latest_same_label == prior_index and len(head) >= PERCEPTION_MARKER_MIN_RATIO * len(marker):
+ self.markers[index] = (prior_index, turn, delta)
+ self.unchanged_marks += 1
+ return marker + delta
+ else:
+ # On the NEW read: labelling the old one would rewrite a message the provider has cached.
+ content = (
+ f"[{label} returned different content from your read at turn {prior.turn}; that one "
+ f"predates your later actions, this is the current one]\n{content}"
+ )
+ self.entries[index] = entry
+ return content
+
+ def compact(self, messages: list[dict[str, Any]]) -> None:
+ """Run before every LLM call. The unread round (after the last assistant message) is never elided
+ and never charged: one batched turn must not evict every earlier read."""
+ last_assistant_idx = -1
+ for i in range(len(messages) - 1, -1, -1):
+ if messages[i].get("role") == "assistant":
+ last_assistant_idx = i
break
- key = snapshot_keys[i]
- if i in keep or key in fresh or key in kept_keys:
- continue
- keep.append(i)
- kept_keys.add(key)
- for i in indices:
- if i in keep:
- continue
- messages[i]["content"] = (
- f"{_COMPACTED_PREFIX}{_read_label(cls, snapshot_keys[i].args)} output elided to bound context]"
- )
- del snapshot_keys[i]
-
- # The content of the newest surviving read per (tool, arguments). Every elided read is already gone
- # from `snapshot_keys`, so an earlier survivor that differs from it has a newer different view.
- newest: dict[tuple[str, str], str] = {}
- for i in sorted(snapshot_keys, reverse=True):
- cls = messages[i]["name"]
- key = snapshot_keys[i]
- if newest.setdefault((cls, key.args), key.content) == key.content:
- continue
- if not messages[i]["content"].startswith(_EARLIER_READ_PREFIX):
- messages[i]["content"] = (
- f"{_EARLIER_READ_PREFIX}{_read_label(cls, key.args)} returned different content; this is what "
- f"the page showed when this read was taken, before your later actions]\n{messages[i]['content']}"
+ newest: dict[tuple[str, str, str], int] = {}
+ for i in sorted(self.entries):
+ if self.entries[i].shown:
+ newest[self.entries[i].key] = i
+ for i, e in list(self.entries.items()):
+ if e.shown and e.handles and i < last_assistant_idx and newest[e.key] != i:
+ self._elide(messages, i)
+
+ read = self._read_bodies(last_assistant_idx)
+ charged = sum(len(messages[i]["content"]) for i in read)
+ # Two full-size reads (window plus its continuation notice) exceed HIGH, so without a floor an epoch
+ # would break paging, which the count window kept at two. The floor never triggers an epoch on its own.
+ if charged > PERCEPTION_RETAIN_CHARS_HIGH and len(read) > len(self._floor(read, newest)):
+ self.epoch_rewrites += 1
+ self._move_cited_bodies(messages)
+ newest = {}
+ for i in sorted(self.entries):
+ if self.entries[i].shown:
+ newest[self.entries[i].key] = i
+ read = self._read_bodies(last_assistant_idx)
+ charged = sum(len(messages[i]["content"]) for i in read)
+ floor = self._floor(read, newest)
+ older = [i for i in read if newest[self.entries[i].key] != i]
+ current = [i for i in read if newest[self.entries[i].key] == i]
+ for i in older + current:
+ if charged <= PERCEPTION_RETAIN_CHARS_LOW:
+ break
+ if i in floor:
+ continue
+ charged -= len(messages[i]["content"])
+ self._elide(messages, i)
+ self.retained_chars_peak = max(self.retained_chars_peak, charged)
+
+ def _floor(self, read: list[int], newest: dict[tuple[str, str, str], int]) -> set[int]:
+ """The newest reads the model has seen, one per key, that an epoch never elides."""
+ return set([i for i in read if newest[self.entries[i].key] == i][-PERCEPTION_RETAIN_MIN_READS:])
+
+ def _read_bodies(self, last_assistant_idx: int) -> list[int]:
+ return sorted(i for i, e in self.entries.items() if e.shown and not e.handles and i < last_assistant_idx)
+
+ def _placeholder(self, entry: _PerceptionEntry) -> str:
+ return f"{_COMPACTED_PREFIX}{_read_label(entry.tool, entry.args)} output elided to bound context]"
+
+ def _elide(self, messages: list[dict[str, Any]], index: int) -> None:
+ entry = self.entries[index]
+ messages[index]["content"] = self._placeholder(entry) + entry.delta
+ entry.shown = False
+
+ def _move_cited_bodies(self, messages: list[dict[str, Any]]) -> None:
+ citing: dict[int, list[int]] = {}
+ for marker_index, (body_index, _, _) in self.markers.items():
+ citing.setdefault(body_index, []).append(marker_index)
+ for body_index, marker_indices in citing.items():
+ target = max(marker_indices)
+ body = self.entries[body_index]
+ shown = messages[body_index]["content"]
+ _, turn, target_delta = self.markers[target]
+ messages[target]["content"] = shown[: len(shown) - len(body.delta)] + target_delta
+ self._elide(messages, body_index)
+ for i in marker_indices:
+ if i != target:
+ messages[i]["content"] = self._placeholder(body) + self.markers[i][2]
+ self.entries[target] = _PerceptionEntry(
+ tool=body.tool,
+ args=body.args,
+ page=body.page,
+ raw=body.raw,
+ turn=turn,
+ handles=False,
+ delta=target_delta,
)
+ self.markers.clear()
+
+ def log_fields(self) -> dict[str, int]:
+ return {
+ "perception_unchanged_marks": self.unchanged_marks,
+ "perception_epoch_rewrites": self.epoch_rewrites,
+ "perception_retained_chars_peak": self.retained_chars_peak,
+ }
@dataclass(kw_only=True, slots=True)
@@ -2769,12 +2840,9 @@ class LoopState:
# each call, passing prompt=None: LLMCaller.use_message_history never appends the
# assistant reply or tool results itself, so multi-turn tool use must be threaded here.
messages: list[dict[str, Any]] = field(default_factory=list)
- # Indices into `messages` of successful perception results, recorded as they are appended so
- # compaction can keep the newest without inferring "real snapshot" from content size. The value is
- # the read's supersession key (`_ReadKey`): its declared arguments, plus what it returned for a tool
- # that issues no handles. Keying on the tool name alone would make a read of one region supersede a
- # read of another; keying on arguments alone does the same to one read taken on two pages.
- snapshot_keys: dict[int, _ReadKey] = field(default_factory=dict)
+ # Successful perception results by index into `messages`, recorded as they are appended, so
+ # compaction never infers "real snapshot" from content size and a skip/error result is never one.
+ reads: _PerceptionStore = field(default_factory=_PerceptionStore)
perception: _PerceptionLedger = field(default_factory=_PerceptionLedger)
# Net-progress ledger (additive shadow); None disables it, mirroring the guard's *_after knobs.
progress: _ProgressLedger | None = None
@@ -3303,7 +3371,7 @@ async def run_agent_tool_loop(
# Elide superseded perception results before re-sending the transcript, so a perception-heavy
# run can't balloon the context to the token backstop (the pre-compaction runaway mode).
- _compact_transcript(st.messages, st.snapshot_keys)
+ st.reads.compact(st.messages)
llm_caller.message_history = list(st.messages)
# Consume any pending look image into THIS call only, then clear: the image rides one request
# and is never appended to `messages`, so the turn after carries zero image blocks.
@@ -4061,21 +4129,37 @@ async def run_agent_tool_loop(
}
)
- if spec is not None and spec.compactable and result.status == "ok":
- # The index this successful snapshot will occupy, pre-append, against the read's
- # identity — the tool half is `messages[i]["name"]`, which compaction reads there.
- st.snapshot_keys[len(st.messages)] = _ReadKey(
- _declared_args_key(spec, args), "" if spec.issues_handles else (content_digest or "")
- )
model_facing_content = result.content
skyvern_ctx = skyvern_context.current()
if skyvern_ctx is not None:
model_facing_content = skyvern_ctx.hide_from_model(model_facing_content)
+ # Masking can change lengths, so the tool's offset is re-found as the masked delta's suffix
+ # position; if masking broke the suffix, the whole result counts as the read.
+ reported_delta_at = reported.get("delta_at")
+ delta_at: int | None = None
+ if type(reported_delta_at) is int and 0 <= reported_delta_at <= len(result.content):
+ delta = result.content[reported_delta_at:]
+ if skyvern_ctx is not None:
+ delta = skyvern_ctx.hide_from_model(delta)
+ if model_facing_content.endswith(delta):
+ delta_at = len(model_facing_content) - len(delta)
+ transcript_content = model_facing_content
+ if spec is not None and spec.compactable and result.status == "ok":
+ transcript_content = st.reads.admit(
+ len(st.messages),
+ tool=tool_name,
+ args=_declared_args_key(spec, args),
+ page=probe_before or "",
+ handles=spec.issues_handles,
+ content=model_facing_content,
+ turn=st.turns,
+ delta_at=delta_at,
+ )
# A refresh raised by this call re-baselines the stall and repeat ledgers below, so neither
# guard may end the run on it first.
refresh_pending = skyvern_ctx is not None and skyvern_ctx.refresh_working_page
st.messages.append(
- {"role": "tool", "tool_call_id": tool_call_id, "name": tool_name, "content": model_facing_content}
+ {"role": "tool", "tool_call_id": tool_call_id, "name": tool_name, "content": transcript_content}
)
if tool_trail is not None and spec is not None and not spec.terminal:
tool_trail.record(
@@ -4603,6 +4687,8 @@ async def run_agent_tool_loop(
# observable from logs today.
peak_probe_revisits=st.perception.peak_probe_revisits if st.perception.revisit_chances else None,
semantic_commit=semantic_commit_stats,
+ # Present only for a run that took a perception read, the population the counters describe.
+ perception_reads=st.reads.log_fields() if st.reads.entries else None,
)
st.outcome.turns = st.turns
diff --git a/skyvern/forge/taskv3/tools.py b/skyvern/forge/taskv3/tools.py
index 315c9584f..5989b4156 100644
--- a/skyvern/forge/taskv3/tools.py
+++ b/skyvern/forge/taskv3/tools.py
@@ -16730,9 +16730,8 @@ def build_browser_tools(
# consume the action-step budget or meter like one that does.
_tool_spec.recordable = True
if _tool_spec.name in ("observe", "get_html", "look"):
- # Large perception dumps: only the latest snapshot is relevant, so let the loop elide older
- # ones from the re-sent transcript (bounds context on perception-heavy pages). look's legend
- # (not its ephemeral image, which never enters the transcript) rides the same rule.
+ # Large perception dumps the loop may elide from the re-sent transcript (`_PerceptionStore`).
+ # look's legend (not its ephemeral image, which never enters the transcript) rides the same rule.
_tool_spec.compactable = True
if _tool_spec.name in ("observe", "look"):
_tool_spec.issues_handles = True
diff --git a/tests/unit/test_taskv3_loop.py b/tests/unit/test_taskv3_loop.py
index 374b796ce..5a59b74f1 100644
--- a/tests/unit/test_taskv3_loop.py
+++ b/tests/unit/test_taskv3_loop.py
@@ -3369,89 +3369,66 @@ def _assistant_turn(*ids: str) -> dict[str, Any]:
return {"role": "assistant", "content": None, "tool_calls": [{"id": i} for i in ids]}
-def test_compact_transcript_elides_superseded_perception() -> None:
- # Keep the newest snapshot of each tracked READ; elide older ones' content (never remove the message),
- # and leave untracked results untouched. Round 2 (after the last assistant) re-takes round 1's two
- # reads with the same arguments, so it supersedes them. `snapshot_keys` maps the successful-perception
- # message indices the loop records to each read's (args) identity.
- from skyvern.forge.taskv3.loop import _compact_transcript, _ReadKey
-
- messages = [
- {"role": "system", "content": "sys"},
- {"role": "user", "content": "goal"},
- _assistant_turn("a", "b", "c"), # round 1
- _tool_msg("a", "observe", "OBSERVE_1 " + "x" * 300), # idx 3
- _tool_msg("b", "get_html", "HTML_1 " + "y" * 300), # idx 4
- _tool_msg("c", "click", "clicked #x"), # idx 5 (not a snapshot)
- _assistant_turn("d", "e"), # round 2 (latest)
- _tool_msg("d", "observe", "OBSERVE_2 latest " + "z" * 300), # idx 7
- _tool_msg("e", "get_html", "HTML_2 latest " + "w" * 300), # idx 8
- ]
- # The observe/get_html successes keyed by their args; the click (5) is not a snapshot at all.
- snapshots = {
- 3: _ReadKey("{}", ""),
- 4: _ReadKey('{"selector": "#rows"}', "d1"),
- 7: _ReadKey("{}", ""),
- 8: _ReadKey('{"selector": "#rows"}', "d1"),
- }
- _compact_transcript(messages, snapshots)
+def _store_round(
+ store: Any, messages: list[dict[str, Any]], turn: int, reads: list[tuple[str, str, str]], *, page: str = "p"
+) -> None:
+ """One assistant turn and its successful reads, appended the way the loop appends them."""
+ ids = [f"c{turn}_{k}" for k in range(len(reads))]
+ messages.append(_assistant_turn(*ids))
+ for call_id, (tool, args, content) in zip(ids, reads):
+ shown = store.admit(
+ len(messages),
+ tool=tool,
+ args=args,
+ page=page,
+ handles=tool in ("observe", "look"),
+ content=content,
+ turn=turn,
+ )
+ messages.append(_tool_msg(call_id, tool, shown))
+
+
+def test_perception_store_elides_superseded_handle_reads_and_names_them() -> None:
+ # A newer observe disposes the refs of the older one, so the older one elides at once; the placeholder
+ # names the read it dropped, and an untracked result is never touched. A second pass changes nothing.
+ from skyvern.forge.taskv3.loop import _PerceptionStore
+
+ store = _PerceptionStore()
+ messages: list[dict[str, Any]] = [{"role": "system", "content": "sys"}, {"role": "user", "content": "goal"}]
+ _store_round(store, messages, 1, [("observe", "{}", "OBSERVE_1 " + "x" * 300)])
+ messages.append(_tool_msg("click", "click", "clicked #x"))
+ _store_round(store, messages, 2, [("observe", "{}", "OBSERVE_2 " + "z" * 300)])
+ _store_round(store, messages, 3, [])
+ store.compact(messages)
by_id = {m["tool_call_id"]: m["content"] for m in messages if m.get("role") == "tool"}
- assert by_id["a"].startswith("[superseded observe") # older observe elided
- assert by_id["b"].startswith("[superseded get_html") # older get_html elided
- assert by_id["c"] == "clicked #x" # untracked result untouched
- assert by_id["d"].startswith("OBSERVE_2 latest") # newest observe kept intact
- assert by_id["e"].startswith("HTML_2 latest") # newest get_html kept intact
- assert set(snapshots) == {7, 8} # elided indices are dropped so a re-run can't re-anchor them
- # The placeholder names the read it dropped, so the model can re-take it deliberately.
- assert by_id["b"] == "[superseded get_html(selector=#rows) output elided to bound context]"
-
- # Idempotent: a second pass over the (now-reduced) index set changes nothing.
+ assert by_id["c1_0"] == "[superseded observe output elided to bound context]"
+ assert by_id["click"] == "clicked #x"
+ assert by_id["c2_0"].startswith("OBSERVE_2")
+
snapshot = [m.get("content") for m in messages]
- _compact_transcript(messages, snapshots)
+ store.compact(messages)
assert [m.get("content") for m in messages] == snapshot
-def test_compact_transcript_keeps_unread_latest_round() -> None:
+def test_perception_store_keeps_unread_latest_round() -> None:
# A single turn can batch several perception calls; compaction runs before the model reads them, so
- # the latest round must be kept intact even when it repeats a compactable tool (would otherwise drop
- # a result the model requested but never saw).
- from skyvern.forge.taskv3.loop import _compact_transcript, _ReadKey
-
- messages = [
- {"role": "user", "content": "goal"},
- _assistant_turn("a", "b"),
- _tool_msg("a", "get_html", "HTML_A " + "a" * 300), # idx 2
- _tool_msg("b", "get_html", "HTML_B " + "b" * 300), # idx 3
- ]
- _compact_transcript(messages, {2: _ReadKey('{"selector": "#a"}', "d1"), 3: _ReadKey('{"selector": "#a"}', "d1")})
- assert messages[2]["content"].startswith("HTML_A") # both unread → neither elided
- assert messages[3]["content"].startswith("HTML_B")
-
-
-def test_compact_transcript_skip_stub_does_not_shadow_real_snapshot() -> None:
- # A skipped/errored perception result is never recorded as a snapshot, so it can't shadow the real
- # observe from an earlier round — else a failed batch would leave the agent with no page view. The
- # skip stub (idx 4) is simply absent from the index set regardless of its content.
- from skyvern.forge.taskv3.loop import _compact_transcript, _ReadKey
+ # the latest round stays intact even when it repeats a tool that keeps only one read.
+ from skyvern.forge.taskv3.loop import _PerceptionStore
- messages = [
- _assistant_turn("o1"),
- _tool_msg("o1", "observe", "REAL_OBSERVE " + "p" * 300), # idx 1 (the only real snapshot)
- _assistant_turn("c1", "o2"), # latest round: a click that failed, so the batched observe was skipped
- _tool_msg("c1", "click", "tool_error: TimeoutError: click failed"), # idx 3
- _tool_msg("o2", "observe", "skipped: earlier tool call in this batch failed"), # idx 4 (not tracked)
- ]
- _compact_transcript(messages, {1: _ReadKey("{}", "")})
- assert messages[1]["content"].startswith("REAL_OBSERVE") # real snapshot preserved as the live view
- assert messages[4]["content"].startswith("skipped:") # skip stub left as-is, never elided or promoted
+ store = _PerceptionStore()
+ messages: list[dict[str, Any]] = [{"role": "user", "content": "goal"}]
+ _store_round(store, messages, 1, [("observe", "{}", "OBS_A " + "a" * 300), ("observe", "{}", "OBS_B " + "b" * 300)])
+ store.compact(messages)
+ assert messages[2]["content"].startswith("OBS_A")
+ assert messages[3]["content"].startswith("OBS_B")
-def test_compact_transcript_noop_without_tracked_snapshots() -> None:
- from skyvern.forge.taskv3.loop import _compact_transcript
+def test_perception_store_noop_without_tracked_reads() -> None:
+ from skyvern.forge.taskv3.loop import _PerceptionStore
- messages = [_assistant_turn("a"), _tool_msg("a", "observe", "big " + "x" * 500)]
- _compact_transcript(messages, {})
- assert messages[1]["content"].startswith("big ") # nothing elided when nothing is tracked
+ messages = [_assistant_turn("a"), _tool_msg("a", "observe", "big " + "x" * 500), _assistant_turn()]
+ _PerceptionStore().compact(messages)
+ assert messages[1]["content"].startswith("big ")
def _observe_tool(handler: ToolHandler) -> ToolSpec:
@@ -10367,17 +10344,17 @@ async def test_every_click_row_carries_a_hit_class_and_no_other_tool_does() -> N
assert all("hit_class" not in e for e in unregistered)
-def _get_html_tool() -> ToolSpec:
+def _get_html_tool(pad: int = 300) -> ToolSpec:
"""A get_html whose answer is a function of its arguments — disjoint reads return disjoint bytes."""
async def handler(args: dict[str, Any]) -> ToolResult:
- return ToolResult.ok(f"HTML[{args.get('selector') or 'page'}@{args.get('offset', 0)}] " + "y" * 300)
+ return ToolResult.ok(f"HTML[{args.get('selector') or 'page'}@{args.get('offset', 0)}] " + "y" * pad)
spec = ToolSpec(
name="get_html",
description="get_html",
- # Declared, like the real spec: the supersession key is built from a tool's DECLARED arguments,
- # so a fixture that declares none would collapse every read onto one key.
+ # Declared, like the real spec: the read key is built from a tool's DECLARED arguments, so a
+ # fixture that declares none would collapse every read onto one key.
parameters={"type": "object", "properties": {"selector": {"type": "string"}, "offset": {"type": "integer"}}},
handler=handler,
)
@@ -10385,6 +10362,64 @@ def _get_html_tool() -> ToolSpec:
return spec
+class _RequestRecordingCaller(_ScriptedCaller):
+ """Records each request as the provider receives it: one serialized string per message."""
+
+ def __init__(self, script: list[list[tuple[str, dict[str, Any]]]]) -> None:
+ super().__init__(script)
+ self.requests: list[list[str]] = []
+
+ async def call(self, **kwargs: Any) -> dict[str, Any]:
+ self.requests.append([json.dumps(m, sort_keys=True, default=str) for m in self.message_history])
+ return await super().call(**kwargs)
+
+
+async def _run_recording(
+ script: list[list[tuple[str, dict[str, Any]]]], tools: list[ToolSpec], **kwargs: Any
+) -> tuple[LoopOutcome, _RequestRecordingCaller]:
+ caller = _RequestRecordingCaller(script)
+ defaults: dict[str, Any] = {"max_turns": 60, "max_tool_calls": 300}
+ defaults.update(kwargs)
+ outcome = await run_agent_tool_loop(
+ llm_caller=caller, system_prompt="sys", user_prompt="goal", tools=tools, **defaults
+ )
+ return outcome, caller
+
+
+_SAME_BYTES_MARKER_RE = re.compile(
+ r"^\[get_html(\([^\n]*\))? returned the same \d+ chars as your read at turn \d+, still shown above\]$"
+)
+_DIFFERS_LABEL_RE = re.compile(r"^\[get_html(\([^\n]*\))? returned different content from your read at turn \d+;")
+
+
+def _read_body(content: str) -> str | None:
+ """The page bytes a transcript read shows in full, or None for a placeholder or a same-bytes marker."""
+ if content.startswith("[superseded ") or _SAME_BYTES_MARKER_RE.match(content):
+ return None
+ if _DIFFERS_LABEL_RE.match(content):
+ return content.partition("\n")[2]
+ return content
+
+
+def _prefix_breaks(requests: list[list[str]]) -> int:
+ return sum(1 for a, b in zip(requests, requests[1:]) if b[: len(a)] != a)
+
+
+def _charged_read_chars(request: list[str]) -> int:
+ messages = [json.loads(m) for m in request]
+ last_assistant = max((i for i, m in enumerate(messages) if m.get("role") == "assistant"), default=-1)
+ return sum(
+ len(m["content"])
+ for m in messages[:last_assistant]
+ if m.get("role") == "tool" and m.get("name") == "get_html" and _read_body(m["content"]) is not None
+ )
+
+
+def _perception_fields(outcome: LoopOutcome) -> dict[str, Any]:
+ assert outcome.telemetry is not None
+ return outcome.telemetry.log_fields()
+
+
@pytest.mark.asyncio
async def test_loop_keeps_reads_of_disjoint_regions_readable_together() -> None:
# SKY-16330. A document too large for one read is gathered in parts, and the parts have to be
@@ -10406,23 +10441,29 @@ async def test_loop_keeps_reads_of_disjoint_regions_readable_together() -> None:
@pytest.mark.asyncio
-async def test_loop_still_elides_a_re_read_of_the_same_region() -> None:
- # The other half of the same rule, and the reason the rule exists: re-reading the SAME region IS
- # a supersession — the older copy is a stale view of the same bytes — so context stays bounded on
- # a run that hammers one read.
+async def test_loop_marks_a_re_read_that_returned_the_same_bytes() -> None:
+ # SKY-17198. A re-read whose bytes equal a body still shown above says so instead of repeating it; the
+ # body it names stays in place. A short body is repeated instead: there the marker saves nothing.
script = [
[("get_html", {"selector": "#part-1"})],
[("get_html", {"selector": "#part-1"})],
[("finish", {"status": "completed", "reason": "ok"})],
]
- outcome, _ = await _run(script, [_get_html_tool(), make_finish_tool()])
- reads = [m["content"] for m in outcome.messages if m.get("role") == "tool" and m.get("name") == "get_html"]
- assert len(reads) == 2
- assert reads[0].startswith("[superseded get_html"), reads[0]
- assert reads[1].startswith("HTML[#part-1")
+ outcome, _ = await _run(script, [_get_html_tool(pad=2000), make_finish_tool()])
+ reads = _tool_contents(outcome, "get_html")
+ assert reads[0].startswith("HTML[#part-1@0] ")
+ assert reads[1] == (
+ f"[get_html(selector=#part-1) returned the same {len(reads[0])} chars as your read at turn 1, "
+ "still shown above]"
+ )
+ assert _perception_fields(outcome)["perception_unchanged_marks"] == 1
+
+ outcome, _ = await _run(script, [_get_html_tool(pad=100), make_finish_tool()])
+ short = _tool_contents(outcome, "get_html")
+ assert short[0] == short[1] and short[1].startswith("HTML[#part-1@0] ")
-def _tabbed_page_tools() -> list[ToolSpec]:
+def _tabbed_page_tools(pad: int = 300) -> list[ToolSpec]:
"""A page with an in-page tab strip: `click` switches the tab, and every perception returns the ACTIVE tab.
get_html declares its arguments like the real spec; observe issues handles like the real one.
@@ -10434,7 +10475,7 @@ def _tabbed_page_tools() -> list[ToolSpec]:
return ToolResult.ok(f"clicked {args['selector']}")
async def get_html(args: dict[str, Any]) -> ToolResult:
- return ToolResult.ok(f"TAB[{state['tab']}] {args.get('selector') or 'page'} " + "y" * 300)
+ return ToolResult.ok(f"TAB[{state['tab']}] {args.get('selector') or 'page'} " + "y" * pad)
async def observe(_args: dict[str, Any]) -> ToolResult:
return ToolResult.ok(f"OBSERVE[{state['tab']}] ref=1 " + "x" * 300)
@@ -10469,68 +10510,24 @@ def _tool_contents(outcome: Any, name: str) -> list[str]:
@pytest.mark.asyncio
-async def test_loop_keeps_one_read_taken_on_two_tabs() -> None:
- # SKY-16330. The same read on two tabs of one page is two reads: identical arguments, different bytes.
- # Keying on arguments alone made each one erase the other, so a goal comparing two tabs could never
- # hold both, and the model re-read them in alternation until the step budget ran out.
- script = [
- [("get_html", {"format": "text"})],
- [("click", {"selector": "b"})],
- [("get_html", {"format": "text"})],
- [("click", {"selector": "a"})],
- [("finish", {"status": "completed", "reason": "ok"})],
- ]
- from skyvern.forge.taskv3.loop import _EARLIER_READ_PREFIX
-
- outcome, _ = await _run(script, _tabbed_page_tools())
- assert outcome.status == "completed"
- reads = _tool_contents(outcome, "get_html")
- assert len(reads) == 2
- # Kept, and marked as taken before the later actions: the loop cannot tell a tab switch from a write.
- label, _, body = reads[0].partition("\n")
- assert label.startswith(_EARLIER_READ_PREFIX), reads[0][:200]
- assert body.startswith("TAB[a]"), reads[0][:200]
- assert reads[1].startswith("TAB[b]"), reads[1][:120]
-
-
-async def test_loop_never_lets_a_changing_re_read_evict_another_region() -> None:
- # A region re-read after its bytes changed is a second version of one read, not a second read. Versions
- # only take slots the distinct reads leave free, so the other region an arguments-only key kept stays.
+async def test_loop_labels_the_newer_of_two_differing_reads_and_rewrites_neither() -> None:
+ # A read, a write to what it read (or a tab switch), and the same read again: both stay, and the model
+ # must be able to tell which one is current or it reports the pre-write value. The label rides the NEW
+ # read, so the old message is never rewritten and every request stays a byte prefix of the next.
script = [
- [("get_html", {"selector": "#doc"})],
- [("get_html", {"selector": "#form"})],
+ [("get_html", {"selector": "#total"})],
[("click", {"selector": "b"})],
- [("get_html", {"selector": "#form"})],
+ [("get_html", {"selector": "#total"})],
[("click", {"selector": "c"})],
[("finish", {"status": "completed", "reason": "ok"})],
]
- outcome, _ = await _run(script, _tabbed_page_tools())
+ outcome, caller = await _run_recording(script, _tabbed_page_tools())
reads = _tool_contents(outcome, "get_html")
- assert len(reads) == 3
- assert reads[0].startswith("TAB[a] #doc"), reads[0][:120]
- assert reads[1].startswith("[superseded get_html(selector=#form)"), reads[1][:120]
- assert reads[2].startswith("TAB[b] #form"), reads[2][:120]
-
-
-async def test_loop_elides_a_tab_re_read_that_returned_the_same_bytes() -> None:
- # The other half: returning to a tab and reading it again returns the same bytes, so the older copy
- # says nothing the newer does not, and it goes. This is what keeps an alternating run bounded.
- script = [
- [("get_html", {"format": "text"})],
- [("click", {"selector": "b"})],
- [("get_html", {"format": "text"})],
- [("click", {"selector": "a"})],
- [("get_html", {"format": "text"})],
- [("finish", {"status": "completed", "reason": "ok"})],
- ]
- outcome, _ = await _run(script, _tabbed_page_tools())
- reads = _tool_contents(outcome, "get_html")
- assert len(reads) == 3
- assert reads[0] == "[superseded get_html(format=text) output elided to bound context]"
- assert "TAB[b]" in reads[1] and not reads[1].startswith("[superseded "), reads[1][:120]
- assert reads[2].startswith("TAB[a]")
+ assert reads[0].startswith("TAB[a] #total"), reads[0][:120]
+ label, _, body = reads[1].partition("\n")
+ assert _DIFFERS_LABEL_RE.match(label) and "get_html(selector=#total)" in label and "turn 1" in label, label
+ assert body.startswith("TAB[b] #total")
+ assert _prefix_breaks(caller.requests) == 0
@pytest.mark.asyncio
@@ -10554,56 +10551,368 @@ async def test_loop_keeps_one_observe_across_tabs() -> None:
@pytest.mark.asyncio
-async def test_loop_labels_the_read_its_own_write_made_earlier() -> None:
- # A read, a write to what it read, and the same read again: both are kept now, and the older one shows
- # a value that is no longer on the page. It must say so, or the model reports the pre-write value as
- # current. The label is added once, however many turns the read survives.
+async def test_loop_read_key_separates_pages_and_windows() -> None:
+ # Page identity and every declared argument are part of a read's key. The same selector on another
+ # page, or another window of one document, is a different read, not a newer version of this one: it is
+ # never labelled as superseding it and never elided ahead of it as an older version.
+ state = {"page": "A"}
+
+ async def get_html(args: dict[str, Any]) -> ToolResult:
+ return ToolResult.ok(f"{state['page']} {args.get('selector')}@{args.get('offset', 0)} " + "y" * 300)
+
+ async def navigate(_args: dict[str, Any]) -> ToolResult:
+ state["page"] = "B"
+ return ToolResult.ok("navigated")
+
+ async def probe() -> str:
+ return f"url-{state['page']}|doc"
+
+ reader = _get_html_tool()
+ reader.handler = get_html
+ nav = ToolSpec(name="navigate", description="n", parameters={"type": "object", "properties": {}}, handler=navigate)
script = [
+ [("get_html", {"selector": "#x"})],
+ [("get_html", {"selector": "#x", "offset": 20000})],
+ [("navigate", {})],
+ [("get_html", {"selector": "#x"})],
+ [("finish", {"status": "completed", "reason": "ok"})],
+ ]
+ outcome, _ = await _run(script, [reader, nav, make_finish_tool()], page_probe=probe)
+ reads = _tool_contents(outcome, "get_html")
+ assert [r.split(" ")[0] for r in reads] == ["A", "A", "B"], [r[:40] for r in reads]
+
+
+def _sized_reader_tools(sizes: dict[str, int]) -> list[ToolSpec]:
+ """get_html returns `sizes["<selector> v<version>"]` chars; `click` bumps that selector's version."""
+ versions: dict[str, int] = {}
+
+ async def get_html(args: dict[str, Any]) -> ToolResult:
+ sel = args.get("selector") or "page"
+ head = f"{sel} v{versions.get(sel, 0)}"
+ return ToolResult.ok(head + " " + "y" * (sizes[head] - len(head) - 1))
+
+ async def click(args: dict[str, Any]) -> ToolResult:
+ versions[args["selector"]] = versions.get(args["selector"], 0) + 1
+ return ToolResult.ok("clicked")
+
+ reader = _get_html_tool()
+ reader.handler = get_html
+ clicker = ToolSpec(
+ name="click",
+ description="c",
+ parameters={"type": "object", "properties": {"selector": {"type": "string"}}},
+ handler=click,
+ )
+ return [reader, clicker, make_finish_tool()]
+
+
+async def test_loop_retention_stays_under_the_high_mark_and_keeps_the_newest_per_key() -> None:
+ # SKY-17222. Read bodies the model has seen are bounded by SIZE: once they pass the high mark, one
+ # rewrite (an epoch) elides them down to the low mark, older versions of a read first, so a read whose
+ # older version survives always has its newest version shown too. Between epochs nothing is rewritten.
+ from skyvern.forge.taskv3.loop import PERCEPTION_RETAIN_CHARS_HIGH
+
+ script: list[list[tuple[str, dict[str, Any]]]] = [
+ [("get_html", {"selector": "#rows"})],
[("get_html", {"selector": "#total"})],
- [("click", {"selector": "b"})],
+ [("get_html", {"selector": "#tiny"})],
+ [("click", {"selector": "#total"})],
[("get_html", {"selector": "#total"})],
- [("click", {"selector": "c"})],
- [("click", {"selector": "d"})],
+ [("get_html", {"selector": "#note"})],
[("finish", {"status": "completed", "reason": "ok"})],
]
- outcome, _ = await _run(script, _tabbed_page_tools())
+ tools = _sized_reader_tools(
+ {"#rows v0": 6_000, "#total v0": 30_000, "#tiny v0": 100, "#total v1": 10_000, "#note v0": 100}
+ )
+ outcome, caller = await _run_recording(script, tools)
+ for request in caller.requests:
+ assert _charged_read_chars(request) <= PERCEPTION_RETAIN_CHARS_HIGH
reads = _tool_contents(outcome, "get_html")
- assert len(reads) == 2
- from skyvern.forge.taskv3.loop import _EARLIER_READ_PREFIX
+ # Between the low and high marks nothing is rewritten. Past HIGH the older #total goes first, and that
+ # alone reaches the low mark: #rows, older but the newest (and only) read of its key, stays.
+ assert reads[0].startswith("#rows v0 "), reads[0][:80]
+ assert reads[1] == "[superseded get_html(selector=#total) output elided to bound context]"
+ assert reads[2].startswith("#tiny v0 "), reads[2][:80]
+ assert _DIFFERS_LABEL_RE.match(reads[3]) and _read_body(reads[3]).startswith("#total v1 "), reads[3][:120]
+ fields = _perception_fields(outcome)
+ assert fields["perception_epoch_rewrites"] == _prefix_breaks(caller.requests) == 1
+ assert fields["perception_retained_chars_peak"] <= PERCEPTION_RETAIN_CHARS_HIGH
+
+
+async def test_loop_epoch_moves_a_cited_body_into_its_newest_marker() -> None:
+ # A marker says "still shown above", so the body it names can never be elided under it. An epoch moves
+ # that body into the newest marker's slot, keeping the newest read of the key literal; the older slot
+ # gets the placeholder, and the body then ages from its new position.
+ script: list[list[tuple[str, dict[str, Any]]]] = [
+ [("get_html", {"selector": s})] for s in ("#a", "#b", "#c", "#d", "#e", "#a", "#f")
+ ]
+ script.append([("finish", {"status": "completed", "reason": "ok"})])
+ outcome, _ = await _run(script, [_get_html_tool(pad=8_000), make_finish_tool()])
+ reads = _tool_contents(outcome, "get_html")
+ assert reads[0] == "[superseded get_html(selector=#a) output elided to bound context]"
+ assert reads[5].startswith("HTML[#a@0] "), reads[5][:80]
+ assert [_read_body(r) is not None for r in reads] == [False, False, False, True, True, True, True]
+ assert _perception_fields(outcome)["perception_unchanged_marks"] == 1
+
- label, _, body = reads[0].partition("\n")
- assert label.startswith(_EARLIER_READ_PREFIX) and "get_html(selector=#total)" in label, reads[0][:200]
- assert body.startswith("TAB[a]")
- assert reads[0].count(_EARLIER_READ_PREFIX) == 1
- assert reads[1].startswith("TAB[b]"), reads[1][:120]
+async def test_loop_epoch_keeps_the_two_reads_the_model_saw_last() -> None:
+ # SKY-16330/#17096. A page larger than one read is gathered in full-size windows, and a cut window is
+ # longer than the low mark (the window plus its continuation notice). An epoch must still leave the
+ # window the model read last turn, or the part it is assembling is gone one request after it arrived.
+ async def get_html(args: dict[str, Any]) -> ToolResult:
+ off = int(args.get("offset") or 0)
+ return ToolResult.ok(
+ f"W{off} " + "y" * 19_990 + f"\n[cut: 100000 chars total; continue with offset={off + 20_000}]"
+ )
+
+ reader = _get_html_tool()
+ reader.handler = get_html
+ script: list[list[tuple[str, dict[str, Any]]]] = [
+ [("get_html", {"offset": o})] for o in (0, 20_000, 40_000, 60_000, 80_000)
+ ]
+ script.append([("finish", {"status": "completed", "reason": "ok"})])
+ _, caller = await _run_recording(script, [reader, make_finish_tool()])
+ for turn, request in enumerate(caller.requests[2:], start=2):
+ messages = [json.loads(m) for m in request]
+ last_assistant = max(i for i, m in enumerate(messages) if m.get("role") == "assistant")
+ seen = [m for m in messages[:last_assistant] if m.get("name") == "get_html"]
+ assert _read_body(seen[-1]["content"]) is not None, (turn, seen[-1]["content"][:80])
+ # Main kept two paged windows readable together; two cut windows exceed the high mark, so the
+ # epoch must keep the newest two the model has seen, not just the last one.
+ if len(seen) >= 2:
+ assert _read_body(seen[-2]["content"]) is not None, (turn, seen[-2]["content"][:80])
+
+
+async def test_loop_marks_a_re_read_only_when_no_newer_read_with_its_label_is_shown() -> None:
+ # A marker names its body by label ("get_html(selector=#x)") and a turn number the model never sees. When
+ # a newer read with the same label from another document is shown between them, "still shown above"
+ # points the model at that document's bytes, so the re-read is shown in full instead.
+ state = {"doc": "A"}
+
+ async def get_html(args: dict[str, Any]) -> ToolResult:
+ return ToolResult.ok(
+ f"DOC[{state['doc']}] {args['selector']} " + "y" * (2_000 if state["doc"] == "A" else 2_100)
+ )
+
+ async def switch(args: dict[str, Any]) -> ToolResult:
+ state["doc"] = args["selector"]
+ return ToolResult.ok("switched")
+
+ async def probe() -> str:
+ return f"url-{state['doc']}|doc-{state['doc']}"
+
+ reader = _get_html_tool()
+ reader.handler = get_html
+ clicker = ToolSpec(
+ name="click",
+ description="c",
+ parameters={"type": "object", "properties": {"selector": {"type": "string"}}},
+ handler=switch,
+ )
+ script = [
+ [("get_html", {"selector": "#x"})],
+ [("click", {"selector": "B"})],
+ [("get_html", {"selector": "#x"})],
+ [("click", {"selector": "A"})],
+ [("get_html", {"selector": "#x"})],
+ [("get_html", {"selector": "#x"})],
+ [("finish", {"status": "completed", "reason": "ok"})],
+ ]
+ outcome, _ = await _run_recording(script, [reader, clicker, make_finish_tool()], page_probe=probe)
+ reads = _tool_contents(outcome, "get_html")
+ assert _read_body(reads[2]) is not None and reads[2].startswith("DOC[A] #x"), reads[2][:120]
+ # With no other document's read after it, the same bytes again do get the marker.
+ assert _SAME_BYTES_MARKER_RE.match(reads[3]), reads[3][:120]
@pytest.mark.asyncio
-async def test_loop_bounds_how_many_distinct_reads_it_retains() -> None:
- # Accumulation is bounded, not unbounded: retaining every distinct read would let a run that
- # walks 300 selectors carry 300 snapshots. The oldest distinct reads elide once the retention
- # window is full, so the ceiling stays a constant times the per-read cap.
- from skyvern.forge.taskv3.loop import PERCEPTION_SNAPSHOT_RETAIN
+async def test_loop_never_marks_a_re_read_whose_bytes_changed() -> None:
+ # A marker is sound only on BYTE equality of what this call returned with a body still shown. Here
+ # the page re-mints the engine's own marker attribute between reads: the canonical digest (which folds
+ # it for the stall guard) is unchanged, but the attribute is what the model addresses the element by,
+ # so "same as before" would hand it a dead selector.
+ state = {"mint": 1}
+
+ async def get_html(_args: dict[str, Any]) -> ToolResult:
+ return ToolResult.ok(f'<input data-tv3="t{state["mint"]}" name="email">' + "y" * 2000)
+
+ async def remount(_args: dict[str, Any]) -> ToolResult:
+ state["mint"] += 1
+ return ToolResult.ok("clicked")
+
+ reader = _get_html_tool()
+ reader.handler = get_html
+ clicker = ToolSpec(name="click", description="c", parameters={"type": "object", "properties": {}}, handler=remount)
+ script = [
+ [("get_html", {"selector": "#email"})],
+ [("click", {})],
+ [("get_html", {"selector": "#email"})],
+ [("finish", {"status": "completed", "reason": "ok"})],
+ ]
+ outcome, _ = await _run(script, [reader, clicker, make_finish_tool()])
+ reads = _tool_contents(outcome, "get_html")
+ assert _read_body(reads[1]) == '<input data-tv3="t2" name="email">' + "y" * 2000, reads[1][:120]
+
+
+async def test_loop_perception_store_invariants_hold_on_random_read_sequences() -> None:
+ # Property over random read/mutate sequences, checked on every request the provider receives:
+ # (1) a same-bytes marker is issued only where a body shown ABOVE it in that request has exactly the
+ # bytes the marked call returned — a changed page never gets one; every shown body is its call's bytes;
+ # (2) between epochs each request is a byte prefix of the next;
+ # (3) seen read bodies never exceed the high mark, and a key's older version is never shown without
+ # its newest one.
+ from skyvern.forge.taskv3.loop import PERCEPTION_RETAIN_CHARS_HIGH
+
+ sizes = {"#s0": 200, "#s1": 1500, "#s2": 9000, "#s3": 16000}
+ totals = {"marks": 0, "epochs": 0}
+ for seed in range(25):
+ rng = random.Random(seed)
+ state: dict[str, Any] = {"page": "A", "mint": 0, "v": {}}
+ calls: list[tuple[tuple[str, str, int], str]] = []
+
+ async def get_html(args: dict[str, Any], state: dict[str, Any] = state, calls: list[Any] = calls) -> ToolResult:
+ sel, off = args.get("selector") or "page", int(args.get("offset") or 0)
+ body = (
+ f'<div data-tv3="t{state["mint"]}">{state["page"]} {sel}@{off} v{state["v"].get(sel, 0)}</div>'
+ + "y" * sizes.get(sel, 500)
+ )
+ calls.append(((state["page"], sel, off), body))
+ return ToolResult.ok(body)
+
+ async def click(args: dict[str, Any], state: dict[str, Any] = state) -> ToolResult:
+ target = args["selector"]
+ if target == "nav":
+ state["page"] = "B" if state["page"] == "A" else "A"
+ elif target == "remint":
+ state["mint"] += 1
+ else:
+ state["v"][target] = state["v"].get(target, 0) + 1
+ return ToolResult.ok(f"clicked {target}")
+
+ async def probe(state: dict[str, Any] = state) -> str:
+ return f"url-{state['page']}|doc"
+
+ reader = _get_html_tool()
+ reader.handler = get_html
+ clicker = ToolSpec(
+ name="click",
+ description="c",
+ parameters={"type": "object", "properties": {"selector": {"type": "string"}}},
+ handler=click,
+ )
+ script: list[list[tuple[str, dict[str, Any]]]] = []
+ for _ in range(24):
+ roll = rng.random()
+ if roll < 0.75:
+ script.append(
+ [
+ (
+ "get_html",
+ {"selector": rng.choice(list(sizes)), **({"offset": 20000} if rng.random() < 0.1 else {})},
+ )
+ for _ in range(1 if rng.random() < 0.8 else 3)
+ ]
+ )
+ else:
+ script.append([("click", {"selector": rng.choice(["nav", "remint", *sizes])})])
+ script.append([("finish", {"status": "completed", "reason": "ok"})])
+ outcome, caller = await _run_recording(script, [reader, clicker, make_finish_tool()], page_probe=probe)
+
+ positions = [
+ i for i, m in enumerate(outcome.messages) if m.get("role") == "tool" and m.get("name") == "get_html"
+ ]
+ assert len(positions) == len(calls), seed
+ call_at = dict(zip(positions, calls))
+ for request in caller.requests:
+ messages = [json.loads(m) for m in request]
+ shown_above: set[str] = set()
+ newest: dict[tuple[str, str, int], int] = {}
+ for i, m in enumerate(messages):
+ if i not in call_at:
+ continue
+ key, returned = call_at[i]
+ newest[key] = i
+ body = _read_body(m["content"])
+ if body is not None:
+ assert body == returned, (seed, i)
+ shown_above.add(body)
+ elif _SAME_BYTES_MARKER_RE.match(m["content"]):
+ assert returned in shown_above, (seed, i, m["content"])
+ for i, m in enumerate(messages):
+ if i in call_at and _read_body(m["content"]) is not None:
+ newest_content = messages[newest[call_at[i][0]]]["content"]
+ assert not newest_content.startswith("[superseded "), (seed, i)
+ assert _charged_read_chars(request) <= PERCEPTION_RETAIN_CHARS_HIGH, seed
+ fields = _perception_fields(outcome)
+ assert _prefix_breaks(caller.requests) == fields["perception_epoch_rewrites"], seed
+ totals["marks"] += fields["perception_unchanged_marks"]
+ totals["epochs"] += fields["perception_epoch_rewrites"]
+ # Not vacuous: the sequences reach both mechanisms.
+ assert totals["marks"] > 0 and totals["epochs"] > 0, totals
+
+
+async def test_loop_reads_that_differ_only_in_their_delta_get_the_marker_and_keep_the_delta() -> None:
+ # A tool may append a verbatim delta at a reported `delta_at`. It is not part of the read: two reads
+ # whose bytes differ only there are the same read, so the re-read gets the marker. The delta is still
+ # new information, so every message carries it verbatim: under the marker, under a moved body, and
+ # under the placeholder an epoch leaves.
+ calls = {"n": 0}
+ sizes = {"#a": 2_000, "#b": 15_000, "#c": 15_000, "#d": 15_000, "#e": 100}
- n = PERCEPTION_SNAPSHOT_RETAIN + 2
- script: list[list[tuple[str, dict[str, Any]]]] = [[("get_html", {"selector": f"#p{i}"})] for i in range(n)]
+ async def get_html(args: dict[str, Any]) -> ToolResult:
+ calls["n"] += 1
+ body = f"<form {args['selector']}>" + "y" * sizes[args["selector"]]
+ return ToolResult.ok(body + f"\n[delta {calls['n']}]", data={"delta_at": len(body)})
+
+ reader = _get_html_tool()
+ reader.handler = get_html
+ script: list[list[tuple[str, dict[str, Any]]]] = [
+ [("get_html", {"selector": s})] for s in ("#a", "#a", "#b", "#c", "#d", "#e")
+ ]
script.append([("finish", {"status": "completed", "reason": "ok"})])
- outcome, _ = await _run(script, [_get_html_tool(), make_finish_tool()], max_turns=n + 5, max_tool_calls=n + 5)
- reads = [m["content"] for m in outcome.messages if m.get("role") == "tool" and m.get("name") == "get_html"]
- assert len(reads) == n
- intact = [r for r in reads if r.startswith("HTML[")]
- # The retention window, plus the latest round — which is protected as unread and does not spend a
- # window slot, so one read per turn leaves RETAIN + 1 standing.
- expected = PERCEPTION_SNAPSHOT_RETAIN + 1
- assert len(intact) == expected, [r[:30] for r in reads]
- # The window keeps the NEWEST reads, which are the ones the model is working from.
- assert [r.split("@")[0] for r in intact] == [f"HTML[#p{i}" for i in range(n - expected, n)]
+ outcome, caller = await _run_recording(script, [reader, make_finish_tool()])
+ in_request = [json.loads(m) for m in caller.requests[2]]
+ marked = [m["content"] for m in in_request if m.get("role") == "tool" and m.get("name") == "get_html"][1]
+ assert marked.startswith("[get_html(selector=#a) returned the same "), marked[:120]
+ assert marked.endswith("\n[delta 2]")
+ reads = _tool_contents(outcome, "get_html")
+ assert [r.rsplit("\n", 1)[1] for r in reads] == [f"[delta {n}]" for n in range(1, 7)]
+ assert _perception_fields(outcome)["perception_epoch_rewrites"] == 1
+ assert reads[0].startswith("[superseded get_html(selector=#a)")
+
+
+def test_perception_store_does_not_charge_the_unread_round() -> None:
+ # SKY-16330. The prompt asks the model to batch, so one turn routinely lands several large reads.
+ # Charging that unread round against the budget would let one batch evict every earlier read — and
+ # rewrite the transcript on a turn that has nothing yet to bound.
+ from skyvern.forge.taskv3.loop import PERCEPTION_RETAIN_CHARS_LOW, _PerceptionStore
+
+ store = _PerceptionStore()
+ messages: list[dict[str, Any]] = [{"role": "user", "content": "goal"}]
+ _store_round(store, messages, 1, [("get_html", "{}", "WINDOW@0 " + "a" * 300)])
+ big = PERCEPTION_RETAIN_CHARS_LOW * 3 // 4
+ _store_round(
+ store,
+ messages,
+ 2,
+ [("get_html", json.dumps({"offset": k * 20000}), f"WINDOW@{k} " + "b" * big) for k in (1, 2, 3)],
+ )
+ store.compact(messages)
+ assert messages[2]["content"].startswith("WINDOW@0")
+ assert all(messages[i]["content"].startswith("WINDOW@") for i in (4, 5, 6))
+ assert store.epoch_rewrites == 0
@pytest.mark.asyncio
async def test_loop_retention_window_does_not_widen_observe() -> None:
# observe takes no arguments, so every observe is a read of the same thing: the whole page as it
- # is now. Exactly one survives, as before — the window can only widen a tool whose reads differ.
+ # is now. Exactly one survives, as before — the size budget applies only to reads without handles.
script = [
[("observe", {})],
[("observe", {})],
@@ -10616,51 +10925,6 @@ async def test_loop_retention_window_does_not_widen_observe() -> None:
assert len([o for o in obs if o.startswith("[superseded ")]) == 2
-def test_compact_transcript_batched_turn_does_not_evict_every_earlier_read() -> None:
- # SKY-16330. The prompt asks the model to batch aggressively, so a single turn routinely lands
- # several reads. Counting the still-unread round against the retention window would let one
- # batched turn evict every earlier read and make accumulation a no-op on exactly that behaviour.
- from skyvern.forge.taskv3.loop import PERCEPTION_SNAPSHOT_RETAIN, _compact_transcript, _ReadKey
-
- messages: list[dict[str, Any]] = [
- _assistant_turn("a"),
- _tool_msg("a", "get_html", "WINDOW@0 " + "a" * 300), # idx 1
- _assistant_turn("b", "c", "d"), # latest round, batched
- _tool_msg("b", "get_html", "WINDOW@20000 " + "b" * 300), # idx 3
- _tool_msg("c", "get_html", "WINDOW@40000 " + "c" * 300), # idx 4
- _tool_msg("d", "get_html", "WINDOW@60000 " + "d" * 300), # idx 5
- ]
- keys = {
- 1: _ReadKey("{}", "d0"),
- 3: _ReadKey('{"offset": 20000}', "d1"),
- 4: _ReadKey('{"offset": 40000}', "d2"),
- 5: _ReadKey('{"offset": 60000}', "d3"),
- }
- _compact_transcript(messages, keys)
- assert PERCEPTION_SNAPSHOT_RETAIN >= 1
- assert messages[1]["content"].startswith("WINDOW@0") # the earlier read survives the batch
- for idx in (3, 4, 5):
- assert messages[idx]["content"].startswith("WINDOW@") # the unread round is untouched
-
-
-def test_compact_transcript_unread_round_still_supersedes_its_own_earlier_read() -> None:
- # The other half: not spending a retention slot must not turn into not superseding. A read the
- # latest round has just re-taken is a stale view of the same bytes and still elides.
- from skyvern.forge.taskv3.loop import _compact_transcript, _ReadKey
-
- messages: list[dict[str, Any]] = [
- _assistant_turn("a"),
- _tool_msg("a", "get_html", "OLD@0 " + "a" * 300), # idx 1
- _assistant_turn("b"),
- _tool_msg("b", "get_html", "NEW@0 " + "b" * 300), # idx 3, same read, re-taken
- ]
- keys = {1: _ReadKey("{}", "d0"), 3: _ReadKey("{}", "d0")}
- _compact_transcript(messages, keys)
- assert messages[1]["content"] == "[superseded get_html output elided to bound context]"
- assert messages[3]["content"].startswith("NEW@0")
- assert set(keys) == {3}
-
-
def test_canonicalization_drops_a_windowed_reads_own_cut_notice_and_head_fragment() -> None:
# SKY-16330. A windowed read's digest must depend on the WINDOW, not on the rest of the document.
# The cut notice carries the document's total size, which moves when bytes outside the window
@@ -10701,11 +10965,16 @@ async def test_loop_undeclared_argument_cannot_split_an_argumentless_tools_key()
# the mark manifest and renumbers, so a retained older legend addresses controls that have moved,
# and a click following it acts on the wrong one. It fails open, because the stale legend looks
# valid. Exactly one look snapshot survives, whatever arrives on the call.
+ calls = {"n": 0}
+
async def handler(args: dict[str, Any]) -> ToolResult:
- return ToolResult.ok("LEGEND " + "x" * 300)
+ # Renumbered on every call, like the real legend, so no same-bytes marker can stand in for elision.
+ calls["n"] += 1
+ return ToolResult.ok(f"LEGEND {calls['n']} " + "x" * 300)
spec = ToolSpec(name="look", description="look", parameters={"type": "object", "properties": {}}, handler=handler)
spec.compactable = True
+ spec.issues_handles = True
script = [
[("look", {})],
[("look", {"reason": "checking the form"})],