Skip to content

High-performance browser automation bridge and multi-instance orchestrator with advanced stealth injection and real-time dashboard.

fix: nth: is one-based over role/label/testid selectors and zero-based over css/xpath/text, so the documented nth:0 finds nothing on a whole selector family (PIN-163)

Luigi Agostib3a300ba25b7
diff --git a/docs/commands.md b/docs/commands.md
index ef1a4ab1..2aef13d2 100644
--- a/docs/commands.md
+++ b/docs/commands.md
@@ -107,7 +107,7 @@ Most element commands accept a unified selector:
- label, placeholder, alt, title, or test id selectors such as `label:Email`, `placeholder:Search`, `alt:Logo`, `title:Close`, `testid:submit`
- positional wrappers such as `first:button`, `last:role:button`, or `nth:2:button` (`nth` is zero-based)
-Positional wrappers index the candidates in **document order** over `css:`, `xpath:`, and `text:` selectors, so `nth:0:` is the first match in the page and `nth:1:` always comes after it. A bare `text:` selector is the one form of those that does not index: it picks the most control-like match among the smallest ones, so `text:Save` prefers a `<button>` over a `<div>` carrying the same label — which means `text:X` and `first:text:X` can resolve to different elements. A wrapper only ever chooses among the matches a bare selector would find; it never changes which matches exist.
+Positional wrappers index the candidates in **document order** for every selector kind, so `nth:0:` is the first match in the page and `nth:1:` always comes after it. A bare `text:` selector is the one form that does not index: it picks the most control-like match among the smallest ones, so `text:Save` prefers a `<button>` over a `<div>` carrying the same label — which means `text:X` and `first:text:X` can resolve to different elements. A wrapper only ever chooses among the matches a bare selector would find; it never changes which matches exist.
Selector prefixes are case-insensitive, so `CSS:#login` and `css:#login` mean the same thing. Only the prefix is case-folded; the value after it is passed through unchanged.
diff --git a/docs/reference/find.md b/docs/reference/find.md
index 69e158fb..27650fcf 100644
--- a/docs/reference/find.md
+++ b/docs/reference/find.md
@@ -158,7 +158,7 @@ curl -X POST http://localhost:9867/tabs/<tabId>/action \
## Operational Notes
- `/find` uses the tab's accessibility snapshot, not raw DOM selectors.
-- Structured `/find` queries such as `role:button Save`, `text:Submit`, `label:Email`, `placeholder:Search`, `alt:Logo`, `title:Close`, `testid:submit`, `first:role:button`, `last:text:Submit`, and `nth:2:label:Email` are matched by the semantic engine against enriched descriptors.
+- Structured `/find` queries such as `role:button Save`, `text:Submit`, `label:Email`, `placeholder:Search`, `alt:Logo`, `title:Close`, `testid:submit`, `first:role:button`, `last:text:Submit`, and `nth:2:label:Email` are matched by the semantic engine against enriched descriptors. A wrapper index is zero-based here too, exactly as it is over `css:`/`xpath:`/`text:` — `nth:0:label:Email` is the first match in document order — and an index past the last match refuses by saying so rather than reporting that nothing matched.
- In action commands, `role:`, `label:`, `placeholder:`, `alt:`, `title:`, `testid:`, and wrappers around those forms use semantic matching. CSS, XPath, refs, the existing `text:` action selector, and bare CSS/text wrappers such as `first:button` remain browser-side selector resolution.
- If there is no cached snapshot, PinchTab tries to refresh it automatically before matching.
- Successful matches are useful inputs to `/action`, `/actions`, and higher-level recovery logic.
diff --git a/docs/reference/mcp-tools.md b/docs/reference/mcp-tools.md
index a342c161..3087747a 100644
--- a/docs/reference/mcp-tools.md
+++ b/docs/reference/mcp-tools.md
@@ -17,7 +17,7 @@ Selector forms include:
- `label:Email`, `placeholder:Search`, `alt:Logo`, `title:Close`, `testid:submit`
- `first:button`, `last:button`, `nth:2:button`
-Positional wrappers index the candidates in **document order** over `css:`, `xpath:`, and `text:` selectors, so `nth:0:` is the first match in the page and `nth:1:` always comes after it. A bare `text:` selector is the one form of those that does not index: it picks the most control-like match among the smallest ones, so `text:Save` prefers a `<button>` over a `<div>` carrying the same label — which means `text:X` and `first:text:X` can resolve to different elements. A wrapper only ever chooses among the matches a bare selector would find; it never changes which matches exist.
+Positional wrappers index the candidates in **document order** for every selector kind, so `nth:0:` is the first match in the page and `nth:1:` always comes after it. A bare `text:` selector is the one form that does not index: it picks the most control-like match among the smallest ones, so `text:Save` prefers a `<button>` over a `<div>` carrying the same label — which means `text:X` and `first:text:X` can resolve to different elements. A wrapper only ever chooses among the matches a bare selector would find; it never changes which matches exist.
Structured semantic locators are matched by the semantic engine; CSS, XPath, refs, the existing `text:` action selector, and bare CSS/text wrappers stay browser-side.
diff --git a/internal/handlers/semantic_selectors.go b/internal/handlers/semantic_selectors.go
index 2f18f90f..fa80de6a 100644
--- a/internal/handlers/semantic_selectors.go
+++ b/internal/handlers/semantic_selectors.go
@@ -70,7 +70,7 @@ func (h *Handlers) applySemanticActionSelectorInScope(ctx context.Context, tabID
return true, fmt.Errorf("semantic selector: %w", err)
}
if result.BestRef == "" {
- return true, fmt.Errorf("semantic selector %q: no matching element found", query)
+ return true, h.emptySemanticMatchError(ctx, sel, query, descs)
}
if scopeBackendNodeID != 0 {
@@ -138,6 +138,7 @@ func semanticSelectorHTTPStatus(err error) int {
case strings.Contains(msg, "not configured"):
return http.StatusNotImplemented
case strings.Contains(msg, "no matching element found"),
+ strings.Contains(msg, "is out of range"),
strings.Contains(msg, "no node is available"):
return http.StatusNotFound
case strings.Contains(msg, "no snapshot available"),
@@ -147,3 +148,32 @@ func semanticSelectorHTTPStatus(err error) int {
return http.StatusInternalServerError
}
}
+
+// emptySemanticMatchError distinguishes an empty page from an index nobody can satisfy.
+// A positional nth: wrapper that runs off the end of the match set otherwise reports
+// "no matching element found" while the elements are plainly there, which reads as a
+// broken page rather than a bad index. The message names the index the CALLER wrote —
+// zero-based, the spelling this project publishes — never the translated one the matcher
+// received.
+func (h *Handlers) emptySemanticMatchError(ctx context.Context, sel selector.Selector, query string, descs []semantic.ElementDescriptor) error {
+ index, base, ok := sel.SemanticNthBase()
+ if !ok {
+ return fmt.Errorf("semantic selector %q: no matching element found", query)
+ }
+ matches := h.countSemanticMatches(ctx, base, descs)
+ if matches == 0 {
+ return fmt.Errorf("semantic selector %q: no matching element found", sel.String())
+ }
+ return fmt.Errorf("semantic selector %q: index %d is out of range, %q matched %d element(s)", sel.String(), index, base, matches)
+}
+
+func (h *Handlers) countSemanticMatches(ctx context.Context, base string, descs []semantic.ElementDescriptor) int {
+ result, err := h.Matcher.Find(ctx, base, descs, semantic.FindOptions{
+ Threshold: 0.3,
+ TopK: len(descs),
+ })
+ if err != nil {
+ return 0
+ }
+ return len(result.Matches)
+}
diff --git a/internal/handlers/semantic_wrapper_index_test.go b/internal/handlers/semantic_wrapper_index_test.go
new file mode 100644
index 00000000..260b4640
--- /dev/null
+++ b/internal/handlers/semantic_wrapper_index_test.go
@@ -0,0 +1,213 @@
+package handlers
+
+import (
+ "context"
+ "encoding/base64"
+ "fmt"
+ "os"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/chromedp/chromedp"
+ "github.com/pinchtab/pinchtab/internal/bridge"
+ "github.com/pinchtab/pinchtab/internal/config"
+ "github.com/pinchtab/pinchtab/internal/selector"
+ "github.com/pinchtab/pinchtab/internal/testbrowser"
+ "github.com/pinchtab/semantic"
+)
+
+func newWrapperIndexHandler(t *testing.T) *Handlers {
+ t.Helper()
+
+ cache := &bridge.RefCache{
+ Nodes: []bridge.A11yNode{
+ {Ref: "e0", Role: "button", Name: "Save"},
+ {Ref: "e1", Role: "button", Name: "Save"},
+ {Ref: "e2", Role: "button", Name: "Save"},
+ },
+ Refs: map[string]int64{"e0": 1, "e1": 2, "e2": 3},
+ }
+ h := New(&findMockBridge{refCache: cache}, &config.RuntimeConfig{ActionTimeout: 10 * time.Second}, nil, nil, nil)
+ h.Matcher = semantic.NewCombinedMatcher(semantic.NewHashingEmbedder(128))
+ return h
+}
+
+func resolveWrapperSelector(t *testing.T, h *Handlers, raw string) (string, error) {
+ t.Helper()
+
+ req := bridge.ActionRequest{}
+ handled, err := h.applySemanticActionSelector(context.Background(), "tab1", selector.Parse(raw), &req)
+ if !handled {
+ t.Fatalf("%s did not route to the semantic matcher, so this test is not exercising the wrapper path", raw)
+ }
+ return req.Ref, err
+}
+
+// TestAPositionalWrapperOverASemanticFormIsZeroBased is the defect: the matcher
+// publishes nth as one-based, this project publishes it as zero-based, and without the
+// translation at the boundary the documented nth:0 matched nothing at all while the
+// elements were plainly on the page.
+func TestAPositionalWrapperOverASemanticFormIsZeroBased(t *testing.T) {
+ h := newWrapperIndexHandler(t)
+
+ for _, tc := range []struct{ raw, wantRef string }{
+ {"role:button Save", "e0"},
+ {"first:role:button Save", "e0"},
+ {"nth:0:role:button Save", "e0"},
+ {"nth:1:role:button Save", "e1"},
+ {"nth:2:role:button Save", "e2"},
+ {"last:role:button Save", "e2"},
+ } {
+ ref, err := resolveWrapperSelector(t, h, tc.raw)
+ if err != nil {
+ t.Errorf("%s: %v", tc.raw, err)
+ continue
+ }
+ if ref != tc.wantRef {
+ t.Errorf("%s resolved %s, want %s", tc.raw, ref, tc.wantRef)
+ }
+ }
+}
+
+// TestAnOutOfRangeWrapperIndexSaysSoInsteadOfNoMatch: an agent cannot act on "no
+// matching element found" when the elements exist and only the index is wrong. The
+// message must name the index the caller wrote — zero-based, this project's spelling —
+// and must not leak the translated one-based index the matcher received.
+func TestAnOutOfRangeWrapperIndexSaysSoInsteadOfNoMatch(t *testing.T) {
+ h := newWrapperIndexHandler(t)
+
+ _, err := resolveWrapperSelector(t, h, "nth:7:role:button Save")
+ if err == nil {
+ t.Fatal("nth:7 over three matches must refuse")
+ }
+ msg := err.Error()
+ for _, needle := range []string{"index 7", "out of range", "3 element(s)"} {
+ if !strings.Contains(msg, needle) {
+ t.Errorf("refusal %q does not carry %q", msg, needle)
+ }
+ }
+ if strings.Contains(msg, "no matching element found") {
+ t.Errorf("refusal %q reports an empty page for what is only a bad index", msg)
+ }
+ if strings.Contains(msg, "nth:8") {
+ t.Errorf("refusal %q names the translated one-based index; it must name what the caller sent", msg)
+ }
+ if status := semanticSelectorHTTPStatus(err); status != 404 {
+ t.Errorf("out-of-range refusal maps to HTTP %d, want 404 rather than a 500 that reads as a server fault", status)
+ }
+}
+
+func TestAWrapperOverASemanticFormWithNoMatchesStillReportsNoMatch(t *testing.T) {
+ h := newWrapperIndexHandler(t)
+
+ _, err := resolveWrapperSelector(t, h, "nth:0:role:slider Volume")
+ if err == nil {
+ t.Fatal("a wrapper over a selector nothing matches must refuse")
+ }
+ if !strings.Contains(err.Error(), "no matching element found") {
+ t.Errorf("refusal %q should report an empty match set, not an index problem", err.Error())
+ }
+}
+
+// TestTheSameNthSelectsTheSameOrdinalOnBothPaths is the agreement the card asks for:
+// one page, one index, both resolution paths. A one-sided test cannot catch the base
+// drifting again, and the base is what drifted — the semantic path goes through the
+// matcher's one-based nth while css goes through the browser-side zero-based resolver.
+func TestTheSameNthSelectsTheSameOrdinalOnBothPaths(t *testing.T) {
+ chromePath := testbrowser.Path(t)
+ alloc, cancelAlloc := chromedp.NewExecAllocator(context.Background(), append(
+ chromedp.DefaultExecAllocatorOptions[:],
+ chromedp.ExecPath(chromePath),
+ chromedp.UserDataDir(t.TempDir()),
+ chromedp.Flag("headless", true),
+ chromedp.Flag("no-sandbox", true),
+ )...)
+ ctx, cancelBrowser := chromedp.NewContext(alloc)
+ ctx, cancelTimeout := context.WithTimeout(ctx, 30*time.Second)
+ t.Cleanup(func() {
+ cancelTimeout()
+ cancelBrowser()
+ cancelAlloc()
+ })
+
+ html := `<button id="first">Save</button><button id="second">Save</button><button id="third">Save</button>`
+ if err := chromedp.Run(ctx,
+ chromedp.Navigate("data:text/html;base64,"+base64.StdEncoding.EncodeToString([]byte(html))),
+ chromedp.WaitVisible("#third", chromedp.ByID),
+ ); err != nil {
+ t.Fatal(err)
+ }
+
+ cfg := &config.RuntimeConfig{ActionTimeout: 10 * time.Second, DefaultBrowser: config.BrowserChrome, StateDir: t.TempDir()}
+ b := bridge.New(context.Background(), ctx, cfg)
+ b.RegisterTab("tab-nth", ctx)
+ h := New(b, cfg, nil, nil, nil)
+ h.Matcher = semantic.NewCombinedMatcher(semantic.NewHashingEmbedder(128))
+
+ elementID := func(t *testing.T, raw string) string {
+ t.Helper()
+ req := bridge.ActionRequest{Selector: raw}
+ if _, err := h.resolveActionRequestSelector(ctx, "tab-nth", &req); err != nil {
+ t.Fatalf("resolve %s: %v", raw, err)
+ }
+ if req.NodeID == 0 {
+ t.Fatalf("resolve %s produced no node", raw)
+ }
+ var id string
+ if err := b.CallFunctionOnNode(ctx, req.NodeID, `function() { return this.id; }`, nil, &id); err != nil {
+ t.Fatalf("describe %s: %v", raw, err)
+ }
+ return id
+ }
+
+ for index, wantID := range []string{"first", "second", "third"} {
+ css := fmt.Sprintf("nth:%d:css:button", index)
+ role := fmt.Sprintf("nth:%d:role:button Save", index)
+
+ gotCSS := elementID(t, css)
+ gotRole := elementID(t, role)
+ if gotCSS != wantID {
+ t.Errorf("%s selected #%s, want #%s", css, gotCSS, wantID)
+ }
+ if gotRole != gotCSS {
+ t.Errorf("%s selected #%s but %s selected #%s; the same index must mean the same ordinal on both paths", role, gotRole, css, gotCSS)
+ }
+ }
+}
+
+// TestOnlyTheSelectorPackageTouchesTheWrapperIndex keeps the base translation in one
+// owner. Both SemanticQuery consumers in this package — the action path and count.go —
+// get the converted index for free; a second adjustment here is how the two bases
+// drifted apart in the first place.
+func TestOnlyTheSelectorPackageTouchesTheWrapperIndex(t *testing.T) {
+ entries, err := os.ReadDir(".")
+ if err != nil {
+ t.Fatal(err)
+ }
+ consumers, scanned := 0, 0
+ for _, entry := range entries {
+ name := entry.Name()
+ if entry.IsDir() || !strings.HasSuffix(name, ".go") || strings.HasSuffix(name, "_test.go") {
+ continue
+ }
+ scanned++
+ src, err := os.ReadFile(name)
+ if err != nil {
+ t.Fatal(err)
+ }
+ body := string(src)
+ if strings.Contains(body, `"nth:`) {
+ t.Errorf("%s builds an nth: query itself; the index base is translated once, in internal/selector", name)
+ }
+ if strings.Contains(body, ".SemanticQuery()") {
+ consumers++
+ }
+ }
+ if scanned == 0 {
+ t.Fatal("no non-test files scanned, so this census read nothing")
+ }
+ if consumers < 2 {
+ t.Errorf("found %d SemanticQuery consumers, want at least 2 (the action path and count.go); a consumer that stopped routing through it would keep the old base", consumers)
+ }
+}
diff --git a/internal/mcp/tools.go b/internal/mcp/tools.go
index 257ea497..bf525f62 100644
--- a/internal/mcp/tools.go
+++ b/internal/mcp/tools.go
@@ -2,7 +2,7 @@ package mcp
import "github.com/mark3labs/mcp-go/mcp"
-const positionalWrapperGrammar = " first/last/nth index matches in document order over css:, xpath: and text: selectors, so nth:0 is the first match in the page. A bare text: selector instead picks the most control-like smallest match, so text:X and first:text:X can differ."
+const positionalWrapperGrammar = " first/last/nth index matches in document order for every selector kind, so nth:0 is the first match in the page. A bare text: selector instead picks the most control-like smallest match, so text:X and first:text:X can differ."
// allTools returns every MCP tool exposed by the PinchTab MCP server.
func allTools() []mcp.Tool {
diff --git a/internal/selector/selector.go b/internal/selector/selector.go
index 82e34ca8..73ebdd1c 100644
--- a/internal/selector/selector.go
+++ b/internal/selector/selector.go
@@ -233,14 +233,39 @@ func (s Selector) SemanticQuery() (string, bool) {
return s.String(), true
}
case KindNth:
- _, raw, err := ParseNth(s.Value)
+ index, raw, err := ParseNth(s.Value)
if err == nil && rawSelectorCanUseSemantic(raw) {
- return s.String(), true
+ return fmt.Sprintf("%s%d:%s", nthPrefix, index+semanticNthOffset, raw), true
}
}
return "", false
}
+// semanticNthOffset translates PinchTab's zero-based nth into the one-based nth the
+// semantic matcher publishes. It is an adapter between two documented grammars, not an
+// off-by-one to be simplified away: the matcher's README states that nth:<n> is 1-based,
+// nth:1 selects the first ordered candidate and nth:0 is not the first match, while this
+// project documents nth as zero-based for every selector kind. Both remain correct; the
+// boundary converts. Removing it makes the documented nth:0 match nothing.
+const semanticNthOffset = 1
+
+const nthPrefix = "nth:"
+
+// SemanticNthBase splits a positional nth wrapper over a semantic form into the
+// caller's ZERO-based index and the bare query underneath it, for a caller that has to
+// say how many matches the unwrapped selector had. It answers false for every other
+// selector, including first:/last:, which cannot be out of range while any match exists.
+func (s Selector) SemanticNthBase() (index int, base string, ok bool) {
+ if s.Kind != KindNth {
+ return 0, "", false
+ }
+ index, raw, err := ParseNth(s.Value)
+ if err != nil || !rawSelectorCanUseSemantic(raw) {
+ return 0, "", false
+ }
+ return index, raw, true
+}
+
func rawSelectorCanUseSemantic(raw string) bool {
sel := Parse(raw)
switch sel.Kind {
diff --git a/internal/selector/selector_test.go b/internal/selector/selector_test.go
index 92a22c5a..1b808d63 100644
--- a/internal/selector/selector_test.go
+++ b/internal/selector/selector_test.go
@@ -371,7 +371,7 @@ func TestSelector_SemanticQuery(t *testing.T) {
{"testid:submit", "testid:submit", true},
{"first:text:Submit", "", false},
{"last:role:button Save", "last:role:button Save", true},
- {"nth:2:label:Email", "nth:2:label:Email", true},
+ {"nth:2:label:Email", "nth:3:label:Email", true},
{"first:button", "", false},
{"last:css:button", "", false},
{"nth:2:button", "", false},
@@ -402,14 +402,19 @@ func TestPositionalWrapperOverSemanticFormReachesTheMatcherWithItsIndex(t *testi
for _, prefix := range semanticPrefixes {
bare := prefix + "Save"
- for _, wrapped := range []string{"first:" + bare, "last:" + bare, "nth:0:" + bare, "nth:2:" + bare} {
- query, ok := Parse(wrapped).SemanticQuery()
+ for _, tc := range []struct{ wrapped, want string }{
+ {"first:" + bare, "first:" + bare},
+ {"last:" + bare, "last:" + bare},
+ {"nth:0:" + bare, "nth:1:" + bare},
+ {"nth:2:" + bare, "nth:3:" + bare},
+ } {
+ query, ok := Parse(tc.wrapped).SemanticQuery()
if !ok {
- t.Errorf("%s no longer routes to the semantic matcher, so the wrapper it carries is never applied", wrapped)
+ t.Errorf("%s no longer routes to the semantic matcher, so the wrapper it carries is never applied", tc.wrapped)
continue
}
- if query != wrapped {
- t.Errorf("SemanticQuery(%q) = %q: the wrapper must reach the matcher intact, since the semantic engine parses the index itself and stripping it here drops the position the caller asked for", wrapped, query)
+ if query != tc.want {
+ t.Errorf("SemanticQuery(%q) = %q, want %q: the wrapper must reach the matcher with its index, translated into the one-based nth the matcher publishes", tc.wrapped, query, tc.want)
}
}
}
@@ -666,3 +671,44 @@ func TestParseMixedCasePrefixesMatchTheirLowercaseForm(t *testing.T) {
t.Errorf(`Parse("CSS:#id") = %+v, want css #id`, got)
}
}
+
+// TestTheSemanticNthOffsetIsAnAdapterNotAnOffByOne states why the +1 exists, so it is
+// not "simplified" away by someone who sees the arithmetic and assumes a bug. The two
+// bases are both documented: this project publishes nth as zero-based for every
+// selector kind, and the semantic matcher's README publishes nth as one-based, where
+// nth:0 is not the first match. Deleting the offset makes the documented nth:0 select
+// nothing on the whole semantic family.
+func TestTheSemanticNthOffsetIsAnAdapterNotAnOffByOne(t *testing.T) {
+ if semanticNthOffset != 1 {
+ t.Fatalf("semanticNthOffset = %d, want 1: PinchTab's public nth is zero-based and the matcher's is one-based", semanticNthOffset)
+ }
+
+ query, ok := Parse("nth:0:role:button Save").SemanticQuery()
+ if !ok {
+ t.Fatal("nth over a semantic form must reach the matcher")
+ }
+ if query != "nth:1:role:button Save" {
+ t.Errorf("the public first match nth:0 arrives as %q; the matcher treats nth:0 as out of range, so it must arrive as nth:1", query)
+ }
+}
+
+func TestSemanticNthBaseReportsTheCallersOwnIndex(t *testing.T) {
+ for _, tc := range []struct {
+ input string
+ wantIndex int
+ wantBase string
+ wantOK bool
+ }{
+ {"nth:0:role:button Save", 0, "role:button Save", true},
+ {"nth:2:label:Email", 2, "label:Email", true},
+ {"first:role:button", 0, "", false},
+ {"last:role:button", 0, "", false},
+ {"nth:2:css:button", 0, "", false},
+ {"role:button", 0, "", false},
+ } {
+ index, base, ok := Parse(tc.input).SemanticNthBase()
+ if ok != tc.wantOK || index != tc.wantIndex || base != tc.wantBase {
+ t.Errorf("SemanticNthBase(%q) = (%d, %q, %v), want (%d, %q, %v)", tc.input, index, base, ok, tc.wantIndex, tc.wantBase, tc.wantOK)
+ }
+ }
+}