High-performance browser automation bridge and multi-instance orchestrator with advanced stealth injection and real-time dashboard.
fix: nth: is one-based over role/label/testid selectors and zero-based over css/xpath/text, so the documented nth:0 finds nothing on a whole selector family (PIN-163)
Luigi Agostib3a300ba25b7
diff --git a/docs/commands.md b/docs/commands.mdindex ef1a4ab1..2aef13d2 100644--- a/docs/commands.md+++ b/docs/commands.md@@ -107,7 +107,7 @@ Most element commands accept a unified selector:- label, placeholder, alt, title, or test id selectors such as `label:Email`, `placeholder:Search`, `alt:Logo`, `title:Close`, `testid:submit`- positional wrappers such as `first:button`, `last:role:button`, or `nth:2:button` (`nth` is zero-based)-Positional wrappers index the candidates in **document order** over `css:`, `xpath:`, and `text:` selectors, so `nth:0:` is the first match in the page and `nth:1:` always comes after it. A bare `text:` selector is the one form of those that does not index: it picks the most control-like match among the smallest ones, so `text:Save` prefers a `<button>` over a `<div>` carrying the same label — which means `text:X` and `first:text:X` can resolve to different elements. A wrapper only ever chooses among the matches a bare selector would find; it never changes which matches exist.+Positional wrappers index the candidates in **document order** for every selector kind, so `nth:0:` is the first match in the page and `nth:1:` always comes after it. A bare `text:` selector is the one form that does not index: it picks the most control-like match among the smallest ones, so `text:Save` prefers a `<button>` over a `<div>` carrying the same label — which means `text:X` and `first:text:X` can resolve to different elements. A wrapper only ever chooses among the matches a bare selector would find; it never changes which matches exist.Selector prefixes are case-insensitive, so `CSS:#login` and `css:#login` mean the same thing. Only the prefix is case-folded; the value after it is passed through unchanged.diff --git a/docs/reference/find.md b/docs/reference/find.mdindex 69e158fb..27650fcf 100644--- a/docs/reference/find.md+++ b/docs/reference/find.md@@ -158,7 +158,7 @@ curl -X POST http://localhost:9867/tabs/<tabId>/action \## Operational Notes- `/find` uses the tab's accessibility snapshot, not raw DOM selectors.-- Structured `/find` queries such as `role:button Save`, `text:Submit`, `label:Email`, `placeholder:Search`, `alt:Logo`, `title:Close`, `testid:submit`, `first:role:button`, `last:text:Submit`, and `nth:2:label:Email` are matched by the semantic engine against enriched descriptors.+- Structured `/find` queries such as `role:button Save`, `text:Submit`, `label:Email`, `placeholder:Search`, `alt:Logo`, `title:Close`, `testid:submit`, `first:role:button`, `last:text:Submit`, and `nth:2:label:Email` are matched by the semantic engine against enriched descriptors. A wrapper index is zero-based here too, exactly as it is over `css:`/`xpath:`/`text:` — `nth:0:label:Email` is the first match in document order — and an index past the last match refuses by saying so rather than reporting that nothing matched.- In action commands, `role:`, `label:`, `placeholder:`, `alt:`, `title:`, `testid:`, and wrappers around those forms use semantic matching. CSS, XPath, refs, the existing `text:` action selector, and bare CSS/text wrappers such as `first:button` remain browser-side selector resolution.- If there is no cached snapshot, PinchTab tries to refresh it automatically before matching.- Successful matches are useful inputs to `/action`, `/actions`, and higher-level recovery logic.diff --git a/docs/reference/mcp-tools.md b/docs/reference/mcp-tools.mdindex a342c161..3087747a 100644--- a/docs/reference/mcp-tools.md+++ b/docs/reference/mcp-tools.md@@ -17,7 +17,7 @@ Selector forms include:- `label:Email`, `placeholder:Search`, `alt:Logo`, `title:Close`, `testid:submit`- `first:button`, `last:button`, `nth:2:button`-Positional wrappers index the candidates in **document order** over `css:`, `xpath:`, and `text:` selectors, so `nth:0:` is the first match in the page and `nth:1:` always comes after it. A bare `text:` selector is the one form of those that does not index: it picks the most control-like match among the smallest ones, so `text:Save` prefers a `<button>` over a `<div>` carrying the same label — which means `text:X` and `first:text:X` can resolve to different elements. A wrapper only ever chooses among the matches a bare selector would find; it never changes which matches exist.+Positional wrappers index the candidates in **document order** for every selector kind, so `nth:0:` is the first match in the page and `nth:1:` always comes after it. A bare `text:` selector is the one form that does not index: it picks the most control-like match among the smallest ones, so `text:Save` prefers a `<button>` over a `<div>` carrying the same label — which means `text:X` and `first:text:X` can resolve to different elements. A wrapper only ever chooses among the matches a bare selector would find; it never changes which matches exist.Structured semantic locators are matched by the semantic engine; CSS, XPath, refs, the existing `text:` action selector, and bare CSS/text wrappers stay browser-side.diff --git a/internal/handlers/semantic_selectors.go b/internal/handlers/semantic_selectors.goindex 2f18f90f..fa80de6a 100644--- a/internal/handlers/semantic_selectors.go+++ b/internal/handlers/semantic_selectors.go@@ -70,7 +70,7 @@ func (h *Handlers) applySemanticActionSelectorInScope(ctx context.Context, tabIDreturn true, fmt.Errorf("semantic selector: %w", err)}if result.BestRef == "" {- return true, fmt.Errorf("semantic selector %q: no matching element found", query)+ return true, h.emptySemanticMatchError(ctx, sel, query, descs)}if scopeBackendNodeID != 0 {@@ -138,6 +138,7 @@ func semanticSelectorHTTPStatus(err error) int {case strings.Contains(msg, "not configured"):return http.StatusNotImplementedcase strings.Contains(msg, "no matching element found"),+ strings.Contains(msg, "is out of range"),strings.Contains(msg, "no node is available"):return http.StatusNotFoundcase strings.Contains(msg, "no snapshot available"),@@ -147,3 +148,32 @@ func semanticSelectorHTTPStatus(err error) int {return http.StatusInternalServerError}}++// emptySemanticMatchError distinguishes an empty page from an index nobody can satisfy.+// A positional nth: wrapper that runs off the end of the match set otherwise reports+// "no matching element found" while the elements are plainly there, which reads as a+// broken page rather than a bad index. The message names the index the CALLER wrote —+// zero-based, the spelling this project publishes — never the translated one the matcher+// received.+func (h *Handlers) emptySemanticMatchError(ctx context.Context, sel selector.Selector, query string, descs []semantic.ElementDescriptor) error {+ index, base, ok := sel.SemanticNthBase()+ if !ok {+ return fmt.Errorf("semantic selector %q: no matching element found", query)+ }+ matches := h.countSemanticMatches(ctx, base, descs)+ if matches == 0 {+ return fmt.Errorf("semantic selector %q: no matching element found", sel.String())+ }+ return fmt.Errorf("semantic selector %q: index %d is out of range, %q matched %d element(s)", sel.String(), index, base, matches)+}++func (h *Handlers) countSemanticMatches(ctx context.Context, base string, descs []semantic.ElementDescriptor) int {+ result, err := h.Matcher.Find(ctx, base, descs, semantic.FindOptions{+ Threshold: 0.3,+ TopK: len(descs),+ })+ if err != nil {+ return 0+ }+ return len(result.Matches)+}diff --git a/internal/handlers/semantic_wrapper_index_test.go b/internal/handlers/semantic_wrapper_index_test.gonew file mode 100644index 00000000..260b4640--- /dev/null+++ b/internal/handlers/semantic_wrapper_index_test.go@@ -0,0 +1,213 @@+package handlers++import (+ "context"+ "encoding/base64"+ "fmt"+ "os"+ "strings"+ "testing"+ "time"++ "github.com/chromedp/chromedp"+ "github.com/pinchtab/pinchtab/internal/bridge"+ "github.com/pinchtab/pinchtab/internal/config"+ "github.com/pinchtab/pinchtab/internal/selector"+ "github.com/pinchtab/pinchtab/internal/testbrowser"+ "github.com/pinchtab/semantic"+)++func newWrapperIndexHandler(t *testing.T) *Handlers {+ t.Helper()++ cache := &bridge.RefCache{+ Nodes: []bridge.A11yNode{+ {Ref: "e0", Role: "button", Name: "Save"},+ {Ref: "e1", Role: "button", Name: "Save"},+ {Ref: "e2", Role: "button", Name: "Save"},+ },+ Refs: map[string]int64{"e0": 1, "e1": 2, "e2": 3},+ }+ h := New(&findMockBridge{refCache: cache}, &config.RuntimeConfig{ActionTimeout: 10 * time.Second}, nil, nil, nil)+ h.Matcher = semantic.NewCombinedMatcher(semantic.NewHashingEmbedder(128))+ return h+}++func resolveWrapperSelector(t *testing.T, h *Handlers, raw string) (string, error) {+ t.Helper()++ req := bridge.ActionRequest{}+ handled, err := h.applySemanticActionSelector(context.Background(), "tab1", selector.Parse(raw), &req)+ if !handled {+ t.Fatalf("%s did not route to the semantic matcher, so this test is not exercising the wrapper path", raw)+ }+ return req.Ref, err+}++// TestAPositionalWrapperOverASemanticFormIsZeroBased is the defect: the matcher+// publishes nth as one-based, this project publishes it as zero-based, and without the+// translation at the boundary the documented nth:0 matched nothing at all while the+// elements were plainly on the page.+func TestAPositionalWrapperOverASemanticFormIsZeroBased(t *testing.T) {+ h := newWrapperIndexHandler(t)++ for _, tc := range []struct{ raw, wantRef string }{+ {"role:button Save", "e0"},+ {"first:role:button Save", "e0"},+ {"nth:0:role:button Save", "e0"},+ {"nth:1:role:button Save", "e1"},+ {"nth:2:role:button Save", "e2"},+ {"last:role:button Save", "e2"},+ } {+ ref, err := resolveWrapperSelector(t, h, tc.raw)+ if err != nil {+ t.Errorf("%s: %v", tc.raw, err)+ continue+ }+ if ref != tc.wantRef {+ t.Errorf("%s resolved %s, want %s", tc.raw, ref, tc.wantRef)+ }+ }+}++// TestAnOutOfRangeWrapperIndexSaysSoInsteadOfNoMatch: an agent cannot act on "no+// matching element found" when the elements exist and only the index is wrong. The+// message must name the index the caller wrote — zero-based, this project's spelling —+// and must not leak the translated one-based index the matcher received.+func TestAnOutOfRangeWrapperIndexSaysSoInsteadOfNoMatch(t *testing.T) {+ h := newWrapperIndexHandler(t)++ _, err := resolveWrapperSelector(t, h, "nth:7:role:button Save")+ if err == nil {+ t.Fatal("nth:7 over three matches must refuse")+ }+ msg := err.Error()+ for _, needle := range []string{"index 7", "out of range", "3 element(s)"} {+ if !strings.Contains(msg, needle) {+ t.Errorf("refusal %q does not carry %q", msg, needle)+ }+ }+ if strings.Contains(msg, "no matching element found") {+ t.Errorf("refusal %q reports an empty page for what is only a bad index", msg)+ }+ if strings.Contains(msg, "nth:8") {+ t.Errorf("refusal %q names the translated one-based index; it must name what the caller sent", msg)+ }+ if status := semanticSelectorHTTPStatus(err); status != 404 {+ t.Errorf("out-of-range refusal maps to HTTP %d, want 404 rather than a 500 that reads as a server fault", status)+ }+}++func TestAWrapperOverASemanticFormWithNoMatchesStillReportsNoMatch(t *testing.T) {+ h := newWrapperIndexHandler(t)++ _, err := resolveWrapperSelector(t, h, "nth:0:role:slider Volume")+ if err == nil {+ t.Fatal("a wrapper over a selector nothing matches must refuse")+ }+ if !strings.Contains(err.Error(), "no matching element found") {+ t.Errorf("refusal %q should report an empty match set, not an index problem", err.Error())+ }+}++// TestTheSameNthSelectsTheSameOrdinalOnBothPaths is the agreement the card asks for:+// one page, one index, both resolution paths. A one-sided test cannot catch the base+// drifting again, and the base is what drifted — the semantic path goes through the+// matcher's one-based nth while css goes through the browser-side zero-based resolver.+func TestTheSameNthSelectsTheSameOrdinalOnBothPaths(t *testing.T) {+ chromePath := testbrowser.Path(t)+ alloc, cancelAlloc := chromedp.NewExecAllocator(context.Background(), append(+ chromedp.DefaultExecAllocatorOptions[:],+ chromedp.ExecPath(chromePath),+ chromedp.UserDataDir(t.TempDir()),+ chromedp.Flag("headless", true),+ chromedp.Flag("no-sandbox", true),+ )...)+ ctx, cancelBrowser := chromedp.NewContext(alloc)+ ctx, cancelTimeout := context.WithTimeout(ctx, 30*time.Second)+ t.Cleanup(func() {+ cancelTimeout()+ cancelBrowser()+ cancelAlloc()+ })++ html := `<button id="first">Save</button><button id="second">Save</button><button id="third">Save</button>`+ if err := chromedp.Run(ctx,+ chromedp.Navigate("data:text/html;base64,"+base64.StdEncoding.EncodeToString([]byte(html))),+ chromedp.WaitVisible("#third", chromedp.ByID),+ ); err != nil {+ t.Fatal(err)+ }++ cfg := &config.RuntimeConfig{ActionTimeout: 10 * time.Second, DefaultBrowser: config.BrowserChrome, StateDir: t.TempDir()}+ b := bridge.New(context.Background(), ctx, cfg)+ b.RegisterTab("tab-nth", ctx)+ h := New(b, cfg, nil, nil, nil)+ h.Matcher = semantic.NewCombinedMatcher(semantic.NewHashingEmbedder(128))++ elementID := func(t *testing.T, raw string) string {+ t.Helper()+ req := bridge.ActionRequest{Selector: raw}+ if _, err := h.resolveActionRequestSelector(ctx, "tab-nth", &req); err != nil {+ t.Fatalf("resolve %s: %v", raw, err)+ }+ if req.NodeID == 0 {+ t.Fatalf("resolve %s produced no node", raw)+ }+ var id string+ if err := b.CallFunctionOnNode(ctx, req.NodeID, `function() { return this.id; }`, nil, &id); err != nil {+ t.Fatalf("describe %s: %v", raw, err)+ }+ return id+ }++ for index, wantID := range []string{"first", "second", "third"} {+ css := fmt.Sprintf("nth:%d:css:button", index)+ role := fmt.Sprintf("nth:%d:role:button Save", index)++ gotCSS := elementID(t, css)+ gotRole := elementID(t, role)+ if gotCSS != wantID {+ t.Errorf("%s selected #%s, want #%s", css, gotCSS, wantID)+ }+ if gotRole != gotCSS {+ t.Errorf("%s selected #%s but %s selected #%s; the same index must mean the same ordinal on both paths", role, gotRole, css, gotCSS)+ }+ }+}++// TestOnlyTheSelectorPackageTouchesTheWrapperIndex keeps the base translation in one+// owner. Both SemanticQuery consumers in this package — the action path and count.go —+// get the converted index for free; a second adjustment here is how the two bases+// drifted apart in the first place.+func TestOnlyTheSelectorPackageTouchesTheWrapperIndex(t *testing.T) {+ entries, err := os.ReadDir(".")+ if err != nil {+ t.Fatal(err)+ }+ consumers, scanned := 0, 0+ for _, entry := range entries {+ name := entry.Name()+ if entry.IsDir() || !strings.HasSuffix(name, ".go") || strings.HasSuffix(name, "_test.go") {+ continue+ }+ scanned+++ src, err := os.ReadFile(name)+ if err != nil {+ t.Fatal(err)+ }+ body := string(src)+ if strings.Contains(body, `"nth:`) {+ t.Errorf("%s builds an nth: query itself; the index base is translated once, in internal/selector", name)+ }+ if strings.Contains(body, ".SemanticQuery()") {+ consumers+++ }+ }+ if scanned == 0 {+ t.Fatal("no non-test files scanned, so this census read nothing")+ }+ if consumers < 2 {+ t.Errorf("found %d SemanticQuery consumers, want at least 2 (the action path and count.go); a consumer that stopped routing through it would keep the old base", consumers)+ }+}diff --git a/internal/mcp/tools.go b/internal/mcp/tools.goindex 257ea497..bf525f62 100644--- a/internal/mcp/tools.go+++ b/internal/mcp/tools.go@@ -2,7 +2,7 @@ package mcpimport "github.com/mark3labs/mcp-go/mcp"-const positionalWrapperGrammar = " first/last/nth index matches in document order over css:, xpath: and text: selectors, so nth:0 is the first match in the page. A bare text: selector instead picks the most control-like smallest match, so text:X and first:text:X can differ."+const positionalWrapperGrammar = " first/last/nth index matches in document order for every selector kind, so nth:0 is the first match in the page. A bare text: selector instead picks the most control-like smallest match, so text:X and first:text:X can differ."// allTools returns every MCP tool exposed by the PinchTab MCP server.func allTools() []mcp.Tool {diff --git a/internal/selector/selector.go b/internal/selector/selector.goindex 82e34ca8..73ebdd1c 100644--- a/internal/selector/selector.go+++ b/internal/selector/selector.go@@ -233,14 +233,39 @@ func (s Selector) SemanticQuery() (string, bool) {return s.String(), true}case KindNth:- _, raw, err := ParseNth(s.Value)+ index, raw, err := ParseNth(s.Value)if err == nil && rawSelectorCanUseSemantic(raw) {- return s.String(), true+ return fmt.Sprintf("%s%d:%s", nthPrefix, index+semanticNthOffset, raw), true}}return "", false}+// semanticNthOffset translates PinchTab's zero-based nth into the one-based nth the+// semantic matcher publishes. It is an adapter between two documented grammars, not an+// off-by-one to be simplified away: the matcher's README states that nth:<n> is 1-based,+// nth:1 selects the first ordered candidate and nth:0 is not the first match, while this+// project documents nth as zero-based for every selector kind. Both remain correct; the+// boundary converts. Removing it makes the documented nth:0 match nothing.+const semanticNthOffset = 1++const nthPrefix = "nth:"++// SemanticNthBase splits a positional nth wrapper over a semantic form into the+// caller's ZERO-based index and the bare query underneath it, for a caller that has to+// say how many matches the unwrapped selector had. It answers false for every other+// selector, including first:/last:, which cannot be out of range while any match exists.+func (s Selector) SemanticNthBase() (index int, base string, ok bool) {+ if s.Kind != KindNth {+ return 0, "", false+ }+ index, raw, err := ParseNth(s.Value)+ if err != nil || !rawSelectorCanUseSemantic(raw) {+ return 0, "", false+ }+ return index, raw, true+}+func rawSelectorCanUseSemantic(raw string) bool {sel := Parse(raw)switch sel.Kind {diff --git a/internal/selector/selector_test.go b/internal/selector/selector_test.goindex 92a22c5a..1b808d63 100644--- a/internal/selector/selector_test.go+++ b/internal/selector/selector_test.go@@ -371,7 +371,7 @@ func TestSelector_SemanticQuery(t *testing.T) {{"testid:submit", "testid:submit", true},{"first:text:Submit", "", false},{"last:role:button Save", "last:role:button Save", true},- {"nth:2:label:Email", "nth:2:label:Email", true},+ {"nth:2:label:Email", "nth:3:label:Email", true},{"first:button", "", false},{"last:css:button", "", false},{"nth:2:button", "", false},@@ -402,14 +402,19 @@ func TestPositionalWrapperOverSemanticFormReachesTheMatcherWithItsIndex(t *testifor _, prefix := range semanticPrefixes {bare := prefix + "Save"- for _, wrapped := range []string{"first:" + bare, "last:" + bare, "nth:0:" + bare, "nth:2:" + bare} {- query, ok := Parse(wrapped).SemanticQuery()+ for _, tc := range []struct{ wrapped, want string }{+ {"first:" + bare, "first:" + bare},+ {"last:" + bare, "last:" + bare},+ {"nth:0:" + bare, "nth:1:" + bare},+ {"nth:2:" + bare, "nth:3:" + bare},+ } {+ query, ok := Parse(tc.wrapped).SemanticQuery()if !ok {- t.Errorf("%s no longer routes to the semantic matcher, so the wrapper it carries is never applied", wrapped)+ t.Errorf("%s no longer routes to the semantic matcher, so the wrapper it carries is never applied", tc.wrapped)continue}- if query != wrapped {- t.Errorf("SemanticQuery(%q) = %q: the wrapper must reach the matcher intact, since the semantic engine parses the index itself and stripping it here drops the position the caller asked for", wrapped, query)+ if query != tc.want {+ t.Errorf("SemanticQuery(%q) = %q, want %q: the wrapper must reach the matcher with its index, translated into the one-based nth the matcher publishes", tc.wrapped, query, tc.want)}}}@@ -666,3 +671,44 @@ func TestParseMixedCasePrefixesMatchTheirLowercaseForm(t *testing.T) {t.Errorf(`Parse("CSS:#id") = %+v, want css #id`, got)}}++// TestTheSemanticNthOffsetIsAnAdapterNotAnOffByOne states why the +1 exists, so it is+// not "simplified" away by someone who sees the arithmetic and assumes a bug. The two+// bases are both documented: this project publishes nth as zero-based for every+// selector kind, and the semantic matcher's README publishes nth as one-based, where+// nth:0 is not the first match. Deleting the offset makes the documented nth:0 select+// nothing on the whole semantic family.+func TestTheSemanticNthOffsetIsAnAdapterNotAnOffByOne(t *testing.T) {+ if semanticNthOffset != 1 {+ t.Fatalf("semanticNthOffset = %d, want 1: PinchTab's public nth is zero-based and the matcher's is one-based", semanticNthOffset)+ }++ query, ok := Parse("nth:0:role:button Save").SemanticQuery()+ if !ok {+ t.Fatal("nth over a semantic form must reach the matcher")+ }+ if query != "nth:1:role:button Save" {+ t.Errorf("the public first match nth:0 arrives as %q; the matcher treats nth:0 as out of range, so it must arrive as nth:1", query)+ }+}++func TestSemanticNthBaseReportsTheCallersOwnIndex(t *testing.T) {+ for _, tc := range []struct {+ input string+ wantIndex int+ wantBase string+ wantOK bool+ }{+ {"nth:0:role:button Save", 0, "role:button Save", true},+ {"nth:2:label:Email", 2, "label:Email", true},+ {"first:role:button", 0, "", false},+ {"last:role:button", 0, "", false},+ {"nth:2:css:button", 0, "", false},+ {"role:button", 0, "", false},+ } {+ index, base, ok := Parse(tc.input).SemanticNthBase()+ if ok != tc.wantOK || index != tc.wantIndex || base != tc.wantBase {+ t.Errorf("SemanticNthBase(%q) = (%d, %q, %v), want (%d, %q, %v)", tc.input, index, base, ok, tc.wantIndex, tc.wantBase, tc.wantOK)+ }+ }+}