High-performance browser automation bridge and multi-instance orchestrator with advanced stealth injection and real-time dashboard.
test: add replicable token efficiency test templates for agents
Marioe008ddf70531
diff --git a/tests/AGENT-TEST-TEMPLATE.md b/tests/AGENT-TEST-TEMPLATE.mdnew file mode 100644index 00000000..34f60a87--- /dev/null+++ b/tests/AGENT-TEST-TEMPLATE.md@@ -0,0 +1,185 @@+# Agent Test Replication Template++**For agents without Pinchtab to replicate token efficiency testing.**++## Quick Start++Copy these steps into your session and run sequentially.++---++## Test: Extract News Headlines + Measure Tokens++### Prerequisites+```bash+# 1. Verify pinchtab is running+curl http://localhost:9867/health+# Expected: {"status": "ok"}++# 2. Have jq installed (for JSON parsing)+which jq+```++### Test Steps (Copy + Paste)++**Step 1: Navigate**+```bash+BASE_URL="http://localhost:9867"+curl -s -X POST $BASE_URL/navigate \+ -H "Content-Type: application/json" \+ -d '{"url": "https://www.corriere.it"}' | jq '.tabId'+# Save the tabId for next steps+```++**Step 2: Wait for page load**+```bash+sleep 3+```++**Step 3: Test Method A — Full snapshot (baseline)**+```bash+echo "=== FULL SNAPSHOT ==="+curl -s $BASE_URL/snapshot > /tmp/snap-full.json+SIZE=$(wc -c < /tmp/snap-full.json)+TOKENS=$((SIZE / 4))+echo "Size: $SIZE bytes"+echo "Est. tokens: ~$TOKENS"+```++**Step 4: Test Method B — Compact snapshot (optimized)**+```bash+echo "=== COMPACT + FILTER SNAPSHOT ==="+curl -s "$BASE_URL/snapshot?format=compact&filter=interactive" > /tmp/snap-compact.json+SIZE=$(wc -c < /tmp/snap-compact.json)+TOKENS=$((SIZE / 4))+echo "Size: $SIZE bytes"+echo "Est. tokens: ~$TOKENS"+```++**Step 5: Test Method C — Text only (cheapest)**+```bash+echo "=== TEXT ONLY ==="+curl -s $BASE_URL/text > /tmp/text.json+SIZE=$(wc -c < /tmp/text.json)+TOKENS=$((SIZE / 4))+echo "Size: $SIZE bytes"+echo "Est. tokens: ~$TOKENS"+```++**Step 6: Compare results**+```bash+echo ""+echo "=== COMPARISON ==="+FULL=$(wc -c < /tmp/snap-full.json)+COMPACT=$(wc -c < /tmp/snap-compact.json)+TEXT=$(wc -c < /tmp/text.json)++SAVINGS=$((100 - (COMPACT * 100 / FULL)))+echo "Full snapshot: ~$((FULL / 4)) tokens (baseline)"+echo "Compact snapshot: ~$((COMPACT / 4)) tokens (savings: $SAVINGS%)"+echo "Text only: ~$((TEXT / 4)) tokens (savings: $((100 - (TEXT * 100 / FULL)))%)"+```++---++## Sample Output++```+=== FULL SNAPSHOT ===+Size: 4250 bytes+Est. tokens: ~1062++=== COMPACT + FILTER SNAPSHOT ===+Size: 1850 bytes+Est. tokens: ~462++=== TEXT ONLY ===+Size: 925 bytes+Est. tokens: ~231++=== COMPARISON ===+Full snapshot: ~1062 tokens (baseline)+Compact snapshot: ~462 tokens (savings: 56%)+Text only: ~231 tokens (savings: 78%)+```++---++## Instructions for Another Agent++**To replicate this test without any prior knowledge:**++1. **Get the test file:**+ ```bash+ curl -s https://raw.githubusercontent.com/pinchtab/pinchtab/main/tests/AGENT-TEST-TEMPLATE.md+ ```++2. **Run each "Test Steps" command one by one** (they're independent)++3. **Record the token counts** in a table like above++4. **Compare:**+ - Compact should be ~50-70% smaller than full+ - Text should be ~75-85% smaller than full++5. **Report result:**+ ```+ ✅ Token efficiency verified+ - Full: 1062 tokens+ - Compact: 462 tokens (56% savings)+ - Text: 231 tokens (78% savings)+ - Recommended: Use /text for reading, /snapshot?format=compact&filter=interactive for interaction+ ```++---++## Troubleshooting++| Issue | Solution |+|-------|----------|+| `curl: (7) Failed to connect` | Pinchtab not running. Run `pinchtab &` first |+| `jq not found` | Install: `brew install jq` (macOS) or `apt-get install jq` (Linux) |+| `404 not found` | Wrong endpoint. Check: `curl http://localhost:9867/health` |+| Token counts seem off | Use `echo $TEXT \| wc -c` for exact byte count; divide by 4 for rough token estimate |++---++## For Exact Token Counting++Use Python + tiktoken (OpenAI's official tokenizer):++```python+import tiktoken+import json++with open('/tmp/snap-full.json') as f:+ data = json.load(f)+ text = json.dumps(data)++enc = tiktoken.encoding_for_model("gpt-4")+tokens = enc.encode(text)+print(f"Exact tokens: {len(tokens)}")+```++Or online: https://platform.openai.com/tokenizer++---++## Next Steps++After confirming token efficiency:++1. **Extract specific headlines:**+ ```bash+ curl -s "$BASE_URL/snapshot?format=compact&filter=interactive" | \+ jq '.nodes[] | select(.role == "heading") | {ref, title: .name}' | \+ head -10+ ```++2. **Use diff mode for repeated snapshots:**+ ```bash+ curl -s "$BASE_URL/snapshot?diff=true&format=compact" | wc -c+ # Much smaller (only changes shown)+ ```++3. **Test on other news sites:** Replace corriere.it with any site and rerundiff --git a/tests/token-efficiency-corriere.md b/tests/token-efficiency-corriere.mdnew file mode 100644index 00000000..ddee8dd5--- /dev/null+++ b/tests/token-efficiency-corriere.md@@ -0,0 +1,179 @@+# Token Efficiency Test: Corriere.it News Extraction++**Goal:** Extract news titles from corriere.it homepage and measure token cost efficiency.++**Test Date:** [Run date]+**Tester:** [Agent name]+**Environment:** Pinchtab running on localhost:9867++---++## Prerequisites++1. Pinchtab server running:+ ```bash+ pinchtab &+ ```++2. `jq` installed (for JSON parsing + token counting):+ ```bash+ which jq || echo "Install jq: brew install jq"+ ```++3. Token counter script (estimate OpenAI tokens):+ ```bash+ # Rough estimation: ~1 token per 4 chars or 1 token per ~0.75 words+ # For exact counts, use tiktoken (Python) or check OpenAI tokenizer+ ```++---++## Test Steps++### Step 1: Navigate to Corriere.it+```bash+curl -X POST http://localhost:9867/navigate \+ -H "Content-Type: application/json" \+ -d '{"url": "https://www.corriere.it"}'+```++Expected: `{"tabId": "...", "status": "ok"}`++### Step 2: Get News Titles — Method A (Full Snapshot)+```bash+curl http://localhost:9867/snapshot | jq '.nodes | map(select(.role == "heading")) | .[0:5]' > /tmp/corriere-snapshot-full.json+```++**Capture output:**+```bash+curl http://localhost:9867/snapshot > /tmp/corriere-snapshot-full.json+echo "Full snapshot size:"+wc -c < /tmp/corriere-snapshot-full.json+echo "Estimated tokens:"+stat -f "%.0f" /tmp/corriere-snapshot-full.json | awk '{printf("~%.0f tokens (4 chars per token)\n", $1/4)}'+```++### Step 3: Get News Titles — Method B (Compact Snapshot)+```bash+curl "http://localhost:9867/snapshot?format=compact&filter=interactive" > /tmp/corriere-snapshot-compact.json+echo "Compact snapshot size:"+wc -c < /tmp/corriere-snapshot-compact.json+echo "Estimated tokens:"+stat -f "%.0f" /tmp/corriere-snapshot-compact.json | awk '{printf("~%.0f tokens\n", $1/4)}'+```++### Step 4: Get News Titles — Method C (Text Only)+```bash+curl http://localhost:9867/text | jq '.text' > /tmp/corriere-text.txt+echo "Text extraction size:"+wc -c < /tmp/corriere-text.txt+echo "Estimated tokens:"+stat -f "%.0f" /tmp/corriere-text.txt | awk '{printf("~%.0f tokens\n", $1/4)}'+```++### Step 5: Get Specific Headlines with CSS Selector+```bash+curl "http://localhost:9867/snapshot?selector=.article-headline" | jq '.nodes[] | select(.name | length > 10) | {ref: .ref, title: .name}' > /tmp/corriere-headlines.json+wc -c < /tmp/corriere-headlines.json+```++### Step 6: Extract via CLI+```bash+pinchtab nav https://www.corriere.it+sleep 3 # Wait for page load+pinchtab snap --format=compact --filter=interactive | tee /tmp/cli-snapshot.json | wc -c+```++---++## Analysis Template++| Method | Bytes | Est. Tokens | Time | Notes |+|--------|-------|-------------|------|-------|+| Full snapshot | ? | ? | ? | Baseline (all nodes) |+| Compact + filter | ? | ? | ? | ~60% fewer tokens |+| Text only | ? | ? | ? | Cheapest option (~800 tokens) |+| CSS selector | ? | ? | ? | Most efficient (targeted) |+| CLI snapshot | ? | ? | ? | Native pinchtab tool |++---++## Expected Results++Based on typical news site structure:++| Method | Token Cost | Token Savings |+|--------|------------|---------------|+| Full `/snapshot` | ~3,500–5,000 | Baseline |+| `/snapshot?format=compact&filter=interactive` | ~1,500–2,000 | 60–70% savings |+| `/text` (readability) | ~800–1,200 | 80–90% savings |+| `/text?mode=raw` | ~1,000–1,500 | ~70% savings |++---++## Replication for Other Agents++**To run this test as another agent:**++1. **Clone or copy this file** to your workspace+2. **Ensure Pinchtab is running:** `pinchtab &` (or ask host to start it)+3. **Run each step sequentially:**+ ```bash+ bash -x token-efficiency-corriere.md # Won't work, but shows intent+ ```+ Or manually run each curl command++4. **Compare results:**+ - Save all outputs to `/tmp/corriere-*`+ - Compare file sizes+ - Calculate token savings++5. **Report format:**+ ```+ **Token Efficiency Test: corriere.it**+ - Full snapshot: 4,250 bytes (~1,062 tokens)+ - Compact snapshot: 1,850 bytes (~462 tokens)+ - Text only: 925 bytes (~231 tokens)+ - Savings with compact: 56% fewer tokens+ - Recommended method: /text for reading, /snapshot?format=compact for interaction+ ```++---++## Docker / Remote Setup++If pinchtab isn't local:++```bash+# Replace localhost:9867 with actual host+PINCHTAB_URL="http://pinchtab-server:9867"++curl -X POST $PINCHTAB_URL/navigate \+ -H "Content-Type: application/json" \+ -d '{"url": "https://www.corriere.it"}'+```++---++## Notes++- **Token estimation:** `bytes / 4` is rough. Use tiktoken (Python) for exact counts:+ ```python+ import tiktoken+ enc = tiktoken.encoding_for_model("gpt-4")+ tokens = enc.encode(text)+ print(len(tokens))+ ```++- **Page load time:** Always wait 3+ seconds after navigate before snapshot+- **Caching:** Snapshot refs are stable within a session; re-use them for actions+- **Stealth:** Test with/without `?displayHeaderFooter=false` for cleaner output++---++## Success Criteria++✅ Agent can extract headlines from corriere.it+✅ Agent can measure token cost difference between methods+✅ Agent demonstrates 50%+ token savings with optimized approach+✅ Results reproducible across multiple runs