High-performance browser automation bridge and multi-instance orchestrator with advanced stealth injection and real-time dashboard.
Scheduler
This page describes how PinchTab's optional scheduler works internally.
Scope
The scheduler is an opt-in dispatch layer for queued browser tasks. It sits in front of the existing tab action execution path and adds:
- queue admission limits
- per-agent fairness
- bounded concurrency
- cooperative cancellation
- result retention with TTL
It does not replace the direct action endpoints.
Runtime Placement
In server mode, the scheduler is created only when scheduler.enabled is true. It is registered directly on the main mux and exposes:
POST /tasksGET /tasksGET /tasks/{id}POST /tasks/{id}/cancelGET /scheduler/statsPOST /tasks/batch
High-Level Flow
client
-> POST /tasks
-> scheduler admission
-> in-memory task queue
-> worker dispatch
-> resolve tab -> instance port
-> POST /tabs/{tabId}/action on the owning instance
-> result store
Core Components
Scheduler
The scheduler owns:
- config
- task queue
- result store
- task lookup for live tasks
- cancellation map
- worker lifecycle
TaskQueue
The queue is:
- in-memory
- global-limit aware
- per-agent-limit aware
- priority ordered within an agent
- fairness aware across agents
Within an agent queue:
- lower
priorityvalue wins - equal priority falls back to FIFO by
CreatedAt
Across agents:
- the agent with the fewest in-flight tasks is chosen first
ResultStore
The result store keeps snapshots of tasks and evicts terminal tasks after the configured TTL.
Resolver
The scheduler takes the orchestrator directly as its InstanceResolver: ResolveTabInstance(tabId) maps a tabId to the owning instance port, which is how the scheduler knows where to forward execution.
The orchestrator also implements RequestAuthorizer. Before dispatch the executor calls AuthorizeTabRequest(tabId, req), which routes through the orchestrator's single hop-auth owner applyInstanceAuth: the request carries the instance's bearer token and, on a trusted child hop, the internal token. That is what lets the instance honor the X-PinchTab-* identity headers instead of stripping them at ingress, so the instance attributes the action to scheduler rather than client. Managed child instances run with activity recording off, so that attribution is persisted only by an instance that records activity. A resolver that does not implement RequestAuthorizer sends no credential, and an instance that requires one refuses the task with 401.
Activity Recording
The scheduler runs in the same process as the server's activity recorder, so it records each dispatched task there directly through an ActivitySink (internal/server passes the dashboard-feed recorder into New). One event per task carries source scheduler, the task's agentId and tabId, the action kind, and the outcome status and duration once the executor returns. The event always reaches the live dashboard stream (/api/events); it is written to the activity log, and so returned by /api/activity after a reload, only when observability.activity.events.scheduler is enabled, which is off by default like the other non-client sources. A failure response from the instance is recorded with the instance's own status (a 404 for a selector that matched nothing) and the executor's error text; a transport failure that never produced a response records as 502.
This sink is the single source of a scheduled action in the dashboard feed, recorded once and independent of any instance-level recording:
- Managed child instances run with activity recording off, so the child never records the forwarded action and there is no duplicate.
- An attached external bridge records the forwarded action into its own recorder and feed, not the server's, so it does not double-count in the dashboard stream either.
Dispatch Lifecycle
A task moves through these internal states:
queued -> assigned -> running -> done
-> failed
queued -> cancelled
queued -> rejected
Implementation details:
assignedis set just before worker execution startsrunningis set immediately before the proxied action requestdone,failed, andcancelledare terminal- rejected tasks are stored as terminal results even though they never enter active execution
Admission And Limits
Admission checks include:
- global queue size
- per-agent queue size
Execution checks include:
- max global in-flight count
- max per-agent in-flight count
These are enforced in the queue and worker dispatch path rather than by external infrastructure.
Cancellation Model
Each running task gets its own context.WithDeadline(...).
The scheduler stores the corresponding cancel function so that:
POST /tasks/{id}/cancelcan stop a running task- shutdown can cancel in-flight work
- deadlines propagate naturally to the proxied request
Deadline Handling
Two deadline paths exist:
- queued task expiry: a background reaper scans queued tasks every 5 seconds and marks expired ones as failed
- running task deadline: the per-task context deadline is enforced by the HTTP request to the executor
Queued expiry currently records:
deadline exceeded while queued
Execution Contract
The scheduler does not invent a separate execution protocol. It converts each task into the normal action request body:
{
"kind": "<action>",
"ref": "<ref>",
"...params": "..."
}
and forwards it to:
POST /tabs/{tabId}/action
That keeps the immediate path and scheduled path aligned.
Error Model
Common failure sources:
- admission rejection because the queue is full
- tab-to-instance resolution failure
- executor HTTP failure
- browser-side action failure
- deadline expiry
Scheduler task snapshots keep the final error string for these cases.
Result Retention
Terminal task snapshots are stored in memory and evicted after the configured TTL. This makes GET /tasks/{id} and GET /tasks useful for short-lived inspection, not for long-term audit storage.
Design Tradeoffs
The current scheduler favors:
- simple in-memory operation
- low dependency count
- reuse of the existing action executor
That means it does not currently provide:
- persistent queue storage
- durable recovery after process restart
- a separate task execution DSL
Phase 2 -- Observability Layer
Metrics
The scheduler maintains global and per-agent counters via the Metrics struct. Global counters use atomic.Uint64 for lock-free increments. Per-agent counters use a sharded mutex pattern (agentMetricEntry) to avoid global lock contention.
Tracked counters:
TasksSubmitted,TasksCompleted,TasksFailedTasksCancelled,TasksRejected,TasksExpiredDispatchTotaland cumulativeDispatchLatency(for average dispatch latency calculation)
Metrics.Snapshot() returns a MetricsSnapshot -- a point-in-time serializable copy that includes per-agent breakdowns.
Stats Endpoint
GET /scheduler/stats exposes three sections:
- queue -- current queued/inflight counts from
QueueStats() - metrics -- snapshot from
Metrics.Snapshot() - config -- current scheduler configuration values
Lifecycle Logging
Key scheduler events are logged via slog:
- task submission, dispatch, completion, failure, cancellation
- queue admission rejections
- config reload events
- webhook delivery outcomes
Webhook Delivery
When a task has a callbackUrl and reaches a terminal state, the scheduler sends a POST with the task snapshot as JSON. This runs asynchronously in a goroutine from finishTask().
Security constraints:
- only
httpandhttpsschemes are accepted (SSRF mitigation) - a dedicated
http.Clientwith a 10-second timeout prevents hanging connections - delivery failures are logged but do not affect task state
Custom headers are sent: X-PinchTab-Event: task.completed and X-PinchTab-Task-ID.
Phase 3 -- Hardening Layer
Batch Submission
POST /tasks/batch accepts an array of task definitions (up to scheduler.maxBatchSize, default 50) sharing a single agentId and optional callbackUrl. Each task is submitted individually through Submit(), so queue admission limits apply per-task.
The batch endpoint supports partial failure: if some tasks are rejected (queue full), the accepted tasks are still submitted and the response includes per-task status.
Config Hot-Reload
ReloadConfig(cfg Config) updates tuning knobs at runtime:
- queue limits via
queue.SetLimits(maxQueue, maxPerAgent) - inflight limits via
cfgMu-protected config fields - result TTL via
results.SetTTL(ttl)
Zero values in the reload config are ignored, preserving the existing setting.
ConfigWatcher is a background goroutine that periodically calls a user-supplied loadFn() (Config, error) and applies changes via ReloadConfig. It can be started and stopped cleanly.
SetLimits and SetTTL
The queue and result store expose safe runtime mutators:
TaskQueue.SetLimits(maxQueue, maxPerAgent)-- atomically updates admission thresholdsResultStore.SetTTL(ttl)-- updates the eviction window for terminal task snapshots