Skip to content

High-performance browser automation bridge and multi-instance orchestrator with advanced stealth injection and real-time dashboard.

9.0 KB
Raw Blame

Scheduler

This page describes how PinchTab's optional scheduler works internally.

Scope

The scheduler is an opt-in dispatch layer for queued browser tasks. It sits in front of the existing tab action execution path and adds:

  • queue admission limits
  • per-agent fairness
  • bounded concurrency
  • cooperative cancellation
  • result retention with TTL

It does not replace the direct action endpoints.

Runtime Placement

In server mode, the scheduler is created only when scheduler.enabled is true. It is registered directly on the main mux and exposes:

  • POST /tasks
  • GET /tasks
  • GET /tasks/{id}
  • POST /tasks/{id}/cancel
  • GET /scheduler/stats
  • POST /tasks/batch

High-Level Flow

client
  -> POST /tasks
  -> scheduler admission
  -> in-memory task queue
  -> worker dispatch
  -> resolve tab -> instance port
  -> POST /tabs/{tabId}/action on the owning instance
  -> result store

Core Components

Scheduler

The scheduler owns:

  • config
  • task queue
  • result store
  • task lookup for live tasks
  • cancellation map
  • worker lifecycle

TaskQueue

The queue is:

  • in-memory
  • global-limit aware
  • per-agent-limit aware
  • priority ordered within an agent
  • fairness aware across agents

Within an agent queue:

  • lower priority value wins
  • equal priority falls back to FIFO by CreatedAt

Across agents:

  • the agent with the fewest in-flight tasks is chosen first

ResultStore

The result store keeps snapshots of tasks and evicts terminal tasks after the configured TTL.

Resolver

The scheduler takes the orchestrator directly as its InstanceResolver: ResolveTabInstance(tabId) maps a tabId to the owning instance port, which is how the scheduler knows where to forward execution.

The orchestrator also implements RequestAuthorizer. Before dispatch the executor calls AuthorizeTabRequest(tabId, req), which routes through the orchestrator's single hop-auth owner applyInstanceAuth: the request carries the instance's bearer token and, on a trusted child hop, the internal token. That is what lets the instance honor the X-PinchTab-* identity headers instead of stripping them at ingress, so the instance attributes the action to scheduler rather than client. Managed child instances run with activity recording off, so that attribution is persisted only by an instance that records activity. A resolver that does not implement RequestAuthorizer sends no credential, and an instance that requires one refuses the task with 401.

Activity Recording

The scheduler runs in the same process as the server's activity recorder, so it records each dispatched task there directly through an ActivitySink (internal/server passes the dashboard-feed recorder into New). One event per task carries source scheduler, the task's agentId and tabId, the action kind, and the outcome status and duration once the executor returns. The event always reaches the live dashboard stream (/api/events); it is written to the activity log, and so returned by /api/activity after a reload, only when observability.activity.events.scheduler is enabled, which is off by default like the other non-client sources. A failure response from the instance is recorded with the instance's own status (a 404 for a selector that matched nothing) and the executor's error text; a transport failure that never produced a response records as 502.

This sink is the single source of a scheduled action in the dashboard feed, recorded once and independent of any instance-level recording:

  • Managed child instances run with activity recording off, so the child never records the forwarded action and there is no duplicate.
  • An attached external bridge records the forwarded action into its own recorder and feed, not the server's, so it does not double-count in the dashboard stream either.

Dispatch Lifecycle

A task moves through these internal states:

queued -> assigned -> running -> done
                           -> failed
queued -> cancelled
queued -> rejected

Implementation details:

  • assigned is set just before worker execution starts
  • running is set immediately before the proxied action request
  • done, failed, and cancelled are terminal
  • rejected tasks are stored as terminal results even though they never enter active execution

Admission And Limits

Admission checks include:

  • global queue size
  • per-agent queue size

Execution checks include:

  • max global in-flight count
  • max per-agent in-flight count

These are enforced in the queue and worker dispatch path rather than by external infrastructure.

Cancellation Model

Each running task gets its own context.WithDeadline(...).

The scheduler stores the corresponding cancel function so that:

  • POST /tasks/{id}/cancel can stop a running task
  • shutdown can cancel in-flight work
  • deadlines propagate naturally to the proxied request

Deadline Handling

Two deadline paths exist:

  • queued task expiry: a background reaper scans queued tasks every 5 seconds and marks expired ones as failed
  • running task deadline: the per-task context deadline is enforced by the HTTP request to the executor

Queued expiry currently records:

deadline exceeded while queued

Execution Contract

The scheduler does not invent a separate execution protocol. It converts each task into the normal action request body:

{
  "kind": "<action>",
  "ref": "<ref>",
  "...params": "..."
}

and forwards it to:

POST /tabs/{tabId}/action

That keeps the immediate path and scheduled path aligned.

Error Model

Common failure sources:

  • admission rejection because the queue is full
  • tab-to-instance resolution failure
  • executor HTTP failure
  • browser-side action failure
  • deadline expiry

Scheduler task snapshots keep the final error string for these cases.

Result Retention

Terminal task snapshots are stored in memory and evicted after the configured TTL. This makes GET /tasks/{id} and GET /tasks useful for short-lived inspection, not for long-term audit storage.

Design Tradeoffs

The current scheduler favors:

  • simple in-memory operation
  • low dependency count
  • reuse of the existing action executor

That means it does not currently provide:

  • persistent queue storage
  • durable recovery after process restart
  • a separate task execution DSL

Phase 2 -- Observability Layer

Metrics

The scheduler maintains global and per-agent counters via the Metrics struct. Global counters use atomic.Uint64 for lock-free increments. Per-agent counters use a sharded mutex pattern (agentMetricEntry) to avoid global lock contention.

Tracked counters:

  • TasksSubmitted, TasksCompleted, TasksFailed
  • TasksCancelled, TasksRejected, TasksExpired
  • DispatchTotal and cumulative DispatchLatency (for average dispatch latency calculation)

Metrics.Snapshot() returns a MetricsSnapshot -- a point-in-time serializable copy that includes per-agent breakdowns.

Stats Endpoint

GET /scheduler/stats exposes three sections:

  • queue -- current queued/inflight counts from QueueStats()
  • metrics -- snapshot from Metrics.Snapshot()
  • config -- current scheduler configuration values

Lifecycle Logging

Key scheduler events are logged via slog:

  • task submission, dispatch, completion, failure, cancellation
  • queue admission rejections
  • config reload events
  • webhook delivery outcomes

Webhook Delivery

When a task has a callbackUrl and reaches a terminal state, the scheduler sends a POST with the task snapshot as JSON. This runs asynchronously in a goroutine from finishTask().

Security constraints:

  • only http and https schemes are accepted (SSRF mitigation)
  • a dedicated http.Client with a 10-second timeout prevents hanging connections
  • delivery failures are logged but do not affect task state

Custom headers are sent: X-PinchTab-Event: task.completed and X-PinchTab-Task-ID.


Phase 3 -- Hardening Layer

Batch Submission

POST /tasks/batch accepts an array of task definitions (up to scheduler.maxBatchSize, default 50) sharing a single agentId and optional callbackUrl. Each task is submitted individually through Submit(), so queue admission limits apply per-task.

The batch endpoint supports partial failure: if some tasks are rejected (queue full), the accepted tasks are still submitted and the response includes per-task status.

Config Hot-Reload

ReloadConfig(cfg Config) updates tuning knobs at runtime:

  • queue limits via queue.SetLimits(maxQueue, maxPerAgent)
  • inflight limits via cfgMu-protected config fields
  • result TTL via results.SetTTL(ttl)

Zero values in the reload config are ignored, preserving the existing setting.

ConfigWatcher is a background goroutine that periodically calls a user-supplied loadFn() (Config, error) and applies changes via ReloadConfig. It can be started and stopped cleanly.

SetLimits and SetTTL

The queue and result store expose safe runtime mutators:

  • TaskQueue.SetLimits(maxQueue, maxPerAgent) -- atomically updates admission thresholds
  • ResultStore.SetTTL(ttl) -- updates the eviction window for terminal task snapshots