Self-improving AI Employees for everyone
Python ai-agentsai-employeeai-employeesassistantautonomous-agentsco-employeescompanygrok-botinstinctloop-agentsmuseown-your-datapersonalself-hosted
feat(scraper): add the tikhubAction node over the official tikhub SDK
trohitg6436bd6d7c7e
diff --git a/CLAUDE.md b/CLAUDE.mdindex 61c55b89..54f20344 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -34,6 +34,7 @@ This is a React Flow-based workflow automation platform implementing n8n-inspire| **[GitHub Service](./docs-internal/github_service.md)** | GitHub CLI integration — `githubAction` (repo_clone / pr_create / pr_list / pr_merge / issue_create / issue_list + `custom` passthrough; AI tool `github`). **Strict Stripe pattern: the gh CLI owns its own auth** — no token stored/injected by OpenCompany, no auth pre-flight (gh's own error surfaces), marker OAuth row for the modal badge, fieldless catalogue entry. Login = spawned `gh auth login --hostname github.com --git-protocol https --web` — source-verified: headless takes the `isInteractive=false` branch (no Press-Enter), prints one-time code + `github.com/login/device` URL on stderr; the modal shows the code via the generic `verificationCode` plumbing (`useCredentialPanel` → `OAuthConnect`). `login_env()` strips `GH_TOKEN`/`GITHUB_TOKEN` (login aborts otherwise). Success gate = `gh auth status` exit 0, then best-effort **`gh auth setup-git`** (official bridge — the future git node needs zero auth code). gh binary: **project-local, pooch-driven** (temporal `_install.py` idiom) — pinned release under `package_dir("gh")`, system gh never consulted (auth state still shared: gh config paths are user-level). Palette group `vcs`. || **[Cloudflare Service](./docs-internal/cloudflare_service.md)** | Cloudflare CLI integration — `cloudflareAction` (whoami / zones_list / dns_records CRUD / `graphql_query` / `custom` passthrough; AI tool `cloudflare`) wrapping the official `cf` CLI (npm `[email protected]`, technical preview — "the next version of Wrangler"). **Two cf-specific variants of the CLI-managed-auth pattern**: (1) the CLI opens the browser ITSELF (loopback PKCE callback on fixed port 8877) so the login handler proxies nothing — single-flight guard (concurrent logins collide on the fixed port) + never-kill-the-shim invariant (killing the npm `.cmd` wrapper orphans the node child holding 8877); success gate = `cf auth whoami` JSON (`authenticated: true` — cf exits 0 in BOTH auth states). (2) **Fixed 86-scope OAuth grant with no `--scopes` flag** — only `dns_analytics:read` for analytics; no RUM/zone-analytics OAuth scope exists anywhere, so the optional `cloudflare_api_token` field (injected as `CLOUDFLARE_API_TOKEN`, cf's documented first-priority credential) is the only path to Web Analytics/RUM + the GraphQL Analytics API. `graphql_query` op POSTs the official `client/v4/graphql` endpoint directly (outside the OpenAPI schema cf/SDKs are generated from) — the replacement for the sunsetted Zone Analytics REST API; needs `Account > Account Analytics > Read`. Pinned npm install into the shared packages tree, system cf never consulted (preview argv surface drifts across versions). Palette group `deployment`. || **[Google Cloud Service](./docs-internal/gcloud_service.md)** | Google Cloud CLI integration — `gcloudAction` (auth/config/projects + set_project / Compute Engine instances list-start-stop-describe / Cloud Run deploy-list-describe / Cloud Storage ls-cp-rm / `custom` passthrough; AI tool `gcloud`) wrapping the official gcloud CLI (pinned versioned archive via pooch into `package_dir("gcloud")` — NOT npm; the Windows bundled-python zip keeps the legacy `google-cloud-sdk-` filename prefix, the `-cli-` spelling 404s, locked by test). Cloudflare login variant (CLI opens the browser itself, random loopback port — no fixed-port hazard) + the gh marker-token pattern; success gate = `gcloud auth list --filter=status:ACTIVE --format=json` non-empty (exit codes never trusted). **Config isolation divergence from gh**: every invocation pins `CLOUDSDK_CONFIG=<DATA_DIR>/gcloud/`, so terminal logins against the operator's global gcloud are deliberately NOT visible to the node. ADC note: node login mints gcloud USER creds only (CLI commands never read ADC); client-library code gets ADC via the `custom` op `auth application-default login` (lands isolated in the pinned dir). Cold install ~100 MB → login answers `pending: True` within the 22 s WS budget. Provider id `gcloud` is orthogonal to the Workspace `google` OAuth2 provider. Palette group `deployment`. |+| **[TikHub Service](./docs-internal/tikhub_service.md)** | TikHub social-scraping integration — `tikhubAction` (`call` / `list_endpoints` / `fetch_url` / `account`; AI tool `tikhub_action`; palette group `scraper`, visible in normal mode) over the **official `tikhub` SDK** (`tikhub>=2.1.2,<3`, auto-generated from TikHub's `openapi.json`; ~1,100 methods across TikTok / Douyin / Instagram / YouTube / Twitter / Xiaohongshu / Bilibili / Kuaishou / Weibo / Reddit / Threads / LinkedIn / Zhihu). **The node is the "flattened CLI"**: one `call` op addressed by the SDK's own `resource.method` id (`douyin_web.fetch_one_video`, also `resource/method` or the `/api/v1/...` path) with **reflective `getattr` dispatch** and `inspect.signature(...).bind` pre-flight so unknown kwargs fail before any paid request; **nothing is vendored** — `endpoint_index()` introspects the installed SDK at runtime (resources = `AsyncResource` instance attrs, params = keyword-only signature, route from the docstring's ``GET /api/v1/...`` literal), cached per process, and feeds both `list_endpoints` and the `platform`-dependent dynamic `endpoint` dropdown (`tikhubEndpoints` loader; `all` returns no options, the LLM path stays free text). `tikhub` is imported lazily inside `_sdk.py` only (plugin imports with `sys.modules["tikhub"] = None`). `TikHubCredential` = bearer `ApiKeyCredential` with a declarative httpx probe against `/api/v1/tikhub/user/get_user_info` (never the unauthenticated health route); runtime key via `TikHubCredential.resolve` so a missing key yields the structured `PermissionDeniedError` envelope. SDK `TikHubError` subclasses map to `NodeUserError` (401 key rejected -> Credentials; 403 scope-or-balance -> `account`; 429 + `retry_after`; 400/422 with TikHub `detail`; 404; 5xx upstream after the SDK's own retries) and HTTP-200-with-in-body-`code >= 400` is raised too. **Flat `$0.001`/request tracking** (`pricing.json: api.tikhub` + `operation_map.tikhub`, `track_tikhub_usage` -> `api_usage_metrics`, only after a 2xx, never for `list_endpoints`). SDK drift is guarded by two contract tests (every `PLATFORMS` value matches a live resource; every backticked id in the skill resolves). || **[Discord Service](./docs-internal/discord_service.md)** | Discord bot integration — `discordSend` / `discordAction` (AI tools) + `discordReceive` / `discordInteraction` triggers. **discord.py owns the gateway, REST is plugin-owned** (nodes must work with no gateway, `discordAction` needs arbitrary routes, and the invalid-request guard is process-wide across accounts). **Multi-account rides the existing `session_id` credential column** — `_accounts.py` is the only file that knows `account_id -> "discord:<app_id>"`, `"default"` is the row the modal already writes, and `credential_customer_id` is deliberately NOT used (it is per-execution-context tenancy, so two nodes in one workflow could never target different bots). **`AccountScopedNode` exists because `server_controlled_fields` is enforced only in `execute_as_tool`'s ToolNode branch** — a dual-purpose ActionNode takes an earlier `{**node_params, **tool_args}` merge with model args winning, so locked fields are stripped before it. Attachments are metadata on the trigger and downloaded by `discordAction` (deployed triggers never run their node body). Interaction tokens never leave the server — the trigger emits an opaque ref. The interactions endpoint is a plugin router, not a `WebhookSource`: Discord's validation probe requires **401** (the generic intake raises 400) and the 3-second ACK deadline precludes awaiting `emit`. Rate limiting is reactive; the invalid-request guard is **process-global** because Cloudflare bans the source IP, not the token. || **[Sarvam AI Service](./docs-internal/sarvam_service.md)** | Sarvam AI (Indic-first). **`nodes/sarvam/` no longer exists** — every capability it served is now a *provider* inside an abstracted node: speech in `nodes/speech/`, and translate / transliterate / detect-language in `nodes/translate/`. What remains Sarvam-specific is `sarvamChatModel` (an **OpenAI-compat provider**: one `_COMPAT_PROVIDERS` entry + a JSON block, no provider subclass) under `nodes/model/`. **One stored key spans two auth styles** — the chat route takes `Authorization: Bearer` (what the openai SDK sends) while every other endpoint takes only `api-subscription-key`, so `SarvamCredential` declares the native header and every surface shares it. Introduces the generic **`supports_model_listing: false`** flag: Sarvam ships no `/v1/models` route, and unhandled that 404 becomes a `NodeUserError` that `AIService.fetch_models` re-raises *before* its curated fallback, breaking key validation and the model dropdown — so `OpenAIProvider.fetch_models` serves the curated list and probes with a one-token completion instead (defaults `true`; the other 12 providers are untouched, locked by test). Also added generic **`Connection.request(files=)`** multipart support. || **[Speech Provider RFC](./docs-internal/speech_provider_rfc.md)** | Provider-abstracted speech — two nodes (`textToSpeech`, `speechToText`) with a `provider` dropdown, backed by **two registries, one per direction** (registry membership IS the capability: a synthesis-only vendor simply never appears in the transcription enum). Everything vendor-specific lives in the plugin folder [`server/nodes/speech/`](./server/nodes/speech/) — protocol, registries, dispatch, per-vendor modules; only the generic `services/provider_registry.py` and the vendor-neutral `services/media/` are shared. v1 providers: OpenAI (both directions), ElevenLabs (TTS), Deepgram + Groq (STT), Sarvam (both). Capabilities are JSON (`server/config/speech_defaults.json`) with per-model overrides, so no shared code branches on a provider name. **Audio never travels as bytes** — nodes return an `AudioRef` (a ~400 B reference) because a 12 MB base64 result blows Temporal's 2 MiB blob limit and retries 3× re-billing the provider each time. Backing routes: `GET/POST /api/workspace/{workflow_id}/…` in [`routers/workspace.py`](./server/routers/workspace.py). |diff --git a/README.md b/README.mdindex 27206e4a..549d2cee 100644--- a/README.md+++ b/README.md@@ -74,7 +74,7 @@ Three example workflows load automatically on first launch. Open them on the can- **Messaging bots.** Send and receive on **WhatsApp** (personal — groups, contacts, newsletter channels), **WhatsApp Business** (official Meta Cloud API: templates, media, interactive messages, signed webhooks), **Telegram** (bots with owner detection), **Discord** (bot with gateway message triggers, slash commands, and OAuth2), and **Twitter/X** (post, reply, search). A unified social node normalizes incoming messages into one format so the same workflow handles them all.- **Voice and language.** Provider-abstracted text-to-speech and speech-to-text (OpenAI, ElevenLabs, Deepgram, Groq, Sarvam) with reference-based audio that flows between nodes, plus translate / transliterate / detect-language nodes (DeepL, Sarvam, or any connected LLM).- **Phone control from a workflow.** Pair your Android phone via QR code and control it from any agent: battery and network status, app launching, WiFi / Bluetooth / airplane toggles, camera, sensors, media playback — 16 device services.-- **Web automation and research.** An interactive browser with accessibility-tree navigation (click, type, screenshot); an alpha harness that drives your *real* Chrome over CDP; scraping with Crawlee and Apify actors (Instagram, TikTok, LinkedIn, Facebook, YouTube, Google Search); search via DuckDuckGo (free), Brave, Serper, and Perplexity; residential proxies with geo-targeting and rotation.+- **Web automation and research.** An interactive browser with accessibility-tree navigation (click, type, screenshot); an alpha harness that drives your *real* Chrome over CDP; scraping with Crawlee, Apify actors (Instagram, TikTok, LinkedIn, Facebook, YouTube, Google Search), and TikHub's ~1000-endpoint social API (TikTok, Douyin, Instagram, YouTube, Twitter/X, Xiaohongshu, Bilibili and more); search via DuckDuckGo (free), Brave, Serper, and Perplexity; residential proxies with geo-targeting and rotation.- **Code, deploys, and pull requests.** Run Python / JavaScript / TypeScript in per-workflow sandboxed workspaces, keep dev servers alive with the Process Manager node (output streams to the Terminal tab), open and merge PRs with the **GitHub** node, ship with the **Vercel** node, manage DNS and analytics with **Cloudflare**, and drive Compute Engine / Cloud Run / Storage with the **Google Cloud** node — all four authenticate through their own CLIs, no token pasting required. A **Gallery** node gives every workflow a visual file explorer with previews and drag-to-parameter assignment.- **Local data and vision.** Give agents typed, bounded access to workspace files and operator-approved external folders through the dataSource tool, and vision for every host model via the visionAnalyze delegate — images travel as native content blocks, never pasted base64.- **Payments.** **Stripe** action node (charges, subscriptions) plus a signed-webhook receiver for reacting to payment events in real time.@@ -170,7 +170,7 @@ Want to add a node, LLM provider, skill, or integration? One Python file = one nIssues and pull requests are welcome. [CONTRIBUTING.md](CONTRIBUTING.md) has the fork/branch/PR workflow, the repository map, and recipes for adding a node, LLM provider, or skill.-One note on scope: connector and provider lists are kept deliberately narrow. The Apify node runs any actor through its `custom` option, and agents reach any OpenAI-compatible endpoint through the existing provider path — so a new first-class preset needs a reason beyond "my service could be in the dropdown too."+One note on scope: connector and provider lists are kept deliberately narrow. The Apify node runs any actor through its `custom` option, the TikHub node calls any of its endpoints through `call`, and agents reach any OpenAI-compatible endpoint through the existing provider path — so a new first-class preset needs a reason beyond "my service could be in the dropdown too."## Communitydiff --git a/docs-internal/node-logic-flows/README.md b/docs-internal/node-logic-flows/README.mdindex 936728a2..460abaf3 100644--- a/docs-internal/node-logic-flows/README.md+++ b/docs-internal/node-logic-flows/README.md@@ -224,6 +224,7 @@ match its doc here, and each doc must still describe what the code does.- [Browser (`browser`)](./web_automation/browser.md)- [Browser Harness (`browserHarness`)](./web_automation/browserHarness.md)- [Crawlee Scraper (`crawleeScraper`)](./web_automation/crawleeScraper.md)+- [TikHub (`tikhubAction`)](./web_automation/tikhubAction.md)diff --git a/docs-internal/node-logic-flows/web_automation/tikhubAction.md b/docs-internal/node-logic-flows/web_automation/tikhubAction.mdnew file mode 100644index 00000000..291990a6--- /dev/null+++ b/docs-internal/node-logic-flows/web_automation/tikhubAction.md@@ -0,0 +1,235 @@+# TikHub (`tikhubAction`)++| Field | Value |+|------|-------|+| **Category** | web_automation / scraper / tool (dual-purpose) |+| **Backend handler** | [`server/nodes/scraper/tikhub_action/__init__.py::TikHubActionNode`](../../../server/nodes/scraper/tikhub_action/__init__.py) - dispatch via `BaseNode.execute()` -> `@Operation("call" / "list_endpoints" / "fetch_url" / "account")`; SDK glue in [`_sdk.py`](../../../server/nodes/scraper/tikhub_action/_sdk.py) |+| **Tests** | [`server/tests/nodes/test_tikhub.py`](../../../server/tests/nodes/test_tikhub.py) |+| **Skill (if any)** | [`server/skills/web_agent/tikhub-skill/SKILL.md`](../../../server/skills/web_agent/tikhub-skill/SKILL.md) |+| **Dual-purpose tool** | yes - tool name `tikhub_action` (`usable_as_tool = True`; both canvas handles explicitly kept visible) |++## Purpose++Call any endpoint of the TikHub social-scraping API (TikTok, Douyin,+Instagram, YouTube, Twitter/X, Xiaohongshu, Bilibili, Kuaishou, Weibo,+Reddit, Threads, LinkedIn, Zhihu, ...) through the official `tikhub`+Python SDK. The node is a "flattened CLI": instead of one operation per+endpoint it exposes `call`, addressed by the SDK's own `resource.method`+id (e.g. `douyin_web.fetch_one_video`), plus `list_endpoints` to discover+ids and their typed parameters without spending a request, `fetch_url`+for the share-URL hybrid parser, and `account` for balance / daily usage.+Used both as a workflow node and as an AI-agent tool; the agent runs the+discover-then-call loop documented in the skill.++## Inputs (handles)++| Handle | Connection type | Required | Purpose |+|--------|-----------------|----------|---------|+| `input-main` | main | no | Upstream data; not consumed directly - all inputs come from `parameters` (or the LLM's tool args when used as a tool) |++## Parameters++(Pydantic `TikHubActionParams`, `model_config = {"extra": "ignore"}`; a+`@model_validator(mode="before")` runs `coerce_blank_params(...,+object_fields=("params",))` so the panel's `""` and LLM-stringified JSON+both become a dict.)++| Name | Type | Default | Required | displayOptions.show | Description |+|------|------|---------|----------|---------------------|-------------|+| `operation` | options | `call` | no | - | `call` / `list_endpoints` / `fetch_url` / `account` |+| `platform` | options (`PLATFORMS` Literal) | `all` | no | `operation` in [`call`, `list_endpoints`] | Drives the endpoint dropdown and the listing filter (`all`, `tiktok`, `douyin`, `instagram`, `youtube`, `twitter`, `xiaohongshu`, `bilibili`, `kuaishou`, `weibo`, `reddit`, `threads`, `linkedin`, `zhihu`, `toutiao`, `xigua`, `wechat`, `lemon8`, `pipixia`, `hybrid`, `tikhub`) |+| `endpoint` | string (dynamic options: `loadOptionsMethod: "tikhubEndpoints"`, depends on `platform`) | `""` | yes for `call` | `operation` = `call` | `resource.method` id, `resource/method`, or a `/api/v1/...` path. Plain `str` (not a Literal) so the LLM can pass any id |+| `params` | object (JSON editor) | `{}` | no | `operation` = `call` | SDK keyword arguments for the endpoint; POST batch endpoints take `{"body": [...]}` |+| `search` | string | `""` | no | `operation` = `list_endpoints` | Substring filter over endpoint id / summary / path |+| `limit` | int (`ge=1, le=2000`) | `100` | no | `operation` = `list_endpoints` | Max endpoints returned |+| `url` | string | `""` | yes for `fetch_url` | `operation` = `fetch_url` | Share URL (TikTok / Douyin, incl. short links) |+| `minimal` | boolean | `false` | no | `operation` = `fetch_url` | Forwarded as `hybrid_parsing.video_data(minimal=...)` |++Field names `model`, `api_key`, `parameters` and `action` are deliberately+avoided (name-based magic in `ParameterRenderer.tsx`).++## Outputs (handles)++| Handle | Shape | Description |+|--------|-------|-------------|+| `output-main` | object | `_shape(operation, **fields)` payload (see below); the same payload is returned to the LLM when wired to `input-tools` |++### Output payload++`TikHubActionOutput` (`extra="allow"`, every field Optional; `_shape`+drops `None` values but keeps `{}` / `[]` - an empty answer from TikHub is+real data):++```ts+// operation = "call" | "fetch_url"+{+ operation: 'call' | 'fetch_url';+ endpoint: string; // resolved resource.method id+ path: string | null; // REST path from the SDK docstring (null if not introspectable)+ url?: string; // fetch_url only+ data: unknown; // TikHub's `data` field, platform-specific shape+ code: number; // TikHub envelope code (200 on success)+ router: string; // TikHub envelope router+ cost_usd: number; // what track_tikhub_usage recorded (0.001 flat)+}++// operation = "list_endpoints"+{+ operation: 'list_endpoints';+ platform: string;+ count: number; // after search + limit+ total: number; // endpoints on that platform+ endpoints: Array<{+ endpoint: string; resource: string; method: string;+ http_method: string | null; path: string | null; summary: string;+ params: Array<{ name: string; type: string; required: boolean }>;+ }>;+}++// operation = "account"+{+ operation: 'account';+ account: unknown; // tikhub_user.get_user_info() data+ usage: unknown; // tikhub_user.get_user_daily_usage() data+ cost_usd: number; // 0.0+}+```++Wrapped by `BaseNode._serialize_result` in the standard envelope+`{ success: true, result: <payload>, ... }`. Runtime output schema+registered via `register_output_schema("tikhubAction", ...)`.++## Logic Flow++```mermaid+flowchart TD+ A[BaseNode.execute -> TikHubActionNode.<operation>] --> B{operation}++ B -- list_endpoints --> L1[endpoint_index - cached SDK introspection, no network]+ L1 --> L2[filter platform == info.platform<br/>substring search over id / summary / path<br/>slice to limit]+ L2 --> L3[_shape list_endpoints: platform, count, total, endpoints]++ B -- call --> C1{endpoint given?}+ C1 -- no --> Ereq[NodeUserError: endpoint required]+ C1 -- yes --> C2[resolve_endpoint - id / slash / REST path via _BY_ENDPOINT + _BY_PATH]+ C2 -- unknown --> Eunk[NodeUserError: Unknown TikHub endpoint + difflib suggestions + hint list_endpoints]+ C2 -- ok --> C3[make_client: TikHubCredential.resolve -> AsyncTikHub api_key, timeout, max_retries]+ C3 -- key missing --> Eperm[PermissionError annotated provider=tikhub<br/>-> PermissionDeniedError envelope + credential.api_key.runtime_failed]+ C3 -- tikhub ImportError --> Eimp[NodeUserError: package not installed, run uv sync]+ C3 -- ok --> C4[fn = getattr getattr client, resource , method]+ C4 --> C5[bind_params: inspect.signature fn .bind **params]+ C5 -- TypeError --> Ebind[NodeUserError: endpoint rejected params + accepted list]+ C5 -- ok --> C6[await fn **params - SDK retries 429 / 5xx up to _MAX_RETRIES]+ C6 -- TikHubError --> Emap[raise_user_error: 401 / 403 / 429 retry_after / 400-422 detail / 404 / 5xx -> NodeUserError]+ C6 -- ok --> C7[to_plain: model_dump if present, unwrap data / code / router]+ C7 -- in-body code >= 400 --> Ebody[NodeUserError with body message]+ C7 -- ok --> C8[track_tikhub_usage ctx, call, path -> api_usage_metrics row, cost_usd]+ C8 --> C9[_shape call: endpoint, path, data, code, router, cost_usd]++ B -- fetch_url --> F1{url given?}+ F1 -- no --> Eurl[NodeUserError: url required]+ F1 -- yes --> F2[same path as call with hybrid_parsing.video_data url, minimal]+ F2 --> C8++ B -- account --> A1[make_client] --> A2[tikhub_user.get_user_info + get_user_daily_usage]+ A2 --> A3[track_tikhub_usage action=account -> cost 0.0]+ A3 --> A4[_shape account: account, usage, cost_usd]+```++## Decision Logic++- **Validation**: `call` requires `endpoint`; `fetch_url` requires `url`.+ `limit` is bounded `1..2000` by Pydantic. `params` must be an object+ (blank / JSON-string inputs are coerced first).+- **Endpoint resolution** accepts three spellings (`resource.method`,+ `resource/method`, `/api/v1/...` path or URL containing it) and fails+ before any network call with up to five close matches.+- **Parameter binding** happens against the live SDK signature before+ the request; unknown or missing-required kwargs raise with the accepted+ parameter list.+- **Platform filter** in `list_endpoints` is `info.platform == platform`+ (first underscore segment of the resource name); `all` disables the+ filter. `search` is a case-insensitive substring over id, summary and+ path.+- **Dropdown loader** (`load_tikhub_endpoints`) returns `[]` for+ `platform == "all"` (the UI prompts to pick a platform) and `[]` with a+ WARN log when the SDK cannot be imported - discovery never breaks the+ parameter panel.+- **Error paths**: pre-flight (`Unknown TikHub endpoint`, `rejected+ params`), credential (`PermissionDeniedError` envelope), SDK+ `TikHubError` subclasses mapped to `NodeUserError` messages (401 key+ rejected -> Credentials; 403 scope-or-balance -> run `account`; 429+ with `retry_after`; 400/422 with TikHub `detail` and ~500 chars of body;+ 404 route gone; 5xx / connection -> upstream after retries;+ `FeatureRemoved` / `Config` -> `str(e)`), in-body `code >= 400` ->+ `NodeUserError`. Anything else propagates with a traceback.+- **Fallbacks**: `path` is `None` when neither the docstring nor the+ source carries the route literal (endpoint still callable by id).++## Side Effects++- **Database writes**: one `api_usage_metrics` row per successful `call` /+ `fetch_url` (`service="tikhub"`, `operation` from `operation_map`,+ `endpoint=<REST path>`, `resource_count=1`, `cost=0.001`) and per+ `account` (`cost=0.0`); none for `list_endpoints`; none after a 4xx /+ 5xx. Written by `track_tikhub_usage` in a `try/except` - a metrics+ failure logs a WARN and never fails the call.+- **Broadcasts**: per-node status via `BaseNode.execute`+ (`update_node_status` executing / success / error); the credential+ path emits `credential.api_key.runtime_failed` when the key is missing.+ No plugin-specific broadcasts.+- **External API calls**: `GET|POST https://api.tikhub.io/api/v1/<platform>/<api>/<action>`+ with `Authorization: Bearer <key>` via the SDK's internal httpx client;+ `fetch_url` -> `GET /api/v1/hybrid/video_data`; `account` ->+ `GET /api/v1/tikhub/user/get_user_info` ++ `GET /api/v1/tikhub/user/get_user_daily_usage`. `list_endpoints` makes+ no call.+- **File I/O**: none.+- **Subprocess**: none.++## External Dependencies++- **Credentials**: `TikHubCredential` (`ApiKeyCredential`, id `tikhub`,+ bearer `Authorization`) resolved via `TikHubCredential.resolve(user_id=ctx.credential_customer_id)`;+ stored in `EncryptedAPIKey`. Modal probe: `GET /api/v1/tikhub/user/get_user_info`.+- **Services**: `PricingService` (`api.tikhub` + `operation_map.tikhub`+ in `pricing.json`), `Database.save_api_usage_metric`, option-loader+ registry (`tikhubEndpoints`).+- **Python packages**: `tikhub>=2.1.2,<3` (imported lazily inside `_sdk.py`+ functions only; brings `httpx`, `pydantic`, `anyio`, `typer`,+ `shellingham`).+- **Environment variables**: none. `TIKHUB_API_KEY` in the process+ environment is deliberately ignored - the key is always passed+ explicitly.++## Edge cases & known limits++- **First use pays introspection**: `endpoint_index()` reflects ~1,100+ SDK methods (~100-300 ms) once per process, behind an `asyncio.Lock`.+- **`platform: all` yields no dropdown options** by design; the LLM path+ is unaffected because `endpoint` is free text.+- **Flat pricing is a floor**: TikHub prices some endpoints above+ `$0.001`; `cost_usd` records the flat rate, `account` reports the true+ balance.+- **403 is ambiguous** (plan scope vs exhausted balance) - the message+ defers to `account`.+- **In-body errors**: TikHub can answer HTTP 200 with `code: 400` in the+ JSON; the SDK does not raise on it, `to_plain` does.+- **Non-idempotent retries**: the SDK retries 429 / 5xx itself+ (`_MAX_RETRIES`, default 2); mapped errors are `NodeUserError` so+ Temporal does not re-run a paid call on top of that.+- **Pagination is caller-driven**: cursor names differ per platform+ (`max_cursor`, `cursor`, `pagination_token`, `continuation_token`,+ `after`, `lastCursor`, `pcursor`); the node passes whatever `params`+ carries and never loops.+- **SDK drift**: resources regenerate between SDK minors; `PLATFORMS`+ and the skill's cited ids are guarded by contract tests 12 and 18+ (see [TikHub Service](../../tikhub_service.md) for the bump recipe).+- **Batch POST endpoints** take `params: {"body": [...]}`; there is no+ separate body field.++## Related++- **Skills using this as a tool**: [`tikhub-skill/SKILL.md`](../../../server/skills/web_agent/tikhub-skill/SKILL.md)+- **Companion nodes**: [`apifyActor`](./apifyActor.md) (actor-based scraping with datasets), [`browser`](./browser.md), [`crawleeScraper`](./crawleeScraper.md)+- **Architecture docs**: [TikHub Service](../../tikhub_service.md), [Plugin System](../../plugin_system.md), [Pricing Service](../../pricing_service.md), [Credentials Encryption](../../credentials_encryption.md)diff --git a/docs-internal/plugin_system.md b/docs-internal/plugin_system.mdindex f12961d0..ee87fed8 100644--- a/docs-internal/plugin_system.md+++ b/docs-internal/plugin_system.md@@ -396,7 +396,7 @@ on a plugin resolves to a registered class.| `nodes/location/_credentials.py` | `GoogleMapsCredential` | api_key (query) | gmaps_create / gmaps_locations / gmaps_nearby_places || `nodes/twitter/_credentials.py` | `TwitterCredential` | oauth2 | twitterSend / twitterSearch / twitterUser / twitterReceive || `nodes/telegram/_credentials.py` | `TelegramCredential` | api_key | telegramSend / telegramReceive |-| `nodes/scraper/_credentials.py` | `ApifyCredential` | api_key (bearer) | apifyActor |+| `nodes/scraper/_credentials.py` | `ApifyCredential` / `TikHubCredential` | api_key (bearer) | apifyActor / tikhubAction (TikHub probes `tikhub/user/get_user_info` declaratively — see [tikhub_service.md](./tikhub_service.md)) || `nodes/model/_credentials.py` | `OpenAI / Anthropic / Gemini / OpenRouter / Groq / Cerebras / DeepSeek / Kimi / Mistral / Xai / Sarvam / Ollama / LMStudio` | api_key | 13 credential classes covering the 13 agent-selectable providers and the 12 standalone chat-model nodes (`ls server/nodes/model/*_chat_model`). Ollama / LM Studio store a local server URL; xAI is agent-selectable but has no standalone node; `SarvamCredential` also serves the speech / translate plugins. || `nodes/search/<name>/__init__.py` (inline) | `BraveSearch / Serper / Perplexity` | api_key | single-use search nodes |@@ -710,7 +710,7 @@ server/│ ├── proxy/ # proxy_request / proxy_config / proxy_status│ │ └── _usage.py # Shared track_proxy_usage│ ├── search/ # brave / serper / perplexity / duckduckgo-│ ├── scraper/ # apify / crawlee+│ ├── scraper/ # apify / crawlee / tikhub_action (SDK-backed, reflective resource.method dispatch)│ ├── tool/ # calculator / currentTime / taskManager / writeTodos / agent_builder / canvas / data_source / simple_memory│ ├── trigger/ # webhookTrigger / chatTrigger / taskTrigger│ ├── workflow/ # startdiff --git a/docs-internal/pricing_service.md b/docs-internal/pricing_service.mdindex 75ab42aa..ffdc634c 100644--- a/docs-internal/pricing_service.md+++ b/docs-internal/pricing_service.md@@ -2,10 +2,10 @@## Overview-The pricing service provides centralized cost tracking for both LLM tokens and external API services (Twitter/X, Google Maps). It supports:+The pricing service provides centralized cost tracking for both LLM tokens and external API services (Twitter/X, Google Maps, TikHub). It supports:- **LLM Token Costs**: Per-model pricing with input/output/cache/reasoning token breakdown-- **API Service Costs**: Per-request/resource pricing for third-party APIs (Twitter/X, Google Maps, Search APIs)+- **API Service Costs**: Per-request/resource pricing for third-party APIs (Twitter/X, Google Maps, TikHub, Search APIs)- **Automatic Tracking**: HTTPX event hooks for transparent API call tracking- **Manual Tracking**: Helper functions for services that don't use HTTPX@@ -286,6 +286,10 @@ async def track_google_usage(node_id, service, action, resource_count, context):# ... save to database ...```+### Example: TikHub Plugin (flat per-request)++TikHub bills roughly $0.001 per successful request and does not charge non-2xx responses, so `server/nodes/scraper/tikhub_action/_sdk.py::track_tikhub_usage(ctx, action, endpoint_path)` records one `resource_count=1` metric only after a 2xx and never for the local `list_endpoints` operation. It takes the `NodeContext` (not the raw dict), reads `session_id` / `workflow_id` from it, wraps everything in `try/except` + `logger.warning` so a metrics failure never fails a call that already cost money, and returns the USD figure for the node's `cost_usd` output key. `pricing.json` carries `api.tikhub` (`request` 0.001, `meta` 0.0) and `operation_map.tikhub` (`call` / `fetch_url` -> `request`, `account` -> `meta`); the credential entry sets `"usage_service": "tikhub"` so the spend shows in the Credentials modal usage table. See [tikhub_service.md](./tikhub_service.md).+### Example: Search Plugins (declarative cost)The search plugins (`server/nodes/search/{brave_search,serper_search,perplexity_search}`) no longer call a `_track_search_usage()` helper. Each declares its cost inline on the `@Operation` decorator instead:diff --git a/docs-internal/tikhub_service.md b/docs-internal/tikhub_service.mdnew file mode 100644index 00000000..e86df8a8--- /dev/null+++ b/docs-internal/tikhub_service.md@@ -0,0 +1,352 @@+# TikHub Service (`tikhub` SDK)++The `tikhubAction` node wraps the **official TikHub Python SDK**+(`pip install tikhub`) for [api.tikhub.io](https://api.tikhub.io), a+pay-per-request scraping API covering TikTok, Douyin, Instagram, YouTube,+Twitter/X, Xiaohongshu, Bilibili, Kuaishou, Weibo, Reddit, Threads,+LinkedIn, Zhihu and more (~1,100 methods in the pinned SDK). One+self-contained plugin folder at+[`server/nodes/scraper/tikhub_action/`](../server/nodes/scraper/tikhub_action/).++The SDK is auto-generated from TikHub's `openapi.json`: every OpenAPI tag+becomes a resource attribute on the client (`douyin_web`,+`tiktok_app_v3`, `instagram_v3`, `hybrid_parsing`, `tikhub_user`, ...)+and every route becomes a coroutine method whose keyword-only+parameters mirror the query / body schema. The SDK's only logic is a+retry policy and a typed exception hierarchy; it returns the raw JSON+envelope `{"code": 200, "router": "...", "params": {...}, "data": {...}}`.+TikHub's own CLI has four commands (`health`, `fetch <url>`, `user info`,+`user usage`), so the node is the "flattened CLI": one `call` operation+addressed by the SDK's own `resource.method` id, plus discovery, hybrid+URL fetch, and account ops. **Nothing is vendored** - the endpoint index+is introspected from the installed SDK at runtime, so bumping the pin is+how new endpoints arrive.++| | |+|---|---|+| Node type | `tikhubAction` (palette group `scraper`, dual-purpose AI tool `tikhub_action`; visible in normal mode) |+| Operations | `call` / `list_endpoints` / `fetch_url` / `account` |+| SDK pin | `tikhub>=2.1.2,<3` in `server/pyproject.toml` (resources regenerate between minors; `<3` caps the surface) |+| Auth / probe | `TikHubCredential(ApiKeyCredential)` - bearer key; declarative httpx probe against `GET /api/v1/tikhub/user/get_user_info` (never `/health/check`, which is unauthenticated) |+| Task queue | `TaskQueue.REST_API` |+| Output | `_shape(operation, **fields)` - `{operation, endpoint, path, data, code, router, cost_usd}` for calls; `{operation, platform, count, total, endpoints[]}` for listing; `{operation, account, usage}` for account. `None` fields dropped, `{}` / `[]` kept (an empty answer from TikHub is real) |+| Cost | Flat `$0.001` per successful request (`pricing.json: api.tikhub`), recorded by `track_tikhub_usage` into `api_usage_metrics`; `list_endpoints` never bills, `account` bills at `0.0` |+| Tests | [`server/tests/nodes/test_tikhub.py`](../server/tests/nodes/test_tikhub.py) (`pytestmark = node_contract`; `pytest --collect-only -q` for the live count) |+| Paired skill | [`server/skills/web_agent/tikhub-skill/SKILL.md`](../server/skills/web_agent/tikhub-skill/SKILL.md) |+| Node card | [`docs-internal/node-logic-flows/web_automation/tikhubAction.md`](./node-logic-flows/web_automation/tikhubAction.md) |++## Folder map++```+server/nodes/scraper/+├── _credentials.py # ApifyCredential + TikHubCredential (probe + _handle_probe_response)+├── tikhub.svg # Credential brand icon -> /api/schemas/credentials/tikhub/icon+└── tikhub_action/+ ├── __init__.py # TikHubActionParams / TikHubActionOutput / TikHubActionNode+ │ # (4 @Operations), _shape, register_option_loader("tikhubEndpoints"),+ │ # register_output_schema. Must NOT import `tikhub` at module import.+ ├── _sdk.py # Everything that touches `tikhub`, lazily inside functions:+ │ # make_client, EndpointInfo/ParamInfo, endpoint_index (cached),+ │ # resolve_endpoint, bind_params, to_plain, raise_user_error,+ │ # load_tikhub_endpoints, track_tikhub_usage, PLATFORMS,+ │ # _MAX_RETRIES / _TIMEOUT+ ├── meta.json # {"color": "#ff79c6"} (scraper group colour, matches apify_actor)+ └── icon.svg # Node brand mark+```++Config touched outside the folder: `server/config/credential_providers.json`+(`providers.tikhub`, category `scrapers`, `usage_service: "tikhub"`),+`server/config/pricing.json` (`api.tikhub` + `operation_map.tikhub`),+`server/config/node_allowlist.json` (`tikhubAction` in `enabled_nodes`),+`server/nodes/visuals.json` (`"tikhubAction": {"skill": "tikhub-skill"}`),+`server/tests/fixtures/tool_names_snapshot.json`+(`"tikhubAction": "tikhub_action"`). `server/nodes/__init__.py` is not+touched - pkgutil discovery picks up the folder, and `_sdk.py` is skipped+by the underscore rule.++## Why the SDK, and why import it lazily++Two paths were on the table: hand-rolled httpx against a vendored+endpoint list, or the official SDK with runtime introspection. The SDK+won because the endpoint surface is the product (TikHub adds and retires+routes monthly) and a vendored index would rot within weeks; the SDK pin+is the single knob that tracks upstream. The costs are (a) `typer` ++`shellingham` arrive as transitive deps for the SDK's own CLI (click /+rich were already present; nothing imports typer at runtime), and (b)+resources regenerate between minors, so anything that names an endpoint+(the `PLATFORMS` tuple, the skill cheat sheet, the tool description) can+drift - see "Invariants".++`tikhub` is imported **only inside functions in `_sdk.py`**. The plugin+module itself must import cleanly with `sys.modules["tikhub"] = None`+(test-locked in a subprocess), for the same reason the LLM providers keep+their SDK imports lazy: plugin registration runs at boot for every node+whether or not the operator ever uses TikHub, and an SDK import failure+must degrade to a `NodeUserError` at execution time+("The 'tikhub' package is not installed - run `uv sync` in server/"),+never to a boot failure or a missing node.++## Dispatch - reflective `resource.method`++`call` never names an SDK method in code. The flow is:++1. `resolve_endpoint(raw)` - accepts `resource.method`, `resource/method`,+ a full `/api/v1/...` path, or a URL containing one (via `_BY_PATH`).+ Unknown input raises `NodeUserError` with up to five+ `difflib.get_close_matches` suggestions and the hint to run+ `list_endpoints`. No network call happens before resolution succeeds.+2. `make_client(ctx)` - see "Credential" below. Used as+ `async with` per operation; `AsyncTikHub(api_key=..., timeout=_TIMEOUT,+ max_retries=_MAX_RETRIES)`. `http_client` is **not** passed: respx+ patches httpx globally and the SDK's own tests are respx-based, so the+ contract tests intercept the SDK's internal client.+3. `fn = getattr(getattr(client, info.resource), info.method)`.+4. `bind_params(info, fn, params)` - `inspect.signature(fn).bind(**params)`+ before calling. A `TypeError` becomes+ `NodeUserError(f"{endpoint} rejected params: {e}. Accepted: aweme_id (str, required), ...")`+ so the LLM sees the accepted names without burning a request.+5. `result = to_plain(await fn(**params))` inside+ `try/except TikHubError -> raise_user_error(e, endpoint)`.+ `to_plain` calls `model_dump(mode="json")` if the SDK ever returns+ models, unwraps `data` / `code` / `router`, and raises `NodeUserError`+ when the in-body `code` is an int >= 400 (the SDK only raises on HTTP+ status; TikHub sometimes answers 200 with `code: 400` in the body).+6. `track_tikhub_usage(ctx, "call", info.path)` after success only.+7. `_shape("call", endpoint=..., path=..., data=..., code=..., router=..., cost_usd=...)`.++`fetch_url` is the same path hard-wired to+`hybrid_parsing.video_data(url=..., minimal=...)`. `account` calls+`tikhub_user.get_user_info()` + `tikhub_user.get_user_daily_usage()` and+returns them as `account` / `usage`, tracked at zero cost.++### Batch POST endpoints++A few SDK methods take a single `body: list[Any]` keyword (e.g.+`douyin_web.fetch_multi_video`). They ride the same `params` field -+`{"body": [...]}` - and bind like any other keyword; nothing in the node+special-cases HTTP method.++## Runtime introspection - `endpoint_index()`++Module-level `_INDEX` cache behind an `asyncio.Lock`, built once per+process on first use (`list_endpoints`, the dropdown loader, or the first+`call`). Cost is ~100-300 ms for ~1,100 methods.++- **Resources**: a throwaway `async with AsyncTikHub(api_key="introspection")`+ (no network); iterate `vars(client)` and keep non-underscore attributes+ that are `tikhub.resources._base.AsyncResource` instances (fallback:+ `type(obj).__module__.startswith("tikhub.resources")`).+- **Methods**: `inspect.getmembers(type(obj), inspect.iscoroutinefunction)`,+ skipping `_`-prefixed names.+- **Params**: the `KEYWORD_ONLY` parameters of `inspect.signature(fn)`.+ Annotations are strings (the generated modules use+ `from __future__ import annotations`) and are kept as display text -+ no `eval_str`. `required = default is Parameter.empty`.+- **HTTP method + path**: regex over `inspect.getdoc(fn)`:+ ``r"``(GET|POST|PUT|PATCH|DELETE)\s+(/\S+?)``"`` - the generated+ docstrings carry the route literal as their last line+ (verified on 2.1.2: every one of the 1,100+ methods matched).+ Fallback: the same regex over `inspect.getsource(fn)` guarded by+ `try/except (OSError, TypeError)`; else `path=None` (display only -+ such an endpoint can still be called by id, just not by path).+- **Summary**: first docstring line; the generated summaries are+ bilingual (`"获取单个作品数据/Get single video data"`), so the option+ label prefers the English part after the last `/`.+- **Record**: `EndpointInfo(endpoint="douyin_web.fetch_one_video", resource,+ method, platform=resource.split("_")[0], http_method, path, summary,+ params=(ParamInfo(name, type, required), ...))`, sorted by `endpoint`,+ plus `_BY_ENDPOINT` and `_BY_PATH` dicts.++`PLATFORMS` is a static tuple - `("all", "tiktok", "douyin", "instagram",+"youtube", "twitter", "xiaohongshu", "bilibili", "kuaishou", "weibo",+"reddit", "threads", "linkedin", "zhihu", "toutiao", "xigua", "wechat",+"lemon8", "pipixia", "hybrid", "tikhub")` - because it feeds a Pydantic+`Literal` that must exist at class-definition time, before any SDK import.+Filter rule is `info.platform == platform`, i.e. the first underscore+segment of the resource name. A test asserts every value except `all`+matches at least one live resource, so an SDK rename fails loudly rather+than producing an empty dropdown.++### The dynamic dropdown++`register_option_loader("tikhubEndpoints", load_tikhub_endpoints)`; the+`endpoint` field declares `dynamicOptions: True`,+`loadOptionsMethod: "tikhubEndpoints"`, `loadOptionsDependsOn: ["platform"]`.+The loader wraps `endpoint_index()` in `try/except Exception ->+logger.warning + []` (an SDK problem must never break the parameter+panel). For `platform == "all"` it returns `[]` rather than ~1,000 options+- the UI prompts the user to pick a platform, while the LLM path still+accepts any id as free text because `endpoint` is `str`, not a `Literal`.+Options are `{"value": e.endpoint, "label": e.endpoint,+"description": f"{summary} - {http_method} {path}"}`.++## Credential and probe++```python+class TikHubCredential(ApiKeyCredential):+ id = "tikhub"; display_name = "TikHub"; category = "Scrapers"+ key_name = "Authorization"; key_location = "bearer"+ docs_url = "https://docs.tikhub.io"+ probe_url = "https://api.tikhub.io/api/v1/tikhub/user/get_user_info"+```++- **Runtime resolution** goes through+ `await TikHubCredential.resolve(user_id=ctx.credential_customer_id)` -+ the sanctioned SDK-node path from the `services/plugin/connection.py`+ docstring. A missing key raises the annotated `PermissionError`, which+ `BaseNode.execute` turns into the `PermissionDeniedError` envelope plus+ the `credential.api_key.runtime_failed` broadcast (the modal badge+ flips red). This deliberately does NOT copy Apify's "return None then+ `NodeUserError`" shape, which loses the structured envelope.+- The key is passed to `AsyncTikHub(api_key=...)` explicitly so an ambient+ `TIKHUB_API_KEY` in the server's environment can never authenticate a+ node implicitly.+- **Probe** is the declarative httpx path of `ApiKeyCredential` (not the+ SDK) so the credentials modal keeps working even if the SDK import+ fails. `_handle_probe_response` reads `user_data.email / balance /+ free_credit` and `api_key_data.api_key_name` defensively (field names+ were inferred from the CLI's `user info` output; absence is tolerated)+ into `ProbeResult(extra=...)`, message+ `"TikHub key validated - {email} (balance ${balance})"`. A 401 surfaces+ through the base `raise_for_status` path.++Catalogue entry (`credential_providers.json`, same shape as `apify`, no+`validate_as`): `kind: "apiKey"`, one required password field `apiKey`+with placeholder "TikHub API key (user portal -> API keys)", icon ref+`/api/schemas/credentials/tikhub/icon`, `usage_service: "tikhub"` so the+API usage panel groups the rows.++## Error mapping - `raise_user_error(e, endpoint)`++Exception classes are imported lazily from `tikhub`. Every mapped case+becomes a `NodeUserError` (one WARN line, no traceback, structured+envelope). `request_id` is appended when the SDK exposes it.++| SDK exception | HTTP | User message (shape) | Why this shape |+|---|---|---|---|+| `TikHubAuthError` | 401 | `TikHub rejected the API key (401) while calling {endpoint}. Update it in Credentials -> TikHub.` | Not retryable; points at the fix |+| `TikHubPermissionError` | 403 | `TikHub refused {endpoint} (403): {detail}. The key may lack access ... or the account balance may be exhausted - run operation=account to check.` | 403 means EITHER the key's plan excludes the endpoint OR balance is exhausted; the node cannot tell which, `account` can |+| `TikHubRateLimitError` | 429 | `TikHub rate-limited {endpoint} (429); retry after {retry_after}s. Each endpoint allows about 10 requests/s.` | 10 req/s per endpoint path; SDK already retried with backoff |+| `TikHubBadRequestError` / `TikHubValidationError` | 400 / 422 | `TikHub rejected the request to {endpoint} ({status}): {detail}. Response: {body}` (body truncated to ~500 chars; `TikHubValidationError` reads `unparseable response`) | The detail is what the LLM needs to fix the args |+| `TikHubNotFoundError` | 404 | `TikHub returned 404 for {endpoint}: {detail}. The route or the requested resource does not exist ...` | Route retired or resource id gone; points at `list_endpoints` |+| `TikHubUpstreamError` / `TikHubServerError` / `TikHubConnectionError` | 5xx / network | `TikHub upstream error while calling {endpoint} (HTTP {status}): {detail}. Retries were exhausted - try again later.` | SDK already retried; a WARN envelope beats a traceback plus Temporal re-runs on a non-idempotent paid call |+| `TikHubFeatureRemovedError` / `TikHubConfigError` | - | `str(e)` | SDK-side messages are already user-facing |+| anything else | - | propagates | Genuine bugs keep the full traceback through `BaseNode.execute` |++In-body errors (HTTP 200, `code >= 400` in the JSON) are raised by+`to_plain` with the body's message. Pre-flight failures (`Unknown TikHub+endpoint ...`, `... rejected params ...`) are raised before any HTTP call.++## Cost tracking++`pricing.json`:++```json+"api": { "tikhub": { "_description": "TikHub pay-per-request (non-2xx not billed)",+ "_source": "https://tikhub.io/pricing", "_updated": "2026-09",+ "request": 0.001, "meta": 0.0, "_default": 0.001 } },+"operation_map": { "tikhub": { "call": "request", "fetch_url": "request", "account": "meta" } }+```++Both halves are required - `calculate_api_cost` returns `0` silently on a+missing `operation_map` line. The `$0.001` figure is TikHub's list price+for the common tier; per-endpoint pricing exists (`tikhub_user.get_endpoint_info`+returns it) but a flat rate was the user's decision (2026-09-04) because+the per-endpoint table changes independently of the SDK and would need+its own refresh loop.++`track_tikhub_usage(ctx, action, endpoint_path)` in `_sdk.py` is modelled+on `track_twitter_usage` (`server/nodes/twitter/_base.py`):+`get_pricing_service().calculate_api_cost("tikhub", action, 1)` then+`get_database().save_api_usage_metric({session_id, node_id, workflow_id,+service: "tikhub", operation, endpoint: <REST path>, resource_count: 1,+cost})`, wrapped in `try/except -> logger.warning` (a metrics failure+never fails a paid call that already succeeded). Called **only after a+successful call**, never for `list_endpoints`, never after a 4xx/5xx+(TikHub does not bill those either). Returns `total_cost` for the+`cost_usd` output key. The `cost=` metadata on each `@Operation` is+dormant - the manual tracker does the work, because the framework's+cost hook fires per operation and cannot know whether the call was+billed.++## SDK version-bump recipe++1. Edit the pin in `server/pyproject.toml` (`"tikhub>=2.1.2,<3"` - keep+ the `<3` cap unless you have re-verified the introspection contract+ against the new major), then `cd server && uv sync`.+2. Smoke the two things the node relies on:+ ```+ uv run python -c "import sys, tikhub; print(tikhub.__version__, 'typer' in sys.modules)" # version, False+ uv run python -c "import sys; import nodes.scraper.tikhub_action; print('tikhub' in sys.modules)" # False+ uv run python -c "import asyncio; from nodes.scraper.tikhub_action._sdk import endpoint_index; ix=asyncio.run(endpoint_index()); print(len(ix), ix[0])"+ ```+ Eyeball the count against the previous release (2.1.2 indexes ~1,100+ methods across 53 resources). A large drop means a resource stopped+ being an `AsyncResource` instance attribute or the docstring route+ convention changed - fix the introspection fallbacks before shipping.+3. Run the two drift guards from the contract suite:+ `uv run pytest tests/nodes/test_tikhub.py -k "platforms or skill_endpoint"`+ (test 12: every `PLATFORMS` value except `all` matches at least one+ live resource and the index exceeds 500 entries; test 18: every+ backticked `x.y` id in `SKILL.md` resolves in the live index). Then+ the full file: `uv run pytest tests/nodes/test_tikhub.py -v`.+4. If test 18 fails, update the skill's cheat sheet and the ids in the+ node's `tool_description` - prefer the highest `_vN` variant the new+ SDK exposes. If test 12 fails, adjust `PLATFORMS`.+5. `uv run ruff check nodes/scraper tests/nodes/test_tikhub.py` and+ `python -m cli docs nodes --check` from the repo root.++## Invariants locked by `test_tikhub.py`++Registration (type / tool name / group / credentials / `usable_as_tool` /+both handles visible - `usable_as_tool = True` auto-hides both handles+unless `hide_input_handle = False` and `hide_output_handle = False` are+declared explicitly); `get_option_loader("tikhubEndpoints")` registered;+`tikhub` in `CREDENTIAL_REGISTRY`; tool schema has no `$defs`. `call`+sends the bearer header and the keyword args as query params, drops+`None` args, unwraps `data`. `params` accepted as a JSON string and as+`""` from the panel (`coerce_blank_params`). REST-path alias resolves.+Unknown endpoint and unknown kwarg fail **before** any HTTP call. Missing+key produces `error_type == "PermissionDeniedError"` with+`credential.provider == "tikhub"`. 401 / 403 / 429 (+ `Retry-After`) /+422 detail / 502 map to `NodeUserError`; HTTP 200 with in-body `code: 400`+is a user error. `fetch_url` forwards `url` + `minimal`; `account` makes+two GETs. `list_endpoints` filters by platform + search and reports+`douyin_web.fetch_one_video` as `GET` with its path and a required+`aweme_id`. Every `PLATFORMS` value except `all` matches a live resource;+index length > 500. The loader returns options for a platform and `[]`+when `tikhub` is missing. The plugin imports in a subprocess with+`sys.modules["tikhub"] = None`, and importing the plugin does not import+`tikhub`. Probe 200 yields `ProbeResult.valid` + `extra["balance"]`;+401 raises. Catalogue / assets present (`credential_providers.json`+entry, `visuals.json` skill, `node_allowlist.json`, `SKILL.md`,+`icon.svg`, `tikhub.svg`, `meta.json` colour). `save_api_usage_metric`+awaited once with `service == "tikhub"` after a 2xx, not for+`list_endpoints`, not after a 4xx. Every skill endpoint id resolves.++## Risks++- **SDK churn.** Resources regenerate between minors; `PLATFORMS`, the+ skill cheat sheet and the tool description can rot. Guarded by tests 12+ and 18 and the `<3` cap - the bump recipe above is the maintenance+ loop.+- **Introspection assumptions.** Resources must be `AsyncResource`+ instance attributes and docstrings must carry the route literal. Both+ have fallbacks (module-name check; `getsource` regex; `path=None`), but+ a generator rewrite could still shrink the index - hence the "eyeball+ the count" step.+- **First-use latency.** The first `list_endpoints` / dropdown open+ reflects ~1,100 methods (~100-300 ms), cached per process; a worker+ restart pays it again.+- **Probe field names** were inferred from the CLI's output, not from a+ schema; extraction is defensive and should be tightened after a+ real-key run.+- **403 ambiguity.** TikHub uses 403 for both "not in your plan" and+ "balance exhausted"; the message points the user at `account` rather+ than guessing.+- **Flat pricing is an approximation.** Some endpoints cost more than+ `$0.001`; the recorded cost is a floor, and the balance reported by+ `account` is the truth.+- **New transitive deps.** `typer` and `shellingham` arrive for the SDK's+ own CLI; nothing in OpenCompany imports them and no version conflicts+ were found at 2.1.2.diff --git a/server/config/credential_providers.json b/server/config/credential_providers.jsonindex aca0acbe..9fb27d51 100644--- a/server/config/credential_providers.json+++ b/server/config/credential_providers.json@@ -717,6 +717,25 @@}]},+ "tikhub": {+ "name": "TikHub",+ "category": "scrapers",+ "category_label": "Scrapers",+ "color": "theme.categoryAI",+ "kind": "apiKey",+ "icon_ref": "/api/schemas/credentials/tikhub/icon",+ "usage_service": "tikhub",+ "fields": [+ {+ "key": "apiKey",+ "label": "API Key",+ "type": "password",+ "secret": true,+ "placeholder": "TikHub API key (user portal → API keys)",+ "required": true+ }+ ]+ },"serper": {"name": "Serper","category": "scrapers",diff --git a/server/config/model_registry.json b/server/config/model_registry.jsonindex aa19365b..810f2164 100644--- a/server/config/model_registry.json+++ b/server/config/model_registry.json@@ -1,8 +1,192 @@{- "_generated": "2026-09-03T11:26:38.057459+00:00",+ "_generated": "2026-09-05T05:04:06.930651+00:00","_source": "openrouter",- "_model_count": 424,+ "_model_count": 431,"models": {+ "openai/gpt-6-astra": {+ "id": "openai/gpt-6-astra",+ "name": "OpenAI: GPT-6 Astra",+ "provider": "openai",+ "local_id": "gpt-6-astra",+ "context_length": 1050000,+ "max_output_tokens": 128000,+ "input_price_per_mtok": 10.0,+ "output_price_per_mtok": 50.0,+ "supports_thinking": true,+ "thinking_type": "effort",+ "temperature_range": [+ 0.0,+ 2.0+ ],+ "is_reasoning_model": false,+ "supported_parameters": [+ "include_reasoning",+ "max_completion_tokens",+ "max_tokens",+ "reasoning",+ "reasoning_effort",+ "response_format",+ "seed",+ "structured_outputs",+ "tool_choice",+ "tools"+ ]+ },+ "openai/gpt-6-astra:batch": {+ "id": "openai/gpt-6-astra:batch",+ "name": "OpenAI: GPT-6 Astra (batch)",+ "provider": "openai",+ "local_id": "gpt-6-astra:batch",+ "context_length": 1050000,+ "max_output_tokens": 128000,+ "input_price_per_mtok": 5.0,+ "output_price_per_mtok": 25.0,+ "supports_thinking": true,+ "thinking_type": "effort",+ "temperature_range": [+ 0.0,+ 2.0+ ],+ "is_reasoning_model": false,+ "supported_parameters": [+ "include_reasoning",+ "max_tokens",+ "reasoning",+ "reasoning_effort",+ "response_format",+ "seed",+ "structured_outputs",+ "tool_choice",+ "tools"+ ]+ },+ "openai/gpt-6-astra-pro": {+ "id": "openai/gpt-6-astra-pro",+ "name": "OpenAI: GPT-6 Astra Pro",+ "provider": "openai",+ "local_id": "gpt-6-astra-pro",+ "context_length": 1050000,+ "max_output_tokens": 128000,+ "input_price_per_mtok": 10.0,+ "output_price_per_mtok": 50.0,+ "supports_thinking": true,+ "thinking_type": "effort",+ "temperature_range": [+ 0.0,+ 2.0+ ],+ "is_reasoning_model": false,+ "supported_parameters": [+ "include_reasoning",+ "max_completion_tokens",+ "max_tokens",+ "reasoning",+ "reasoning_effort",+ "response_format",+ "seed",+ "structured_outputs",+ "tool_choice",+ "tools"+ ]+ },+ "openai/gpt-6-astra-pro:batch": {+ "id": "openai/gpt-6-astra-pro:batch",+ "name": "OpenAI: GPT-6 Astra Pro (batch)",+ "provider": "openai",+ "local_id": "gpt-6-astra-pro:batch",+ "context_length": 1050000,+ "max_output_tokens": 128000,+ "input_price_per_mtok": 5.0,+ "output_price_per_mtok": 25.0,+ "supports_thinking": true,+ "thinking_type": "effort",+ "temperature_range": [+ 0.0,+ 2.0+ ],+ "is_reasoning_model": false,+ "supported_parameters": [+ "include_reasoning",+ "max_tokens",+ "reasoning",+ "reasoning_effort",+ "response_format",+ "seed",+ "structured_outputs",+ "tool_choice",+ "tools"+ ]+ },+ "inclusionai/ling-3.0-flash-sante:free": {+ "id": "inclusionai/ling-3.0-flash-sante:free",+ "name": "inclusionAI: Ling 3.0 Flash Sante (free)",+ "provider": "inclusionai",+ "local_id": "ling-3.0-flash-sante:free",+ "context_length": 262144,+ "max_output_tokens": 32768,+ "input_price_per_mtok": 0.0,+ "output_price_per_mtok": 0.0,+ "supports_thinking": false,+ "thinking_type": "none",+ "temperature_range": [+ 0.0,+ 2.0+ ],+ "is_reasoning_model": false,+ "supported_parameters": [+ "frequency_penalty",+ "include_reasoning",+ "logprobs",+ "max_tokens",+ "presence_penalty",+ "reasoning",+ "repetition_penalty",+ "seed",+ "stop",+ "temperature",+ "tool_choice",+ "tools",+ "top_k",+ "top_logprobs",+ "top_p"+ ]+ },+ "qwen/qwen3.8-max-0902": {+ "id": "qwen/qwen3.8-max-0902",+ "name": "Qwen: Qwen3.8 Max (0902)",+ "provider": "qwen",+ "local_id": "qwen3.8-max-0902",+ "context_length": 1000000,+ "max_output_tokens": 131072,+ "input_price_per_mtok": 2.0,+ "output_price_per_mtok": 6.0,+ "supports_thinking": false,+ "thinking_type": "none",+ "temperature_range": [+ 0.0,+ 2.0+ ],+ "is_reasoning_model": false,+ "supported_parameters": [+ "frequency_penalty",+ "include_reasoning",+ "logprobs",+ "max_tokens",+ "presence_penalty",+ "reasoning",+ "reasoning_effort",+ "response_format",+ "seed",+ "stop",+ "structured_outputs",+ "temperature",+ "tool_choice",+ "tools",+ "top_k",+ "top_logprobs",+ "top_p"+ ]+ },"meta/muse-spark-1.3-contributor": {"id": "meta/muse-spark-1.3-contributor","name": "Meta: Muse Spark 1.3 Contributor",@@ -281,9 +465,45 @@"tools"]},+ "inclusionai/ling-3.0-flash-fin": {+ "id": "inclusionai/ling-3.0-flash-fin",+ "name": "inclusionAI: Ling 3.0 Flash Fin",+ "provider": "inclusionai",+ "local_id": "ling-3.0-flash-fin",+ "context_length": 262144,+ "max_output_tokens": 235929,+ "input_price_per_mtok": 0.06,+ "output_price_per_mtok": 0.18,+ "supports_thinking": false,+ "thinking_type": "none",+ "temperature_range": [+ 0.0,+ 2.0+ ],+ "is_reasoning_model": false,+ "supported_parameters": [+ "frequency_penalty",+ "include_reasoning",+ "logit_bias",+ "max_tokens",+ "min_p",+ "presence_penalty",+ "reasoning",+ "repetition_penalty",+ "response_format",+ "seed",+ "stop",+ "structured_outputs",+ "temperature",+ "tool_choice",+ "tools",+ "top_k",+ "top_p"+ ]+ },"inclusionai/ling-3.0-flash-fin:free": {"id": "inclusionai/ling-3.0-flash-fin:free",- "name": "Ling 3.0 Flash Fin (free)",+ "name": "inclusionAI: Ling 3.0 Flash Fin (free)","provider": "inclusionai","local_id": "ling-3.0-flash-fin:free","context_length": 262144,@@ -590,9 +810,9 @@"provider": "z-ai","local_id": "glm-latest","context_length": 1310720,- "max_output_tokens": 943718,- "input_price_per_mtok": 1.12,- "output_price_per_mtok": 3.52,+ "max_output_tokens": 235929,+ "input_price_per_mtok": 1.15,+ "output_price_per_mtok": 3.5,"supports_thinking": false,"thinking_type": "none","temperature_range": [@@ -696,8 +916,8 @@"local_id": "qwen3.8-27b","context_length": 1000000,"max_output_tokens": 131072,- "input_price_per_mtok": 0.425,- "output_price_per_mtok": 2.55,+ "input_price_per_mtok": 0.42,+ "output_price_per_mtok": 3.0,"supports_thinking": false,"thinking_type": "none","temperature_range": [@@ -961,8 +1181,8 @@"local_id": "deepseek-v4-pro-0813","context_length": 1048576,"max_output_tokens": 384000,- "input_price_per_mtok": 1.1154,- "output_price_per_mtok": 3.3462,+ "input_price_per_mtok": 1.1207,+ "output_price_per_mtok": 3.362,"supports_thinking": false,"thinking_type": "none","temperature_range": [@@ -1212,6 +1432,7 @@"frequency_penalty","include_reasoning","max_tokens",+ "parallel_tool_calls","presence_penalty","reasoning","response_format",@@ -1328,51 +1549,15 @@"top_p"]},- "qwen/qwen3.8-max": {- "id": "qwen/qwen3.8-max",- "name": "Qwen: Qwen3.8 Max",- "provider": "qwen",- "local_id": "qwen3.8-max",- "context_length": 1000000,- "max_output_tokens": 131072,- "input_price_per_mtok": 2.0,- "output_price_per_mtok": 6.0,- "supports_thinking": false,- "thinking_type": "none",- "temperature_range": [- 0.0,- 2.0- ],- "is_reasoning_model": false,- "supported_parameters": [- "frequency_penalty",- "include_reasoning",- "logprobs",- "max_tokens",- "presence_penalty",- "reasoning",- "reasoning_effort",- "response_format",- "seed",- "stop",- "structured_outputs",- "temperature",- "tool_choice",- "tools",- "top_k",- "top_logprobs",- "top_p"- ]- },"deepseek/deepseek-v4-flash-latest": {"id": "~deepseek/deepseek-v4-flash-latest","name": "DeepSeek V4 Flash Latest","provider": "deepseek","local_id": "deepseek-v4-flash-latest","context_length": 1310720,- "max_output_tokens": 393216,+ "max_output_tokens": 131072,"input_price_per_mtok": 0.05,- "output_price_per_mtok": 0.16,+ "output_price_per_mtok": 0.1,"supports_thinking": false,"thinking_type": "none","temperature_range": [@@ -1678,7 +1863,7 @@},"inclusionai/ling-3.0-flash": {"id": "inclusionai/ling-3.0-flash",- "name": "Ling-3.0-flash",+ "name": "inclusionAI: Ling 3.0 Flash","provider": "inclusionai","local_id": "ling-3.0-flash","context_length": 262144,@@ -3232,6 +3417,39 @@"top_p"]},+ "nvidia/nemotron-3.5-content-safety": {+ "id": "nvidia/nemotron-3.5-content-safety",+ "name": "NVIDIA: Nemotron 3.5 Content Safety",+ "provider": "nvidia",+ "local_id": "nemotron-3.5-content-safety",+ "context_length": 131072,+ "max_output_tokens": 117964,+ "input_price_per_mtok": 0.2,+ "output_price_per_mtok": 0.2,+ "supports_thinking": false,+ "thinking_type": "none",+ "temperature_range": [+ 0.0,+ 2.0+ ],+ "is_reasoning_model": false,+ "supported_parameters": [+ "frequency_penalty",+ "include_reasoning",+ "logit_bias",+ "max_tokens",+ "min_p",+ "presence_penalty",+ "reasoning",+ "repetition_penalty",+ "response_format",+ "seed",+ "stop",+ "temperature",+ "top_k",+ "top_p"+ ]+ },"nvidia/nemotron-3.5-content-safety:free": {"id": "nvidia/nemotron-3.5-content-safety:free","name": "NVIDIA: Nemotron 3.5 Content Safety (free)",@@ -3263,9 +3481,9 @@"provider": "nvidia","local_id": "nemotron-3-ultra-550b-a55b","context_length": 262144,- "max_output_tokens": 182520,- "input_price_per_mtok": 0.6,- "output_price_per_mtok": 2.4,+ "max_output_tokens": 32768,+ "input_price_per_mtok": 0.625,+ "output_price_per_mtok": 3.125,"supports_thinking": false,"thinking_type": "none","temperature_range": [@@ -3832,15 +4050,15 @@"top_p"]},- "ibm-granite/granite-4.1-8b": {- "id": "ibm-granite/granite-4.1-8b",- "name": "IBM: Granite 4.1 8B",- "provider": "ibm-granite",- "local_id": "granite-4.1-8b",- "context_length": 131072,- "max_output_tokens": 117964,- "input_price_per_mtok": 0.05,- "output_price_per_mtok": 0.1,+ "xai/grok-4.3:batch": {+ "id": "x-ai/grok-4.3:batch",+ "name": "SpaceXAI: Grok 4.3 (batch)",+ "provider": "xai",+ "local_id": "grok-4.3:batch",+ "context_length": 1000000,+ "max_output_tokens": 900000,+ "input_price_per_mtok": 1.0,+ "output_price_per_mtok": 2.0,"supports_thinking": false,"thinking_type": "none","temperature_range": [@@ -3849,19 +4067,17 @@],"is_reasoning_model": false,"supported_parameters": [- "frequency_penalty",+ "include_reasoning","logprobs","max_tokens",- "presence_penalty",- "repetition_penalty",+ "reasoning",+ "reasoning_effort","response_format","seed",- "stop","structured_outputs","temperature","tool_choice","tools",- "top_k","top_logprobs","top_p"]@@ -3904,8 +4120,8 @@"name": "Mistral: Mistral Medium 3.5 (batch)","provider": "mistralai","local_id": "mistral-medium-3-5:batch",- "context_length": 262144,- "max_output_tokens": 209715,+ "context_length": 32768,+ "max_output_tokens": 26214,"input_price_per_mtok": 0.75,"output_price_per_mtok": 3.75,"supports_thinking": false,@@ -4057,8 +4273,8 @@"local_id": "kimi-latest","context_length": 1048576,"max_output_tokens": 943718,- "input_price_per_mtok": 2.55,- "output_price_per_mtok": 12.75,+ "input_price_per_mtok": 2.5,+ "output_price_per_mtok": 14.0,"supports_thinking": false,"thinking_type": "none","temperature_range": [@@ -4328,9 +4544,9 @@"provider": "qwen","local_id": "qwen3.6-27b","context_length": 262144,- "max_output_tokens": 235929,- "input_price_per_mtok": 0.6,- "output_price_per_mtok": 3.6,+ "max_output_tokens": 65536,+ "input_price_per_mtok": 0.3,+ "output_price_per_mtok": 2.0,"supports_thinking": false,"thinking_type": "none","temperature_range": [@@ -4480,8 +4696,8 @@"local_id": "deepseek-v4-pro","context_length": 1048576,"max_output_tokens": 384000,- "input_price_per_mtok": 1.0423,- "output_price_per_mtok": 2.0845,+ "input_price_per_mtok": 0.9613,+ "output_price_per_mtok": 1.9227,"supports_thinking": false,"thinking_type": "none","temperature_range": [@@ -4520,8 +4736,8 @@"local_id": "deepseek-v4-flash","context_length": 1048576,"max_output_tokens": 384000,- "input_price_per_mtok": 0.0886,- "output_price_per_mtok": 0.1772,+ "input_price_per_mtok": 0.0867,+ "output_price_per_mtok": 0.1733,"supports_thinking": false,"thinking_type": "none","temperature_range": [@@ -5958,9 +6174,9 @@"provider": "qwen","local_id": "qwen3.5-35b-a3b","context_length": 262144,- "max_output_tokens": 235929,- "input_price_per_mtok": 0.25,- "output_price_per_mtok": 1.25,+ "max_output_tokens": 16384,+ "input_price_per_mtok": 0.08,+ "output_price_per_mtok": 0.75,"supports_thinking": false,"thinking_type": "none","temperature_range": [@@ -6719,6 +6935,7 @@"frequency_penalty","include_reasoning","max_tokens",+ "parallel_tool_calls","presence_penalty","reasoning","response_format",@@ -9029,9 +9246,9 @@"provider": "deepseek","local_id": "deepseek-chat-v3.1","context_length": 163840,- "max_output_tokens": 32768,- "input_price_per_mtok": 0.25,- "output_price_per_mtok": 0.95,+ "max_output_tokens": 144900,+ "input_price_per_mtok": 0.55,+ "output_price_per_mtok": 1.65,"supports_thinking": false,"thinking_type": "none","temperature_range": [@@ -9829,9 +10046,9 @@"provider": "qwen","local_id": "qwen3-235b-a22b-2507","context_length": 262144,- "max_output_tokens": 235929,- "input_price_per_mtok": 0.0875,- "output_price_per_mtok": 0.35,+ "max_output_tokens": 16384,+ "input_price_per_mtok": 0.09,+ "output_price_per_mtok": 0.55,"supports_thinking": false,"thinking_type": "none","temperature_range": [@@ -11452,9 +11669,9 @@"provider": "qwen","local_id": "qwen2.5-vl-72b-instruct","context_length": 128000,- "max_output_tokens": 28800,- "input_price_per_mtok": 0.25,- "output_price_per_mtok": 0.75,+ "max_output_tokens": 115200,+ "input_price_per_mtok": 0.8,+ "output_price_per_mtok": 1.0,"supports_thinking": false,"thinking_type": "none","temperature_range": [@@ -12518,7 +12735,6 @@"supported_parameters": ["frequency_penalty","logit_bias",- "logprobs","max_tokens","min_p","presence_penalty",@@ -12531,7 +12747,6 @@"tool_choice","tools","top_k",- "top_logprobs","top_p"]},@@ -13243,8 +13458,8 @@"provider": "undi95","local_id": "remm-slerp-l2-13b","context_length": 6144,- "max_output_tokens": 4096,- "input_price_per_mtok": 0.45,+ "max_output_tokens": 5529,+ "input_price_per_mtok": 0.35,"output_price_per_mtok": 0.65,"supports_thinking": false,"thinking_type": "none",diff --git a/server/config/node_allowlist.json b/server/config/node_allowlist.jsonindex bdacb4df..89f8c4c5 100644--- a/server/config/node_allowlist.json+++ b/server/config/node_allowlist.json@@ -32,6 +32,8 @@"timer","cronScheduler",+ "tikhubAction",+"telegramSend","telegramReceive",diff --git a/server/config/pricing.json b/server/config/pricing.jsonindex 1985c7b8..cbc7eced 100644--- a/server/config/pricing.json+++ b/server/config/pricing.json@@ -255,6 +255,14 @@"web_search": 0.003,"_default": 0.003},+ "tikhub": {+ "_description": "TikHub pay-per-request (non-2xx not billed)",+ "_source": "https://tikhub.io/pricing",+ "_updated": "2026-09",+ "request": 0.001,+ "meta": 0.0,+ "_default": 0.001+ },"serper": {"_description": "Serper Google Search API pricing (per query)","_source": "https://serper.dev/pricing",@@ -420,6 +428,11 @@"brave_search": {"web_search": "web_search"},+ "tikhub": {+ "call": "request",+ "fetch_url": "request",+ "account": "meta"+ },"serper": {"web_search": "web_search"},diff --git a/server/nodes/README.md b/server/nodes/README.mdindex acac67b9..c609d34b 100644--- a/server/nodes/README.md+++ b/server/nodes/README.md@@ -123,7 +123,8 @@ social/ — Unified social (send / receive). Names no platform: eachonto its own parameter shape.email/ — IMAP/SMTP via Himalaya CLIsearch/ — Web search APIs (brave / serper / perplexity / duckduckgo)-scraper/ — Apify / Crawlee+scraper/ — Apify / Crawlee / TikHub (SDK-backed, flattened like a+ CLI: list_endpoints + call over resource.method ids)document/ — RAG pipeline (scrape / download / parse / chunk / embed / store)code/ — Python / Monty (sandboxed Python) / JS / TS executorsfilesystem/ — file_read / file_modify / shell / fs_search / gallery@@ -210,7 +211,7 @@ from ._credentials import TwitterCredential # shared with 3 sibling| `nodes/twitter/` | `TwitterCredential` (OAuth2 + PKCE) | twitter_send / _search / _user / _receive || `nodes/telegram/` | `TelegramCredential` (bot token + owner chat id) | telegram_send / _receive || `nodes/discord/` | `DiscordBotCredential` (bot token; overrides `inject()` because Discord uses `Bot <token>`, not the inherited `Bearer `) + `DiscordUserCredential` (OAuth2 user context, separate id so connecting a user never overwrites the bot) | discord_send / _action / _receive / _interaction |-| `nodes/scraper/` | `ApifyCredential` (Bearer) | apify_actor |+| `nodes/scraper/` | `ApifyCredential` (Bearer, SDK probe) + `TikHubCredential` (Bearer, declarative httpx probe against `tikhub/user/get_user_info` — kept SDK-free so the modal validates even if the `tikhub` import fails) | apify_actor / tikhub_action || `nodes/model/` | 13 LLM credential classes: 11 cloud (`OpenAI / Anthropic / Gemini / OpenRouter / Groq / Cerebras / DeepSeek / Kimi / Mistral / xAI / Sarvam`) plus Ollama / LM Studio | 12 chat models (xAI has no standalone chat-model node) **plus the 5 `nodes/sarvam/` service nodes**, which import `SarvamCredential` from here — one stored key serves Sarvam's OpenAI-compatible chat endpoint *and* its `api-subscription-key` REST APIs || `nodes/search/` | `BraveSearch / Serper / Perplexity` inlined in each plugin file | single-use per plugin |diff --git a/server/nodes/scraper/_credentials.py b/server/nodes/scraper/_credentials.pyindex 16bf170c..864d5be7 100644--- a/server/nodes/scraper/_credentials.py+++ b/server/nodes/scraper/_credentials.py@@ -1,11 +1,17 @@-"""Apify credential (Wave 11.E.1 — per-domain).+"""Scraper credentials (Wave 11.E.1 — per-domain).-The crawlee_scraper plugin in this folder may also add its own Credential-subclasses here in future.+``ApifyCredential`` probes through the Apify SDK; ``TikHubCredential`` uses+the declarative httpx probe on purpose — the credentials modal must keep+working even if the ``tikhub`` SDK import fails, so this module stays+SDK-free."""from __future__ import annotations+from typing import Any, Dict, Optional++import httpx+from services.plugin.credential import ApiKeyCredential, ProbeResult@@ -44,3 +50,82 @@ class ApifyCredential(ApiKeyCredential):"plan": result.get("plan"),},)+++_ENVELOPE_MESSAGE_KEYS = ("message", "error", "detail", "msg")+++def _envelope_message(body: Dict[str, Any], fallback: str) -> str:+ for key in _ENVELOPE_MESSAGE_KEYS:+ value = body.get(key)+ if isinstance(value, str) and value:+ return value+ return fallback+++class TikHubCredential(ApiKeyCredential):+ """TikHub (api.tikhub.io) pay-per-request social scraping API.++ The probe hits the authenticated user-info route rather than+ ``/api/v1/health/check`` (which answers without a key). A 401 surfaces+ through the base ``raise_for_status`` path; a 2xx is inspected for the+ in-body ``code`` TikHub uses for API-level failures.+ """++ id = "tikhub"+ display_name = "TikHub"+ category = "Scrapers"+ key_name = "Authorization"+ key_location = "bearer"+ docs_url = "https://docs.tikhub.io"+ probe_url = "https://api.tikhub.io/api/v1/tikhub/user/get_user_info"++ @classmethod+ def _handle_probe_response(cls, response: httpx.Response) -> ProbeResult:+ try:+ body: Any = response.json()+ except ValueError:+ body = None+ if not isinstance(body, dict):+ return ProbeResult(valid=True, message="TikHub key validated")++ code = body.get("code")+ if isinstance(code, int) and not isinstance(code, bool) and code >= 400:+ return ProbeResult(+ valid=False,+ message=f"TikHub rejected the API key: {_envelope_message(body, f'code {code}')}",+ )++ # Field names come from the CLI's `user info` output; the envelope+ # may carry them at the top level or under ``data``. Tolerate both+ # and tolerate absence — a missing field must not fail validation.+ data = body.get("data") if isinstance(body.get("data"), dict) else {}++ def section(name: str) -> Dict[str, Any]:+ for source in (body, data):+ value = source.get(name)+ if isinstance(value, dict):+ return value+ return {}++ user = section("user_data")+ key_data = section("api_key_data")+ email: Optional[str] = user.get("email")+ balance = user.get("balance")+ free_credit = user.get("free_credit")+ api_key_name = key_data.get("api_key_name")++ who = email or api_key_name or "account"+ summary = f"TikHub key validated — {who}"+ if balance is not None:+ summary += f" (balance ${balance})"+ return ProbeResult(+ valid=True,+ message=summary,+ extra={+ "email": email,+ "balance": balance,+ "free_credit": free_credit,+ "api_key_name": api_key_name,+ },+ )diff --git a/server/nodes/scraper/tikhub.svg b/server/nodes/scraper/tikhub.svgnew file mode 100644index 00000000..6140e456--- /dev/null+++ b/server/nodes/scraper/tikhub.svg@@ -0,0 +1,5 @@+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none">+ <rect x="2" y="2" width="20" height="20" rx="5" fill="#ff79c6"/>+ <path d="M6.5 7h9v2.4h-3.2V17h-2.6V9.4H6.5z" fill="#ffffff"/>+ <path d="M15.2 11.2l4.3 2.6-4.3 2.6z" fill="#ffffff"/>+</svg>diff --git a/server/nodes/scraper/tikhub_action/__init__.py b/server/nodes/scraper/tikhub_action/__init__.pynew file mode 100644index 00000000..570eb180--- /dev/null+++ b/server/nodes/scraper/tikhub_action/__init__.py@@ -0,0 +1,254 @@+"""TikHub — the ``tikhubAction`` node, an SDK-backed social scraper "flattened like a CLI".++TikHub (https://api.tikhub.io) is a pay-per-request REST API over ~1000+scraping endpoints for TikTok, Douyin, Instagram, YouTube, Twitter/X,+Xiaohongshu, Bilibili, Kuaishou, Weibo, Reddit, Threads, LinkedIn and more.+Rather than one node per platform, this node exposes the SDK's own+``resource.method`` ids: ``list_endpoints`` discovers them (no network),+``call`` invokes one with keyword arguments, ``fetch_url`` is the hybrid+share-URL parser, and ``account`` reports balance and daily usage.++This module must not import ``tikhub`` at import time — every SDK touch+lives in :mod:`._sdk`, lazily, so the plugin registers and the credentials+modal validates even when the SDK is absent.+"""++from __future__ import annotations++from typing import Any, Dict, List, Literal, Optional++from pydantic import BaseModel, ConfigDict, Field, model_validator++from core.logging import get_logger+from services.node_output_schemas import register_output_schema+from services.plugin import ActionNode, NodeContext, NodeUserError, Operation, TaskQueue, coerce_blank_params+from services.ws_handler_registry import register_option_loader++from .._credentials import TikHubCredential+from ._sdk import (+ PLATFORMS,+ call_endpoint,+ list_endpoints as sdk_list_endpoints,+ load_tikhub_endpoints,+ make_client,+ resolve_endpoint,+ track_tikhub_usage,+)++logger = get_logger(__name__)++_CALL = {"displayOptions": {"show": {"operation": ["call"]}}}+_LIST = {"displayOptions": {"show": {"operation": ["list_endpoints"]}}}+_CALL_OR_LIST = {"displayOptions": {"show": {"operation": ["call", "list_endpoints"]}}}+_FETCH_URL = {"displayOptions": {"show": {"operation": ["fetch_url"]}}}++_FETCH_URL_ENDPOINT = "hybrid_parsing.video_data"+_USER_INFO_ENDPOINT = "tikhub_user.get_user_info"+_USER_USAGE_ENDPOINT = "tikhub_user.get_user_daily_usage"+++class TikHubActionParams(BaseModel):+ operation: Literal["call", "list_endpoints", "fetch_url", "account"] = "call"++ platform: Literal[PLATFORMS] = Field(+ default="all",+ description="Platform filter for the endpoint dropdown and for list_endpoints.",+ json_schema_extra=_CALL_OR_LIST,+ )+ # ``str`` rather than a Literal on purpose: the LLM may pass any id the+ # SDK knows, and the dropdown is only a convenience over the same string.+ endpoint: str = Field(+ default="",+ description="Endpoint id as <resource>.<method> (a /api/v1/... path or full API URL also works).",+ json_schema_extra={+ "placeholder": "douyin_web.fetch_one_video",+ "dynamicOptions": True,+ "loadOptionsMethod": "tikhubEndpoints",+ "loadOptionsDependsOn": ["platform"],+ **_CALL,+ },+ )+ params: Dict[str, Any] = Field(+ default_factory=dict,+ description="Keyword arguments for the endpoint as a JSON object (see list_endpoints for names). Batch POST endpoints take {\"body\": [...]}.",+ json_schema_extra={"editor": "json", "rows": 6, **_CALL},+ )++ search: str = Field(+ default="",+ description="Case-insensitive substring over endpoint id, summary and route.",+ json_schema_extra={"placeholder": "one_video", **_LIST},+ )+ limit: int = Field(default=100, ge=1, le=2000, json_schema_extra=_LIST)++ url: str = Field(+ default="",+ description="Share URL of a post/video to parse (TikTok, Douyin, Instagram, ...).",+ json_schema_extra={"placeholder": "https://www.tiktok.com/@user/video/123", **_FETCH_URL},+ )+ minimal: bool = Field(+ default=False,+ description="Return the trimmed payload instead of the full platform response.",+ json_schema_extra=_FETCH_URL,+ )++ model_config = ConfigDict(extra="ignore")++ @model_validator(mode="before")+ @classmethod+ def _coerce(cls, values: Any) -> Any:+ # The panel stores "" for cleared fields and renders ``params`` as+ # text; an LLM may also stringify the JSON object.+ return coerce_blank_params(cls, values, object_fields=("params",))+++class TikHubActionOutput(BaseModel):+ operation: Optional[str] = None+ endpoint: Optional[str] = None+ path: Optional[str] = None+ url: Optional[str] = None+ data: Optional[Any] = None+ code: Optional[Any] = None+ router: Optional[str] = None+ platform: Optional[str] = None+ endpoints: Optional[List[Dict[str, Any]]] = None+ count: Optional[int] = None+ total: Optional[int] = None+ account: Optional[Any] = None+ usage: Optional[Any] = None+ cost_usd: Optional[float] = None++ model_config = ConfigDict(extra="allow")+++class TikHubActionNode(ActionNode):+ type = "tikhubAction"+ display_name = "TikHub"+ subtitle = "Social Scraper API"+ group = ("scraper", "tool")+ description = (+ "Scrape TikTok, Douyin, Instagram, YouTube, Twitter/X, Xiaohongshu, Bilibili and more "+ "through TikHub's ~1000-endpoint API — discover endpoints, call any of them, parse a share URL, or check the account"+ )+ component_kind = "square"+ tool_name = "tikhub_action"+ tool_description = (+ "Scrape public social-media data through TikHub (api.tikhub.io): TikTok, Douyin, Instagram, "+ "YouTube, Twitter/X, Xiaohongshu, Bilibili, Kuaishou, Weibo, Reddit, Threads, LinkedIn, Zhihu "+ "and more (~1000 endpoints). Two-step flow: (1) operation=list_endpoints with platform (e.g. "+ "'tiktok') and an optional search term to discover endpoint ids, their HTTP route and accepted "+ "params; (2) operation=call with endpoint='<resource>.<method>' and params as a JSON object of "+ "keyword arguments. Examples: douyin_web.fetch_one_video {\"aweme_id\": \"...\"}; "+ "twitter_web.fetch_search_timeline {\"keyword\": \"...\"}; hybrid_parsing.video_data "+ "{\"url\": \"<share url>\"} (or simply operation=fetch_url with url=). Batch POST endpoints take "+ "one 'body' list param. operation=account returns the TikHub account (email, balance) and daily "+ "usage (tikhub_user resource). Results return as data (unwrapped from TikHub's envelope) plus "+ "code/router. Each successful call costs about $0.001 and every endpoint allows ~10 requests/s. "+ "Needs a TikHub API key in Credentials → TikHub."+ )+ handles = (+ {"name": "input-main", "kind": "input", "position": "left", "label": "Input", "role": "main"},+ {"name": "output-main", "kind": "output", "position": "right", "label": "Output", "role": "main"},+ )+ annotations = {"destructive": False, "readonly": True, "open_world": True}+ credentials = (TikHubCredential,)+ task_queue = TaskQueue.REST_API+ usable_as_tool = True+ # usable_as_tool auto-hides both handles unless declared; this node+ # must stay wirable on the canvas as well as callable as a tool.+ hide_input_handle = False+ hide_output_handle = False++ Params = TikHubActionParams+ Output = TikHubActionOutput++ @staticmethod+ def _shape(operation: str, **fields: Any) -> Dict[str, Any]:+ """``{"operation": ...}`` plus every kwarg that is not ``None``.++ Only ``None`` is dropped — an empty ``{}`` / ``[]`` ``data`` is a real+ answer from TikHub and must reach the panel.+ """+ shaped: Dict[str, Any] = {"operation": operation}+ shaped.update({k: v for k, v in fields.items() if v is not None})+ return shaped++ # ---- operations ------------------------------------------------------++ @Operation("call", cost={"service": "tikhub", "action": "call", "count": 1})+ async def call(self, ctx: NodeContext, params: TikHubActionParams) -> Any:+ endpoint = params.endpoint.strip()+ if not endpoint:+ raise NodeUserError(+ "endpoint is required (e.g. douyin_web.fetch_one_video). "+ "Use operation=list_endpoints to discover endpoint ids."+ )+ info = await resolve_endpoint(endpoint)+ async with make_client(ctx) as client:+ result = await call_endpoint(client, info, params.params)+ cost = await track_tikhub_usage(ctx, "call", info.path)+ return self._shape(+ "call",+ endpoint=info.endpoint,+ path=info.path,+ data=result["data"],+ code=result["code"],+ router=result["router"],+ cost_usd=cost,+ )++ @Operation("fetch_url", cost={"service": "tikhub", "action": "fetch_url", "count": 1})+ async def fetch_url(self, ctx: NodeContext, params: TikHubActionParams) -> Any:+ url = params.url.strip()+ if not url:+ raise NodeUserError("url is required (a share URL of the post or video to parse)")+ info = await resolve_endpoint(_FETCH_URL_ENDPOINT)+ kwargs: Dict[str, Any] = {"url": url}+ if params.minimal:+ kwargs["minimal"] = True+ async with make_client(ctx) as client:+ result = await call_endpoint(client, info, kwargs)+ cost = await track_tikhub_usage(ctx, "fetch_url", info.path)+ return self._shape(+ "fetch_url",+ url=url,+ endpoint=info.endpoint,+ path=info.path,+ data=result["data"],+ code=result["code"],+ router=result["router"],+ cost_usd=cost,+ )++ @Operation("list_endpoints")+ async def list_endpoints(self, ctx: NodeContext, params: TikHubActionParams) -> Any:+ # Pure introspection — no network, no billing, no tracking.+ matches, total = await sdk_list_endpoints(platform=params.platform, search=params.search, limit=params.limit)+ return self._shape(+ "list_endpoints",+ platform=params.platform,+ count=len(matches),+ total=total,+ endpoints=[m.as_dict() for m in matches],+ )++ @Operation("account", cost={"service": "tikhub", "action": "account", "count": 1})+ async def account(self, ctx: NodeContext, params: TikHubActionParams) -> Any:+ info = await resolve_endpoint(_USER_INFO_ENDPOINT)+ usage: Any = None+ async with make_client(ctx) as client:+ account = (await call_endpoint(client, info, {}))["data"]+ try:+ usage_info = await resolve_endpoint(_USER_USAGE_ENDPOINT)+ usage = (await call_endpoint(client, usage_info, {}))["data"]+ except NodeUserError as exc:+ # Balance is the answer that matters; usage is a bonus.+ logger.warning("tikhub daily usage unavailable", error=str(exc))+ cost = await track_tikhub_usage(ctx, "account", info.path)+ return self._shape("account", account=account, usage=usage, cost_usd=cost)+++register_option_loader("tikhubEndpoints", load_tikhub_endpoints)+register_output_schema(TikHubActionNode.type, TikHubActionOutput)++__all__ = ["TikHubActionNode", "TikHubActionOutput", "TikHubActionParams"]diff --git a/server/nodes/scraper/tikhub_action/_sdk.py b/server/nodes/scraper/tikhub_action/_sdk.pynew file mode 100644index 00000000..10d8f797--- /dev/null+++ b/server/nodes/scraper/tikhub_action/_sdk.py@@ -0,0 +1,565 @@+"""Everything that touches the ``tikhub`` SDK, imported lazily.++The node in ``__init__.py`` never imports ``tikhub`` at module level — the+plugin must register (and the credentials modal must keep working) even when+the SDK is missing. Every function here that needs the SDK imports it inside+its body.++The SDK is auto-generated from TikHub's OpenAPI spec: ``client.<resource>.+<method>(**kwargs)`` is ``GET|POST /api/v1/<platform>/<api>/<action>``, all+parameters are keyword-only, and each method's docstring carries the route+literal on its last line. That convention is what :func:`endpoint_index`+introspects — there is no vendored endpoint list, so an SDK upgrade is the+only thing needed to pick up new routes.+"""++from __future__ import annotations++import asyncio+import contextlib+import difflib+import inspect+import json+import re+from dataclasses import asdict, dataclass+from typing import Any, AsyncIterator, Dict, List, Optional, Tuple+from urllib.parse import urlparse++from core.logging import get_logger+from services.plugin import NodeContext, NodeUserError++from .._credentials import TikHubCredential++logger = get_logger(__name__)++# Per-request timeout handed to the SDK's httpx client.+_TIMEOUT = 30.0+# ``RetryPolicy.should_retry`` returns False once ``attempt >= max_retries``,+# so 3 means one initial attempt plus two retries (connection / 5xx / 429+# only). Tests monkeypatch this to 0 to skip the anyio backoff sleeps.+_MAX_RETRIES = 3+# Any non-empty key works for introspection: the throwaway client never+# sends a request, it only exists so ``vars(client)`` lists the resources.+_INTROSPECTION_KEY = "introspection"+_BODY_PREVIEW_CHARS = 500+_MAX_SUGGESTIONS = 5+_ROUTE_RE = re.compile(r"``(GET|POST|PUT|PATCH|DELETE)\s+(/\S+?)``")+_SOURCE_ROUTE_RE = re.compile(r"_request\(\s*['\"](GET|POST|PUT|PATCH|DELETE)['\"]\s*,\s*['\"](/[^'\"]+)['\"]")++# Static platform list for the ``platform`` dropdown. Filter rule is+# ``info.platform == platform`` where platform is the first ``_`` segment of+# the SDK resource name (``tiktok_app_v3`` -> ``tiktok``). A test asserts every+# value except ``all`` matches at least one live resource so SDK drift fails+# loudly instead of silently emptying a dropdown.+PLATFORMS: Tuple[str, ...] = (+ "all",+ "tiktok",+ "douyin",+ "instagram",+ "youtube",+ "twitter",+ "xiaohongshu",+ "bilibili",+ "kuaishou",+ "weibo",+ "reddit",+ "threads",+ "linkedin",+ "zhihu",+ "toutiao",+ "xigua",+ "wechat",+ "lemon8",+ "pipixia",+ "hybrid",+ "tikhub",+)+++@dataclass(frozen=True)+class ParamInfo:+ name: str+ type: str+ required: bool+++@dataclass(frozen=True)+class EndpointInfo:+ endpoint: str # "douyin_web.fetch_one_video" — the SDK's own id+ resource: str # "douyin_web"+ method: str # "fetch_one_video"+ platform: str # "douyin"+ http_method: Optional[str] # "GET" / "POST" (None when undiscoverable)+ path: Optional[str] # "/api/v1/douyin/web/fetch_one_video"+ summary: str # English half of the docstring's first line+ params: Tuple[ParamInfo, ...]++ def as_dict(self) -> Dict[str, Any]:+ return asdict(self) | {"params": [asdict(p) for p in self.params]}+++_INDEX: Optional[List[EndpointInfo]] = None+_BY_ENDPOINT: Dict[str, EndpointInfo] = {}+_BY_PATH: Dict[str, EndpointInfo] = {}+_INDEX_LOCK = asyncio.Lock()+++# ---------------------------------------------------------------------------+# Introspection+# ---------------------------------------------------------------------------+++def platform_of(resource: str) -> str:+ """``tiktok_app_v3`` -> ``tiktok``; ``hybrid_parsing`` -> ``hybrid``."""+ return resource.split("_")[0]+++def _is_resource(obj: Any) -> bool:+ try:+ from tikhub.resources._base import AsyncResource++ if isinstance(obj, AsyncResource):+ return True+ except ImportError:+ pass+ module = getattr(type(obj), "__module__", "") or ""+ return module.startswith("tikhub.resources")+++def _annotation_text(param: inspect.Parameter) -> str:+ ann = param.annotation+ if ann is inspect.Parameter.empty:+ return "Any"+ # Under ``from __future__ import annotations`` these are already strings.+ return ann if isinstance(ann, str) else getattr(ann, "__name__", repr(ann))+++def _route_of(fn: Any) -> Tuple[Optional[str], Optional[str]]:+ doc = inspect.getdoc(fn) or ""+ match = _ROUTE_RE.search(doc)+ if match:+ return match.group(1), match.group(2)+ try:+ match = _SOURCE_ROUTE_RE.search(inspect.getsource(fn))+ except (OSError, TypeError):+ match = None+ if match:+ return match.group(1), match.group(2)+ return None, None+++def _summary_of(fn: Any) -> str:+ doc = inspect.getdoc(fn) or ""+ first = doc.strip().splitlines()[0].strip() if doc.strip() else ""+ # Generated docstrings read "<Chinese>/<English>"; keep the English half.+ if "/" in first:+ english = first.split("/", 1)[1].strip()+ if english:+ return english+ return first+++def _describe(resource_name: str, method_name: str, fn: Any) -> EndpointInfo:+ params: List[ParamInfo] = []+ try:+ signature = inspect.signature(fn)+ except (TypeError, ValueError):+ signature = None+ if signature is not None:+ for p in signature.parameters.values():+ if p.kind is not inspect.Parameter.KEYWORD_ONLY:+ continue+ params.append(+ ParamInfo(+ name=p.name,+ type=_annotation_text(p),+ required=p.default is inspect.Parameter.empty,+ )+ )+ http_method, path = _route_of(fn)+ return EndpointInfo(+ endpoint=f"{resource_name}.{method_name}",+ resource=resource_name,+ method=method_name,+ platform=platform_of(resource_name),+ http_method=http_method,+ path=path,+ summary=_summary_of(fn),+ params=tuple(params),+ )+++def _build_index() -> List[EndpointInfo]:+ """Reflect every resource method on a throwaway client. No network."""+ from tikhub import AsyncTikHub++ client = AsyncTikHub(api_key=_INTROSPECTION_KEY, max_retries=0)+ entries: List[EndpointInfo] = []+ for resource_name, resource in vars(client).items():+ if resource_name.startswith("_") or not _is_resource(resource):+ continue+ for method_name, fn in inspect.getmembers(type(resource), inspect.iscoroutinefunction):+ if method_name.startswith("_"):+ continue+ entries.append(_describe(resource_name, method_name, fn))+ entries.sort(key=lambda e: e.endpoint)+ return entries+++async def endpoint_index() -> List[EndpointInfo]:+ """Cached, process-wide endpoint catalogue (built once, ~1000 entries)."""+ global _INDEX+ if _INDEX is not None:+ return _INDEX+ async with _INDEX_LOCK:+ if _INDEX is None:+ entries = await asyncio.to_thread(_build_index)+ _BY_ENDPOINT.clear()+ _BY_PATH.clear()+ for entry in entries:+ _BY_ENDPOINT[entry.endpoint] = entry+ if entry.path:+ _BY_PATH.setdefault(entry.path, entry)+ _INDEX = entries+ logger.info("tikhub endpoint index built", endpoints=len(entries))+ return _INDEX+++async def list_endpoints(+ platform: str = "all",+ search: str = "",+ limit: Optional[int] = None,+) -> Tuple[List[EndpointInfo], int]:+ """Filter the index by platform and a case-insensitive substring.++ Returns ``(matches, total)`` where ``total`` counts matches before+ ``limit`` is applied, so a caller can tell the list was cut.+ """+ index = await endpoint_index()+ needle = (search or "").strip().lower()+ matches = [+ e+ for e in index+ if (platform in ("", "all") or e.platform == platform)+ and (not needle or needle in e.endpoint.lower() or needle in e.summary.lower() or needle in (e.path or "").lower())+ ]+ total = len(matches)+ if limit is not None and limit >= 0:+ matches = matches[:limit]+ return matches, total+++# ---------------------------------------------------------------------------+# Resolution + argument binding+# ---------------------------------------------------------------------------+++def _normalise_key(raw: str) -> Tuple[Optional[str], List[str]]:+ """Return ``(dotted_id, candidate_paths)`` for a user-supplied endpoint.++ Accepts ``resource.method``, ``resource/method``, an ``/api/v1/...`` path+ (with or without the leading slash / prefix) or a full URL.+ """+ key = raw.strip()+ if key.startswith(("http://", "https://")):+ key = urlparse(key).path+ key = key.rstrip("/")+ if key.startswith("/"):+ return None, [key, f"/api/v1{key}" if not key.startswith("/api/") else key]+ if key.count("/") == 1:+ return key.replace("/", "."), [f"/{key}", f"/api/v1/{key}"]+ if "/" in key:+ return None, [f"/{key}", f"/api/v1/{key}"]+ return key, []+++async def resolve_endpoint(raw: str) -> EndpointInfo:+ """Map any accepted spelling of an endpoint to its :class:`EndpointInfo`."""+ if not (raw or "").strip():+ raise NodeUserError(+ "endpoint is required (e.g. douyin_web.fetch_one_video). "+ "Use operation=list_endpoints to discover endpoint ids."+ )+ await endpoint_index()+ dotted, paths = _normalise_key(raw)+ if dotted and dotted in _BY_ENDPOINT:+ return _BY_ENDPOINT[dotted]+ for path in paths:+ if path in _BY_PATH:+ return _BY_PATH[path]++ probe = dotted or raw.strip()+ suggestions = difflib.get_close_matches(probe, list(_BY_ENDPOINT), n=_MAX_SUGGESTIONS, cutoff=0.5)+ if not suggestions and "." not in probe:+ # A bare method name: offer every resource that has it.+ tail = "." + probe.rsplit("/", 1)[-1]+ suggestions = [e for e in _BY_ENDPOINT if e.endswith(tail)][:_MAX_SUGGESTIONS]+ hint = f" Did you mean: {', '.join(suggestions)}?" if suggestions else ""+ raise NodeUserError(+ f"Unknown TikHub endpoint '{raw.strip()}'.{hint} "+ "Use operation=list_endpoints (with platform and search) to discover endpoint ids."+ )+++def _accepted_params(info: EndpointInfo) -> str:+ if not info.params:+ return "none (this endpoint takes no parameters)"+ return ", ".join(f"{p.name} ({p.type}, {'required' if p.required else 'optional'})" for p in info.params)+++def bind_params(info: EndpointInfo, fn: Any, params: Any) -> Dict[str, Any]:+ """Validate ``params`` against the SDK method signature before any HTTP.++ A wrong or missing keyword is the most common LLM mistake; rejecting it+ here (with the accepted list) beats a 422 from TikHub that was already+ billed as a request.+ """+ if params is None:+ params = {}+ if not isinstance(params, dict):+ raise NodeUserError(+ f"params for {info.endpoint} must be a JSON object of keyword arguments. Accepted: {_accepted_params(info)}"+ )+ if any(not isinstance(k, str) for k in params):+ raise NodeUserError(f"params for {info.endpoint} must use string keys. Accepted: {_accepted_params(info)}")+ try:+ bound = inspect.signature(fn).bind(**params)+ except TypeError as exc:+ raise NodeUserError(f"{info.endpoint} rejected params: {exc}. Accepted: {_accepted_params(info)}") from exc+ return dict(bound.arguments)+++# ---------------------------------------------------------------------------+# Response + error shaping+# ---------------------------------------------------------------------------+++def _preview(value: Any) -> str:+ if value is None:+ return ""+ if isinstance(value, str):+ text = value+ else:+ try:+ text = json.dumps(value, ensure_ascii=False, default=str)+ except (TypeError, ValueError):+ text = str(value)+ return text if len(text) <= _BODY_PREVIEW_CHARS else text[:_BODY_PREVIEW_CHARS] + "…"+++def _envelope_message(body: Any, fallback: str) -> str:+ if isinstance(body, dict):+ for key in ("message", "error", "detail", "msg"):+ value = body.get(key)+ if isinstance(value, str) and value:+ return value+ return fallback+++def to_plain(result: Any) -> Dict[str, Any]:+ """Unwrap TikHub's ``{code, router, params, data}`` envelope.++ The SDK returns ``response.json()`` verbatim and never looks at the+ in-body ``code``, so an API-level failure arrives here as a 2xx dict —+ treat an int ``code >= 400`` as a user error. A body without a ``data``+ key (or a non-JSON body) is passed through whole under ``data``.+ """+ if hasattr(result, "model_dump"):+ result = result.model_dump(mode="json")+ if not isinstance(result, dict):+ return {"data": result, "code": None, "router": None}+ code = result.get("code")+ if isinstance(code, int) and not isinstance(code, bool) and code >= 400:+ raise NodeUserError(f"TikHub returned code {code}: {_envelope_message(result, 'request failed')}")+ data = result["data"] if "data" in result else result+ return {"data": data, "code": code, "router": result.get("router")}+++def raise_user_error(exc: BaseException, endpoint: str) -> None:+ """Translate an SDK exception into a ``NodeUserError``.++ Only ``TikHubError`` subclasses are mapped; anything else propagates so+ genuine bugs keep their traceback. Always raises.+ """+ from tikhub import (+ TikHubAuthError,+ TikHubBadRequestError,+ TikHubConfigError,+ TikHubConnectionError,+ TikHubFeatureRemovedError,+ TikHubHTTPError,+ TikHubNotFoundError,+ TikHubPermissionError,+ TikHubRateLimitError,+ TikHubServerError,+ TikHubUpstreamError,+ TikHubValidationError,+ )++ status = getattr(exc, "status_code", None)+ detail = exc.args[0] if exc.args else str(exc)+ body = _preview(getattr(exc, "response_body", None))++ if isinstance(exc, TikHubAuthError):+ message = f"TikHub rejected the API key (401) while calling {endpoint}. Update it in Credentials → TikHub."+ elif isinstance(exc, TikHubPermissionError):+ message = (+ f"TikHub refused {endpoint} (403): {detail}. The key may lack access to this endpoint "+ "or the account balance may be exhausted — run operation=account to check."+ )+ elif isinstance(exc, TikHubRateLimitError):+ retry_after = getattr(exc, "retry_after", None)+ wait = f"; retry after {retry_after:g}s" if isinstance(retry_after, (int, float)) else ""+ message = f"TikHub rate-limited {endpoint} (429){wait}. Each endpoint allows about 10 requests/s."+ elif isinstance(exc, TikHubNotFoundError):+ message = (+ f"TikHub returned 404 for {endpoint}: {detail}. The route or the requested resource does not "+ "exist — check the endpoint id with operation=list_endpoints."+ )+ elif isinstance(exc, TikHubBadRequestError):+ message = f"TikHub rejected the request to {endpoint} ({status}): {detail}."+ if body:+ message += f" Response: {body}"+ elif isinstance(exc, TikHubValidationError):+ raw = _preview(getattr(exc, "raw", None))+ message = f"TikHub returned an unparseable response for {endpoint}: {detail}."+ if raw:+ message += f" Raw: {raw}"+ elif isinstance(exc, (TikHubUpstreamError, TikHubServerError, TikHubConnectionError)):+ where = f" (HTTP {status})" if status else ""+ message = f"TikHub upstream error while calling {endpoint}{where}: {detail}. Retries were exhausted — try again later."+ elif isinstance(exc, (TikHubFeatureRemovedError, TikHubConfigError)):+ message = str(exc)+ elif isinstance(exc, TikHubHTTPError):+ message = f"TikHub returned HTTP {status} for {endpoint}: {detail}."+ if body:+ message += f" Response: {body}"+ else:+ raise exc++ request_id = getattr(exc, "request_id", None)+ if request_id:+ message += f" [request_id={request_id}]"+ raise NodeUserError(message) from exc+++# ---------------------------------------------------------------------------+# Client + dispatch+# ---------------------------------------------------------------------------+++async def make_client(ctx: NodeContext) -> AsyncIterator[Any]:+ """Yield an authenticated ``AsyncTikHub`` for one operation.++ The key is resolved FIRST so a missing credential raises the annotated+ ``PermissionError`` untouched — ``BaseNode.execute`` turns that into the+ ``PermissionDeniedError`` envelope plus the ``credential.api_key.+ runtime_failed`` broadcast. The key is always passed explicitly so a+ ``$TIKHUB_API_KEY`` in the server environment can never authenticate a+ node implicitly.+ """+ secrets = await TikHubCredential.resolve(user_id=ctx.credential_customer_id)+ try:+ from tikhub import AsyncTikHub+ except ImportError as exc:+ raise NodeUserError("The 'tikhub' package is not installed — run `uv sync` in server/.") from exc+ async with AsyncTikHub(api_key=secrets["api_key"], timeout=_TIMEOUT, max_retries=_MAX_RETRIES) as client:+ yield client+++async def call_endpoint(client: Any, info: EndpointInfo, params: Any) -> Dict[str, Any]:+ """``getattr`` dispatch + bind + typed-error mapping + envelope unwrap."""+ from tikhub import TikHubError++ resource = getattr(client, info.resource, None)+ fn = getattr(resource, info.method, None) if resource is not None else None+ if fn is None:+ raise NodeUserError(f"{info.endpoint} is not available in the installed tikhub SDK. Run operation=list_endpoints.")+ kwargs = bind_params(info, fn, params)+ try:+ raw = await fn(**kwargs)+ except TikHubError as exc:+ raise_user_error(exc, info.endpoint)+ return to_plain(raw)+++# ---------------------------------------------------------------------------+# Usage tracking + dropdown loader+# ---------------------------------------------------------------------------+++async def track_tikhub_usage(ctx: NodeContext, action: str, endpoint_path: Optional[str]) -> float:+ """Record one billed TikHub request in ``api_usage_metrics``.++ Called only after a successful call (non-2xx is not billed by TikHub)+ and never for ``list_endpoints``. Best-effort: a tracking failure logs a+ warning and returns 0.0 rather than failing a call that already cost+ money. Returns the computed USD cost for the ``cost_usd`` output key.+ """+ try:+ from services.plugin.deps import get_database+ from services.pricing import get_pricing_service++ cost_data = get_pricing_service().calculate_api_cost("tikhub", action, 1)+ raw = ctx.raw if isinstance(ctx.raw, dict) else {}+ await get_database().save_api_usage_metric(+ {+ "session_id": ctx.session_id or raw.get("session_id", "default"),+ "node_id": ctx.node_id,+ "workflow_id": ctx.workflow_id or raw.get("workflow_id"),+ "service": "tikhub",+ "operation": cost_data.get("operation", action),+ "endpoint": endpoint_path or action,+ "resource_count": 1,+ "cost": cost_data.get("total_cost", 0.0),+ }+ )+ return float(cost_data.get("total_cost", 0.0) or 0.0)+ except Exception as exc: # noqa: BLE001 — tracking must never fail a billed call+ logger.warning("tikhub usage tracking failed", action=action, error=str(exc))+ return 0.0+++async def load_tikhub_endpoints(params: Dict[str, Any]) -> List[Dict[str, Any]]:+ """``loadOptionsMethod`` for the ``endpoint`` dropdown.++ ``all`` deliberately returns no options — ~1000 items is not a usable+ dropdown, and the LLM path accepts any id as free text anyway. The+ panel prompts to pick a platform first. A missing SDK must never break+ discovery, so failures degrade to an empty list with a warning.+ """+ platform = str((params or {}).get("platform") or "all")+ if platform == "all":+ return []+ try:+ matches, _ = await list_endpoints(platform=platform)+ except Exception as exc: # noqa: BLE001 — SDK missing / introspection failure+ logger.warning("tikhub endpoint loader failed", platform=platform, error=str(exc))+ return []+ return [+ {+ "value": e.endpoint,+ "label": e.endpoint,+ "description": f"{e.summary} — {e.http_method or '?'} {e.path or ''}".strip(),+ }+ for e in matches+ ]+++__all__ = [+ "PLATFORMS",+ "EndpointInfo",+ "ParamInfo",+ "bind_params",+ "call_endpoint",+ "endpoint_index",+ "list_endpoints",+ "load_tikhub_endpoints",+ "make_client",+ "platform_of",+ "raise_user_error",+ "resolve_endpoint",+ "to_plain",+ "track_tikhub_usage",+]diff --git a/server/nodes/scraper/tikhub_action/icon.svg b/server/nodes/scraper/tikhub_action/icon.svgnew file mode 100644index 00000000..6140e456--- /dev/null+++ b/server/nodes/scraper/tikhub_action/icon.svg@@ -0,0 +1,5 @@+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none">+ <rect x="2" y="2" width="20" height="20" rx="5" fill="#ff79c6"/>+ <path d="M6.5 7h9v2.4h-3.2V17h-2.6V9.4H6.5z" fill="#ffffff"/>+ <path d="M15.2 11.2l4.3 2.6-4.3 2.6z" fill="#ffffff"/>+</svg>diff --git a/server/nodes/scraper/tikhub_action/meta.json b/server/nodes/scraper/tikhub_action/meta.jsonnew file mode 100644index 00000000..02961037--- /dev/null+++ b/server/nodes/scraper/tikhub_action/meta.json@@ -0,0 +1,3 @@+{+ "color": "#ff79c6"+}diff --git a/server/nodes/visuals.json b/server/nodes/visuals.jsonindex 1256c033..cd6101c4 100644--- a/server/nodes/visuals.json+++ b/server/nodes/visuals.json@@ -21,6 +21,9 @@"apifyActor": {"skill": "apify-skill"},+ "tikhubAction": {+ "skill": "tikhub-skill"+ },"appLauncher": {"icon": "🚀","skill": "app-launcher-skill"diff --git a/server/pyproject.toml b/server/pyproject.tomlindex 23da2ff3..b85b10f3 100644--- a/server/pyproject.toml+++ b/server/pyproject.toml@@ -100,6 +100,11 @@ dependencies = [# Apify Web Scraping"apify-client>=1.8.0",+ # TikHub social scraping SDK. Auto-generated from TikHub's openapi.json;+ # capped below 3 because the resource/method surface regenerates per+ # release. Pulls typer for its own `tikhub` CLI (not imported by us).+ "tikhub>=2.1.2,<3",+# Crawlee Web Scraping"crawlee[beautifulsoup]>=1.4.0","html2text>=2024.2.26",diff --git a/server/skills/GUIDE.md b/server/skills/GUIDE.mdindex f1021ffb..537d9a2d 100644--- a/server/skills/GUIDE.md+++ b/server/skills/GUIDE.md@@ -40,7 +40,7 @@ server/skills/├── vertex_agent/ # vertex-agent-skill, vertex-agent-admin-skill└── web_agent/ # Web automation skillsbrowser-skill, browser-harness-skill, http-request-skill, apify-skill,- crawlee-scraper-skill, proxy-config-skill, duckduckgo-search-skill,+ crawlee-scraper-skill, tikhub-skill, proxy-config-skill, duckduckgo-search-skill,brave-search-skill, serper-search-skill, perplexity-search-skill```diff --git a/server/skills/web_agent/tikhub-skill/SKILL.md b/server/skills/web_agent/tikhub-skill/SKILL.mdnew file mode 100644index 00000000..a920b5cf--- /dev/null+++ b/server/skills/web_agent/tikhub-skill/SKILL.md@@ -0,0 +1,355 @@+---+name: tikhub-skill+description: Scrape TikTok, Douyin, Instagram, YouTube, Twitter/X, Xiaohongshu, Bilibili, Kuaishou, Weibo, Reddit, Threads, LinkedIn and more through the TikHub pay-per-request API - discover endpoints, call them by id, parse any share URL, and check account balance.+allowed-tools: "tikhub_action"+metadata:+ author: opencompany+ version: "1.0"+ category: web++---++# TikHub Skill++TikHub (api.tikhub.io) is a pay-per-request REST API that exposes roughly+1,000 scraping endpoints across TikTok, Douyin, Instagram, YouTube,+Twitter/X, Xiaohongshu, Bilibili, Kuaishou, Weibo, Reddit, Threads,+LinkedIn, Zhihu, Lemon8 and others. Each endpoint is one HTTP call that+returns the platform's raw data; there are no long-running jobs, no+actors, no datasets to poll. The `tikhub_action` tool is a thin, flattened+wrapper over the official `tikhub` Python SDK: every SDK method is+addressable by its resource-dot-method id (for example+`douyin_web.fetch_one_video`), and the tool passes your `params` straight+through as the SDK's keyword arguments.++## Tool: tikhub_action++### Operations++| Operation | Purpose | Key fields |+|---|---|---|+| `list_endpoints` | Discover endpoints (no API call, free) - returns each endpoint's id, HTTP method, path, summary and typed params with `required` flags | `platform` (default `all`), `search` (substring over id / summary / path), `limit` |+| `call` | Invoke one endpoint by id | `endpoint` (resource-dot-method id or a `/api/v1/...` path), `params` (JSON object of the SDK keyword args) |+| `fetch_url` | Parse any TikTok / Douyin share URL through `hybrid_parsing.video_data` | `url`, `minimal` (trimmed payload) |+| `account` | Balance, free credit and today's usage for the stored key | - |++### Response++`call` and `fetch_url`:++```json+{+ "operation": "call",+ "endpoint": "douyin_web.fetch_one_video",+ "path": "/api/v1/douyin/web/fetch_one_video",+ "data": { "aweme_detail": { "aweme_id": "7372484719365098803", "desc": "...", "statistics": { "digg_count": 12345 } } },+ "code": 200,+ "router": "/api/v1/douyin/web/fetch_one_video",+ "cost_usd": 0.001+}+```++`data` is the platform payload exactly as TikHub returns it - the shape+differs per endpoint and per platform. `code` / `router` are TikHub's own+envelope fields. `cost_usd` is what this call was billed.++`list_endpoints`:++```json+{+ "operation": "list_endpoints",+ "platform": "twitter",+ "count": 13,+ "total": 13,+ "endpoints": [+ {+ "endpoint": "twitter_web.fetch_search_timeline",+ "resource": "twitter_web",+ "method": "fetch_search_timeline",+ "http_method": "GET",+ "path": "/api/v1/twitter/web/fetch_search_timeline",+ "summary": "Search",+ "params": [+ { "name": "keyword", "type": "str", "required": true },+ { "name": "search_type", "type": "str | None", "required": false },+ { "name": "cursor", "type": "str | None", "required": false }+ ]+ }+ ]+}+```++`count` is how many endpoints came back after `search` + `limit`; `total`+is how many the platform has. With `platform: "all"` and no `search` the+listing is capped, so always narrow by platform or search term.++On failure the tool raises an error carrying TikHub's own message -+surface it verbatim (see "Errors" below).++## The discover-then-call loop++Never guess an endpoint id or its parameter names. Endpoints are+introspected from the installed SDK at runtime, so `list_endpoints` is+always exact, free, and returns the `required` flag for every parameter.++1. Narrow by platform and a search term:++```json+{ "operation": "list_endpoints", "platform": "douyin", "search": "comments" }+```++2. Read the `params` list of the endpoint you picked, then call it with+ exactly those keyword names:++```json+{ "operation": "call", "endpoint": "douyin_web.fetch_video_comments", "params": { "aweme_id": "7372484719365098803", "count": 20 } }+```++3. Paginate by feeding the cursor that came back in `data` into the next+ call (see "Pagination").++The `endpoint` field also accepts the REST path+(`/api/v1/douyin/web/fetch_video_comments`) or the slash spelling+(`douyin_web/fetch_video_comments`); all three resolve to the same SDK+method. An unknown id fails before any network call and the error lists+the closest matches.++Unknown keyword names are rejected before the request is sent, and the+error lists the accepted parameters - fix the names and retry, do not+retry the same call.++### POST batch endpoints++A few endpoints take a JSON body instead of query parameters (the SDK+signature shows a single `body` argument). Pass it as the `body` key+inside `params`:++```json+{ "operation": "call", "endpoint": "douyin_web.fetch_multi_video", "params": { "body": ["7372484719365098803", "7372484719365098804"] } }+```++## Share URLs: prefer fetch_url++For any TikTok or Douyin share link (including short links on+v.douyin.com and vm.tiktok.com), use `fetch_url` first. It resolves the+link and returns the full video payload in one billed request without you+having to extract an `aweme_id`:++```json+{ "operation": "fetch_url", "url": "https://www.tiktok.com/@scout2015/video/7372484719365098803" }+{ "operation": "fetch_url", "url": "https://v.douyin.com/iRNBho6u/", "minimal": true }+```++`minimal: true` trims the payload to the fields most tasks need (id,+description, author, stats, media URLs). For other platforms use the+platform's own URL-accepting endpoint (`instagram_v3.get_post_info` takes a+`url`, `kuaishou_web.fetch_one_video_by_url` takes a `url`,+`xiaohongshu_web.get_note_info_v7` takes `share_text`).++## Per-platform cheat sheet++Ids below are verified against the installed SDK. Parameter names are the+SDK keyword names; `list_endpoints` remains the authority for the full+signature.++### TikTok++Two resource families: `tiktok_web` (web-client fields, camelCase params+such as `secUid` / `uniqueId` / `itemId`) and `tiktok_app_v3` (app-client+fields, snake_case params such as `sec_user_id` / `unique_id` /+`aweme_id`). Both return TikTok data; the app family tends to carry richer+statistics, the web family is lighter.++| Endpoint | Required params | Notes |+|---|---|---|+| `tiktok_web.fetch_user_profile` | `uniqueId` or `secUid` | Profile by @handle (`uniqueId`) or sec id |+| `tiktok_web.fetch_user_post` | `secUid` | Paginate with `cursor` (int); `count` |+| `tiktok_web.fetch_post_detail` | `itemId` | Single video (web fields) |+| `tiktok_web.fetch_search_video` | `keyword` | Paginate with `offset` + `search_id` from the previous page |+| `tiktok_web.fetch_post_comment` | `aweme_id` | Paginate with `cursor`; `count` |+| `tiktok_app_v3.fetch_one_video` | `aweme_id` | Single video (app fields) |+| `tiktok_app_v3.fetch_one_video_by_share_url` | `share_url` | Video from a share link |+| `tiktok_app_v3.fetch_user_post_videos` | `sec_user_id` or `unique_id` | Paginate with `max_cursor`; `count` |+| `tiktok_app_v3.fetch_video_search_result` | `keyword` | `offset`, `count`, `sort_type`, `publish_time`, `region` |+| `tiktok_app_v3.handler_user_profile` | one of `user_id` / `sec_user_id` / `unique_id` | Profile (app fields) |+| `tiktok_app_v3.fetch_hashtag_video_list` | `ch_id` | Videos under a hashtag; get `ch_id` from `tiktok_app_v3.fetch_hashtag_search_result` |++### Douyin++`douyin_web` is the primary family; `douyin_app_v3` mirrors it with app+fields; `douyin_search` holds the current search endpoints (POST, but+still plain keyword params).++| Endpoint | Required params | Notes |+|---|---|---|+| `douyin_web.fetch_one_video` | `aweme_id` | Single video; `need_anchor_info` optional |+| `douyin_web.fetch_one_video_by_share_url` | `share_url` | Video from a share link (or use `fetch_url`) |+| `douyin_web.handler_user_profile` | `sec_user_id` | Profile; `douyin_web.handler_user_profile_v2` takes `unique_id` (the Douyin handle) |+| `douyin_web.fetch_user_post_videos` | `sec_user_id` | Paginate with `max_cursor` (string); `count` |+| `douyin_web.fetch_video_comments` | `aweme_id` | Paginate with `cursor`; `count` |+| `douyin_web.fetch_hot_search_result` | - | Hot search board |+| `douyin_search.fetch_video_search_v2` | `keyword` | Paginate with `cursor` + `search_id`; `sort_type`, `publish_time`, `filter_duration` |+| `douyin_app_v3.fetch_one_video_v3` | `aweme_id` | Video without copyright restrictions on media URLs |+++Three generations exist: `instagram_v1`, `instagram_v2`, `instagram_v3`.+Prefer `instagram_v3` (newest; most endpoints accept either `user_id` or+`username`). `instagram_v2` remains useful for hashtag / location / music+feeds and paginates with `pagination_token`.++| Endpoint | Required params | Notes |+|---|---|---|+| `instagram_v3.get_user_profile` | `user_id` or `username` | Profile |+| `instagram_v3.get_user_posts` | `username` | Paginate with `after` (cursor) and `first` / `count` |+| `instagram_v3.get_user_reels` | `user_id` or `username` | Paginate with `after`; `page_size` |+| `instagram_v3.get_post_info` | `media_id` or `url` | Post from a URL or media id |+| `instagram_v3.get_post_comments` | `code` (the shortcode from the post URL) | Paginate with `min_id`; `sort_order` |+| `instagram_v3.general_search` | `query` | Paginate with `next_max_id` + `rank_token` |+| `instagram_v3.get_user_followers` | `user_id` or `username` | Paginate with `max_id`; `count` |+| `instagram_v2.fetch_hashtag_posts` | `keyword` | Paginate with `pagination_token`; `feed_type` |++### YouTube++`youtube_web` is the primary family; `youtube_web_v2` adds captions,+stream URLs, community posts and URL-accepting variants.++| Endpoint | Required params | Notes |+|---|---|---|+| `youtube_web.get_video_info_v3` | `video_id` | Video metadata; `language_code` |+| `youtube_web.get_channel_id` | `channel_name` | Resolve @handle to a channel id |+| `youtube_web.get_channel_info` | `channel_id` | Channel metadata |+| `youtube_web.get_channel_videos_v3` | `channel_id` | Paginate with `continuation_token` |+| `youtube_web.get_video_comments` | `video_id` | Paginate with `continuation_token`; `sort_by` |+| `youtube_web.get_general_search` | `search_query` | Filters: `upload_time`, `duration`, `content_type`, `sort_by`; paginate with `continuation_token` |+| `youtube_web_v2.get_video_captions` | `video_id` or `video_url` | Subtitle text; `language_code`, `format` |++### Twitter / X++| Endpoint | Required params | Notes |+|---|---|---|+| `twitter_web.fetch_tweet_detail` | `tweet_id` | Single tweet |+| `twitter_web.fetch_user_profile` | `screen_name` or `rest_id` | Profile |+| `twitter_web.fetch_user_post_tweet` | `screen_name` or `rest_id` | User timeline; paginate with `cursor` |+| `twitter_web.fetch_search_timeline` | `keyword` | `search_type` (Top / Latest / Media / People / Lists); paginate with `cursor` |+| `twitter_web.fetch_post_comments` | `tweet_id` | Replies; paginate with `cursor` |+| `twitter_web.fetch_trending` | - | `country` optional |++### Xiaohongshu (RedNote)++Families: `xiaohongshu_web` (note detail, search, user), `xiaohongshu_app_v2`+(comments, topic feeds, search variants), plus older `xiaohongshu_app`,+`xiaohongshu_web_v2`, `xiaohongshu_web_v3`. Most note endpoints accept+either a `note_id` or the raw `share_text` copied from the app.++| Endpoint | Required params | Notes |+|---|---|---|+| `xiaohongshu_web.get_note_info_v7` | `note_id` or `share_text` | Note detail (newest) |+| `xiaohongshu_web.get_note_id_and_xsec_token` | `share_text` | Resolve a share link to `note_id` + `xsec_token` |+| `xiaohongshu_web.search_notes_v3` | `keyword` | Paginate with `page`; `sort`, `noteType`, `noteTime` |+| `xiaohongshu_web.get_user_info` | `user_id` | Profile |+| `xiaohongshu_web.get_user_notes_v2` | `user_id` | Paginate with `lastCursor` |+| `xiaohongshu_app_v2.get_note_comments` | `note_id` or `share_text` | Paginate with `cursor`; `sort_strategy` |++### Other platforms++Every other family follows the same shape - run `list_endpoints` with the+platform to see them. Starting points: `bilibili_web.fetch_one_video`+(`bv_id`), `kuaishou_web.fetch_one_video_by_url` (`url`),+`weibo_web_v2.fetch_post_detail` (`id`), `reddit_app.fetch_post_details`+(`post_id`), `threads_web.fetch_post_detail` (`post_id`),+`linkedin_web_v2.get_user_profile` (`username`),+`zhihu_web.fetch_question_answers` (`question_id`).++### Hybrid parsing and account++| Endpoint | Required params | Notes |+|---|---|---|+| `hybrid_parsing.video_data` | `url` | What `fetch_url` calls; `minimal`, `base64_url` |+| `tikhub_user.get_user_info` | - | Balance, free credit, email (what `account` calls) |+| `tikhub_user.get_user_daily_usage` | - | Today's request count and spend (what `account` calls) |+| `tikhub_user.get_endpoint_info` | `endpoint` (a `/api/v1/...` path) | Price and description of one endpoint |+| `tikhub_user.calculate_price` | `endpoint` | Projected cost; `request_per_day` |++## Pagination++Cursor names vary by platform. Read the cursor from `data` and pass it+back under the parameter name the endpoint declares:++| Platform | Request param | Where the next cursor lives in `data` | Stop when |+|---|---|---|---|+| TikTok app / Douyin | `max_cursor` (user posts), `cursor` (comments, hashtags) | `max_cursor` / `cursor` | `has_more` is `false` or `0` |+| TikTok web | `cursor` (posts, comments), `offset` + `search_id` (search) | `cursor`, `search_id` | `hasMore` / `has_more` is false |+| Instagram v3 | `after` (posts, reels), `max_id` (followers), `min_id` (comments), `next_max_id` (search) | `end_cursor` under `page_info`, or `next_max_id` / `next_min_id` | `has_next_page` false or cursor absent |+| Instagram v2 | `pagination_token` | `pagination_token` | token absent |+| YouTube | `continuation_token` | `continuation_token` | token absent |+| Twitter | `cursor` | `cursor` (bottom cursor) | cursor repeats or no new entries |+| Xiaohongshu | `cursor` (app_v2), `lastCursor` (web), `page` (search) | `cursor` / `last_cursor` | `has_more` false |+| Weibo | `page` or `max_id` | `max_id` | empty page |+| Kuaishou | `pcursor` | `pcursor` | `pcursor` is `"no_more"` |++Fetch pages sequentially - each page is one billed request, and the same+endpoint path is limited to 10 requests per second.++## Cost and rate limits++- Roughly $0.001 per successful request (TikHub bills only 2xx responses;+ failed calls are free). `list_endpoints` never calls the API and costs+ nothing.+- 10 requests per second per endpoint path. Bursting past it returns 429;+ the SDK retries with backoff, and the tool reports `retry_after` when+ TikHub sends it.+- Before a large crawl (hundreds of pages), run `account` and tell the+ user the balance; `tikhub_user.calculate_price` gives a projection for+ one endpoint.+- Every `call` result carries `cost_usd`; costs are also recorded in+ OpenCompany's API usage panel under the `tikhub` service.++## Errors++The tool raises a plain error message; relay it to the user as-is and act+on the class of failure:++| Message contains | Meaning | What to do |+|---|---|---|+| `rejected the API key (401)` | Key missing, revoked or wrong | Tell the user to update it in Credentials -> TikHub; do not retry |+| `refused <endpoint> (403)` | Key lacks the endpoint's scope OR balance is exhausted | Run `account`; if balance is 0 tell the user to top up, otherwise the endpoint is outside the key's plan |+| `rate-limited <endpoint> (429)` | More than 10 req/s on one path | Wait `retry_after` seconds (or a few seconds), then retry once |+| `rejected the request to <endpoint> (400 or 422)` | Wrong or malformed argument values | The message carries TikHub's `detail`; fix the value and retry |+| `returned 404 for <endpoint>` | Route removed or id / resource no longer exists | Run `list_endpoints` and pick the current id |+| `upstream error while calling <endpoint>` (5xx / connection) | Platform-side failure after the SDK's retries | Try again later; try the `_v2` / `_v3` variant of the endpoint if one exists |+| `Unknown TikHub endpoint` | Id typo | Use the suggestions in the message or `list_endpoints` |+| `... rejected params` | Keyword name not in the SDK signature | Use the accepted names listed in the message |++A 200 whose body says `code: 4xx` is also raised as an error with+TikHub's message - the platform returned nothing useful for that id.++## Authentication++OpenCompany stores the TikHub API key encrypted; you never see it and+must never ask the user to paste it into the chat. If a call fails with+the 401 message, tell the user: open **Credentials -> Scrapers -> TikHub**,+paste the key from the TikHub user portal (tikhub.io -> API Keys) and+click Validate - the modal shows the account email and balance when the+key is accepted.++## Best practices++1. **Prefer `fetch_url` for any share URL.** One request, no id+ extraction, works for TikTok and Douyin links including short links.+2. **Use `list_endpoints` before guessing.** It is free, exact for the+ installed SDK, and shows which params are required. Pick the highest+ `_vN` suffix when several variants exist unless the user needs a+ specific field set.+3. **Pass params by their SDK names**, as a JSON object, values typed as+ the signature says (`count` is an int, cursors are usually strings).+4. **Paginate deliberately.** Report how many pages you fetched and stop+ when `has_more` (or the platform's equivalent) turns false; do not loop+ without a page cap the user agreed to.+5. **Surface errors verbatim.** TikHub's `detail` strings are precise;+ do not paraphrase them or retry blindly on 401 / 403 / 404.+6. **Keep `data` intact when relaying to downstream nodes** - the raw+ platform payload is what other nodes and the user's exports expect.+7. **Respect the platforms' terms and the user's intent**: scrape public+ content only, do not collect personal data beyond the task, and do not+ hammer one endpoint past 10 req/s.diff --git a/server/tests/fixtures/tool_names_snapshot.json b/server/tests/fixtures/tool_names_snapshot.jsonindex c2fd065f..ffb82350 100644--- a/server/tests/fixtures/tool_names_snapshot.json+++ b/server/tests/fixtures/tool_names_snapshot.json@@ -75,6 +75,7 @@"gcloudAction": "gcloud","typescriptExecutor": "typescript_code","apifyActor": "apify_actor",+ "tikhubAction": "tikhub_action","agentBuilder": "agent_builder","twitterSend": "twitter_send","twitterSearch": "twitter_search",diff --git a/server/tests/nodes/test_tikhub.py b/server/tests/nodes/test_tikhub.pynew file mode 100644index 00000000..38e1f2a4--- /dev/null+++ b/server/tests/nodes/test_tikhub.py@@ -0,0 +1,1011 @@+"""Contract tests for the TikHub scraper node (``tikhubAction``).++The node is the "flattened CLI" over the official ``tikhub`` SDK: one+``call`` operation addressed by the SDK's own ``resource.method`` id, plus+``list_endpoints`` (pure introspection, no network), ``fetch_url`` (hybrid+parser) and ``account``. Everything vendor-specific lives in+``nodes/scraper/tikhub_action/_sdk.py`` and is imported lazily, so the+plugin registers without the SDK present.++The SDK speaks httpx against ``https://api.tikhub.io``, so ``respx`` sees+every request the node makes; nothing here touches the network. The+endpoint index is built by reflecting over the installed SDK, so the+introspection tests exercise the real package — SDK drift (a renamed+resource, a platform that vanished) fails here rather than in a workflow.+"""++from __future__ import annotations++import json+import re+import subprocess+import sys+from pathlib import Path+from typing import get_args+from unittest.mock import patch++import httpx+import pytest+import respx++from tests.nodes._mocks import patched_container, patched_pricing++pytestmark = pytest.mark.node_contract++import nodes.scraper.tikhub_action as tikhub_mod # noqa: E402+from nodes.scraper._credentials import TikHubCredential # noqa: E402+from nodes.scraper.tikhub_action import _sdk # noqa: E402+from services.node_registry import get_node_class # noqa: E402+from services.plugin.base import NodeUserError # noqa: E402+from services.plugin.credential import CREDENTIAL_REGISTRY, ProbeResult # noqa: E402+from services.ws_handler_registry import get_option_loader # noqa: E402++BASE = "https://api.tikhub.io"+FETCH_ONE_VIDEO = f"{BASE}/api/v1/douyin/web/fetch_one_video"+HYBRID_VIDEO_DATA = f"{BASE}/api/v1/hybrid/video_data"+USER_INFO = f"{BASE}/api/v1/tikhub/user/get_user_info"+USER_DAILY_USAGE = f"{BASE}/api/v1/tikhub/user/get_user_daily_usage"++PLUGIN_DIR = Path(tikhub_mod.__file__).parent+SCRAPER_DIR = PLUGIN_DIR.parent+SERVER_DIR = PLUGIN_DIR.parents[2]+SKILL_MD = SERVER_DIR / "skills" / "web_agent" / "tikhub-skill" / "SKILL.md"++KEYS = {"tikhub": "tk"}+++def _ok(data, router="/api/v1/douyin/web/fetch_one_video", params=None):+ """A TikHub success envelope as the API returns it."""+ return {"code": 200, "router": router, "params": params or {}, "data": data}+++def _reset_cache() -> None:+ _sdk._INDEX = None+ for name in ("_BY_ENDPOINT", "_BY_PATH"):+ cache = getattr(_sdk, name, None)+ if isinstance(cache, dict):+ cache.clear()+ elif cache is not None:+ setattr(_sdk, name, None)++[email protected](autouse=True)+def reset_index():+ """The index is a module-level cache; every test starts cold so the+ SDK-missing degrade test cannot be masked by a warm cache."""+ _reset_cache()+ yield+ _reset_cache()++[email protected](autouse=True)+def no_retries(monkeypatch):+ """The SDK retries 401/403/429/5xx with anyio backoff sleeps; the+ error-mapping tests only need the first response."""+ monkeypatch.setattr(_sdk, "_MAX_RETRIES", 0)+++async def _execute(harness, params, *, keys=KEYS, total_cost=0.001, **kwargs):+ with patched_container(auth_api_keys=keys), patched_pricing(total_cost=total_cost):+ return await harness.execute("tikhubAction", params, **kwargs)+++# ============================================================================+# 1. Registration+# ============================================================================+++class TestRegistration:+ def test_class_attributes(self):+ cls = get_node_class("tikhubAction")+ assert cls is not None, "tikhubAction is not registered"+ assert cls.type == "tikhubAction"+ assert cls.tool_name == "tikhub_action"+ assert tuple(cls.group) == ("scraper", "tool")+ assert cls.usable_as_tool is True+ assert TikHubCredential in tuple(cls.credentials)++ def test_both_canvas_handles_stay_visible(self):+ # usable_as_tool=True auto-hides both handles unless the class+ # declares them; the node must stay wirable on the canvas.+ cls = get_node_class("tikhubAction")+ assert cls.hide_input_handle is False+ assert cls.hide_output_handle is False+ assert "hide_input_handle" in cls.__dict__+ assert "hide_output_handle" in cls.__dict__++ def test_option_loader_and_credential_registered(self):+ assert get_option_loader("tikhubEndpoints") is not None+ assert CREDENTIAL_REGISTRY["tikhub"] is TikHubCredential++ def test_tool_schema_is_flat(self):+ # Dual-purpose ActionNode: ``AIService._get_tool_schema`` builds the+ # LLM schema as ``inline_schema_refs(Params.model_json_schema())``.+ from services.plugin.tool import inline_schema_refs++ cls = get_node_class("tikhubAction")+ schema = inline_schema_refs(cls.Params.model_json_schema())+ dumped = json.dumps(schema)+ assert "$defs" not in dumped+ assert "definitions" not in schema+ assert '"$ref"' not in dumped+ assert set(schema["properties"]) >= {"operation", "platform", "endpoint", "params", "search", "limit", "url", "minimal"}++ def test_params_defaults(self):+ cls = get_node_class("tikhubAction")+ params = cls.Params()+ assert params.operation == "call"+ assert params.platform == "all"+ assert params.endpoint == ""+ assert params.params == {}+ assert params.minimal is False+ assert set(get_args(cls.Params.model_fields["operation"].annotation)) == {+ "call",+ "list_endpoints",+ "fetch_url",+ "account",+ }++ def test_reserved_field_names_absent(self):+ # ParameterRenderer keys magic on these literal names.+ cls = get_node_class("tikhubAction")+ assert not {"model", "api_key", "parameters", "action"} & set(cls.Params.model_fields)+++# ============================================================================+# 2-4. call+# ============================================================================+++class TestCall:+ @respx.mock+ async def test_happy_path(self, harness):+ route = respx.get(FETCH_ONE_VIDEO).mock(+ return_value=httpx.Response(200, json=_ok({"aweme_id": "7123", "desc": "hello"}))+ )++ result = await _execute(+ harness,+ {+ "operation": "call",+ "endpoint": "douyin_web.fetch_one_video",+ "params": {"aweme_id": "7123"},+ },+ )++ harness.assert_envelope(result, success=True)+ harness.assert_output_shape(result, ["operation", "endpoint", "path", "data", "code", "router", "cost_usd"])+ payload = result["result"]+ assert payload["operation"] == "call"+ assert payload["endpoint"] == "douyin_web.fetch_one_video"+ assert payload["path"] == "/api/v1/douyin/web/fetch_one_video"+ assert payload["data"] == {"aweme_id": "7123", "desc": "hello"}+ assert payload["code"] == 200+ assert payload["router"] == "/api/v1/douyin/web/fetch_one_video"+ assert payload["cost_usd"] == 0.001++ assert route.call_count == 1+ sent = respx.calls.last.request+ assert sent.method == "GET"+ assert sent.headers["Authorization"] == "Bearer tk"+ assert sent.url.params["aweme_id"] == "7123"+ # optional kwargs left at None are dropped, not sent as "None"+ assert "need_anchor_info" not in sent.url.params++ async def test_dispatch_reads_the_operation_key(self, harness):+ # ``BaseNode._pick_operation`` reads the raw ``operation`` key for+ # every multi-op node; the Params default ("call") is what the panel+ # and the tool schema advertise, not what dispatch falls back to.+ with respx.mock:+ result = await _execute(harness, {"endpoint": "douyin_web.fetch_one_video", "params": {"aweme_id": "1"}})+ assert respx.calls.call_count == 0++ harness.assert_envelope(result, success=False)+ assert "operation" in result["error"].lower()++ @respx.mock+ async def test_params_as_json_string(self, harness):+ respx.get(FETCH_ONE_VIDEO).mock(return_value=httpx.Response(200, json=_ok({"x": 1})))++ result = await _execute(+ harness,+ {"operation": "call", "endpoint": "douyin_web.fetch_one_video", "params": '{"aweme_id": "999"}'},+ )++ harness.assert_envelope(result, success=True)+ assert respx.calls.last.request.url.params["aweme_id"] == "999"++ @respx.mock+ async def test_params_blank_string_from_panel(self, harness):+ respx.get(USER_INFO).mock(+ return_value=httpx.Response(200, json=_ok({"email": "[email protected]"}, router="/api/v1/tikhub/user/get_user_info"))+ )++ result = await _execute(harness, {"operation": "call", "endpoint": "tikhub_user.get_user_info", "params": ""})++ harness.assert_envelope(result, success=True)+ assert result["result"]["data"] == {"email": "[email protected]"}+ assert respx.calls.call_count == 1++ @respx.mock+ async def test_endpoint_as_rest_path_alias(self, harness):+ respx.get(FETCH_ONE_VIDEO).mock(return_value=httpx.Response(200, json=_ok({"x": 1})))++ result = await _execute(+ harness,+ {"operation": "call", "endpoint": "/api/v1/douyin/web/fetch_one_video", "params": {"aweme_id": "1"}},+ )++ harness.assert_envelope(result, success=True)+ assert result["result"]["endpoint"] == "douyin_web.fetch_one_video"+ assert respx.calls.call_count == 1++ @respx.mock+ async def test_endpoint_as_slash_alias(self, harness):+ respx.get(FETCH_ONE_VIDEO).mock(return_value=httpx.Response(200, json=_ok({"x": 1})))++ result = await _execute(+ harness,+ {"operation": "call", "endpoint": "douyin_web/fetch_one_video", "params": {"aweme_id": "1"}},+ )++ harness.assert_envelope(result, success=True)+ assert result["result"]["endpoint"] == "douyin_web.fetch_one_video"++ @respx.mock+ async def test_empty_data_is_kept(self, harness):+ # An empty answer from TikHub is a real answer, not a missing key.+ respx.get(FETCH_ONE_VIDEO).mock(return_value=httpx.Response(200, json=_ok({})))++ result = await _execute(+ harness,+ {"operation": "call", "endpoint": "douyin_web.fetch_one_video", "params": {"aweme_id": "1"}},+ )++ harness.assert_envelope(result, success=True)+ assert "data" in result["result"]+ assert result["result"]["data"] == {}+++# ============================================================================+# 5-9. errors+# ============================================================================+++class TestErrors:+ @respx.mock+ async def test_unknown_endpoint_is_user_error_without_http(self, harness):+ result = await _execute(harness, {"operation": "call", "endpoint": "nope.nothing", "params": {}})++ harness.assert_envelope(result, success=False)+ assert result["error_type"] == "NodeUserError"+ assert "list_endpoints" in result["error"]+ assert respx.calls.call_count == 0++ @respx.mock+ async def test_missing_endpoint_is_user_error_without_http(self, harness):+ result = await _execute(harness, {"operation": "call", "endpoint": ""})++ harness.assert_envelope(result, success=False)+ assert result["error_type"] == "NodeUserError"+ assert respx.calls.call_count == 0++ @respx.mock+ async def test_unknown_kwarg_rejected_before_http(self, harness):+ respx.get(FETCH_ONE_VIDEO).mock(return_value=httpx.Response(200, json=_ok({"x": 1})))++ result = await _execute(+ harness,+ {"operation": "call", "endpoint": "douyin_web.fetch_one_video", "params": {"aweme_id": "1", "bogus": True}},+ )++ harness.assert_envelope(result, success=False)+ assert result["error_type"] == "NodeUserError"+ assert "bogus" in result["error"]+ # the message lists what the endpoint accepts+ assert "aweme_id" in result["error"]+ assert respx.calls.call_count == 0++ @respx.mock+ async def test_missing_required_kwarg_rejected_before_http(self, harness):+ respx.get(FETCH_ONE_VIDEO).mock(return_value=httpx.Response(200, json=_ok({"x": 1})))++ result = await _execute(harness, {"operation": "call", "endpoint": "douyin_web.fetch_one_video", "params": {}})++ harness.assert_envelope(result, success=False)+ assert result["error_type"] == "NodeUserError"+ assert "aweme_id" in result["error"]+ assert respx.calls.call_count == 0++ @respx.mock+ async def test_missing_key_is_permission_denied_envelope(self, harness):+ respx.get(FETCH_ONE_VIDEO).mock(return_value=httpx.Response(200, json=_ok({"x": 1})))++ result = await _execute(+ harness,+ {"operation": "call", "endpoint": "douyin_web.fetch_one_video", "params": {"aweme_id": "1"}},+ keys={},+ )++ harness.assert_envelope(result, success=False)+ assert result["error_type"] == "PermissionDeniedError"+ assert result["credential"]["provider"] == "tikhub"+ assert result["credential"]["reason"] == "missing"+ assert result["credential"]["remediation"] == "add_key"+ assert respx.calls.call_count == 0++ @pytest.mark.parametrize(+ ("status", "body", "headers", "needles"),+ [+ (401, {"detail": "invalid api key"}, {}, ("401",)),+ (403, {"detail": "insufficient balance"}, {}, ("403",)),+ (429, {"detail": "slow down"}, {"Retry-After": "7"}, ("429", "7")),+ (+ 422,+ {"detail": [{"loc": ["query", "aweme_id"], "msg": "field required"}]},+ {},+ ("422", "aweme_id"),+ ),+ (502, {"detail": "upstream failed"}, {}, ("502",)),+ ],+ )+ @respx.mock+ async def test_http_errors_map_to_user_errors(self, harness, status, body, headers, needles):+ route = respx.get(FETCH_ONE_VIDEO).mock(return_value=httpx.Response(status, json=body, headers=headers))++ result = await _execute(+ harness,+ {"operation": "call", "endpoint": "douyin_web.fetch_one_video", "params": {"aweme_id": "1"}},+ )++ harness.assert_envelope(result, success=False)+ assert result["error_type"] == "NodeUserError", result+ for needle in needles:+ assert needle in result["error"], result["error"]+ # _MAX_RETRIES = 0: exactly one attempt+ assert route.call_count == 1++ @respx.mock+ async def test_401_points_at_credentials(self, harness):+ respx.get(FETCH_ONE_VIDEO).mock(return_value=httpx.Response(401, json={"detail": "bad"}))++ result = await _execute(+ harness,+ {"operation": "call", "endpoint": "douyin_web.fetch_one_video", "params": {"aweme_id": "1"}},+ )++ assert result["success"] is False+ assert "credential" in result["error"].lower()++ @respx.mock+ async def test_in_body_error_code_on_http_200(self, harness):+ # The SDK ignores in-body codes; the node must not.+ respx.get(FETCH_ONE_VIDEO).mock(+ return_value=httpx.Response(+ 200,+ json={"code": 400, "router": "/api/v1/douyin/web/fetch_one_video", "message": "aweme not found"},+ )+ )++ result = await _execute(+ harness,+ {"operation": "call", "endpoint": "douyin_web.fetch_one_video", "params": {"aweme_id": "1"}},+ )++ harness.assert_envelope(result, success=False)+ assert result["error_type"] == "NodeUserError"+ assert "aweme not found" in result["error"]+++# ============================================================================+# 10. fetch_url + account+# ============================================================================+++class TestFetchUrlAndAccount:+ @respx.mock+ async def test_fetch_url_uses_hybrid_parser(self, harness):+ route = respx.get(HYBRID_VIDEO_DATA).mock(+ return_value=httpx.Response(200, json=_ok({"id": "v1"}, router="/api/v1/hybrid/video_data"))+ )+ share_url = "https://www.tiktok.com/@someone/video/7123"++ result = await _execute(+ harness,+ {"operation": "fetch_url", "url": share_url, "minimal": True},+ )++ harness.assert_envelope(result, success=True)+ harness.assert_output_shape(result, ["operation", "url", "data", "cost_usd"])+ payload = result["result"]+ assert payload["operation"] == "fetch_url"+ assert payload["url"] == share_url+ assert payload["data"] == {"id": "v1"}++ assert route.call_count == 1+ sent = respx.calls.last.request+ assert sent.headers["Authorization"] == "Bearer tk"+ assert sent.url.params["url"] == share_url+ assert sent.url.params["minimal"] == "true"+ assert "base64_url" not in sent.url.params++ @respx.mock+ async def test_fetch_url_default_minimal(self, harness):+ respx.get(HYBRID_VIDEO_DATA).mock(return_value=httpx.Response(200, json=_ok({}, router="/api/v1/hybrid/video_data")))++ result = await _execute(harness, {"operation": "fetch_url", "url": "https://v.douyin.com/abc/"})++ harness.assert_envelope(result, success=True)+ assert respx.calls.last.request.url.params["url"] == "https://v.douyin.com/abc/"++ @respx.mock+ async def test_fetch_url_requires_url(self, harness):+ result = await _execute(harness, {"operation": "fetch_url", "url": ""})++ harness.assert_envelope(result, success=False)+ assert result["error_type"] == "NodeUserError"+ assert respx.calls.call_count == 0++ @respx.mock+ async def test_account_makes_two_gets(self, harness):+ info = respx.get(USER_INFO).mock(+ return_value=httpx.Response(+ 200,+ json=_ok({"email": "[email protected]", "balance": 12.5}, router="/api/v1/tikhub/user/get_user_info"),+ )+ )+ usage = respx.get(USER_DAILY_USAGE).mock(+ return_value=httpx.Response(+ 200,+ json=_ok({"requests": 42}, router="/api/v1/tikhub/user/get_user_daily_usage"),+ )+ )++ result = await _execute(harness, {"operation": "account"})++ harness.assert_envelope(result, success=True)+ harness.assert_output_shape(result, ["operation", "account", "usage"])+ payload = result["result"]+ assert payload["operation"] == "account"+ assert payload["account"] == {"email": "[email protected]", "balance": 12.5}+ assert payload["usage"] == {"requests": 42}+ assert info.call_count == 1+ assert usage.call_count == 1+ assert respx.calls.call_count == 2+ for call in respx.calls:+ assert call.request.headers["Authorization"] == "Bearer tk"+++# ============================================================================+# 11. list_endpoints+# ============================================================================+++class TestListEndpoints:+ async def test_platform_and_search_filter(self, harness):+ with respx.mock:+ result = await _execute(+ harness,+ {"operation": "list_endpoints", "platform": "douyin", "search": "fetch_one_video"},+ )+ assert respx.calls.call_count == 0++ harness.assert_envelope(result, success=True)+ harness.assert_output_shape(result, ["operation", "platform", "endpoints", "count", "total"])+ payload = result["result"]+ assert payload["operation"] == "list_endpoints"+ assert payload["platform"] == "douyin"+ endpoints = payload["endpoints"]+ assert endpoints+ assert payload["count"] == len(endpoints)+ assert payload["total"] >= payload["count"]+ assert all(e["endpoint"].startswith("douyin") for e in endpoints)++ by_id = {e["endpoint"]: e for e in endpoints}+ entry = by_id["douyin_web.fetch_one_video"]+ assert entry["resource"] == "douyin_web"+ assert entry["method"] == "fetch_one_video"+ assert entry["http_method"] == "GET"+ assert entry["path"] == "/api/v1/douyin/web/fetch_one_video"+ assert entry["summary"]+ params = {p["name"]: p for p in entry["params"]}+ assert params["aweme_id"]["required"] is True+ assert params["aweme_id"]["type"]+ assert params["need_anchor_info"]["required"] is False++ async def test_all_platforms_respects_limit(self, harness):+ with respx.mock:+ result = await _execute(harness, {"operation": "list_endpoints", "platform": "all", "limit": 5})+ assert respx.calls.call_count == 0++ harness.assert_envelope(result, success=True)+ payload = result["result"]+ assert payload["count"] == 5+ assert len(payload["endpoints"]) == 5+ assert payload["total"] > 1000++ async def test_default_limit_caps_at_100(self, harness):+ with respx.mock:+ result = await _execute(harness, {"operation": "list_endpoints"})+ assert respx.calls.call_count == 0++ harness.assert_envelope(result, success=True)+ assert result["result"]["count"] == 100++ async def test_search_over_path(self, harness):+ with respx.mock:+ result = await _execute(+ harness,+ {"operation": "list_endpoints", "platform": "hybrid", "search": "/api/v1/hybrid/video_data"},+ )+ assert respx.calls.call_count == 0++ harness.assert_envelope(result, success=True)+ ids = [e["endpoint"] for e in result["result"]["endpoints"]]+ assert "hybrid_parsing.video_data" in ids++ async def test_no_match_is_empty_success(self, harness):+ with respx.mock:+ result = await _execute(+ harness,+ {"operation": "list_endpoints", "platform": "douyin", "search": "zzz_no_such_endpoint_zzz"},+ )++ harness.assert_envelope(result, success=True)+ assert result["result"]["endpoints"] == []+ assert result["result"]["count"] == 0++ async def test_list_never_needs_a_key(self, harness):+ # Discovery is introspection over the installed SDK; a missing key+ # must not block it.+ with respx.mock:+ result = await _execute(harness, {"operation": "list_endpoints", "platform": "douyin"}, keys={})++ harness.assert_envelope(result, success=True)+ assert result["result"]["count"] > 0+++# ============================================================================+# 12. index / introspection+# ============================================================================+++class TestIndex:+ async def test_index_is_large_and_sorted(self):+ index = await _sdk.endpoint_index()+ assert len(index) > 1000+ ids = [info.endpoint for info in index]+ assert ids == sorted(ids)+ assert len(set(ids)) == len(ids)++ async def test_index_is_cached(self):+ first = await _sdk.endpoint_index()+ second = await _sdk.endpoint_index()+ assert first is second++ async def test_every_platform_matches_a_live_resource(self):+ assert _sdk.PLATFORMS[0] == "all"+ index = await _sdk.endpoint_index()+ live = {info.platform for info in index}+ missing = [p for p in _sdk.PLATFORMS if p != "all" and p not in live]+ assert not missing, f"PLATFORMS entries with no SDK resource (SDK drift): {missing}"++ def test_params_platform_literal_mirrors_platforms(self):+ cls = get_node_class("tikhubAction")+ assert tuple(get_args(cls.Params.model_fields["platform"].annotation)) == tuple(_sdk.PLATFORMS)++ async def test_endpoint_info_shape(self):+ index = await _sdk.endpoint_index()+ info = next(i for i in index if i.endpoint == "douyin_web.fetch_one_video")+ assert info.resource == "douyin_web"+ assert info.method == "fetch_one_video"+ assert info.platform == "douyin"+ assert info.http_method == "GET"+ assert info.path == "/api/v1/douyin/web/fetch_one_video"+ assert info.summary+ params = {p.name: p for p in info.params}+ assert params["aweme_id"].required is True+ assert params["aweme_id"].type+ assert params["need_anchor_info"].required is False++ async def test_lookup_caches_populated(self):+ await _sdk.endpoint_index()+ assert _sdk._BY_ENDPOINT["douyin_web.fetch_one_video"].path == "/api/v1/douyin/web/fetch_one_video"+ assert _sdk._BY_PATH["/api/v1/douyin/web/fetch_one_video"].endpoint == "douyin_web.fetch_one_video"++ @pytest.mark.parametrize(+ "raw",+ [+ "douyin_web.fetch_one_video",+ "douyin_web/fetch_one_video",+ "/api/v1/douyin/web/fetch_one_video",+ "https://api.tikhub.io/api/v1/douyin/web/fetch_one_video",+ " douyin_web.fetch_one_video ",+ ],+ )+ async def test_resolve_endpoint_aliases(self, raw):+ info = await _sdk.resolve_endpoint(raw)+ assert info.endpoint == "douyin_web.fetch_one_video"++ async def test_resolve_unknown_suggests_close_matches(self):+ with pytest.raises(NodeUserError) as excinfo:+ await _sdk.resolve_endpoint("douyin_web.fetch_one_vidoe")+ message = str(excinfo.value)+ assert "fetch_one_video" in message+ assert "list_endpoints" in message++ async def test_resolve_blank_is_user_error(self):+ with pytest.raises(NodeUserError):+ await _sdk.resolve_endpoint("")+++# ============================================================================+# 13. option loader+# ============================================================================+++class TestOptionLoader:+ async def test_returns_options_for_a_platform(self):+ loader = get_option_loader("tikhubEndpoints")+ options = await loader({"platform": "douyin"})+ assert options+ for option in options:+ assert set(option) >= {"value", "label"}+ assert option["value"].startswith("douyin")+ values = [o["value"] for o in options]+ assert "douyin_web.fetch_one_video" in values+ entry = next(o for o in options if o["value"] == "douyin_web.fetch_one_video")+ assert "/api/v1/douyin/web/fetch_one_video" in entry.get("description", "")++ async def test_all_platform_returns_no_options(self):+ # ~1100 items would swamp the dropdown; the panel prompts for a+ # platform instead. The LLM path accepts any id as free text.+ loader = get_option_loader("tikhubEndpoints")+ assert await loader({"platform": "all"}) == []+ assert await loader({}) == []++ async def test_loader_matches_direct_loader_function(self):+ assert get_option_loader("tikhubEndpoints") is _sdk.load_tikhub_endpoints++ async def test_degrades_to_empty_when_sdk_missing(self):+ loader = get_option_loader("tikhubEndpoints")+ _reset_cache()+ blocked = {+ "tikhub": None,+ "tikhub.resources": None,+ "tikhub.resources._base": None,+ "tikhub.errors": None,+ "tikhub.client": None,+ }+ with patch.dict(sys.modules, blocked):+ options = await loader({"platform": "douyin"})+ assert options == []+++# ============================================================================+# 14. boot-path purity+# ============================================================================+++class TestNoEagerSdkImport:+ """Importing the plugin must never import ``tikhub``.++ Runs in a clean interpreter because the pytest process already has the+ SDK loaded by the introspection tests above.+ """++ def _probe(self, code: str) -> str:+ result = subprocess.run(+ [sys.executable, "-c", code],+ cwd=SERVER_DIR,+ capture_output=True,+ text=True,+ timeout=180,+ )+ assert result.returncode == 0, f"probe subprocess failed (rc={result.returncode}):\n{result.stderr}"+ return result.stdout++ def test_plugin_imports_without_sdk(self):+ out = self._probe(+ "import sys\n"+ "sys.modules['tikhub'] = None\n"+ "import nodes.scraper.tikhub_action\n"+ "from services.node_registry import get_node_class\n"+ "print('REGISTERED=' + str(get_node_class('tikhubAction') is not None))\n"+ )+ assert "REGISTERED=True" in out++ def test_plugin_import_does_not_load_sdk(self):+ out = self._probe(+ "import sys\n"+ "import nodes.scraper.tikhub_action\n"+ "import nodes.scraper.tikhub_action._sdk\n"+ "print('LEAKED=' + str('tikhub' in sys.modules))\n"+ )+ assert "LEAKED=False" in out+++# ============================================================================+# 15. credential probe+# ============================================================================+++class _ProbeResponse:+ def __init__(self, status: int, body: dict):+ self.status_code = status+ self._body = body+ self.request = httpx.Request("GET", USER_INFO)++ def raise_for_status(self):+ if self.status_code >= 400:+ raise httpx.HTTPStatusError(+ f"HTTP {self.status_code}",+ request=self.request,+ response=httpx.Response(self.status_code, request=self.request),+ )+ return None++ def json(self):+ return self._body+++def _fake_client(captured: dict, response: _ProbeResponse):+ class _Client:+ def __init__(self, **kwargs):+ captured["client_kwargs"] = kwargs++ async def __aenter__(self):+ return self++ async def __aexit__(self, *args):+ return False++ async def request(self, method, url, **kwargs):+ captured["method"] = method+ captured["url"] = url+ captured["headers"] = kwargs.get("headers") or {}+ captured["params"] = kwargs.get("params")+ captured["json"] = kwargs.get("json")+ return response++ async def get(self, url, **kwargs):+ return await self.request("GET", url, **kwargs)++ return _Client+++class TestCredentialProbe:+ def test_declaration(self):+ assert TikHubCredential.id == "tikhub"+ assert TikHubCredential.auth == "api_key"+ assert TikHubCredential.key_location == "bearer"+ assert TikHubCredential.probe_url == USER_INFO+ assert TikHubCredential.category == "Scrapers"+ assert TikHubCredential.docs_url++ def test_inject_is_bearer(self):+ request = TikHubCredential.inject({"api_key": "tk"}, {"headers": {}, "params": {}})+ assert request["headers"]["Authorization"] == "Bearer tk"++ async def test_probe_200_reads_balance(self, monkeypatch):+ import services.plugin.credential as credential_mod++ captured: dict = {}+ body = {+ "code": 200,+ "router": "/api/v1/tikhub/user/get_user_info",+ "data": {+ "user_data": {"email": "[email protected]", "balance": 12.5, "free_credit": 1.0},+ "api_key_data": {"api_key_name": "default"},+ },+ }+ monkeypatch.setattr(credential_mod.httpx, "AsyncClient", _fake_client(captured, _ProbeResponse(200, body)))++ result = await TikHubCredential._probe("tk123")++ assert isinstance(result, ProbeResult)+ assert result.valid is True+ assert result.extra["balance"] == 12.5+ assert "[email protected]" in result.message+ assert captured["method"] == "GET"+ assert captured["url"] == USER_INFO+ assert captured["headers"]["Authorization"] == "Bearer tk123"++ async def test_probe_tolerates_sparse_body(self, monkeypatch):+ import services.plugin.credential as credential_mod++ captured: dict = {}+ monkeypatch.setattr(+ credential_mod.httpx,+ "AsyncClient",+ _fake_client(captured, _ProbeResponse(200, {"code": 200, "data": {}})),+ )++ result = await TikHubCredential._probe("tk123")+ assert result.valid is True++ async def test_probe_401_raises(self, monkeypatch):+ import services.plugin.credential as credential_mod++ captured: dict = {}+ monkeypatch.setattr(+ credential_mod.httpx,+ "AsyncClient",+ _fake_client(captured, _ProbeResponse(401, {"detail": "invalid api key"})),+ )++ with pytest.raises(httpx.HTTPStatusError):+ await TikHubCredential._probe("bad")++ def test_handle_probe_response_reads_balance(self):+ response = httpx.Response(+ 200,+ json={"code": 200, "data": {"user_data": {"email": "[email protected]", "balance": 3.25}}},+ request=httpx.Request("GET", USER_INFO),+ )+ result = TikHubCredential._handle_probe_response(response)+ assert result.valid is True+ assert result.extra["balance"] == 3.25+++# ============================================================================+# 16. catalogue + assets+# ============================================================================+++class TestCatalogueAndAssets:+ def test_credential_providers_entry(self):+ config = json.loads((SERVER_DIR / "config" / "credential_providers.json").read_text(encoding="utf-8"))+ entry = config["providers"]["tikhub"]+ assert entry["name"] == "TikHub"+ assert entry["kind"] == "apiKey"+ assert entry["category"] == "scrapers"+ assert entry["usage_service"] == "tikhub"+ assert entry["icon_ref"] == "/api/schemas/credentials/tikhub/icon"+ assert "validate_as" not in entry+ field = entry["fields"][0]+ assert field["key"] == "apiKey"+ assert field["secret"] is True+ assert field["required"] is True++ def test_pricing_has_both_halves(self):+ pricing = json.loads((SERVER_DIR / "config" / "pricing.json").read_text(encoding="utf-8"))+ api = pricing["api"]["tikhub"]+ assert api["request"] == 0.001+ assert api["meta"] == 0.0+ op_map = pricing["operation_map"]["tikhub"]+ assert op_map["call"] == "request"+ assert op_map["fetch_url"] == "request"+ assert op_map["account"] == "meta"+ # every mapped operation resolves to a price line+ for target in op_map.values():+ assert target in api++ def test_node_allowlist_enables_node(self):+ allowlist = json.loads((SERVER_DIR / "config" / "node_allowlist.json").read_text(encoding="utf-8"))+ assert "tikhubAction" in allowlist["enabled_nodes"]+ assert "tikhubAction" not in allowlist.get("disabled_nodes", [])++ def test_visuals_skill_binding(self):+ visuals = json.loads((SERVER_DIR / "nodes" / "visuals.json").read_text(encoding="utf-8"))+ assert visuals["tikhubAction"]["skill"] == "tikhub-skill"++ def test_tool_name_snapshot(self):+ snapshot = json.loads((SERVER_DIR / "tests" / "fixtures" / "tool_names_snapshot.json").read_text(encoding="utf-8"))+ assert snapshot["tikhubAction"] == "tikhub_action"++ def test_plugin_folder_assets(self):+ meta = json.loads((PLUGIN_DIR / "meta.json").read_text(encoding="utf-8"))+ assert re.fullmatch(r"#[0-9a-fA-F]{6}", meta["color"])+ icon = PLUGIN_DIR / "icon.svg"+ assert icon.exists()+ assert "<svg" in icon.read_text(encoding="utf-8")++ def test_credential_icon_is_colocated(self):+ brand = SCRAPER_DIR / "tikhub.svg"+ assert brand.exists()+ assert "<svg" in brand.read_text(encoding="utf-8")+ assert TikHubCredential.get_icon_path() == brand++ def test_skill_exists_with_tool_binding(self):+ if not SKILL_MD.exists():+ pytest.skip(f"{SKILL_MD} not written yet (owned by the docs/skill agent)")+ text = SKILL_MD.read_text(encoding="utf-8")+ assert text.startswith("---")+ frontmatter = text.split("---", 2)[1]+ assert "name: tikhub-skill" in frontmatter+ assert "tikhub_action" in frontmatter+++# ============================================================================+# 17. usage tracking+# ============================================================================+++class TestUsageTracking:+ @respx.mock+ async def test_call_records_one_metric_after_2xx(self, harness):+ respx.get(FETCH_ONE_VIDEO).mock(return_value=httpx.Response(200, json=_ok({"x": 1})))++ result = await _execute(+ harness,+ {"operation": "call", "endpoint": "douyin_web.fetch_one_video", "params": {"aweme_id": "1"}},+ total_cost=0.001,+ )++ harness.assert_envelope(result, success=True)+ harness.database.save_api_usage_metric.assert_awaited_once()+ metric = harness.database.save_api_usage_metric.await_args.args[0]+ assert metric["service"] == "tikhub"+ assert metric["resource_count"] == 1+ assert metric["cost"] == 0.001+ assert metric["workflow_id"] == "test_workflow"+ assert metric["session_id"] == "test_session"+ assert metric["node_id"]+ assert "/api/v1/douyin/web/fetch_one_video" in str(metric.get("endpoint"))+ assert result["result"]["cost_usd"] == 0.001++ @respx.mock+ async def test_fetch_url_records_metric(self, harness):+ respx.get(HYBRID_VIDEO_DATA).mock(return_value=httpx.Response(200, json=_ok({}, router="/api/v1/hybrid/video_data")))++ result = await _execute(harness, {"operation": "fetch_url", "url": "https://v.douyin.com/abc/"})++ harness.assert_envelope(result, success=True)+ harness.database.save_api_usage_metric.assert_awaited_once()+ assert harness.database.save_api_usage_metric.await_args.args[0]["service"] == "tikhub"++ async def test_list_endpoints_records_nothing(self, harness):+ with respx.mock:+ result = await _execute(harness, {"operation": "list_endpoints", "platform": "douyin"})++ harness.assert_envelope(result, success=True)+ harness.database.save_api_usage_metric.assert_not_awaited()++ @respx.mock+ async def test_4xx_records_nothing(self, harness):+ respx.get(FETCH_ONE_VIDEO).mock(return_value=httpx.Response(403, json={"detail": "no balance"}))++ result = await _execute(+ harness,+ {"operation": "call", "endpoint": "douyin_web.fetch_one_video", "params": {"aweme_id": "1"}},+ )++ harness.assert_envelope(result, success=False)+ harness.database.save_api_usage_metric.assert_not_awaited()++ @respx.mock+ async def test_tracking_failure_never_fails_the_call(self, harness):+ respx.get(FETCH_ONE_VIDEO).mock(return_value=httpx.Response(200, json=_ok({"x": 1})))+ harness.database.save_api_usage_metric.side_effect = RuntimeError("db down")++ result = await _execute(+ harness,+ {"operation": "call", "endpoint": "douyin_web.fetch_one_video", "params": {"aweme_id": "1"}},+ )++ harness.assert_envelope(result, success=True)+ assert result["result"]["data"] == {"x": 1}+++# ============================================================================+# 18. skill endpoint ids+# ============================================================================+++class TestSkillEndpointIds:+ async def test_every_cited_endpoint_resolves(self):+ if not SKILL_MD.exists():+ pytest.skip(f"{SKILL_MD} not written yet (owned by the docs/skill agent)")+ text = SKILL_MD.read_text(encoding="utf-8")+ index = await _sdk.endpoint_index()+ resources = {info.resource for info in index}+ known = {info.endpoint for info in index}++ cited = set(re.findall(r"`([a-z0-9_]+\.[a-z0-9_]+)`", text))+ # Only ids whose resource half is a real SDK resource are endpoint+ # citations; `params.aweme_id`-style dotted paths are prose.+ candidates = {c for c in cited if c.split(".", 1)[0] in resources}+ assert candidates, "SKILL.md cites no endpoint ids"+ unknown = sorted(candidates - known)+ assert not unknown, f"SKILL.md cites endpoint ids missing from the SDK index: {unknown}"