Skip to content

Self-improving AI Employees for everyone

fix(apify): port the Apify node to apify-client 3.x

trohitgc76d62155ad0
diff --git a/docs-internal/node-logic-flows/web_automation/apifyActor.md b/docs-internal/node-logic-flows/web_automation/apifyActor.md
index 84aaf90c..ad87f0de 100644
--- a/docs-internal/node-logic-flows/web_automation/apifyActor.md
+++ b/docs-internal/node-logic-flows/web_automation/apifyActor.md
@@ -46,7 +46,7 @@ Params model is `ApifyActorParams` (field names are snake_case).
| `crawler_max_depth` | int (0-20) | `2` | no | `actor_id=apify/website-content-crawler` | -> `maxCrawlDepth` |
| `crawler_max_pages` | int (1-10000) | `50` | no | `actor_id=apify/website-content-crawler` | -> `maxCrawlPages` |
| `max_results` | int (1-10000) | `100` | no | - | Forwarded to `dataset.list_items(limit=...)` |
-| `timeout` | int (1-3600) | `300` | no | - | Run timeout in seconds (SDK forwards as `timeout_secs`) |
+| `timeout` | int (1-3600) | `300` | no | - | Run timeout in seconds (passed to the SDK as `run_timeout=timedelta(seconds=...)`) |
| `memory` | options (Literal: 128/256/512/1024/2048/4096/8192) | `1024` | no | - | Actor memory in MB (`memory_mbytes`) |
Actor presets (`actor_id`): `apify/instagram-scraper`, `clockworks/tiktok-scraper`,
@@ -72,9 +72,9 @@ Actor presets (`actor_id`): `apify/instagram-scraper`, `clockworks/tiktok-scrape
items: any[]; // dataset items up to maxResults
item_count: number;
dataset_id: string;
- compute_units: number; // mapped from usageTotalUsd on the run
- started_at: string;
- finished_at: string;
+ compute_units: number; // mapped from usage_total_usd on the run
+ started_at: string; // ISO 8601 with offset
+ finished_at: string; // ISO 8601 with offset, or "" while unset
}
```
@@ -94,16 +94,16 @@ flowchart TD
C1 -- yes --> Eaid[raise NodeUserError: Actor ID is required]
C1 -- no --> D[_build_actor_input params.model_dump<br/>parse actor_input JSON else {}<br/>merge per-actor quick helpers]
D --> E[Read timeout, max_results, memory_mbytes]
- E --> F[client.actor id .call<br/>run_input, timeout_secs, memory_mbytes]
- F -- run_info None --> Enone[raise NodeUserError:<br/>no result returned]
- F -- ok --> G{run_status?}
- G -- FAILED --> Gf[raise NodeUserError errorMessage]
+ E --> F[client.actor id .call<br/>run_input, run_timeout, memory_mbytes, logger=None]
+ F -- run None --> Enone[raise NodeUserError:<br/>no result returned]
+ F -- ok --> G{run.status?}
+ G -- FAILED --> Gf[raise NodeUserError status_message]
G -- TIMED-OUT --> Gt[raise NodeUserError: Actor timed out]
G -- ABORTED --> Ga[raise NodeUserError: Actor run was aborted]
G -- other --> H{dataset_id truthy?}
H -- yes --> H1[client.dataset id .list_items limit=max_results<br/>items = listing.items]
H -- no --> H2[items = empty list]
- H1 & H2 --> I[Build ApifyActorOutput:<br/>run_id, actor_id, status, items, item_count,<br/>dataset_id, compute_units=usageTotalUsd,<br/>started_at, finished_at]
+ H1 & H2 --> I[Build ApifyActorOutput:<br/>run_id, actor_id, status, items, item_count,<br/>dataset_id, compute_units=usage_total_usd,<br/>started_at, finished_at]
I --> J[Return BaseNode success envelope]
```
@@ -147,20 +147,22 @@ flowchart TD
- **Credentials**: `auth_service.get_api_key("apify", "default")` -> stored in
the `EncryptedAPIKey` table under provider `apify`.
- **Services**: Apify platform (api.apify.com).
-- **Python packages**: `apify-client` (>= async variant `ApifyClientAsync`).
+- **Python packages**: `apify-client` 3.1 or later (`ApifyClientAsync`). Since 3.x, `ActorClientAsync.call` takes `run_timeout` as a `timedelta` (1.x's `timeout_secs` raises `TypeError`). It returns a pydantic `Run`, and `user("me").get()` returns `UserPrivateInfo`, or `UserPublicInfo` when the token may not read private info. The tests build their fakes with `create_autospec` of the SDK's resource clients, so a call the installed SDK would reject fails them too.
- **Environment variables**: none (token lives in credentials DB).
## Edge cases & known limits
- **Invalid `actorInput` JSON is silent**: the run is sent with `{}` and may
succeed unexpectedly (actor uses defaults) or fail with a validation error
- that surfaces in `errorMessage`.
+ that surfaces in the run's `status_message`.
- **Quick helpers overwrite raw input**: setting both `actorInput` JSON AND
the per-actor helper means the helper wins. This is not documented in the
frontend.
- **No streaming**: the handler blocks until the actor finishes or the SDK
- raises. `timeout` is the hard cap.
-- **`compute_units` is misleading**: the value returned is `run.usageTotalUsd`
+ raises, and passes `logger=None` so the SDK does not copy the actor's log
+ into the app's. `timeout` caps only the time the run spends RUNNING; a run
+ queued as READY (the account's concurrent runs are full) can wait longer.
+- **`compute_units` is misleading**: the value returned is `run.usage_total_usd`
(USD), not compute units. The key name is preserved for backward compat.
- **Error rewriting is substring-based (validation only)**: `validate_apify_token`
pattern-matches `401`/`Unauthorized` to a generic message; the runtime `run`
diff --git a/server/nodes/scraper/apify_actor/__init__.py b/server/nodes/scraper/apify_actor/__init__.py
index 7fefeb29..053f092a 100644
--- a/server/nodes/scraper/apify_actor/__init__.py
+++ b/server/nodes/scraper/apify_actor/__init__.py
@@ -8,6 +8,7 @@ raw ``actorInput`` JSON for common actors.
from __future__ import annotations
import json
+from datetime import timedelta
from typing import Any, Dict, List, Literal, Optional
from pydantic import BaseModel, ConfigDict, Field
@@ -101,12 +102,14 @@ async def validate_apify_token(api_token: str) -> Dict[str, Any]:
user_info = await client.user("me").get()
if not user_info:
return {"valid": False, "error": "Could not fetch user info"}
- plan = user_info.get("plan")
+ # A token that may not read private info gets the public model,
+ # which has no email and no plan.
+ plan = getattr(user_info, "plan", None)
return {
"valid": True,
- "username": user_info.get("username", ""),
- "email": user_info.get("email", ""),
- "plan": plan.get("id", "free") if isinstance(plan, dict) else "free",
+ "username": user_info.username or "",
+ "email": getattr(user_info, "email", None) or "",
+ "plan": (plan.id if plan is not None else None) or "free",
}
except Exception as e:
logger.error(f"[Apify] Token validation error: {e}")
@@ -284,21 +287,23 @@ class ApifyActorNode(ActionNode):
logger.info(
f"[Apify] Running actor {actor_id} " f"timeout={timeout_secs}s memory={memory_mbytes}MB",
)
- run_info = await client.actor(actor_id).call(
+ run = await client.actor(actor_id).call(
run_input=actor_input,
- timeout_secs=timeout_secs,
+ run_timeout=timedelta(seconds=timeout_secs),
memory_mbytes=memory_mbytes,
+ # The SDK would otherwise stream the actor's own log into ours.
+ logger=None,
)
- if run_info is None:
+ if run is None:
raise NodeUserError("Actor run failed - no result returned")
- status = run_info.get("status", "UNKNOWN")
- run_id = run_info.get("id", "")
- dataset_id = run_info.get("defaultDatasetId", "")
+ status = run.status
+ run_id = run.id
+ dataset_id = run.default_dataset_id
if status == "FAILED":
- raise NodeUserError(run_info.get("errorMessage", "Actor run failed"))
+ raise NodeUserError(run.status_message or "Actor run failed")
if status == "TIMED-OUT":
raise NodeUserError("Actor timed out. Try increasing the timeout.")
if status == "ABORTED":
@@ -317,7 +322,7 @@ class ApifyActorNode(ActionNode):
items=items,
item_count=len(items),
dataset_id=dataset_id,
- compute_units=run_info.get("usageTotalUsd", 0),
- started_at=run_info.get("startedAt", ""),
- finished_at=run_info.get("finishedAt", ""),
+ compute_units=run.usage_total_usd or 0,
+ started_at=run.started_at.isoformat(),
+ finished_at=run.finished_at.isoformat() if run.finished_at else "",
)
diff --git a/server/pyproject.toml b/server/pyproject.toml
index ad54bf52..1ce4abad 100644
--- a/server/pyproject.toml
+++ b/server/pyproject.toml
@@ -134,8 +134,9 @@ dependencies = [
# Twitter/X API
"xdk>=0.1.0",
- # Apify Web Scraping
- "apify-client>=1.8.0",
+ # Apify Web Scraping. Floor 3.1: the node uses the 3.x API (run_timeout,
+ # pydantic Run and user models), which 1.x and 2.x do not have.
+ "apify-client>=3.1.0",
# TikHub social scraping SDK. Auto-generated from TikHub's openapi.json;
# capped below 3 because the resource/method surface regenerates per
diff --git a/server/tests/credentials/test_onepassword_runtime_adapters.py b/server/tests/credentials/test_onepassword_runtime_adapters.py
index 38636f2f..89a9e2ec 100644
--- a/server/tests/credentials/test_onepassword_runtime_adapters.py
+++ b/server/tests/credentials/test_onepassword_runtime_adapters.py
@@ -23,9 +23,16 @@ def runtime_auth(monkeypatch, key="runtime-key"):
async def test_apify_activity_passes_trusted_principal_to_private_resolution(monkeypatch):
+ from datetime import datetime, timezone
+
+ from apify_client._models import Run
from nodes.scraper.apify_actor import ApifyActorNode, ApifyActorParams
auth = runtime_auth(monkeypatch)
- actor = SimpleNamespace(call=AsyncMock(return_value={"id": "run", "status": "SUCCEEDED"}))
+ run = Run.model_construct(
+ id="run", status="SUCCEEDED", status_message=None, default_dataset_id="", usage_total_usd=None,
+ started_at=datetime(2026, 4, 15, tzinfo=timezone.utc), finished_at=None,
+ )
+ actor = SimpleNamespace(call=AsyncMock(return_value=run))
sdk = MagicMock(return_value=SimpleNamespace(actor=lambda name: actor))
monkeypatch.setitem(sys.modules, "apify_client", SimpleNamespace(ApifyClientAsync=sdk))
ctx = NodeContext(node_id="scraper", node_type="apifyActor", raw={"user_id": "alice"})
diff --git a/server/tests/nodes/test_web_automation.py b/server/tests/nodes/test_web_automation.py
index cc036c6c..047ca33e 100644
--- a/server/tests/nodes/test_web_automation.py
+++ b/server/tests/nodes/test_web_automation.py
@@ -9,13 +9,14 @@ these indicates the docs (and the user-visible contract) need to be updated too.
All external side-effects are mocked:
- crawleeScraper: patches the crawlee crawler classes the handler imports
lazily (`BeautifulSoupCrawler`, `PlaywrightCrawler`).
- - apifyActor: patches the ApifyClientAsync singleton returned from
- `services.handlers.apify._get_apify_client`.
+ - apifyActor: patches `apify_client.ApifyClientAsync` with a client whose
+ resource clients are autospecs of the installed SDK's.
"""
from __future__ import annotations
-from unittest.mock import AsyncMock, MagicMock, patch
+from datetime import datetime, timedelta, timezone
+from unittest.mock import AsyncMock, MagicMock, create_autospec, patch
import pytest
@@ -150,26 +151,52 @@ class TestCrawleeScraper:
# ============================================================================
+def _run(**fields):
+ """An SDK ``Run`` holding the fields the node reads; the API sends many more."""
+ from apify_client._models import Run
+
+ values = {
+ "id": "run_1",
+ "status": "SUCCEEDED",
+ "status_message": None,
+ "default_dataset_id": "",
+ "usage_total_usd": None,
+ "started_at": datetime(2026, 4, 15, tzinfo=timezone.utc),
+ "finished_at": None,
+ }
+ values.update(fields)
+ return Run.model_construct(**values)
+
+
def _make_fake_apify_client(
*,
- run_info: dict | None = None,
+ run=None,
items: list | None = None,
call_exc: Exception | None = None,
):
- """Build a MagicMock that mimics ApifyClientAsync surface used by the handler."""
+ """A client whose resource clients carry the installed SDK's signatures.
+
+ ``create_autospec`` fails a call with a keyword the SDK does not take, as
+ the SDK itself would: apify-client 3.x dropped 1.x's ``timeout_secs``, and
+ every run raised TypeError while these fakes accepted anything.
+ """
+ from apify_client._resource_clients import ActorClientAsync, DatasetClientAsync
+ from apify_client._resource_clients.dataset import DatasetItemsPage
+
client = MagicMock(name="ApifyClientAsync")
- actor_client = MagicMock(name="ActorClient")
+ actor_client = create_autospec(ActorClientAsync, instance=True)
if call_exc:
- actor_client.call = AsyncMock(side_effect=call_exc)
+ actor_client.call.side_effect = call_exc
else:
- actor_client.call = AsyncMock(return_value=run_info)
+ actor_client.call.return_value = run
client.actor = MagicMock(return_value=actor_client)
- list_result = MagicMock()
- list_result.items = items or []
- dataset_client = MagicMock(name="DatasetClient")
- dataset_client.list_items = AsyncMock(return_value=list_result)
+ page_items = items or []
+ dataset_client = create_autospec(DatasetClientAsync, instance=True)
+ dataset_client.list_items.return_value = DatasetItemsPage(
+ items=page_items, total=len(page_items), offset=0, count=len(page_items), limit=len(page_items), desc=False,
+ )
client.dataset = MagicMock(return_value=dataset_client)
return client
@@ -177,16 +204,15 @@ def _make_fake_apify_client(
class TestApifyActor:
async def test_happy_path_returns_dataset_items(self, harness):
- run_info = {
- "id": "run_123",
- "status": "SUCCEEDED",
- "defaultDatasetId": "ds_456",
- "usageTotalUsd": 0.02,
- "startedAt": "2026-04-15T00:00:00Z",
- "finishedAt": "2026-04-15T00:00:05Z",
- }
+ run = _run(
+ id="run_123",
+ default_dataset_id="ds_456",
+ usage_total_usd=0.02,
+ started_at=datetime(2026, 4, 15, tzinfo=timezone.utc),
+ finished_at=datetime(2026, 4, 15, 0, 0, 5, tzinfo=timezone.utc),
+ )
items = [{"url": "https://a.example"}, {"url": "https://b.example"}]
- fake_client = _make_fake_apify_client(run_info=run_info, items=items)
+ fake_client = _make_fake_apify_client(run=run, items=items)
with patched_container(auth_api_keys={"apify": "tk_apify"}), patch("apify_client.ApifyClientAsync", return_value=fake_client):
result = await harness.execute(
@@ -221,6 +247,8 @@ class TestApifyActor:
assert payload["items"] == items
assert payload["dataset_id"] == "ds_456"
assert payload["compute_units"] == 0.02
+ assert payload["started_at"] == "2026-04-15T00:00:00+00:00"
+ assert payload["finished_at"] == "2026-04-15T00:00:05+00:00"
# Quick-helper merge: call_args run_input should contain directUrls
call_kwargs = fake_client.actor.return_value.call.await_args.kwargs
@@ -230,6 +258,8 @@ class TestApifyActor:
"https://instagram.com/b",
]
assert call_kwargs["memory_mbytes"] == 1024 # default
+ assert call_kwargs["run_timeout"] == timedelta(seconds=300) # default
+ assert call_kwargs["logger"] is None # the actor's log is not streamed into ours
# dataset list called with the right limit
fake_client.dataset.return_value.list_items.assert_awaited_once_with(limit=50)
@@ -245,7 +275,7 @@ class TestApifyActor:
assert "apify api token not configured" in result["error"].lower()
async def test_missing_actor_id_returns_validation_error(self, harness):
- fake_client = _make_fake_apify_client(run_info={})
+ fake_client = _make_fake_apify_client(run=_run())
with patched_container(auth_api_keys={"apify": "tk"}), patch("apify_client.ApifyClientAsync", return_value=fake_client):
result = await harness.execute("apifyActor", {"actor_id": ""})
@@ -254,13 +284,8 @@ class TestApifyActor:
assert "invalid parameters" in result["error"].lower()
async def test_failed_run_status_returns_error_envelope(self, harness):
- run_info = {
- "id": "run_err",
- "status": "FAILED",
- "errorMessage": "Actor crashed at step 3",
- "defaultDatasetId": "",
- }
- fake_client = _make_fake_apify_client(run_info=run_info, items=[])
+ run = _run(id="run_err", status="FAILED", status_message="Actor crashed at step 3")
+ fake_client = _make_fake_apify_client(run=run, items=[])
with patched_container(auth_api_keys={"apify": "tk"}), patch("apify_client.ApifyClientAsync", return_value=fake_client):
result = await harness.execute("apifyActor", {"actor_id": "apify/instagram-scraper"})
@@ -278,15 +303,8 @@ class TestApifyActor:
assert "401" in result["error"].lower() or "unauthorized" in result["error"].lower()
async def test_invalid_actor_input_json_silently_becomes_empty_dict(self, harness):
- run_info = {
- "id": "run_ok",
- "status": "SUCCEEDED",
- "defaultDatasetId": "ds",
- "usageTotalUsd": 0,
- "startedAt": "",
- "finishedAt": "",
- }
- fake_client = _make_fake_apify_client(run_info=run_info, items=[])
+ run = _run(id="run_ok", default_dataset_id="ds", usage_total_usd=0)
+ fake_client = _make_fake_apify_client(run=run, items=[])
with patched_container(auth_api_keys={"apify": "tk"}), patch("apify_client.ApifyClientAsync", return_value=fake_client):
result = await harness.execute(
@@ -301,3 +319,38 @@ class TestApifyActor:
call_kwargs = fake_client.actor.return_value.call.await_args.kwargs
# malformed JSON becomes {} not a parse error
assert call_kwargs["run_input"] == {}
+
+
+class TestApifyTokenValidation:
+ """The Credentials modal's key check. On apify-client 3.x the SDK
+ returns models, not dicts, so a valid key read as invalid."""
+
+ async def _validate(self, user_info):
+ from apify_client._resource_clients import UserClientAsync
+ from nodes.scraper.apify_actor import validate_apify_token
+
+ user_client = create_autospec(UserClientAsync, instance=True)
+ user_client.get.return_value = user_info
+ client = MagicMock(name="ApifyClientAsync")
+ client.user = MagicMock(return_value=user_client)
+ with patch("apify_client.ApifyClientAsync", return_value=client):
+ return await validate_apify_token("tk")
+
+ async def test_private_info_gives_username_email_and_plan(self):
+ from apify_client._models import Plan, UserPrivateInfo
+
+ info = UserPrivateInfo.model_construct(
+ username="ada", email="[email protected]", plan=Plan.model_construct(id="PERSONAL"),
+ )
+ assert await self._validate(info) == {
+ "valid": True, "username": "ada", "email": "[email protected]", "plan": "PERSONAL",
+ }
+
+ async def test_public_info_has_no_email_or_plan(self):
+ from apify_client._models import UserPublicInfo
+
+ info = UserPublicInfo.model_construct(username="ada")
+ assert await self._validate(info) == {"valid": True, "username": "ada", "email": "", "plan": "free"}
+
+ async def test_missing_user_info_is_invalid(self):
+ assert (await self._validate(None))["valid"] is False
diff --git a/server/uv.lock b/server/uv.lock
index e4af6848..41204f38 100644
--- a/server/uv.lock
+++ b/server/uv.lock
@@ -2042,7 +2042,7 @@ requires-dist = [
{ name = "aiohttp", specifier = ">=3.14.4" },
{ name = "aiosqlite", specifier = ">=0.20.0" },
{ name = "anthropic", specifier = "==0.96.0" },
- { name = "apify-client", specifier = ">=1.8.0" },
+ { name = "apify-client", specifier = ">=3.1.0" },
{ name = "asyncpg", specifier = ">=0.30.0,<1" },
{ name = "bcrypt", specifier = ">=4.2.0" },
{ name = "beautifulsoup4", marker = "extra == 'docs'", specifier = ">=4.12.0" },