Self-improving AI Employees for everyone
Python ai-agentsai-employeeai-employeesassistantautonomous-agentsco-employeescompanygrok-botinstinctloop-agentsmuseown-your-datapersonalself-hosted
fix(apify): port the Apify node to apify-client 3.x
trohitgc76d62155ad0
diff --git a/docs-internal/node-logic-flows/web_automation/apifyActor.md b/docs-internal/node-logic-flows/web_automation/apifyActor.mdindex 84aaf90c..ad87f0de 100644--- a/docs-internal/node-logic-flows/web_automation/apifyActor.md+++ b/docs-internal/node-logic-flows/web_automation/apifyActor.md@@ -46,7 +46,7 @@ Params model is `ApifyActorParams` (field names are snake_case).| `crawler_max_depth` | int (0-20) | `2` | no | `actor_id=apify/website-content-crawler` | -> `maxCrawlDepth` || `crawler_max_pages` | int (1-10000) | `50` | no | `actor_id=apify/website-content-crawler` | -> `maxCrawlPages` || `max_results` | int (1-10000) | `100` | no | - | Forwarded to `dataset.list_items(limit=...)` |-| `timeout` | int (1-3600) | `300` | no | - | Run timeout in seconds (SDK forwards as `timeout_secs`) |+| `timeout` | int (1-3600) | `300` | no | - | Run timeout in seconds (passed to the SDK as `run_timeout=timedelta(seconds=...)`) || `memory` | options (Literal: 128/256/512/1024/2048/4096/8192) | `1024` | no | - | Actor memory in MB (`memory_mbytes`) |Actor presets (`actor_id`): `apify/instagram-scraper`, `clockworks/tiktok-scraper`,@@ -72,9 +72,9 @@ Actor presets (`actor_id`): `apify/instagram-scraper`, `clockworks/tiktok-scrapeitems: any[]; // dataset items up to maxResultsitem_count: number;dataset_id: string;- compute_units: number; // mapped from usageTotalUsd on the run- started_at: string;- finished_at: string;+ compute_units: number; // mapped from usage_total_usd on the run+ started_at: string; // ISO 8601 with offset+ finished_at: string; // ISO 8601 with offset, or "" while unset}```@@ -94,16 +94,16 @@ flowchart TDC1 -- yes --> Eaid[raise NodeUserError: Actor ID is required]C1 -- no --> D[_build_actor_input params.model_dump<br/>parse actor_input JSON else {}<br/>merge per-actor quick helpers]D --> E[Read timeout, max_results, memory_mbytes]- E --> F[client.actor id .call<br/>run_input, timeout_secs, memory_mbytes]- F -- run_info None --> Enone[raise NodeUserError:<br/>no result returned]- F -- ok --> G{run_status?}- G -- FAILED --> Gf[raise NodeUserError errorMessage]+ E --> F[client.actor id .call<br/>run_input, run_timeout, memory_mbytes, logger=None]+ F -- run None --> Enone[raise NodeUserError:<br/>no result returned]+ F -- ok --> G{run.status?}+ G -- FAILED --> Gf[raise NodeUserError status_message]G -- TIMED-OUT --> Gt[raise NodeUserError: Actor timed out]G -- ABORTED --> Ga[raise NodeUserError: Actor run was aborted]G -- other --> H{dataset_id truthy?}H -- yes --> H1[client.dataset id .list_items limit=max_results<br/>items = listing.items]H -- no --> H2[items = empty list]- H1 & H2 --> I[Build ApifyActorOutput:<br/>run_id, actor_id, status, items, item_count,<br/>dataset_id, compute_units=usageTotalUsd,<br/>started_at, finished_at]+ H1 & H2 --> I[Build ApifyActorOutput:<br/>run_id, actor_id, status, items, item_count,<br/>dataset_id, compute_units=usage_total_usd,<br/>started_at, finished_at]I --> J[Return BaseNode success envelope]```@@ -147,20 +147,22 @@ flowchart TD- **Credentials**: `auth_service.get_api_key("apify", "default")` -> stored inthe `EncryptedAPIKey` table under provider `apify`.- **Services**: Apify platform (api.apify.com).-- **Python packages**: `apify-client` (>= async variant `ApifyClientAsync`).+- **Python packages**: `apify-client` 3.1 or later (`ApifyClientAsync`). Since 3.x, `ActorClientAsync.call` takes `run_timeout` as a `timedelta` (1.x's `timeout_secs` raises `TypeError`). It returns a pydantic `Run`, and `user("me").get()` returns `UserPrivateInfo`, or `UserPublicInfo` when the token may not read private info. The tests build their fakes with `create_autospec` of the SDK's resource clients, so a call the installed SDK would reject fails them too.- **Environment variables**: none (token lives in credentials DB).## Edge cases & known limits- **Invalid `actorInput` JSON is silent**: the run is sent with `{}` and maysucceed unexpectedly (actor uses defaults) or fail with a validation error- that surfaces in `errorMessage`.+ that surfaces in the run's `status_message`.- **Quick helpers overwrite raw input**: setting both `actorInput` JSON ANDthe per-actor helper means the helper wins. This is not documented in thefrontend.- **No streaming**: the handler blocks until the actor finishes or the SDK- raises. `timeout` is the hard cap.-- **`compute_units` is misleading**: the value returned is `run.usageTotalUsd`+ raises, and passes `logger=None` so the SDK does not copy the actor's log+ into the app's. `timeout` caps only the time the run spends RUNNING; a run+ queued as READY (the account's concurrent runs are full) can wait longer.+- **`compute_units` is misleading**: the value returned is `run.usage_total_usd`(USD), not compute units. The key name is preserved for backward compat.- **Error rewriting is substring-based (validation only)**: `validate_apify_token`pattern-matches `401`/`Unauthorized` to a generic message; the runtime `run`diff --git a/server/nodes/scraper/apify_actor/__init__.py b/server/nodes/scraper/apify_actor/__init__.pyindex 7fefeb29..053f092a 100644--- a/server/nodes/scraper/apify_actor/__init__.py+++ b/server/nodes/scraper/apify_actor/__init__.py@@ -8,6 +8,7 @@ raw ``actorInput`` JSON for common actors.from __future__ import annotationsimport json+from datetime import timedeltafrom typing import Any, Dict, List, Literal, Optionalfrom pydantic import BaseModel, ConfigDict, Field@@ -101,12 +102,14 @@ async def validate_apify_token(api_token: str) -> Dict[str, Any]:user_info = await client.user("me").get()if not user_info:return {"valid": False, "error": "Could not fetch user info"}- plan = user_info.get("plan")+ # A token that may not read private info gets the public model,+ # which has no email and no plan.+ plan = getattr(user_info, "plan", None)return {"valid": True,- "username": user_info.get("username", ""),- "email": user_info.get("email", ""),- "plan": plan.get("id", "free") if isinstance(plan, dict) else "free",+ "username": user_info.username or "",+ "email": getattr(user_info, "email", None) or "",+ "plan": (plan.id if plan is not None else None) or "free",}except Exception as e:logger.error(f"[Apify] Token validation error: {e}")@@ -284,21 +287,23 @@ class ApifyActorNode(ActionNode):logger.info(f"[Apify] Running actor {actor_id} " f"timeout={timeout_secs}s memory={memory_mbytes}MB",)- run_info = await client.actor(actor_id).call(+ run = await client.actor(actor_id).call(run_input=actor_input,- timeout_secs=timeout_secs,+ run_timeout=timedelta(seconds=timeout_secs),memory_mbytes=memory_mbytes,+ # The SDK would otherwise stream the actor's own log into ours.+ logger=None,)- if run_info is None:+ if run is None:raise NodeUserError("Actor run failed - no result returned")- status = run_info.get("status", "UNKNOWN")- run_id = run_info.get("id", "")- dataset_id = run_info.get("defaultDatasetId", "")+ status = run.status+ run_id = run.id+ dataset_id = run.default_dataset_idif status == "FAILED":- raise NodeUserError(run_info.get("errorMessage", "Actor run failed"))+ raise NodeUserError(run.status_message or "Actor run failed")if status == "TIMED-OUT":raise NodeUserError("Actor timed out. Try increasing the timeout.")if status == "ABORTED":@@ -317,7 +322,7 @@ class ApifyActorNode(ActionNode):items=items,item_count=len(items),dataset_id=dataset_id,- compute_units=run_info.get("usageTotalUsd", 0),- started_at=run_info.get("startedAt", ""),- finished_at=run_info.get("finishedAt", ""),+ compute_units=run.usage_total_usd or 0,+ started_at=run.started_at.isoformat(),+ finished_at=run.finished_at.isoformat() if run.finished_at else "",)diff --git a/server/pyproject.toml b/server/pyproject.tomlindex ad54bf52..1ce4abad 100644--- a/server/pyproject.toml+++ b/server/pyproject.toml@@ -134,8 +134,9 @@ dependencies = [# Twitter/X API"xdk>=0.1.0",- # Apify Web Scraping- "apify-client>=1.8.0",+ # Apify Web Scraping. Floor 3.1: the node uses the 3.x API (run_timeout,+ # pydantic Run and user models), which 1.x and 2.x do not have.+ "apify-client>=3.1.0",# TikHub social scraping SDK. Auto-generated from TikHub's openapi.json;# capped below 3 because the resource/method surface regenerates perdiff --git a/server/tests/credentials/test_onepassword_runtime_adapters.py b/server/tests/credentials/test_onepassword_runtime_adapters.pyindex 38636f2f..89a9e2ec 100644--- a/server/tests/credentials/test_onepassword_runtime_adapters.py+++ b/server/tests/credentials/test_onepassword_runtime_adapters.py@@ -23,9 +23,16 @@ def runtime_auth(monkeypatch, key="runtime-key"):async def test_apify_activity_passes_trusted_principal_to_private_resolution(monkeypatch):+ from datetime import datetime, timezone++ from apify_client._models import Runfrom nodes.scraper.apify_actor import ApifyActorNode, ApifyActorParamsauth = runtime_auth(monkeypatch)- actor = SimpleNamespace(call=AsyncMock(return_value={"id": "run", "status": "SUCCEEDED"}))+ run = Run.model_construct(+ id="run", status="SUCCEEDED", status_message=None, default_dataset_id="", usage_total_usd=None,+ started_at=datetime(2026, 4, 15, tzinfo=timezone.utc), finished_at=None,+ )+ actor = SimpleNamespace(call=AsyncMock(return_value=run))sdk = MagicMock(return_value=SimpleNamespace(actor=lambda name: actor))monkeypatch.setitem(sys.modules, "apify_client", SimpleNamespace(ApifyClientAsync=sdk))ctx = NodeContext(node_id="scraper", node_type="apifyActor", raw={"user_id": "alice"})diff --git a/server/tests/nodes/test_web_automation.py b/server/tests/nodes/test_web_automation.pyindex cc036c6c..047ca33e 100644--- a/server/tests/nodes/test_web_automation.py+++ b/server/tests/nodes/test_web_automation.py@@ -9,13 +9,14 @@ these indicates the docs (and the user-visible contract) need to be updated too.All external side-effects are mocked:- crawleeScraper: patches the crawlee crawler classes the handler importslazily (`BeautifulSoupCrawler`, `PlaywrightCrawler`).- - apifyActor: patches the ApifyClientAsync singleton returned from- `services.handlers.apify._get_apify_client`.+ - apifyActor: patches `apify_client.ApifyClientAsync` with a client whose+ resource clients are autospecs of the installed SDK's."""from __future__ import annotations-from unittest.mock import AsyncMock, MagicMock, patch+from datetime import datetime, timedelta, timezone+from unittest.mock import AsyncMock, MagicMock, create_autospec, patchimport pytest@@ -150,26 +151,52 @@ class TestCrawleeScraper:# ============================================================================+def _run(**fields):+ """An SDK ``Run`` holding the fields the node reads; the API sends many more."""+ from apify_client._models import Run++ values = {+ "id": "run_1",+ "status": "SUCCEEDED",+ "status_message": None,+ "default_dataset_id": "",+ "usage_total_usd": None,+ "started_at": datetime(2026, 4, 15, tzinfo=timezone.utc),+ "finished_at": None,+ }+ values.update(fields)+ return Run.model_construct(**values)++def _make_fake_apify_client(*,- run_info: dict | None = None,+ run=None,items: list | None = None,call_exc: Exception | None = None,):- """Build a MagicMock that mimics ApifyClientAsync surface used by the handler."""+ """A client whose resource clients carry the installed SDK's signatures.++ ``create_autospec`` fails a call with a keyword the SDK does not take, as+ the SDK itself would: apify-client 3.x dropped 1.x's ``timeout_secs``, and+ every run raised TypeError while these fakes accepted anything.+ """+ from apify_client._resource_clients import ActorClientAsync, DatasetClientAsync+ from apify_client._resource_clients.dataset import DatasetItemsPage+client = MagicMock(name="ApifyClientAsync")- actor_client = MagicMock(name="ActorClient")+ actor_client = create_autospec(ActorClientAsync, instance=True)if call_exc:- actor_client.call = AsyncMock(side_effect=call_exc)+ actor_client.call.side_effect = call_excelse:- actor_client.call = AsyncMock(return_value=run_info)+ actor_client.call.return_value = runclient.actor = MagicMock(return_value=actor_client)- list_result = MagicMock()- list_result.items = items or []- dataset_client = MagicMock(name="DatasetClient")- dataset_client.list_items = AsyncMock(return_value=list_result)+ page_items = items or []+ dataset_client = create_autospec(DatasetClientAsync, instance=True)+ dataset_client.list_items.return_value = DatasetItemsPage(+ items=page_items, total=len(page_items), offset=0, count=len(page_items), limit=len(page_items), desc=False,+ )client.dataset = MagicMock(return_value=dataset_client)return client@@ -177,16 +204,15 @@ def _make_fake_apify_client(class TestApifyActor:async def test_happy_path_returns_dataset_items(self, harness):- run_info = {- "id": "run_123",- "status": "SUCCEEDED",- "defaultDatasetId": "ds_456",- "usageTotalUsd": 0.02,- "startedAt": "2026-04-15T00:00:00Z",- "finishedAt": "2026-04-15T00:00:05Z",- }+ run = _run(+ id="run_123",+ default_dataset_id="ds_456",+ usage_total_usd=0.02,+ started_at=datetime(2026, 4, 15, tzinfo=timezone.utc),+ finished_at=datetime(2026, 4, 15, 0, 0, 5, tzinfo=timezone.utc),+ )items = [{"url": "https://a.example"}, {"url": "https://b.example"}]- fake_client = _make_fake_apify_client(run_info=run_info, items=items)+ fake_client = _make_fake_apify_client(run=run, items=items)with patched_container(auth_api_keys={"apify": "tk_apify"}), patch("apify_client.ApifyClientAsync", return_value=fake_client):result = await harness.execute(@@ -221,6 +247,8 @@ class TestApifyActor:assert payload["items"] == itemsassert payload["dataset_id"] == "ds_456"assert payload["compute_units"] == 0.02+ assert payload["started_at"] == "2026-04-15T00:00:00+00:00"+ assert payload["finished_at"] == "2026-04-15T00:00:05+00:00"# Quick-helper merge: call_args run_input should contain directUrlscall_kwargs = fake_client.actor.return_value.call.await_args.kwargs@@ -230,6 +258,8 @@ class TestApifyActor:"https://instagram.com/b",]assert call_kwargs["memory_mbytes"] == 1024 # default+ assert call_kwargs["run_timeout"] == timedelta(seconds=300) # default+ assert call_kwargs["logger"] is None # the actor's log is not streamed into ours# dataset list called with the right limitfake_client.dataset.return_value.list_items.assert_awaited_once_with(limit=50)@@ -245,7 +275,7 @@ class TestApifyActor:assert "apify api token not configured" in result["error"].lower()async def test_missing_actor_id_returns_validation_error(self, harness):- fake_client = _make_fake_apify_client(run_info={})+ fake_client = _make_fake_apify_client(run=_run())with patched_container(auth_api_keys={"apify": "tk"}), patch("apify_client.ApifyClientAsync", return_value=fake_client):result = await harness.execute("apifyActor", {"actor_id": ""})@@ -254,13 +284,8 @@ class TestApifyActor:assert "invalid parameters" in result["error"].lower()async def test_failed_run_status_returns_error_envelope(self, harness):- run_info = {- "id": "run_err",- "status": "FAILED",- "errorMessage": "Actor crashed at step 3",- "defaultDatasetId": "",- }- fake_client = _make_fake_apify_client(run_info=run_info, items=[])+ run = _run(id="run_err", status="FAILED", status_message="Actor crashed at step 3")+ fake_client = _make_fake_apify_client(run=run, items=[])with patched_container(auth_api_keys={"apify": "tk"}), patch("apify_client.ApifyClientAsync", return_value=fake_client):result = await harness.execute("apifyActor", {"actor_id": "apify/instagram-scraper"})@@ -278,15 +303,8 @@ class TestApifyActor:assert "401" in result["error"].lower() or "unauthorized" in result["error"].lower()async def test_invalid_actor_input_json_silently_becomes_empty_dict(self, harness):- run_info = {- "id": "run_ok",- "status": "SUCCEEDED",- "defaultDatasetId": "ds",- "usageTotalUsd": 0,- "startedAt": "",- "finishedAt": "",- }- fake_client = _make_fake_apify_client(run_info=run_info, items=[])+ run = _run(id="run_ok", default_dataset_id="ds", usage_total_usd=0)+ fake_client = _make_fake_apify_client(run=run, items=[])with patched_container(auth_api_keys={"apify": "tk"}), patch("apify_client.ApifyClientAsync", return_value=fake_client):result = await harness.execute(@@ -301,3 +319,38 @@ class TestApifyActor:call_kwargs = fake_client.actor.return_value.call.await_args.kwargs# malformed JSON becomes {} not a parse errorassert call_kwargs["run_input"] == {}+++class TestApifyTokenValidation:+ """The Credentials modal's key check. On apify-client 3.x the SDK+ returns models, not dicts, so a valid key read as invalid."""++ async def _validate(self, user_info):+ from apify_client._resource_clients import UserClientAsync+ from nodes.scraper.apify_actor import validate_apify_token++ user_client = create_autospec(UserClientAsync, instance=True)+ user_client.get.return_value = user_info+ client = MagicMock(name="ApifyClientAsync")+ client.user = MagicMock(return_value=user_client)+ with patch("apify_client.ApifyClientAsync", return_value=client):+ return await validate_apify_token("tk")++ async def test_private_info_gives_username_email_and_plan(self):+ from apify_client._models import Plan, UserPrivateInfo++ info = UserPrivateInfo.model_construct(+ username="ada", email="[email protected]", plan=Plan.model_construct(id="PERSONAL"),+ )+ assert await self._validate(info) == {+ "valid": True, "username": "ada", "email": "[email protected]", "plan": "PERSONAL",+ }++ async def test_public_info_has_no_email_or_plan(self):+ from apify_client._models import UserPublicInfo++ info = UserPublicInfo.model_construct(username="ada")+ assert await self._validate(info) == {"valid": True, "username": "ada", "email": "", "plan": "free"}++ async def test_missing_user_info_is_invalid(self):+ assert (await self._validate(None))["valid"] is Falsediff --git a/server/uv.lock b/server/uv.lockindex e4af6848..41204f38 100644--- a/server/uv.lock+++ b/server/uv.lock@@ -2042,7 +2042,7 @@ requires-dist = [{ name = "aiohttp", specifier = ">=3.14.4" },{ name = "aiosqlite", specifier = ">=0.20.0" },{ name = "anthropic", specifier = "==0.96.0" },- { name = "apify-client", specifier = ">=1.8.0" },+ { name = "apify-client", specifier = ">=3.1.0" },{ name = "asyncpg", specifier = ">=0.30.0,<1" },{ name = "bcrypt", specifier = ">=4.2.0" },{ name = "beautifulsoup4", marker = "extra == 'docs'", specifier = ">=4.12.0" },