Self-improving AI Employees for everyone
Python ai-agentsai-employeeai-employeesassistantautonomous-agentsco-employeescompanygrok-botinstinctloop-agentsmuseown-your-datapersonalself-hosted
fix: Cross-platform script fixes, Docker improvements, and UI cleanup
trohitgbe2b40a23c84
diff --git a/CLAUDE.md b/CLAUDE.mdindex bea49afa..8f8dc1c4 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -374,7 +374,7 @@ class CacheEntry(SQLModel, table=True):## Codebase Summary- **Hybrid architecture**: Node.js + Python + React TypeScript-- **92 implemented workflow nodes** with clean service separation (6 AI models + 3 AI agents/memory + 13 specialized agents + 1 skill + 4 dedicated tools + 3 search + 16 Android + 3 WhatsApp + 4 Twitter + 2 Telegram + 2 Social + 3 Location + 3 Code + 6 Utility + 6 Document + 2 Chat + 2 Scheduler + 2 Workflow + 7 Google Workspace + 1 Apify + 3 Proxy)+- **93 implemented workflow nodes** with clean service separation (6 AI models + 3 AI agents/memory + 13 specialized agents + 1 skill + 4 dedicated tools + 3 search + 16 Android + 3 WhatsApp + 4 Twitter + 2 Telegram + 2 Social + 3 Location + 3 Code + 6 Utility + 6 Document + 2 Chat + 2 Scheduler + 2 Workflow + 7 Google Workspace + 1 Apify + 1 Crawlee + 3 Proxy)- **WebSocket-First Architecture**: WebSocket as primary frontend-backend communication (120 message handlers)- **Recent optimizations**: REST APIs replaced with WebSocket, AI endpoints migrated to Python, Android automation integrated@@ -412,6 +412,7 @@ The project was completely refactored from schema-based node definitions to expl- `src/nodeDefinitions/codeNodes.ts` - Python, JavaScript, and TypeScript code execution nodes- `src/nodeDefinitions/proxyNodes.ts` - Proxy nodes (proxyRequest, proxyConfig, proxyStatus) and shared PROXY_PARAMETERS- `src/nodeDefinitions/utilityNodes.ts` - HTTP, Webhook, chatTrigger, and console nodes+- `src/nodeDefinitions/crawleeNodes.ts` - Crawlee web scraping node (crawleeScraper) with static/Playwright modes- `src/nodeDefinitions/documentNodes.ts` - Document processing nodes (httpScraper, fileDownloader, documentParser, textChunker, embeddingGenerator, vectorStore)- `src/nodeDefinitions/chatNodes.ts` - Chat send and history nodes- `src/nodeDefinitions/schedulerNodes.ts` - Timer and cron scheduler nodes@@ -603,7 +604,8 @@ server/skills/├── web-search-skill/SKILL.md├── http-request-skill/SKILL.md├── proxy-config-skill/SKILL.md- └── apify-skill/SKILL.md+ ├── apify-skill/SKILL.md+ └── crawlee-scraper-skill/SKILL.md```**SKILL.md Format:**@@ -1374,12 +1376,12 @@ All scripts in `scripts/` are cross-platform Node.js (Windows, macOS, Linux, WSL- `stop.js` - Kills processes on configured ports- `build.js` - Full production build (client, Python, WhatsApp)- `clean.js` - Removes node_modules, dist, .venv-- `docker.js` - Docker Compose wrapper with Redis profile detection+- `docker.js` - Docker Compose wrapper with v2 detection and Redis profile supportSee **[Scripts Reference](./docs-internal/SCRIPTS.md)** for full documentation.## Current Status-✅ **INodeProperties System**: Fully implemented with 92 functional node components+✅ **INodeProperties System**: Fully implemented with 93 functional node components✅ **WebSocket-First Architecture**: 120 message handlers replacing REST APIs✅ **Code Editor**: Python, JavaScript, and TypeScript executors with syntax-highlighted editor (react-simple-code-editor + prismjs) and console output✅ **Node.js Executor**: Persistent Node.js server (Express + tsx) for fast JS/TS execution, replacing subprocess spawning@@ -2230,12 +2232,13 @@ The project deploys using Docker Compose with nginx reverse proxy.**Backend (`server/Dockerfile`):**- Python 3.12-slim base with Node.js 22.x for JS/TS execution+- Includes Playwright chromium for JS-rendered web scraping (crawleeScraper node)- Includes persistent Node.js server (Express + tsx) on port 3020- Optimized bytecode compilation (`python -O -m compileall`)- Health check endpoint on port 3010-- Startup script (`start.sh`) runs both Python and Node.js servers+- Startup script (`start.sh`) runs both Python and Node.js servers (must have LF line endings, not CRLF)- Depends on: redis, whatsapp-- Size: ~600 MB+- Size: ~800 MB (includes Playwright chromium)**WhatsApp (`docker/Dockerfile.whatsapp`):**- Uses npm package `whatsapp-rpc` with pre-built binaries@@ -2267,6 +2270,9 @@ services:backend:build: ./serverports: ["${PYTHON_BACKEND_PORT:-3010}:${PYTHON_BACKEND_PORT:-3010}"]+ volumes:+ - ./server:/app+ - /app/nodejs/node_modules # Preserve Linux binaries (prevents Windows esbuild conflict)depends_on:whatsapp: { condition: service_healthy } # No Redis dependencyenvironment:diff --git a/client/src/components/CredentialsModal.tsx b/client/src/components/CredentialsModal.tsxindex 4e51d7a9..227b00b7 100644--- a/client/src/components/CredentialsModal.tsx+++ b/client/src/components/CredentialsModal.tsx@@ -215,7 +215,7 @@ const { validateApiKey, saveApiKey, getStoredApiKey, hasStoredKey, removeApiKey,const [loading, setLoading] = useState<Record<string, boolean>>({});const [models, setModels] = useState<Record<string, string[]>>({});const [keys, setKeys] = useState<Record<string, string>>({});- const [proxyUrls, setProxyUrls] = useState<Record<string, string>>({});+ // const [proxyUrls, setProxyUrls] = useState<Record<string, string>>({}); // Proxy disabled for nowconst [selectedItem, setSelectedItem] = useState<CredentialItem | null>(null);// Provider defaults state@@ -240,8 +240,6 @@ const { validateApiKey, saveApiKey, getStoredApiKey, hasStoredKey, removeApiKey,const allItems = CATEGORIES.flatMap(c => c.items);const newKeys: Record<string, string> = {};const newValid: Record<string, boolean> = {};- const newProxyUrls: Record<string, string> = {};-for (const item of allItems) {if (item.isSpecial) continue;if (await hasStoredKey(item.id)) {@@ -251,18 +249,9 @@ const { validateApiKey, saveApiKey, getStoredApiKey, hasStoredKey, removeApiKey,newValid[item.id] = true;}}- // Load proxy URL for AI providers (stored as {provider}_proxy)- const proxyKey = `${item.id}_proxy`;- if (await hasStoredKey(proxyKey)) {- const proxyUrl = await getStoredApiKey(proxyKey);- if (proxyUrl) {- newProxyUrls[item.id] = proxyUrl;- }- }}setKeys(newKeys);setValidKeys(newValid);- setProxyUrls(newProxyUrls);}, [hasStoredKey, getStoredApiKey]);useEffect(() => {@@ -2297,58 +2286,7 @@ const { validateApiKey, saveApiKey, getStoredApiKey, hasStoredKey, removeApiKey,/></div>- {/* Proxy URL Input - Only for AI providers */}- {CATEGORIES.find(c => c.key === 'ai')?.items.some(i => i.id === item.id) && (- <div style={{ marginBottom: theme.spacing.xl }}>- <label style={{- display: 'block',- fontSize: theme.fontSize.sm,- fontWeight: theme.fontWeight.medium,- color: theme.colors.text,- marginBottom: theme.spacing.sm,- }}>- Proxy URL <span style={{ fontWeight: theme.fontWeight.normal, color: theme.colors.textMuted }}>(optional)</span>- </label>- <Space.Compact style={{ width: '100%' }}>- <Input- value={proxyUrls[item.id] || ''}- onChange={(e: React.ChangeEvent<HTMLInputElement>) => setProxyUrls(p => ({ ...p, [item.id]: e.target.value }))}- placeholder="http://localhost:11434"- style={{- fontFamily: 'monospace',- backgroundColor: theme.colors.background,- borderColor: theme.colors.border,- color: theme.colors.text,- }}- />- <Button- onClick={async () => {- const proxyUrl = proxyUrls[item.id]?.trim();- if (proxyUrl) {- await saveApiKey(`${item.id}_proxy`, proxyUrl);- } else {- await removeApiKey(`${item.id}_proxy`);- setProxyUrls(p => ({ ...p, [item.id]: '' }));- }- }}- style={{- backgroundColor: `${theme.dracula.purple}25`,- borderColor: `${theme.dracula.purple}60`,- color: theme.dracula.purple,- }}- >- {proxyUrls[item.id]?.trim() ? 'Save' : 'Clear'}- </Button>- </Space.Compact>- <div style={{- fontSize: theme.fontSize.xs,- color: theme.colors.textMuted,- marginTop: theme.spacing.sm,- }}>- Route requests through a proxy (e.g., Ollama). When set, the proxy handles authentication.- </div>- </div>- )}+ {/* Proxy URL Input - Disabled for now */}{/* Default Parameters Section - Only for AI providers */}{CATEGORIES.find(c => c.key === 'ai')?.items.some(i => i.id === item.id) && (diff --git a/client/src/nodeDefinitions/crawleeNodes.ts b/client/src/nodeDefinitions/crawleeNodes.tsindex 99a96b26..de8653d7 100644--- a/client/src/nodeDefinitions/crawleeNodes.ts+++ b/client/src/nodeDefinitions/crawleeNodes.ts@@ -45,7 +45,7 @@ export const crawleeNodes: Record<string, INodeTypeDescription> = {displayName: 'Tool Name',name: 'toolName',type: 'string',- default: 'web_scraper',+ default: 'web_reader',description: 'Name of this tool when used by AI Agent',},{@@ -53,7 +53,7 @@ export const crawleeNodes: Record<string, INodeTypeDescription> = {name: 'toolDescription',type: 'string',default:- 'Scrape web pages to extract text content, links, and structured data. Supports static HTML and JavaScript-rendered pages.',+ 'Read and extract content from web pages. Fetches page text, links, and data. You MUST use this tool when asked to read or fetch any URL.',description: 'Description shown to AI Agent',},diff --git a/docker-compose.yml b/docker-compose.ymlindex 26b94906..da0251a1 100644--- a/docker-compose.yml+++ b/docker-compose.yml@@ -31,6 +31,7 @@ services:- ./server:/app- backend-data:/app/data- credentials-data:/app/credentials # Encrypted credentials database (separate volume)+ - /app/nodejs/node_modules # Preserve container-built node_modules (Linux binaries, not Windows)networks:- app-networkrestart: unless-stoppeddiff --git a/docs-internal/SCRIPTS.md b/docs-internal/SCRIPTS.mdindex 4cd76d5a..eeb78ee1 100644--- a/docs-internal/SCRIPTS.md+++ b/docs-internal/SCRIPTS.md@@ -28,7 +28,7 @@ Open http://localhost:3000### Dependency ChecksThe `start` and `build` commands verify these dependencies before running:-- Node.js 18++- Node.js 22+- Python 3.12+- uv (Python package manager)diff --git a/install.sh b/install.shindex 19bedacc..45fdfa29 100755--- a/install.sh+++ b/install.sh@@ -278,7 +278,15 @@ main() {echo ""# Install machinaos from npm- npm install -g machinaos+ # On Linux/WSL without nvm, global npm install needs sudo unless prefix is user-writable+ if npm install -g machinaos 2>/dev/null; then+ : # Installed successfully+ elif command -v sudo &> /dev/null; then+ info "Retrying with sudo..."+ sudo npm install -g machinaos+ else+ error_exit "npm install -g failed. Try: sudo npm install -g machinaos"+ fiecho ""echo -e "${GREEN}============================================${NC}"@@ -291,6 +299,9 @@ main() {echo " Open in browser:"echo " http://localhost:3000"echo ""+ echo " Optional: Enable JS-rendered web scraping:"+ echo " playwright install chromium"+ echo ""}# Run maindiff --git a/package-lock.json b/package-lock.jsonindex 322c2a1a..c16121d6 100644--- a/package-lock.json+++ b/package-lock.json@@ -1,12 +1,12 @@{"name": "machinaos",- "version": "0.0.35",+ "version": "0.0.38","lockfileVersion": 3,"requires": true,"packages": {"": {"name": "machinaos",- "version": "0.0.35",+ "version": "0.0.38","hasInstallScript": true,"license": "MIT","workspaces": [@@ -16,13 +16,13 @@"concurrently": "^9.2.1","cross-env": "^7.0.3","rimraf": "^6.0.1",- "whatsapp-rpc": "^0.0.10"+ "whatsapp-rpc": "^0.0.12"},"bin": {"machina": "bin/cli.js"},"engines": {- "node": ">=20.0.0"+ "node": ">=22.0.0"}},"client": {@@ -11369,9 +11369,9 @@}},"node_modules/whatsapp-rpc": {- "version": "0.0.10",- "resolved": "https://registry.npmjs.org/whatsapp-rpc/-/whatsapp-rpc-0.0.10.tgz",- "integrity": "sha512-F7pfeYzmP5f0pyAng2Ej+Fmh0PmpOmd2Hp6Vr5eYQtTp9YBsUX2dp4RTURupVeFai6oUipCaNdqQ9u/jlmzKqg==",+ "version": "0.0.12",+ "resolved": "https://registry.npmjs.org/whatsapp-rpc/-/whatsapp-rpc-0.0.12.tgz",+ "integrity": "sha512-Rad4VVlE7WDMZ+A+Mo5lW0ZhfJb7kmp7yVZP2xyQ3Qu1JWJSN012yw74srzEyGpHJC3lSh+wa4XmItSn8xFpyg==","hasInstallScript": true,"license": "MIT","dependencies": {diff --git a/package.json b/package.jsonindex 4a308c02..884cde14 100644--- a/package.json+++ b/package.json@@ -30,7 +30,7 @@"machina": "./bin/cli.js"},"engines": {- "node": ">=20.0.0"+ "node": ">=22.0.0"},"files": ["bin/",diff --git a/scripts/build.js b/scripts/build.jsindex 9ec6cbd2..c0915e9d 100755--- a/scripts/build.js+++ b/scripts/build.js@@ -89,11 +89,12 @@ const npmVersion = getVersion('npm --version');console.log(` npm: ${npmVersion}`);// Python (required, user must install)+// Check python3 first to avoid picking up Python 2.x on Linux/macOS/WSLlet pyCmd = null;-if (runSilent('python --version')) {- pyCmd = 'python';-} else if (runSilent('python3 --version')) {+if (runSilent('python3 --version')) {pyCmd = 'python3';+} else if (runSilent('python --version')) {+ pyCmd = 'python';}if (pyCmd) {@@ -143,32 +144,44 @@ try {const templatePath = resolve(ROOT, '.env.template');if (!existsSync(envPath) && existsSync(templatePath)) {copyFileSync(templatePath, envPath);- console.log('[0/6] Created .env from template');+ console.log('[0/5] Created .env from template');}// Step 1: Install root + client dependencies (workspaces handles both)if (!isPostInstall) {- console.log('[1/4] Installing dependencies...');+ console.log('[1/5] Installing dependencies...');npmInstall(ROOT);} else {- console.log('[1/4] Dependencies already installed by npm');+ console.log('[1/5] Dependencies already installed by npm');}// Step 2: Build client- console.log('[2/4] Building client...');+ console.log('[2/5] Building client...');run('npm run build', resolve(ROOT, 'client'));// Step 3: Install Python dependencies- console.log('[3/4] Installing Python dependencies...');+ console.log('[3/5] Installing Python dependencies...');const serverDir = resolve(ROOT, 'server');if (!existsSync(resolve(serverDir, '.venv'))) {run('uv venv', serverDir);}run('uv sync', serverDir);- // Step 4: Verify WhatsApp RPC package- console.log('[4/4] Verifying WhatsApp RPC...');- run('whatsapp-rpc status', ROOT);+ // Step 4: Install Playwright browser for web scraping (non-fatal)+ console.log('[4/5] Installing Playwright browser...');+ try {+ run('playwright install chromium', serverDir);+ } catch {+ console.log(' Warning: Playwright browser install failed. JS-rendered web scraping will be unavailable.');+ }++ // Step 5: Verify WhatsApp RPC package (non-fatal)+ console.log('[5/5] Verifying WhatsApp RPC...');+ try {+ run('whatsapp-rpc status', ROOT);+ } catch {+ console.log(' Warning: whatsapp-rpc not available. Use --skip-whatsapp when starting.');+ }console.log('\nBuild complete.');diff --git a/scripts/daemon.js b/scripts/daemon.jsindex 7c46fd2e..6631f7b2 100755--- a/scripts/daemon.js+++ b/scripts/daemon.js@@ -17,7 +17,7 @@* --memory=LIMIT Memory limit (default: 2G, Linux only)*/import { execSync, spawnSync } from 'child_process';-import { existsSync, writeFileSync, mkdirSync, unlinkSync } from 'fs';+import { existsSync, readFileSync, writeFileSync, mkdirSync, unlinkSync } from 'fs';import { resolve, dirname } from 'path';import { fileURLToPath } from 'url';import { homedir } from 'os';@@ -115,7 +115,20 @@ function windowsInstall() {// Install servicerun(`nssm install ${SERVICE_NAME} "${pythonExe}" -m uvicorn main:app --host 0.0.0.0 --port 3010 --log-level warning`);run(`nssm set ${SERVICE_NAME} AppDirectory "${serverDir}"`);- run(`nssm set ${SERVICE_NAME} AppEnvironmentExtra "PYTHONPATH=${serverDir}"`);+ // Load .env variables for the service+ const envFile = resolve(installDir, '.env');+ let envExtra = `PYTHONPATH=${serverDir} PYTHONUTF8=1`;+ if (existsSync(envFile)) {+ for (const line of readFileSync(envFile, 'utf-8').split('\n')) {+ const match = line.match(/^([^#=]+)=(.*)$/);+ if (match) {+ const key = match[1].trim();+ const val = match[2].trim().replace(/^["']|["']$/g, '');+ if (key && val) envExtra += ` ${key}=${val}`;+ }+ }+ }+ run(`nssm set ${SERVICE_NAME} AppEnvironmentExtra "${envExtra}"`);run(`nssm set ${SERVICE_NAME} Start SERVICE_AUTO_START`);run(`nssm set ${SERVICE_NAME} AppStdout "${logsDir}\\service.log"`);run(`nssm set ${SERVICE_NAME} AppStderr "${logsDir}\\service.log"`);@@ -169,7 +182,10 @@ After=network.targetType=simpleUser=${serviceUser}WorkingDirectory=${serverDir}-EnvironmentFile=${installDir}/.env+EnvironmentFile=-${installDir}/.env+Environment=PYTHONPATH=${serverDir}+Environment=PYTHONUTF8=1+Environment=PATH=${venvPath}/bin:/usr/local/bin:/usr/bin:/binExecStart=${venvPath}/bin/uvicorn main:app --host 0.0.0.0 --port 3010 --log-level warningRestart=alwaysRestartSec=5@@ -269,6 +285,10 @@ function macGeneratePlist() {<dict><key>PATH</key><string>${venvPath}/bin:/usr/local/bin:/usr/bin:/bin</string>+ <key>PYTHONPATH</key>+ <string>${serverDir}</string>+ <key>PYTHONUTF8</key>+ <string>1</string></dict><key>RunAtLoad</key><true/>diff --git a/scripts/dev.js b/scripts/dev.jsindex 76ace58b..0ec30b18 100644--- a/scripts/dev.js+++ b/scripts/dev.js@@ -100,7 +100,8 @@ async function main() {const readySet = new Set();const totalExpected = readyPatterns.length;- const proc = spawn('npx', ['concurrently', '--raw', '--kill-others', ...services], {+ // No --kill-others: uvicorn hot-reloads (exit code 1) would cascade-kill frontend+ const proc = spawn('npx', ['concurrently', '--raw', ...services], {cwd: ROOT,stdio: ['inherit', 'pipe', 'pipe'],shell: true,diff --git a/scripts/docker.js b/scripts/docker.jsindex df245e9e..d6ddbd96 100755--- a/scripts/docker.js+++ b/scripts/docker.js@@ -4,7 +4,7 @@* Usage: node scripts/docker.js <command> [args...]* Commands: up, down, build, logs, restart*/-import { spawn } from 'child_process';+import { spawn, execSync } from 'child_process';import {ROOT,loadEnvConfig,@@ -53,10 +53,21 @@ if (command === 'logs') {// Add any additional argscomposeArgs.push(...args);-console.log(`[Docker] Running: docker-compose ${composeArgs.join(' ')}`);+// Detect docker compose v2 (docker compose) vs v1 (docker-compose)+let useV2 = false;+try {+ execSync('docker compose version', { stdio: 'pipe', timeout: 5000 });+ useV2 = true;+} catch {+ // Fall back to docker-compose v1+}++const composeCmd = useV2 ? 'docker' : 'docker-compose';+const composeFinalArgs = useV2 ? ['compose', ...composeArgs] : composeArgs;++console.log(`[Docker] Running: ${composeCmd} ${composeFinalArgs.join(' ')}`);-// Run docker-compose-const proc = spawn('docker-compose', composeArgs, {+const proc = spawn(composeCmd, composeFinalArgs, {cwd: ROOT,stdio: 'inherit',shell: truediff --git a/scripts/port_kill.py b/scripts/port_kill.pyindex 506cd03a..f18d6804 100644--- a/scripts/port_kill.py+++ b/scripts/port_kill.py@@ -121,8 +121,10 @@ def cmd_kill_machina(root_dir, exclude_script=None):continueif proc.pid == my_pid:continue- proc.kill()- killed.append({"pid": proc.pid, "name": name})+ # Graceful termination first, then force kill after timeout+ result = kill_pid(proc.pid, graceful_timeout=2)+ if result["killed"]:+ killed.append({"pid": proc.pid, "name": name})except (psutil.NoSuchProcess, psutil.AccessDenied):passdiff --git a/scripts/serve-client.js b/scripts/serve-client.jsindex 733c8ed4..01493951 100755--- a/scripts/serve-client.js+++ b/scripts/serve-client.js@@ -34,7 +34,14 @@ const server = createServer((req, res) => {// Remove query stringurl = url.split('?')[0];- const filePath = join(DIST_DIR, url);+ const filePath = resolve(DIST_DIR, '.' + url);++ // Prevent path traversal outside DIST_DIR+ if (!filePath.startsWith(DIST_DIR)) {+ res.writeHead(403);+ res.end('Forbidden');+ return;+ }readFile(filePath, (err, content) => {if (err) {diff --git a/scripts/utils.js b/scripts/utils.jsindex 2bbe5653..2527225d 100644--- a/scripts/utils.js+++ b/scripts/utils.js@@ -71,7 +71,12 @@ export function loadEnvConfig() {for (const line of readFileSync(envPath, 'utf-8').split('\n')) {const match = line.match(/^([^#=]+)=(.*)$/);if (match) {- env[match[1].trim()] = match[2].trim().replace(/^["']|["']$/g, '');+ let value = match[2].trim();+ // Strip inline comments (but not inside quoted values)+ if (!value.startsWith('"') && !value.startsWith("'")) {+ value = value.replace(/\s+#.*$/, '');+ }+ env[match[1].trim()] = value.replace(/^["']|["']$/g, '');}}}@@ -219,7 +224,33 @@ function killPortNative(port) {}} catch {}- return { killed, portFree: killed.length > 0 || true };+ // Re-check if the port is actually free after killing+ if (killed.length > 0) {+ try {+ let stillInUse = false;+ if (isWindows) {+ const recheck = execSync('netstat -ano', { encoding: 'utf-8', timeout: 5000, stdio: ['pipe', 'pipe', 'pipe'] });+ const portStr = `:${port}`;+ for (const line of recheck.split('\n')) {+ if (line.includes(portStr) && line.includes('LISTENING')) {+ const parts = line.trim().split(/\s+/);+ if (parts[1]?.endsWith(portStr)) { stillInUse = true; break; }+ }+ }+ } else {+ try {+ execSync(`lsof -ti :${port}`, { encoding: 'utf-8', timeout: 3000, stdio: ['pipe', 'pipe', 'pipe'] });+ stillInUse = true;+ } catch {+ stillInUse = false; // lsof exits non-zero when no process found+ }+ }+ return { killed, portFree: !stillInUse };+ } catch {+ return { killed, portFree: true }; // Assume freed if verification fails+ }+ }+ return { killed, portFree: true }; // No processes found = port was already free}/**diff --git a/server/Dockerfile b/server/Dockerfileindex 25583078..95994ce4 100644--- a/server/Dockerfile+++ b/server/Dockerfile@@ -18,6 +18,9 @@ COPY pyproject.toml .# Install Python dependencies using uvRUN uv pip install --system -e .+# Install Playwright browser (chromium) and its system dependencies+RUN playwright install --with-deps chromium+# Copy application codeCOPY . .diff --git a/server/config/model_registry.json b/server/config/model_registry.jsonindex 4a70c317..85c46f52 100644--- a/server/config/model_registry.json+++ b/server/config/model_registry.json@@ -1,7 +1,7 @@{- "_generated": "2026-02-28T07:14:28.108966+00:00",+ "_generated": "2026-03-02T05:51:12.590903+00:00","_source": "openrouter",- "_model_count": 343,+ "_model_count": 342,"models": {"bytedance-seed/seed-2.0-mini": {"id": "bytedance-seed/seed-2.0-mini",@@ -3620,36 +3620,6 @@"top_p"]},- "opengvlab/internvl3-78b": {- "id": "opengvlab/internvl3-78b",- "name": "OpenGVLab: InternVL3 78B",- "provider": "opengvlab",- "local_id": "internvl3-78b",- "context_length": 32768,- "max_output_tokens": 32768,- "input_price_per_mtok": 0.15,- "output_price_per_mtok": 0.6,- "supports_thinking": false,- "thinking_type": "none",- "temperature_range": [- 0.0,- 2.0- ],- "is_reasoning_model": false,- "supported_parameters": [- "frequency_penalty",- "max_tokens",- "presence_penalty",- "repetition_penalty",- "response_format",- "seed",- "stop",- "structured_outputs",- "temperature",- "top_k",- "top_p"- ]- },"qwen/qwen3-next-80b-a3b-thinking": {"id": "qwen/qwen3-next-80b-a3b-thinking","name": "Qwen: Qwen3 Next 80B A3B Thinking",diff --git a/server/pyproject.toml b/server/pyproject.tomlindex 78b9e7e6..9468b842 100644--- a/server/pyproject.toml+++ b/server/pyproject.toml@@ -67,7 +67,7 @@ dependencies = ["apify-client>=1.8.0",# Crawlee Web Scraping- "crawlee[beautifulsoup]>=1.4.0",+ "crawlee[beautifulsoup,playwright]>=1.4.0","html2text>=2024.2.26",# Telegram Bot APIdiff --git a/server/services/ai.py b/server/services/ai.pyindex 3e61858f..1ecf0cf7 100644--- a/server/services/ai.py+++ b/server/services/ai.py@@ -2500,7 +2500,7 @@ class AIService:'proxyConfig': 'proxy_config','proxyStatus': 'proxy_status',# Crawlee web scraping (dual-purpose tool)- 'crawleeScraper': 'web_scraper',+ 'crawleeScraper': 'web_reader',}DEFAULT_TOOL_DESCRIPTIONS = {'calculatorTool': 'Perform mathematical calculations. Operations: add, subtract, multiply, divide, power, sqrt, mod, abs',@@ -2565,7 +2565,7 @@ class AIService:'proxyConfig': 'Manage proxy providers and routing rules. Operations: list_providers, add_provider, update_provider, remove_provider, set_credentials, test_provider, get_stats, add_routing_rule, list_routing_rules, remove_routing_rule.','proxyStatus': 'Get proxy provider health stats, scores, and usage statistics. Returns enabled status, provider list with health scores, and aggregated stats.',# Crawlee web scraping (dual-purpose tool)- 'crawleeScraper': 'Scrape web pages to extract text, links, and structured data. Supports static HTML (beautifulsoup) and JavaScript-rendered pages (playwright). Use crawlerType to select engine.',+ 'crawleeScraper': 'Read and extract content from web pages. Fetches page text, links, and data. Use beautifulsoup for static pages or playwright for JS-rendered pages. You MUST use this tool when the user asks to read, fetch, or get content from any URL.',}try:@@ -3443,12 +3443,12 @@ class AIService:return ApifyActorSchema- # Crawlee web scraper tool schema (dual-purpose: workflow node + AI tool)+ # Crawlee web reader tool schema (dual-purpose: workflow node + AI tool)if node_type == 'crawleeScraper':class CrawleeScraperSchema(BaseModel):- """Scrape web pages to extract text, links, and structured data.- Supports static HTML (beautifulsoup) and JS-rendered pages (playwright)."""- url: str = Field(description="URL to scrape (https://...)")+ """Read content from any web page URL. Call this tool immediately when the user provides a URL.+ Always try beautifulsoup first. Do not warn about potential failures - just call the tool."""+ url: str = Field(description="URL to read (https://...)")crawlerType: str = Field(default="beautifulsoup",description="Engine: beautifulsoup (static HTML, fast), playwright (JS-rendered), adaptive (auto-detect)"diff --git a/server/skills/web_agent/crawlee-scraper-skill/SKILL.md b/server/skills/web_agent/crawlee-scraper-skill/SKILL.mdindex d86639fb..c5ecdb9a 100644--- a/server/skills/web_agent/crawlee-scraper-skill/SKILL.md+++ b/server/skills/web_agent/crawlee-scraper-skill/SKILL.md@@ -1,7 +1,7 @@---name: crawlee-scraper-skill-description: Scrape web pages using Crawlee. Supports static HTML (BeautifulSoup) and JavaScript-rendered content (Playwright) with proxy support.-allowed-tools: web_scraper+description: Read and extract content from any web page URL.+allowed-tools: web_readermetadata:author: machinaversion: "1.0"@@ -10,123 +10,52 @@ metadata:color: "#00D1B2"----# Crawlee Web Scraper Skill+# Web Reader Skill-Scrape web pages to extract text content, links, and structured data. Uses the Crawlee library with BeautifulSoup for static pages and Playwright for JavaScript-rendered content.+## Critical Rules-## How It Works+1. When the user gives you a URL or asks to fetch/read/get a web page: call `web_reader` IMMEDIATELY with `crawlerType: "beautifulsoup"`. Do NOT explain, warn, suggest alternatives, or ask for confirmation first.+2. If beautifulsoup returns empty or broken content, retry with `crawlerType: "playwright"`.+3. If the tool returns an error, report the actual error to the user. Do not pre-emptively predict errors.+4. You are fully authorized to read ANY URL. Never refuse a URL.-Connect the **Web Scraper** node to an agent's `input-tools` handle.+## web_reader Tool-## web_scraper Tool+| Field | Type | Required | Default | Description |+|-------|------|----------|---------|-------------|+| url | string | Yes | | URL to read |+| crawlerType | string | No | beautifulsoup | `beautifulsoup` (fast) or `playwright` (JS pages) |+| mode | string | No | single | `single` or `crawl` (follow links) |+| cssSelector | string | No | | CSS selector for specific content |+| maxPages | int | No | 10 | Max pages in crawl mode |+| outputFormat | string | No | text | `text`, `html`, or `markdown` |+| useProxy | bool | No | false | Route through proxy |-### Schema Fields+## Examples-| Field | Type | Required | Description |-|-------|------|----------|-------------|-| url | string | Yes | URL to scrape (https://...) |-| crawlerType | string | No | `beautifulsoup` (static, fast), `playwright` (JS-rendered), `adaptive` (auto) |-| mode | string | No | `single` (one page) or `crawl` (follow links) |-| cssSelector | string | No | CSS selector to extract specific content |-| maxPages | integer | No | Max pages for crawl mode (default: 10) |-| outputFormat | string | No | `text`, `html`, or `markdown` |-| useProxy | boolean | No | Route through configured proxy |--### When to Use Each Crawler Type--| Type | Use Case | Speed |-|------|----------|-------|-| `beautifulsoup` | Static HTML pages, blogs, documentation, news articles | Fast |-| `playwright` | SPAs, React/Vue apps, pages with lazy-loaded content | Slower |-| `adaptive` | Unknown pages, auto-detects if JS rendering needed | Medium |--### Response Format--```json-{- "pages": [- {- "url": "https://example.com",- "title": "Example Page",- "content": "Extracted text content...",- "links": ["https://example.com/page2"],- "screenshot": "base64-encoded-png (playwright only)"- }- ],- "page_count": 1,- "crawler_type": "beautifulsoup",- "mode": "single",- "proxied": false-}-```--### Examples--**Scrape a single page:**+Read a page:```json-{- "url": "https://example.com/article",- "crawlerType": "beautifulsoup",- "outputFormat": "text"-}+{"url": "https://example.com", "crawlerType": "beautifulsoup"}```-**Scrape a JS-rendered SPA:**+Read a JS-rendered page:```json-{- "url": "https://app.example.com/dashboard",- "crawlerType": "playwright"-}+{"url": "https://app.example.com", "crawlerType": "playwright"}```-**Crawl documentation site:**+Crawl a site:```json-{- "url": "https://docs.example.com",- "mode": "crawl",- "maxPages": 50,- "outputFormat": "markdown"-}+{"url": "https://docs.example.com", "mode": "crawl", "maxPages": 20, "outputFormat": "markdown"}```-**Extract specific content with CSS selector:**+Extract specific content:```json-{- "url": "https://blog.example.com",- "cssSelector": "article .post-content",- "outputFormat": "text"-}+{"url": "https://blog.example.com", "cssSelector": "article .content"}```-**Scrape through proxy:**-```json-{- "url": "https://example.com",- "useProxy": true,- "crawlerType": "beautifulsoup"-}-```--## Guidelines--1. **Start with beautifulsoup** - it is much faster. Only use playwright when content is JS-rendered.-2. **Use CSS selectors** to extract specific content instead of scraping entire pages.-3. **Set reasonable maxPages** when crawling to avoid excessive scraping.-4. **Use proxy** for sites that rate-limit or geo-restrict.-5. **Respect robots.txt** and website terms of service.--## Error Handling--| Error | Cause | Fix |-|-------|-------|-----|-| "Crawlee not installed" | Missing dependency | `pip install 'crawlee[beautifulsoup]'` |-| "Playwright not installed" | Missing browser | `pip install 'crawlee[playwright]' && playwright install chromium` |-| Timeout | Page too slow or too many pages | Reduce maxPages or increase timeout |-| Empty content | Wrong CSS selector or JS-only page | Try playwright or adjust cssSelector |--## Setup+## Tips-1. Connect **Web Scraper** node to agent's `input-tools` handle-2. BeautifulSoup works immediately (no extra setup)-3. For Playwright: `pip install 'crawlee[playwright]' && playwright install chromium`-4. For proxy: Configure proxy providers in Credentials Modal+- Always try beautifulsoup first, it works on most sites and is fast.+- Use playwright only if beautifulsoup returns empty/broken content.+- Use CSS selectors when you know the page structure.+- Use proxy for geo-restricted or rate-limited sites.diff --git a/server/start.sh b/server/start.shindex e7a6fc2f..aca1152d 100644--- a/server/start.sh+++ b/server/start.sh@@ -5,4 +5,4 @@cd /app/nodejs && npm start &# Start Python server (foreground)-python -m uvicorn main:app --host 0.0.0.0 --port ${PORT:-3010}+python -m uvicorn main:app --host 0.0.0.0 --port ${PORT:-3010} --log-level warning