Skip to content

Self-improving AI Employees for everyone

fix: Cross-platform script fixes, Docker improvements, and UI cleanup

trohitgbe2b40a23c84
diff --git a/CLAUDE.md b/CLAUDE.md
index bea49afa..8f8dc1c4 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -374,7 +374,7 @@ class CacheEntry(SQLModel, table=True):
## Codebase Summary
- **Hybrid architecture**: Node.js + Python + React TypeScript
-- **92 implemented workflow nodes** with clean service separation (6 AI models + 3 AI agents/memory + 13 specialized agents + 1 skill + 4 dedicated tools + 3 search + 16 Android + 3 WhatsApp + 4 Twitter + 2 Telegram + 2 Social + 3 Location + 3 Code + 6 Utility + 6 Document + 2 Chat + 2 Scheduler + 2 Workflow + 7 Google Workspace + 1 Apify + 3 Proxy)
+- **93 implemented workflow nodes** with clean service separation (6 AI models + 3 AI agents/memory + 13 specialized agents + 1 skill + 4 dedicated tools + 3 search + 16 Android + 3 WhatsApp + 4 Twitter + 2 Telegram + 2 Social + 3 Location + 3 Code + 6 Utility + 6 Document + 2 Chat + 2 Scheduler + 2 Workflow + 7 Google Workspace + 1 Apify + 1 Crawlee + 3 Proxy)
- **WebSocket-First Architecture**: WebSocket as primary frontend-backend communication (120 message handlers)
- **Recent optimizations**: REST APIs replaced with WebSocket, AI endpoints migrated to Python, Android automation integrated
@@ -412,6 +412,7 @@ The project was completely refactored from schema-based node definitions to expl
- `src/nodeDefinitions/codeNodes.ts` - Python, JavaScript, and TypeScript code execution nodes
- `src/nodeDefinitions/proxyNodes.ts` - Proxy nodes (proxyRequest, proxyConfig, proxyStatus) and shared PROXY_PARAMETERS
- `src/nodeDefinitions/utilityNodes.ts` - HTTP, Webhook, chatTrigger, and console nodes
+- `src/nodeDefinitions/crawleeNodes.ts` - Crawlee web scraping node (crawleeScraper) with static/Playwright modes
- `src/nodeDefinitions/documentNodes.ts` - Document processing nodes (httpScraper, fileDownloader, documentParser, textChunker, embeddingGenerator, vectorStore)
- `src/nodeDefinitions/chatNodes.ts` - Chat send and history nodes
- `src/nodeDefinitions/schedulerNodes.ts` - Timer and cron scheduler nodes
@@ -603,7 +604,8 @@ server/skills/
├── web-search-skill/SKILL.md
├── http-request-skill/SKILL.md
├── proxy-config-skill/SKILL.md
- └── apify-skill/SKILL.md
+ ├── apify-skill/SKILL.md
+ └── crawlee-scraper-skill/SKILL.md
```
**SKILL.md Format:**
@@ -1374,12 +1376,12 @@ All scripts in `scripts/` are cross-platform Node.js (Windows, macOS, Linux, WSL
- `stop.js` - Kills processes on configured ports
- `build.js` - Full production build (client, Python, WhatsApp)
- `clean.js` - Removes node_modules, dist, .venv
-- `docker.js` - Docker Compose wrapper with Redis profile detection
+- `docker.js` - Docker Compose wrapper with v2 detection and Redis profile support
See **[Scripts Reference](./docs-internal/SCRIPTS.md)** for full documentation.
## Current Status
-✅ **INodeProperties System**: Fully implemented with 92 functional node components
+✅ **INodeProperties System**: Fully implemented with 93 functional node components
✅ **WebSocket-First Architecture**: 120 message handlers replacing REST APIs
✅ **Code Editor**: Python, JavaScript, and TypeScript executors with syntax-highlighted editor (react-simple-code-editor + prismjs) and console output
✅ **Node.js Executor**: Persistent Node.js server (Express + tsx) for fast JS/TS execution, replacing subprocess spawning
@@ -2230,12 +2232,13 @@ The project deploys using Docker Compose with nginx reverse proxy.
**Backend (`server/Dockerfile`):**
- Python 3.12-slim base with Node.js 22.x for JS/TS execution
+- Includes Playwright chromium for JS-rendered web scraping (crawleeScraper node)
- Includes persistent Node.js server (Express + tsx) on port 3020
- Optimized bytecode compilation (`python -O -m compileall`)
- Health check endpoint on port 3010
-- Startup script (`start.sh`) runs both Python and Node.js servers
+- Startup script (`start.sh`) runs both Python and Node.js servers (must have LF line endings, not CRLF)
- Depends on: redis, whatsapp
-- Size: ~600 MB
+- Size: ~800 MB (includes Playwright chromium)
**WhatsApp (`docker/Dockerfile.whatsapp`):**
- Uses npm package `whatsapp-rpc` with pre-built binaries
@@ -2267,6 +2270,9 @@ services:
backend:
build: ./server
ports: ["${PYTHON_BACKEND_PORT:-3010}:${PYTHON_BACKEND_PORT:-3010}"]
+ volumes:
+ - ./server:/app
+ - /app/nodejs/node_modules # Preserve Linux binaries (prevents Windows esbuild conflict)
depends_on:
whatsapp: { condition: service_healthy } # No Redis dependency
environment:
diff --git a/client/src/components/CredentialsModal.tsx b/client/src/components/CredentialsModal.tsx
index 4e51d7a9..227b00b7 100644
--- a/client/src/components/CredentialsModal.tsx
+++ b/client/src/components/CredentialsModal.tsx
@@ -215,7 +215,7 @@ const { validateApiKey, saveApiKey, getStoredApiKey, hasStoredKey, removeApiKey,
const [loading, setLoading] = useState<Record<string, boolean>>({});
const [models, setModels] = useState<Record<string, string[]>>({});
const [keys, setKeys] = useState<Record<string, string>>({});
- const [proxyUrls, setProxyUrls] = useState<Record<string, string>>({});
+ // const [proxyUrls, setProxyUrls] = useState<Record<string, string>>({}); // Proxy disabled for now
const [selectedItem, setSelectedItem] = useState<CredentialItem | null>(null);
// Provider defaults state
@@ -240,8 +240,6 @@ const { validateApiKey, saveApiKey, getStoredApiKey, hasStoredKey, removeApiKey,
const allItems = CATEGORIES.flatMap(c => c.items);
const newKeys: Record<string, string> = {};
const newValid: Record<string, boolean> = {};
- const newProxyUrls: Record<string, string> = {};
-
for (const item of allItems) {
if (item.isSpecial) continue;
if (await hasStoredKey(item.id)) {
@@ -251,18 +249,9 @@ const { validateApiKey, saveApiKey, getStoredApiKey, hasStoredKey, removeApiKey,
newValid[item.id] = true;
}
}
- // Load proxy URL for AI providers (stored as {provider}_proxy)
- const proxyKey = `${item.id}_proxy`;
- if (await hasStoredKey(proxyKey)) {
- const proxyUrl = await getStoredApiKey(proxyKey);
- if (proxyUrl) {
- newProxyUrls[item.id] = proxyUrl;
- }
- }
}
setKeys(newKeys);
setValidKeys(newValid);
- setProxyUrls(newProxyUrls);
}, [hasStoredKey, getStoredApiKey]);
useEffect(() => {
@@ -2297,58 +2286,7 @@ const { validateApiKey, saveApiKey, getStoredApiKey, hasStoredKey, removeApiKey,
/>
</div>
- {/* Proxy URL Input - Only for AI providers */}
- {CATEGORIES.find(c => c.key === 'ai')?.items.some(i => i.id === item.id) && (
- <div style={{ marginBottom: theme.spacing.xl }}>
- <label style={{
- display: 'block',
- fontSize: theme.fontSize.sm,
- fontWeight: theme.fontWeight.medium,
- color: theme.colors.text,
- marginBottom: theme.spacing.sm,
- }}>
- Proxy URL <span style={{ fontWeight: theme.fontWeight.normal, color: theme.colors.textMuted }}>(optional)</span>
- </label>
- <Space.Compact style={{ width: '100%' }}>
- <Input
- value={proxyUrls[item.id] || ''}
- onChange={(e: React.ChangeEvent<HTMLInputElement>) => setProxyUrls(p => ({ ...p, [item.id]: e.target.value }))}
- placeholder="http://localhost:11434"
- style={{
- fontFamily: 'monospace',
- backgroundColor: theme.colors.background,
- borderColor: theme.colors.border,
- color: theme.colors.text,
- }}
- />
- <Button
- onClick={async () => {
- const proxyUrl = proxyUrls[item.id]?.trim();
- if (proxyUrl) {
- await saveApiKey(`${item.id}_proxy`, proxyUrl);
- } else {
- await removeApiKey(`${item.id}_proxy`);
- setProxyUrls(p => ({ ...p, [item.id]: '' }));
- }
- }}
- style={{
- backgroundColor: `${theme.dracula.purple}25`,
- borderColor: `${theme.dracula.purple}60`,
- color: theme.dracula.purple,
- }}
- >
- {proxyUrls[item.id]?.trim() ? 'Save' : 'Clear'}
- </Button>
- </Space.Compact>
- <div style={{
- fontSize: theme.fontSize.xs,
- color: theme.colors.textMuted,
- marginTop: theme.spacing.sm,
- }}>
- Route requests through a proxy (e.g., Ollama). When set, the proxy handles authentication.
- </div>
- </div>
- )}
+ {/* Proxy URL Input - Disabled for now */}
{/* Default Parameters Section - Only for AI providers */}
{CATEGORIES.find(c => c.key === 'ai')?.items.some(i => i.id === item.id) && (
diff --git a/client/src/nodeDefinitions/crawleeNodes.ts b/client/src/nodeDefinitions/crawleeNodes.ts
index 99a96b26..de8653d7 100644
--- a/client/src/nodeDefinitions/crawleeNodes.ts
+++ b/client/src/nodeDefinitions/crawleeNodes.ts
@@ -45,7 +45,7 @@ export const crawleeNodes: Record<string, INodeTypeDescription> = {
displayName: 'Tool Name',
name: 'toolName',
type: 'string',
- default: 'web_scraper',
+ default: 'web_reader',
description: 'Name of this tool when used by AI Agent',
},
{
@@ -53,7 +53,7 @@ export const crawleeNodes: Record<string, INodeTypeDescription> = {
name: 'toolDescription',
type: 'string',
default:
- 'Scrape web pages to extract text content, links, and structured data. Supports static HTML and JavaScript-rendered pages.',
+ 'Read and extract content from web pages. Fetches page text, links, and data. You MUST use this tool when asked to read or fetch any URL.',
description: 'Description shown to AI Agent',
},
diff --git a/docker-compose.yml b/docker-compose.yml
index 26b94906..da0251a1 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -31,6 +31,7 @@ services:
- ./server:/app
- backend-data:/app/data
- credentials-data:/app/credentials # Encrypted credentials database (separate volume)
+ - /app/nodejs/node_modules # Preserve container-built node_modules (Linux binaries, not Windows)
networks:
- app-network
restart: unless-stopped
diff --git a/docs-internal/SCRIPTS.md b/docs-internal/SCRIPTS.md
index 4cd76d5a..eeb78ee1 100644
--- a/docs-internal/SCRIPTS.md
+++ b/docs-internal/SCRIPTS.md
@@ -28,7 +28,7 @@ Open http://localhost:3000
### Dependency Checks
The `start` and `build` commands verify these dependencies before running:
-- Node.js 18+
+- Node.js 22+
- Python 3.12+
- uv (Python package manager)
diff --git a/install.sh b/install.sh
index 19bedacc..45fdfa29 100755
--- a/install.sh
+++ b/install.sh
@@ -278,7 +278,15 @@ main() {
echo ""
# Install machinaos from npm
- npm install -g machinaos
+ # On Linux/WSL without nvm, global npm install needs sudo unless prefix is user-writable
+ if npm install -g machinaos 2>/dev/null; then
+ : # Installed successfully
+ elif command -v sudo &> /dev/null; then
+ info "Retrying with sudo..."
+ sudo npm install -g machinaos
+ else
+ error_exit "npm install -g failed. Try: sudo npm install -g machinaos"
+ fi
echo ""
echo -e "${GREEN}============================================${NC}"
@@ -291,6 +299,9 @@ main() {
echo " Open in browser:"
echo " http://localhost:3000"
echo ""
+ echo " Optional: Enable JS-rendered web scraping:"
+ echo " playwright install chromium"
+ echo ""
}
# Run main
diff --git a/package-lock.json b/package-lock.json
index 322c2a1a..c16121d6 100644
--- a/package-lock.json
+++ b/package-lock.json
@@ -1,12 +1,12 @@
{
"name": "machinaos",
- "version": "0.0.35",
+ "version": "0.0.38",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "machinaos",
- "version": "0.0.35",
+ "version": "0.0.38",
"hasInstallScript": true,
"license": "MIT",
"workspaces": [
@@ -16,13 +16,13 @@
"concurrently": "^9.2.1",
"cross-env": "^7.0.3",
"rimraf": "^6.0.1",
- "whatsapp-rpc": "^0.0.10"
+ "whatsapp-rpc": "^0.0.12"
},
"bin": {
"machina": "bin/cli.js"
},
"engines": {
- "node": ">=20.0.0"
+ "node": ">=22.0.0"
}
},
"client": {
@@ -11369,9 +11369,9 @@
}
},
"node_modules/whatsapp-rpc": {
- "version": "0.0.10",
- "resolved": "https://registry.npmjs.org/whatsapp-rpc/-/whatsapp-rpc-0.0.10.tgz",
- "integrity": "sha512-F7pfeYzmP5f0pyAng2Ej+Fmh0PmpOmd2Hp6Vr5eYQtTp9YBsUX2dp4RTURupVeFai6oUipCaNdqQ9u/jlmzKqg==",
+ "version": "0.0.12",
+ "resolved": "https://registry.npmjs.org/whatsapp-rpc/-/whatsapp-rpc-0.0.12.tgz",
+ "integrity": "sha512-Rad4VVlE7WDMZ+A+Mo5lW0ZhfJb7kmp7yVZP2xyQ3Qu1JWJSN012yw74srzEyGpHJC3lSh+wa4XmItSn8xFpyg==",
"hasInstallScript": true,
"license": "MIT",
"dependencies": {
diff --git a/package.json b/package.json
index 4a308c02..884cde14 100644
--- a/package.json
+++ b/package.json
@@ -30,7 +30,7 @@
"machina": "./bin/cli.js"
},
"engines": {
- "node": ">=20.0.0"
+ "node": ">=22.0.0"
},
"files": [
"bin/",
diff --git a/scripts/build.js b/scripts/build.js
index 9ec6cbd2..c0915e9d 100755
--- a/scripts/build.js
+++ b/scripts/build.js
@@ -89,11 +89,12 @@ const npmVersion = getVersion('npm --version');
console.log(` npm: ${npmVersion}`);
// Python (required, user must install)
+// Check python3 first to avoid picking up Python 2.x on Linux/macOS/WSL
let pyCmd = null;
-if (runSilent('python --version')) {
- pyCmd = 'python';
-} else if (runSilent('python3 --version')) {
+if (runSilent('python3 --version')) {
pyCmd = 'python3';
+} else if (runSilent('python --version')) {
+ pyCmd = 'python';
}
if (pyCmd) {
@@ -143,32 +144,44 @@ try {
const templatePath = resolve(ROOT, '.env.template');
if (!existsSync(envPath) && existsSync(templatePath)) {
copyFileSync(templatePath, envPath);
- console.log('[0/6] Created .env from template');
+ console.log('[0/5] Created .env from template');
}
// Step 1: Install root + client dependencies (workspaces handles both)
if (!isPostInstall) {
- console.log('[1/4] Installing dependencies...');
+ console.log('[1/5] Installing dependencies...');
npmInstall(ROOT);
} else {
- console.log('[1/4] Dependencies already installed by npm');
+ console.log('[1/5] Dependencies already installed by npm');
}
// Step 2: Build client
- console.log('[2/4] Building client...');
+ console.log('[2/5] Building client...');
run('npm run build', resolve(ROOT, 'client'));
// Step 3: Install Python dependencies
- console.log('[3/4] Installing Python dependencies...');
+ console.log('[3/5] Installing Python dependencies...');
const serverDir = resolve(ROOT, 'server');
if (!existsSync(resolve(serverDir, '.venv'))) {
run('uv venv', serverDir);
}
run('uv sync', serverDir);
- // Step 4: Verify WhatsApp RPC package
- console.log('[4/4] Verifying WhatsApp RPC...');
- run('whatsapp-rpc status', ROOT);
+ // Step 4: Install Playwright browser for web scraping (non-fatal)
+ console.log('[4/5] Installing Playwright browser...');
+ try {
+ run('playwright install chromium', serverDir);
+ } catch {
+ console.log(' Warning: Playwright browser install failed. JS-rendered web scraping will be unavailable.');
+ }
+
+ // Step 5: Verify WhatsApp RPC package (non-fatal)
+ console.log('[5/5] Verifying WhatsApp RPC...');
+ try {
+ run('whatsapp-rpc status', ROOT);
+ } catch {
+ console.log(' Warning: whatsapp-rpc not available. Use --skip-whatsapp when starting.');
+ }
console.log('\nBuild complete.');
diff --git a/scripts/daemon.js b/scripts/daemon.js
index 7c46fd2e..6631f7b2 100755
--- a/scripts/daemon.js
+++ b/scripts/daemon.js
@@ -17,7 +17,7 @@
* --memory=LIMIT Memory limit (default: 2G, Linux only)
*/
import { execSync, spawnSync } from 'child_process';
-import { existsSync, writeFileSync, mkdirSync, unlinkSync } from 'fs';
+import { existsSync, readFileSync, writeFileSync, mkdirSync, unlinkSync } from 'fs';
import { resolve, dirname } from 'path';
import { fileURLToPath } from 'url';
import { homedir } from 'os';
@@ -115,7 +115,20 @@ function windowsInstall() {
// Install service
run(`nssm install ${SERVICE_NAME} "${pythonExe}" -m uvicorn main:app --host 0.0.0.0 --port 3010 --log-level warning`);
run(`nssm set ${SERVICE_NAME} AppDirectory "${serverDir}"`);
- run(`nssm set ${SERVICE_NAME} AppEnvironmentExtra "PYTHONPATH=${serverDir}"`);
+ // Load .env variables for the service
+ const envFile = resolve(installDir, '.env');
+ let envExtra = `PYTHONPATH=${serverDir} PYTHONUTF8=1`;
+ if (existsSync(envFile)) {
+ for (const line of readFileSync(envFile, 'utf-8').split('\n')) {
+ const match = line.match(/^([^#=]+)=(.*)$/);
+ if (match) {
+ const key = match[1].trim();
+ const val = match[2].trim().replace(/^["']|["']$/g, '');
+ if (key && val) envExtra += ` ${key}=${val}`;
+ }
+ }
+ }
+ run(`nssm set ${SERVICE_NAME} AppEnvironmentExtra "${envExtra}"`);
run(`nssm set ${SERVICE_NAME} Start SERVICE_AUTO_START`);
run(`nssm set ${SERVICE_NAME} AppStdout "${logsDir}\\service.log"`);
run(`nssm set ${SERVICE_NAME} AppStderr "${logsDir}\\service.log"`);
@@ -169,7 +182,10 @@ After=network.target
Type=simple
User=${serviceUser}
WorkingDirectory=${serverDir}
-EnvironmentFile=${installDir}/.env
+EnvironmentFile=-${installDir}/.env
+Environment=PYTHONPATH=${serverDir}
+Environment=PYTHONUTF8=1
+Environment=PATH=${venvPath}/bin:/usr/local/bin:/usr/bin:/bin
ExecStart=${venvPath}/bin/uvicorn main:app --host 0.0.0.0 --port 3010 --log-level warning
Restart=always
RestartSec=5
@@ -269,6 +285,10 @@ function macGeneratePlist() {
<dict>
<key>PATH</key>
<string>${venvPath}/bin:/usr/local/bin:/usr/bin:/bin</string>
+ <key>PYTHONPATH</key>
+ <string>${serverDir}</string>
+ <key>PYTHONUTF8</key>
+ <string>1</string>
</dict>
<key>RunAtLoad</key>
<true/>
diff --git a/scripts/dev.js b/scripts/dev.js
index 76ace58b..0ec30b18 100644
--- a/scripts/dev.js
+++ b/scripts/dev.js
@@ -100,7 +100,8 @@ async function main() {
const readySet = new Set();
const totalExpected = readyPatterns.length;
- const proc = spawn('npx', ['concurrently', '--raw', '--kill-others', ...services], {
+ // No --kill-others: uvicorn hot-reloads (exit code 1) would cascade-kill frontend
+ const proc = spawn('npx', ['concurrently', '--raw', ...services], {
cwd: ROOT,
stdio: ['inherit', 'pipe', 'pipe'],
shell: true,
diff --git a/scripts/docker.js b/scripts/docker.js
index df245e9e..d6ddbd96 100755
--- a/scripts/docker.js
+++ b/scripts/docker.js
@@ -4,7 +4,7 @@
* Usage: node scripts/docker.js <command> [args...]
* Commands: up, down, build, logs, restart
*/
-import { spawn } from 'child_process';
+import { spawn, execSync } from 'child_process';
import {
ROOT,
loadEnvConfig,
@@ -53,10 +53,21 @@ if (command === 'logs') {
// Add any additional args
composeArgs.push(...args);
-console.log(`[Docker] Running: docker-compose ${composeArgs.join(' ')}`);
+// Detect docker compose v2 (docker compose) vs v1 (docker-compose)
+let useV2 = false;
+try {
+ execSync('docker compose version', { stdio: 'pipe', timeout: 5000 });
+ useV2 = true;
+} catch {
+ // Fall back to docker-compose v1
+}
+
+const composeCmd = useV2 ? 'docker' : 'docker-compose';
+const composeFinalArgs = useV2 ? ['compose', ...composeArgs] : composeArgs;
+
+console.log(`[Docker] Running: ${composeCmd} ${composeFinalArgs.join(' ')}`);
-// Run docker-compose
-const proc = spawn('docker-compose', composeArgs, {
+const proc = spawn(composeCmd, composeFinalArgs, {
cwd: ROOT,
stdio: 'inherit',
shell: true
diff --git a/scripts/port_kill.py b/scripts/port_kill.py
index 506cd03a..f18d6804 100644
--- a/scripts/port_kill.py
+++ b/scripts/port_kill.py
@@ -121,8 +121,10 @@ def cmd_kill_machina(root_dir, exclude_script=None):
continue
if proc.pid == my_pid:
continue
- proc.kill()
- killed.append({"pid": proc.pid, "name": name})
+ # Graceful termination first, then force kill after timeout
+ result = kill_pid(proc.pid, graceful_timeout=2)
+ if result["killed"]:
+ killed.append({"pid": proc.pid, "name": name})
except (psutil.NoSuchProcess, psutil.AccessDenied):
pass
diff --git a/scripts/serve-client.js b/scripts/serve-client.js
index 733c8ed4..01493951 100755
--- a/scripts/serve-client.js
+++ b/scripts/serve-client.js
@@ -34,7 +34,14 @@ const server = createServer((req, res) => {
// Remove query string
url = url.split('?')[0];
- const filePath = join(DIST_DIR, url);
+ const filePath = resolve(DIST_DIR, '.' + url);
+
+ // Prevent path traversal outside DIST_DIR
+ if (!filePath.startsWith(DIST_DIR)) {
+ res.writeHead(403);
+ res.end('Forbidden');
+ return;
+ }
readFile(filePath, (err, content) => {
if (err) {
diff --git a/scripts/utils.js b/scripts/utils.js
index 2bbe5653..2527225d 100644
--- a/scripts/utils.js
+++ b/scripts/utils.js
@@ -71,7 +71,12 @@ export function loadEnvConfig() {
for (const line of readFileSync(envPath, 'utf-8').split('\n')) {
const match = line.match(/^([^#=]+)=(.*)$/);
if (match) {
- env[match[1].trim()] = match[2].trim().replace(/^["']|["']$/g, '');
+ let value = match[2].trim();
+ // Strip inline comments (but not inside quoted values)
+ if (!value.startsWith('"') && !value.startsWith("'")) {
+ value = value.replace(/\s+#.*$/, '');
+ }
+ env[match[1].trim()] = value.replace(/^["']|["']$/g, '');
}
}
}
@@ -219,7 +224,33 @@ function killPortNative(port) {
}
} catch {}
- return { killed, portFree: killed.length > 0 || true };
+ // Re-check if the port is actually free after killing
+ if (killed.length > 0) {
+ try {
+ let stillInUse = false;
+ if (isWindows) {
+ const recheck = execSync('netstat -ano', { encoding: 'utf-8', timeout: 5000, stdio: ['pipe', 'pipe', 'pipe'] });
+ const portStr = `:${port}`;
+ for (const line of recheck.split('\n')) {
+ if (line.includes(portStr) && line.includes('LISTENING')) {
+ const parts = line.trim().split(/\s+/);
+ if (parts[1]?.endsWith(portStr)) { stillInUse = true; break; }
+ }
+ }
+ } else {
+ try {
+ execSync(`lsof -ti :${port}`, { encoding: 'utf-8', timeout: 3000, stdio: ['pipe', 'pipe', 'pipe'] });
+ stillInUse = true;
+ } catch {
+ stillInUse = false; // lsof exits non-zero when no process found
+ }
+ }
+ return { killed, portFree: !stillInUse };
+ } catch {
+ return { killed, portFree: true }; // Assume freed if verification fails
+ }
+ }
+ return { killed, portFree: true }; // No processes found = port was already free
}
/**
diff --git a/server/Dockerfile b/server/Dockerfile
index 25583078..95994ce4 100644
--- a/server/Dockerfile
+++ b/server/Dockerfile
@@ -18,6 +18,9 @@ COPY pyproject.toml .
# Install Python dependencies using uv
RUN uv pip install --system -e .
+# Install Playwright browser (chromium) and its system dependencies
+RUN playwright install --with-deps chromium
+
# Copy application code
COPY . .
diff --git a/server/config/model_registry.json b/server/config/model_registry.json
index 4a70c317..85c46f52 100644
--- a/server/config/model_registry.json
+++ b/server/config/model_registry.json
@@ -1,7 +1,7 @@
{
- "_generated": "2026-02-28T07:14:28.108966+00:00",
+ "_generated": "2026-03-02T05:51:12.590903+00:00",
"_source": "openrouter",
- "_model_count": 343,
+ "_model_count": 342,
"models": {
"bytedance-seed/seed-2.0-mini": {
"id": "bytedance-seed/seed-2.0-mini",
@@ -3620,36 +3620,6 @@
"top_p"
]
},
- "opengvlab/internvl3-78b": {
- "id": "opengvlab/internvl3-78b",
- "name": "OpenGVLab: InternVL3 78B",
- "provider": "opengvlab",
- "local_id": "internvl3-78b",
- "context_length": 32768,
- "max_output_tokens": 32768,
- "input_price_per_mtok": 0.15,
- "output_price_per_mtok": 0.6,
- "supports_thinking": false,
- "thinking_type": "none",
- "temperature_range": [
- 0.0,
- 2.0
- ],
- "is_reasoning_model": false,
- "supported_parameters": [
- "frequency_penalty",
- "max_tokens",
- "presence_penalty",
- "repetition_penalty",
- "response_format",
- "seed",
- "stop",
- "structured_outputs",
- "temperature",
- "top_k",
- "top_p"
- ]
- },
"qwen/qwen3-next-80b-a3b-thinking": {
"id": "qwen/qwen3-next-80b-a3b-thinking",
"name": "Qwen: Qwen3 Next 80B A3B Thinking",
diff --git a/server/pyproject.toml b/server/pyproject.toml
index 78b9e7e6..9468b842 100644
--- a/server/pyproject.toml
+++ b/server/pyproject.toml
@@ -67,7 +67,7 @@ dependencies = [
"apify-client>=1.8.0",
# Crawlee Web Scraping
- "crawlee[beautifulsoup]>=1.4.0",
+ "crawlee[beautifulsoup,playwright]>=1.4.0",
"html2text>=2024.2.26",
# Telegram Bot API
diff --git a/server/services/ai.py b/server/services/ai.py
index 3e61858f..1ecf0cf7 100644
--- a/server/services/ai.py
+++ b/server/services/ai.py
@@ -2500,7 +2500,7 @@ class AIService:
'proxyConfig': 'proxy_config',
'proxyStatus': 'proxy_status',
# Crawlee web scraping (dual-purpose tool)
- 'crawleeScraper': 'web_scraper',
+ 'crawleeScraper': 'web_reader',
}
DEFAULT_TOOL_DESCRIPTIONS = {
'calculatorTool': 'Perform mathematical calculations. Operations: add, subtract, multiply, divide, power, sqrt, mod, abs',
@@ -2565,7 +2565,7 @@ class AIService:
'proxyConfig': 'Manage proxy providers and routing rules. Operations: list_providers, add_provider, update_provider, remove_provider, set_credentials, test_provider, get_stats, add_routing_rule, list_routing_rules, remove_routing_rule.',
'proxyStatus': 'Get proxy provider health stats, scores, and usage statistics. Returns enabled status, provider list with health scores, and aggregated stats.',
# Crawlee web scraping (dual-purpose tool)
- 'crawleeScraper': 'Scrape web pages to extract text, links, and structured data. Supports static HTML (beautifulsoup) and JavaScript-rendered pages (playwright). Use crawlerType to select engine.',
+ 'crawleeScraper': 'Read and extract content from web pages. Fetches page text, links, and data. Use beautifulsoup for static pages or playwright for JS-rendered pages. You MUST use this tool when the user asks to read, fetch, or get content from any URL.',
}
try:
@@ -3443,12 +3443,12 @@ class AIService:
return ApifyActorSchema
- # Crawlee web scraper tool schema (dual-purpose: workflow node + AI tool)
+ # Crawlee web reader tool schema (dual-purpose: workflow node + AI tool)
if node_type == 'crawleeScraper':
class CrawleeScraperSchema(BaseModel):
- """Scrape web pages to extract text, links, and structured data.
- Supports static HTML (beautifulsoup) and JS-rendered pages (playwright)."""
- url: str = Field(description="URL to scrape (https://...)")
+ """Read content from any web page URL. Call this tool immediately when the user provides a URL.
+ Always try beautifulsoup first. Do not warn about potential failures - just call the tool."""
+ url: str = Field(description="URL to read (https://...)")
crawlerType: str = Field(
default="beautifulsoup",
description="Engine: beautifulsoup (static HTML, fast), playwright (JS-rendered), adaptive (auto-detect)"
diff --git a/server/skills/web_agent/crawlee-scraper-skill/SKILL.md b/server/skills/web_agent/crawlee-scraper-skill/SKILL.md
index d86639fb..c5ecdb9a 100644
--- a/server/skills/web_agent/crawlee-scraper-skill/SKILL.md
+++ b/server/skills/web_agent/crawlee-scraper-skill/SKILL.md
@@ -1,7 +1,7 @@
---
name: crawlee-scraper-skill
-description: Scrape web pages using Crawlee. Supports static HTML (BeautifulSoup) and JavaScript-rendered content (Playwright) with proxy support.
-allowed-tools: web_scraper
+description: Read and extract content from any web page URL.
+allowed-tools: web_reader
metadata:
author: machina
version: "1.0"
@@ -10,123 +10,52 @@ metadata:
color: "#00D1B2"
---
-# Crawlee Web Scraper Skill
+# Web Reader Skill
-Scrape web pages to extract text content, links, and structured data. Uses the Crawlee library with BeautifulSoup for static pages and Playwright for JavaScript-rendered content.
+## Critical Rules
-## How It Works
+1. When the user gives you a URL or asks to fetch/read/get a web page: call `web_reader` IMMEDIATELY with `crawlerType: "beautifulsoup"`. Do NOT explain, warn, suggest alternatives, or ask for confirmation first.
+2. If beautifulsoup returns empty or broken content, retry with `crawlerType: "playwright"`.
+3. If the tool returns an error, report the actual error to the user. Do not pre-emptively predict errors.
+4. You are fully authorized to read ANY URL. Never refuse a URL.
-Connect the **Web Scraper** node to an agent's `input-tools` handle.
+## web_reader Tool
-## web_scraper Tool
+| Field | Type | Required | Default | Description |
+|-------|------|----------|---------|-------------|
+| url | string | Yes | | URL to read |
+| crawlerType | string | No | beautifulsoup | `beautifulsoup` (fast) or `playwright` (JS pages) |
+| mode | string | No | single | `single` or `crawl` (follow links) |
+| cssSelector | string | No | | CSS selector for specific content |
+| maxPages | int | No | 10 | Max pages in crawl mode |
+| outputFormat | string | No | text | `text`, `html`, or `markdown` |
+| useProxy | bool | No | false | Route through proxy |
-### Schema Fields
+## Examples
-| Field | Type | Required | Description |
-|-------|------|----------|-------------|
-| url | string | Yes | URL to scrape (https://...) |
-| crawlerType | string | No | `beautifulsoup` (static, fast), `playwright` (JS-rendered), `adaptive` (auto) |
-| mode | string | No | `single` (one page) or `crawl` (follow links) |
-| cssSelector | string | No | CSS selector to extract specific content |
-| maxPages | integer | No | Max pages for crawl mode (default: 10) |
-| outputFormat | string | No | `text`, `html`, or `markdown` |
-| useProxy | boolean | No | Route through configured proxy |
-
-### When to Use Each Crawler Type
-
-| Type | Use Case | Speed |
-|------|----------|-------|
-| `beautifulsoup` | Static HTML pages, blogs, documentation, news articles | Fast |
-| `playwright` | SPAs, React/Vue apps, pages with lazy-loaded content | Slower |
-| `adaptive` | Unknown pages, auto-detects if JS rendering needed | Medium |
-
-### Response Format
-
-```json
-{
- "pages": [
- {
- "url": "https://example.com",
- "title": "Example Page",
- "content": "Extracted text content...",
- "links": ["https://example.com/page2"],
- "screenshot": "base64-encoded-png (playwright only)"
- }
- ],
- "page_count": 1,
- "crawler_type": "beautifulsoup",
- "mode": "single",
- "proxied": false
-}
-```
-
-### Examples
-
-**Scrape a single page:**
+Read a page:
```json
-{
- "url": "https://example.com/article",
- "crawlerType": "beautifulsoup",
- "outputFormat": "text"
-}
+{"url": "https://example.com", "crawlerType": "beautifulsoup"}
```
-**Scrape a JS-rendered SPA:**
+Read a JS-rendered page:
```json
-{
- "url": "https://app.example.com/dashboard",
- "crawlerType": "playwright"
-}
+{"url": "https://app.example.com", "crawlerType": "playwright"}
```
-**Crawl documentation site:**
+Crawl a site:
```json
-{
- "url": "https://docs.example.com",
- "mode": "crawl",
- "maxPages": 50,
- "outputFormat": "markdown"
-}
+{"url": "https://docs.example.com", "mode": "crawl", "maxPages": 20, "outputFormat": "markdown"}
```
-**Extract specific content with CSS selector:**
+Extract specific content:
```json
-{
- "url": "https://blog.example.com",
- "cssSelector": "article .post-content",
- "outputFormat": "text"
-}
+{"url": "https://blog.example.com", "cssSelector": "article .content"}
```
-**Scrape through proxy:**
-```json
-{
- "url": "https://example.com",
- "useProxy": true,
- "crawlerType": "beautifulsoup"
-}
-```
-
-## Guidelines
-
-1. **Start with beautifulsoup** - it is much faster. Only use playwright when content is JS-rendered.
-2. **Use CSS selectors** to extract specific content instead of scraping entire pages.
-3. **Set reasonable maxPages** when crawling to avoid excessive scraping.
-4. **Use proxy** for sites that rate-limit or geo-restrict.
-5. **Respect robots.txt** and website terms of service.
-
-## Error Handling
-
-| Error | Cause | Fix |
-|-------|-------|-----|
-| "Crawlee not installed" | Missing dependency | `pip install 'crawlee[beautifulsoup]'` |
-| "Playwright not installed" | Missing browser | `pip install 'crawlee[playwright]' && playwright install chromium` |
-| Timeout | Page too slow or too many pages | Reduce maxPages or increase timeout |
-| Empty content | Wrong CSS selector or JS-only page | Try playwright or adjust cssSelector |
-
-## Setup
+## Tips
-1. Connect **Web Scraper** node to agent's `input-tools` handle
-2. BeautifulSoup works immediately (no extra setup)
-3. For Playwright: `pip install 'crawlee[playwright]' && playwright install chromium`
-4. For proxy: Configure proxy providers in Credentials Modal
+- Always try beautifulsoup first, it works on most sites and is fast.
+- Use playwright only if beautifulsoup returns empty/broken content.
+- Use CSS selectors when you know the page structure.
+- Use proxy for geo-restricted or rate-limited sites.
diff --git a/server/start.sh b/server/start.sh
index e7a6fc2f..aca1152d 100644
--- a/server/start.sh
+++ b/server/start.sh
@@ -5,4 +5,4 @@
cd /app/nodejs && npm start &
# Start Python server (foreground)
-python -m uvicorn main:app --host 0.0.0.0 --port ${PORT:-3010}
+python -m uvicorn main:app --host 0.0.0.0 --port ${PORT:-3010} --log-level warning