Self-improving AI Employees for everyone
Python ai-agentsai-employeeai-employeesassistantautonomous-agentsco-employeescompanygrok-botinstinctloop-agentsmuseown-your-datapersonalself-hosted
fix: unified data_dir config, sqlalchemy preflight probe, errors docs
trohitg0c7028fd2692
diff --git a/.env.template b/.env.templateindex 07da43c1..3839a43e 100644--- a/.env.template+++ b/.env.template@@ -46,8 +46,11 @@ TEMPORAL_SERVER_ADDRESS=localhost:7233TEMPORAL_NAMESPACE=defaultTEMPORAL_TASK_QUEUE=machina-tasks-# Per-workflow workspace (nodes save files here, agents access via filesystem tools)-WORKSPACE_BASE_DIR=data/workspaces+# Data directory (base for all persistent storage: DBs, workspaces, logs)+DATA_DIR=data++# Per-workflow workspace (nodes save files here, resolved under DATA_DIR)+WORKSPACE_BASE_DIR=workspaces# LoggingLOG_LEVEL=INFOdiff --git a/CLAUDE.md b/CLAUDE.mdindex 0d81eb85..9280d152 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -28,6 +28,7 @@ This is a React Flow-based workflow automation platform implementing n8n-inspire| **[Scripts Reference](./docs-internal/SCRIPTS.md)** | Available npm/shell scripts and their usage || **[Server Documentation](./docs-internal/server-readme.md)** | Python backend architecture and API documentation || **[Skill Creation Guide](./server/skills/GUIDE.md)** | How to create new skills (folder structure, SKILL.md format, metadata, supporting files) |+| **[Known Errors & Troubleshooting](./docs-internal/errors.md)** | Documented root causes and fixes for common errors (SQLAlchemy Windows hang, Temporal issues, WhatsApp timeouts) || **[New Service Integration](./docs-internal/new_service_integration.md)** | Complete guide for integrating external services (OAuth, database, handlers, nodes, AI tools) - use Google Workspace as reference || **[Onboarding Service](./docs-internal/onboarding.md)** | First-launch welcome wizard with 5 steps, database persistence, and replay from Settings || **[CLI Services Integration](./docs-internal/cli_services_integration.md)** | Guide for integrating CLI-based services (Temporal, etc.) with proper lifecycle management |diff --git a/docs-internal/errors.md b/docs-internal/errors.mdnew file mode 100644index 00000000..19840482--- /dev/null+++ b/docs-internal/errors.md@@ -0,0 +1,252 @@+# Known Errors & Troubleshooting++Documented root causes and fixes for errors encountered in MachinaOs development and production.++---++## 1. SQLAlchemy Import Hang (Windows)++**Symptom**: Backend hangs at startup with no output after `Importing DI container + all services...`. The process is alive but never binds port 3010. `import sqlalchemy` blocks indefinitely.++**Root cause**: Git worktrees nested inside the project root (e.g., `.claude/worktrees/`) cause Windows Defender real-time scanning to fan out across all worktree directories when Python loads `.pyd` (native DLL) files. SQLAlchemy has 5 Cython `.pyd` files (`collections`, `immutabledict`, `processors`, `resultproxy`, `util`) loaded sequentially during import. Defender's scan queue backs up across the worktree copies, blocking `LoadLibrary()` for minutes per file.++**Contributing factors**:+- Each worktree contains its own `.venv/`, `node_modules/`, and source tree (thousands of files)+- Windows Search Indexer and Defender monitor directory trees recursively from the project root+- pnpm hardlinks from `.pnpm-store` into each worktree's `node_modules/` create additional file-system contention+- Killing the hung Python process does NOT help -- the next attempt restarts the scan queue from scratch++**Fix**: Move or remove worktrees from inside the project root.++```bash+# Remove worktrees+git worktree remove .claude/worktrees/<name>++# Or move them outside the project root+git worktree move .claude/worktrees/<name> ../machinaos-worktrees/<name>+```++**Prevention**: Keep git worktrees as siblings of the project, not nested inside it. For example:+```+d:/startup/projects/+ MachinaOs/ # main project+ machinaos-worktrees/ # worktrees outside the project root+ credentials-scaling/+ native-llm-sdk/+```++**Verification**: After removing worktrees, `import sqlalchemy` should complete in <1 second:+```bash+cd server && uv run python -u -c "import time; t=time.time(); import sqlalchemy; print(f'{time.time()-t:.2f}s')"+```++### 1a. SQLAlchemy Hang Persists After Removing Worktrees++**Symptom**: Even after moving worktrees outside the project root and adding the `.venv` to Defender exclusions, `import sqlalchemy` still hangs. Only a system reboot resolves it.++**Root cause**: Defender's minifilter driver (`MpFilter.sys`) caches scan verdicts in a kernel-mode cache keyed by file identity (volume + file reference number + USN). When scans were previously backed up, some entries stay in "pending scan" state indefinitely. Defender exclusions added at runtime do NOT evict existing pending cache entries -- only a Defender service restart or full reboot clears the minifilter's in-memory state.++Contributing factors:+- Killing stuck Python processes via `Stop-Process` can trigger Defender to re-scan the DLL handles those processes had mapped+- SysMain/Superfetch prefetch contention on cold venv imports+- NTFS USN journal backlog from large file-churn operations (pip installs, worktree moves)++**Fix** (without reboot):+```powershell+# Restart Defender service (admin required)+Restart-Service WinDefend++# Or restart SysMain+Restart-Service SysMain+```++**Fix** (if admin is unavailable): **Reboot**. This is what clears the stuck kernel cache reliably.++**Prevention** (`scripts/start.js`): A preflight probe times `import sqlalchemy`. If it exceeds 8 seconds, it fails fast with actionable remediation steps instead of letting uvicorn hang silently.++---++## 2. Temporal `context canceled` / `UpdateTaskQueue` Errors++**Symptom**: Temporal server logs show recurring errors even when no workflows are running:+```+level=ERROR msg="Operation failed with internal error."+error="UpdateTaskQueue failed. Failed to start transaction. Error: context canceled"+component=matching-engine wf-namespace=temporal-system+```++**Root cause**: Temporal's SQLite database runs in DELETE journal mode by default, which allows only one writer at a time. Temporal's internal system workflows (namespace replication, queue metadata maintenance, backlog counters) contend for write access. When multiple internal workflows try to update task queue metadata concurrently, `BeginTx` blocks, the gRPC context deadline elapses, and the transaction is cancelled.++**Fix**: Enable WAL (Write-Ahead Logging) mode via `--sqlite-pragma` flags. This is configured in the `temporal-server` npm package (v0.0.10+) via `configs/server.json`:++```json+{+ "sqlitePragma": ["journal_mode=wal", "busy_timeout=5000"]+}+```++- `journal_mode=wal` -- concurrent read/write access instead of single-writer DELETE mode+- `busy_timeout=5000` -- wait up to 5s for a write lock instead of failing immediately++After changing the pragma config, delete the existing `temporal.db` so the new DB is created with WAL mode from the start:++```bash+temporal stop+rm <temporal-server-package>/data/temporal.db+temporal start+```++**Verification**: After restart, check the metrics endpoint. `persistence_error_with_type` counters for `serviceerror_Unavailable` should stop incrementing:+```bash+curl -s http://localhost:9090/metrics | grep "persistence_error_with_type" | grep "Unavailable"+```++WAL mode is confirmed when `.db-wal` and `.db-shm` sidecar files appear next to `temporal.db`.++---++## 3. Temporal Activity `CancelledError` on Long-Running Nodes++**Symptom**: Nodes that run for more than ~2 minutes (Deep Agent, browser automation, AI multi-tool loops) fail with:+```+asyncio.exceptions.CancelledError+```+in the Temporal activity at `activities.py` line `async for msg in ws:`.++The Temporal UI shows the activity failed with `TIMEOUT_TYPE_HEARTBEAT`.++**Root cause**: The activity WebSocket read loop only sent heartbeats when a non-matching WebSocket message arrived. During long-running operations where the backend processes internally without broadcasting any WS messages, no heartbeats fire. Temporal's 2-minute `heartbeat_timeout` expires and cancels the activity.++**Fix** (applied in activities.py): Replace the `async for msg in ws:` iterator with an explicit `asyncio.wait_for(ws.receive(), timeout=30.0)` loop. On timeout (no message in 30s), a heartbeat fires and the loop continues. This guarantees heartbeats every 30 seconds regardless of WebSocket traffic.++```python+# Before (broken for long-running nodes):+async for msg in ws:+ if msg.type == aiohttp.WSMsgType.TEXT:+ response = json.loads(msg.data)+ if response.get("request_id") == request_id:+ return response+ activity.heartbeat(f"Waiting for {node_id}") # Only fires on messages++# After (heartbeats even when no messages arrive):+while True:+ try:+ msg = await asyncio.wait_for(ws.receive(), timeout=30.0)+ except asyncio.TimeoutError:+ activity.heartbeat(f"Waiting for {node_id}") # Fires every 30s guaranteed+ continue+ # ... handle msg+```++Also changed `receive_timeout=540` to `receive_timeout=None` on `ws_connect()` -- the old 9-minute aiohttp-level timeout was a second hard cap that could kill activities independently of the heartbeat mechanism. Liveness is now managed entirely by Temporal heartbeats.++**Timeout configuration reference**:++| Parameter | Value | Purpose |+|-----------|-------|---------|+| `start_to_close_timeout` | 10 min | Maximum total time for an activity (workflow.py) |+| `heartbeat_timeout` | 2 min | Maximum gap between heartbeats before Temporal cancels (workflow.py) |+| `asyncio.wait_for` timeout | 30s | Periodic heartbeat interval in the WS read loop (activities.py) |+| `ws_connect heartbeat` | 30s | WebSocket protocol-level ping/pong keepalive (activities.py) |+| `receive_timeout` | None | No aiohttp-level hard cap (was 540s) |++---++## 4. WhatsApp RPC Timeout++**Symptom**: Backend logs show:+```+WhatsApp RPC timeout - Go service not responding at ws://localhost:9400/ws/rpc+```++WhatsApp service health check (`/health`) returns 200 OK, but the WebSocket RPC connection fails.++**Root cause**: The RPCClient WebSocket connect timeout was set to 2.0 seconds (`routers/whatsapp.py`). The Go whatsmeow service's WebSocket handshake can take 2-3 seconds on Windows, especially on cold start or when Defender is scanning the binary. A 2.1s handshake exceeds the 2.0s deadline.++**Fix**: Increased the connect timeout from 2.0s to 5.0s in `RPCClient.connect()`:++```python+self.ws = await asyncio.wait_for(+ websockets.connect(self.url, ping_interval=30, max_size=100*1024*1024),+ timeout=5.0 # Was 2.0 -- too tight for Windows cold start+)+```++**Note**: This was also triggered by upgrading `edgymeow` from 0.0.18 to 0.0.19, where the newer Go binary had a slightly slower WebSocket handshake. Reverted to 0.0.18 pending investigation of the Go-side slowdown. The 5.0s timeout fix is correct regardless of version.++---++## 5. `ERR_CONNECTION_REFUSED` on Frontend Auth Check++**Symptom**: After `pnpm run dev`, browser console shows repeated errors:+```+GET http://localhost:3010/api/auth/status net::ERR_CONNECTION_REFUSED+Failed to check auth status (attempt 4/6): TypeError: Failed to fetch+```++**Root cause**: The FastAPI lifespan blocks on Temporal client connection (`await temporal_client_wrapper.connect(retries=10, delay=3.0)`) for up to 30 seconds. During this window, uvicorn has not called `yield` yet, so it is not accepting HTTP connections. The frontend retry window (1+2+4+8+16 = 31s) barely overlaps with the blocking window and can exhaust before the backend starts serving.++**Fix**: Move the Temporal initialization into a background `asyncio.create_task()` so the lifespan yields immediately. WorkflowService falls back to parallel/sequential execution until Temporal connects in the background. See the stashed refactor in `git stash list` for the implementation.++**Workaround** (if the refactor is not applied): Increase the frontend retry count or delay in `AuthContext.tsx`, or ensure `temporal-server` is already running before starting the Python backend.++---++## 6. `temporal-server` Binary Not Found++**Symptom**:+```+Binary not found. Run: npm run postinstall+```++**Root cause**: The `temporal-server` npm package downloads the Temporal CLI binary (~60MB) during its `postinstall` script. If `pnpm install` was interrupted, if `--ignore-scripts` was used, or if a previous install was corrupted, the binary at `<package>/bin/temporal.exe` is missing.++**Fix**:+```bash+# Re-run the postinstall download+npx --no-install temporal postinstall++# Or reinstall the package+pnpm install+```++---++## 7. Python Version Mismatch Warning++**Symptom**: Every `uv run` command shows:+```+warning: `VIRTUAL_ENV=C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13...`+does not match the project environment path `.venv` and will be ignored+```++**Root cause**: A parent process (e.g., Claude Code's harness) leaked a `VIRTUAL_ENV` environment variable pointing to the Windows Store Python 3.13 installation. This is NOT a virtualenv -- it's a system Python install directory incorrectly set as `VIRTUAL_ENV`.++**Impact**: None. uv correctly ignores the leaked env var and uses the project's `.venv` (Python 3.12.8). The warning is cosmetic noise from the parent shell environment.++**Fix**: Unset the variable in your shell:+```bash+unset VIRTUAL_ENV+```++Or start a fresh terminal that doesn't inherit from the Claude Code harness.++---++## 8. `install.js` Python Version Check Accepts 3.13+++**Symptom**: `scripts/install.js` reports `Python: Python 3.13.7` as valid, but `pyproject.toml` requires `>=3.11,<3.13`.++**Root cause**: The version check at `install.js:59` uses `minor >= 12` with no upper bound:+```js+if (major >= 3 && minor >= 12) { return { cmd, version }; }+```++This accepts Python 3.13, 3.14, etc. even though the project constraint is `<3.13`.++**Impact**: Low -- `uv sync` independently enforces `requires-python` from `pyproject.toml` and downloads a compatible Python (3.12.x) regardless of what `install.js` reports. The user sees misleading output but the .venv is built correctly.++**Fix**: Update the check to match `pyproject.toml`:+```js+if (major === 3 && minor >= 11 && minor < 13) { return { cmd, version }; }+```diff --git a/scripts/clean.js b/scripts/clean.jsindex d3c2cc09..b87947c8 100755--- a/scripts/clean.js+++ b/scripts/clean.js@@ -21,10 +21,9 @@ const targets = ['client/node_modules','client/dist','client/.vite',- 'server/data',+ 'server/data', // DBs (workflow.db, credentials.db) + workspaces'server/.venv',- 'server/credentials.db', // Encrypted credentials database- '.venv', // Root venv (stale, should not exist)+ '.venv', // Root venv (stale, should not exist)];async function main() {diff --git a/scripts/start.js b/scripts/start.jsindex c95abd3a..8634d29a 100755--- a/scripts/start.js+++ b/scripts/start.js@@ -44,6 +44,36 @@ async function main() {process.exit(1);}+ // Preflight: time-boxed sqlalchemy import probe.+ // On Windows, Defender's minifilter driver (MpFilter.sys) can cache stale+ // "pending scan" entries that block .pyd LoadLibrary calls even after+ // adding exclusions. See docs-internal/errors.md #1 and #1a.+ const pyExe = resolve(ROOT, 'server', '.venv', 'Scripts', isWindows ? 'python.exe' : 'python');+ const pyFallback = resolve(ROOT, 'server', '.venv', 'bin', 'python');+ const py = existsSync(pyExe) ? pyExe : pyFallback;+ if (existsSync(py)) {+ const probeStart = Date.now();+ try {+ execSync(`"${py}" -c "import sqlalchemy"`, { timeout: 15000, stdio: 'pipe' });+ const elapsed = Date.now() - probeStart;+ if (elapsed > 5000) {+ console.warn(`Warning: sqlalchemy import took ${(elapsed / 1000).toFixed(1)}s (expected <1s).`);+ console.warn(' See docs-internal/errors.md #1 for Windows Defender remediation.');+ }+ } catch (e) {+ const elapsed = Date.now() - probeStart;+ console.error(`Error: Python venv health check failed (${(elapsed / 1000).toFixed(1)}s).`);+ console.error(' sqlalchemy import hung or crashed.');+ console.error(' Likely cause: Windows Defender scan cache or stale kernel state.');+ console.error(' Fix options:');+ console.error(' 1. Restart-Service WinDefend (admin PowerShell)');+ console.error(' 2. Reboot the machine');+ console.error(' 3. Add D:\\...\\server\\.venv to Defender exclusions');+ console.error(' See docs-internal/errors.md #1 and #1a for details.');+ process.exit(1);+ }+ }+const config = loadEnvConfig();process.env.PYTHONUTF8 = '1';ensureEnvFile();diff --git a/server/core/config.py b/server/core/config.pyindex 2928fae1..3be10076 100644--- a/server/core/config.py+++ b/server/core/config.py@@ -86,7 +86,11 @@ class Settings(BaseSettings):api_key_encryption_key: str = Field(env="API_KEY_ENCRYPTION_KEY", min_length=32)api_key_cache_ttl: int = Field(default=2592000, env="API_KEY_CACHE_TTL", ge=3600)+ # Data directory (base for all persistent storage: DBs, workspaces, logs)+ data_dir: str = Field(default="data", env="DATA_DIR")+# Credentials Database (separate encrypted database for API keys and OAuth tokens)+ # Resolved relative to data_dir unless absolutecredentials_db_path: str = Field(default="credentials.db", env="CREDENTIALS_DB_PATH")# Credential Backend Selection@@ -126,7 +130,8 @@ class Settings(BaseSettings):ws_logging_enabled: bool = Field(default=True, env="WS_LOGGING_ENABLED")# Workspace Configuration (per-workflow file storage for nodes and agents)- workspace_base_dir: str = Field(default="data/workspaces", env="WORKSPACE_BASE_DIR")+ # Workspace base -- relative to data_dir unless absolute+ workspace_base_dir: str = Field(default="workspaces", env="WORKSPACE_BASE_DIR")# Compaction Configurationcompaction_enabled: bool = Field(default=True, env="COMPACTION_ENABLED")@@ -160,6 +165,23 @@ class Settings(BaseSettings):"""Check if running in production mode."""return not self.debug+ def _resolve_under_data(self, path: str) -> str:+ """Resolve a path relative to data_dir, unless already absolute."""+ p = Path(path)+ if p.is_absolute():+ return str(p)+ return str(Path(self.data_dir) / p)++ @property+ def credentials_db_resolved(self) -> str:+ """Full credentials DB path, rooted under data_dir."""+ return self._resolve_under_data(self.credentials_db_path)++ @property+ def workspace_base_resolved(self) -> str:+ """Full workspace base path, rooted under data_dir."""+ return self._resolve_under_data(self.workspace_base_dir)+model_config = {"env_file": "../.env","env_file_encoding": "utf-8",diff --git a/server/core/container.py b/server/core/container.pyindex f647b69f..a957df95 100644--- a/server/core/container.py+++ b/server/core/container.py@@ -70,7 +70,7 @@ class Container(containers.DeclarativeContainer):# Credentials database (separate encrypted database for API keys and OAuth tokens)credentials_database = providers.Singleton(CredentialsDatabase,- db_path=settings.provided.credentials_db_path,+ db_path=settings.provided.credentials_db_resolved,encryption=encryption_service)diff --git a/server/services/claude_code_service.py b/server/services/claude_code_service.pyindex a142a102..5d17bc24 100644--- a/server/services/claude_code_service.py+++ b/server/services/claude_code_service.py@@ -60,7 +60,7 @@ class ClaudeCodeService:if not cwd:from core.config import Settings- cwd = os.path.join(Settings().workspace_base_dir, 'default')+ cwd = os.path.join(Settings().workspace_base_resolved, 'default')os.makedirs(cwd, exist_ok=True)work_dir = cwddiff --git a/server/services/handlers/filesystem.py b/server/services/handlers/filesystem.pyindex ed4bbbef..0ead9530 100644--- a/server/services/handlers/filesystem.py+++ b/server/services/handlers/filesystem.py@@ -24,7 +24,7 @@ def _get_backend(parameters: Dict[str, Any], context: Dict[str, Any] = None):from core.config import Settingsparam_dir = parameters.get('working_directory')ctx_dir = context.get('workspace_dir') if context else None- root = param_dir or ctx_dir or os.path.join(Settings().workspace_base_dir, 'default')+ root = param_dir or ctx_dir or os.path.join(Settings().workspace_base_resolved, 'default')os.makedirs(root, exist_ok=True)logger.info("[Filesystem] root=%s", root)return LocalShellBackend(root_dir=root, virtual_mode=True)diff --git a/server/services/process_service.py b/server/services/process_service.pyindex 58f45c0c..97286792 100644--- a/server/services/process_service.py+++ b/server/services/process_service.py@@ -104,7 +104,7 @@ class ProcessService:argv = shlex.split(command)env = {**os.environ, "PYTHONUNBUFFERED": "1"}from core.config import Settings- workspace_base = Path(Settings().workspace_base_dir).resolve()+ workspace_base = Path(Settings().workspace_base_resolved).resolve()if not working_directory:working_directory = str(workspace_base / 'default')diff --git a/server/services/workflow.py b/server/services/workflow.pyindex e2548931..00f3f5f6 100644--- a/server/services/workflow.py+++ b/server/services/workflow.py@@ -135,7 +135,7 @@ class WorkflowService:def _get_workspace_dir(self, workflow_id: Optional[str]) -> str:"""Get or create workspace directory for a workflow."""- base = Path(self.settings.workspace_base_dir)+ base = Path(self.settings.workspace_base_resolved)wf_id = workflow_id or "default"workspace = base / wf_idworkspace.mkdir(parents=True, exist_ok=True)