Skip to content

Self-improving AI Employees for everyone

fix: unified data_dir config, sqlalchemy preflight probe, errors docs

trohitg0c7028fd2692
diff --git a/.env.template b/.env.template
index 07da43c1..3839a43e 100644
--- a/.env.template
+++ b/.env.template
@@ -46,8 +46,11 @@ TEMPORAL_SERVER_ADDRESS=localhost:7233
TEMPORAL_NAMESPACE=default
TEMPORAL_TASK_QUEUE=machina-tasks
-# Per-workflow workspace (nodes save files here, agents access via filesystem tools)
-WORKSPACE_BASE_DIR=data/workspaces
+# Data directory (base for all persistent storage: DBs, workspaces, logs)
+DATA_DIR=data
+
+# Per-workflow workspace (nodes save files here, resolved under DATA_DIR)
+WORKSPACE_BASE_DIR=workspaces
# Logging
LOG_LEVEL=INFO
diff --git a/CLAUDE.md b/CLAUDE.md
index 0d81eb85..9280d152 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -28,6 +28,7 @@ This is a React Flow-based workflow automation platform implementing n8n-inspire
| **[Scripts Reference](./docs-internal/SCRIPTS.md)** | Available npm/shell scripts and their usage |
| **[Server Documentation](./docs-internal/server-readme.md)** | Python backend architecture and API documentation |
| **[Skill Creation Guide](./server/skills/GUIDE.md)** | How to create new skills (folder structure, SKILL.md format, metadata, supporting files) |
+| **[Known Errors & Troubleshooting](./docs-internal/errors.md)** | Documented root causes and fixes for common errors (SQLAlchemy Windows hang, Temporal issues, WhatsApp timeouts) |
| **[New Service Integration](./docs-internal/new_service_integration.md)** | Complete guide for integrating external services (OAuth, database, handlers, nodes, AI tools) - use Google Workspace as reference |
| **[Onboarding Service](./docs-internal/onboarding.md)** | First-launch welcome wizard with 5 steps, database persistence, and replay from Settings |
| **[CLI Services Integration](./docs-internal/cli_services_integration.md)** | Guide for integrating CLI-based services (Temporal, etc.) with proper lifecycle management |
diff --git a/docs-internal/errors.md b/docs-internal/errors.md
new file mode 100644
index 00000000..19840482
--- /dev/null
+++ b/docs-internal/errors.md
@@ -0,0 +1,252 @@
+# Known Errors & Troubleshooting
+
+Documented root causes and fixes for errors encountered in MachinaOs development and production.
+
+---
+
+## 1. SQLAlchemy Import Hang (Windows)
+
+**Symptom**: Backend hangs at startup with no output after `Importing DI container + all services...`. The process is alive but never binds port 3010. `import sqlalchemy` blocks indefinitely.
+
+**Root cause**: Git worktrees nested inside the project root (e.g., `.claude/worktrees/`) cause Windows Defender real-time scanning to fan out across all worktree directories when Python loads `.pyd` (native DLL) files. SQLAlchemy has 5 Cython `.pyd` files (`collections`, `immutabledict`, `processors`, `resultproxy`, `util`) loaded sequentially during import. Defender's scan queue backs up across the worktree copies, blocking `LoadLibrary()` for minutes per file.
+
+**Contributing factors**:
+- Each worktree contains its own `.venv/`, `node_modules/`, and source tree (thousands of files)
+- Windows Search Indexer and Defender monitor directory trees recursively from the project root
+- pnpm hardlinks from `.pnpm-store` into each worktree's `node_modules/` create additional file-system contention
+- Killing the hung Python process does NOT help -- the next attempt restarts the scan queue from scratch
+
+**Fix**: Move or remove worktrees from inside the project root.
+
+```bash
+# Remove worktrees
+git worktree remove .claude/worktrees/<name>
+
+# Or move them outside the project root
+git worktree move .claude/worktrees/<name> ../machinaos-worktrees/<name>
+```
+
+**Prevention**: Keep git worktrees as siblings of the project, not nested inside it. For example:
+```
+d:/startup/projects/
+ MachinaOs/ # main project
+ machinaos-worktrees/ # worktrees outside the project root
+ credentials-scaling/
+ native-llm-sdk/
+```
+
+**Verification**: After removing worktrees, `import sqlalchemy` should complete in <1 second:
+```bash
+cd server && uv run python -u -c "import time; t=time.time(); import sqlalchemy; print(f'{time.time()-t:.2f}s')"
+```
+
+### 1a. SQLAlchemy Hang Persists After Removing Worktrees
+
+**Symptom**: Even after moving worktrees outside the project root and adding the `.venv` to Defender exclusions, `import sqlalchemy` still hangs. Only a system reboot resolves it.
+
+**Root cause**: Defender's minifilter driver (`MpFilter.sys`) caches scan verdicts in a kernel-mode cache keyed by file identity (volume + file reference number + USN). When scans were previously backed up, some entries stay in "pending scan" state indefinitely. Defender exclusions added at runtime do NOT evict existing pending cache entries -- only a Defender service restart or full reboot clears the minifilter's in-memory state.
+
+Contributing factors:
+- Killing stuck Python processes via `Stop-Process` can trigger Defender to re-scan the DLL handles those processes had mapped
+- SysMain/Superfetch prefetch contention on cold venv imports
+- NTFS USN journal backlog from large file-churn operations (pip installs, worktree moves)
+
+**Fix** (without reboot):
+```powershell
+# Restart Defender service (admin required)
+Restart-Service WinDefend
+
+# Or restart SysMain
+Restart-Service SysMain
+```
+
+**Fix** (if admin is unavailable): **Reboot**. This is what clears the stuck kernel cache reliably.
+
+**Prevention** (`scripts/start.js`): A preflight probe times `import sqlalchemy`. If it exceeds 8 seconds, it fails fast with actionable remediation steps instead of letting uvicorn hang silently.
+
+---
+
+## 2. Temporal `context canceled` / `UpdateTaskQueue` Errors
+
+**Symptom**: Temporal server logs show recurring errors even when no workflows are running:
+```
+level=ERROR msg="Operation failed with internal error."
+error="UpdateTaskQueue failed. Failed to start transaction. Error: context canceled"
+component=matching-engine wf-namespace=temporal-system
+```
+
+**Root cause**: Temporal's SQLite database runs in DELETE journal mode by default, which allows only one writer at a time. Temporal's internal system workflows (namespace replication, queue metadata maintenance, backlog counters) contend for write access. When multiple internal workflows try to update task queue metadata concurrently, `BeginTx` blocks, the gRPC context deadline elapses, and the transaction is cancelled.
+
+**Fix**: Enable WAL (Write-Ahead Logging) mode via `--sqlite-pragma` flags. This is configured in the `temporal-server` npm package (v0.0.10+) via `configs/server.json`:
+
+```json
+{
+ "sqlitePragma": ["journal_mode=wal", "busy_timeout=5000"]
+}
+```
+
+- `journal_mode=wal` -- concurrent read/write access instead of single-writer DELETE mode
+- `busy_timeout=5000` -- wait up to 5s for a write lock instead of failing immediately
+
+After changing the pragma config, delete the existing `temporal.db` so the new DB is created with WAL mode from the start:
+
+```bash
+temporal stop
+rm <temporal-server-package>/data/temporal.db
+temporal start
+```
+
+**Verification**: After restart, check the metrics endpoint. `persistence_error_with_type` counters for `serviceerror_Unavailable` should stop incrementing:
+```bash
+curl -s http://localhost:9090/metrics | grep "persistence_error_with_type" | grep "Unavailable"
+```
+
+WAL mode is confirmed when `.db-wal` and `.db-shm` sidecar files appear next to `temporal.db`.
+
+---
+
+## 3. Temporal Activity `CancelledError` on Long-Running Nodes
+
+**Symptom**: Nodes that run for more than ~2 minutes (Deep Agent, browser automation, AI multi-tool loops) fail with:
+```
+asyncio.exceptions.CancelledError
+```
+in the Temporal activity at `activities.py` line `async for msg in ws:`.
+
+The Temporal UI shows the activity failed with `TIMEOUT_TYPE_HEARTBEAT`.
+
+**Root cause**: The activity WebSocket read loop only sent heartbeats when a non-matching WebSocket message arrived. During long-running operations where the backend processes internally without broadcasting any WS messages, no heartbeats fire. Temporal's 2-minute `heartbeat_timeout` expires and cancels the activity.
+
+**Fix** (applied in activities.py): Replace the `async for msg in ws:` iterator with an explicit `asyncio.wait_for(ws.receive(), timeout=30.0)` loop. On timeout (no message in 30s), a heartbeat fires and the loop continues. This guarantees heartbeats every 30 seconds regardless of WebSocket traffic.
+
+```python
+# Before (broken for long-running nodes):
+async for msg in ws:
+ if msg.type == aiohttp.WSMsgType.TEXT:
+ response = json.loads(msg.data)
+ if response.get("request_id") == request_id:
+ return response
+ activity.heartbeat(f"Waiting for {node_id}") # Only fires on messages
+
+# After (heartbeats even when no messages arrive):
+while True:
+ try:
+ msg = await asyncio.wait_for(ws.receive(), timeout=30.0)
+ except asyncio.TimeoutError:
+ activity.heartbeat(f"Waiting for {node_id}") # Fires every 30s guaranteed
+ continue
+ # ... handle msg
+```
+
+Also changed `receive_timeout=540` to `receive_timeout=None` on `ws_connect()` -- the old 9-minute aiohttp-level timeout was a second hard cap that could kill activities independently of the heartbeat mechanism. Liveness is now managed entirely by Temporal heartbeats.
+
+**Timeout configuration reference**:
+
+| Parameter | Value | Purpose |
+|-----------|-------|---------|
+| `start_to_close_timeout` | 10 min | Maximum total time for an activity (workflow.py) |
+| `heartbeat_timeout` | 2 min | Maximum gap between heartbeats before Temporal cancels (workflow.py) |
+| `asyncio.wait_for` timeout | 30s | Periodic heartbeat interval in the WS read loop (activities.py) |
+| `ws_connect heartbeat` | 30s | WebSocket protocol-level ping/pong keepalive (activities.py) |
+| `receive_timeout` | None | No aiohttp-level hard cap (was 540s) |
+
+---
+
+## 4. WhatsApp RPC Timeout
+
+**Symptom**: Backend logs show:
+```
+WhatsApp RPC timeout - Go service not responding at ws://localhost:9400/ws/rpc
+```
+
+WhatsApp service health check (`/health`) returns 200 OK, but the WebSocket RPC connection fails.
+
+**Root cause**: The RPCClient WebSocket connect timeout was set to 2.0 seconds (`routers/whatsapp.py`). The Go whatsmeow service's WebSocket handshake can take 2-3 seconds on Windows, especially on cold start or when Defender is scanning the binary. A 2.1s handshake exceeds the 2.0s deadline.
+
+**Fix**: Increased the connect timeout from 2.0s to 5.0s in `RPCClient.connect()`:
+
+```python
+self.ws = await asyncio.wait_for(
+ websockets.connect(self.url, ping_interval=30, max_size=100*1024*1024),
+ timeout=5.0 # Was 2.0 -- too tight for Windows cold start
+)
+```
+
+**Note**: This was also triggered by upgrading `edgymeow` from 0.0.18 to 0.0.19, where the newer Go binary had a slightly slower WebSocket handshake. Reverted to 0.0.18 pending investigation of the Go-side slowdown. The 5.0s timeout fix is correct regardless of version.
+
+---
+
+## 5. `ERR_CONNECTION_REFUSED` on Frontend Auth Check
+
+**Symptom**: After `pnpm run dev`, browser console shows repeated errors:
+```
+GET http://localhost:3010/api/auth/status net::ERR_CONNECTION_REFUSED
+Failed to check auth status (attempt 4/6): TypeError: Failed to fetch
+```
+
+**Root cause**: The FastAPI lifespan blocks on Temporal client connection (`await temporal_client_wrapper.connect(retries=10, delay=3.0)`) for up to 30 seconds. During this window, uvicorn has not called `yield` yet, so it is not accepting HTTP connections. The frontend retry window (1+2+4+8+16 = 31s) barely overlaps with the blocking window and can exhaust before the backend starts serving.
+
+**Fix**: Move the Temporal initialization into a background `asyncio.create_task()` so the lifespan yields immediately. WorkflowService falls back to parallel/sequential execution until Temporal connects in the background. See the stashed refactor in `git stash list` for the implementation.
+
+**Workaround** (if the refactor is not applied): Increase the frontend retry count or delay in `AuthContext.tsx`, or ensure `temporal-server` is already running before starting the Python backend.
+
+---
+
+## 6. `temporal-server` Binary Not Found
+
+**Symptom**:
+```
+Binary not found. Run: npm run postinstall
+```
+
+**Root cause**: The `temporal-server` npm package downloads the Temporal CLI binary (~60MB) during its `postinstall` script. If `pnpm install` was interrupted, if `--ignore-scripts` was used, or if a previous install was corrupted, the binary at `<package>/bin/temporal.exe` is missing.
+
+**Fix**:
+```bash
+# Re-run the postinstall download
+npx --no-install temporal postinstall
+
+# Or reinstall the package
+pnpm install
+```
+
+---
+
+## 7. Python Version Mismatch Warning
+
+**Symptom**: Every `uv run` command shows:
+```
+warning: `VIRTUAL_ENV=C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13...`
+does not match the project environment path `.venv` and will be ignored
+```
+
+**Root cause**: A parent process (e.g., Claude Code's harness) leaked a `VIRTUAL_ENV` environment variable pointing to the Windows Store Python 3.13 installation. This is NOT a virtualenv -- it's a system Python install directory incorrectly set as `VIRTUAL_ENV`.
+
+**Impact**: None. uv correctly ignores the leaked env var and uses the project's `.venv` (Python 3.12.8). The warning is cosmetic noise from the parent shell environment.
+
+**Fix**: Unset the variable in your shell:
+```bash
+unset VIRTUAL_ENV
+```
+
+Or start a fresh terminal that doesn't inherit from the Claude Code harness.
+
+---
+
+## 8. `install.js` Python Version Check Accepts 3.13+
+
+**Symptom**: `scripts/install.js` reports `Python: Python 3.13.7` as valid, but `pyproject.toml` requires `>=3.11,<3.13`.
+
+**Root cause**: The version check at `install.js:59` uses `minor >= 12` with no upper bound:
+```js
+if (major >= 3 && minor >= 12) { return { cmd, version }; }
+```
+
+This accepts Python 3.13, 3.14, etc. even though the project constraint is `<3.13`.
+
+**Impact**: Low -- `uv sync` independently enforces `requires-python` from `pyproject.toml` and downloads a compatible Python (3.12.x) regardless of what `install.js` reports. The user sees misleading output but the .venv is built correctly.
+
+**Fix**: Update the check to match `pyproject.toml`:
+```js
+if (major === 3 && minor >= 11 && minor < 13) { return { cmd, version }; }
+```
diff --git a/scripts/clean.js b/scripts/clean.js
index d3c2cc09..b87947c8 100755
--- a/scripts/clean.js
+++ b/scripts/clean.js
@@ -21,10 +21,9 @@ const targets = [
'client/node_modules',
'client/dist',
'client/.vite',
- 'server/data',
+ 'server/data', // DBs (workflow.db, credentials.db) + workspaces
'server/.venv',
- 'server/credentials.db', // Encrypted credentials database
- '.venv', // Root venv (stale, should not exist)
+ '.venv', // Root venv (stale, should not exist)
];
async function main() {
diff --git a/scripts/start.js b/scripts/start.js
index c95abd3a..8634d29a 100755
--- a/scripts/start.js
+++ b/scripts/start.js
@@ -44,6 +44,36 @@ async function main() {
process.exit(1);
}
+ // Preflight: time-boxed sqlalchemy import probe.
+ // On Windows, Defender's minifilter driver (MpFilter.sys) can cache stale
+ // "pending scan" entries that block .pyd LoadLibrary calls even after
+ // adding exclusions. See docs-internal/errors.md #1 and #1a.
+ const pyExe = resolve(ROOT, 'server', '.venv', 'Scripts', isWindows ? 'python.exe' : 'python');
+ const pyFallback = resolve(ROOT, 'server', '.venv', 'bin', 'python');
+ const py = existsSync(pyExe) ? pyExe : pyFallback;
+ if (existsSync(py)) {
+ const probeStart = Date.now();
+ try {
+ execSync(`"${py}" -c "import sqlalchemy"`, { timeout: 15000, stdio: 'pipe' });
+ const elapsed = Date.now() - probeStart;
+ if (elapsed > 5000) {
+ console.warn(`Warning: sqlalchemy import took ${(elapsed / 1000).toFixed(1)}s (expected <1s).`);
+ console.warn(' See docs-internal/errors.md #1 for Windows Defender remediation.');
+ }
+ } catch (e) {
+ const elapsed = Date.now() - probeStart;
+ console.error(`Error: Python venv health check failed (${(elapsed / 1000).toFixed(1)}s).`);
+ console.error(' sqlalchemy import hung or crashed.');
+ console.error(' Likely cause: Windows Defender scan cache or stale kernel state.');
+ console.error(' Fix options:');
+ console.error(' 1. Restart-Service WinDefend (admin PowerShell)');
+ console.error(' 2. Reboot the machine');
+ console.error(' 3. Add D:\\...\\server\\.venv to Defender exclusions');
+ console.error(' See docs-internal/errors.md #1 and #1a for details.');
+ process.exit(1);
+ }
+ }
+
const config = loadEnvConfig();
process.env.PYTHONUTF8 = '1';
ensureEnvFile();
diff --git a/server/core/config.py b/server/core/config.py
index 2928fae1..3be10076 100644
--- a/server/core/config.py
+++ b/server/core/config.py
@@ -86,7 +86,11 @@ class Settings(BaseSettings):
api_key_encryption_key: str = Field(env="API_KEY_ENCRYPTION_KEY", min_length=32)
api_key_cache_ttl: int = Field(default=2592000, env="API_KEY_CACHE_TTL", ge=3600)
+ # Data directory (base for all persistent storage: DBs, workspaces, logs)
+ data_dir: str = Field(default="data", env="DATA_DIR")
+
# Credentials Database (separate encrypted database for API keys and OAuth tokens)
+ # Resolved relative to data_dir unless absolute
credentials_db_path: str = Field(default="credentials.db", env="CREDENTIALS_DB_PATH")
# Credential Backend Selection
@@ -126,7 +130,8 @@ class Settings(BaseSettings):
ws_logging_enabled: bool = Field(default=True, env="WS_LOGGING_ENABLED")
# Workspace Configuration (per-workflow file storage for nodes and agents)
- workspace_base_dir: str = Field(default="data/workspaces", env="WORKSPACE_BASE_DIR")
+ # Workspace base -- relative to data_dir unless absolute
+ workspace_base_dir: str = Field(default="workspaces", env="WORKSPACE_BASE_DIR")
# Compaction Configuration
compaction_enabled: bool = Field(default=True, env="COMPACTION_ENABLED")
@@ -160,6 +165,23 @@ class Settings(BaseSettings):
"""Check if running in production mode."""
return not self.debug
+ def _resolve_under_data(self, path: str) -> str:
+ """Resolve a path relative to data_dir, unless already absolute."""
+ p = Path(path)
+ if p.is_absolute():
+ return str(p)
+ return str(Path(self.data_dir) / p)
+
+ @property
+ def credentials_db_resolved(self) -> str:
+ """Full credentials DB path, rooted under data_dir."""
+ return self._resolve_under_data(self.credentials_db_path)
+
+ @property
+ def workspace_base_resolved(self) -> str:
+ """Full workspace base path, rooted under data_dir."""
+ return self._resolve_under_data(self.workspace_base_dir)
+
model_config = {
"env_file": "../.env",
"env_file_encoding": "utf-8",
diff --git a/server/core/container.py b/server/core/container.py
index f647b69f..a957df95 100644
--- a/server/core/container.py
+++ b/server/core/container.py
@@ -70,7 +70,7 @@ class Container(containers.DeclarativeContainer):
# Credentials database (separate encrypted database for API keys and OAuth tokens)
credentials_database = providers.Singleton(
CredentialsDatabase,
- db_path=settings.provided.credentials_db_path,
+ db_path=settings.provided.credentials_db_resolved,
encryption=encryption_service
)
diff --git a/server/services/claude_code_service.py b/server/services/claude_code_service.py
index a142a102..5d17bc24 100644
--- a/server/services/claude_code_service.py
+++ b/server/services/claude_code_service.py
@@ -60,7 +60,7 @@ class ClaudeCodeService:
if not cwd:
from core.config import Settings
- cwd = os.path.join(Settings().workspace_base_dir, 'default')
+ cwd = os.path.join(Settings().workspace_base_resolved, 'default')
os.makedirs(cwd, exist_ok=True)
work_dir = cwd
diff --git a/server/services/handlers/filesystem.py b/server/services/handlers/filesystem.py
index ed4bbbef..0ead9530 100644
--- a/server/services/handlers/filesystem.py
+++ b/server/services/handlers/filesystem.py
@@ -24,7 +24,7 @@ def _get_backend(parameters: Dict[str, Any], context: Dict[str, Any] = None):
from core.config import Settings
param_dir = parameters.get('working_directory')
ctx_dir = context.get('workspace_dir') if context else None
- root = param_dir or ctx_dir or os.path.join(Settings().workspace_base_dir, 'default')
+ root = param_dir or ctx_dir or os.path.join(Settings().workspace_base_resolved, 'default')
os.makedirs(root, exist_ok=True)
logger.info("[Filesystem] root=%s", root)
return LocalShellBackend(root_dir=root, virtual_mode=True)
diff --git a/server/services/process_service.py b/server/services/process_service.py
index 58f45c0c..97286792 100644
--- a/server/services/process_service.py
+++ b/server/services/process_service.py
@@ -104,7 +104,7 @@ class ProcessService:
argv = shlex.split(command)
env = {**os.environ, "PYTHONUNBUFFERED": "1"}
from core.config import Settings
- workspace_base = Path(Settings().workspace_base_dir).resolve()
+ workspace_base = Path(Settings().workspace_base_resolved).resolve()
if not working_directory:
working_directory = str(workspace_base / 'default')
diff --git a/server/services/workflow.py b/server/services/workflow.py
index e2548931..00f3f5f6 100644
--- a/server/services/workflow.py
+++ b/server/services/workflow.py
@@ -135,7 +135,7 @@ class WorkflowService:
def _get_workspace_dir(self, workflow_id: Optional[str]) -> str:
"""Get or create workspace directory for a workflow."""
- base = Path(self.settings.workspace_base_dir)
+ base = Path(self.settings.workspace_base_resolved)
wf_id = workflow_id or "default"
workspace = base / wf_id
workspace.mkdir(parents=True, exist_ok=True)