Self-improving AI Employees for everyone
Python ai-agentsai-employeeai-employeesassistantautonomous-agentsco-employeescompanygrok-botinstinctloop-agentsmuseown-your-datapersonalself-hosted
fix(agent): JSON-driven agent defaults + nushell shell + recursion handling
trohitgc281e4b3db44
diff --git a/CLAUDE.md b/CLAUDE.mdindex 4a647e6f..9ae54641 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -224,7 +224,7 @@ server/models/└── database.py # ConversationMessage, NodeParameter, ToolSchema, ChatMessage, TokenUsageMetric, APIUsageMetric, CompactionEvent, SessionTokenState, UserSettings, ProviderDefaults, AgentTeam, TeamMember, TeamTask, AgentMessage tablesserver/config/-├── llm_defaults.json # Default models per provider (edit to change defaults)+├── llm_defaults.json # Per-provider defaults (model, base_url, max_output_tokens, context_length, temperature_range, reasoning_models, thinking_type, ...) AND a top-level `agent` block (recursion_limit, default_temperature, compaction.ratio) that drives the LangGraph agent loop and CompactionService — no env-var defaults; this is the source of truth.├── model_registry.json # Cached model data from OpenRouter (auto-refreshed)├── pricing.json # LLM and API pricing config├── google_apis.json # Google Workspace API endpoints, scopes, OAuth callback paths@@ -1457,19 +1457,23 @@ Residential proxy provider management with geo-targeting, session control, and a- **typescriptExecutor**: **Dual-purpose node** - Execute TypeScript code via persistent Node.js server with type safety, syntax-highlighted editor and console output. Works as workflow node OR AI Agent tool (`typescript_code`).### Filesystem & Shell Nodes (4 nodes)-Dual-purpose tool nodes delegating to `deepagents.backends.LocalShellBackend`. Uses per-workflow workspace from execution context (`context["workspace_dir"]` = `data/workspaces/<workflow_id>/`). `virtual_mode=True` confines paths within the workspace. Fallback uses `Settings().workspace_base_dir` (never `os.getcwd()`).+Dual-purpose tool nodes wrapping `deepagents.backends.LocalShellBackend`. Per-workflow workspace from execution context (`context["workspace_dir"]` = `data/workspaces/<workflow_id>/`). Fallback uses `Settings().workspace_base_dir` (never `os.getcwd()`).++**Path safety**: `nodes/filesystem/_backend.py` exposes `normalize_virtual_path()` which uses `pathlib.PureWindowsPath` (host-OS independent) to strip Windows drives, POSIX root, and UNC anchors uniformly, then delegates to `deepagents.backends.utils.validate_path` for `..`/`~` rejection. Wired into `file_read`, `file_modify`, and `fs_search` so LLM-emitted paths in any flavour (`C:\foo`, `/tmp/foo`, `\\server\share\x`, `foo\bar`) all map to a virtual path under the workspace. `virtual_mode=True` only sandboxes filesystem ops — `execute()` itself is never path-restricted (deepagents documents this).++**Shell backend is Nushell** (`NushellBackend` subclassing `LocalShellBackend`). Cross-platform parity — same grammar on Windows/macOS/Linux. `inherit_env=True` so `npm`, `node`, `python`, `git`, etc. are reachable on PATH. Nu stdlib is loaded (no `--no-std-lib`). Falls back to upstream `LocalShellBackend.execute()` (POSIX `sh` / cmd.exe) when `nu` isn't on PATH. **Bash idioms (`&&`, `||`, `$VAR`, backticks, `>`) do not work** — see `server/skills/terminal/shell-skill/SKILL.md` for the Nu equivalents.- **fileRead**: **Dual-purpose node** - Read file contents with line numbers and pagination. Works as workflow node OR AI Agent tool (`file_read`). Parameters: file_path, offset, limit.-- **fileModify**: **Dual-purpose node** - Write new files or edit existing files with string replacement. Works as workflow node OR AI Agent tool (`file_modify`). Operations: write (create/overwrite), edit (find and replace with old_string/new_string/replace_all).-- **shell**: **Dual-purpose node** - Execute shell commands with timeout. Works as workflow node OR AI Agent tool (`shell_execute`). **Sandboxed: no system PATH** -- use process_manager for npm/python/node commands. All backend calls wrapped in `asyncio.to_thread()` to avoid blocking the event loop. Returns stdout, exit_code, truncated flag.+- **fileModify**: **Dual-purpose node** - Write new files or edit existing files with string replacement. Works as workflow node OR AI Agent tool (`file_modify`). Operations: write (wholesale create-or-replace, no overwrite flag), edit (find and replace with old_string/new_string/replace_all).+- **shell**: **Dual-purpose node** - Execute Nushell commands with timeout. Works as workflow node OR AI Agent tool (`shell_execute`). PATH inherited; external tools (npm, node, python, git, ...) work. All backend calls wrapped in `asyncio.to_thread()` to avoid blocking the event loop. Returns stdout, exit_code, truncated flag.- **fsSearch**: **Dual-purpose node** - Search the filesystem with three modes. Works as workflow node OR AI Agent tool (`fs_search`). Modes: ls (list directory), glob (pattern match), grep (search file contents).**Key Files:**| File | Description ||------|-------------|-| `server/services/handlers/filesystem.py` | Handlers delegating to `deepagents.backends.LocalShellBackend` |+| `server/nodes/filesystem/_backend.py` | `NushellBackend`, `get_backend()`, `normalize_virtual_path()` || `server/skills/coding_agent/` | Skills: file-read-skill, file-modify-skill, fs-search-skill |-| `server/skills/terminal/` | Skills: shell-skill, process-manager-skill, bash-skill, powershell-skill, wsl-skill |+| `server/skills/terminal/` | Skills: shell-skill (Nushell), process-manager-skill, bash-skill, powershell-skill, wsl-skill |### Process Manager Node (1 node)Cross-platform process manager for long-running subprocesses (dev servers, watchers, build tools). Uses `asyncio.create_subprocess_exec` with full system PATH (`env={**os.environ}`). Output streams to Terminal tab via `broadcast_terminal_log()` and persists to log files in the workspace.@@ -2292,6 +2296,13 @@ Adding a new node type's output shape: define one Pydantic model, register it in## AI Agent Node Architecture+### LangGraph Loop Termination+The standard agent path (`build_agent_graph` in `services/ai.py`) routes purely on `pending_tool_calls` — no custom iteration counter. Termination is bounded by LangGraph's built-in `recursion_limit`, sourced from `agent.recursion_limit` in `llm_defaults.json` (currently 500 — generous backstop, not the load-bearing termination signal).++The token-based compaction threshold (`agent.compaction.ratio` × context_length, ≈100K for claude-sonnet-4-6) is the real termination signal: `_track_token_usage` runs after each agent turn, and when cumulative tokens cross the threshold it invokes `CompactionService.compact_context` to summarise the transcript before the next turn.++If the recursion cap is ever hit anyway, the `GraphRecursionError` handler appends a terminal `AIMessage` carrying a truncation note so `_extract_text_content` returns a usable partial response and the workflow continues — `_track_token_usage` and any post-loop persistence still run.+### Config-Driven Component DesignThe `AIAgentNode.tsx` component uses a configuration-driven pattern to support multiple agent types from a single component.@@ -3673,13 +3684,24 @@ pricing_info = pricing.get_pricing("anthropic", "claude-3-5-sonnet-20241022")### Configuration-Environment variables in `.env`:+The compaction threshold is fully JSON-driven via `server/config/llm_defaults.json`:++```json+"agent": {+ "recursion_limit": 500,+ "default_temperature": 0.7,+ "compaction": { "ratio": 0.5 }+}+```++Effective threshold = `providers.<provider>.context_length.<model>` × `agent.compaction.ratio` (e.g. claude-sonnet-4-6 → 200K × 0.5 = 100K input tokens). When model/provider are unknown, `model_registry.get_context_length` falls through to the provider's `_default` entry in the same JSON — never a Python constant.++Only the global on/off toggle stays in `.env`:```bashCOMPACTION_ENABLED=true # Enable/disable compaction globally-COMPACTION_THRESHOLD=100000 # Global default threshold (min: 10000, used when model context unknown)```-**Threshold priority:** per-session `custom_threshold` > model-aware (50% of context window) > `COMPACTION_THRESHOLD`+**Threshold priority:** per-session `custom_threshold` > model-aware (`context_length × ratio`).### Client-Side Compactiondiff --git a/server/config/llm_defaults.json b/server/config/llm_defaults.jsonindex 76a76dea..d174fff9 100644--- a/server/config/llm_defaults.json+++ b/server/config/llm_defaults.json@@ -4,8 +4,8 @@"_description": "LLM provider defaults. Connection config (base_url, models_endpoint) drives both native SDK and LangChain paths. Model specs auto-enriched by model_registry.json (OpenRouter cache).","agent": {- "_description": "Agent execution defaults shared across the LangGraph + compaction layers.",- "recursion_limit": 50,+ "_description": "Agent execution defaults shared across the LangGraph + compaction layers. recursion_limit is intentionally generous so the token-based compaction.ratio is the load-bearing termination signal, not the step cap.",+ "recursion_limit": 500,"default_temperature": 0.7,"compaction": {"ratio": 0.5diff --git a/server/core/config.py b/server/core/config.pyindex 97699460..f5b25389 100644--- a/server/core/config.py+++ b/server/core/config.py@@ -143,9 +143,10 @@ class Settings(BaseSettings):# silently allows. Override to "0.0.0.0" for container deployments.whatsapp_bind_host: str = Field(default="localhost", env="WHATSAPP_BIND_HOST")- # Compaction Configuration+ # Compaction Configuration. The threshold is fully model-aware and+ # sourced from server/config/llm_defaults.json (context_length ×+ # agent.compaction.ratio). Only the global on/off toggle remains here.compaction_enabled: bool = Field(default=True, env="COMPACTION_ENABLED")- compaction_threshold: int = Field(default=100000, env="COMPACTION_THRESHOLD", ge=10000)# Gunicorn Configuration (for production deployment)gunicorn_timeout: int = Field(default=120, env="GUNICORN_TIMEOUT", ge=30)diff --git a/server/nodes/filesystem/_backend.py b/server/nodes/filesystem/_backend.pyindex e8c0dcdb..24aacc04 100644--- a/server/nodes/filesystem/_backend.py+++ b/server/nodes/filesystem/_backend.py@@ -102,11 +102,11 @@ class NushellBackend(LocalShellBackend):# Flags per nushell/src/command.rs (long forms):# -n / --no-config-file Skip user/global config (deterministic).- # --no-std-lib Don't load the std library (faster, fewer- # surprises in a one-shot invocation).# --no-history Don't read or write shell history.# -c / --commands Run the next argument as a single command.- argv = [nu, "-n", "--no-history", "--no-std-lib", "-c", command]+ # The std library is intentionally left loaded — agents need+ # ``path``, ``str``, ``http``, etc. without per-command imports.+ argv = [nu, "-n", "--no-history", "-c", command]try:result = subprocess.run(@@ -181,4 +181,9 @@ def get_backend()os.makedirs(root, exist_ok=True)get_logger(__name__).info("[Filesystem] root=%s", root)- return NushellBackend(root_dir=root, virtual_mode=True)+ # ``inherit_env=True`` makes the host PATH available so the agent can+ # invoke npm, node, python, git, pwd (POSIX), where (Win), etc. as+ # external commands. ``virtual_mode`` only sandboxes filesystem ops+ # per deepagents docs — shell ``execute()`` was never path-restricted —+ # so inheriting env doesn't loosen any existing security boundary.+ return NushellBackend(root_dir=root, virtual_mode=True, inherit_env=True)diff --git a/server/services/ai.py b/server/services/ai.pyindex 3e1c934f..c1cc486f 100644--- a/server/services/ai.py+++ b/server/services/ai.py@@ -1920,8 +1920,10 @@ class AIService:# Execute the graph. ``recursion_limit`` (cap on agent/tool# supersteps) is sourced from llm_defaults.json so deployments- # tune it without code changes. On hit, surface what the agent- # produced before stopping rather than the raw library error.+ # tune it without code changes. On hit, synthesise a terminal+ # AIMessage from whatever the agent produced before stopping+ # so downstream extraction returns a usable partial response+ # instead of failing the workflow.from services.model_registry import get_model_registryrecursion_limit = int(get_model_registry().get_agent_defaults()["recursion_limit"])final_state = None@@ -1933,13 +1935,19 @@ class AIService:):final_state = snapshotexcept GraphRecursionError:- msgs = (final_state or {}).get("messages") or []- raise ValueError(- f"Agent reached the {recursion_limit}-superstep recursion "- f"limit without producing a final answer "- f"({len(msgs)} message(s) exchanged). "+ # Append a terminal AIMessage so downstream extraction (and+ # the post-loop _track_token_usage / compact_context call)+ # still has a usable state instead of a half-built one.+ msgs = list((final_state or {}).get("messages") or [])+ msgs.append(AIMessage(content=(+ f"[Recursion limit reached: {recursion_limit} supersteps. "f"Adjust agent.recursion_limit in llm_defaults.json or "- f"simplify the task."+ f"simplify the task.]"+ )))+ final_state = {**(final_state or {}), "messages": msgs}+ logger.warning(+ f"[LangGraph] Recursion limit hit ({recursion_limit}); "+ f"returning partial response with {len(msgs)} message(s)")# Extract the AI response (last message in the accumulated messages)@@ -2391,13 +2399,16 @@ class AIService:):final_state = snapshotexcept GraphRecursionError:- msgs = (final_state or {}).get("messages") or []- raise ValueError(- f"Agent reached the {recursion_limit}-superstep "- f"recursion limit without producing a final answer "- f"({len(msgs)} message(s) exchanged). "- f"Adjust agent.recursion_limit in llm_defaults.json "- f"or simplify the task."+ msgs = list((final_state or {}).get("messages") or [])+ msgs.append(AIMessage(content=(+ f"[Recursion limit reached: {recursion_limit} supersteps. "+ f"Adjust agent.recursion_limit in llm_defaults.json or "+ f"simplify the task.]"+ )))+ final_state = {**(final_state or {}), "messages": msgs}+ logger.warning(+ f"[LangGraph] Recursion limit hit ({recursion_limit}); "+ f"returning partial response with {len(msgs)} message(s)")# Extract responsediff --git a/server/services/compaction.py b/server/services/compaction.pyindex 30f293f4..c502e474 100644--- a/server/services/compaction.py+++ b/server/services/compaction.py@@ -58,7 +58,6 @@ def _extract_text_from_response(content) -> str:class CompactionConfig(BaseModel):"""Provider-agnostic compaction configuration."""enabled: bool = True- threshold: int = 100000class CompactionService:@@ -66,10 +65,7 @@ class CompactionService:def __init__(self, database: "Database", settings: "Settings"):self._db = database- self._config = CompactionConfig(- enabled=settings.compaction_enabled,- threshold=settings.compaction_threshold- )+ self._config = CompactionConfig(enabled=settings.compaction_enabled)self._ai_service = Nonedef set_ai_service(self, ai_service) -> None:@@ -226,11 +222,9 @@ class CompactionService:custom = state.get("custom_threshold")if custom:threshold = custom- elif model and provider:+ else:ratio = await self._get_compaction_ratio()threshold = self.get_model_threshold(model, provider, ratio=ratio)- else:- threshold = self._config.threshold# Get context window size for frontend displaycontext_length = 0if model and provider:diff --git a/server/skills/terminal/shell-skill/SKILL.md b/server/skills/terminal/shell-skill/SKILL.mdindex affd14cf..759d9e91 100644--- a/server/skills/terminal/shell-skill/SKILL.md+++ b/server/skills/terminal/shell-skill/SKILL.md@@ -1,17 +1,19 @@---name: shell-skill-description: Execute short-lived shell commands in a sandboxed environment. No PATH access -- use process_manager for npm/python/node commands.+description: Execute short-lived shell commands inside the per-workflow workspace. The shell is Nushell (cross-platform, same syntax on Windows/macOS/Linux). External tools on PATH (npm, node, python, git, ...) are available.allowed-tools: "shell"metadata:author: machina- version: "3.0"+ version: "4.0"category: execution----# Shell Tool+# Shell Tool (Nushell)-Execute short-lived shell commands in a sandboxed workspace. The shell runs with a **restricted environment** (no system PATH). For commands that need `npm`, `python`, `node`, or other system tools, use the **process_manager** tool instead.+Execute short-lived shell commands in the workflow workspace. **The shell is [Nushell](https://www.nushell.sh/) — the same grammar runs on Windows, macOS, and Linux.** Do not write `cmd.exe`, PowerShell, or Bash idioms; they will fail or behave wrong.++External binaries on `PATH` (`npm`, `node`, `python`, `git`, `pwd`, etc.) are available — Nu invokes them as external commands automatically.## shell_execute Tool@@ -19,8 +21,8 @@ Execute short-lived shell commands in a sandboxed workspace. The shell runs with| Field | Type | Required | Description ||-------|------|----------|-------------|-| command | string | Yes | Shell command to execute |-| timeout | int | No | Timeout in seconds (default: 30, max: 300) |+| command | string | Yes | Nushell command (single or `;`-chained) |+| timeout | int | No | Seconds (default 30, max 300) |### Response@@ -29,81 +31,88 @@ Execute short-lived shell commands in a sandboxed workspace. The shell runs with"stdout": "command output","exit_code": 0,"truncated": false,- "command": "dir"+ "command": "ls"}```-### Exit Codes--| Code | Meaning |-|------|---------|+| Exit code | Meaning |+|---|---|| 0 | Success || 124 | Timed out || non-zero | Failure |-## OS-Specific Commands--The shell uses the system's native shell (`cmd.exe` on Windows, `/bin/sh` on Linux/macOS). Use the correct commands for the platform.--### Detect OS first--```json-{"command": "ver"}+## Critical: Nushell ≠ Bash++| Bash / cmd.exe (do NOT use) | Nushell (correct) |+|---|---|+| `cmd1 && cmd2` (and-then) | `cmd1; cmd2` *(unconditional sequential — see below for short-circuit)* |+| `cmd1 || cmd2` (or-else) | `try { cmd1 } catch { cmd2 }` |+| `$VAR` substitution | `$env.VAR` |+| `` `cmd` `` or `$(cmd)` | `(cmd)` *(parens, no dollar)* |+| `cmd > file.txt` | `cmd \| save file.txt` |+| `cmd >> file.txt` | `cmd \| save --append file.txt` |+| `cmd 2>&1` | `cmd \| complete \| get stdout` *(all output is captured anyway)* |+| `if [ -f x.txt ]; then ...` | `if ('x.txt' \| path exists) { ... }` |+| `for f in *.py; do ...` | `glob '*.py' \| each { \|f\| ... }` |+| `*` glob in argv (auto-expand) | wrap in quotes or use `glob` |+| `~/path` | `('~/path' \| path expand)` |++### Short-circuit "and-then" (the `&&` replacement)++The user log showed `pwd && ls -la` failing — the parser explicitly rejects `&&`. Use one of:++```nu+# A: just sequential, doesn't short-circuit on failure+pwd; ls -la++# B: short-circuit using exit code via try/catch+try { npm install } catch { print 'install failed'; exit 1 }+ls -la++# C: explicit conditional on the previous command's success+let r = (do { npm install } | complete)+if $r.exit_code == 0 { ls -la } else { print $r.stderr }```-If output contains `Windows`, use Windows commands. Otherwise use Unix commands.-### Windows (cmd.exe)+Use **A** for "run these in order regardless of outcome", **B/C** when you must stop on failure.-| Task | Command |-|------|---------|-| List files | `dir` |-| List with details | `dir /a` |-| Read file | `type README.md` |-| Write to file | `echo hello > output.txt` |-| Find files | `dir /s /b *.py` |-| Search content | `findstr /s /i "pattern" *.py` |-| Copy file | `copy src.txt dst.txt` |-| Move file | `move src.txt dst.txt` |-| Delete file | `del output.txt` |-| Delete folder | `rmdir /s /q folder` |-| Create folder | `mkdir newfolder` |-| Show current dir | `cd` |--### Linux / macOS (sh)+## Common tasks (cross-platform, Nushell)| Task | Command |-|------|---------|-| List files | `ls -la` |-| Read file | `cat README.md` |-| Write to file | `echo hello > output.txt` |-| Find files | `find . -name '*.py' -type f` |-| Search content | `grep -r "pattern" --include='*.py' .` |-| Copy file | `cp src.txt dst.txt` |-| Move file | `mv src.txt dst.txt` |-| Delete file | `rm output.txt` |-| Delete folder | `rm -rf folder` |-| Create folder | `mkdir -p newfolder` |-| Show current dir | `pwd` |--## Shell vs Process Manager+|---|---|+| Show current dir | `pwd` *(nu builtin)* |+| List files | `ls` *(returns a table — pipe further)* |+| List recursively | `ls **/*` |+| Read file | `open README.md` *(text/json/csv auto-parsed)* or `cat README.md` |+| Write to file | `'hello' \| save -f output.txt` |+| Append | `'more' \| save --append output.txt` |+| Find files by name | `glob '**/*.py'` |+| Search content | `rg 'pattern' .` *(if ripgrep on PATH)* or `open file.txt \| find 'pattern'` |+| Copy / move / delete | `cp a b`, `mv a b`, `rm a` |+| Make folder | `mkdir new` |+| Run npm / node / python | `npm install`, `node app.js`, `python -V` *(via PATH)* |+| Capture command output into a var | `let v = (npm -v \| str trim)` |+| Conditional on a binary existing | `if (which git \| is-empty) { print 'no git' }` |++## Workspace and paths++- The cwd is the per-workflow workspace; relative paths resolve there.+- Filesystem operations elsewhere on this tool (read/write/edit via `file_*`) honour `virtual_mode=True` and reject `..`/`~` traversal. Shell `execute()` itself is **not** path-restricted (deepagents documents this), so prefer `file_read` / `file_modify` / `fs_search` for actual filesystem work.++## Use the right tool| Need | Tool | Why |-|------|------|-----|-| List/read/copy/move files | **shell_execute** | Sandboxed, safe |-| Delete files | **shell_execute** | Confined to workspace |-| Search file content | **shell_execute** | Fast, no PATH needed |-| `npm install`, `pip install` | **process_manager** | Needs PATH |-| `python script.py`, `node app.js` | **process_manager** | Needs PATH |-| Dev servers, watchers | **process_manager** | Long-running |--The shell is the **only safe tool for file deletion**. It runs inside the agent's workspace with `virtual_mode=True` (path traversal blocked). The process_manager blocks destructive commands since it has full PATH.+|---|---|---|+| List / search / one-shot file ops | **shell_execute** | Fast, in-workspace |+| Reading or editing a specific file | **file_read** / **file_modify** | Path-sandboxed, no shell parsing surprises |+| Long-running processes (dev servers, watchers, `npm run dev`) | **process_manager** | Streams output, restartable, doesn't tie up the agent |+| Recursive code search | **fs_search** | grep mode, structured results |## Guidelines-1. **Detect OS first** -- use `ver` to check Windows vs Unix, then use correct commands-2. **Short-lived commands only** -- the shell waits for completion-3. **No system PATH** -- `npm`, `python`, `node` will not be found. Use process_manager-4. **No daemons/servers** -- commands that don't exit will hang until timeout-5. Use relative paths (workspace is the working directory)-6. Chain with `&&` for sequential execution-7. Avoid interactive commands requiring user input+1. **Never use `&&`, `||`, backticks, `$VAR`, or `>` redirection.** Use the Nushell equivalent on the right side of the table above.+2. **One command (or `;`-chain) per call.** No multi-line scripts; if you need control flow, use `if` / `try` / `each` inline.+3. **Short-lived only.** If the command runs longer than ~30s or is a daemon, switch to `process_manager`.+4. **Don't pre-detect the OS.** Nu syntax is identical on Windows, macOS, Linux — write one command, not platform branches.+5. **Quote glob patterns** (`'*.py'`) so Nu's `glob` builtin expands them, not the caller.+6. **Capture command exit code** with `do { … } | complete` if you need to branch on success/failure.