Skip to content

Self-improving AI Employees for everyone

fix(agent): JSON-driven agent defaults + nushell shell + recursion handling

trohitgc281e4b3db44
diff --git a/CLAUDE.md b/CLAUDE.md
index 4a647e6f..9ae54641 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -224,7 +224,7 @@ server/models/
└── database.py # ConversationMessage, NodeParameter, ToolSchema, ChatMessage, TokenUsageMetric, APIUsageMetric, CompactionEvent, SessionTokenState, UserSettings, ProviderDefaults, AgentTeam, TeamMember, TeamTask, AgentMessage tables
server/config/
-├── llm_defaults.json # Default models per provider (edit to change defaults)
+├── llm_defaults.json # Per-provider defaults (model, base_url, max_output_tokens, context_length, temperature_range, reasoning_models, thinking_type, ...) AND a top-level `agent` block (recursion_limit, default_temperature, compaction.ratio) that drives the LangGraph agent loop and CompactionService — no env-var defaults; this is the source of truth.
├── model_registry.json # Cached model data from OpenRouter (auto-refreshed)
├── pricing.json # LLM and API pricing config
├── google_apis.json # Google Workspace API endpoints, scopes, OAuth callback paths
@@ -1457,19 +1457,23 @@ Residential proxy provider management with geo-targeting, session control, and a
- **typescriptExecutor**: **Dual-purpose node** - Execute TypeScript code via persistent Node.js server with type safety, syntax-highlighted editor and console output. Works as workflow node OR AI Agent tool (`typescript_code`).
### Filesystem & Shell Nodes (4 nodes)
-Dual-purpose tool nodes delegating to `deepagents.backends.LocalShellBackend`. Uses per-workflow workspace from execution context (`context["workspace_dir"]` = `data/workspaces/<workflow_id>/`). `virtual_mode=True` confines paths within the workspace. Fallback uses `Settings().workspace_base_dir` (never `os.getcwd()`).
+Dual-purpose tool nodes wrapping `deepagents.backends.LocalShellBackend`. Per-workflow workspace from execution context (`context["workspace_dir"]` = `data/workspaces/<workflow_id>/`). Fallback uses `Settings().workspace_base_dir` (never `os.getcwd()`).
+
+**Path safety**: `nodes/filesystem/_backend.py` exposes `normalize_virtual_path()` which uses `pathlib.PureWindowsPath` (host-OS independent) to strip Windows drives, POSIX root, and UNC anchors uniformly, then delegates to `deepagents.backends.utils.validate_path` for `..`/`~` rejection. Wired into `file_read`, `file_modify`, and `fs_search` so LLM-emitted paths in any flavour (`C:\foo`, `/tmp/foo`, `\\server\share\x`, `foo\bar`) all map to a virtual path under the workspace. `virtual_mode=True` only sandboxes filesystem ops — `execute()` itself is never path-restricted (deepagents documents this).
+
+**Shell backend is Nushell** (`NushellBackend` subclassing `LocalShellBackend`). Cross-platform parity — same grammar on Windows/macOS/Linux. `inherit_env=True` so `npm`, `node`, `python`, `git`, etc. are reachable on PATH. Nu stdlib is loaded (no `--no-std-lib`). Falls back to upstream `LocalShellBackend.execute()` (POSIX `sh` / cmd.exe) when `nu` isn't on PATH. **Bash idioms (`&&`, `||`, `$VAR`, backticks, `>`) do not work** — see `server/skills/terminal/shell-skill/SKILL.md` for the Nu equivalents.
- **fileRead**: **Dual-purpose node** - Read file contents with line numbers and pagination. Works as workflow node OR AI Agent tool (`file_read`). Parameters: file_path, offset, limit.
-- **fileModify**: **Dual-purpose node** - Write new files or edit existing files with string replacement. Works as workflow node OR AI Agent tool (`file_modify`). Operations: write (create/overwrite), edit (find and replace with old_string/new_string/replace_all).
-- **shell**: **Dual-purpose node** - Execute shell commands with timeout. Works as workflow node OR AI Agent tool (`shell_execute`). **Sandboxed: no system PATH** -- use process_manager for npm/python/node commands. All backend calls wrapped in `asyncio.to_thread()` to avoid blocking the event loop. Returns stdout, exit_code, truncated flag.
+- **fileModify**: **Dual-purpose node** - Write new files or edit existing files with string replacement. Works as workflow node OR AI Agent tool (`file_modify`). Operations: write (wholesale create-or-replace, no overwrite flag), edit (find and replace with old_string/new_string/replace_all).
+- **shell**: **Dual-purpose node** - Execute Nushell commands with timeout. Works as workflow node OR AI Agent tool (`shell_execute`). PATH inherited; external tools (npm, node, python, git, ...) work. All backend calls wrapped in `asyncio.to_thread()` to avoid blocking the event loop. Returns stdout, exit_code, truncated flag.
- **fsSearch**: **Dual-purpose node** - Search the filesystem with three modes. Works as workflow node OR AI Agent tool (`fs_search`). Modes: ls (list directory), glob (pattern match), grep (search file contents).
**Key Files:**
| File | Description |
|------|-------------|
-| `server/services/handlers/filesystem.py` | Handlers delegating to `deepagents.backends.LocalShellBackend` |
+| `server/nodes/filesystem/_backend.py` | `NushellBackend`, `get_backend()`, `normalize_virtual_path()` |
| `server/skills/coding_agent/` | Skills: file-read-skill, file-modify-skill, fs-search-skill |
-| `server/skills/terminal/` | Skills: shell-skill, process-manager-skill, bash-skill, powershell-skill, wsl-skill |
+| `server/skills/terminal/` | Skills: shell-skill (Nushell), process-manager-skill, bash-skill, powershell-skill, wsl-skill |
### Process Manager Node (1 node)
Cross-platform process manager for long-running subprocesses (dev servers, watchers, build tools). Uses `asyncio.create_subprocess_exec` with full system PATH (`env={**os.environ}`). Output streams to Terminal tab via `broadcast_terminal_log()` and persists to log files in the workspace.
@@ -2292,6 +2296,13 @@ Adding a new node type's output shape: define one Pydantic model, register it in
## AI Agent Node Architecture
+### LangGraph Loop Termination
+The standard agent path (`build_agent_graph` in `services/ai.py`) routes purely on `pending_tool_calls` — no custom iteration counter. Termination is bounded by LangGraph's built-in `recursion_limit`, sourced from `agent.recursion_limit` in `llm_defaults.json` (currently 500 — generous backstop, not the load-bearing termination signal).
+
+The token-based compaction threshold (`agent.compaction.ratio` × context_length, ≈100K for claude-sonnet-4-6) is the real termination signal: `_track_token_usage` runs after each agent turn, and when cumulative tokens cross the threshold it invokes `CompactionService.compact_context` to summarise the transcript before the next turn.
+
+If the recursion cap is ever hit anyway, the `GraphRecursionError` handler appends a terminal `AIMessage` carrying a truncation note so `_extract_text_content` returns a usable partial response and the workflow continues — `_track_token_usage` and any post-loop persistence still run.
+
### Config-Driven Component Design
The `AIAgentNode.tsx` component uses a configuration-driven pattern to support multiple agent types from a single component.
@@ -3673,13 +3684,24 @@ pricing_info = pricing.get_pricing("anthropic", "claude-3-5-sonnet-20241022")
### Configuration
-Environment variables in `.env`:
+The compaction threshold is fully JSON-driven via `server/config/llm_defaults.json`:
+
+```json
+"agent": {
+ "recursion_limit": 500,
+ "default_temperature": 0.7,
+ "compaction": { "ratio": 0.5 }
+}
+```
+
+Effective threshold = `providers.<provider>.context_length.<model>` × `agent.compaction.ratio` (e.g. claude-sonnet-4-6 → 200K × 0.5 = 100K input tokens). When model/provider are unknown, `model_registry.get_context_length` falls through to the provider's `_default` entry in the same JSON — never a Python constant.
+
+Only the global on/off toggle stays in `.env`:
```bash
COMPACTION_ENABLED=true # Enable/disable compaction globally
-COMPACTION_THRESHOLD=100000 # Global default threshold (min: 10000, used when model context unknown)
```
-**Threshold priority:** per-session `custom_threshold` > model-aware (50% of context window) > `COMPACTION_THRESHOLD`
+**Threshold priority:** per-session `custom_threshold` > model-aware (`context_length × ratio`).
### Client-Side Compaction
diff --git a/server/config/llm_defaults.json b/server/config/llm_defaults.json
index 76a76dea..d174fff9 100644
--- a/server/config/llm_defaults.json
+++ b/server/config/llm_defaults.json
@@ -4,8 +4,8 @@
"_description": "LLM provider defaults. Connection config (base_url, models_endpoint) drives both native SDK and LangChain paths. Model specs auto-enriched by model_registry.json (OpenRouter cache).",
"agent": {
- "_description": "Agent execution defaults shared across the LangGraph + compaction layers.",
- "recursion_limit": 50,
+ "_description": "Agent execution defaults shared across the LangGraph + compaction layers. recursion_limit is intentionally generous so the token-based compaction.ratio is the load-bearing termination signal, not the step cap.",
+ "recursion_limit": 500,
"default_temperature": 0.7,
"compaction": {
"ratio": 0.5
diff --git a/server/core/config.py b/server/core/config.py
index 97699460..f5b25389 100644
--- a/server/core/config.py
+++ b/server/core/config.py
@@ -143,9 +143,10 @@ class Settings(BaseSettings):
# silently allows. Override to "0.0.0.0" for container deployments.
whatsapp_bind_host: str = Field(default="localhost", env="WHATSAPP_BIND_HOST")
- # Compaction Configuration
+ # Compaction Configuration. The threshold is fully model-aware and
+ # sourced from server/config/llm_defaults.json (context_length ×
+ # agent.compaction.ratio). Only the global on/off toggle remains here.
compaction_enabled: bool = Field(default=True, env="COMPACTION_ENABLED")
- compaction_threshold: int = Field(default=100000, env="COMPACTION_THRESHOLD", ge=10000)
# Gunicorn Configuration (for production deployment)
gunicorn_timeout: int = Field(default=120, env="GUNICORN_TIMEOUT", ge=30)
diff --git a/server/nodes/filesystem/_backend.py b/server/nodes/filesystem/_backend.py
index e8c0dcdb..24aacc04 100644
--- a/server/nodes/filesystem/_backend.py
+++ b/server/nodes/filesystem/_backend.py
@@ -102,11 +102,11 @@ class NushellBackend(LocalShellBackend):
# Flags per nushell/src/command.rs (long forms):
# -n / --no-config-file Skip user/global config (deterministic).
- # --no-std-lib Don't load the std library (faster, fewer
- # surprises in a one-shot invocation).
# --no-history Don't read or write shell history.
# -c / --commands Run the next argument as a single command.
- argv = [nu, "-n", "--no-history", "--no-std-lib", "-c", command]
+ # The std library is intentionally left loaded — agents need
+ # ``path``, ``str``, ``http``, etc. without per-command imports.
+ argv = [nu, "-n", "--no-history", "-c", command]
try:
result = subprocess.run(
@@ -181,4 +181,9 @@ def get_backend(
)
os.makedirs(root, exist_ok=True)
get_logger(__name__).info("[Filesystem] root=%s", root)
- return NushellBackend(root_dir=root, virtual_mode=True)
+ # ``inherit_env=True`` makes the host PATH available so the agent can
+ # invoke npm, node, python, git, pwd (POSIX), where (Win), etc. as
+ # external commands. ``virtual_mode`` only sandboxes filesystem ops
+ # per deepagents docs — shell ``execute()`` was never path-restricted —
+ # so inheriting env doesn't loosen any existing security boundary.
+ return NushellBackend(root_dir=root, virtual_mode=True, inherit_env=True)
diff --git a/server/services/ai.py b/server/services/ai.py
index 3e1c934f..c1cc486f 100644
--- a/server/services/ai.py
+++ b/server/services/ai.py
@@ -1920,8 +1920,10 @@ class AIService:
# Execute the graph. ``recursion_limit`` (cap on agent/tool
# supersteps) is sourced from llm_defaults.json so deployments
- # tune it without code changes. On hit, surface what the agent
- # produced before stopping rather than the raw library error.
+ # tune it without code changes. On hit, synthesise a terminal
+ # AIMessage from whatever the agent produced before stopping
+ # so downstream extraction returns a usable partial response
+ # instead of failing the workflow.
from services.model_registry import get_model_registry
recursion_limit = int(get_model_registry().get_agent_defaults()["recursion_limit"])
final_state = None
@@ -1933,13 +1935,19 @@ class AIService:
):
final_state = snapshot
except GraphRecursionError:
- msgs = (final_state or {}).get("messages") or []
- raise ValueError(
- f"Agent reached the {recursion_limit}-superstep recursion "
- f"limit without producing a final answer "
- f"({len(msgs)} message(s) exchanged). "
+ # Append a terminal AIMessage so downstream extraction (and
+ # the post-loop _track_token_usage / compact_context call)
+ # still has a usable state instead of a half-built one.
+ msgs = list((final_state or {}).get("messages") or [])
+ msgs.append(AIMessage(content=(
+ f"[Recursion limit reached: {recursion_limit} supersteps. "
f"Adjust agent.recursion_limit in llm_defaults.json or "
- f"simplify the task."
+ f"simplify the task.]"
+ )))
+ final_state = {**(final_state or {}), "messages": msgs}
+ logger.warning(
+ f"[LangGraph] Recursion limit hit ({recursion_limit}); "
+ f"returning partial response with {len(msgs)} message(s)"
)
# Extract the AI response (last message in the accumulated messages)
@@ -2391,13 +2399,16 @@ class AIService:
):
final_state = snapshot
except GraphRecursionError:
- msgs = (final_state or {}).get("messages") or []
- raise ValueError(
- f"Agent reached the {recursion_limit}-superstep "
- f"recursion limit without producing a final answer "
- f"({len(msgs)} message(s) exchanged). "
- f"Adjust agent.recursion_limit in llm_defaults.json "
- f"or simplify the task."
+ msgs = list((final_state or {}).get("messages") or [])
+ msgs.append(AIMessage(content=(
+ f"[Recursion limit reached: {recursion_limit} supersteps. "
+ f"Adjust agent.recursion_limit in llm_defaults.json or "
+ f"simplify the task.]"
+ )))
+ final_state = {**(final_state or {}), "messages": msgs}
+ logger.warning(
+ f"[LangGraph] Recursion limit hit ({recursion_limit}); "
+ f"returning partial response with {len(msgs)} message(s)"
)
# Extract response
diff --git a/server/services/compaction.py b/server/services/compaction.py
index 30f293f4..c502e474 100644
--- a/server/services/compaction.py
+++ b/server/services/compaction.py
@@ -58,7 +58,6 @@ def _extract_text_from_response(content) -> str:
class CompactionConfig(BaseModel):
"""Provider-agnostic compaction configuration."""
enabled: bool = True
- threshold: int = 100000
class CompactionService:
@@ -66,10 +65,7 @@ class CompactionService:
def __init__(self, database: "Database", settings: "Settings"):
self._db = database
- self._config = CompactionConfig(
- enabled=settings.compaction_enabled,
- threshold=settings.compaction_threshold
- )
+ self._config = CompactionConfig(enabled=settings.compaction_enabled)
self._ai_service = None
def set_ai_service(self, ai_service) -> None:
@@ -226,11 +222,9 @@ class CompactionService:
custom = state.get("custom_threshold")
if custom:
threshold = custom
- elif model and provider:
+ else:
ratio = await self._get_compaction_ratio()
threshold = self.get_model_threshold(model, provider, ratio=ratio)
- else:
- threshold = self._config.threshold
# Get context window size for frontend display
context_length = 0
if model and provider:
diff --git a/server/skills/terminal/shell-skill/SKILL.md b/server/skills/terminal/shell-skill/SKILL.md
index affd14cf..759d9e91 100644
--- a/server/skills/terminal/shell-skill/SKILL.md
+++ b/server/skills/terminal/shell-skill/SKILL.md
@@ -1,17 +1,19 @@
---
name: shell-skill
-description: Execute short-lived shell commands in a sandboxed environment. No PATH access -- use process_manager for npm/python/node commands.
+description: Execute short-lived shell commands inside the per-workflow workspace. The shell is Nushell (cross-platform, same syntax on Windows/macOS/Linux). External tools on PATH (npm, node, python, git, ...) are available.
allowed-tools: "shell"
metadata:
author: machina
- version: "3.0"
+ version: "4.0"
category: execution
---
-# Shell Tool
+# Shell Tool (Nushell)
-Execute short-lived shell commands in a sandboxed workspace. The shell runs with a **restricted environment** (no system PATH). For commands that need `npm`, `python`, `node`, or other system tools, use the **process_manager** tool instead.
+Execute short-lived shell commands in the workflow workspace. **The shell is [Nushell](https://www.nushell.sh/) — the same grammar runs on Windows, macOS, and Linux.** Do not write `cmd.exe`, PowerShell, or Bash idioms; they will fail or behave wrong.
+
+External binaries on `PATH` (`npm`, `node`, `python`, `git`, `pwd`, etc.) are available — Nu invokes them as external commands automatically.
## shell_execute Tool
@@ -19,8 +21,8 @@ Execute short-lived shell commands in a sandboxed workspace. The shell runs with
| Field | Type | Required | Description |
|-------|------|----------|-------------|
-| command | string | Yes | Shell command to execute |
-| timeout | int | No | Timeout in seconds (default: 30, max: 300) |
+| command | string | Yes | Nushell command (single or `;`-chained) |
+| timeout | int | No | Seconds (default 30, max 300) |
### Response
@@ -29,81 +31,88 @@ Execute short-lived shell commands in a sandboxed workspace. The shell runs with
"stdout": "command output",
"exit_code": 0,
"truncated": false,
- "command": "dir"
+ "command": "ls"
}
```
-### Exit Codes
-
-| Code | Meaning |
-|------|---------|
+| Exit code | Meaning |
+|---|---|
| 0 | Success |
| 124 | Timed out |
| non-zero | Failure |
-## OS-Specific Commands
-
-The shell uses the system's native shell (`cmd.exe` on Windows, `/bin/sh` on Linux/macOS). Use the correct commands for the platform.
-
-### Detect OS first
-
-```json
-{"command": "ver"}
+## Critical: Nushell ≠ Bash
+
+| Bash / cmd.exe (do NOT use) | Nushell (correct) |
+|---|---|
+| `cmd1 && cmd2` (and-then) | `cmd1; cmd2` *(unconditional sequential — see below for short-circuit)* |
+| `cmd1 || cmd2` (or-else) | `try { cmd1 } catch { cmd2 }` |
+| `$VAR` substitution | `$env.VAR` |
+| `` `cmd` `` or `$(cmd)` | `(cmd)` *(parens, no dollar)* |
+| `cmd > file.txt` | `cmd \| save file.txt` |
+| `cmd >> file.txt` | `cmd \| save --append file.txt` |
+| `cmd 2>&1` | `cmd \| complete \| get stdout` *(all output is captured anyway)* |
+| `if [ -f x.txt ]; then ...` | `if ('x.txt' \| path exists) { ... }` |
+| `for f in *.py; do ...` | `glob '*.py' \| each { \|f\| ... }` |
+| `*` glob in argv (auto-expand) | wrap in quotes or use `glob` |
+| `~/path` | `('~/path' \| path expand)` |
+
+### Short-circuit "and-then" (the `&&` replacement)
+
+The user log showed `pwd && ls -la` failing — the parser explicitly rejects `&&`. Use one of:
+
+```nu
+# A: just sequential, doesn't short-circuit on failure
+pwd; ls -la
+
+# B: short-circuit using exit code via try/catch
+try { npm install } catch { print 'install failed'; exit 1 }
+ls -la
+
+# C: explicit conditional on the previous command's success
+let r = (do { npm install } | complete)
+if $r.exit_code == 0 { ls -la } else { print $r.stderr }
```
-If output contains `Windows`, use Windows commands. Otherwise use Unix commands.
-### Windows (cmd.exe)
+Use **A** for "run these in order regardless of outcome", **B/C** when you must stop on failure.
-| Task | Command |
-|------|---------|
-| List files | `dir` |
-| List with details | `dir /a` |
-| Read file | `type README.md` |
-| Write to file | `echo hello > output.txt` |
-| Find files | `dir /s /b *.py` |
-| Search content | `findstr /s /i "pattern" *.py` |
-| Copy file | `copy src.txt dst.txt` |
-| Move file | `move src.txt dst.txt` |
-| Delete file | `del output.txt` |
-| Delete folder | `rmdir /s /q folder` |
-| Create folder | `mkdir newfolder` |
-| Show current dir | `cd` |
-
-### Linux / macOS (sh)
+## Common tasks (cross-platform, Nushell)
| Task | Command |
-|------|---------|
-| List files | `ls -la` |
-| Read file | `cat README.md` |
-| Write to file | `echo hello > output.txt` |
-| Find files | `find . -name '*.py' -type f` |
-| Search content | `grep -r "pattern" --include='*.py' .` |
-| Copy file | `cp src.txt dst.txt` |
-| Move file | `mv src.txt dst.txt` |
-| Delete file | `rm output.txt` |
-| Delete folder | `rm -rf folder` |
-| Create folder | `mkdir -p newfolder` |
-| Show current dir | `pwd` |
-
-## Shell vs Process Manager
+|---|---|
+| Show current dir | `pwd` *(nu builtin)* |
+| List files | `ls` *(returns a table — pipe further)* |
+| List recursively | `ls **/*` |
+| Read file | `open README.md` *(text/json/csv auto-parsed)* or `cat README.md` |
+| Write to file | `'hello' \| save -f output.txt` |
+| Append | `'more' \| save --append output.txt` |
+| Find files by name | `glob '**/*.py'` |
+| Search content | `rg 'pattern' .` *(if ripgrep on PATH)* or `open file.txt \| find 'pattern'` |
+| Copy / move / delete | `cp a b`, `mv a b`, `rm a` |
+| Make folder | `mkdir new` |
+| Run npm / node / python | `npm install`, `node app.js`, `python -V` *(via PATH)* |
+| Capture command output into a var | `let v = (npm -v \| str trim)` |
+| Conditional on a binary existing | `if (which git \| is-empty) { print 'no git' }` |
+
+## Workspace and paths
+
+- The cwd is the per-workflow workspace; relative paths resolve there.
+- Filesystem operations elsewhere on this tool (read/write/edit via `file_*`) honour `virtual_mode=True` and reject `..`/`~` traversal. Shell `execute()` itself is **not** path-restricted (deepagents documents this), so prefer `file_read` / `file_modify` / `fs_search` for actual filesystem work.
+
+## Use the right tool
| Need | Tool | Why |
-|------|------|-----|
-| List/read/copy/move files | **shell_execute** | Sandboxed, safe |
-| Delete files | **shell_execute** | Confined to workspace |
-| Search file content | **shell_execute** | Fast, no PATH needed |
-| `npm install`, `pip install` | **process_manager** | Needs PATH |
-| `python script.py`, `node app.js` | **process_manager** | Needs PATH |
-| Dev servers, watchers | **process_manager** | Long-running |
-
-The shell is the **only safe tool for file deletion**. It runs inside the agent's workspace with `virtual_mode=True` (path traversal blocked). The process_manager blocks destructive commands since it has full PATH.
+|---|---|---|
+| List / search / one-shot file ops | **shell_execute** | Fast, in-workspace |
+| Reading or editing a specific file | **file_read** / **file_modify** | Path-sandboxed, no shell parsing surprises |
+| Long-running processes (dev servers, watchers, `npm run dev`) | **process_manager** | Streams output, restartable, doesn't tie up the agent |
+| Recursive code search | **fs_search** | grep mode, structured results |
## Guidelines
-1. **Detect OS first** -- use `ver` to check Windows vs Unix, then use correct commands
-2. **Short-lived commands only** -- the shell waits for completion
-3. **No system PATH** -- `npm`, `python`, `node` will not be found. Use process_manager
-4. **No daemons/servers** -- commands that don't exit will hang until timeout
-5. Use relative paths (workspace is the working directory)
-6. Chain with `&&` for sequential execution
-7. Avoid interactive commands requiring user input
+1. **Never use `&&`, `||`, backticks, `$VAR`, or `>` redirection.** Use the Nushell equivalent on the right side of the table above.
+2. **One command (or `;`-chain) per call.** No multi-line scripts; if you need control flow, use `if` / `try` / `each` inline.
+3. **Short-lived only.** If the command runs longer than ~30s or is a daemon, switch to `process_manager`.
+4. **Don't pre-detect the OS.** Nu syntax is identical on Windows, macOS, Linux — write one command, not platform branches.
+5. **Quote glob patterns** (`'*.py'`) so Nu's `glob` builtin expands them, not the caller.
+6. **Capture command exit code** with `do { … } | complete` if you need to branch on success/failure.