Skip to content

ServerKit is a lightweight, modern server control panel for managing web applications, databases, and services on your VPS or dedicated server. Built with Python/Flask backend and React frontend.

test(ai): verify management UI and document completed integration

Juan Denis8468ab79f2da
diff --git a/docs/AI_MANAGEMENT_REVIEW.md b/docs/AI_MANAGEMENT_REVIEW.md
deleted file mode 100644
index b424918a..00000000
--- a/docs/AI_MANAGEMENT_REVIEW.md
+++ /dev/null
@@ -1,173 +0,0 @@
-# ServerKit AI management: source review and integration proposal
-
-Reviewed 2026-09-08. Primary references are the local Prompture checkout
-(`d36872b`) and AgentSite checkout (`db323d4`). CachiBot is a secondary reference
-for its model-slot UI. This is a source review, not a live provider benchmark.
-Neither reference repository was modified.
-
-## Main finding
-
-ServerKit should expose more of Prompture's existing capabilities through a
-small, coherent settings UI. It does not need another model-routing or token
-accounting engine. However, Prompture does not supply ServerKit's user/workspace
-authorization, saved-connection policy, or reporting views automatically.
-
-ServerKit pins and has installed Prompture **1.11.0**. Normalized source-text
-comparison found these local Prompture files identical to the installed package:
-`agents/conversation.py`, `agents/async_agent.py`, `pipeline/resolver.py`,
-`pipeline/routing.py`, and `infra/{budget,session,tracker,ledger,capabilities}.py`.
-The checkout's `VERSION` file says 1.10.0; it is not evidence that the installed
-runtime lacks these features. No SDK upgrade is needed for the capabilities
-reviewed below. AgentSite declares `prompture[all]>=1.9.1`, not an exact pin.
-
-## What Prompture already provides
-
-| Capability | Existing implementation | ServerKit integration |
-| --- | --- | --- |
-| Model roles and inheritance | `ModelResolver`, ordered configuration layers, utility-to-default fallback | Store explicit connection/model choices for Utility, Standard, and Advanced roles; resolve request override → task role → default. |
-| Routing | `ModelRouter`, cost-optimized, balanced, quality-first, fast strategies; routing explanations | Select only from administrator-approved connection/model candidates before a job starts. Strategies are heuristics, not a reliability or operational-risk classifier. |
-| Catalog and capabilities | Provider descriptors, discovery, model rates, context limits, tool/vision/structured-output flags | Enrich the existing model picker with verified metadata, filters, and input/output prices per 1M tokens. Keep manual gateway aliases. Unknown capability/pricing stays unknown. |
-| Generation controls | Conversation options, history cap, tool timeout/round/result limits; Agent iteration limits, guardrails, structured output | Add a validated Advanced section for supported controls. Distinguish lifetime token budget from per-response output-token limit. Do not expose arbitrary driver JSON or imply all models accept temperature/reasoning settings. |
-| Budget policies | `hard_stop`, `warn_and_continue`, `degrade`, token/cost ceilings, remaining budget | Use the existing hard-stop mechanism now. Add degradation only after connection-aware fallback is implemented and tested. |
-| Per-run observability | `UsageSession`, driver callbacks, tokens/cache tokens, costs, errors, latency, per-model summaries | Attach usage collection to each actual run; preserve ServerKit context and show the result in chat and reports. |
-| Persistent observability | `UsageTracker` events, filters, tags/metadata, daily/monthly budget records; `ModelUsageLedger` aggregates | Reuse event collection/calculation, with a ServerKit-owned data location and access-controlled reporting. Global SDK aggregates are not per-user accounting. |
-| Larger workflows | Agent/DeepAgent abstractions, guardrails, structured outputs, delegation and context hooks | Consider separate read-only diagnostic workflows later. Keep existing plugin tools, RBAC checks, and write confirmations in control of server operations. |
-
-Sources: Prompture [resolver](../../prompture/prompture/pipeline/resolver.py),
-[routing](../../prompture/prompture/pipeline/routing.py),
-[Conversation](../../prompture/prompture/agents/conversation.py),
-[AsyncAgent](../../prompture/prompture/agents/async_agent.py),
-[capabilities](../../prompture/prompture/infra/capabilities.py),
-[budget](../../prompture/prompture/infra/budget.py),
-[session](../../prompture/prompture/infra/session.py),
-[tracker](../../prompture/prompture/infra/tracker.py), and
-[ledger](../../prompture/prompture/infra/ledger.py).
-
-## What to borrow from AgentSite
-
-Its [model picker](../../AgentSite/frontend/src/components/shared/ModelSelect.jsx)
-and [model endpoint](../../AgentSite/agentsite/api/routes/models.py) demonstrate
-searchable provider groups, context/output sizes, tools, vision, reasoning, and
-pricing. The useful idea is showing enough information to choose a model,
-without requiring users to memorize IDs. Port the interaction into ServerKit's
-shared SCSS controls, not AgentSite's Tailwind markup.
-
-Its [configuration](../../AgentSite/agentsite/config.py) assigns cheaper strategies
-to triage and some specialist reviewers, and quality-first to the final judge.
-That is a better model for ServerKit than treating every task as equally costly.
-AgentSite also has [analytics endpoints](../../AgentSite/agentsite/api/routes/agents.py)
-and a [routing-cost view](../../AgentSite/frontend/src/components/analytics/CostByRoutingChart.jsx).
-Borrow the questions they answer: which model did the work, how much did it cost,
-how often did it fail, and which task/strategy caused the spend?
-
-There are important limits to copying its implementation:
-
-- `route_for()` selects the first syntactically valid model in a configured
- strategy pool. It does not attempt runtime failover. Searches found calls only
- in tests; the production orchestrator calls `resolve_agent_model()` instead.
- Strategy configuration alone is therefore not proof of routed generation in
- this checkout. See [resolver](../../AgentSite/agentsite/engine/model_resolver.py)
- and [orchestrator](../../AgentSite/agentsite/agents/orchestrator.py).
-- AgentSite's utility resolver takes the first budget fallback model. In
- ServerKit, an inexpensive task model and an outage fallback should be separate
- settings; they solve different problems.
-- Model-cost endpoints collapse ledger rows into a dictionary keyed only by model,
- although the ledger key is model plus API-key hash. Multiple credential rows
- can overwrite one another. The default ledger is also process-user-wide,
- while the configured tracker uses AgentSite's data directory. ServerKit should
- aggregate its own scoped run events instead of copying those endpoints.
-
-## Recommended settings structure
-
-Keep five sections within AI Assistant, using the shared card, field, switch,
-model-picker, and footer patterns in [SETTINGS_UI.md](SETTINGS_UI.md):
-
-1. **Connections:** credentials, endpoint, connection status, model discovery.
-2. **Task models:** three labeled roles, each showing its saved connection and
- model. Show inherited defaults explicitly; no empty dropdown with hidden behavior.
-3. **Limits and behavior:** spend/token limits and supported generation controls.
- Keep technical controls in an Advanced disclosure with sensible defaults.
-4. **Routing and fallback:** opt-in strategy, approved candidates, fallback order,
- and a preview explaining the selected model. Publish this only with working
- consumers and connection-bound driver creation.
-5. **Usage:** conversation/run totals and administrator reports by period, task
- role, model, connection, user, and workspace.
-
-| Task role | Intended work | Selection rule |
-| --- | --- | --- |
-| Utility / economical | Short summaries, classification, structured extraction | Prefer an explicitly configured inexpensive model with required capabilities; inherit Standard if unset. Do not add model calls for tasks already handled deterministically, such as current chat titles. |
-| Standard | Everyday questions and routine tool-assisted diagnosis | Default assistant connection/model. |
-| Advanced | Complex incident analysis and multi-step diagnosis | Explicit user selection or a defined task type; inherit Standard if unset. Higher capability never changes permissions. |
-
-Persist each choice as `{connection_id, model}`. The model portion is an opaque
-provider/gateway ID, including embedded slashes. Resolve before the run, record
-the decision and its reason, and preserve the selected connection for resumed
-chats. A selected strategy must not switch credentials midway through a tool loop.
-
-## Integration issues to account for
-
-**Budget enforcement was disabled in ServerKit.** `build_conversation()` supplied
-`max_cost` with `budget_policy=None`, which makes Prompture's check a no-op.
-This change enables `hard_stop` for a nonzero ceiling on both fresh and resumed
-conversations, validates saved limits, translates budget stops, and preserves
-usage even when a turn ends without an answer. Zero disables the ceiling. The
-default is $0.50, matching the settings endpoint. These checks use reported
-usage: the last call can overshoot and unpriced gateway aliases need a provider
-spending limit.
-
-**Routing options are not all enforcement mechanisms.** In the reviewed
-`routing.py`, `max_cost_per_call` is declared/documented but is not used by the
-selection logic. Preferred providers are ordering preferences, not an allowlist.
-Discovery uses the global environment. ServerKit must validate candidates against
-saved connections and enforce spending through an actual budget check.
-
-**Automatic fallback needs an adapter.** Conversation's `_switch_model()` calls
-`get_driver_for_model(new_model)` without forwarding the saved driver/environment.
-Do not enable it across ServerKit connections as-is. An explicit driver factory
-must retain connection identity, endpoint, and credentials. Failed write tools
-must not be silently replayed after fallback. ServerKit's legacy `fallback_models`
-setting is currently stored but not applied by its conversation factory.
-
-**CachiBot's router argument is not portable.** Its agent construction can pass
-`model_router`; the reviewed Prompture `AsyncAgent` constructor accepts no such
-argument. Use the public routing API as a separate selection step, rather than
-copying that constructor call.
-
-**Usage needs run attribution.** ServerKit persists Conversation usage snapshots,
-which are cumulative; its chat context currently drops that usage when loading
-messages and handling completion. Summing those snapshots double-counts spend.
-Capture per-call/run usage with Prompture callbacks, or explicitly calculate
-turn deltas, while retaining the cumulative value for conversation limits.
-
-**SDK tracker scopes are not tenant isolation.** The reviewed tracker
-`check_budget(scope)` selects a budget record by scope but sums usage events over
-the chosen period without filtering events by that scope. Do not advertise it as
-a per-user/workspace quota unchanged. Scope accounting, permissions, and any
-concurrent spend reservations require ServerKit integration.
-
-## Delivery sequence and acceptance criteria
-
-1. **Current repair:** consistent settings surfaces and controls, model discovery
- picker, persistent enable switch, clear notification footer, enforced
- conversation ceiling. Offline tests cover both chat modes and budget resume.
-2. **Profiles and visible usage:** implement the three roles and their actual task
- consumers together. Add a migration-backed run record with unique run ID,
- conversation/user/workspace, requested role, actual connection/model, routing
- reason, tokens, cost source, duration, outcome, and fallback attempts. Reuse
- Prompture counters; keep accounting idempotent. Test inheritance, permissions,
- exact gateway IDs, unknown pricing, and no double counting after resume.
-3. **Routing and reports:** add approved model pools and a routing preview; show
- daily/monthly totals, model/task breakdowns, latency/failure rates, and remaining
- limits. Separate estimated cost from provider-reported cost. Missing prices
- must read “Unknown,” not “Free.” Test that disallowed connections cannot be
- selected and that report totals reconcile with recorded runs.
-4. **Fallback and richer workflows:** add credential-aware failover/degradation,
- bounded diagnostic workflows, and supported context/history controls after
- the accounting and permission tests pass. Avoid shared response caching for
- live server state; any future cache needs user/workspace/connection scoping
- and an explicit freshness policy.
-
-The task-role/routing/reporting work in steps 2–4 is a proposal, not implemented
-UI. This review is an internal FYI; no upstream issue or external message was sent.
-Useful upstream follow-ups are a connection-aware fallback factory, enforcement
-or clarification of `max_cost_per_call`, and scoped tracker budget aggregation.
diff --git a/docs/AI_PROVIDERS.md b/docs/AI_PROVIDERS.md
index f84faeae..adfd0df3 100644
--- a/docs/AI_PROVIDERS.md
+++ b/docs/AI_PROVIDERS.md
@@ -10,12 +10,14 @@ and refreshes the chat UI. Disabling blocks new messages in both chat endpoints
while retaining conversation history. Limits and protections have their own Save
action; saving a connection preserves unsaved changes to those settings.
-New chats offer a connection selector and an optional model override. Once a
-chat starts, its connection and model stay fixed. Changing the panel default
-does not reroute existing chats. You can rotate a connection's credentials, but
+New chats offer a task, model role, and optional connection/model override.
+The selected connection and model are saved with the conversation. Changing
+the panel default does not reroute existing chats. Configured fallback or budget
+degradation may switch the saved binding before a turn produces output.
+You can rotate a connection's credentials, but
changing a provider or endpoint used by existing chats requires a new connection.
-Delete those chats before deleting their connection, and select another default
-before deleting the current default.
+Delete those chats and remove task/routing/fallback references before deleting
+their connection, and select another default before deleting the current default.
## Native providers and gateways
@@ -60,18 +62,111 @@ it. Vertex Gemini requires an explicit API key; Vertex Claude requires a project
location, and access token. Ambient cloud credentials are not a connection option.
Existing assistant RBAC, redaction, and write-action confirmations remain active.
-Gateway routing aliases may have unknown pricing; enforce authoritative spending
-limits at the gateway. Automatic fallback to another connection is not enabled.
+The picker includes catalog prices per million input/output tokens, context and
+output limits, and tools, vision, reasoning, and structured-output capabilities.
+Capability filters show only confirmed support. Missing metadata stays unknown.
+Gateway aliases are not assigned an upstream model's price or capabilities.
+Discovering a saved connection caches its nonsecret catalog; changing its
+configuration clears that cache. Manual IDs remain supported.
-The per-conversation cost ceiling uses Prompture's `hard_stop` policy for new
-and resumed conversations (default $0.50; zero disables it). It checks reported
-cumulative usage before each model call, so a final call can overshoot the
-remaining amount. Budget stops return a clear message, and usage is persisted
-even if the turn stops before producing an answer. This is not an authoritative
-billing limit when provider pricing is unavailable.
+## Task models and execution
-See [AI management review](AI_MANAGEMENT_REVIEW.md) for the Prompture and
-AgentSite source review and the proposed task profiles, routing, and reporting.
+**Task models and behavior** assigns a saved connection and exact model ID to
+Utility, Standard, and Advanced. Standard inherits the panel default; Utility
+and Advanced inherit Standard when unset. The UI shows the inherited binding.
+Prompture's `ModelResolver` resolves these slots before execution.
+
+| Task | Default role | Runtime behavior |
+| --- | --- | --- |
+| Chat | Standard | Existing assistant tools and confirmations, or simple text mode |
+| Summarize | Utility | Summarizes supplied text and attached context without tools |
+| Extract structured facts | Utility | Prompture validates an object with `summary`, `facts`, and `warnings`; no tools |
+| Read-only diagnosis | Advanced | Only tools marked read-only and authorized for the caller; bounded tool rounds |
+
+An explicit connection/model override wins, followed by an explicit role.
+Without either, enabled automatic routing uses the approved pool; otherwise
+the task's role is used. Existing conversations retain their task and binding.
+Advanced models never grant additional permissions.
+
+Routing uses Prompture's cost-optimized, balanced, quality-first, or fast
+strategy, with discovery restricted to administrator-approved saved bindings.
+Candidate capability tiers guide its heuristics; “fast” is not a measured latency
+guarantee. The routing preview runs against the unsaved draft without making
+a model call. Each run records the selection strategy and explanation.
+
+Fallback is opt-in and tries the ordered list with each candidate's own endpoint
+and credentials. It only runs before any provider output has been delivered.
+The unused legacy `fallback_models` settings field is rejected; fallback
+candidates must include a saved connection ID in `/ai/management`.
+Once output or a tool response starts, a failure ends that turn; actions are
+never replayed on another connection. The adapter avoids Prompture's ambient
+credential fallback path. The actual selected model appears in run reporting.
+
+## Generation and spending controls
+
+Generation controls are applied to actual conversations: maximum output tokens
+per response, lifetime conversation tokens, history messages, tool rounds, and
+tool-result characters. Temperature is forwarded only with catalog-confirmed
+support; reasoning effort is forwarded only to supported OpenAI reasoning models.
+Unsupported settings retain provider defaults. Arbitrary driver JSON and detached
+tool timeouts are not exposed.
+
+The conversation cost ceiling defaults to $0.50; zero disables it. Select:
+
+- **Stop at the limit:** check cumulative cost/tokens before every provider call.
+- **Warn and continue:** retain usage and show the exceeded allowance in chat.
+- **Use a cheaper model near the limit, then stop:** at 80% of the cost allowance,
+ switch before a turn to the first configured fallback with lower known combined
+ input/output rates, then enforce the hard ceiling. Unknown prices are not compared.
+
+Monthly allowances can apply panel-wide, per user, and per workspace. Workspace
+attribution is captured from the active workspace when a chat starts and access
+is rechecked on resume. All-workspace chats still use panel/user allowances.
+Reservations prevent concurrent runs from all consuming the same remaining
+allowance in ServerKit's single worker. Each run stops further calls once its
+reservation is spent. Unknown-cost attempts conservatively consume that
+reservation. A restarted worker marks abandoned runs interrupted and retains
+their conservative charge.
+
+Limits use reported/estimated usage; a final call can overshoot. Unknown gateway
+pricing cannot enforce a billing ceiling. Use provider/gateway limits for
+authoritative spending enforcement.
+
+## Usage and plugin integration
+
+Each response shows its actual model, per-run tokens and cost source, duration,
+provider calls/errors, routing reason, and remaining conversation allowance.
+Reloading history retains those run details. **AI usage** provides administrator
+reports by period, role, model, connection, user, and workspace, with daily
+breakdowns, remaining monthly allowances, and a drawer for provider attempts.
+Unknown/partial pricing is labeled separately from known cost.
+
+Prompture `UsageSession` collects each actual driver call; its cost calculation
+and `BudgetState` checks are reused. ServerKit stores one migration-backed
+`AiRun` per turn, with idempotent completion. Cumulative Conversation snapshots
+remain for lifetime limits and are never summed as per-run costs. Run records
+survive transcript deletion. Reporting starts with this migration; older
+transcripts are not backfilled with invented per-call costs.
+
+The plugin SDK `ask()` and `ask_stream()` use the same enable state, role/routing
+policy, guardrails, limits, driver adapter, and ledger without creating chat
+transcripts. Both accept optional `profile` and `workflow` arguments:
+
+```python
+from app.plugins_sdk import ai
+
+summary = ai.ask(text, workflow='summarize')
+facts = ai.ask(text, workflow='extract', profile='utility')
+```
+
+The implementation was checked against local Prompture `d36872b` and AgentSite
+`db323d4`. AgentSite informed the model-picker/reporting interactions. Prompture
+1.11.0 already supplies the relevant resolver, router, structured output, budget,
+and usage machinery. ServerKit supplies credential binding, authorization,
+scope-filtered accounting, persistence, and UI. In this SDK version, router
+preferences are not an allowlist and tracker budget scopes do not filter usage
+by tenant; ServerKit explicitly enforces both boundaries. No upstream issue
+or external message was sent.
## Upgrade behavior
@@ -82,3 +177,7 @@ different provider cannot borrow the current connection's credentials: start a
new chat using an explicitly configured connection. Resumed chats reconstruct
their Prompture driver with their saved connection, and restore history and
usage without exporting credentials or using global provider configuration.
+
+Migration `099_ai_management` adds cached model metadata, conversation task
+snapshots, and the durable run ledger. Apply normal Alembic upgrades before
+starting the updated backend. No SDK upgrade or manual schema creation is needed.
diff --git a/docs/SETTINGS_UI.md b/docs/SETTINGS_UI.md
index 89f05e89..85485491 100644
--- a/docs/SETTINGS_UI.md
+++ b/docs/SETTINGS_UI.md
@@ -36,6 +36,15 @@ model discovery and exact IDs, secret preservation, assistant enable persistence
and failures, notification scope, and section action placement. CI runs it and
uploads screenshots from `frontend/test-results/`.
+AI management reuses `ModelBinding`/`ModelPicker`, `SegControl`, `FormField`,
+`Switch`, `DataTable`, `DataTableFooter`, and `Drawer`. Task-role forms, routing
+and fallback candidates, and usage reports share those controls. Keep connection
+credentials in their own editor; reference saved connections everywhere else.
+Inherited model roles must show their effective connection/model, and monetary
+figures must show their currency. Browser coverage includes saving/reloading
+roles, previewing draft routing, model metadata, usage details, chat task payloads,
+per-response allowance display, and narrow-screen overflow.
+
Provider discovery tests do not contact external providers. Live connectivity
must be verified separately with a configured connection. Model capabilities and
pricing should only be shown when supplied by a trustworthy provider catalog;
diff --git a/frontend/scripts/settings-ui-browser.mjs b/frontend/scripts/settings-ui-browser.mjs
index 44445788..f604df0d 100644
--- a/frontend/scripts/settings-ui-browser.mjs
+++ b/frontend/scripts/settings-ui-browser.mjs
@@ -22,6 +22,17 @@ try {
let rejectToggle = false;
let savedConnection;
let savedPreferences;
+ let sentChat;
+ let management = {
+ profiles: { utility: null, standard: null, advanced: null }, routing_enabled: false, strategy: 'balanced', pool: [],
+ fallback_enabled: false, fallbacks: [], budget_policy: 'hard_stop', max_tokens: 0, max_output_tokens: 2048,
+ max_tool_rounds: 6, max_tool_result_length: 4000, max_history_messages: 40, temperature: null, reasoning_effort: null,
+ monthly_limit_usd: 0, user_monthly_limit_usd: 0, workspace_monthly_limit_usd: 0, reservation_usd: 0.5,
+ };
+ const usage = { run_id: 'run-1', model: 'openai_compatible/team/cheap', connection_id: 'connection-1', profile: 'utility', workflow: 'summarize',
+ reason: 'Selected Utility task model', cost: 0.02, total_tokens: 120, cost_source: 'estimated', call_count: 1, errors: 0, duration_ms: 1200,
+ budget: { cost_remaining: 0.48, tokens_remaining: null },
+ attempts: [{ connection_id: 'connection-1', model: 'team/cheap', status: 'success', total_tokens: 120 }] };
let connection = { id: 'connection-1', name: 'Local gateway', provider: 'openai_compatible', model: 'team/default', config: { endpoint: 'http://gateway.test/v1' }, secrets_set: ['api_key'] };
await page.route('**/api/v1/**', async (route) => {
const path = new URL(route.request().url()).pathname;
@@ -30,6 +41,21 @@ try {
if (path.endsWith('/auth/setup-status')) payload = { needs_setup: false };
else if (path.endsWith('/auth/me')) payload = { user: { id: 1, username: 'operator', email: '[email protected]', role: 'admin', created_at: '2026-01-01' } };
else if (path.endsWith('/ai/status')) payload = { enabled, configured: true };
+ else if (path.endsWith('/ai/management/preview')) payload = { connection_id: connection.id, model: 'team/cheap', reason: 'Approved Utility candidate' };
+ else if (path.endsWith('/ai/management')) {
+ if (method === 'PUT') management = route.request().postDataJSON();
+ payload = management;
+ } else if (path.endsWith('/ai/usage')) payload = {
+ totals: { cost: 0.02, tokens: 120, runs: 1, errors: 0, unknown_cost_runs: 0, average_duration_ms: 1200 },
+ groups: { model: [{ label: usage.model, cost: 0.02, tokens: 120, runs: 1 }], profile: [{ label: 'utility', cost: 0.02, tokens: 120, runs: 1 }] },
+ runs: [{ id: 'run-1', model: usage.model, profile: 'utility', status: 'success', cost: 0.02, reason: usage.reason, created_at: '2026-09-08T12:00:00Z', usage }],
+ allowances: [{ scope: 'Panel', remaining: 9.98, limit: 10 }], has_more: false, next_offset: 50,
+ };
+ else if (path.endsWith('/ai/chat/stream')) {
+ sentChat = route.request().postDataJSON();
+ const events = [['open', { conversation_id: 'chat-1' }], ['run_start', usage], ['text_delta', { text: 'The server is healthy.' }], ['done', { conversation_id: 'chat-1', usage }]];
+ return route.fulfill({ contentType: 'text/event-stream', body: events.map(([name, data]) => `event: ${name}\ndata: ${JSON.stringify(data)}\n\n`).join('') });
+ }
else if (path.endsWith('/ai/settings')) {
if (method === 'PUT') {
const body = route.request().postDataJSON();
@@ -43,7 +69,8 @@ try {
{ name: 'endpoint', label: 'API base URL', type: 'url', required: true },
{ name: 'api_key', label: 'API key', type: 'text', secret: true },
] }] };
- else if (path.endsWith('/ai/connections/probe')) payload = { models: ['team/default', 'team/reasoning', 'team/fast'], source: 'live' };
+ else if (path.endsWith('/ai/connections/probe')) payload = { models: ['team/default', 'team/reasoning', 'team/fast'], source: 'live',
+ model_details: [{ id: 'team/default' }, { id: 'team/reasoning', reasoning: true, tools: true, context_window: 128000, max_output_tokens: 8192, pricing: { input: 1, output: 3 } }, { id: 'team/fast' }] };
else if (path.endsWith('/ai/connections/connection-1')) {
savedConnection = route.request().postDataJSON();
connection = { ...connection, ...savedConnection, secrets_set: ['api_key'] };
@@ -62,8 +89,11 @@ try {
await page.locator('#ai-max-cost').fill('1.25');
await page.getByRole('button', { name: 'Discover models', exact: true }).click();
await page.getByRole('button', { name: 'Browse models (3)', exact: true }).click();
+ await page.getByLabel('Capability', { exact: true }).selectOption('reasoning');
+ await page.getByText('1 USD input / 3 USD output per 1M tokens', { exact: false }).waitFor();
+ assert.equal(await page.getByRole('option').filter({ hasText: 'team/default' }).count(), 0);
await page.getByPlaceholder('Search models…').fill('reasoning');
- await page.getByRole('option', { name: 'team/reasoning', exact: true }).click();
+ await page.getByRole('option', { name: /^team\/reasoning/ }).click();
assert.equal(await page.locator('#ai-model').inputValue(), 'team/reasoning');
await page.getByRole('button', { name: 'Save connection', exact: true }).click();
await page.getByText('Connection saved.', { exact: false }).waitFor();
@@ -83,6 +113,46 @@ try {
await page.locator('[data-testid="ai-state"][data-enabled="false"][data-ready="false"]').waitFor({ state: 'attached' });
await page.screenshot({ path: `test-results/settings-ai-${theme}.png`, fullPage: true });
+ await page.getByRole('tab', { name: 'Task models and behavior' }).click();
+ await page.locator('#ai-profile-utility-connection').selectOption('connection-1');
+ await page.locator('#ai-profile-utility-model').fill('team/cheap');
+ await page.getByRole('button', { name: 'Add model candidate' }).first().click();
+ await page.locator('#ai-pool-0-model').fill('team/cheap');
+ await page.locator('#ai-tier-0').selectOption('budget');
+ await page.locator('#ai-routing_enabled').click();
+ await page.locator('#ai-routing-sample').fill('Summarize server health');
+ await page.getByRole('button', { name: 'Preview routing' }).click();
+ await page.getByText('Approved Utility candidate').waitFor();
+ await page.locator('#ai-max_output_tokens').fill('1024');
+ await page.getByRole('button', { name: 'Save AI management settings' }).click();
+ await page.getByText('AI management settings saved.').waitFor();
+ assert.equal(management.profiles.utility.model, 'team/cheap');
+ assert.equal(management.routing_enabled, true);
+ assert.equal(Number(management.max_output_tokens), 1024);
+ await page.screenshot({ path: `test-results/settings-ai-management-${theme}.png`, fullPage: true });
+ await page.reload();
+ await page.getByRole('tab', { name: 'Task models and behavior' }).click();
+ assert.equal(await page.locator('#ai-profile-utility-model').inputValue(), 'team/cheap');
+ await page.getByRole('tab', { name: 'AI usage', exact: true }).click();
+ await page.getByRole('heading', { name: 'Run details' }).waitFor();
+ await page.getByRole('button', { name: 'Details', exact: true }).click();
+ await page.getByRole('dialog').getByText('Cost source: estimated').waitFor();
+ await page.screenshot({ path: `test-results/settings-ai-usage-${theme}.png`, fullPage: true });
+ await page.keyboard.press('Escape');
+ enabled = true;
+ await visit('chat');
+ await page.locator('#ai-chat-workflow').selectOption('summarize');
+ await page.locator('#ai-chat-profile').selectOption('utility');
+ await page.getByRole('textbox', { name: 'Message the assistant' }).fill('Summarize server health');
+ await page.getByRole('textbox', { name: 'Message the assistant' }).press('Enter');
+ await page.locator('.sk-ai-message__usage summary').click();
+ await page.locator('.sk-ai-message__usage summary').getByText('0.02 USD', { exact: false }).waitFor();
+ await page.getByText('Conversation cost allowance remaining:', { exact: false }).waitFor();
+ assert.equal(sentChat.workflow, 'summarize');
+ assert.equal(sentChat.profile, 'utility');
+ assert(!sentChat.connection_id, 'A default connection override must not bypass task roles');
+ await page.screenshot({ path: `test-results/ai-chat-usage-${theme}.png`, fullPage: true });
+
await visit('notifications');
await page.getByRole('heading', { name: 'Notification Channels' }).waitFor();
assert.equal(await page.getByRole('switch', { name: 'Slack', exact: true }).count(), 0);
@@ -136,6 +206,13 @@ try {
await visit(pane);
await page.locator('.settings-card, .settings-form').first().waitFor();
assert(await page.evaluate(() => document.documentElement.scrollWidth <= window.innerWidth), `${pane} overflows on mobile`);
+ if (pane === 'ai') {
+ for (const tab of ['Task models and behavior', 'AI usage']) {
+ await page.getByRole('tab', { name: tab, exact: true }).click();
+ assert(await page.evaluate(() => document.documentElement.scrollWidth <= window.innerWidth), `${tab} overflows on mobile`);
+ }
+ await page.screenshot({ path: `test-results/settings-ai-mobile-${theme}.png`, fullPage: true });
+ }
if (pane === 'notifications') {
await page.getByRole('button', { name: 'Save Preferences' }).waitFor();
const actions = await page.locator('.settings-actions--footer > button').evaluateAll((nodes) => nodes.map((node) => ({ top: node.getBoundingClientRect().top, bottom: node.getBoundingClientRect().bottom })));
diff --git a/frontend/tests/browser/settings-ui.jsx b/frontend/tests/browser/settings-ui.jsx
index 41a309ce..3dfdc34f 100644
--- a/frontend/tests/browser/settings-ui.jsx
+++ b/frontend/tests/browser/settings-ui.jsx
@@ -5,12 +5,16 @@ import { LocaleProvider } from '../../src/contexts/LocaleContext';
import { ToastProvider } from '../../src/contexts/ToastContext';
import { ConfirmProvider } from '../../src/contexts/ConfirmContext';
import { AIProvider } from '../../src/contexts/AIContext';
+import { WorkspaceProvider } from '../../src/contexts/WorkspaceContext';
import { useServerkitAI } from '../../src/contexts/useServerkitAI';
import AISettingsTab from '../../src/components/settings/AISettingsTab';
import ProfileTab from '../../src/components/settings/ProfileTab';
import NotificationsTab from '../../src/components/settings/NotificationsTab';
import ApiSettingsTab from '../../src/components/settings/ApiSettingsTab';
import { FormField } from '../../src/components/FormField';
+import ConnectionSelector from '../../src/components/ai/ConnectionSelector';
+import Composer from '../../src/components/ai/Composer';
+import Message from '../../src/components/ai/Message';
import '../../src/styles/main.scss';
export function Fixture() {
@@ -23,6 +27,7 @@ export function Fixture() {
{pane === 'profile' && <ProfileTab />}
{pane === 'notifications' && <NotificationsTab />}
{pane === 'api' && <ApiSettingsTab />}
+ {pane === 'chat' && <div className="settings-card"><ConnectionSelector /><Composer />{ai.messages.map((message) => <Message key={message.id} message={message} />)}</div>}
{pane === 'controls' && <div className="settings-card">
<FormField label="Native field" htmlFor="native-field"><input id="native-field" /></FormField>
<FormField label="Native select" htmlFor="native-select"><select id="native-select"><option>Option</option></select></FormField>
@@ -33,7 +38,7 @@ export function Fixture() {
</>;
}
createRoot(document.getElementById('root')).render(
- <MemoryRouter><LocaleProvider><AuthProvider><ToastProvider><ConfirmProvider><AIProvider>
+ <MemoryRouter><LocaleProvider><AuthProvider><WorkspaceProvider><ToastProvider><ConfirmProvider><AIProvider>
<Fixture />
- </AIProvider></ConfirmProvider></ToastProvider></AuthProvider></LocaleProvider></MemoryRouter>
+ </AIProvider></ConfirmProvider></ToastProvider></WorkspaceProvider></AuthProvider></LocaleProvider></MemoryRouter>
);