ServerKit is a lightweight, modern server control panel for managing web applications, databases, and services on your VPS or dedicated server. Built with Python/Flask backend and React frontend.
Python astracloudflarecontrol-panelcyberpaneldnsfablehosting-control-panelhosting-panelkimi-k3server-managementserverkitubuntu-web-panelwebserver
docs(ai): document settings standards and Prompture integration review
Juan Denis9bb474e1116b
diff --git a/docs/AI_MANAGEMENT_REVIEW.md b/docs/AI_MANAGEMENT_REVIEW.mdnew file mode 100644index 00000000..b424918a--- /dev/null+++ b/docs/AI_MANAGEMENT_REVIEW.md@@ -0,0 +1,173 @@+# ServerKit AI management: source review and integration proposal++Reviewed 2026-09-08. Primary references are the local Prompture checkout+(`d36872b`) and AgentSite checkout (`db323d4`). CachiBot is a secondary reference+for its model-slot UI. This is a source review, not a live provider benchmark.+Neither reference repository was modified.++## Main finding++ServerKit should expose more of Prompture's existing capabilities through a+small, coherent settings UI. It does not need another model-routing or token+accounting engine. However, Prompture does not supply ServerKit's user/workspace+authorization, saved-connection policy, or reporting views automatically.++ServerKit pins and has installed Prompture **1.11.0**. Normalized source-text+comparison found these local Prompture files identical to the installed package:+`agents/conversation.py`, `agents/async_agent.py`, `pipeline/resolver.py`,+`pipeline/routing.py`, and `infra/{budget,session,tracker,ledger,capabilities}.py`.+The checkout's `VERSION` file says 1.10.0; it is not evidence that the installed+runtime lacks these features. No SDK upgrade is needed for the capabilities+reviewed below. AgentSite declares `prompture[all]>=1.9.1`, not an exact pin.++## What Prompture already provides++| Capability | Existing implementation | ServerKit integration |+| --- | --- | --- |+| Model roles and inheritance | `ModelResolver`, ordered configuration layers, utility-to-default fallback | Store explicit connection/model choices for Utility, Standard, and Advanced roles; resolve request override → task role → default. |+| Routing | `ModelRouter`, cost-optimized, balanced, quality-first, fast strategies; routing explanations | Select only from administrator-approved connection/model candidates before a job starts. Strategies are heuristics, not a reliability or operational-risk classifier. |+| Catalog and capabilities | Provider descriptors, discovery, model rates, context limits, tool/vision/structured-output flags | Enrich the existing model picker with verified metadata, filters, and input/output prices per 1M tokens. Keep manual gateway aliases. Unknown capability/pricing stays unknown. |+| Generation controls | Conversation options, history cap, tool timeout/round/result limits; Agent iteration limits, guardrails, structured output | Add a validated Advanced section for supported controls. Distinguish lifetime token budget from per-response output-token limit. Do not expose arbitrary driver JSON or imply all models accept temperature/reasoning settings. |+| Budget policies | `hard_stop`, `warn_and_continue`, `degrade`, token/cost ceilings, remaining budget | Use the existing hard-stop mechanism now. Add degradation only after connection-aware fallback is implemented and tested. |+| Per-run observability | `UsageSession`, driver callbacks, tokens/cache tokens, costs, errors, latency, per-model summaries | Attach usage collection to each actual run; preserve ServerKit context and show the result in chat and reports. |+| Persistent observability | `UsageTracker` events, filters, tags/metadata, daily/monthly budget records; `ModelUsageLedger` aggregates | Reuse event collection/calculation, with a ServerKit-owned data location and access-controlled reporting. Global SDK aggregates are not per-user accounting. |+| Larger workflows | Agent/DeepAgent abstractions, guardrails, structured outputs, delegation and context hooks | Consider separate read-only diagnostic workflows later. Keep existing plugin tools, RBAC checks, and write confirmations in control of server operations. |++Sources: Prompture [resolver](../../prompture/prompture/pipeline/resolver.py),+[routing](../../prompture/prompture/pipeline/routing.py),+[Conversation](../../prompture/prompture/agents/conversation.py),+[AsyncAgent](../../prompture/prompture/agents/async_agent.py),+[capabilities](../../prompture/prompture/infra/capabilities.py),+[budget](../../prompture/prompture/infra/budget.py),+[session](../../prompture/prompture/infra/session.py),+[tracker](../../prompture/prompture/infra/tracker.py), and+[ledger](../../prompture/prompture/infra/ledger.py).++## What to borrow from AgentSite++Its [model picker](../../AgentSite/frontend/src/components/shared/ModelSelect.jsx)+and [model endpoint](../../AgentSite/agentsite/api/routes/models.py) demonstrate+searchable provider groups, context/output sizes, tools, vision, reasoning, and+pricing. The useful idea is showing enough information to choose a model,+without requiring users to memorize IDs. Port the interaction into ServerKit's+shared SCSS controls, not AgentSite's Tailwind markup.++Its [configuration](../../AgentSite/agentsite/config.py) assigns cheaper strategies+to triage and some specialist reviewers, and quality-first to the final judge.+That is a better model for ServerKit than treating every task as equally costly.+AgentSite also has [analytics endpoints](../../AgentSite/agentsite/api/routes/agents.py)+and a [routing-cost view](../../AgentSite/frontend/src/components/analytics/CostByRoutingChart.jsx).+Borrow the questions they answer: which model did the work, how much did it cost,+how often did it fail, and which task/strategy caused the spend?++There are important limits to copying its implementation:++- `route_for()` selects the first syntactically valid model in a configured+ strategy pool. It does not attempt runtime failover. Searches found calls only+ in tests; the production orchestrator calls `resolve_agent_model()` instead.+ Strategy configuration alone is therefore not proof of routed generation in+ this checkout. See [resolver](../../AgentSite/agentsite/engine/model_resolver.py)+ and [orchestrator](../../AgentSite/agentsite/agents/orchestrator.py).+- AgentSite's utility resolver takes the first budget fallback model. In+ ServerKit, an inexpensive task model and an outage fallback should be separate+ settings; they solve different problems.+- Model-cost endpoints collapse ledger rows into a dictionary keyed only by model,+ although the ledger key is model plus API-key hash. Multiple credential rows+ can overwrite one another. The default ledger is also process-user-wide,+ while the configured tracker uses AgentSite's data directory. ServerKit should+ aggregate its own scoped run events instead of copying those endpoints.++## Recommended settings structure++Keep five sections within AI Assistant, using the shared card, field, switch,+model-picker, and footer patterns in [SETTINGS_UI.md](SETTINGS_UI.md):++1. **Connections:** credentials, endpoint, connection status, model discovery.+2. **Task models:** three labeled roles, each showing its saved connection and+ model. Show inherited defaults explicitly; no empty dropdown with hidden behavior.+3. **Limits and behavior:** spend/token limits and supported generation controls.+ Keep technical controls in an Advanced disclosure with sensible defaults.+4. **Routing and fallback:** opt-in strategy, approved candidates, fallback order,+ and a preview explaining the selected model. Publish this only with working+ consumers and connection-bound driver creation.+5. **Usage:** conversation/run totals and administrator reports by period, task+ role, model, connection, user, and workspace.++| Task role | Intended work | Selection rule |+| --- | --- | --- |+| Utility / economical | Short summaries, classification, structured extraction | Prefer an explicitly configured inexpensive model with required capabilities; inherit Standard if unset. Do not add model calls for tasks already handled deterministically, such as current chat titles. |+| Standard | Everyday questions and routine tool-assisted diagnosis | Default assistant connection/model. |+| Advanced | Complex incident analysis and multi-step diagnosis | Explicit user selection or a defined task type; inherit Standard if unset. Higher capability never changes permissions. |++Persist each choice as `{connection_id, model}`. The model portion is an opaque+provider/gateway ID, including embedded slashes. Resolve before the run, record+the decision and its reason, and preserve the selected connection for resumed+chats. A selected strategy must not switch credentials midway through a tool loop.++## Integration issues to account for++**Budget enforcement was disabled in ServerKit.** `build_conversation()` supplied+`max_cost` with `budget_policy=None`, which makes Prompture's check a no-op.+This change enables `hard_stop` for a nonzero ceiling on both fresh and resumed+conversations, validates saved limits, translates budget stops, and preserves+usage even when a turn ends without an answer. Zero disables the ceiling. The+default is $0.50, matching the settings endpoint. These checks use reported+usage: the last call can overshoot and unpriced gateway aliases need a provider+spending limit.++**Routing options are not all enforcement mechanisms.** In the reviewed+`routing.py`, `max_cost_per_call` is declared/documented but is not used by the+selection logic. Preferred providers are ordering preferences, not an allowlist.+Discovery uses the global environment. ServerKit must validate candidates against+saved connections and enforce spending through an actual budget check.++**Automatic fallback needs an adapter.** Conversation's `_switch_model()` calls+`get_driver_for_model(new_model)` without forwarding the saved driver/environment.+Do not enable it across ServerKit connections as-is. An explicit driver factory+must retain connection identity, endpoint, and credentials. Failed write tools+must not be silently replayed after fallback. ServerKit's legacy `fallback_models`+setting is currently stored but not applied by its conversation factory.++**CachiBot's router argument is not portable.** Its agent construction can pass+`model_router`; the reviewed Prompture `AsyncAgent` constructor accepts no such+argument. Use the public routing API as a separate selection step, rather than+copying that constructor call.++**Usage needs run attribution.** ServerKit persists Conversation usage snapshots,+which are cumulative; its chat context currently drops that usage when loading+messages and handling completion. Summing those snapshots double-counts spend.+Capture per-call/run usage with Prompture callbacks, or explicitly calculate+turn deltas, while retaining the cumulative value for conversation limits.++**SDK tracker scopes are not tenant isolation.** The reviewed tracker+`check_budget(scope)` selects a budget record by scope but sums usage events over+the chosen period without filtering events by that scope. Do not advertise it as+a per-user/workspace quota unchanged. Scope accounting, permissions, and any+concurrent spend reservations require ServerKit integration.++## Delivery sequence and acceptance criteria++1. **Current repair:** consistent settings surfaces and controls, model discovery+ picker, persistent enable switch, clear notification footer, enforced+ conversation ceiling. Offline tests cover both chat modes and budget resume.+2. **Profiles and visible usage:** implement the three roles and their actual task+ consumers together. Add a migration-backed run record with unique run ID,+ conversation/user/workspace, requested role, actual connection/model, routing+ reason, tokens, cost source, duration, outcome, and fallback attempts. Reuse+ Prompture counters; keep accounting idempotent. Test inheritance, permissions,+ exact gateway IDs, unknown pricing, and no double counting after resume.+3. **Routing and reports:** add approved model pools and a routing preview; show+ daily/monthly totals, model/task breakdowns, latency/failure rates, and remaining+ limits. Separate estimated cost from provider-reported cost. Missing prices+ must read “Unknown,” not “Free.” Test that disallowed connections cannot be+ selected and that report totals reconcile with recorded runs.+4. **Fallback and richer workflows:** add credential-aware failover/degradation,+ bounded diagnostic workflows, and supported context/history controls after+ the accounting and permission tests pass. Avoid shared response caching for+ live server state; any future cache needs user/workspace/connection scoping+ and an explicit freshness policy.++The task-role/routing/reporting work in steps 2–4 is a proposal, not implemented+UI. This review is an internal FYI; no upstream issue or external message was sent.+Useful upstream follow-ups are a connection-aware fallback factory, enforcement+or clarification of `max_cost_per_call`, and scoped tracker budget aggregation.diff --git a/docs/AI_PROVIDERS.md b/docs/AI_PROVIDERS.mdindex ce02debd..f84faeae 100644--- a/docs/AI_PROVIDERS.md+++ b/docs/AI_PROVIDERS.md@@ -1,12 +1,14 @@# AI provider connectionsIn **Settings → AI Assistant**, add a named connection, choose a provider, fill-in its configuration, and enter a default model ID. **Discover models** uses the-current form values; manual model entry remains available when a provider does-not support discovery. **Save connection** stores credentials encrypted. The+in its configuration, and choose a default model. **Discover models** uses the+current form values. **Browse models** opens a searchable picker; manual model+entry remains available for gateways and providers without discovery. **Save connection** stores credentials encrypted. Thefirst connection becomes the default automatically; another connection can be-made default when saving it. Enable the assistant using the settings below the-connection editor and save those settings.+made default when saving it. The **Enable AI assistant** switch saves immediately+and refreshes the chat UI. Disabling blocks new messages in both chat endpoints+while retaining conversation history. Limits and protections have their own Save+action; saving a connection preserves unsaved changes to those settings.New chats offer a connection selector and an optional model override. Once achat starts, its connection and model stay fixed. Changing the panel default@@ -61,6 +63,16 @@ Existing assistant RBAC, redaction, and write-action confirmations remain activeGateway routing aliases may have unknown pricing; enforce authoritative spendinglimits at the gateway. Automatic fallback to another connection is not enabled.+The per-conversation cost ceiling uses Prompture's `hard_stop` policy for new+and resumed conversations (default $0.50; zero disables it). It checks reported+cumulative usage before each model call, so a final call can overshoot the+remaining amount. Budget stops return a clear message, and usage is persisted+even if the turn stops before producing an answer. This is not an authoritative+billing limit when provider pricing is unavailable.++See [AI management review](AI_MANAGEMENT_REVIEW.md) for the Prompture and+AgentSite source review and the proposed task profiles, routing, and reporting.+## Upgrade behaviorMigration `098_ai_provider_connections` creates encrypted connection records,diff --git a/docs/SETTINGS_UI.md b/docs/SETTINGS_UI.mdnew file mode 100644index 00000000..89f05e89--- /dev/null+++ b/docs/SETTINGS_UI.md@@ -0,0 +1,42 @@+# Settings UI contract++Use the existing IBM Plex / SCSS design system and shared components. Settings+must pass the same checks as the rest of the panel.++- Wrap each form section in `settings-card` or use `settings-form` on the form.+ A form inside a card inherits that card rather than drawing a second surface.+- Use `FormField` with `Input`, `Select`, and `Switch`. Native text fields inside+ `FormField` receive shared styling too; checkboxes and radios keep their size.+- Put the label on the left and the switch on the right for a boolean preference.+ Use checkboxes for row selection and acknowledgements. Do not put colored pills+ around a control or style every descendant `span` (Radix uses internal spans).+- Keep text readable on tinted backgrounds. Use the primary text token for+ settings alerts and selected navigation; reserve accent color for controls.+- For section-specific create actions, use the right side of the section header+ or the existing view picker's action slot. Page actions still use the topbar.+ Keep save actions in the form footer with the shared spacing classes.+ Use `settings-actions settings-actions--footer` for paired footer actions:+ secondary first, primary rightmost, 12px gap, stacked on small screens.+- Use `EmptyState`. Inside a card it inherits the surface and drops its border.+ Use `stat-strip` for a group of metrics, with a distinct muted cell surface.+- Sidebar, topbar, and statusbar all use `--bg-sidebar`. The statusbar has a+ divider, without an extra bright surface or shadow.+- Name scope explicitly. Personal notification destinations differ from shared+ server delivery settings. Slack currently has no per-user destination.+- Never reset an unrelated draft after saving another section. A switch that+ saves immediately must say so, keep its previous state if the save fails, and+ refresh consumers after success.++## Verification++Run `npm run lint`, `npm run build`, and `npm run test:settings-ui` in `frontend`.+The browser regression renders actual components and the complete SCSS cascade,+with synthetic API responses. It checks light/dark surfaces, mobile overflow,+model discovery and exact IDs, secret preservation, assistant enable persistence+and failures, notification scope, and section action placement. CI runs it and+uploads screenshots from `frontend/test-results/`.++Provider discovery tests do not contact external providers. Live connectivity+must be verified separately with a configured connection. Model capabilities and+pricing should only be shown when supplied by a trustworthy provider catalog;+do not infer them from a model name.