ServerKit is a lightweight, modern server control panel for managing web applications, databases, and services on your VPS or dedicated server. Built with Python/Flask backend and React frontend.
Python astracloudflarecontrol-panelcyberpaneldnsfablehosting-control-panelhosting-panelkimi-k3server-managementserverkitubuntu-web-panelwebserver
Fleet monitor + security and agent hardening
Juan Denisf6f08ae9d146
diff --git a/ROADMAP.md b/ROADMAP.mdindex 55fd4786..d7f9e076 100644--- a/ROADMAP.md+++ b/ROADMAP.md@@ -352,42 +352,42 @@ Complete the workflow engine with proper execution logic, missing triggers, and----## Phase 26: Agent Fleet Management (Planned)+## Phase 26: Agent Fleet Management (Completed)**Priority: High**Level up agent management from "connect and monitor" to full fleet control.-- [ ] Agent version tracking and compatibility matrix (panel version ↔ agent version)-- [ ] Push agent upgrades from the panel (single server or fleet-wide rollout)-- [ ] Staged rollout support — upgrade agents in batches with health checks between waves-- [ ] Agent health dashboard — connection uptime, heartbeat latency, command success rate per agent-- [ ] Auto-discovery of new servers on the local network (mDNS/broadcast scan)-- [ ] Agent registration approval workflow (admin must approve before agent joins fleet)-- [ ] Bulk agent operations — restart, upgrade, rotate keys across selected servers-- [ ] Agent changelog and release notes visible in UI-- [ ] Offline agent command queue — persist commands and deliver when agent reconnects-- [ ] Command retry with configurable backoff for failed/timed-out operations-- [ ] Agent connection diagnostics — test connectivity, latency, firewall check from panel+- [x] Agent version tracking and compatibility matrix (panel version ↔ agent version)+- [x] Push agent upgrades from the panel (single server or fleet-wide rollout)+- [x] Staged rollout support — upgrade agents in batches with health checks between waves+- [x] Agent health dashboard — connection uptime, heartbeat latency, command success rate per agent+- [x] Auto-discovery of new servers on the local network (mDNS/broadcast scan)+- [x] Agent registration approval workflow (admin must approve before agent joins fleet)+- [x] Bulk agent operations — restart, upgrade, rotate keys across selected servers+- [x] Agent changelog and release notes visible in UI+- [x] Offline agent command queue — persist commands and deliver when agent reconnects+- [x] Command retry with configurable backoff for failed/timed-out operations+- [x] Agent connection diagnostics — test connectivity, latency, firewall check from panel----## Phase 27: Cross-Server Monitoring Dashboard (Planned)+## Phase 27: Cross-Server Monitoring Dashboard (Completed)**Priority: High**Fleet-wide visibility — see everything at a glance and catch problems early.-- [ ] Fleet overview dashboard — heatmap of all servers by CPU/memory/disk usage-- [ ] Server comparison charts — overlay metrics from multiple servers on one graph-- [ ] Per-server alert thresholds (CPU > 80% for 5 min → warning, > 95% → critical)-- [ ] Anomaly detection — automatic baseline learning, alert on deviations-- [ ] Custom metric dashboards — drag-and-drop widgets, save layouts per user-- [ ] Metric correlation view — spot relationships between metrics across servers-- [ ] Capacity forecasting — trend-based predictions (disk full in X days, memory growth rate)-- [ ] Metrics export — Prometheus endpoint (`/metrics`), CSV download, JSON API-- [ ] Grafana integration guide and pre-built dashboard templates-- [ ] Fleet-wide search — find which server is running a specific container, service, or port+- [x] Fleet overview dashboard — heatmap of all servers by CPU/memory/disk usage+- [x] Server comparison charts — overlay metrics from multiple servers on one graph+- [x] Per-server alert thresholds (CPU > 80% for 5 min → warning, > 95% → critical)+- [x] Anomaly detection — automatic baseline learning, alert on deviations+- [x] Custom metric dashboards — drag-and-drop widgets, save layouts per user+- [x] Metric correlation view — spot relationships between metrics across servers+- [x] Capacity forecasting — trend-based predictions (disk full in X days, memory growth rate)+- [x] Metrics export — Prometheus endpoint (`/metrics`), CSV download, JSON API+- [x] Grafana integration guide and pre-built dashboard templates+- [x] Fleet-wide search — find which server is running a specific container, service, or port---@@ -585,9 +585,9 @@ Spin up and manage cloud servers directly from the panel.| v1.2.0 | Backups, Advanced SSL, Advanced Security | Completed || v1.3.0 | Email server, API enhancements | Completed || v1.4.0 | Team & permissions, SSO & OAuth login | Completed |-| v1.5.0 | New UI, Visual Designer, Services Page | Current |-| v1.6.0 | Workflow triggers & completion, fleet management | Planned |-| v1.7.0 | Cross-server monitoring, agent plugin system | Planned |+| v1.5.0 | New UI, Visual Designer, Services Page | Completed |+| v1.6.0 | Workflow triggers & completion, fleet management | Current |+| v1.7.0 | Cross-server monitoring, agent plugin system | In Progress || v1.8.0 | Server templates, multi-tenancy | Planned || v1.9.0 | Advanced SSL, DNS management, Nginx config | Planned || v2.0.0 | Status pages, server provisioning, performance | Planned |diff --git a/agent/docker-compose.yml b/agent/docker-compose.ymlindex 1c145c97..02422b2f 100644--- a/agent/docker-compose.yml+++ b/agent/docker-compose.yml@@ -15,13 +15,13 @@services:agent:- image: jhd3197/serverkit-agent:latest+ image: jhd3197/serverkit-agent:${AGENT_VERSION:-1.0.0}container_name: serverkit-agentrestart: unless-stopped- # Run as root to access Docker socket- # The agent drops privileges internally where possible- user: root+ # Use docker group for socket access instead of running as root+ group_add:+ - dockervolumes:# Docker socket for container managementdiff --git a/agent/internal/agent/agent.go b/agent/internal/agent/agent.goindex 152fd415..ea946ed8 100644--- a/agent/internal/agent/agent.go+++ b/agent/internal/agent/agent.go@@ -3,12 +3,14 @@ package agentimport ("bufio""context"+ "crypto/hmac"+ "crypto/sha256""encoding/base64"+ "encoding/hex""encoding/json""fmt""net""os"- "runtime""sync""time"@@ -259,43 +261,58 @@ func (a *Agent) discoveryLoop(ctx context.Context) {var req struct {Type string `json:"type"`Timestamp int64 `json:"timestamp"`+ Signature string `json:"signature"`}if err := json.Unmarshal(buf[:n], &req); err != nil || (req.Type != "discovery_request" && req.Type != string(protocol.TypeDiscoveryRequest)) {continue}- // Respond with agent info- hostname, _ := os.Hostname()- info := protocol.SystemInfo{- Hostname: hostname,- OS: runtime.GOOS,- OSVersion: "", // Could populate this- Architecture: runtime.GOARCH,- AgentVersion: Version,+ // If agent has no credentials (not registered), don't respond to discovery+ if a.cfg.Auth.APIKey == "" {+ continue}+ // Validate timestamp is within 60 seconds+ now := time.Now().UnixMilli()+ if req.Timestamp <= 0 || abs(now-req.Timestamp) > 60000 {+ a.log.Debug("Ignoring discovery request with stale timestamp")+ continue+ }++ // Verify HMAC signature+ if req.Signature == "" {+ a.log.Debug("Ignoring discovery request without signature")+ continue+ }+ expectedMessage := fmt.Sprintf("discovery:%d", req.Timestamp)+ mac := hmac.New(sha256.New, []byte(a.cfg.Auth.APIKey))+ mac.Write([]byte(expectedMessage))+ expectedSignature := hex.EncodeToString(mac.Sum(nil))+ if !hmac.Equal([]byte(req.Signature), []byte(expectedSignature)) {+ a.log.Debug("Ignoring discovery request with invalid signature")+ continue+ }++ // Respond with minimal agent info (no detailed hardware specs)+ hostname, _ := os.Hostname()resp := struct {- Type string `json:"type"`- AgentID string `json:"agent_id"`- Hostname string `json:"hostname"`- AgentVersion string `json:"agent_version"`- OS string `json:"os"`- Arch string `json:"arch"`- Timestamp int64 `json:"timestamp"`- Info protocol.SystemInfo `json:"info"`+ Type string `json:"type"`+ AgentID string `json:"agent_id"`+ Hostname string `json:"hostname"`+ Status string `json:"status"`+ AgentVersion string `json:"agent_version"`+ Timestamp int64 `json:"timestamp"`}{Type: "discovery",AgentID: a.cfg.Agent.ID,Hostname: hostname,+ Status: "online",AgentVersion: Version,- OS: runtime.GOOS,- Arch: runtime.GOARCH,Timestamp: time.Now().UnixMilli(),- Info: info,}data, _ := json.Marshal(resp)-+// Send response to remoteAddr on port+1respAddr, _ := net.ResolveUDPAddr("udp", fmt.Sprintf("%s:%d", remoteAddr.IP.String(), port+1))conn.WriteToUDP(data, respAddr)@@ -303,6 +320,14 @@ func (a *Agent) discoveryLoop(ctx context.Context) {}}+// abs returns the absolute value of an int64+func abs(x int64) int64 {+ if x < 0 {+ return -x+ }+ return x+}+// heartbeatLoop sends periodic heartbeatsfunc (a *Agent) heartbeatLoop(ctx context.Context) {ticker := time.NewTicker(a.cfg.Server.PingInterval)@@ -389,15 +414,18 @@ func (a *Agent) handleCommand(data []byte) {return}- // Execute command+ // Execute command with enforced maximum timeoutstart := time.Now()- ctx := context.Background()- if cmd.Timeout > 0 {- var cancel context.CancelFunc- ctx, cancel = context.WithTimeout(ctx, time.Duration(cmd.Timeout)*time.Millisecond)- defer cancel()+ maxTimeout := 5 * time.Minute+ cmdTimeout := time.Duration(cmd.Timeout) * time.Millisecond++ if cmdTimeout <= 0 || cmdTimeout > maxTimeout {+ cmdTimeout = maxTimeout}+ ctx, cancel := context.WithTimeout(context.Background(), cmdTimeout)+ defer cancel()+result, err := handler(ctx, cmd.Params)duration := time.Since(start)diff --git a/agent/internal/agent/registration.go b/agent/internal/agent/registration.goindex e1874765..e55513cc 100644--- a/agent/internal/agent/registration.go+++ b/agent/internal/agent/registration.go@@ -3,7 +3,10 @@ package agentimport ("bytes""context"+ "crypto/hmac"+ "crypto/sha256""crypto/tls"+ "encoding/hex""encoding/json""fmt""io"@@ -96,8 +99,7 @@ func (r *Registration) Register(serverURL, token, name string) (*RegistrationResTimeout: 30 * time.Second,Transport: &http.Transport{TLSClientConfig: &tls.Config{- // Allow insecure for development - in production this should be strict- InsecureSkipVerify: strings.HasPrefix(serverURL, "http://") || strings.Contains(serverURL, "localhost"),+ InsecureSkipVerify: os.Getenv("SERVERKIT_INSECURE_TLS") == "true",},},}@@ -176,8 +178,16 @@ func (r *Registration) Unregister(serverURL, agentID, apiKey, apiSecret string)return fmt.Errorf("failed to create request: %w", err)}- req.Header.Set("X-API-Key", apiKey)- req.Header.Set("X-API-Secret", apiSecret)+ // HMAC-based authentication instead of sending raw secret+ timestamp := fmt.Sprintf("%d", time.Now().UnixMilli())+ message := fmt.Sprintf("%s:%s", agentID, timestamp)+ mac := hmac.New(sha256.New, []byte(apiSecret))+ mac.Write([]byte(message))+ signature := hex.EncodeToString(mac.Sum(nil))++ req.Header.Set("X-Agent-ID", agentID)+ req.Header.Set("X-Timestamp", timestamp)+ req.Header.Set("X-Signature", signature)resp, err := client.Do(req)if err != nil {diff --git a/agent/internal/ws/client.go b/agent/internal/ws/client.goindex 621da67b..47854491 100644--- a/agent/internal/ws/client.go+++ b/agent/internal/ws/client.go@@ -7,6 +7,7 @@ import ("fmt""net/http""net/url"+ "os""strings""sync""time"@@ -108,8 +109,8 @@ func (c *Client) Connect(ctx context.Context) error {HandshakeTimeout: 10 * time.Second,}- // Allow insecure for development- if c.cfg.InsecureSkipVerify {+ // Only allow insecure TLS when explicitly set via environment variable+ if os.Getenv("SERVERKIT_INSECURE_TLS") == "true" {dialer.TLSClientConfig = &tls.Config{InsecureSkipVerify: true}}diff --git a/backend/.env.example b/backend/.env.exampleindex 8230c85e..1ba9c075 100644--- a/backend/.env.example+++ b/backend/.env.example@@ -23,7 +23,7 @@ PORT=5000# IMPORTANT: Generate a unique key for production!# Run: python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"# Or: python -c "import base64; import os; print(base64.urlsafe_b64encode(os.urandom(32)).decode())"-SERVERKIT_ENCRYPTION_KEY=dSenbMzCii4PMYP3Bb_N9jdqm6pa_TOD4l9yAwhhSBA=+SERVERKIT_ENCRYPTION_KEY= # Generate with: python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"# GitHub Repository for agent releases (owner/repo format)# This is used to fetch the latest agent version for the downloads pagediff --git a/backend/app/__init__.py b/backend/app/__init__.pyindex 16248543..10e4aa17 100644--- a/backend/app/__init__.py+++ b/backend/app/__init__.py@@ -1,5 +1,5 @@import os-from flask import Flask, send_from_directory+from flask import Flask, send_from_directory, request, jsonifyfrom flask_sqlalchemy import SQLAlchemyfrom flask_jwt_extended import JWTManagerfrom flask_cors import CORS@@ -222,6 +222,10 @@ def create_app(config_name=None):from app.api.servers import servers_bpapp.register_blueprint(servers_bp, url_prefix='/api/v1/servers')+ # Register blueprints - Fleet Monitor (Cross-server monitoring)+ from app.api.fleet_monitor import fleet_monitor_bp+ app.register_blueprint(fleet_monitor_bp, url_prefix='/api/v1/fleet-monitor')+# Handle database migrations (Alembic)with app.app_context():from app.services.migration_service import MigrationService@@ -250,6 +254,29 @@ def create_app(config_name=None):# Start hourly analytics aggregation and event retry threads_start_api_background_threads(app)+ # Request body size limit+ app.config['MAX_CONTENT_LENGTH'] = 100 * 1024 * 1024 # 100MB limit++ # Reject 2FA pending tokens on non-2FA endpoints+ @app.before_request+ def check_2fa_pending():+ """Reject 2FA pending tokens on non-2FA endpoints."""+ from flask_jwt_extended import verify_jwt_in_request, get_jwt+ if request.endpoint and request.path.startswith('/api/'):+ # Allow 2FA verification endpoints+ if '/two-factor/verify' in request.path or '/two-factor/verify-backup' in request.path:+ return+ # Allow auth endpoints (login, refresh)+ if '/auth/login' in request.path or '/auth/refresh' in request.path:+ return+ try:+ verify_jwt_in_request()+ claims = get_jwt()+ if claims.get('2fa_pending'):+ return jsonify({'error': '2FA verification required'}), 403+ except Exception:+ pass # Let @jwt_required handle actual auth errors+# Serve frontend for root path@app.route('/')def serve_index():diff --git a/backend/app/agent_gateway.py b/backend/app/agent_gateway.pyindex e2420386..6d513197 100644--- a/backend/app/agent_gateway.py+++ b/backend/app/agent_gateway.py@@ -5,6 +5,7 @@ WebSocket gateway for ServerKit agents.Handles agent connections, authentication, and message routing."""+from collections import defaultdictfrom flask import requestfrom flask_socketio import Namespace, emit, disconnectimport json@@ -15,6 +16,22 @@ from app.models.server import Serverfrom app.utils.ip_utils import is_ip_allowedfrom app.services.anomaly_detection_service import anomaly_detection_service+# In-memory rate limiter for agent authentication+_auth_attempts = defaultdict(list)+_AUTH_RATE_LIMIT = 10 # max attempts per window+_AUTH_RATE_WINDOW = 60 # seconds+++def _check_auth_rate_limit(ip_address: str) -> bool:+ """Check if IP has exceeded auth rate limit."""+ now = time.time()+ # Clean old entries+ _auth_attempts[ip_address] = [t for t in _auth_attempts[ip_address] if now - t < _AUTH_RATE_WINDOW]+ if len(_auth_attempts[ip_address]) >= _AUTH_RATE_LIMIT:+ return False+ _auth_attempts[ip_address].append(now)+ return True+class AgentNamespace(Namespace):"""@@ -50,6 +67,11 @@ class AgentNamespace(Namespace):}"""sid = request.sid+ ip_address = request.remote_addr+ if not _check_auth_rate_limit(ip_address):+ emit('auth_response', {'success': False, 'error': 'Rate limit exceeded'}, room=request.sid, namespace='/agent')+ return+agent_id = data.get('agent_id')api_key_prefix = data.get('api_key_prefix')signature = data.get('signature')diff --git a/backend/app/api/auth.py b/backend/app/api/auth.pyindex 471b0131..87f7689e 100644--- a/backend/app/api/auth.py+++ b/backend/app/api/auth.py@@ -1,3 +1,4 @@+import loggingfrom datetime import datetimefrom flask import Blueprint, request, jsonifyfrom flask_jwt_extended import (@@ -12,6 +13,8 @@ from app.models import User, AuditLog, SystemSettingsfrom app.services.settings_service import SettingsServicefrom app.services.audit_service import AuditService+logger = logging.getLogger(__name__)+auth_bp = Blueprint('auth', __name__)@@ -64,8 +67,12 @@ def register():# Check if registration is allowedis_first_user = User.query.count() == 0- if not is_first_user and not invitation and not SettingsService.is_registration_enabled():- return jsonify({'error': 'Registration is disabled'}), 403+ if not is_first_user and not invitation:+ if not SettingsService.needs_setup() and not SettingsService.is_registration_enabled():+ logger.warning(f"Registration attempt blocked - setup already completed. IP: {request.remote_addr}")+ return jsonify({'error': 'Registration is disabled'}), 403+ if not SettingsService.is_registration_enabled():+ return jsonify({'error': 'Registration is disabled'}), 403email = data.get('email')username = data.get('username')@@ -75,10 +82,10 @@ def register():return jsonify({'error': 'Missing required fields'}), 400if User.query.filter_by(email=email).first():- return jsonify({'error': 'Email already registered'}), 409+ return jsonify({'error': 'This email or username is unavailable'}), 409if User.query.filter_by(username=username).first():- return jsonify({'error': 'Username already taken'}), 409+ return jsonify({'error': 'This email or username is unavailable'}), 409if len(password) < 8:return jsonify({'error': 'Password must be at least 8 characters'}), 400@@ -285,6 +292,7 @@ def get_current_user():@auth_bp.route('/me', methods=['PUT'])[email protected]("3 per minute")@jwt_required()def update_current_user():current_user_id = get_jwt_identity()diff --git a/backend/app/api/fleet_monitor.py b/backend/app/api/fleet_monitor.pynew file mode 100644index 00000000..7e88484d--- /dev/null+++ b/backend/app/api/fleet_monitor.py@@ -0,0 +1,198 @@+"""+Fleet Monitor API++Endpoints for fleet-wide monitoring: heatmaps, comparisons,+alert thresholds, anomaly detection, capacity forecasting,+fleet search, and metrics export.+"""++import os+from flask import Blueprint, request, jsonify, Response+from flask_jwt_extended import jwt_required, get_jwt_identity++from app.services.fleet_monitor_service import fleet_monitor_service+from app.middleware.rbac import admin_required++fleet_monitor_bp = Blueprint('fleet_monitor', __name__)+++# ==================== Heatmap ====================++@fleet_monitor_bp.route('/heatmap', methods=['GET'])+@jwt_required()+def get_fleet_heatmap():+ """Get latest metrics for all servers in heatmap format."""+ group_id = request.args.get('group_id')+ return jsonify(fleet_monitor_service.get_fleet_heatmap(group_id))+++# ==================== Comparison ====================++@fleet_monitor_bp.route('/comparison', methods=['GET'])+@jwt_required()+def get_comparison():+ """Get multi-server comparison time-series."""+ ids_param = request.args.get('ids', '')+ server_ids = [s.strip() for s in ids_param.split(',') if s.strip()]+ metric = request.args.get('metric', 'cpu')+ period = request.args.get('period', '24h')++ if not server_ids:+ return jsonify({'error': 'ids parameter is required'}), 400++ return jsonify(fleet_monitor_service.get_comparison_timeseries(server_ids, metric, period))+++# ==================== Alerts ====================++@fleet_monitor_bp.route('/alerts', methods=['GET'])+@jwt_required()+def get_alerts():+ """Get metric alerts."""+ status = request.args.get('status')+ severity = request.args.get('severity')+ server_id = request.args.get('server_id')+ limit = request.args.get('limit', 50, type=int)++ return jsonify(fleet_monitor_service.get_alerts(status, severity, server_id, limit))+++@fleet_monitor_bp.route('/alerts/<alert_id>/acknowledge', methods=['POST'])+@jwt_required()+def acknowledge_alert(alert_id):+ """Acknowledge an active alert."""+ user_id = get_jwt_identity()+ success = fleet_monitor_service.acknowledge_alert(alert_id, user_id)+ if not success:+ return jsonify({'error': 'Cannot acknowledge alert'}), 400+ return jsonify({'success': True})+++@fleet_monitor_bp.route('/alerts/<alert_id>/resolve', methods=['POST'])+@jwt_required()+def resolve_alert(alert_id):+ """Resolve an alert."""+ success = fleet_monitor_service.resolve_alert(alert_id)+ if not success:+ return jsonify({'error': 'Cannot resolve alert'}), 400+ return jsonify({'success': True})+++# ==================== Thresholds ====================++@fleet_monitor_bp.route('/thresholds', methods=['GET'])+@jwt_required()+def get_thresholds():+ """Get alert thresholds."""+ server_id = request.args.get('server_id')+ return jsonify(fleet_monitor_service.get_thresholds(server_id))+++@fleet_monitor_bp.route('/thresholds', methods=['POST'])+@jwt_required()+@admin_required+def create_threshold():+ """Create or update an alert threshold."""+ data = request.get_json()+ result = fleet_monitor_service.upsert_threshold(data)+ if 'error' in result:+ return jsonify(result), 400+ return jsonify(result), 201+++@fleet_monitor_bp.route('/thresholds/<threshold_id>', methods=['DELETE'])+@jwt_required()+@admin_required+def delete_threshold(threshold_id):+ """Delete an alert threshold."""+ success = fleet_monitor_service.delete_threshold(threshold_id)+ if not success:+ return jsonify({'error': 'Threshold not found'}), 404+ return jsonify({'success': True})+++# ==================== Anomalies ====================++@fleet_monitor_bp.route('/anomalies', methods=['GET'])+@jwt_required()+def get_anomalies():+ """Get anomaly detection results."""+ server_id = request.args.get('server_id')+ return jsonify(fleet_monitor_service.detect_anomalies(server_id))+++# ==================== Capacity Forecast ====================++@fleet_monitor_bp.route('/forecast/<server_id>', methods=['GET'])+@jwt_required()+def get_forecast(server_id):+ """Get capacity forecast for a server."""+ metric = request.args.get('metric', 'disk')+ return jsonify(fleet_monitor_service.forecast_capacity(server_id, metric))+++# ==================== Fleet Search ====================++@fleet_monitor_bp.route('/search', methods=['GET'])+@jwt_required()+def search_fleet():+ """Search across fleet for servers, containers, services, ports."""+ query = request.args.get('q', '')+ search_type = request.args.get('type', 'any')++ if not query or len(query) < 2:+ return jsonify({'error': 'Query must be at least 2 characters'}), 400++ return jsonify(fleet_monitor_service.search_fleet(query, search_type))+++# ==================== Export ====================++@fleet_monitor_bp.route('/export/csv', methods=['GET'])+@jwt_required()+def export_csv():+ """Export metrics as CSV download."""+ ids_param = request.args.get('ids', '')+ server_ids = [s.strip() for s in ids_param.split(',') if s.strip()]+ metric = request.args.get('metric', 'cpu')+ period = request.args.get('period', '24h')++ if not server_ids:+ return jsonify({'error': 'ids parameter is required'}), 400++ csv_data = fleet_monitor_service.export_metrics_csv(server_ids, metric, period)+ return Response(+ csv_data,+ mimetype='text/csv',+ headers={'Content-Disposition': f'attachment; filename=fleet_{metric}_{period}.csv'}+ )+++@fleet_monitor_bp.route('/export/json', methods=['GET'])+@jwt_required()+def export_json():+ """Export metrics as JSON."""+ ids_param = request.args.get('ids', '')+ server_ids = [s.strip() for s in ids_param.split(',') if s.strip()]+ metric = request.args.get('metric', 'cpu')+ period = request.args.get('period', '24h')++ if not server_ids:+ return jsonify({'error': 'ids parameter is required'}), 400++ return jsonify(fleet_monitor_service.get_comparison_timeseries(server_ids, metric, period))+++# ==================== Prometheus ====================++@fleet_monitor_bp.route('/prometheus', methods=['GET'])+def prometheus_metrics():+ """Prometheus-compatible metrics endpoint (no JWT, uses token param)."""+ token = request.args.get('token') or request.headers.get('X-Prometheus-Token')+ expected = os.environ.get('PROMETHEUS_TOKEN')++ if expected and token != expected:+ return Response('Unauthorized', status=401)++ metrics = fleet_monitor_service.get_prometheus_metrics()+ return Response(metrics, mimetype='text/plain; version=0.0.4; charset=utf-8')diff --git a/backend/app/api/servers.py b/backend/app/api/servers.pyindex 62bff3e2..b015e40e 100644--- a/backend/app/api/servers.py+++ b/backend/app/api/servers.py@@ -11,7 +11,7 @@ from datetime import datetime, timedeltafrom flask import Blueprint, request, jsonify, Response, current_app, redirectfrom flask_jwt_extended import jwt_required, get_jwt_identity-from app import db+from app import db, limiterfrom app.models import Userfrom app.models.server import Server, ServerGroup, ServerMetrics, ServerCommand, AgentSession, AgentVersion, AgentRolloutfrom app.services.agent_registry import agent_registry@@ -320,6 +320,7 @@ def regenerate_token(server_id):@servers_bp.route('/register', methods=['POST'])[email protected]("5 per minute")def register_agent():"""Agent registration endpoint.@@ -388,6 +389,9 @@ def register_agent():ws_scheme = 'wss' if request.is_secure else 'ws'ws_url = f"{ws_scheme}://{request.host}/agent"+ # Security note: api_secret is returned once during registration so the agent+ # can store it. The server-side copy is stored encrypted. The registration token+ # is already cleared above (single-use), preventing re-registration.return jsonify({'agent_id': server.agent_id,'name': server.name,diff --git a/backend/app/api/sso.py b/backend/app/api/sso.pyindex 2f584488..89fb0103 100644--- a/backend/app/api/sso.py+++ b/backend/app/api/sso.py@@ -10,6 +10,7 @@ from app.models.oauth_identity import OAuthIdentityfrom app.services import sso_servicefrom app.services.settings_service import SettingsServicefrom app.services.audit_service import AuditService+from app.middleware.rbac import admin_requiredsso_bp = Blueprint('sso', __name__)@@ -199,12 +200,10 @@ def unlink_provider(provider):# ------------------------------------------------------------------@sso_bp.route('/admin/config', methods=['GET'])-@jwt_required()+@admin_requireddef get_sso_config():"""All SSO settings (secrets redacted)."""user = User.query.get(get_jwt_identity())- if not user or not user.is_admin:- return jsonify({'error': 'Admin access required'}), 403config = {}for key in SettingsService.DEFAULT_SETTINGS:@@ -220,12 +219,10 @@ def get_sso_config():@sso_bp.route('/admin/config/<provider>', methods=['PUT'])-@jwt_required()+@admin_requireddef update_provider_config(provider):"""Update a provider's SSO config."""user = User.query.get(get_jwt_identity())- if not user or not user.is_admin:- return jsonify({'error': 'Admin access required'}), 403if provider not in VALID_PROVIDERS:return jsonify({'error': f'Invalid provider: {provider}'}), 400@@ -254,24 +251,18 @@ def update_provider_config(provider):@sso_bp.route('/admin/test/<provider>', methods=['POST'])-@jwt_required()+@admin_requireddef test_provider(provider):"""Test provider connectivity."""- user = User.query.get(get_jwt_identity())- if not user or not user.is_admin:- return jsonify({'error': 'Admin access required'}), 403-result = sso_service.test_provider_connectivity(provider)return jsonify(result), 200 if result['ok'] else 400@sso_bp.route('/admin/general', methods=['PUT'])-@jwt_required()+@admin_requireddef update_general_settings():"""Update general SSO settings (auto_provision, force_sso, etc.)."""user = User.query.get(get_jwt_identity())- if not user or not user.is_admin:- return jsonify({'error': 'Admin access required'}), 403data = request.get_json() or {}general_keys = ['sso_auto_provision', 'sso_default_role', 'sso_force_sso', 'sso_allowed_domains']diff --git a/backend/app/api/two_factor.py b/backend/app/api/two_factor.pyindex 2cd5f6a0..65697a23 100644--- a/backend/app/api/two_factor.py+++ b/backend/app/api/two_factor.py@@ -4,11 +4,15 @@ Two-Factor Authentication APIProvides endpoints for enabling, disabling, and managing 2FA."""+import loggingfrom flask import Blueprint, request, jsonifyfrom flask_jwt_extended import jwt_required, get_jwt_identity+from app import limiterfrom app.models import Userfrom app.services.totp_service import TOTPService, TwoFactorSetup+logger = logging.getLogger(__name__)+two_factor_bp = Blueprint('two_factor', __name__)@@ -117,6 +121,8 @@ def disable_2fa():if not success:return jsonify({'error': error}), 400+ logger.info(f"2FA disabled for user {user.id} - existing sessions remain valid (TODO: implement session invalidation)")+return jsonify({'message': '2FA has been disabled successfully'}), 200@@ -156,6 +162,7 @@ def regenerate_backup_codes():@two_factor_bp.route('/verify', methods=['POST'])[email protected]("5 per minute")def verify_2fa_code():"""Verify a 2FA code during login.diff --git a/backend/app/middleware/security.py b/backend/app/middleware/security.pyindex a5303ac3..20a2ce5b 100644--- a/backend/app/middleware/security.py+++ b/backend/app/middleware/security.py@@ -23,8 +23,8 @@ def register_security_headers(app: Flask):# Content Security Policycsp_directives = ["default-src 'self'",- "script-src 'self' 'unsafe-inline' 'unsafe-eval' https://unpkg.com",- "style-src 'self' 'unsafe-inline' https://unpkg.com",+ "script-src 'self'",+ "style-src 'self'","img-src 'self' data: https:","font-src 'self'","connect-src 'self' ws: wss:",diff --git a/backend/app/models/__init__.py b/backend/app/models/__init__.pyindex 6d608ed5..8cc20a6a 100644--- a/backend/app/models/__init__.py+++ b/backend/app/models/__init__.py@@ -21,6 +21,7 @@ from app.models.api_key import ApiKeyfrom app.models.api_usage import ApiUsageLog, ApiUsageSummaryfrom app.models.event_subscription import EventSubscription, EventDeliveryfrom app.models.invitation import Invitation+from app.models.metric_alert import ServerAlertThreshold, MetricAlert__all__ = ['User', 'Application', 'Domain', 'EnvironmentVariable', 'EnvironmentVariableHistory',@@ -31,5 +32,6 @@ __all__ = ['EnvironmentActivity', 'PromotionJob', 'SanitizationProfile','EmailDomain', 'EmailAccount', 'EmailAlias', 'EmailForwardingRule', 'DNSProviderConfig','OAuthIdentity', 'ApiKey', 'ApiUsageLog', 'ApiUsageSummary',- 'EventSubscription', 'EventDelivery', 'Invitation'+ 'EventSubscription', 'EventDelivery', 'Invitation',+ 'ServerAlertThreshold', 'MetricAlert']diff --git a/backend/app/models/invitation.py b/backend/app/models/invitation.pyindex 4655b29b..94a5014e 100644--- a/backend/app/models/invitation.py+++ b/backend/app/models/invitation.py@@ -16,7 +16,7 @@ class Invitation(db.Model):id = db.Column(db.Integer, primary_key=True)email = db.Column(db.String(255), nullable=True) # Nullable for link-only invitestoken = db.Column(db.String(64), unique=True, nullable=False, index=True,- default=lambda: uuid4().hex)+ default=lambda: __import__('secrets').token_urlsafe(32))role = db.Column(db.String(20), nullable=False, default='developer')permissions = db.Column(db.Text, nullable=True) # JSON custom permissionsinvited_by = db.Column(db.Integer, db.ForeignKey('users.id'), nullable=False)diff --git a/backend/app/models/metric_alert.py b/backend/app/models/metric_alert.pynew file mode 100644index 00000000..e74b11c1--- /dev/null+++ b/backend/app/models/metric_alert.py@@ -0,0 +1,76 @@+"""Models for fleet-wide metric monitoring and alerting."""++import uuid+from datetime import datetime+from app import db+++class ServerAlertThreshold(db.Model):+ """Per-server or global alert threshold configuration."""+ __tablename__ = 'server_alert_thresholds'++ id = db.Column(db.String(36), primary_key=True, default=lambda: str(uuid.uuid4()))+ server_id = db.Column(db.String(36), db.ForeignKey('servers.id'), nullable=True, index=True)+ # null = global default++ metric = db.Column(db.String(20), nullable=False) # cpu, memory, disk+ warning_threshold = db.Column(db.Float, default=80.0)+ critical_threshold = db.Column(db.Float, default=95.0)+ duration_seconds = db.Column(db.Integer, default=300) # sustained for N seconds+ enabled = db.Column(db.Boolean, default=True)++ created_at = db.Column(db.DateTime, default=datetime.utcnow)+ updated_at = db.Column(db.DateTime, default=datetime.utcnow, onupdate=datetime.utcnow)++ server = db.relationship('Server', backref='alert_thresholds')++ def to_dict(self):+ return {+ 'id': self.id,+ 'server_id': self.server_id,+ 'server_name': self.server.name if self.server else None,+ 'metric': self.metric,+ 'warning_threshold': self.warning_threshold,+ 'critical_threshold': self.critical_threshold,+ 'duration_seconds': self.duration_seconds,+ 'enabled': self.enabled,+ 'created_at': self.created_at.isoformat() if self.created_at else None,+ }+++class MetricAlert(db.Model):+ """Alert triggered when server metrics exceed thresholds."""+ __tablename__ = 'metric_alerts'++ id = db.Column(db.String(36), primary_key=True, default=lambda: str(uuid.uuid4()))+ server_id = db.Column(db.String(36), db.ForeignKey('servers.id'), nullable=False, index=True)++ metric = db.Column(db.String(20), nullable=False) # cpu, memory, disk+ severity = db.Column(db.String(10), nullable=False) # warning, critical+ value = db.Column(db.Float) # the value that triggered it+ threshold = db.Column(db.Float) # the threshold exceeded+ duration_seconds = db.Column(db.Integer) # how long it was exceeded++ status = db.Column(db.String(20), default='active', index=True) # active, acknowledged, resolved+ acknowledged_by = db.Column(db.Integer, db.ForeignKey('users.id'), nullable=True)++ created_at = db.Column(db.DateTime, default=datetime.utcnow)+ resolved_at = db.Column(db.DateTime)++ server = db.relationship('Server', backref='metric_alerts')++ def to_dict(self):+ return {+ 'id': self.id,+ 'server_id': self.server_id,+ 'server_name': self.server.name if self.server else None,+ 'metric': self.metric,+ 'severity': self.severity,+ 'value': self.value,+ 'threshold': self.threshold,+ 'duration_seconds': self.duration_seconds,+ 'status': self.status,+ 'acknowledged_by': self.acknowledged_by,+ 'created_at': self.created_at.isoformat() if self.created_at else None,+ 'resolved_at': self.resolved_at.isoformat() if self.resolved_at else None,+ }diff --git a/backend/app/services/agent_registry.py b/backend/app/services/agent_registry.pyindex 946df8a5..e3cde688 100644--- a/backend/app/services/agent_registry.py+++ b/backend/app/services/agent_registry.py@@ -7,9 +7,12 @@ and handles agent lifecycle events.import hmacimport hashlib+import loggingimport secretsimport timeimport threading++logger = logging.getLogger(__name__)from datetime import datetime, timedeltafrom typing import Dict, Optional, Callable, Anyfrom dataclasses import dataclass, field@@ -472,10 +475,17 @@ class AgentRegistry:"""Handle command result from agent"""agent = self.get_agent_by_socket(socket_id)if not agent:+ logger.warning(f"Command result from unknown socket: {socket_id}")returncommand_id = result.get('command_id')if not command_id:+ logger.warning(f"Command result missing command_id from agent: {agent.server_id}")+ return++ # Validate command_id format to prevent injection+ if not isinstance(command_id, str) or len(command_id) > 64:+ logger.warning(f"Invalid command_id format from agent: {agent.server_id}")returnwith self._lock:@@ -519,7 +529,7 @@ class AgentRegistry:# Check timestamp (allow 5 minute window)now = int(time.time() * 1000)- if abs(now - timestamp) > 300000: # 5 minutes+ if abs(now - timestamp) > 60000: # 60 secondsif ip_address:anomaly_detection_service.track_auth_attempt(None, False, ip_address)return None@@ -575,6 +585,11 @@ class AgentRegistry:if ip_address:anomaly_detection_service.track_auth_attempt(server.id, True, ip_address)+ # TODO: Implement per-message session token validation. Currently, the session+ # token is issued at auth time but not verified on each subsequent message.+ # Full session-per-message validation requires protocol changes on both the+ # agent (Go) and backend sides.+return server# ==================== System Info ====================diff --git a/backend/app/services/build_service.py b/backend/app/services/build_service.pyindex 0ede960f..35a2cd4f 100644--- a/backend/app/services/build_service.py+++ b/backend/app/services/build_service.py@@ -481,8 +481,7 @@ class BuildService:env.update(env_vars)process = subprocess.Popen(- build_cmd,- shell=True,+ ['bash', '-c', build_cmd],cwd=app_path,env=env,stdout=subprocess.PIPE,diff --git a/backend/app/services/cron_service.py b/backend/app/services/cron_service.pyindex 289210f1..c9ddfb84 100644--- a/backend/app/services/cron_service.py+++ b/backend/app/services/cron_service.py@@ -7,6 +7,7 @@ for cross-platform compatibility.import osimport re+import shleximport subprocessimport platformfrom typing import Dict, List, Optional@@ -245,6 +246,20 @@ class CronService:return ', '.join(descriptions) if descriptions else schedule+ BLOCKED_PATTERNS = [';', '&&', '||', '|', '`', '$(', '>', '<', '\n', '\r']++ @classmethod+ def _validate_command(cls, command: str) -> bool:+ """Validate cron command to prevent injection."""+ for pattern in cls.BLOCKED_PATTERNS:+ if pattern in command:+ return False+ # Require absolute paths+ parts = shlex.split(command)+ if parts and not parts[0].startswith('/'):+ return False+ return True+@classmethoddef add_job(cls, schedule: str, command: str, name: str = None,description: str = None) -> Dict:@@ -257,6 +272,9 @@ class CronService:if not command or not command.strip():return {'success': False, 'error': 'Command cannot be empty'}+ if not cls._validate_command(command):+ return {'success': False, 'error': 'Invalid command: must use absolute paths and cannot contain shell operators (;, &&, ||, |, `, $())'}+# Generate job IDjob_id = f"job_{datetime.now().strftime('%Y%m%d%H%M%S')}"@@ -594,8 +612,7 @@ class CronService:try:# Run the commandresult = subprocess.run(- command,- shell=True,+ ['bash', '-c', command],capture_output=True,text=True,timeout=60diff --git a/backend/app/services/database_service.py b/backend/app/services/database_service.pyindex 23ab8d25..06b4027d 100644--- a/backend/app/services/database_service.py+++ b/backend/app/services/database_service.py@@ -1,5 +1,6 @@import subprocessimport os+import reimport secretsimport stringimport json@@ -8,8 +9,19 @@ from datetime import datetimefrom app import paths+def _validate_identifier(name: str, max_length: int = 64) -> bool:+ """Validate database/user identifiers to prevent SQL injection."""+ return bool(re.match(r'^[a-zA-Z0-9_$]{1,}$', name)) and len(name) <= max_length++class DatabaseService:- """Service for managing MySQL/MariaDB and PostgreSQL databases."""+ """Service for managing MySQL/MariaDB and PostgreSQL databases.++ NOTE (L6): Subprocess timeout values in this service (typically 30s) should be+ reviewed periodically to ensure they are appropriate. Shorter timeouts reduce+ the window for resource exhaustion attacks, but may break legitimate long-running+ operations like large database backups or restores.+ """BACKUP_DIR = paths.DB_BACKUP_DIR@@ -51,14 +63,18 @@ class DatabaseService:"""Execute a MySQL query."""try:cmd = ['mysql', '-u', 'root']- if root_password:- cmd.extend([f'-p{root_password}'])if database:cmd.extend(['-D', database])cmd.extend(['-e', query])+ # Use MYSQL_PWD env var to avoid passing password on CLI+ env = None+ if root_password:+ env = os.environ.copy()+ env['MYSQL_PWD'] = root_password+result = subprocess.run(- cmd, capture_output=True, text=True+ cmd, capture_output=True, text=True, env=env)return {'success': result.returncode == 0,@@ -106,6 +122,12 @@ class DatabaseService:@staticmethoddef mysql_create_database(name, charset='utf8mb4', collation='utf8mb4_unicode_ci', root_password=None):"""Create a MySQL database."""+ if not _validate_identifier(name):+ return {'success': False, 'error': 'Invalid identifier: only alphanumeric characters and underscores allowed'}+ if not _validate_identifier(charset):+ return {'success': False, 'error': 'Invalid charset identifier'}+ if not _validate_identifier(collation, max_length=128):+ return {'success': False, 'error': 'Invalid collation identifier'}query = f"CREATE DATABASE IF NOT EXISTS `{name}` CHARACTER SET {charset} COLLATE {collation};"result = DatabaseService.mysql_execute(query, root_password=root_password)return result@@ -113,6 +135,8 @@ class DatabaseService:@staticmethoddef mysql_drop_database(name, root_password=None):"""Drop a MySQL database."""+ if not _validate_identifier(name):+ return {'success': False, 'error': 'Invalid identifier: only alphanumeric characters and underscores allowed'}query = f"DROP DATABASE IF EXISTS `{name}`;"result = DatabaseService.mysql_execute(query, root_password=root_password)return result@@ -140,13 +164,23 @@ class DatabaseService:@staticmethoddef mysql_create_user(username, password, host='localhost', root_password=None):"""Create a MySQL user."""- query = f"CREATE USER IF NOT EXISTS '{username}'@'{host}' IDENTIFIED BY '{password}';"+ if not _validate_identifier(username):+ return {'success': False, 'error': 'Invalid identifier: only alphanumeric characters and underscores allowed'}+ if not _validate_identifier(host):+ return {'success': False, 'error': 'Invalid host identifier'}+ # Use MYSQL_PWD env var to avoid passing password on CLI+ escaped_password = password.replace("'", "\\'").replace("\\", "\\\\")+ query = f"CREATE USER IF NOT EXISTS '{username}'@'{host}' IDENTIFIED BY '{escaped_password}';"result = DatabaseService.mysql_execute(query, root_password=root_password)return result@staticmethoddef mysql_drop_user(username, host='localhost', root_password=None):"""Drop a MySQL user."""+ if not _validate_identifier(username):+ return {'success': False, 'error': 'Invalid identifier: only alphanumeric characters and underscores allowed'}+ if not _validate_identifier(host):+ return {'success': False, 'error': 'Invalid host identifier'}query = f"DROP USER IF EXISTS '{username}'@'{host}';"result = DatabaseService.mysql_execute(query, root_password=root_password)return result@@ -154,6 +188,12 @@ class DatabaseService:@staticmethoddef mysql_grant_privileges(username, database, privileges='ALL', host='localhost', root_password=None):"""Grant privileges to a MySQL user."""+ if not _validate_identifier(username):+ return {'success': False, 'error': 'Invalid identifier: only alphanumeric characters and underscores allowed'}+ if not _validate_identifier(database):+ return {'success': False, 'error': 'Invalid identifier: only alphanumeric characters and underscores allowed'}+ if not _validate_identifier(host):+ return {'success': False, 'error': 'Invalid host identifier'}query = f"GRANT {privileges} ON `{database}`.* TO '{username}'@'{host}'; FLUSH PRIVILEGES;"result = DatabaseService.mysql_execute(query, root_password=root_password)return result@@ -161,6 +201,12 @@ class DatabaseService:@staticmethoddef mysql_revoke_privileges(username, database, privileges='ALL', host='localhost', root_password=None):"""Revoke privileges from a MySQL user."""+ if not _validate_identifier(username):+ return {'success': False, 'error': 'Invalid identifier: only alphanumeric characters and underscores allowed'}+ if not _validate_identifier(database):+ return {'success': False, 'error': 'Invalid identifier: only alphanumeric characters and underscores allowed'}+ if not _validate_identifier(host):+ return {'success': False, 'error': 'Invalid host identifier'}query = f"REVOKE {privileges} ON `{database}`.* FROM '{username}'@'{host}'; FLUSH PRIVILEGES;"result = DatabaseService.mysql_execute(query, root_password=root_password)return result@@ -168,6 +214,10 @@ class DatabaseService:@staticmethoddef mysql_get_user_privileges(username, host='localhost', root_password=None):"""Get privileges for a MySQL user."""+ if not _validate_identifier(username):+ return []+ if not _validate_identifier(host):+ return []result = DatabaseService.mysql_execute(f"SHOW GRANTS FOR '{username}'@'{host}';",root_password=root_password@@ -194,13 +244,17 @@ class DatabaseService:try:cmd = ['mysqldump', '-u', 'root']- if root_password:- cmd.append(f'-p{root_password}')cmd.append(database)+ # Use MYSQL_PWD env var to avoid passing password on CLI+ env = None+ if root_password:+ env = os.environ.copy()+ env['MYSQL_PWD'] = root_password+# Pipe through gzipwith open(output_path, 'wb') as f:- dump = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)+ dump = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)gzip = subprocess.Popen(['gzip'], stdin=dump.stdout, stdout=f, stderr=subprocess.PIPE)dump.stdout.close()gzip.communicate()@@ -224,14 +278,18 @@ class DatabaseService:try:cmd = ['mysql', '-u', 'root']- if root_password:- cmd.append(f'-p{root_password}')cmd.append(database)+ # Use MYSQL_PWD env var to avoid passing password on CLI+ env = None+ if root_password:+ env = os.environ.copy()+ env['MYSQL_PWD'] = root_password+if backup_path.endswith('.gz'):# Decompress and restoregunzip = subprocess.Popen(['gunzip', '-c', backup_path], stdout=subprocess.PIPE)- restore = subprocess.Popen(cmd, stdin=gunzip.stdout, stdout=subprocess.PIPE, stderr=subprocess.PIPE)+ restore = subprocess.Popen(cmd, stdin=gunzip.stdout, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)gunzip.stdout.close()_, stderr = restore.communicate()@@ -239,7 +297,7 @@ class DatabaseService:return {'success': False, 'error': stderr.decode()}else:with open(backup_path, 'r') as f:- result = subprocess.run(cmd, stdin=f, capture_output=True, text=True)+ result = subprocess.run(cmd, stdin=f, capture_output=True, text=True, env=env)if result.returncode != 0:return {'success': False, 'error': result.stderr}@@ -555,8 +613,12 @@ class DatabaseService:# Build mysql command with JSON output formatcmd = ['mysql', '-u', 'root']++ # Use MYSQL_PWD env var to avoid passing password on CLI+ env = Noneif root_password:- cmd.extend([f'-p{root_password}'])+ env = os.environ.copy()+ env['MYSQL_PWD'] = root_passwordcmd.extend(['-D', database,'-e', query,@@ -570,7 +632,8 @@ class DatabaseService:cmd,capture_output=True,text=True,- timeout=timeout+ timeout=timeout,+ env=env)execution_time = time.time() - start_time@@ -1038,10 +1101,13 @@ class DatabaseService:def docker_mysql_execute(container_name, query, database=None, user='root', password=None):"""Execute a MySQL query inside a Docker container."""try:- cmd = ['docker', 'exec', container_name, 'mysql', '-u', user]+ cmd = ['docker', 'exec']+ # Use MYSQL_PWD env var to avoid passing password on CLIif password:- cmd.append(f'-p{password}')+ cmd.extend(['-e', f'MYSQL_PWD={password}'])++ cmd.extend([container_name, 'mysql', '-u', user])if database:cmd.extend(['-D', database])@@ -1137,9 +1203,11 @@ class DatabaseService:try:start_time = time.time()- cmd = ['docker', 'exec', container_name, 'mysql', '-u', user]+ cmd = ['docker', 'exec']+ # Use MYSQL_PWD env var to avoid passing password on CLIif password:- cmd.append(f'-p{password}')+ cmd.extend(['-e', f'MYSQL_PWD={password}'])+ cmd.extend([container_name, 'mysql', '-u', user])cmd.extend(['-D', database,'-e', query,diff --git a/backend/app/services/discovery_service.py b/backend/app/services/discovery_service.pyindex f6229ca6..0966225e 100644--- a/backend/app/services/discovery_service.py+++ b/backend/app/services/discovery_service.py@@ -7,26 +7,43 @@ Uses UDP broadcast to find agents.import socketimport json+import hmac+import hashlibimport threadingimport time+import loggingfrom datetime import datetime-from typing import List, Dict+from typing import List, Dict, Optionalfrom app import dbfrom app.models.server import Server+logger = logging.getLogger(__name__)+class DiscoveryService:"""Service for discovering new servers/agents on the network."""- def __init__(self, port=9000):+ # Maximum allowed age for discovery responses (seconds)+ MAX_RESPONSE_AGE = 30++ def __init__(self, port=9000, secret_key: Optional[str] = None):self.port = port+ self.secret_key = secret_keyself._discovered_agents = {} # agent_id -> infoself._is_scanning = Falseself._lock = threading.Lock()+ def _sign_discovery_request(self, request_data: dict, secret_key: str) -> dict:+ """Sign a discovery request with HMAC."""+ request_data['timestamp'] = int(time.time() * 1000)+ message = json.dumps(request_data, sort_keys=True)+ signature = hmac.new(secret_key.encode(), message.encode(), hashlib.sha256).hexdigest()+ request_data['signature'] = signature+ return request_data+def start_scan(self, duration=10) -> List[Dict]:"""Start a network scan for agents.@@ -57,16 +74,22 @@ class DiscoveryService:sock.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)sock.settimeout(2)- message = json.dumps({+ request_data = {'type': 'discovery_request','timestamp': int(time.time() * 1000)- })-+ }++ # Sign the request if a shared secret is configured+ if self.secret_key:+ request_data = self._sign_discovery_request(request_data, self.secret_key)++ message = json.dumps(request_data)+# Broadcast to common local networkssock.sendto(message.encode(), ('255.255.255.255', self.port))sock.close()except Exception as e:- print(f"Error sending discovery broadcast: {e}")+ logger.error(f"Error sending discovery broadcast: {e}")def _listen_for_responses(self):"""Listen for UDP discovery responses from agents"""@@ -82,27 +105,37 @@ class DiscoveryService:try:info = json.loads(data.decode())if info.get('type') == 'discovery':+ # Validate timestamp to prevent replay attacks+ response_ts = info.get('timestamp')+ if response_ts:+ age_seconds = abs(time.time() * 1000 - response_ts) / 1000+ if age_seconds > self.MAX_RESPONSE_AGE:+ logger.warning(f"Stale discovery response from {addr[0]} (age: {age_seconds:.0f}s), ignoring")+ continue+agent_id = info.get('agent_id')if agent_id:# Add IP address from senderinfo['ip_address'] = addr[0]-+# Check if already registeredserver = Server.query.filter_by(agent_id=agent_id).first()info['is_registered'] = server is not Noneif server:info['server_name'] = server.nameinfo['server_id'] = server.id-+with self._lock:self._discovered_agents[agent_id] = info+ else:+ logger.warning(f"Discovery response from {addr[0]} missing agent_id")except Exception as e:- print(f"Error parsing discovery response: {e}")+ logger.error(f"Error parsing discovery response: {e}")except socket.timeout:continuesock.close()except Exception as e:- print(f"Error in discovery listener: {e}")+ logger.error(f"Error in discovery listener: {e}")def get_discovered_agents(self) -> List[Dict]:"""Get currently discovered agents"""diff --git a/backend/app/services/docker_service.py b/backend/app/services/docker_service.pyindex 234c0492..0062fce0 100644--- a/backend/app/services/docker_service.py+++ b/backend/app/services/docker_service.py@@ -1,6 +1,7 @@import subprocessimport jsonimport os+import shleximport yamlfrom datetime import datetime@@ -138,7 +139,7 @@ class DockerService:cmd.append(image)if command:- cmd.extend(command.split())+ cmd.extend(shlex.split(command))result = subprocess.run(cmd, capture_output=True, text=True)@@ -183,7 +184,7 @@ class DockerService:cmd.append(image)if command:- cmd.extend(command.split())+ cmd.extend(shlex.split(command))result = subprocess.run(cmd, capture_output=True, text=True)@@ -490,7 +491,7 @@ class DockerService:if tty:cmd.append('-t')cmd.append(container_id)- cmd.extend(command.split())+ cmd.extend(shlex.split(command))result = subprocess.run(cmd, capture_output=True, text=True, timeout=60)return {diff --git a/backend/app/services/file_service.py b/backend/app/services/file_service.pyindex ab02a7fc..82fc0f7f 100644--- a/backend/app/services/file_service.py+++ b/backend/app/services/file_service.py@@ -299,8 +299,13 @@ class FileService:if not os.path.exists(old_path):return {'success': False, 'error': 'Path not found'}+ # Validate new_name has no path separators+ if '/' in new_name or '\\' in new_name or '..' in new_name:+ return {'success': False, 'error': 'Invalid filename: path separators not allowed'}+new_path = os.path.join(os.path.dirname(old_path), new_name)+ # Re-validate the constructed pathif not cls.is_path_allowed(new_path):return {'success': False, 'error': 'Access denied: target path not allowed'}@@ -370,6 +375,9 @@ class FileService:try:# Convert octal string to intmode_int = int(mode, 8)+ # Validate permission mode+ if mode_int < 0o000 or mode_int > 0o777:+ return {'success': False, 'error': 'Invalid permission mode. Must be between 000 and 777.'}os.chmod(path, mode_int)return {'success': True, 'path': path, 'mode': mode}except ValueError:diff --git a/backend/app/services/fleet_monitor_service.py b/backend/app/services/fleet_monitor_service.pynew file mode 100644index 00000000..3dd134a0--- /dev/null+++ b/backend/app/services/fleet_monitor_service.py@@ -0,0 +1,613 @@+"""+Fleet Monitor Service++Provides fleet-wide monitoring: heatmap, comparison charts, alert thresholds,+anomaly detection, capacity forecasting, fleet search, and metrics export.+"""++import io+import csv+import math+import logging+from datetime import datetime, timedelta+from typing import List, Dict, Optional, Any++from sqlalchemy import func, and_++from app import db+from app.models.server import Server, ServerMetrics, ServerGroup+from app.models.metric_alert import ServerAlertThreshold, MetricAlert+from app.services.agent_registry import agent_registry++logger = logging.getLogger(__name__)++METRIC_COLUMNS = {+ 'cpu': 'cpu_percent',+ 'memory': 'memory_percent',+ 'disk': 'disk_percent',+ 'network_rx': 'network_rx_rate',+ 'network_tx': 'network_tx_rate',+}+++class FleetMonitorService:+ """Service for fleet-wide monitoring and alerting."""++ # ==================== Fleet Heatmap ====================++ @staticmethod+ def get_fleet_heatmap(group_id: str = None) -> List[Dict]:+ """Get latest metrics for all servers, shaped for heatmap display."""+ query = Server.query+ if group_id:+ query = query.filter_by(group_id=group_id)++ servers = query.all()+ result = []++ for server in servers:+ latest = ServerMetrics.query.filter_by(+ server_id=server.id+ ).order_by(ServerMetrics.timestamp.desc()).first()++ result.append({+ 'id': server.id,+ 'name': server.name,+ 'status': server.status,+ 'group_id': server.group_id,+ 'group_name': server.group.name if server.group else None,+ 'cpu': round(latest.cpu_percent, 1) if latest and latest.cpu_percent is not None else None,+ 'memory': round(latest.memory_percent, 1) if latest and latest.memory_percent is not None else None,+ 'disk': round(latest.disk_percent, 1) if latest and latest.disk_percent is not None else None,+ 'containers': latest.container_running if latest else None,+ 'last_update': latest.timestamp.isoformat() if latest else None,+ })++ return result++ # ==================== Comparison Timeseries ====================++ @staticmethod+ def get_comparison_timeseries(+ server_ids: List[str], metric: str = 'cpu', period: str = '24h'+ ) -> Dict:+ """Get time-series data for multiple servers for overlay charting."""+ hours_back = {'1h': 1, '6h': 6, '24h': 24, '7d': 168, '30d': 720}.get(period, 24)+ interval = {'1h': 1, '6h': 5, '24h': 15, '7d': 60, '30d': 360}.get(period, 15)+ cutoff = datetime.utcnow() - timedelta(hours=hours_back)+ col_name = METRIC_COLUMNS.get(metric, 'cpu_percent')++ series = []+ for server_id in server_ids:+ server = Server.query.get(server_id)+ if not server:+ continue++ records = ServerMetrics.query.filter(+ ServerMetrics.server_id == server_id,+ ServerMetrics.timestamp >= cutoff+ ).order_by(ServerMetrics.timestamp.asc()).all()++ # Downsample for longer periods+ if interval > 1 and len(records) > 0:+ records = _downsample_simple(records, interval)++ data = []+ for r in records:+ val = getattr(r, col_name, None)+ if val is not None:+ data.append({+ 'timestamp': r.timestamp.isoformat(),+ 'value': round(val, 2)+ })++ series.append({+ 'server_id': server_id,+ 'name': server.name,+ 'data': data+ })++ return {'metric': metric, 'period': period, 'series': series}++ # ==================== Alert Thresholds ====================++ @staticmethod+ def get_thresholds(server_id: str = None) -> List[Dict]:+ """Get alert thresholds, optionally filtered by server."""+ query = ServerAlertThreshold.query+ if server_id:+ query = query.filter(+ (ServerAlertThreshold.server_id == server_id) |+ (ServerAlertThreshold.server_id == None)+ )+ return [t.to_dict() for t in query.all()]++ @staticmethod+ def upsert_threshold(data: Dict) -> Dict:+ """Create or update an alert threshold."""+ threshold_id = data.get('id')+ if threshold_id:+ threshold = ServerAlertThreshold.query.get(threshold_id)+ if not threshold:+ return {'error': 'Threshold not found'}+ else:+ threshold = ServerAlertThreshold()+ db.session.add(threshold)++ threshold.server_id = data.get('server_id')+ threshold.metric = data.get('metric', 'cpu')+ threshold.warning_threshold = data.get('warning_threshold', 80.0)+ threshold.critical_threshold = data.get('critical_threshold', 95.0)+ threshold.duration_seconds = data.get('duration_seconds', 300)+ threshold.enabled = data.get('enabled', True)++ db.session.commit()+ return threshold.to_dict()++ @staticmethod+ def delete_threshold(threshold_id: str) -> bool:+ """Delete an alert threshold."""+ threshold = ServerAlertThreshold.query.get(threshold_id)+ if not threshold:+ return False+ db.session.delete(threshold)+ db.session.commit()+ return True++ # ==================== Alert Checking ====================++ @staticmethod+ def check_fleet_thresholds():+ """Check all online servers against their thresholds. Call periodically."""+ thresholds = ServerAlertThreshold.query.filter_by(enabled=True).all()+ if not thresholds:+ return++ # Group thresholds: per-server overrides + global defaults+ global_thresholds = {}+ server_thresholds = {}+ for t in thresholds:+ if t.server_id:+ server_thresholds.setdefault(t.server_id, {})[t.metric] = t+ else:+ global_thresholds[t.metric] = t++ servers = Server.query.filter_by(status='online').all()++ for server in servers:+ for metric_name, col_name in METRIC_COLUMNS.items():+ # Find applicable threshold (server-specific > global)+ threshold = server_thresholds.get(server.id, {}).get(metric_name)+ if not threshold:+ threshold = global_thresholds.get(metric_name)+ if not threshold:+ continue++ # Get recent metrics for the duration window+ cutoff = datetime.utcnow() - timedelta(seconds=threshold.duration_seconds)+ recent = ServerMetrics.query.filter(+ ServerMetrics.server_id == server.id,+ ServerMetrics.timestamp >= cutoff+ ).all()++ if not recent:+ continue++ values = [getattr(r, col_name) for r in recent if getattr(r, col_name) is not None]+ if not values:+ continue++ avg_val = sum(values) / len(values)++ # Check if there's already an active alert for this server+metric+ existing = MetricAlert.query.filter_by(+ server_id=server.id, metric=metric_name, status='active'+ ).first()++ if avg_val >= threshold.critical_threshold:+ if not existing or existing.severity != 'critical':+ if existing:+ existing.resolved_at = datetime.utcnow()+ existing.status = 'resolved'+ alert = MetricAlert(+ server_id=server.id,+ metric=metric_name,+ severity='critical',+ value=round(avg_val, 1),+ threshold=threshold.critical_threshold,+ duration_seconds=threshold.duration_seconds+ )+ db.session.add(alert)+ elif avg_val >= threshold.warning_threshold:+ if not existing or existing.severity == 'critical':+ if existing and existing.severity == 'critical':+ existing.resolved_at = datetime.utcnow()+ existing.status = 'resolved'+ if not existing or existing.severity == 'critical':+ alert = MetricAlert(+ server_id=server.id,+ metric=metric_name,+ severity='warning',+ value=round(avg_val, 1),+ threshold=threshold.warning_threshold,+ duration_seconds=threshold.duration_seconds+ )+ db.session.add(alert)+ else:+ # Below thresholds - resolve any active alert+ if existing:+ existing.resolved_at = datetime.utcnow()+ existing.status = 'resolved'++ try:+ db.session.commit()+ except Exception as e:+ logger.error(f"Error checking fleet thresholds: {e}")+ db.session.rollback()++ @staticmethod+ def get_alerts(+ status: str = None, severity: str = None,+ server_id: str = None, limit: int = 50+ ) -> List[Dict]:+ """Get metric alerts with filters."""+ query = MetricAlert.query.order_by(MetricAlert.created_at.desc())+ if status:+ query = query.filter_by(status=status)+ if severity:+ query = query.filter_by(severity=severity)+ if server_id:+ query = query.filter_by(server_id=server_id)+ return [a.to_dict() for a in query.limit(limit).all()]++ @staticmethod+ def acknowledge_alert(alert_id: str, user_id: int) -> bool:+ alert = MetricAlert.query.get(alert_id)+ if not alert or alert.status != 'active':+ return False+ alert.status = 'acknowledged'+ alert.acknowledged_by = user_id+ db.session.commit()+ return True++ @staticmethod+ def resolve_alert(alert_id: str) -> bool:+ alert = MetricAlert.query.get(alert_id)+ if not alert or alert.status == 'resolved':+ return False+ alert.status = 'resolved'+ alert.resolved_at = datetime.utcnow()+ db.session.commit()+ return True++ # ==================== Anomaly Detection ====================++ @staticmethod+ def detect_anomalies(server_id: str = None) -> List[Dict]:+ """Simple z-score anomaly detection on 7-day hourly averages."""+ servers = [Server.query.get(server_id)] if server_id else Server.query.filter_by(status='online').all()+ cutoff = datetime.utcnow() - timedelta(days=7)+ anomalies = []++ for server in servers:+ if not server:+ continue++ records = ServerMetrics.query.filter(+ ServerMetrics.server_id == server.id,+ ServerMetrics.timestamp >= cutoff+ ).all()++ if len(records) < 20:+ continue++ latest = records[-1] if records else None+ if not latest:+ continue++ for metric_name, col_name in [('cpu', 'cpu_percent'), ('memory', 'memory_percent'), ('disk', 'disk_percent')]:+ values = [getattr(r, col_name) for r in records if getattr(r, col_name) is not None]+ if len(values) < 10:+ continue++ mean = sum(values) / len(values)+ variance = sum((v - mean) ** 2 for v in values) / len(values)+ stddev = math.sqrt(variance) if variance > 0 else 0++ current = getattr(latest, col_name)+ if current is None or stddev == 0:+ continue++ z_score = (current - mean) / stddev++ if abs(z_score) > 2.5:+ anomalies.append({+ 'server_id': server.id,+ 'server_name': server.name,+ 'metric': metric_name,+ 'current_value': round(current, 1),+ 'mean': round(mean, 1),+ 'stddev': round(stddev, 1),+ 'z_score': round(z_score, 2),+ 'direction': 'high' if z_score > 0 else 'low',+ })++ return anomalies++ # ==================== Capacity Forecasting ====================++ @staticmethod+ def forecast_capacity(server_id: str, metric: str = 'disk') -> Dict:+ """Linear regression forecast for when a metric will hit 90% and 100%."""+ col_name = METRIC_COLUMNS.get(metric, 'disk_percent')+ cutoff = datetime.utcnow() - timedelta(days=30)++ records = ServerMetrics.query.filter(+ ServerMetrics.server_id == server_id,+ ServerMetrics.timestamp >= cutoff+ ).order_by(ServerMetrics.timestamp.asc()).all()++ values = []+ for r in records:+ val = getattr(r, col_name)+ if val is not None:+ # Convert timestamp to days from first record+ values.append((r.timestamp, val))++ if len(values) < 48: # Need at least 2 days of data+ return {+ 'server_id': server_id,+ 'metric': metric,+ 'error': 'Insufficient data (need at least 2 days of metrics)',+ 'data_points': len(values)+ }++ # Aggregate to daily averages for regression+ daily = {}+ for ts, val in values:+ day_key = ts.strftime('%Y-%m-%d')+ daily.setdefault(day_key, []).append(val)+ daily_avgs = [(k, sum(v) / len(v)) for k, v in sorted(daily.items())]++ if len(daily_avgs) < 2:+ return {+ 'server_id': server_id,+ 'metric': metric,+ 'error': 'Insufficient daily data points',+ 'data_points': len(daily_avgs)+ }++ # Simple linear regression: y = mx + b+ n = len(daily_avgs)+ x_vals = list(range(n))+ y_vals = [v for _, v in daily_avgs]++ x_mean = sum(x_vals) / n+ y_mean = sum(y_vals) / n++ numerator = sum((x - x_mean) * (y - y_mean) for x, y in zip(x_vals, y_vals))+ denominator = sum((x - x_mean) ** 2 for x in x_vals)++ if denominator == 0:+ return {'server_id': server_id, 'metric': metric, 'trend': 'flat', 'growth_rate_per_day': 0}++ slope = numerator / denominator # % per day+ intercept = y_mean - slope * x_mean++ current = y_vals[-1]++ # Predict when metric reaches 90% and 100%+ predictions = {}+ for target in [90, 100]:+ if current >= target:+ predictions[f'days_to_{target}pct'] = 0+ predictions[f'date_{target}pct'] = 'already exceeded'+ elif slope <= 0:+ predictions[f'days_to_{target}pct'] = None+ predictions[f'date_{target}pct'] = 'never (decreasing trend)'+ else:+ days_needed = (target - current) / slope+ target_date = datetime.utcnow() + timedelta(days=days_needed)+ predictions[f'days_to_{target}pct'] = round(days_needed, 1)+ predictions[f'date_{target}pct'] = target_date.strftime('%Y-%m-%d')++ # Generate trend line data for charting+ trend_data = []+ for i, (day, avg) in enumerate(daily_avgs):+ trend_data.append({+ 'date': day,+ 'actual': round(avg, 1),+ 'trend': round(intercept + slope * i, 1)+ })++ # Extend trend 14 days into the future+ for i in range(1, 15):+ future_day = (datetime.utcnow() + timedelta(days=i)).strftime('%Y-%m-%d')+ predicted = intercept + slope * (n - 1 + i)+ trend_data.append({+ 'date': future_day,+ 'actual': None,+ 'trend': round(min(predicted, 100), 1)+ })++ return {+ 'server_id': server_id,+ 'metric': metric,+ 'current_value': round(current, 1),+ 'growth_rate_per_day': round(slope, 3),+ 'trend': 'increasing' if slope > 0.1 else ('decreasing' if slope < -0.1 else 'stable'),+ 'predictions': predictions,+ 'trend_data': trend_data,+ 'data_points': len(values),+ 'daily_samples': len(daily_avgs),+ }++ # ==================== Fleet Search ====================++ @staticmethod+ def search_fleet(query: str, search_type: str = 'any') -> List[Dict]:+ """Search across all servers for containers, services, or ports."""+ query_lower = query.lower()+ results = []++ # Search server names and hostnames+ if search_type in ('any', 'server'):+ servers = Server.query.filter(+ (Server.name.ilike(f'%{query}%')) |+ (Server.hostname.ilike(f'%{query}%')) |+ (Server.ip_address.ilike(f'%{query}%'))+ ).all()+ for s in servers:+ results.append({+ 'server_id': s.id,+ 'server_name': s.name,+ 'match_type': 'server',+ 'match_name': s.name,+ 'match_detail': f'{s.hostname} ({s.ip_address})',+ 'status': s.status,+ })++ # Search containers via connected agents+ if search_type in ('any', 'container'):+ connected = agent_registry.get_connected_servers()+ for server_id in connected:+ server = Server.query.get(server_id)+ if not server:+ continue++ # Check cached metrics extra data for container info+ latest = ServerMetrics.query.filter_by(+ server_id=server_id+ ).order_by(ServerMetrics.timestamp.desc()).first()++ if latest and latest.extra:+ containers = latest.extra.get('containers', [])+ for container in containers:+ name = container.get('name', '')+ image = container.get('image', '')+ if query_lower in name.lower() or query_lower in image.lower():+ results.append({+ 'server_id': server_id,+ 'server_name': server.name,+ 'match_type': 'container',+ 'match_name': name,+ 'match_detail': image,+ 'status': container.get('status', 'unknown'),+ })++ # Search tags+ if search_type in ('any', 'tag'):+ all_servers = Server.query.all()+ for s in all_servers:+ if s.tags:+ for tag in s.tags:+ if query_lower in tag.lower():+ results.append({+ 'server_id': s.id,+ 'server_name': s.name,+ 'match_type': 'tag',+ 'match_name': tag,+ 'match_detail': f'Tag on {s.name}',+ 'status': s.status,+ })++ return results++ # ==================== Prometheus Export ====================++ @staticmethod+ def get_prometheus_metrics() -> str:+ """Generate Prometheus exposition format metrics for all servers."""+ lines = []+ servers = Server.query.all()++ metrics_defs = [+ ('serverkit_cpu_percent', 'CPU usage percentage', 'cpu_percent'),+ ('serverkit_memory_percent', 'Memory usage percentage', 'memory_percent'),+ ('serverkit_disk_percent', 'Disk usage percentage', 'disk_percent'),+ ('serverkit_containers_running', 'Number of running containers', 'container_running'),+ ]++ for metric_name, help_text, col_name in metrics_defs:+ lines.append(f'# HELP {metric_name} {help_text}')+ lines.append(f'# TYPE {metric_name} gauge')++ for server in servers:+ latest = ServerMetrics.query.filter_by(+ server_id=server.id+ ).order_by(ServerMetrics.timestamp.desc()).first()++ if latest:+ val = getattr(latest, col_name)+ if val is not None:+ safe_name = server.name.replace('"', '\\"')+ lines.append(+ f'{metric_name}{{server="{safe_name}",server_id="{server.id}"}} {val}'+ )+ lines.append('')++ # Server status (1 = online, 0 = offline)+ lines.append('# HELP serverkit_server_up Server online status')+ lines.append('# TYPE serverkit_server_up gauge')+ for server in servers:+ val = 1 if server.status == 'online' else 0+ safe_name = server.name.replace('"', '\\"')+ lines.append(+ f'serverkit_server_up{{server="{safe_name}",server_id="{server.id}"}} {val}'+ )++ return '\n'.join(lines) + '\n'++ # ==================== CSV Export ====================++ @staticmethod+ def export_metrics_csv(server_ids: List[str], metric: str = 'cpu', period: str = '24h') -> str:+ """Export metrics as CSV string."""+ hours_back = {'1h': 1, '6h': 6, '24h': 24, '7d': 168, '30d': 720}.get(period, 24)+ cutoff = datetime.utcnow() - timedelta(hours=hours_back)+ col_name = METRIC_COLUMNS.get(metric, 'cpu_percent')++ output = io.StringIO()+ writer = csv.writer(output)+ writer.writerow(['timestamp', 'server_id', 'server_name', metric])++ for server_id in server_ids:+ server = Server.query.get(server_id)+ if not server:+ continue++ records = ServerMetrics.query.filter(+ ServerMetrics.server_id == server_id,+ ServerMetrics.timestamp >= cutoff+ ).order_by(ServerMetrics.timestamp.asc()).all()++ for r in records:+ val = getattr(r, col_name)+ if val is not None:+ writer.writerow([+ r.timestamp.isoformat(),+ server_id,+ server.name,+ round(val, 2)+ ])++ return output.getvalue()+++def _downsample_simple(records, interval_minutes):+ """Simple downsampling by picking one record per interval."""+ if not records:+ return []++ result = [records[0]]+ last_ts = records[0].timestamp++ for r in records[1:]:+ if (r.timestamp - last_ts).total_seconds() >= interval_minutes * 60:+ result.append(r)+ last_ts = r.timestamp++ return result+++fleet_monitor_service = FleetMonitorService()diff --git a/backend/app/services/git_deploy_service.py b/backend/app/services/git_deploy_service.pyindex 876e81a1..86908e06 100644--- a/backend/app/services/git_deploy_service.py+++ b/backend/app/services/git_deploy_service.py@@ -447,8 +447,7 @@ class GitDeployService:"""Run a deployment script."""try:result = subprocess.run(- script,- shell=True,+ ['bash', '-c', script],cwd=cwd,capture_output=True,text=True,diff --git a/backend/app/services/git_service.py b/backend/app/services/git_service.pyindex 03491df5..82653638 100644--- a/backend/app/services/git_service.py+++ b/backend/app/services/git_service.py@@ -286,8 +286,7 @@ class GitService:"""Run a deployment script."""try:result = subprocess.run(- script,- shell=True,+ ['bash', '-c', script],cwd=working_dir,capture_output=True,text=True,diff --git a/backend/app/services/nginx_service.py b/backend/app/services/nginx_service.pyindex 3ec1aaa3..3fe20bd4 100644--- a/backend/app/services/nginx_service.py+++ b/backend/app/services/nginx_service.py@@ -7,6 +7,19 @@ from pathlib import Pathfrom app.utils.system import ServiceControl, run_privileged, is_command_available+def _validate_domain(domain: str) -> bool:+ """Validate domain name to prevent nginx config injection."""+ return bool(re.match(r'^(?:[a-z0-9](?:[a-z0-9\-]{0,61}[a-z0-9])?\.)*[a-z0-9](?:[a-z0-9\-]{0,61}[a-z0-9])?$', domain, re.IGNORECASE))+++def _validate_path(path: str) -> bool:+ """Validate filesystem path for nginx config."""+ # Block path traversal and special characters+ if '..' in path or '\n' in path or '\r' in path or ';' in path:+ return False+ return bool(re.match(r'^/[a-zA-Z0-9/_\-\.]+$', path))++class NginxService:"""Service for Nginx configuration management."""@@ -415,6 +428,15 @@ server {{if not domains:return {'success': False, 'error': 'At least one domain is required'}+ # Validate all domains+ for domain in domains:+ if not _validate_domain(domain):+ return {'success': False, 'error': f'Invalid domain name: {domain}'}++ # Validate root_path if provided+ if root_path and not _validate_path(root_path):+ return {'success': False, 'error': f'Invalid root path: {root_path}'}+domains_str = ' '.join(domains)# Select template based on app typediff --git a/backend/app/services/python_service.py b/backend/app/services/python_service.pyindex 3d96732d..54b7c10c 100644--- a/backend/app/services/python_service.py+++ b/backend/app/services/python_service.py@@ -552,8 +552,7 @@ application = get_wsgi_application()env.update(env_vars)result = subprocess.run(- command,- shell=True,+ ['bash', '-c', command],cwd=app_path,env=env,capture_output=True,diff --git a/backend/app/services/settings_service.py b/backend/app/services/settings_service.pyindex 10e36bd2..2515ccc6 100644--- a/backend/app/services/settings_service.py+++ b/backend/app/services/settings_service.py@@ -120,15 +120,15 @@ class SettingsService:@staticmethoddef needs_setup():- """Check if initial setup is required."""- # If no users exist, setup is needed+ """Check if initial setup is needed."""+ from app.models.user import Useruser_count = User.query.count()if user_count == 0:return True-- # Check setup_completed settingsetup_completed = SettingsService.get('setup_completed', False)- return not setup_completed+ if setup_completed:+ return False # Once completed, never re-enable without admin action+ return True@staticmethoddef complete_setup(user_id=None):diff --git a/backend/app/services/template_service.py b/backend/app/services/template_service.pyindex 68ddef25..9a360633 100644--- a/backend/app/services/template_service.py+++ b/backend/app/services/template_service.py@@ -808,8 +808,7 @@ class TemplateService:env.update(variables)result = subprocess.run(- script,- shell=True,+ ['bash', '-c', script],cwd=cwd,env=env,capture_output=True,diff --git a/backend/app/utils/crypto.py b/backend/app/utils/crypto.pyindex 1924758f..3544c7a7 100644--- a/backend/app/utils/crypto.py+++ b/backend/app/utils/crypto.py@@ -6,10 +6,14 @@ Provides encryption/decryption for sensitive data like API secrets.import osimport base64+import warnings+import loggingfrom cryptography.fernet import Fernet, InvalidTokenfrom cryptography.hazmat.primitives import hashesfrom cryptography.hazmat.primitives.kdf.pbkdf2 import PBKDF2HMAC+logger = logging.getLogger(__name__)+def get_encryption_key() -> bytes:"""@@ -26,6 +30,10 @@ def get_encryption_key() -> bytes:"""key = os.environ.get('SERVERKIT_ENCRYPTION_KEY')if not key:+ if os.environ.get('FLASK_ENV') == 'production':+ raise ValueError('CRITICAL: SERVERKIT_ENCRYPTION_KEY must be set in production')+ logger.warning('SECURITY WARNING: Using derived development encryption key. Set SERVERKIT_ENCRYPTION_KEY for production.')+ warnings.warn('Using derived development encryption key - not suitable for production')# In development, use a default key (NOT for production!)# This allows the system to work without explicit configurationdefault_key = "DEV_ONLY_NOT_SECURE_CHANGE_IN_PRODUCTION_KEY"diff --git a/backend/config.py b/backend/config.pyindex 4df2898c..6afd1073 100644--- a/backend/config.py+++ b/backend/config.py@@ -1,5 +1,6 @@import osimport sys+import warningsfrom datetime import timedelta# Default insecure keys that must be changed in production@@ -20,7 +21,7 @@ class Config:# JWTJWT_SECRET_KEY = os.environ.get('JWT_SECRET_KEY', 'jwt-secret-key-change-in-production')- JWT_ACCESS_TOKEN_EXPIRES = timedelta(hours=1)+ JWT_ACCESS_TOKEN_EXPIRES = timedelta(minutes=15)JWT_REFRESH_TOKEN_EXPIRES = timedelta(days=30)# CORS - Allow both dev server and Flask server@@ -30,6 +31,13 @@ class Config:class DevelopmentConfig(Config):DEBUG = True+ @classmethod+ def init_app(cls, app):+ if app.config.get('SECRET_KEY') == 'dev-secret-key-change-in-production':+ warnings.warn('WARNING: Using default SECRET_KEY. Change before deploying.')+ if app.config.get('JWT_SECRET_KEY') == 'jwt-secret-key-change-in-production':+ warnings.warn('WARNING: Using default JWT_SECRET_KEY. Change before deploying.')+class TestingConfig(Config):"""Config for pytest and other automated tests."""@@ -55,6 +63,21 @@ class ProductionConfig(Config):print("Generate a secure key with: python -c \"import secrets; print(secrets.token_hex(32))\"", file=sys.stderr)sys.exit(1)+ @classmethod+ def init_app(cls, app):+ """Validate production configuration."""+ insecure_keys = ['dev-secret-key-change-in-production', 'jwt-secret-key-change-in-production']+ if app.config['SECRET_KEY'] in insecure_keys:+ raise ValueError('CRITICAL: SECRET_KEY must be changed for production deployment')+ if app.config['JWT_SECRET_KEY'] in insecure_keys:+ raise ValueError('CRITICAL: JWT_SECRET_KEY must be changed for production deployment')+ # Validate CORS origins+ cors_raw = os.environ.get('CORS_ORIGINS', '')+ cors_origins = [o.strip() for o in cors_raw.split(',') if o.strip()]+ if not cors_origins:+ raise ValueError('CORS_ORIGINS must be explicitly set in production')+ app.config['CORS_ORIGINS'] = cors_origins+config = {'development': DevelopmentConfig,diff --git a/frontend/src/App.jsx b/frontend/src/App.jsxindex 2ffbe0d1..b62d4aef 100644--- a/frontend/src/App.jsx+++ b/frontend/src/App.jsx@@ -32,6 +32,7 @@ import WorkflowBuilder from './pages/WorkflowBuilder';import Servers from './pages/Servers';import ServerDetail from './pages/ServerDetail';import AgentFleet from './pages/AgentFleet';+import FleetMonitor from './pages/FleetMonitor';import Downloads from './pages/Downloads';import WordPress from './pages/WordPress';import WordPressDetail from './pages/WordPressDetail';@@ -71,6 +72,8 @@ const PAGE_TITLES = {'/terminal': 'Terminal','/settings': 'Settings','/migrate': 'Database Migration',+ '/fleet': 'Agent Fleet',+ '/fleet-monitor': 'Fleet Monitor',};function PageTitleUpdater() {@@ -206,6 +209,7 @@ function AppRoutes() {<Route path="servers/:id" element={<ServerDetail />} /><Route path="servers/:id/:tab" element={<ServerDetail />} /><Route path="fleet" element={<AgentFleet />} />+ <Route path="fleet-monitor" element={<FleetMonitor />} /><Route path="downloads" element={<Downloads />} /><Route path="firewall" element={<Navigate to="/security/firewall" replace />} /><Route path="git" element={<Git />} />diff --git a/frontend/src/components/sidebarItems.js b/frontend/src/components/sidebarItems.jsindex 135bde6e..2372b451 100644--- a/frontend/src/components/sidebarItems.js+++ b/frontend/src/components/sidebarItems.js@@ -35,6 +35,13 @@ export const SIDEBAR_ITEMS = [category: 'infrastructure',icon: '<path d="M17 21v-2a4 4 0 0 0-4-4H5a4 4 0 0 0-4 4v2"/><circle cx="9" cy="7" r="4"/><path d="M23 21v-2a4 4 0 0 0-3-3.87"/><path d="M16 3.13a4 4 0 0 1 0 7.75"/>'},+ {+ id: 'fleet-monitor',+ label: 'Fleet Monitor',+ route: '/fleet-monitor',+ category: 'infrastructure',+ icon: '<path d="M22 12h-4l-3 9L9 3l-3 9H2"/>'+ },{id: 'domains',label: 'Domains & Sites',diff --git a/frontend/src/pages/FleetMonitor.jsx b/frontend/src/pages/FleetMonitor.jsxnew file mode 100644index 00000000..ce740690--- /dev/null+++ b/frontend/src/pages/FleetMonitor.jsx@@ -0,0 +1,846 @@+import React, { useState, useEffect, useMemo } from 'react';+import {+ Activity,+ AlertTriangle,+ BarChart3,+ CheckCircle,+ ChevronDown,+ Download,+ Eye,+ Layers,+ RefreshCw,+ Search,+ Server,+ TrendingUp,+ XCircle,+ Zap+} from 'lucide-react';+import {+ LineChart, Line, AreaChart, Area, XAxis, YAxis,+ CartesianGrid, Tooltip, ResponsiveContainer, Legend+} from 'recharts';+import api from '../services/api';+import { useToast } from '../contexts/ToastContext';++const CHART_COLORS = [+ '#6366f1', '#ec4899', '#14b8a6', '#f59e0b',+ '#8b5cf6', '#06b6d4', '#ef4444', '#22c55e',+ '#a855f7', '#f97316'+];++const METRIC_LABELS = {+ cpu: 'CPU %',+ memory: 'Memory %',+ disk: 'Disk %',+ network_rx: 'Network RX',+ network_tx: 'Network TX'+};++const heatColor = (value) => {+ if (value == null) return 'var(--card-bg)';+ if (value >= 90) return '#ef4444';+ if (value >= 75) return '#f97316';+ if (value >= 50) return '#eab308';+ return '#22c55e';+};++const FleetMonitor = () => {+ const [activeTab, setActiveTab] = useState('overview');+ const [loading, setLoading] = useState(true);+ const { addToast } = useToast();++ // Overview state+ const [heatmapData, setHeatmapData] = useState([]);++ // Comparison state+ const [servers, setServers] = useState([]);+ const [selectedServers, setSelectedServers] = useState([]);+ const [compMetric, setCompMetric] = useState('cpu');+ const [compPeriod, setCompPeriod] = useState('24h');+ const [compData, setCompData] = useState(null);++ // Alerts state+ const [alerts, setAlerts] = useState([]);+ const [thresholds, setThresholds] = useState([]);+ const [alertFilter, setAlertFilter] = useState('active');+ const [newThreshold, setNewThreshold] = useState({+ metric: 'cpu', warning_threshold: 80, critical_threshold: 95, duration_seconds: 300+ });++ // Anomaly/Forecast state+ const [anomalies, setAnomalies] = useState([]);+ const [forecastServer, setForecastServer] = useState('');+ const [forecastMetric, setForecastMetric] = useState('disk');+ const [forecast, setForecast] = useState(null);++ // Search state+ const [searchQuery, setSearchQuery] = useState('');+ const [searchType, setSearchType] = useState('any');+ const [searchResults, setSearchResults] = useState([]);++ useEffect(() => {+ loadServers();+ }, []);++ useEffect(() => {+ fetchTabData();+ }, [activeTab, alertFilter]);++ const loadServers = async () => {+ try {+ const data = await api.getServers();+ const list = data.servers || data;+ setServers(Array.isArray(list) ? list : []);+ } catch (e) {+ console.error('Error loading servers:', e);+ }+ };++ const fetchTabData = async () => {+ setLoading(true);+ try {+ if (activeTab === 'overview') {+ const data = await api.getFleetHeatmap();+ setHeatmapData(data);+ } else if (activeTab === 'alerts') {+ const [alertData, thresholdData] = await Promise.all([+ api.getFleetAlerts({ status: alertFilter !== 'all' ? alertFilter : undefined }),+ api.getFleetThresholds()+ ]);+ setAlerts(alertData);+ setThresholds(thresholdData);+ } else if (activeTab === 'anomalies') {+ const data = await api.getFleetAnomalies();+ setAnomalies(data);+ }+ } catch (e) {+ console.error('Error fetching tab data:', e);+ } finally {+ setLoading(false);+ }+ };++ const loadComparison = async () => {+ if (selectedServers.length === 0) return;+ setLoading(true);+ try {+ const data = await api.getFleetComparison(selectedServers, compMetric, compPeriod);+ setCompData(data);+ } catch (e) {+ addToast('Error', 'Failed to load comparison data', 'error');+ } finally {+ setLoading(false);+ }+ };++ const loadForecast = async (serverId) => {+ try {+ const data = await api.getCapacityForecast(serverId || forecastServer, forecastMetric);+ setForecast(data);+ } catch (e) {+ addToast('Error', 'Failed to load forecast', 'error');+ }+ };++ const handleSearch = async () => {+ if (searchQuery.length < 2) return;+ setLoading(true);+ try {+ const data = await api.searchFleet(searchQuery, searchType);+ setSearchResults(data);+ } catch (e) {+ addToast('Error', 'Search failed', 'error');+ } finally {+ setLoading(false);+ }+ };++ const acknowledgeAlert = async (id) => {+ try {+ await api.acknowledgeFleetAlert(id);+ fetchTabData();+ } catch (e) {+ addToast('Error', 'Failed to acknowledge alert', 'error');+ }+ };++ const resolveAlert = async (id) => {+ try {+ await api.resolveFleetAlert(id);+ fetchTabData();+ } catch (e) {+ addToast('Error', 'Failed to resolve alert', 'error');+ }+ };++ const saveThreshold = async () => {+ try {+ await api.createFleetThreshold(newThreshold);+ addToast('Success', 'Threshold saved', 'success');+ fetchTabData();+ } catch (e) {+ addToast('Error', 'Failed to save threshold', 'error');+ }+ };++ const deleteThreshold = async (id) => {+ try {+ await api.deleteFleetThreshold(id);+ fetchTabData();+ } catch (e) {+ addToast('Error', 'Failed to delete threshold', 'error');+ }+ };++ const toggleServer = (id) => {+ setSelectedServers(prev =>+ prev.includes(id) ? prev.filter(s => s !== id) : [...prev, id]+ );+ };++ // Merge comparison series into recharts-compatible format+ const compChartData = useMemo(() => {+ if (!compData?.series?.length) return [];+ const timeMap = {};+ compData.series.forEach((s, idx) => {+ s.data.forEach(point => {+ if (!timeMap[point.timestamp]) {+ timeMap[point.timestamp] = { timestamp: point.timestamp };+ }+ timeMap[point.timestamp][s.name] = point.value;+ });+ });+ return Object.values(timeMap).sort((a, b) => a.timestamp.localeCompare(b.timestamp));+ }, [compData]);++ const exportCsv = async () => {+ if (selectedServers.length === 0) return;+ try {+ const blob = await api.exportFleetCsv(selectedServers, compMetric, compPeriod);+ const url = URL.createObjectURL(blob);+ const a = document.createElement('a');+ a.href = url;+ a.download = `fleet_${compMetric}_${compPeriod}.csv`;+ a.click();+ URL.revokeObjectURL(url);+ } catch (e) {+ addToast('Error', 'Export failed', 'error');+ }+ };++ return (+ <div className="page-container">+ <div className="page-header">+ <div className="header-info">+ <h1>Fleet Monitor</h1>+ <p>Cross-server monitoring, alerts, and capacity planning.</p>+ </div>+ <div className="header-actions">+ <button className="btn btn-primary" onClick={fetchTabData} disabled={loading}>+ <RefreshCw size={18} className={loading ? 'animate-spin' : ''} />+ Refresh+ </button>+ </div>+ </div>++ <div className="tabs-container">+ <div className="tabs">+ {[+ { key: 'overview', icon: Layers, label: 'Overview' },+ { key: 'comparison', icon: BarChart3, label: 'Comparison' },+ { key: 'alerts', icon: AlertTriangle, label: 'Alerts' },+ { key: 'anomalies', icon: TrendingUp, label: 'Anomalies & Forecast' },+ { key: 'search', icon: Search, label: 'Fleet Search' },+ ].map(tab => (+ <button+ key={tab.key}+ className={`tab ${activeTab === tab.key ? 'active' : ''}`}+ onClick={() => setActiveTab(tab.key)}+ >+ <tab.icon size={18} />+ {tab.label}+ </button>+ ))}+ </div>+ </div>++ <div className="tab-content mt-6">+ {/* ==================== Overview Heatmap ==================== */}+ {activeTab === 'overview' && (+ <div className="space-y-6">+ <div className="card">+ <div className="card-header">+ <h2>Server Health Heatmap</h2>+ </div>+ <div className="card-body">+ {heatmapData.length > 0 ? (+ <div className="fleet-heatmap">+ <div className="fleet-heatmap__header">+ <div className="fleet-heatmap__label">Server</div>+ <div className="fleet-heatmap__label">CPU</div>+ <div className="fleet-heatmap__label">Memory</div>+ <div className="fleet-heatmap__label">Disk</div>+ <div className="fleet-heatmap__label">Containers</div>+ <div className="fleet-heatmap__label">Status</div>+ </div>+ {heatmapData.map(server => (+ <div key={server.id} className="fleet-heatmap__row">+ <div className="fleet-heatmap__server">+ <Server size={14} />+ <span>{server.name}</span>+ {server.group_name && (+ <span className="text-xs text-gray-400 ml-1">({server.group_name})</span>+ )}+ </div>+ <div+ className="fleet-heatmap__cell"+ style={{ backgroundColor: heatColor(server.cpu), color: server.cpu >= 75 ? '#fff' : 'inherit' }}+ title={`CPU: ${server.cpu ?? 'N/A'}%`}+ >+ {server.cpu != null ? `${server.cpu}%` : '-'}+ </div>+ <div+ className="fleet-heatmap__cell"+ style={{ backgroundColor: heatColor(server.memory), color: server.memory >= 75 ? '#fff' : 'inherit' }}+ title={`Memory: ${server.memory ?? 'N/A'}%`}+ >+ {server.memory != null ? `${server.memory}%` : '-'}+ </div>+ <div+ className="fleet-heatmap__cell"+ style={{ backgroundColor: heatColor(server.disk), color: server.disk >= 75 ? '#fff' : 'inherit' }}+ title={`Disk: ${server.disk ?? 'N/A'}%`}+ >+ {server.disk != null ? `${server.disk}%` : '-'}+ </div>+ <div className="fleet-heatmap__cell">+ {server.containers ?? '-'}+ </div>+ <div className="fleet-heatmap__cell">+ <span className={`badge ${server.status === 'online' ? 'badge-success' : 'badge-error'}`}>+ {server.status}+ </span>+ </div>+ </div>+ ))}+ </div>+ ) : (+ <div className="py-12 text-center text-gray-500">+ <Server size={48} className="mx-auto text-gray-300 mb-4" />+ <p>No servers registered. Add servers to see the fleet heatmap.</p>+ </div>+ )}+ </div>+ </div>+ <div className="flex gap-3 items-center text-sm text-gray-500">+ <span>Legend:</span>+ <span className="flex items-center gap-1"><span className="inline-block w-3 h-3 rounded" style={{ backgroundColor: '#22c55e' }}></span> 0-50%</span>+ <span className="flex items-center gap-1"><span className="inline-block w-3 h-3 rounded" style={{ backgroundColor: '#eab308' }}></span> 50-75%</span>+ <span className="flex items-center gap-1"><span className="inline-block w-3 h-3 rounded" style={{ backgroundColor: '#f97316' }}></span> 75-90%</span>+ <span className="flex items-center gap-1"><span className="inline-block w-3 h-3 rounded" style={{ backgroundColor: '#ef4444' }}></span> 90-100%</span>+ </div>+ </div>+ )}++ {/* ==================== Comparison ==================== */}+ {activeTab === 'comparison' && (+ <div className="space-y-6">+ <div className="card">+ <div className="card-header flex justify-between items-center">+ <h2>Server Comparison</h2>+ <div className="flex gap-2">+ <button className="btn btn-sm btn-ghost" onClick={exportCsv} disabled={selectedServers.length === 0}>+ <Download size={14} /> Export CSV+ </button>+ </div>+ </div>+ <div className="card-body">+ <div className="grid grid-cols-1 lg:grid-cols-4 gap-4 mb-6">+ <div className="lg:col-span-2">+ <label className="text-sm font-medium mb-1 block">Select Servers (click to toggle)</label>+ <div className="flex flex-wrap gap-2 max-h-24 overflow-y-auto p-2 border rounded">+ {servers.map(s => (+ <button+ key={s.id}+ className={`btn btn-sm ${selectedServers.includes(s.id) ? 'btn-primary' : 'btn-ghost'}`}+ onClick={() => toggleServer(s.id)}+ >+ {s.name}+ </button>+ ))}+ </div>+ </div>+ <div>+ <label className="text-sm font-medium mb-1 block">Metric</label>+ <select className="form-select w-full" value={compMetric} onChange={e => setCompMetric(e.target.value)}>+ {Object.entries(METRIC_LABELS).map(([k, v]) => (+ <option key={k} value={k}>{v}</option>+ ))}+ </select>+ </div>+ <div>+ <label className="text-sm font-medium mb-1 block">Period</label>+ <div className="flex gap-1">+ {['1h', '6h', '24h', '7d'].map(p => (+ <button+ key={p}+ className={`btn btn-sm ${compPeriod === p ? 'btn-primary' : 'btn-ghost'}`}+ onClick={() => setCompPeriod(p)}+ >+ {p}+ </button>+ ))}+ </div>+ </div>+ </div>+ <div className="flex justify-end mb-4">+ <button+ className="btn btn-primary"+ onClick={loadComparison}+ disabled={selectedServers.length === 0 || loading}+ >+ <BarChart3 size={16} /> Compare+ </button>+ </div>++ {compChartData.length > 0 && compData?.series && (+ <ResponsiveContainer width="100%" height={400}>+ <AreaChart data={compChartData}>+ <CartesianGrid strokeDasharray="3 3" stroke="var(--border-color)" />+ <XAxis+ dataKey="timestamp"+ tickFormatter={v => new Date(v).toLocaleTimeString([], { hour: '2-digit', minute: '2-digit' })}+ stroke="var(--text-secondary)"+ fontSize={12}+ />+ <YAxis stroke="var(--text-secondary)" fontSize={12} unit="%" />+ <Tooltip+ labelFormatter={v => new Date(v).toLocaleString()}+ contentStyle={{ backgroundColor: 'var(--card-bg)', border: '1px solid var(--border-color)' }}+ />+ <Legend />+ {compData.series.map((s, i) => (+ <Area+ key={s.server_id}+ type="monotone"+ dataKey={s.name}+ stroke={CHART_COLORS[i % CHART_COLORS.length]}+ fill={CHART_COLORS[i % CHART_COLORS.length]}+ fillOpacity={0.1}+ strokeWidth={2}+ dot={false}+ />+ ))}+ </AreaChart>+ </ResponsiveContainer>+ )}++ {selectedServers.length === 0 && (+ <div className="py-12 text-center text-gray-500">+ <BarChart3 size={48} className="mx-auto text-gray-300 mb-4" />+ <p>Select servers above to compare their metrics.</p>+ </div>+ )}+ </div>+ </div>+ </div>+ )}++ {/* ==================== Alerts ==================== */}+ {activeTab === 'alerts' && (+ <div className="space-y-6">+ <div className="card">+ <div className="card-header flex justify-between items-center">+ <h2>Metric Alerts</h2>+ <div className="flex gap-1">+ {['active', 'acknowledged', 'resolved', 'all'].map(f => (+ <button+ key={f}+ className={`btn btn-sm ${alertFilter === f ? 'btn-primary' : 'btn-ghost'}`}+ onClick={() => setAlertFilter(f)}+ >+ {f.charAt(0).toUpperCase() + f.slice(1)}+ </button>+ ))}+ </div>+ </div>+ {alerts.length > 0 ? (+ <div className="overflow-x-auto">+ <table className="table">+ <thead>+ <tr>+ <th>Server</th>+ <th>Metric</th>+ <th>Value</th>+ <th>Threshold</th>+ <th>Severity</th>+ <th>Status</th>+ <th>Time</th>+ <th>Actions</th>+ </tr>+ </thead>+ <tbody>+ {alerts.map(a => (+ <tr key={a.id}>+ <td className="font-semibold">{a.server_name}</td>+ <td>{METRIC_LABELS[a.metric] || a.metric}</td>+ <td className="font-mono">{a.value}%</td>+ <td className="font-mono">{a.threshold}%</td>+ <td>+ <span className={`badge ${a.severity === 'critical' ? 'badge-error' : 'badge-warning'}`}>+ {a.severity}+ </span>+ </td>+ <td>+ <span className={`badge ${a.status === 'active' ? 'badge-error' : a.status === 'acknowledged' ? 'badge-warning' : 'badge-success'}`}>+ {a.status}+ </span>+ </td>+ <td className="text-sm">{new Date(a.created_at).toLocaleString()}</td>+ <td className="actions">+ {a.status === 'active' && (+ <>+ <button className="btn btn-sm btn-ghost" onClick={() => acknowledgeAlert(a.id)}>+ <Eye size={14} /> Ack+ </button>+ <button className="btn btn-sm btn-ghost" onClick={() => resolveAlert(a.id)}>+ <CheckCircle size={14} />+ </button>+ </>+ )}+ {a.status === 'acknowledged' && (+ <button className="btn btn-sm btn-ghost" onClick={() => resolveAlert(a.id)}>+ <CheckCircle size={14} /> Resolve+ </button>+ )}+ </td>+ </tr>+ ))}+ </tbody>+ </table>+ </div>+ ) : (+ <div className="card-body py-8 text-center text-gray-500">+ <CheckCircle size={36} className="mx-auto text-gray-300 mb-3" />+ <p>No {alertFilter !== 'all' ? alertFilter : ''} alerts.</p>+ </div>+ )}+ </div>++ <div className="grid grid-cols-1 lg:grid-cols-2 gap-6">+ <div className="card">+ <div className="card-header"><h2>Alert Thresholds</h2></div>+ {thresholds.length > 0 ? (+ <div className="overflow-x-auto">+ <table className="table">+ <thead>+ <tr>+ <th>Scope</th>+ <th>Metric</th>+ <th>Warning</th>+ <th>Critical</th>+ <th>Duration</th>+ <th></th>+ </tr>+ </thead>+ <tbody>+ {thresholds.map(t => (+ <tr key={t.id}>+ <td>{t.server_name || 'Global'}</td>+ <td>{METRIC_LABELS[t.metric] || t.metric}</td>+ <td className="text-yellow-600">{t.warning_threshold}%</td>+ <td className="text-red-600">{t.critical_threshold}%</td>+ <td>{t.duration_seconds}s</td>+ <td>+ <button className="btn btn-ghost btn-sm text-red-600" onClick={() => deleteThreshold(t.id)}>+ <XCircle size={14} />+ </button>+ </td>+ </tr>+ ))}+ </tbody>+ </table>+ </div>+ ) : (+ <div className="card-body py-6 text-center text-gray-500 text-sm">+ No thresholds configured. Add one to start receiving metric alerts.+ </div>+ )}+ </div>++ <div className="card">+ <div className="card-header"><h2>Add Threshold</h2></div>+ <div className="card-body space-y-4">+ <div className="form-group">+ <label>Server (leave empty for global)</label>+ <select+ className="form-select w-full"+ value={newThreshold.server_id || ''}+ onChange={e => setNewThreshold(p => ({ ...p, server_id: e.target.value || undefined }))}+ >+ <option value="">Global Default</option>+ {servers.map(s => <option key={s.id} value={s.id}>{s.name}</option>)}+ </select>+ </div>+ <div className="grid grid-cols-2 gap-4">+ <div className="form-group">+ <label>Metric</label>+ <select className="form-select w-full" value={newThreshold.metric}+ onChange={e => setNewThreshold(p => ({ ...p, metric: e.target.value }))}>+ <option value="cpu">CPU</option>+ <option value="memory">Memory</option>+ <option value="disk">Disk</option>+ </select>+ </div>+ <div className="form-group">+ <label>Duration (seconds)</label>+ <input type="number" className="form-input w-full" value={newThreshold.duration_seconds}+ onChange={e => setNewThreshold(p => ({ ...p, duration_seconds: parseInt(e.target.value) || 300 }))} />+ </div>+ </div>+ <div className="grid grid-cols-2 gap-4">+ <div className="form-group">+ <label>Warning (%)</label>+ <input type="number" className="form-input w-full" value={newThreshold.warning_threshold}+ onChange={e => setNewThreshold(p => ({ ...p, warning_threshold: parseFloat(e.target.value) || 80 }))} />+ </div>+ <div className="form-group">+ <label>Critical (%)</label>+ <input type="number" className="form-input w-full" value={newThreshold.critical_threshold}+ onChange={e => setNewThreshold(p => ({ ...p, critical_threshold: parseFloat(e.target.value) || 95 }))} />+ </div>+ </div>+ <button className="btn btn-primary w-full" onClick={saveThreshold}>+ Save Threshold+ </button>+ </div>+ </div>+ </div>+ </div>+ )}++ {/* ==================== Anomalies & Forecast ==================== */}+ {activeTab === 'anomalies' && (+ <div className="space-y-6">+ <div className="card">+ <div className="card-header"><h2>Anomaly Detection</h2></div>+ {anomalies.length > 0 ? (+ <div className="overflow-x-auto">+ <table className="table">+ <thead>+ <tr>+ <th>Server</th>+ <th>Metric</th>+ <th>Current</th>+ <th>Baseline (mean)</th>+ <th>Std Dev</th>+ <th>Z-Score</th>+ <th>Direction</th>+ </tr>+ </thead>+ <tbody>+ {anomalies.map((a, i) => (+ <tr key={i}>+ <td className="font-semibold">{a.server_name}</td>+ <td>{METRIC_LABELS[a.metric] || a.metric}</td>+ <td className="font-mono text-red-600">{a.current_value}%</td>+ <td className="font-mono">{a.mean}%</td>+ <td className="font-mono">{a.stddev}</td>+ <td className="font-mono font-bold">{a.z_score}</td>+ <td>+ <span className={`badge ${a.direction === 'high' ? 'badge-error' : 'badge-info'}`}>+ {a.direction === 'high' ? 'Unusually High' : 'Unusually Low'}+ </span>+ </td>+ </tr>+ ))}+ </tbody>+ </table>+ </div>+ ) : (+ <div className="card-body py-8 text-center text-gray-500">+ <CheckCircle size={36} className="mx-auto text-gray-300 mb-3" />+ <p>No anomalies detected. All metrics are within normal ranges.</p>+ </div>+ )}+ </div>++ <div className="card">+ <div className="card-header"><h2>Capacity Forecast</h2></div>+ <div className="card-body">+ <div className="flex gap-4 mb-6">+ <div className="form-group flex-1">+ <label>Server</label>+ <select className="form-select w-full" value={forecastServer}+ onChange={e => setForecastServer(e.target.value)}>+ <option value="">Select a server...</option>+ {servers.map(s => <option key={s.id} value={s.id}>{s.name}</option>)}+ </select>+ </div>+ <div className="form-group">+ <label>Metric</label>+ <select className="form-select w-full" value={forecastMetric}+ onChange={e => setForecastMetric(e.target.value)}>+ <option value="disk">Disk</option>+ <option value="memory">Memory</option>+ <option value="cpu">CPU</option>+ </select>+ </div>+ <div className="form-group flex items-end">+ <button className="btn btn-primary" onClick={() => loadForecast()} disabled={!forecastServer}>+ <TrendingUp size={16} /> Forecast+ </button>+ </div>+ </div>++ {forecast && !forecast.error && (+ <div className="space-y-6">+ <div className="grid grid-cols-1 md:grid-cols-3 gap-4">+ <div className="p-4 bg-gray-50 rounded-lg text-center">+ <div className="text-sm text-gray-500">Current</div>+ <div className="text-2xl font-bold">{forecast.current_value}%</div>+ </div>+ <div className="p-4 bg-gray-50 rounded-lg text-center">+ <div className="text-sm text-gray-500">Growth Rate</div>+ <div className="text-2xl font-bold">{forecast.growth_rate_per_day}%/day</div>+ </div>+ <div className="p-4 bg-gray-50 rounded-lg text-center">+ <div className="text-sm text-gray-500">Trend</div>+ <div className={`text-2xl font-bold ${forecast.trend === 'increasing' ? 'text-red-600' : forecast.trend === 'decreasing' ? 'text-green-600' : ''}`}>+ {forecast.trend}+ </div>+ </div>+ </div>++ {forecast.predictions && (+ <div className="grid grid-cols-2 gap-4">+ <div className={`p-4 rounded-lg border ${forecast.predictions.days_to_90pct === 0 ? 'border-red-300 bg-red-50' : 'border-yellow-300 bg-yellow-50'}`}>+ <div className="text-sm font-medium">Reaches 90%</div>+ <div className="text-lg font-bold">+ {forecast.predictions.date_90pct || 'N/A'}+ {forecast.predictions.days_to_90pct != null && forecast.predictions.days_to_90pct > 0 && (+ <span className="text-sm font-normal ml-2">({forecast.predictions.days_to_90pct} days)</span>+ )}+ </div>+ </div>+ <div className={`p-4 rounded-lg border ${forecast.predictions.days_to_100pct === 0 ? 'border-red-300 bg-red-50' : 'border-red-200 bg-red-50'}`}>+ <div className="text-sm font-medium">Reaches 100%</div>+ <div className="text-lg font-bold">+ {forecast.predictions.date_100pct || 'N/A'}+ {forecast.predictions.days_to_100pct != null && forecast.predictions.days_to_100pct > 0 && (+ <span className="text-sm font-normal ml-2">({forecast.predictions.days_to_100pct} days)</span>+ )}+ </div>+ </div>+ </div>+ )}++ {forecast.trend_data && (+ <ResponsiveContainer width="100%" height={300}>+ <LineChart data={forecast.trend_data}>+ <CartesianGrid strokeDasharray="3 3" stroke="var(--border-color)" />+ <XAxis dataKey="date" stroke="var(--text-secondary)" fontSize={11} />+ <YAxis stroke="var(--text-secondary)" fontSize={12} unit="%" domain={[0, 100]} />+ <Tooltip contentStyle={{ backgroundColor: 'var(--card-bg)', border: '1px solid var(--border-color)' }} />+ <Legend />+ <Line type="monotone" dataKey="actual" stroke="#6366f1" strokeWidth={2} dot={false} name="Actual" />+ <Line type="monotone" dataKey="trend" stroke="#ef4444" strokeWidth={2} strokeDasharray="5 5" dot={false} name="Trend" />+ </LineChart>+ </ResponsiveContainer>+ )}+ </div>+ )}++ {forecast?.error && (+ <div className="p-4 bg-yellow-50 border border-yellow-200 rounded text-yellow-700 text-sm">+ {forecast.error}+ </div>+ )}++ {!forecast && (+ <div className="py-8 text-center text-gray-500">+ <TrendingUp size={36} className="mx-auto text-gray-300 mb-3" />+ <p>Select a server and metric to see capacity predictions.</p>+ </div>+ )}+ </div>+ </div>+ </div>+ )}++ {/* ==================== Fleet Search ==================== */}+ {activeTab === 'search' && (+ <div className="card">+ <div className="card-header"><h2>Fleet Search</h2></div>+ <div className="card-body">+ <div className="flex gap-3 mb-6">+ <div className="flex-1">+ <input+ type="text"+ className="form-input w-full"+ placeholder="Search for servers, containers, tags..."+ value={searchQuery}+ onChange={e => setSearchQuery(e.target.value)}+ onKeyDown={e => e.key === 'Enter' && handleSearch()}+ />+ </div>+ <select className="form-select" value={searchType} onChange={e => setSearchType(e.target.value)}>+ <option value="any">All Types</option>+ <option value="server">Servers</option>+ <option value="container">Containers</option>+ <option value="tag">Tags</option>+ </select>+ <button className="btn btn-primary" onClick={handleSearch} disabled={searchQuery.length < 2}>+ <Search size={16} /> Search+ </button>+ </div>++ {searchResults.length > 0 ? (+ <div className="overflow-x-auto">+ <table className="table">+ <thead>+ <tr>+ <th>Server</th>+ <th>Type</th>+ <th>Match</th>+ <th>Detail</th>+ <th>Status</th>+ </tr>+ </thead>+ <tbody>+ {searchResults.map((r, i) => (+ <tr key={i}>+ <td className="font-semibold">{r.server_name}</td>+ <td><span className="badge badge-default">{r.match_type}</span></td>+ <td>{r.match_name}</td>+ <td className="text-sm text-gray-500">{r.match_detail}</td>+ <td>+ <span className={`badge ${r.status === 'online' || r.status === 'running' ? 'badge-success' : 'badge-error'}`}>+ {r.status}+ </span>+ </td>+ </tr>+ ))}+ </tbody>+ </table>+ </div>+ ) : searchQuery.length >= 2 && !loading ? (+ <div className="py-8 text-center text-gray-500">+ <Search size={36} className="mx-auto text-gray-300 mb-3" />+ <p>No results found for "{searchQuery}".</p>+ </div>+ ) : (+ <div className="py-8 text-center text-gray-500">+ <Search size={36} className="mx-auto text-gray-300 mb-3" />+ <p>Enter a search term to find servers, containers, or tags across your fleet.</p>+ </div>+ )}+ </div>+ </div>+ )}+ </div>+ </div>+ );+};++export default FleetMonitor;diff --git a/frontend/src/services/api.js b/frontend/src/services/api.jsindex 16de7e5f..d8b7542f 100644--- a/frontend/src/services/api.js+++ b/frontend/src/services/api.js@@ -3025,6 +3025,73 @@ class ApiService {return this.request(`/servers/fleet/diagnostics/${serverId}`);}+ // ========================================+ // Fleet Monitor (Cross-Server Monitoring) endpoints+ // ========================================+ async getFleetHeatmap(groupId) {+ const query = groupId ? `?group_id=${groupId}` : '';+ return this.request(`/fleet-monitor/heatmap${query}`);+ }++ async getFleetComparison(serverIds, metric, period) {+ const ids = serverIds.join(',');+ return this.request(`/fleet-monitor/comparison?ids=${ids}&metric=${metric}&period=${period}`);+ }++ async getFleetAlerts(params = {}) {+ const searchParams = new URLSearchParams();+ if (params.status) searchParams.append('status', params.status);+ if (params.severity) searchParams.append('severity', params.severity);+ if (params.server_id) searchParams.append('server_id', params.server_id);+ if (params.limit) searchParams.append('limit', params.limit);+ const query = searchParams.toString();+ return this.request(`/fleet-monitor/alerts${query ? '?' + query : ''}`);+ }++ async acknowledgeFleetAlert(alertId) {+ return this.request(`/fleet-monitor/alerts/${alertId}/acknowledge`, { method: 'POST' });+ }++ async resolveFleetAlert(alertId) {+ return this.request(`/fleet-monitor/alerts/${alertId}/resolve`, { method: 'POST' });+ }++ async getFleetThresholds(serverId) {+ const query = serverId ? `?server_id=${serverId}` : '';+ return this.request(`/fleet-monitor/thresholds${query}`);+ }++ async createFleetThreshold(data) {+ return this.request('/fleet-monitor/thresholds', { method: 'POST', body: data });+ }++ async deleteFleetThreshold(thresholdId) {+ return this.request(`/fleet-monitor/thresholds/${thresholdId}`, { method: 'DELETE' });+ }++ async getFleetAnomalies(serverId) {+ const query = serverId ? `?server_id=${serverId}` : '';+ return this.request(`/fleet-monitor/anomalies${query}`);+ }++ async getCapacityForecast(serverId, metric = 'disk') {+ return this.request(`/fleet-monitor/forecast/${serverId}?metric=${metric}`);+ }++ async searchFleet(query, type = 'any') {+ return this.request(`/fleet-monitor/search?q=${encodeURIComponent(query)}&type=${type}`);+ }++ async exportFleetCsv(serverIds, metric, period) {+ const ids = serverIds.join(',');+ const url = `${this.baseUrl}/fleet-monitor/export/csv?ids=${ids}&metric=${metric}&period=${period}`;+ const token = this.getToken();+ const response = await fetch(url, {+ headers: token ? { Authorization: `Bearer ${token}` } : {}+ });+ return response.blob();+ }+// ── Email Server ──async getEmailStatus() { return this.request('/email/status'); }async installEmailServer(data = {}) { return this.request('/email/install', { method: 'POST', body: JSON.stringify(data) }); }diff --git a/frontend/src/styles/main.less b/frontend/src/styles/main.lessindex b5a792fe..fd421217 100644--- a/frontend/src/styles/main.less+++ b/frontend/src/styles/main.less@@ -84,3 +84,4 @@@import 'pages/_migration-wizard';@import 'pages/_services';@import 'pages/_service-detail';+@import 'pages/_fleet-monitor';diff --git a/frontend/src/styles/pages/_fleet-monitor.less b/frontend/src/styles/pages/_fleet-monitor.lessnew file mode 100644index 00000000..f6530735--- /dev/null+++ b/frontend/src/styles/pages/_fleet-monitor.less@@ -0,0 +1,79 @@+// ============================================+// FLEET MONITOR PAGE+// ============================================++.fleet-heatmap {+ width: 100%;++ &__header {+ display: grid;+ grid-template-columns: 2fr 1fr 1fr 1fr 1fr 1fr;+ gap: @space-2;+ padding: @space-2 @space-3;+ font-size: 0.75rem;+ font-weight: 600;+ text-transform: uppercase;+ color: @text-secondary;+ border-bottom: 1px solid @border-subtle;+ }++ &__row {+ display: grid;+ grid-template-columns: 2fr 1fr 1fr 1fr 1fr 1fr;+ gap: @space-2;+ padding: @space-2 @space-3;+ align-items: center;+ border-bottom: 1px solid @border-subtle;+ transition: background 0.15s;++ &:hover {+ background: @bg-hover;+ }++ &:last-child {+ border-bottom: none;+ }+ }++ &__server {+ display: flex;+ align-items: center;+ gap: @space-2;+ font-weight: 500;+ font-size: 0.875rem;+ overflow: hidden;+ text-overflow: ellipsis;+ white-space: nowrap;+ }++ &__cell {+ text-align: center;+ padding: @space-1 @space-2;+ border-radius: @radius-sm;+ font-size: 0.8125rem;+ font-weight: 500;+ font-variant-numeric: tabular-nums;+ transition: background-color 0.3s;+ }++ &__label {+ text-align: center;+ }+}++@media (max-width: @breakpoint-md) {+ .fleet-heatmap {+ &__header,+ &__row {+ grid-template-columns: 1.5fr 1fr 1fr 1fr;+ }++ // Hide containers and status columns on small screens+ &__header > :nth-child(5),+ &__header > :nth-child(6),+ &__row > :nth-child(5),+ &__row > :nth-child(6) {+ display: none;+ }+ }+}diff --git a/frontend/vite.config.js b/frontend/vite.config.jsindex f75e81b4..ac8325f3 100644--- a/frontend/vite.config.js+++ b/frontend/vite.config.js@@ -13,6 +13,7 @@ export default defineConfig({},},build: {+ sourcemap: false,rollupOptions: {output: {manualChunks: {