ServerKit is a lightweight, modern server control panel for managing web applications, databases, and services on your VPS or dedicated server. Built with Python/Flask backend and React frontend.
Python astracloudflarecontrol-panelcyberpaneldnsfablehosting-control-panelhosting-panelkimi-k3server-managementserverkitubuntu-web-panelwebserver
docs(22): Complete Container Logs API phase
Juan Denis33fd67579f60
diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.mdindex acfb58ea..aa597c54 100644--- a/.planning/ROADMAP.md+++ b/.planning/ROADMAP.md@@ -30,7 +30,7 @@ Completed 2026-01-19 | 6 phases | React Flow canvas, node types, deploy from wor| Phase | Name | Status | Goal ||-------|------|--------|------|-| 22 | Container Logs API | pending | Backend API to fetch/stream Docker container logs |+| 22 | Container Logs API | **completed** | Backend API to fetch/stream Docker container logs || 23 | Log Viewer UI | pending | Real-time log display with auto-scroll, ANSI color support || 24 | Log Search & Filtering | pending | Search within logs, filter by level/time, export || 25 | Per-App Resource Collection | pending | Collect per-container CPU/memory/network stats via Docker API |@@ -40,7 +40,7 @@ Completed 2026-01-19 | 6 phases | React Flow canvas, node types, deploy from wor----## Phase 22: Container Logs API+## Phase 22: Container Logs API (Completed)**Goal**: Create backend API to fetch and stream Docker container logs**Key Tasks**:@@ -51,10 +51,10 @@ Completed 2026-01-19 | 6 phases | React Flow canvas, node types, deploy from wor- Support multiple output formats (raw, JSON)**Success Criteria**:-- [ ] Can fetch last N lines of container logs-- [ ] Can stream logs in real-time via WebSocket-- [ ] Logs include timestamps-- [ ] Handles containers that aren't running+- [x] Can fetch last N lines of container logs+- [x] Can stream logs in real-time via WebSocket+- [x] Logs include timestamps+- [x] Handles containers that aren't running---diff --git a/.planning/STATE.md b/.planning/STATE.mdindex 1ab7bc9c..a6773995 100644--- a/.planning/STATE.md+++ b/.planning/STATE.md@@ -2,7 +2,7 @@## Current Status- **Milestone**: 4 - Container Logs & Monitoring-- **Current Phase**: 22 - Container Logs API+- **Current Phase**: 23 - Log Viewer UI- **Phase Status**: Not Started- **Last Updated**: 2026-01-19@@ -10,7 +10,7 @@| Phase | Status | Started | Completed ||-------|--------|---------|-----------|-| 22 | pending | - | - |+| 22 | completed | 2026-01-19 | 2026-01-19 || 23 | pending | - | - || 24 | pending | - | - || 25 | pending | - | - |@@ -77,11 +77,14 @@ Infrastructure Fixes, Private URLs & UI Polish:None currently identified.## Notes-- Docker SDK for Python will be used for container logs/stats-- Existing WebSocket infrastructure (Flask-SocketIO) will be extended+- Using subprocess for Docker CLI commands (not Docker SDK)+- WebSocket infrastructure extended with container log streaming- Recharts already installed for graphs- Existing NotificationService will be extended for alerts+## Session Log+- 2026-01-19: Phase 22 completed - Container Logs API (3 commits)+## Quick Links- [ROADMAP.md](./ROADMAP.md)- [PROJECT.md](./PROJECT.md)diff --git a/.planning/phases/22-container-logs-api/PLAN.md b/.planning/phases/22-container-logs-api/PLAN.mdnew file mode 100644index 00000000..4f45d192--- /dev/null+++ b/.planning/phases/22-container-logs-api/PLAN.md@@ -0,0 +1,318 @@+# Phase 22: Container Logs API - Execution Plan++## Goal+Create backend API to fetch and stream Docker container logs for applications, enabling real-time log viewing in the UI (Phase 23).++## Prerequisites+- Docker apps exist with `container_id` or can be resolved via `root_path`+- WebSocket infrastructure exists (Flask-SocketIO with threading async_mode)+- Existing `DockerService.get_container_logs()` method available++## Architecture++### Existing Infrastructure++**DockerService** (`backend/app/services/docker_service.py:237-254`):+```python+@staticmethod+def get_container_logs(container_id, tail=100, since=None, timestamps=True):+ """Get container logs."""+ cmd = ['docker', 'logs']+ if tail:+ cmd.extend(['--tail', str(tail)])+ if since:+ cmd.extend(['--since', since])+ if timestamps:+ cmd.append('-t')+ cmd.append(container_id)+ # Returns {'success': True, 'logs': logs} or {'success': False, 'error': str}+```++**WebSocket Pattern** (`backend/app/sockets.py`):+- Uses room-based streaming: `join_room(f'build_{app_id}')`+- Build logs emit via `emit_build_log(app_id, message, level)`+- Clients subscribe via `subscribe_build` event with `app_id`++**Application Model** (`backend/app/models/application.py`):+- `container_id`: String(100), nullable - stores Docker container ID+- `root_path`: String(500) - path to docker-compose directory+- For compose apps: container names follow `serverkit-app-{app.id}` pattern++### Log Streaming Flow+```+1. Client connects to WebSocket with JWT token+2. Client emits 'subscribe_container_logs' with app_id+3. Server joins room 'logs_{app_id}'+4. Server starts background thread to stream logs via `docker logs -f`+5. Server emits 'container_log' events to room+6. Client emits 'unsubscribe_container_logs' or disconnects+7. Server stops background thread and leaves room+```++## Tasks++### Task 1: Extend DockerService with Streaming Logs+**File**: `backend/app/services/docker_service.py`++Add method for streaming container logs:+```python+@staticmethod+def stream_container_logs(container_id, callback, tail=100, since=None, timestamps=True):+ """Stream container logs in real-time.++ Args:+ container_id: Docker container ID or name+ callback: Function to call with each log line+ tail: Number of existing lines to fetch first+ since: Only logs since this timestamp+ timestamps: Include timestamps in output++ Returns:+ subprocess.Popen object for the streaming process+ """+ cmd = ['docker', 'logs', '--follow']+ if tail:+ cmd.extend(['--tail', str(tail)])+ if since:+ cmd.extend(['--since', since])+ if timestamps:+ cmd.append('-t')+ cmd.append(container_id)++ process = subprocess.Popen(+ cmd,+ stdout=subprocess.PIPE,+ stderr=subprocess.STDOUT,+ text=True,+ bufsize=1+ )+ return process+```++### Task 2: Create Container Logs REST API Endpoint+**File**: `backend/app/api/apps.py`++Extend existing `/apps/<id>/logs` endpoint to support container logs with more options:+```python+@apps_bp.route('/<int:app_id>/container-logs', methods=['GET'])+@jwt_required()+def get_container_logs(app_id):+ """Get container logs for a Docker application.++ Query params:+ - tail: Number of lines from end (default: 100, max: 10000)+ - since: ISO timestamp or duration (e.g., '10m', '1h', '2024-01-01T00:00:00Z')+ - timestamps: Include timestamps (default: true)+ - format: Output format - 'raw' or 'json' (default: 'raw')++ Returns:+ {+ "success": true,+ "logs": "...", // Raw format+ "lines": [...] // JSON format with parsed lines+ "container_id": "...",+ "container_name": "...",+ "app_id": 1+ }+ """+```++### Task 3: Add Container Logs WebSocket Events+**File**: `backend/app/sockets.py`++Add WebSocket handlers for real-time log streaming:+```python+# Store active log streams+container_log_streams = {} # sid -> {'process': Popen, 'app_id': int, 'thread': Thread}+[email protected]('subscribe_container_logs')+def handle_subscribe_container_logs(data):+ """Subscribe to real-time container log streaming.++ data: {+ 'app_id': int,+ 'tail': int (optional, default 100),+ 'since': str (optional)+ }+ """+ sid = request.sid+ app_id = data.get('app_id')+ tail = data.get('tail', 100)+ since = data.get('since')++ # Validate token and app access (already done in connect)+ # Get container ID from app+ # Start streaming thread+ # Join room for this app's logs+ join_room(f'logs_{app_id}')+[email protected]('unsubscribe_container_logs')+def handle_unsubscribe_container_logs():+ """Unsubscribe from container log streaming."""+ sid = request.sid+ # Stop streaming process+ # Leave room+ # Clean up++def emit_container_log(app_id: int, line: str, timestamp: str = None):+ """Emit a container log line to all subscribers."""+ socketio.emit('container_log', {+ 'app_id': app_id,+ 'line': line,+ 'timestamp': timestamp or time.time()+ }, room=f'logs_{app_id}')+```++### Task 4: Handle Container Resolution for Compose Apps+**File**: `backend/app/services/docker_service.py`++Add method to get container name/ID for an app:+```python+@staticmethod+def get_app_container_id(app):+ """Get the main container ID for an application.++ For apps with container_id set, use that directly.+ For compose apps, query docker compose ps to find container.++ Args:+ app: Application model instance++ Returns:+ str: Container ID or name, or None if not found+ """+ if app.container_id:+ return app.container_id++ if app.root_path:+ # Get containers from docker compose+ containers = DockerService.compose_ps(app.root_path)+ if containers:+ # Return first container (main service)+ # For compose apps, container name is typically service name+ return containers[0].get('ID') or containers[0].get('Name')++ return None++@staticmethod+def get_app_container_name(app_id):+ """Get container name by convention: serverkit-app-{app_id}"""+ return f"serverkit-app-{app_id}"+```++### Task 5: Add Log Line Parsing Utility+**File**: `backend/app/services/docker_service.py`++Add utility to parse Docker log lines:+```python+@staticmethod+def parse_log_line(line):+ """Parse a Docker log line into structured format.++ Docker logs with timestamps look like:+ 2024-01-15T10:30:45.123456789Z Log message here++ Returns:+ {+ 'timestamp': '2024-01-15T10:30:45.123456789Z',+ 'message': 'Log message here',+ 'level': 'info' # Detected from content+ }+ """+ # Parse timestamp if present+ # Detect log level from message content (ERROR, WARN, INFO, DEBUG)+ # Return structured object+```++### Task 6: Handle Non-Running Container Edge Cases+**File**: `backend/app/api/apps.py` and `backend/app/sockets.py`++Handle edge cases gracefully:+- Container doesn't exist: Return appropriate error message+- Container is stopped: Return existing logs (no streaming)+- Container is restarting: Handle reconnection+- Multiple containers (compose): Allow selecting specific service++```python+# In REST endpoint+if not container_id:+ return jsonify({+ 'success': False,+ 'error': 'No container found for this application',+ 'hint': 'The application may not have been started yet'+ }), 404++# Check if container is running for streaming+container_info = DockerService.get_container(container_id)+if not container_info:+ return jsonify({+ 'success': False,+ 'error': 'Container not found'+ }), 404++state = container_info.get('State', {}).get('Status', 'unknown')+```++### Task 7: Add API Documentation+**File**: Update API with docstrings and OpenAPI-style docs++Document the new endpoints:+- `GET /apps/{id}/container-logs` - Fetch historical logs+- WebSocket `subscribe_container_logs` - Stream real-time logs+- WebSocket `unsubscribe_container_logs` - Stop streaming+- WebSocket `container_log` event - Log line event++## Success Criteria+- [ ] Can fetch last N lines of container logs via REST API+- [ ] Can stream logs in real-time via WebSocket+- [ ] Logs include timestamps+- [ ] Handles containers that aren't running (returns stored logs or error)+- [ ] Handles compose apps (multiple containers)+- [ ] JSON format option for structured log data+- [ ] Proper cleanup when client disconnects++## Files to Create/Modify++| File | Action | Purpose |+|------|--------|---------|+| `backend/app/services/docker_service.py` | Modify | Add streaming logs, container resolution, log parsing |+| `backend/app/api/apps.py` | Modify | Add `/container-logs` endpoint |+| `backend/app/sockets.py` | Modify | Add container log streaming WebSocket events |++## Technical Notes++### Docker Logs Command Reference+```bash+# Basic logs+docker logs <container>++# With options+docker logs --tail 100 --since "10m" -t --follow <container>++# Options:+# --tail N Show last N lines+# --since TIME Show logs since timestamp (RFC3339) or duration (10m, 1h)+# -t, --timestamps Show timestamps+# -f, --follow Stream new logs+```++### WebSocket Room Naming Convention+- Build logs: `build_{app_id}`+- Container logs: `logs_{app_id}`+- Metrics: broadcast to all subscribers++### Log Buffer Limits+- REST API: Max 10,000 lines per request+- WebSocket: No buffer limit (streaming)+- Frontend (Phase 23): Should implement virtual scrolling for large log volumes++### ANSI Color Codes+Docker logs may contain ANSI color codes. These will be preserved in the output and handled by the frontend (Phase 23) using an ANSI parser.++## Estimated Commits+- feat(22-1): Add streaming logs method to DockerService+- feat(22-2): Add container-logs REST endpoint+- feat(22-3): Add WebSocket container log streaming+- feat(22-4): Add container resolution for compose apps+- feat(22-5): Add log parsing utility+- feat(22-6): Handle edge cases for non-running containersdiff --git a/.planning/phases/22-container-logs-api/SUMMARY.md b/.planning/phases/22-container-logs-api/SUMMARY.mdnew file mode 100644index 00000000..de907fe2--- /dev/null+++ b/.planning/phases/22-container-logs-api/SUMMARY.md@@ -0,0 +1,133 @@+# Phase 22: Container Logs API - Summary++## Overview+Created backend API and WebSocket infrastructure to fetch and stream Docker container logs for applications, enabling real-time log viewing in the UI.++## Completed Tasks++### Task 1: DockerService Streaming Extensions+**Commit**: `f3010d9`++Added methods to DockerService for container log operations:+- `stream_container_logs()` - Start streaming logs via subprocess with `docker logs --follow`+- `get_app_container_id()` - Resolve container ID from Application model (handles both direct container_id and compose apps)+- `get_all_app_containers()` - List all containers for compose apps+- `parse_log_line()` - Parse Docker log line to extract timestamp and detect level (error/warn/info/debug)+- `parse_logs_to_lines()` - Batch parse logs text into structured lines+- `get_container_state()` - Check if container is running, get status info++### Task 2: REST API Endpoints+**Commit**: `744c626`++Added endpoints to `backend/app/api/apps.py`:++**GET /apps/{id}/container-logs**+- Query params: `tail`, `since`, `timestamps`, `format`, `service`+- Returns logs in 'raw' or 'json' format+- Includes container state and available services+- Caps at 10,000 lines for performance++**GET /apps/{id}/containers**+- Lists all containers for a compose application+- Returns container id, name, service, state++### Task 3: WebSocket Real-Time Streaming+**Commit**: `f55e399`++Added WebSocket events to `backend/app/sockets.py`:++**subscribe_container_logs** event+- Input: `{app_id, tail?, since?, service?}`+- Starts background thread streaming via `docker logs --follow`+- Joins room `logs_{app_id}`+- Emits `container_log` events with parsed log lines++**unsubscribe_container_logs** event+- Stops streaming subprocess+- Leaves room+- Cleans up resources++**Events emitted:**+- `container_log` - Log line with parsed timestamp/level+- `container_log_ended` - When container stops+- `container_log_error` - On stream failure++### Edge Cases Handled+- Container not found: Returns helpful error with hint+- Container stopped: REST returns existing logs, WebSocket emits ended event+- Compose apps: Service selection via `service` parameter+- Process cleanup: Terminates subprocess on disconnect/unsubscribe+- Large logs: Capped at 10,000 lines for REST API++## Files Modified++| File | Changes |+|------|---------|+| `backend/app/services/docker_service.py` | +185 lines: streaming, parsing, container resolution |+| `backend/app/api/apps.py` | +151 lines: container-logs and containers endpoints |+| `backend/app/sockets.py` | +227 lines: WebSocket container log streaming |++## API Reference++### REST Endpoints++```+GET /api/v1/apps/{id}/container-logs+ ?tail=100 # Lines from end (max 10000)+ ?since=10m # Duration or ISO timestamp+ ?timestamps=true # Include timestamps+ ?format=raw # 'raw' or 'json'+ ?service=web # Specific service for compose apps++GET /api/v1/apps/{id}/containers+ # List containers for compose apps+```++### WebSocket Events++```javascript+// Subscribe to logs+socket.emit('subscribe_container_logs', {+ app_id: 1,+ tail: 100,+ since: '10m',+ service: 'web' // optional+});++// Receive logs+socket.on('container_log', (data) => {+ // data: {app_id, line, parsed: {timestamp, message, level}, timestamp}+});++socket.on('container_log_ended', (data) => {+ // Container stopped+});++socket.on('container_log_error', (data) => {+ // Stream error+});++// Unsubscribe+socket.emit('unsubscribe_container_logs');+```++## Testing Notes++- Tested against Docker compose applications+- Log streaming continues until container stops or client disconnects+- ANSI color codes preserved in output (frontend will parse)+- Timestamps parsed from Docker RFC3339 format++## Next Phase++Phase 23: Log Viewer UI+- Create LogViewer component with terminal-style display+- Implement auto-scroll with pause on user scroll+- Add ANSI color code parsing+- Connect to WebSocket for live streaming++## Commits++1. `f3010d9` - feat(22-1): Add streaming logs and container resolution to DockerService+2. `744c626` - feat(22-2): Add container-logs REST API endpoint+3. `f55e399` - feat(22-3): Add WebSocket container log streaming