A modern and elegant dashboard for network traffic visualization and analysis.
Neko Master - Comprehensive Codebase Research Report
Executive Summary
Neko Master is a modern network traffic analytics platform designed for edge gateways. It provides real-time monitoring, traffic auditing, and multi-gateway management capabilities. The system collects traffic data from Clash/Mihomo and Surge proxy gateways, aggregates it in SQLite, and visualizes it through a Next.js web interface.
1. Project Overview
1.1 Purpose & Scope
Neko Master serves as a traffic visibility layer for proxy gateways. It does NOT:
- Provide network access or proxy services
- Handle subscription management
- Perform traffic routing
It DOES:
- Collect and aggregate traffic statistics from gateways
- Provide real-time WebSocket updates to connected clients
- Offer historical traffic analysis and visualization
- Support multiple backend gateways simultaneously
1.2 Technology Stack
| Layer | Technology |
|---|---|
| Frontend | Next.js 16, React 19, TypeScript, Tailwind CSS v4 |
| UI Components | shadcn/ui (new-york style), lucide-react icons |
| Charts | Recharts, D3.js, react-simple-maps |
| State Management | TanStack React Query, WebSocket hooks |
| Backend | Fastify 5, WebSocket (ws), better-sqlite3 |
| GeoIP | maxmind (local MMDB), online API fallback |
| i18n | next-intl (zh/en) |
| Build | Turbo monorepo, pnpm workspaces |
| Deployment | Docker, standalone Next.js output |
1.3 Repository Structure
neko-master/
├── apps/
│ ├── collector/ # Backend data collection service
│ │ └── src/
│ │ ├── index.ts # Entry point, orchestrates all services
│ │ ├── app.ts # Fastify API server
│ │ ├── websocket.ts # WebSocket broadcast server
│ │ ├── collector.ts # Clash WebSocket collector
│ │ ├── surge-collector.ts # Surge HTTP polling collector
│ │ ├── realtime.ts # In-memory traffic aggregation
│ │ ├── batch-buffer.ts # Batch write optimization
│ │ ├── geo-service.ts # GeoIP lookup service
│ │ ├── db.ts # Database facade
│ │ └── database/
│ │ ├── schema.ts # DDL definitions
│ │ └── repositories/ # Repository pattern implementations
│ │
│ └── web/ # Next.js frontend
│ └── app/[locale]/
│ ├── dashboard/ # Main dashboard page
│ └── layout.tsx # Root layout with providers
│
├── packages/
│ └── shared/ # Shared TypeScript types & utilities
│ └── src/
│ ├── index.ts # Type exports
│ ├── gateway-utils.ts # Gateway API helpers
│ └── geo-ip-utils.ts # GeoIP data normalization
│
└── docs/
└── architecture.md # Chinese architecture documentation
2. Core Architecture
2.1 High-Level Data Flow
┌─────────────────────┐
│ Clash / Mihomo │──── WebSocket ────┐
│ Gateway │ │
└─────────────────────┘ │
▼
┌─────────────────────┐ ┌─────────────────┐
│ Surge Gateway │──── HTTP ──▶│ Collector │
│ │ Polling │ Service │
└─────────────────────┘ │ (Port 3001) │
└────────┬────────┘
│
┌─────────────┼─────────────┐
▼ ▼ ▼
┌──────────┐ ┌──────────┐ ┌──────────┐
│ SQLite │ │ Realtime │ │ WS │
│ (WAL) │ │ Store │ │ Server │
└──────────┘ └──────────┘ │(Port 3002)│
│ │ └──────────┘
│ │ │
▼ ▼ │
┌─────────────────────────┐ │
│ Next.js Web UI │◀───┘
│ (Port 3000) │
└─────────────────────────┘
2.2 Three-Tier Architecture
Frontend Tier (Next.js)
- React Query for HTTP API data fetching
- WebSocket hook for real-time updates
- Client-side state management
Collector Tier (Fastify + WebSocket)
- REST API endpoints (
/api/*) - WebSocket server for real-time push
- Gateway collectors (Clash WS + Surge HTTP)
- REST API endpoints (
Storage Tier (SQLite + Memory)
- SQLite with WAL mode for persistence
- In-memory RealtimeStore for low-latency queries
3. Backend Deep Dive
3.1 Entry Point (index.ts)
The main entry point orchestrates all services:
- Database initialization - Creates/opens SQLite database
- GeoIP service - Initializes IP geolocation service
- WebSocket server - Starts WS server on port 3002
- API server - Starts Fastify on port 3001
- Backend management loop - Polls every 5s for backend config changes
- Auto-cleanup job - Runs every 6 hours for data retention
// Key initialization sequence
async function main() {
db = new StatsDatabase(DB_PATH);
geoService = new GeoIPService(db);
wsServer = new StatsWebSocketServer(COLLECTOR_WS_PORT, db);
apiServer = new APIServer(API_PORT, db, realtimeStore, policySyncService);
// Start collectors based on database config
setInterval(manageBackends, 5000);
}
3.2 Clash Collector (collector.ts)
Connection Protocol:
- Connects to
ws://gateway:port/connections - Uses Bearer token authentication if provided
- Auto-reconnects on disconnect with 5s interval
Traffic Tracking Algorithm:
For each WebSocket message:
1. Parse connections array
2. For new connections:
- Track initial traffic values
- Record to batch buffer
- Record to realtime store
3. For existing connections:
- Calculate delta: current - last
- Record delta to buffer and realtime
- Update last seen timestamp
4. For disappeared connections:
- Remove from tracking map
5. Flush batch buffer if >= 5000 items
Stale Connection Cleanup:
- Removes connections not seen for 5 minutes
- Runs every 2 minutes
3.3 Surge Collector (surge-collector.ts)
Different from Clash:
- Uses HTTP polling (GET
/v1/requests/recent) instead of WebSocket - Default 2s polling interval
- Exponential backoff on errors (max 5 retries, up to 60s delay)
Policy Path Extraction:
Surge provides policy decision paths in the notes field:
"[Rule] Policy decision path: rule -> group1 -> group2 -> finalProxy"
This is parsed and reversed to match Clash's chain format:
chains = [finalProxy, ..., group2, group1, rule]
Duplicate Prevention:
- Uses
recentlyCompletedMap with 5-minute TTL - Prevents double-counting completed requests that may reappear in polls
3.4 RealtimeStore (realtime.ts)
The RealtimeStore is a critical component for low-latency queries. It maintains:
| Map | Purpose | Max Entries |
|---|---|---|
summaryByBackend |
Total upload/download/connections delta | Per backend |
minuteByBackend |
Per-minute traffic buckets | 180 minutes default |
domainByBackend |
Per-domain traffic deltas | 50,000 |
ipByBackend |
Per-IP traffic deltas | 50,000 |
proxyByBackend |
Per-proxy traffic deltas | Unlimited |
deviceByBackend |
Per-device (source IP) traffic | Unlimited |
ruleByBackend |
Per-rule traffic deltas | Unlimited |
countryByBackend |
Per-country traffic deltas | Unlimited |
Key Operations:
recordTraffic()- Adds traffic delta to all relevant mapsmerge*()methods - Combines DB data with realtime deltasclearTraffic()- Called after batch flushpruneIfNeeded()- Evicts lowest-traffic entries when limits exceeded
Memory Bounds:
- Maximum 50,000 domain entries
- Maximum 50,000 IP entries
- Maximum 10,000 device detail entries
- Eviction removes bottom 25% by traffic
3.5 BatchBuffer (batch-buffer.ts)
Optimizes database writes by:
Aggregating by composite key:
backendId:minute:domain:ip:chain:fullChain:rule:rulePayload:sourceIPBatching flushes:
- Every 30 seconds (configurable via
FLUSH_INTERVAL_MS) - When buffer reaches 5000 items (configurable via
FLUSH_MAX_BUFFER_SIZE)
- Every 30 seconds (configurable via
Two-phase write:
- Traffic stats →
db.batchUpdateTrafficStats() - Country stats →
db.batchUpdateCountryStats()
- Traffic stats →
3.6 WebSocket Server (websocket.ts)
Connection Lifecycle:
- Client connects with optional token (URL param or cookie)
- Server validates auth if enabled
- Client sends
subscribemessage with options:backendId- Which backend to subscribe tostart/end- Time range for historical dataminPushIntervalMs- Throttle push frequency- Various detail inclusion flags
Broadcast Mechanism:
broadcastStats(changedBackendId?: number, force = false) {
// Throttle: max 1 broadcast per second
if (now - lastBroadcastTime < 1000 && !force) return;
// Per-client push respecting minPushIntervalMs
for each client:
if client.minPushIntervalMs > 0 && elapsed < minPushIntervalMs:
skip
// Resolve backend
if changedBackendId !== client.backendId:
skip
// Get stats and send
stats = getStatsForBackend(...)
ws.send(JSON.stringify({ type: 'stats', data: stats }))
}
Summary Caching:
- 2-second TTL for realtime ranges
- 5-minute TTL for historical ranges
- Cache key includes all query parameters for deduplication
3.7 Database Layer
3.7.1 Schema Overview
Primary Tables:
| Table | Purpose | Primary Key |
|---|---|---|
domain_stats |
Per-domain aggregation | (backend_id, domain) |
ip_stats |
Per-IP aggregation | (backend_id, ip) |
proxy_stats |
Per-proxy aggregation | (backend_id, chain) |
rule_stats |
Per-rule aggregation | (backend_id, rule) |
country_stats |
Per-country aggregation | (backend_id, country) |
device_stats |
Per-source-IP aggregation | (backend_id, source_ip) |
Fact Tables (for range queries):
| Table | Granularity | Use Case |
|---|---|---|
minute_dim_stats |
Per-minute | Short-range queries (<6h) |
hourly_dim_stats |
Per-hour | Long-range queries (>6h) |
minute_country_stats |
Per-minute country | Short-range country queries |
hourly_country_stats |
Per-hour country | Long-range country queries |
Cross-reference Tables:
| Table | Purpose |
|---|---|
domain_proxy_stats |
Domain × Proxy traffic |
ip_proxy_stats |
IP × Proxy traffic |
device_domain_stats |
Device × Domain traffic |
device_ip_stats |
Device × IP traffic |
rule_chain_traffic |
Rule × Chain traffic |
rule_domain_traffic |
Rule × Domain traffic |
rule_ip_traffic |
Rule × IP traffic |
3.7.2 Repository Pattern
The database uses a facade pattern (StatsDatabase) that delegates to specialized repositories:
class StatsDatabase {
public readonly repos = {
auth: AuthRepository,
surge: SurgeRepository,
timeseries: TimeseriesRepository,
country: CountryRepository,
device: DeviceRepository,
proxy: ProxyRepository,
rule: RuleRepository,
ip: IPRepository,
config: ConfigRepository,
trafficWriter: TrafficWriterRepository,
domain: DomainRepository,
backend: BackendRepository,
};
}
Each repository extends BaseRepository which provides shared utilities:
parseMinuteRange()- Time range parsingsplitChainParts()- Proxy chain parsingexpandShortChainsForRules()- Rule chain expansionaggregateProxyStatsByFirstHop()- Proxy aggregation
3.7.3 Query Cache
Database-level range query cache:
- Default TTL: 8 seconds for realtime, 5 minutes for historical
- Maximum 1024 entries
- Configurable via environment variables
3.8 GeoIP Service (geo-service.ts)
Dual Provider Support:
Online API (default)
- Endpoint:
https://api.ipinfo.es/ipinfo?ip=xxx - Rate-limited: 100ms minimum between requests
- 8-second timeout per request
- Endpoint:
Local MMDB (optional)
- Requires:
GeoLite2-City.mmdb,GeoLite2-ASN.mmdb - Optional:
GeoLite2-Country.mmdb - No rate limiting
- Hot-reload when files change
- Requires:
Lookup Flow:
getGeoLocation(ip):
1. Check if private IP → return "LOCAL"
2. Check database cache → return cached
3. Check failed IPs cooldown → return null
4. Check pending queries → return existing promise
5. Queue new lookup
6. Process queue with rate limiting
7. Save to database cache
Failure Handling:
- Failed IPs are cached for 30 minutes
- Queue overflow (100 items) drops new lookups
- Cleanup runs every 10 minutes
4. Frontend Deep Dive
4.1 Next.js Configuration (next.config.ts)
Key Features:
API Proxying:
// Rewrites /api/* to collector service rewrites: [{ source: "/api/:path*", destination: `${API_URL}/api/:path*` }]PWA Support:
- Enabled only in production
- Service worker with skip waiting
- Cache-Control headers for sw.js
Environment Variables:
NEXT_PUBLIC_APP_VERSION- From package.jsonNEXT_PUBLIC_WS_PORT- WebSocket port (default 3002)
4.2 Dashboard Page Structure
DashboardPage
├── Sidebar (navigation, backend status)
├── Header (controls, time range, refresh)
├── Content (tab-based content)
│ ├── Overview Tab (summary cards, top items)
│ ├── Domains Tab (paginated domain list)
│ ├── IPs Tab (paginated IP list with GeoIP)
│ ├── Proxies Tab (proxy traffic breakdown)
│ ├── Rules Tab (rule chain flow visualization)
│ ├── Devices Tab (per-device traffic)
│ └── Regions Tab (world map, country stats)
└── BackendConfigDialog (gateway setup)
4.3 Data Fetching Patterns
HTTP API (React Query):
// Hook pattern
export function useTrafficTrend(backendId, params) {
return useQuery({
queryKey: ['trafficTrend', backendId, params],
queryFn: () => api.getTrafficTrendAggregated(backendId, params),
staleTime: 5000,
});
}
WebSocket (Custom Hook):
// Real-time subscription
useStatsWebSocket({
backendId,
timeRange,
onMessage: (data) => {
// Update local state
}
});
4.4 Internationalization
- Uses
next-intlpackage - Two locales:
zh(default),en - Translation files:
apps/web/messages/{zh,en}.json - Hook:
useTranslations('namespace')
5. Multi-Backend Architecture
5.1 Backend Isolation
All data is isolated by backend_id:
- Database: All tables have
backend_idin primary key - RealtimeStore: Maps are keyed by backend ID
- WebSocket: Clients subscribe to specific backend
5.2 Backend Management
// Backend management loop (every 5s)
manageBackends():
1. Get all backend configs from DB
2. For each backend:
- If config changed → restart collector
- If enabled/listening changed → start/stop collector
3. Stop collectors for deleted backends
5.3 Backend Types
| Type | Protocol | Data Source |
|---|---|---|
clash |
WebSocket | /connections endpoint |
surge |
HTTP Polling | /v1/requests/recent endpoint |
6. Authentication & Security
6.1 Cookie-Based Auth
- User sets token in settings UI
- Server hashes token and stores in
auth_configtable - Login endpoint sets
HttpOnlycookieneko-session - Subsequent requests include cookie automatically
6.2 Auth Middleware
// Fastify hook
app.addHook('onRequest', async (request, reply) => {
// Skip public routes
if (isPublicRoute(request.url)) return;
// Check if auth required
if (!authService.isAuthRequired()) return;
// Validate cookie or Bearer token
const token = request.cookies['neko-session']
|| extractBearerToken(request);
if (!valid) return 401;
});
6.3 Emergency Access
FORCE_ACCESS_CONTROL_OFF=truebypasses auth- Used for password recovery
- Should NOT be enabled in production
7. Performance Optimizations
7.1 Database Level
| Optimization | Setting |
|---|---|
| WAL Mode | journal_mode = WAL |
| Sync Mode | synchronous = NORMAL |
| Cache Size | 64MB (cache_size = -65536) |
| Temp Store | Memory (temp_store = MEMORY) |
| Busy Timeout | 5 seconds |
7.2 Application Level
| Strategy | Implementation |
|---|---|
| Batch Writes | 30s interval or 5000 items |
| Query Cache | 8s realtime / 5min historical |
| RealtimeStore | In-memory delta aggregation |
| WebSocket Throttle | 1s minimum between broadcasts |
| GeoIP Rate Limit | 100ms between online API calls |
7.3 Memory Management
- RealtimeStore entry limits (50K domains, 50K IPs)
- Eviction removes bottom 25% by traffic
- Stale connection cleanup (5-minute timeout)
8. Data Retention
8.1 Configuration
Stored in app_config table:
retention.connection_logs_days- Minute-level data (default: 7)retention.hourly_stats_days- Hourly aggregations (default: 30)retention.auto_cleanup- Enable automatic cleanup (default: true)
8.2 Cleanup Process
// Runs every 6 hours
runAutoCleanup():
1. Get retention config
2. Calculate cutoff dates
3. Delete old minute_stats records
4. Delete old hourly_stats records
9. Environment Variables
9.1 Core Variables
| Variable | Default | Description |
|---|---|---|
NODE_ENV |
- | Environment mode |
DB_PATH |
./stats.db |
SQLite database path |
COOKIE_SECRET |
auto-generated | Cookie signing key |
API_PORT |
3001 | Fastify API port |
COLLECTOR_WS_PORT |
3002 | WebSocket server port |
WEB_PORT |
3000 | Next.js port |
9.2 GeoIP Variables
| Variable | Default | Description |
|---|---|---|
GEOIP_LOOKUP_PROVIDER |
online |
online or local |
GEOIP_ONLINE_API_URL |
https://api.ipinfo.es/ipinfo |
Online API endpoint |
9.3 Tuning Variables
| Variable | Default | Description |
|---|---|---|
FLUSH_INTERVAL_MS |
30000 | Batch flush interval |
FLUSH_MAX_BUFFER_SIZE |
5000 | Max buffer size before flush |
REALTIME_MAX_MINUTES |
180 | RealtimeStore time window |
DB_RANGE_QUERY_CACHE_TTL_MS |
8000 | Query cache TTL (realtime) |
DB_HISTORICAL_QUERY_CACHE_TTL_MS |
300000 | Query cache TTL (historical) |
10. Testing
10.1 Test Framework
- Framework: Vitest
- Location:
apps/collector/src/**/*.test.ts - Patterns:
- Unit tests for repositories
- Service tests with mock databases
- Integration tests for collectors
10.2 Running Tests
# All tests
pnpm --filter @neko-master/collector test
# Single file
pnpm --filter @neko-master/collector test -- src/modules/auth/auth.service.test.ts
# Watch mode
pnpm --filter @neko-master/collector test:watch
11. Docker Deployment
11.1 Container Architecture
Single container runs both services:
- Next.js (port 3000) - Web UI
- Fastify (port 3001) - API
- WebSocket (port 3002) - Real-time
11.2 Docker Compose (Recommended)
services:
neko-master:
image: foru17/neko-master:latest
ports:
- "3000:3000"
- "3002:3002"
volumes:
- ./data:/app/data
- ./geoip:/app/data/geoip:ro
environment:
- COOKIE_SECRET=your-secret
11.3 Reverse Proxy Setup
Nginx example:
location / {
proxy_pass http://localhost:3000;
}
location /_cm_ws {
proxy_pass http://localhost:3002;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
12. Key Design Patterns
12.1 Repository Pattern
Separates data access logic from business logic. Each repository handles a specific domain:
DomainRepository- Domain statisticsIPRepository- IP statistics + GeoIPRuleRepository- Rule statistics + chain flow- etc.
12.2 Facade Pattern
StatsDatabase class acts as a facade, providing a simplified interface to the complex repository subsystem.
12.3 Observer Pattern
WebSocket server broadcasts updates to subscribed clients when new traffic data arrives.
12.4 Strategy Pattern
GeoIP service uses strategy pattern for provider selection (online vs local MMDB).
12.5 Batch Aggregation Pattern
Traffic updates are buffered and aggregated before database writes, reducing I/O by ~90%.
13. Potential Improvements
13.1 Identified Areas
Type Safety:
- Some
anytypes remain in test files (allowed by ESLint config) - GeoIP API responses could use stricter typing
- Some
Error Handling:
- Some error paths log but don't propagate errors
- Could benefit from structured error types
Performance:
- RealtimeStore could use LRU cache instead of eviction
- WebSocket message serialization could be optimized
Testing:
- Frontend lacks test coverage
- Could add E2E tests
Documentation:
- API documentation could be generated (OpenAPI/Swagger)
- Inline code comments could be expanded
14. Conclusion
Neko Master is a well-architected traffic analytics platform with:
- Clean separation of concerns between data collection, storage, and visualization
- Efficient data pipeline with batch aggregation and realtime caching
- Flexible multi-backend support for different gateway types
- Production-ready deployment with Docker and reverse proxy support
- Good performance characteristics through careful optimization
The codebase demonstrates strong TypeScript practices, modular design, and attention to operational concerns like data retention and graceful shutdown.