A modern and elegant dashboard for network traffic visualization and analysis.
docs: Add agent guidance, a documentation index, and initial research reports for GPT-5.3, GLM-5, and Kimi 2.5.
luoleid670f3fcf0d1
diff --git a/AGENTS.md b/AGENTS.mdnew file mode 100644index 0000000..dcd3037--- /dev/null+++ b/AGENTS.md@@ -0,0 +1,183 @@+# AGENTS.md++Guidance for coding agents working in this monorepo.++## Scope and Priorities++1. Follow this file for repo-specific workflows and conventions.+2. Follow direct user instructions over this file when they conflict.+3. Keep changes minimal, targeted, and consistent with existing code.++## Repository Overview++- Package manager: `pnpm` (workspace monorepo).+- Task runner: `turbo` at repo root.+- Main apps:+ - `apps/web`: Next.js 16 + React 19 frontend.+ - `apps/collector`: Fastify + WebSocket + SQLite backend service.+- Shared package:+ - `packages/shared`: shared TypeScript types and utility functions.++## Environment and Toolchain++- Node.js: project docs target Node 22.+- TypeScript is strict across packages.+- Module systems:+ - `apps/collector` and `packages/shared`: ESM (`type: module`, `NodeNext`).+ - `apps/web`: Next.js/bundler TS config.++## Install and Bootstrap++- Install dependencies: `pnpm install`+- Start all apps in dev mode: `pnpm dev`+- Optional helper script: `./start.sh`++## Build, Lint, and Test Commands++### Root (all workspaces)++- Build all packages/apps: `pnpm build`+- Run lint across workspaces: `pnpm lint`+- Start all dev tasks: `pnpm dev`++### Filtered workspace commands (recommended during development)++- Build web only: `pnpm --filter @neko-master/web build`+- Build collector only: `pnpm --filter @neko-master/collector build`+- Build shared only: `pnpm --filter @neko-master/shared build`+- Lint web only: `pnpm --filter @neko-master/web lint`+- Lint collector only: `pnpm --filter @neko-master/collector lint`++### Tests++- This repo currently has tests in `apps/collector` (Vitest).+- Run all collector tests: `pnpm --filter @neko-master/collector test`+- Watch mode: `pnpm --filter @neko-master/collector test:watch`++### Run a single test file (important)++- Run one test file:+ - `pnpm --filter @neko-master/collector test -- src/modules/auth/auth.service.test.ts`+- Run one test file by pattern:+ - `pnpm --filter @neko-master/collector test -- src/**/*.geoip-config.test.ts`+- Run a single test case by name:+ - `pnpm --filter @neko-master/collector test -- -t "should validate token format"`+- Run single file + test name:+ - `pnpm --filter @neko-master/collector test -- src/modules/auth/auth.service.test.ts -t "should validate token format"`++### Writing tests (collector)++- Use Vitest (`describe`/`it`/`expect` imported from `'vitest'`).+- Use `createTestDatabase()` and `createTestBackend()` from `src/__tests__/helpers.ts` to set up a temporary SQLite DB in `beforeEach` and call `cleanup()` in `afterEach`.+- Test files live next to the code they test (e.g., `auth.service.test.ts` beside `auth.service.ts`).++### Type checks and push safety++- Collector type check: `pnpm --filter @neko-master/collector exec tsc --noEmit`+- Note: `.husky/pre-push` runs checks when pushing to `main`:+ - collector `tsc --noEmit`+ - web `build`++## Code Style Guidelines++### General TypeScript Rules++- Prefer explicit types on exported APIs and `import type` for type-only imports.+- Keep strict-null and unknown-safe patterns; narrow with guards.+- Avoid `any`; if unavoidable, keep scope tiny and document briefly.+- Prefix intentionally unused variables/parameters with `_` (e.g., `_unused`).+- Prefer small pure functions for transformations and parsing logic.++### ESLint Key Rules (collector)++- `no-console`: **`console.log` is forbidden** — use `console.info`, `console.warn`, or `console.error` instead.+- `@typescript-eslint/no-unused-vars`: warn-level; variables/args prefixed with `_` are exempted.+- `@typescript-eslint/no-explicit-any`: warn-level in source code.+- **Test files** (`*.test.ts`, `__tests__/**`): `no-explicit-any` and `no-console` are turned off.++### Imports and Module Boundaries++- Keep import groups logically ordered:+ 1) platform/native modules,+ 2) third-party packages,+ 3) workspace/internal modules.+- Use relative imports with `.js` extension in ESM NodeNext packages (`collector`, `shared`).+- In web app code, prefer alias imports via `@/*` for internal modules.+- Do not create cross-app deep imports; use `@neko-master/shared` for shared contracts.++### Formatting and Layout++- Match local file style instead of forcing global formatting changes.+- Existing style is **not uniform**, even within a single package:+ - `apps/web`: double quotes and semicolons.+ - `apps/collector`: mixed — some files use single quotes, others use double quotes. Always follow the style of the file you are editing.+ - `packages/shared`: single quotes and semicolons.+- Preserve existing whitespace and line-wrap style in touched files.++### Naming Conventions++- Filenames: kebab-case (e.g., `auth.service.ts`, `use-gateway.ts`).+- React components: PascalCase exports; file names remain kebab-case.+- React hooks: `useXxx` naming.+- Types/interfaces/enums: PascalCase.+- Variables/functions: camelCase.+- Environment variables/constants: UPPER_SNAKE_CASE.+- Database-origin fields may use snake_case (`is_active`, `created_at`); preserve API contracts.++### Error Handling and Logging++- In `catch`, use `error: unknown`, then narrow via `instanceof Error`.+- Return structured API errors for HTTP handlers (e.g., `{ error: string }`) with proper status codes.+- Add contextual logs for operational failures (network, DB, background jobs).+- Use timeouts for external network calls where appropriate (`AbortSignal.timeout(...)`).+- Prefer resilient flows (`Promise.allSettled`) when partial results are acceptable.++### Backend (collector) Patterns++- Keep business logic in services; route/controller handlers should stay thin.+- Reuse repository/base-repository helpers for DB query logic.+- Maintain backend isolation by `backendId` in data paths and caches.+- For auth/security behavior, preserve cookie/token compatibility paths.+- **Fastify plugin pattern**: controllers export a `FastifyPluginAsync` function, registered in `app.ts` via `app.register(controller, { prefix: '/api/xxx' })`. Services are injected onto the Fastify instance through `declare module 'fastify'` augmentation and accessed as `fastify.xxxService`.++### Frontend (web) Patterns++- Respect Next.js App Router conventions (`app/[locale]/...`).+- Add `"use client"` only for client components/hooks that need it.+- Prefer React Query hooks for server state and caching.+- Keep API contract types aligned with `@neko-master/shared` and `apps/web/lib/api.ts`.+- Reuse shared utility helpers (`cn`, formatting utilities) instead of ad-hoc duplicates.+- **i18n**: uses `next-intl` with two locales (`zh` default, `en`).+ - Translation files: `apps/web/messages/{zh,en}.json`.+ - Routing config: `apps/web/i18n/routing.ts`.+ - Use `useTranslations('namespace')` in components; never hardcode user-facing strings.+- **UI components**: shadcn/ui (`new-york` style) + Tailwind CSS v4 + `lucide-react` icons.+ - Base UI primitives live in `apps/web/components/ui/`.+ - Config: `apps/web/components.json`.+ - Always reuse existing UI components before creating new ones.++### Shared Package Patterns++- `packages/shared` should stay framework-agnostic and side-effect free.+- Keep shared types backward compatible when possible.+- Export shared helpers from `packages/shared/src/index.ts` intentionally.++## Testing Expectations for Changes++- Backend-only changes: run collector lint + targeted tests first, then full collector tests if broad.+- Frontend-only changes: run web lint; also run web build for production-impacting changes.+- Cross-package changes: run `pnpm lint` and `pnpm build` before final handoff when feasible.++## Cursor/Copilot Rules Check++- Checked for Cursor rules:+ - `.cursor/rules/` -> not found+ - `.cursorrules` -> not found+- Checked for Copilot instructions:+ - `.github/copilot-instructions.md` -> not found+- If any of these files are added later, treat them as additional mandatory guidance.++## Agent Working Notes++- Prefer minimal, surgical edits over large refactors.+- Avoid renaming/moving files unless required by the task.diff --git a/docs/README.md b/docs/README.mdnew file mode 100644index 0000000..7da9e12--- /dev/null+++ b/docs/README.md@@ -0,0 +1,23 @@+# Documentation Index++## Architecture++- Chinese: `docs/architecture.md`+- English: `docs/architecture.en.md`++## Research Reports++### Kimi 2.5++- Chinese: `docs/research-kimi2.5.zh.md`+- English: `docs/research-kimi2.5.en.md`++### GPT-5.3++- Chinese: `docs/research-GPT5.3.zh.md`+- English: `docs/research-GPT5.3.en.md`++### GLM-5++- Chinese: `docs/research-glm5.zh.md`+- English: `docs/research-glm5.en.md`diff --git a/docs/research-GPT5.3.en.md b/docs/research-GPT5.3.en.mdnew file mode 100644index 0000000..3df2f92--- /dev/null+++ b/docs/research-GPT5.3.en.md@@ -0,0 +1,451 @@+# Neko Master Monorepo Deep Research (GPT-5.3)++This report documents a deep analysis of `/Users/luolei/DEV/clash-master`, including architecture, runtime behavior, code-level design patterns, edge cases, and operational specifics.++## 1) What This Repository Is++Neko Master is a pnpm monorepo for network traffic analytics and visualization.++- `apps/collector`: backend data collector + API + WebSocket push server (Fastify + ws + SQLite)+- `apps/web`: Next.js dashboard frontend (App Router, next-intl, React Query)+- `packages/shared`: shared contracts/utilities used by both apps++Core capability: ingest traffic from Clash/Mihomo and Surge, aggregate into multi-dimensional stats, and expose near-real-time dashboards.++## 2) Monorepo Build/Dev/Lint Orchestration++### Workspace + task runner++- Workspace config: `pnpm-workspace.yaml`, root `package.json`+- Task orchestration: `turbo.json`+- Root scripts:+ - `pnpm dev` -> `turbo dev`+ - `pnpm build` -> `turbo build`+ - `pnpm lint` -> `turbo lint`++### Turbo behavior specifics++- `build` depends on dependency graph (`^build`), caches `dist/**` and `.next/**` (excluding `.next/cache/**`)+- `dev` is non-cached and persistent+- `lint` also respects dependency graph (`^lint`)+- Global cache invalidation includes `**/.env.*local`++## 3) Collector Deep Dive (`apps/collector`)++## 3.1 Runtime topology and startup sequence++Main orchestrator: `apps/collector/src/index.ts`++Single process hosts all of the following:++- Fastify HTTP API server (`APIServer`)+- Dedicated WebSocket server (`StatsWebSocketServer`)+- One collector instance per backend (Clash WS collector or Surge polling collector)+- GeoIP service (`GeoIPService`)+- Surge policy sync service (`SurgePolicySyncService`)++Startup sequence in `main()`:++1. Load `.env.local` (if present), then `.env`+2. Initialize SQLite facade (`StatsDatabase`)+3. Initialize GeoIP service+4. Start WS server (default `3002`)+5. Start policy sync service+6. Start Fastify API server (default `3001`)+7. Start backend management loop immediately + every 5s+8. Schedule retention cleanup (+30s once, then every 6h)+9. Register graceful shutdown (`SIGINT`/`SIGTERM`)++## 3.2 Multi-backend lifecycle management++`manageBackends()` continuously reconciles DB backend configs with in-memory collectors:++- starts collectors for enabled + listening backends+- restarts collector when config changes (`url`, `token`, `type`, `listening`, `enabled`)+- stops collectors for deleted/disabled/non-listening backends++State maps are keyed by `backendId`, preserving backend isolation.++## 3.3 Ingestion path: Clash (WebSocket stream)++Implementation: `apps/collector/src/collector.ts`++- Connects to upstream `/connections` WS endpoint+- Tracks connection snapshots in `Map<id, TrackedConnection>`+- Per update, computes positive deltas (`max(0, current - previous)`)+- Ignores invalid/empty frames safely+- Removes stale missing IDs and periodically purges stale tracked connections+- Reconnect uses fixed interval behavior (no exponential retry)++## 3.4 Ingestion path: Surge (HTTP polling)++Implementation: `apps/collector/src/surge-collector.ts`++- Polls `/v1/requests/recent` (default interval ~2s)+- Uses exponential retry backoff for request failures+- 4xx is treated as non-retryable class in retry path+- Tracks requests by ID and computes incremental byte deltas+- Handles counter reset edge case (`current < previous`)+- Uses `recentlyCompleted` map (TTL ~5 min) to avoid duplicate counting of completed requests+- Parses policy/rule-chain metadata from Surge notes++## 3.5 Batch writing, aggregation, and realtime interplay++Batch layer: `apps/collector/src/batch-buffer.ts`++- Buffers updates and flushes on:+ - timer (`FLUSH_INTERVAL_MS`, default 30000)+ - buffer size threshold (`FLUSH_MAX_BUFFER_SIZE`, default 5000)+- Flush writes both traffic and country data in batch form++Writer repository: `apps/collector/src/database/repositories/traffic-writer.repository.ts`++- Pre-aggregates in memory before SQL writes+- Writes to multiple aggregate/fact tables in transactional groups++Realtime interaction:++- Deltas are also pushed into in-memory `realtimeStore`+- After successful DB flush, relevant realtime deltas are cleared to avoid DB+memory double counting+- Collectors trigger WS broadcast callback with internal throttling++## 3.6 API architecture and module boundaries++Core server assembly: `apps/collector/src/app.ts`++- Registers CORS, cookie plugin, service injections, and module controllers+- Controller prefixes:+ - `/api/backends`+ - `/api/stats`+ - `/api/auth`+ - `/api/db`+- Adds compatibility routes directly in `app.ts`:+ - `/api/gateway/proxies`+ - `/api/gateway/providers/proxies`+ - `/api/gateway/providers/proxies/refresh`+ - `/api/gateway/rules`+ - `/health`++Pattern:++- Controllers are thin orchestration layers (parse + validate + delegate)+- Services host business logic+- `StatsDatabase` facade + repository classes host data access/transform logic++## 3.7 Authentication and access-control specifics++Auth service: `apps/collector/src/modules/auth/auth.service.ts`++- Token stored as SHA-256 hash in DB+- Runtime checks support:+ - cookie (`neko-session`)+ - bearer fallback (`Authorization: Bearer ...`)+- Public routes skip auth checks (`/health`, `/api/auth/state`, `/api/auth/verify`, `/api/auth/logout`)++Special modes:++- `FORCE_ACCESS_CONTROL_OFF=true`: bypass auth requirement+- `SHOWCASE_SITE_MODE=true`: blocks many mutating endpoints and masks sensitive backend URL output++Cookie secret behavior:++- If `COOKIE_SECRET` missing, server auto-generates one+- In production this causes sessions to invalidate on restart unless secret is persisted++## 3.8 WebSocket server protocol and behavior++Server: `apps/collector/src/websocket.ts`++Highlights:++- Dedicated WS server process inside collector runtime+- Auth via cookie or query token+- Incoming message types include `ping` and `subscribe`+- Subscribe payload supports backend selection, time range, detail flags, pagination/trend options, and min push interval+- Initial data push on connect/subscription+- Broadcast globally throttled; per-client minimum interval also enforced+- Server-side response caching by query shape to avoid recomputation per client+- Base summary cache TTL differs by freshness (near-now short TTL, historical longer TTL)++Realtime merge policy:++- In-memory deltas are merged only for queries whose end-time is near “now” (`REALTIME_RANGE_END_TOLERANCE_MS`)++## 3.9 Database design and query strategy++Schema and migrations:++- `apps/collector/src/database/schema.ts`+- `apps/collector/src/db.ts`++Design traits:++- SQLite with WAL and performance pragmas+- Backend-scoped analytics (`backend_id` across major tables)+- Mix of cumulative aggregate tables + minute/hourly fact tables++Important query optimization behavior:++- Range queries choose minute or hourly fact table based on range span (commonly 6h threshold)+- DB range cache with TTL:+ - near-now short TTL+ - historical longer TTL++Retention:++- Retention config in DB (`app_config` / retention APIs)+- Auto cleanup prunes old minute/hourly and related fact data++## 3.10 GeoIP subsystem specifics++Service: `apps/collector/src/geo-service.ts`++Behavior:++- Private IP short-circuit to local classification+- DB cache first (`geoip_cache`)+- Dedupes concurrent lookups (`pendingQueries`)+- Cooldown for failed IPs to avoid hammering provider+- Queue + pacing controls for online lookups++Provider model:++- Configurable online/local provider+- Local uses MaxMind MMDB files (City + ASN required)+- Falls back to online when local unavailable+- Config endpoints expose configured vs effective provider and validate local prerequisites++## 3.11 Collector tests and coverage profile++Test setup:++- Vitest (`apps/collector/vitest.config.ts`)+- Helper utilities in `apps/collector/src/__tests__/helpers.ts`+- Temporary SQLite DB created per test suite cycle++Covered:++- auth service behavior+- stats service behavior+- traffic writer repository correctness+- geoip config/normalization behavior++Under-covered areas:++- startup orchestration in `index.ts`+- Clash/Surge collector runtime edge behavior+- WS protocol integration and auth edge cases+- backend lifecycle reconciliation loop race scenarios++## 3.12 Notable collector quirks and risks++- Manual route validation in controllers (limited centralized schema validation)+- Potential overlap risk: async `manageBackends` invoked by timer without explicit single-flight guard+- Legacy/parallel code paths exist in some module folders vs active top-level runtime files+- Data model contains denormalized comma-joined fields in places, which may degrade precision at scale++## 4) Web Frontend Deep Dive (`apps/web`)++## 4.1 Routing and locale architecture++Key files:++- `apps/web/app/[locale]/layout.tsx`+- `apps/web/app/[locale]/page.tsx`+- `apps/web/app/[locale]/dashboard/page.tsx`+- `apps/web/proxy.ts`+- `apps/web/i18n/routing.ts`+- `apps/web/i18n/request.ts`++Behavior:++- Locale-aware App Router (`en`, `zh`; default `zh`)+- Locale root page re-exports dashboard component directly+- next-intl middleware (`proxy.ts`) handles locale routing with matcher exclusions++## 4.2 Data layer and API client design++Core client: `apps/web/lib/api.ts`++- One typed API facade for all backend endpoints+- Runtime API base resolution order:+ 1. `window.__RUNTIME_CONFIG__.API_URL`+ 2. env (`NEXT_PUBLIC_API_URL` / `API_URL`)+ 3. `/api`+- GET inflight dedupe to avoid duplicate concurrent requests+- 401 triggers browser event (`api:unauthorized`) for auth state sync+- Uses `ApiError` class (`apps/web/lib/api-error.ts`)++React Query defaults:++- `apps/web/components/providers/query-provider.tsx`+- staleTime ~5s default, gcTime ~5m, retry 1+- Per-hook overrides in `hooks/api/*` + `lib/query-config.ts`++## 4.3 Dashboard state and composition model++Main orchestrator hook: `apps/web/app/[locale]/dashboard/hooks/use-dashboard.ts`++- Coordinates tab state, time range, backend selection, translations, auth awareness, and data fetching strategy+- Uses mixed strategy: HTTP query + websocket updates depending on tab and refresh mode++Composition:++- `dashboard/page.tsx` composes `Sidebar`, `Header`, and `Content`+- `Content` switches feature modules by tab (overview/domains/countries/proxies/rules/devices/network)++## 4.4 Real-time WebSocket client strategy++Client implementation: `apps/web/lib/websocket.ts`++- Candidate endpoint strategy supports runtime/env-configured WS URL and inferred fallbacks+- Environment-sensitive ordering (production tends to try path-based `/_cm_ws` first)+- Heartbeat ping/pong + latency tracking+- Exponential reconnect and fallback endpoint rotation+- Rich `subscribe` payload to control server response shape++Integration:++- Several feature modules consume websocket updates and merge into React Query cache for low-latency UX++## 4.5 Authentication UX and control flow++Core auth state: `apps/web/lib/auth.tsx`++- Global provider determines whether auth is enabled and whether current session is valid+- Guard (`AuthGuard`) displays login dialog when required+- Logout clears session and reloads++Notable behavior:++- API unauthorized events trigger auth state downgrade+- Auth verification uses backend auth endpoints and protected probe calls++## 4.6 UI, styling, theme, and component system++Key pieces:++- Tailwind v4 (`app/globals.css`, `postcss.config.mjs`)+- shadcn/ui config (`apps/web/components.json`, style `new-york`)+- Primitive UI components in `apps/web/components/ui/*`+- Theme runtime via next-themes provider and theme-color sync+- Selective framer-motion usage for key interactions++## 4.7 PWA/service worker setup++Files:++- Manifest route: `apps/web/app/manifest.ts`+- SW script: `apps/web/public/sw.js`+- SW runtime registration: `apps/web/components/common/sw-register.tsx`+- Next config conditional PWA plugin in production: `apps/web/next.config.ts`++Specificity:++- There is both custom service-worker logic and production next-pwa integration, so behavior can differ across environments if not carefully tested.++## 4.8 Frontend risks and edge cases++- Multiple feature modules can establish websocket connections simultaneously depending on view usage+- Strong coupling to backend WS payload shape (`stats` message optional fields)+- Locale switching is path-string based; robust enough now but sensitive to future route shape changes+- Viewport config disables user scaling (accessibility tradeoff)++## 5) Shared Package Deep Dive (`packages/shared`)++Primary file: `packages/shared/src/index.ts`++Provides:++- Shared domain contracts for stats, gateway payloads, websocket updates, auth, surge models+- Utility re-exports:+ - `gateway-utils.ts`+ - `geo-ip-utils.ts`++Important utilities:++- `buildGatewayHeaders`: type-aware header construction for Clash/Surge+- `parseSurgeRule` + `parseGatewayRule`: normalization/parsing helpers+- `getGatewayBaseUrl`: normalizes WS/HTTP gateway base URL extraction+- `normalizeGeoIP`: compatibility normalizer for mixed geo payload shapes++Compatibility note:++- Because apps depend on `workspace:*`, shared contract changes are immediately global and should be treated as coordinated cross-app changes.++## 6) Deployment, Runtime Config, and Ops++## 6.1 Next rewrite + runtime config++- API calls in browser usually target `/api/*`+- Next rewrite forwards `/api/:path*` to `API_URL` destination+- Runtime config script (`public/runtime-config.js`) supports deploy-time override without rebuild++## 6.2 Docker/runtime scripts++Relevant files:++- `Dockerfile`+- `docker-compose.yml`+- `docker-start.sh`+- `.dockerignore`++Notable behavior:++- startup script auto-generates/persists `COOKIE_SECRET` in mounted data volume if missing+- runtime config is generated/injected at container startup+- image build is multi-stage and multi-arch aware via CI workflow++## 7) Git Hooks and CI Quality Gates++Pre-push hook (`.husky/pre-push`):++- only enforced when pushing to `main`+- runs:+ - collector type check+ - web production build++GitHub Actions:++- Docker build/publish pipeline (`.github/workflows/docker-build.yml`)+- dev branch sync guard (`ensure-dev-synced-with-main.yml`)+- preview branch automation (`dev-preview-branch.yml`)++Observation:++- CI gate is stronger on container buildability than lint/test depth.++## 8) Documentation vs Implementation Drift++Architecture docs are useful but partially stale in folder naming/module layout compared to current collector file structure. Use docs for conceptual understanding, but trust code paths for implementation-level decisions.++## 9) Strengths, Risks, and Practical Guidance++### Strengths++- Clear monorepo separation with shared contracts+- Good real-time architecture: DB persistence + in-memory delta merge + WS push+- Runtime-config-aware frontend deployment model+- Backend isolation by `backendId` is consistently embedded in data path++### Key risks++- Some runtime-critical paths (collector orchestration + WS protocol) are lightly tested+- Potential race/overlap patterns in periodic backend manager loops+- Mixed/legacy code paths can confuse contributors+- Strong frontend/backend contract coupling (especially WS payload shape)++### What to know before editing++1. Shared types in `packages/shared` are cross-app contracts.+2. Collector uses ESM + `.js` extension for relative imports.+3. Web API base and WS endpoints can be runtime-overridden.+4. Keep backend isolation (`backendId`) intact in any new query/write path.+5. Validate changes in both realtime and non-realtime (HTTP fallback) modes.+6. For auth/security changes, verify cookie + bearer compatibility paths.++## 10) Recommended Next Research (if needed)++- Build an endpoint matrix (HTTP + WS) with exact request/response schemas and source functions.+- Add a collector reliability review focused on reconnection/backoff consistency between Clash and Surge paths.+- Add a test-gap plan prioritizing `index.ts` orchestrator and `websocket.ts` protocol behavior.diff --git a/docs/research-GPT5.3.zh.md b/docs/research-GPT5.3.zh.mdnew file mode 100644index 0000000..ed3f6ca--- /dev/null+++ b/docs/research-GPT5.3.zh.md@@ -0,0 +1,406 @@+# Neko Master 单体仓库深度研究(GPT-5.3)++本文是对 `/Users/luolei/DEV/clash-master` 的中文研究整理,覆盖架构、运行机制、代码分层、边界场景和运维细节。++## 1)仓库定位++Neko Master 是一个基于 `pnpm` 的网络流量分析与可视化单体仓库:++- `apps/collector`:后端采集 + API + WebSocket 推送(Fastify + ws + SQLite)+- `apps/web`:Next.js 仪表盘前端(App Router、next-intl、React Query)+- `packages/shared`:前后端共享契约与工具++核心能力:采集 Clash/Mihomo 与 Surge 流量,聚合为多维统计,并提供近实时展示。++## 2)Monorepo 构建与任务编排++### 工作区与任务运行器++- 工作区配置:`pnpm-workspace.yaml`、根目录 `package.json`+- 任务编排:`turbo.json`+- 根命令:+ - `pnpm dev` -> `turbo dev`+ - `pnpm build` -> `turbo build`+ - `pnpm lint` -> `turbo lint`++### Turbo 行为要点++- `build` 按依赖图执行(`^build`),缓存 `dist/**` 与 `.next/**`(排除 `.next/cache/**`)+- `dev` 为长驻任务,不走缓存+- `lint` 同样尊重依赖图(`^lint`)+- 全局缓存失效包含 `**/.env.*local`++## 3)Collector 深入分析(`apps/collector`)++### 3.1 运行拓扑与启动顺序++入口:`apps/collector/src/index.ts`++单进程内承载:++- Fastify API 服务(`APIServer`)+- 独立 WebSocket 服务(`StatsWebSocketServer`)+- 每个后端一个采集器实例(Clash WS 或 Surge 轮询)+- GeoIP 服务(`GeoIPService`)+- Surge 策略同步服务(`SurgePolicySyncService`)++`main()` 启动流程:++1. 加载 `.env.local`(若存在)后再加载 `.env`+2. 初始化 SQLite 门面(`StatsDatabase`)+3. 初始化 GeoIP 服务+4. 启动 WebSocket 服务(默认 `3002`)+5. 启动策略同步服务+6. 启动 Fastify API 服务(默认 `3001`)+7. 立即执行后端管理循环,并每 5 秒轮询一次+8. 安排保留期清理任务(+30 秒首次,之后每 6 小时)+9. 注册优雅退出(`SIGINT`/`SIGTERM`)++### 3.2 多后端生命周期管理++`manageBackends()` 持续将数据库配置与内存采集器状态对齐:++- 对启用且 listening 的后端启动采集器+- 当 `url`、`token`、`type`、`listening`、`enabled` 变化时重启采集器+- 对删除/禁用/非 listening 的后端停止采集器++状态以 `backendId` 为键,保证后端隔离。++### 3.3 Clash 采集链路(WebSocket)++实现:`apps/collector/src/collector.ts`++- 连接上游 `/connections` WebSocket+- 使用 `Map<id, TrackedConnection>` 维护连接快照+- 每次更新计算正向增量(`max(0, current - previous)`)+- 安全忽略无效或空帧+- 清理缺失连接 ID,并周期性移除陈旧连接+- 断连重试为固定间隔(非指数退避)++### 3.4 Surge 采集链路(HTTP 轮询)++实现:`apps/collector/src/surge-collector.ts`++- 轮询 `/v1/requests/recent`(默认约 2 秒)+- 请求失败采用指数退避+- 4xx 在重试路径中视为不可重试+- 以请求 ID 跟踪并计算增量字节+- 处理计数器重置场景(`current < previous`)+- 使用 `recentlyCompleted`(约 5 分钟 TTL)避免已完成请求重复计数+- 从 Surge notes 解析策略/规则链++### 3.5 批量写入、聚合与实时态协同++批缓冲层:`apps/collector/src/batch-buffer.ts`++- 触发 flush 条件:+ - 定时(`FLUSH_INTERVAL_MS`,默认 `30000`)+ - 缓冲阈值(`FLUSH_MAX_BUFFER_SIZE`,默认 `5000`)+- Flush 批量写入流量与国家统计++写入仓库:`apps/collector/src/database/repositories/traffic-writer.repository.ts`++- SQL 写入前先在内存预聚合+- 多聚合表按事务组写入++与实时数据协同:++- 增量同步写入 `realtimeStore`+- DB flush 成功后清理对应实时增量,避免双计+- 采集器触发 WS 广播并带内部节流++### 3.6 API 架构与模块边界++组装入口:`apps/collector/src/app.ts`++- 注册 CORS、cookie 插件、服务注入与模块控制器+- 控制器前缀:+ - `/api/backends`+ - `/api/stats`+ - `/api/auth`+ - `/api/db`+- 兼容路由直接在 `app.ts` 挂载:+ - `/api/gateway/proxies`+ - `/api/gateway/providers/proxies`+ - `/api/gateway/providers/proxies/refresh`+ - `/api/gateway/rules`+ - `/health`++模式:控制器薄编排,服务承载业务,`StatsDatabase` + repository 承载数据访问与转换。++### 3.7 认证与访问控制细节++认证服务:`apps/collector/src/modules/auth/auth.service.ts`++- token 以 SHA-256 哈希存储+- 运行期支持:+ - cookie(`neko-session`)+ - Bearer 令牌回退(`Authorization: Bearer ...`)+- 公共路由跳过鉴权(如 `/health`、`/api/auth/state`、`/api/auth/verify`、`/api/auth/logout`)++特殊模式:++- `FORCE_ACCESS_CONTROL_OFF=true`:关闭鉴权+- `SHOWCASE_SITE_MODE=true`:屏蔽多数写操作并隐藏后端敏感 URL++Cookie Secret 行为:++- 未配置 `COOKIE_SECRET` 时服务端自动生成+- 生产环境若不持久化会导致重启后会话失效++### 3.8 WebSocket 协议与行为++服务:`apps/collector/src/websocket.ts`++- 独立 WS 服务运行于 collector 进程+- 认证支持 cookie 或 query token+- 入站消息类型包含 `ping`、`subscribe`+- 订阅 payload 支持后端、时间范围、细节开关、分页/趋势选项、最小推送间隔+- 连接/订阅后会推送初始数据+- 全局广播节流 + 客户端最小推送间隔双重控制+- 服务器按查询形状缓存结果,避免每客户端重复计算+- 摘要缓存 TTL 随“是否接近当前时刻”动态变化++实时合并策略:仅当查询结束时间接近“当前”时,才合并内存增量(`REALTIME_RANGE_END_TOLERANCE_MS`)。++### 3.9 数据库设计与查询策略++相关文件:`apps/collector/src/database/schema.ts`、`apps/collector/src/db.ts`++设计特征:++- SQLite + WAL + 性能 pragma+- 关键表均按 `backend_id` 作用域隔离+- 累计聚合表与分钟/小时事实表组合++优化要点:++- 范围查询按时间跨度自动选择分钟或小时事实表(常见阈值 6 小时)+- DB 范围查询缓存按时间新鲜度分配短/长 TTL++保留策略:++- 保留配置在 `app_config` / retention API+- 自动清理定期裁剪过旧分钟/小时统计及关联事实数据++### 3.10 GeoIP 子系统++服务:`apps/collector/src/geo-service.ts`++关键行为:++- 私网 IP 快速归类+- 优先查询数据库缓存(`geoip_cache`)+- 并发去重(`pendingQueries`)+- 失败 IP 冷却,防止反复打满上游+- 队列 + 节流控制在线查询++提供者模式:++- 支持 online/local 两种+- local 依赖 MaxMind MMDB(City + ASN)+- local 不可用时回退 online+- 配置接口可返回“配置值”与“生效值”,并校验本地依赖完整性++### 3.11 测试覆盖现状++测试体系:Vitest(`apps/collector/vitest.config.ts`)++- 测试辅助:`apps/collector/src/__tests__/helpers.ts`+- 每轮测试使用临时 SQLite 数据库++已覆盖:++- auth service+- stats service+- traffic writer repository+- geoip 配置与归一化逻辑++薄弱区:++- `index.ts` 启动编排+- Clash/Surge 采集运行时边界行为+- WS 协议集成与鉴权边界+- 后端生命周期循环的竞态场景++### 3.12 风险与注意点++- 控制器中仍有部分手工参数校验,集中式 schema 校验不足+- 定时触发的 `manageBackends` 缺少显式 single-flight,存在并发重叠可能+- 部分模块内存在历史并行路径,可能增加理解成本+- 部分字段以逗号拼接存储,规模扩大后可能影响查询精度与维护性++## 4)Web 前端深入分析(`apps/web`)++### 4.1 路由与多语言++关键文件:++- `apps/web/app/[locale]/layout.tsx`+- `apps/web/app/[locale]/page.tsx`+- `apps/web/app/[locale]/dashboard/page.tsx`+- `apps/web/proxy.ts`+- `apps/web/i18n/routing.ts`+- `apps/web/i18n/request.ts`++行为:++- App Router 多语言路由(`en`、`zh`,默认 `zh`)+- locale 根页直接复用 dashboard+- `proxy.ts` 中 next-intl 中间件处理语言路由与 matcher 排除++### 4.2 数据层与 API 客户端++核心客户端:`apps/web/lib/api.ts`++- 提供类型化 API 门面+- API base 解析优先级:+ 1. `window.__RUNTIME_CONFIG__.API_URL`+ 2. 环境变量(`NEXT_PUBLIC_API_URL` / `API_URL`)+ 3. `/api`+- GET 并发去重,避免重复请求+- 401 触发 `api:unauthorized` 事件,同步全局鉴权状态+- 错误模型使用 `ApiError`(`apps/web/lib/api-error.ts`)++React Query 默认:`staleTime ~5s`、`gcTime ~5m`、`retry 1`,在 `hooks/api/*` 与 `lib/query-config.ts` 可按场景覆写。++### 4.3 Dashboard 组合与状态++核心 hook:`apps/web/app/[locale]/dashboard/hooks/use-dashboard.ts`++- 协调 tab、时间范围、后端选择、翻译、鉴权感知与数据拉取策略+- 根据 tab 与刷新模式组合 HTTP + WS+- 页面由 `Sidebar`、`Header`、`Content` 组合;`Content` 按 tab 切换模块++### 4.4 前端 WebSocket 策略++实现:`apps/web/lib/websocket.ts`++- 端点候选支持运行时/env 配置和推断回退+- 生产环境倾向优先尝试路径式 `/_cm_ws`+- 心跳 ping/pong + 延迟跟踪+- 指数重连 + 端点轮换+- `subscribe` payload 支持精细化控制返回数据形状++多模块会将 WS 更新合并回 React Query 缓存,实现低延迟体验。++### 4.5 鉴权 UX 流程++核心:`apps/web/lib/auth.tsx`++- 全局 provider 决定是否开启鉴权与会话状态+- `AuthGuard` 在必要时展示登录对话框+- logout 清理会话并刷新+- 接收到 API 未授权事件后降级本地鉴权状态++### 4.6 UI 与主题系统++- Tailwind v4(`app/globals.css`、`postcss.config.mjs`)+- shadcn/ui(`apps/web/components.json`,`new-york` 风格)+- 基础组件在 `apps/web/components/ui/*`+- next-themes 管理主题与主题色同步+- framer-motion 在关键交互点按需使用++### 4.7 PWA / Service Worker++相关文件:++- `apps/web/app/manifest.ts`+- `apps/web/public/sw.js`+- `apps/web/components/common/sw-register.tsx`+- `apps/web/next.config.ts`++注意:自定义 SW 逻辑与生产环境 next-pwa 插件并存,不同环境行为需专项验证。++### 4.8 前端风险点++- 某些视图可能并发建立多个 WS 连接+- 与后端 WS payload(尤其 `stats` 可选字段)耦合较深+- 语言切换依赖路径字符串规则,未来路由形状变化需谨慎+- viewport 禁止用户缩放,存在可访问性权衡++## 5)共享包分析(`packages/shared`)++主文件:`packages/shared/src/index.ts`++提供:统计、网关 payload、WS 更新、鉴权、Surge 模型等共享契约,以及工具导出:++- `gateway-utils.ts`+- `geo-ip-utils.ts`++关键工具:++- `buildGatewayHeaders`:按网关类型构造请求头+- `parseSurgeRule` / `parseGatewayRule`:规则归一化解析+- `getGatewayBaseUrl`:归一化提取网关基础 URL+- `normalizeGeoIP`:兼容不同 Geo payload 形态++注意:`workspace:*` 依赖意味着 shared 改动会立即影响两端,必须按跨应用契约变更对待。++## 6)部署与运行时配置++### 6.1 Next 重写与运行时配置++- 浏览器 API 常走 `/api/*`+- Next rewrites 将 `/api/:path*` 转发到 `API_URL`+- `public/runtime-config.js` 支持部署时覆盖,无需重建镜像++### 6.2 Docker 与启动脚本++相关文件:`Dockerfile`、`docker-compose.yml`、`docker-start.sh`、`.dockerignore`++关键行为:++- 若缺失 `COOKIE_SECRET`,启动脚本会在挂载数据卷中自动生成并持久化+- 容器启动时动态注入 runtime config+- 镜像采用多阶段构建,CI 支持多架构++## 7)Git Hook 与 CI 质量门禁++`/.husky/pre-push`(仅 push 到 `main` 时生效):++- collector: `tsc --noEmit`+- web: 生产构建++GitHub Actions 包含:++- Docker build/publish+- dev 与 main 同步检查+- dev preview 分支自动化++观察:当前 CI 对容器可构建性要求较强,但 lint/test 深度仍有提升空间。++## 8)文档与实现偏差++架构文档对概念层面有价值,但部分目录命名与 collector 当前实现存在时序差异;做实现级决策时以代码路径为准。++## 9)优势、风险与实操建议++### 优势++- monorepo 分层清晰,shared 契约明确+- 实时架构合理(DB 持久化 + 内存增量 + WS 推送)+- 前端部署支持 runtime config,运维灵活+- `backendId` 隔离贯穿数据链路++### 主要风险++- 采集编排与 WS 协议等关键路径测试覆盖偏薄+- 后端管理循环存在潜在并发重叠+- 历史/并行代码路径增加维护复杂度+- 前后端协议(尤其 WS)耦合较强++### 修改前建议++1. `packages/shared` 类型是跨应用契约,改动要联动验证。+2. collector 为 ESM,相对导入需 `.js` 后缀。+3. Web API 基址与 WS 端点支持运行时覆盖,排查问题先确认生效配置。+4. 新增读写链路必须保持 `backendId` 隔离。+5. 同时验证实时模式与 HTTP 回退模式。+6. 涉及鉴权时同时验证 cookie 与 bearer 路径。++## 10)后续可选研究方向++- 建立 HTTP + WS 端点矩阵(请求/响应 schema + 源函数映射)+- 增加采集可靠性专项评估(Clash 与 Surge 在重连/退避策略的一致性)+- 制定测试补齐计划,优先 `index.ts` 编排与 `websocket.ts` 协议行为diff --git a/docs/research-glm5.en.md b/docs/research-glm5.en.mdnew file mode 100644index 0000000..37e4e5b--- /dev/null+++ b/docs/research-glm5.en.md@@ -0,0 +1,752 @@+# Neko Master - Comprehensive Codebase Research Report++## Executive Summary++**Neko Master** is a modern network traffic analytics platform designed for edge gateways. It provides real-time monitoring, traffic auditing, and multi-gateway management capabilities. The system collects traffic data from Clash/Mihomo and Surge proxy gateways, aggregates it in SQLite, and visualizes it through a Next.js web interface.++---++## 1. Project Overview++### 1.1 Purpose & Scope++Neko Master serves as a traffic visibility layer for proxy gateways. It does NOT:+- Provide network access or proxy services+- Handle subscription management+- Perform traffic routing++It DOES:+- Collect and aggregate traffic statistics from gateways+- Provide real-time WebSocket updates to connected clients+- Offer historical traffic analysis and visualization+- Support multiple backend gateways simultaneously++### 1.2 Technology Stack++| Layer | Technology |+|-------|------------|+| Frontend | Next.js 16, React 19, TypeScript, Tailwind CSS v4 |+| UI Components | shadcn/ui (new-york style), lucide-react icons |+| Charts | Recharts, D3.js, react-simple-maps |+| State Management | TanStack React Query, WebSocket hooks |+| Backend | Fastify 5, WebSocket (ws), better-sqlite3 |+| GeoIP | maxmind (local MMDB), online API fallback |+| i18n | next-intl (zh/en) |+| Build | Turbo monorepo, pnpm workspaces |+| Deployment | Docker, standalone Next.js output |++### 1.3 Repository Structure++```+neko-master/+├── apps/+│ ├── collector/ # Backend data collection service+│ │ └── src/+│ │ ├── index.ts # Entry point, orchestrates all services+│ │ ├── app.ts # Fastify API server+│ │ ├── websocket.ts # WebSocket broadcast server+│ │ ├── collector.ts # Clash WebSocket collector+│ │ ├── surge-collector.ts # Surge HTTP polling collector+│ │ ├── realtime.ts # In-memory traffic aggregation+│ │ ├── batch-buffer.ts # Batch write optimization+│ │ ├── geo-service.ts # GeoIP lookup service+│ │ ├── db.ts # Database facade+│ │ └── database/+│ │ ├── schema.ts # DDL definitions+│ │ └── repositories/ # Repository pattern implementations+│ │+│ └── web/ # Next.js frontend+│ └── app/[locale]/+│ ├── dashboard/ # Main dashboard page+│ └── layout.tsx # Root layout with providers+│+├── packages/+│ └── shared/ # Shared TypeScript types & utilities+│ └── src/+│ ├── index.ts # Type exports+│ ├── gateway-utils.ts # Gateway API helpers+│ └── geo-ip-utils.ts # GeoIP data normalization+│+└── docs/+ └── architecture.md # Chinese architecture documentation+```++---++## 2. Core Architecture++### 2.1 High-Level Data Flow++```+┌─────────────────────┐+│ Clash / Mihomo │──── WebSocket ────┐+│ Gateway │ │+└─────────────────────┘ │+ ▼+┌─────────────────────┐ ┌─────────────────┐+│ Surge Gateway │──── HTTP ──▶│ Collector │+│ │ Polling │ Service │+└─────────────────────┘ │ (Port 3001) │+ └────────┬────────┘+ │+ ┌─────────────┼─────────────┐+ ▼ ▼ ▼+ ┌──────────┐ ┌──────────┐ ┌──────────┐+ │ SQLite │ │ Realtime │ │ WS │+ │ (WAL) │ │ Store │ │ Server │+ └──────────┘ └──────────┘ │(Port 3002)│+ │ │ └──────────┘+ │ │ │+ ▼ ▼ │+ ┌─────────────────────────┐ │+ │ Next.js Web UI │◀───┘+ │ (Port 3000) │+ └─────────────────────────┘+```++### 2.2 Three-Tier Architecture++1. **Frontend Tier** (Next.js)+ - React Query for HTTP API data fetching+ - WebSocket hook for real-time updates+ - Client-side state management++2. **Collector Tier** (Fastify + WebSocket)+ - REST API endpoints (`/api/*`)+ - WebSocket server for real-time push+ - Gateway collectors (Clash WS + Surge HTTP)++3. **Storage Tier** (SQLite + Memory)+ - SQLite with WAL mode for persistence+ - In-memory RealtimeStore for low-latency queries++---++## 3. Backend Deep Dive++### 3.1 Entry Point (`index.ts`)++The main entry point orchestrates all services:++1. **Database initialization** - Creates/opens SQLite database+2. **GeoIP service** - Initializes IP geolocation service+3. **WebSocket server** - Starts WS server on port 3002+4. **API server** - Starts Fastify on port 3001+5. **Backend management loop** - Polls every 5s for backend config changes+6. **Auto-cleanup job** - Runs every 6 hours for data retention++```typescript+// Key initialization sequence+async function main() {+ db = new StatsDatabase(DB_PATH);+ geoService = new GeoIPService(db);+ wsServer = new StatsWebSocketServer(COLLECTOR_WS_PORT, db);+ apiServer = new APIServer(API_PORT, db, realtimeStore, policySyncService);++ // Start collectors based on database config+ setInterval(manageBackends, 5000);+}+```++### 3.2 Clash Collector (`collector.ts`)++**Connection Protocol:**+- Connects to `ws://gateway:port/connections`+- Uses Bearer token authentication if provided+- Auto-reconnects on disconnect with 5s interval++**Traffic Tracking Algorithm:**++```+For each WebSocket message:+ 1. Parse connections array+ 2. For new connections:+ - Track initial traffic values+ - Record to batch buffer+ - Record to realtime store+ 3. For existing connections:+ - Calculate delta: current - last+ - Record delta to buffer and realtime+ - Update last seen timestamp+ 4. For disappeared connections:+ - Remove from tracking map+ 5. Flush batch buffer if >= 5000 items+```++**Stale Connection Cleanup:**+- Removes connections not seen for 5 minutes+- Runs every 2 minutes++### 3.3 Surge Collector (`surge-collector.ts`)++**Different from Clash:**+- Uses HTTP polling (GET `/v1/requests/recent`) instead of WebSocket+- Default 2s polling interval+- Exponential backoff on errors (max 5 retries, up to 60s delay)++**Policy Path Extraction:**+Surge provides policy decision paths in the `notes` field:+```+"[Rule] Policy decision path: rule -> group1 -> group2 -> finalProxy"+```++This is parsed and reversed to match Clash's chain format:+```+chains = [finalProxy, ..., group2, group1, rule]+```++**Duplicate Prevention:**+- Uses `recentlyCompleted` Map with 5-minute TTL+- Prevents double-counting completed requests that may reappear in polls++### 3.4 RealtimeStore (`realtime.ts`)++The RealtimeStore is a critical component for low-latency queries. It maintains:++| Map | Purpose | Max Entries |+|-----|---------|-------------|+| `summaryByBackend` | Total upload/download/connections delta | Per backend |+| `minuteByBackend` | Per-minute traffic buckets | 180 minutes default |+| `domainByBackend` | Per-domain traffic deltas | 50,000 |+| `ipByBackend` | Per-IP traffic deltas | 50,000 |+| `proxyByBackend` | Per-proxy traffic deltas | Unlimited |+| `deviceByBackend` | Per-device (source IP) traffic | Unlimited |+| `ruleByBackend` | Per-rule traffic deltas | Unlimited |+| `countryByBackend` | Per-country traffic deltas | Unlimited |++**Key Operations:**++1. **`recordTraffic()`** - Adds traffic delta to all relevant maps+2. **`merge*()` methods** - Combines DB data with realtime deltas+3. **`clearTraffic()`** - Called after batch flush+4. **`pruneIfNeeded()`** - Evicts lowest-traffic entries when limits exceeded++**Memory Bounds:**+- Maximum 50,000 domain entries+- Maximum 50,000 IP entries+- Maximum 10,000 device detail entries+- Eviction removes bottom 25% by traffic++### 3.5 BatchBuffer (`batch-buffer.ts`)++Optimizes database writes by:++1. **Aggregating by composite key:**+ ```+ backendId:minute:domain:ip:chain:fullChain:rule:rulePayload:sourceIP+ ```++2. **Batching flushes:**+ - Every 30 seconds (configurable via `FLUSH_INTERVAL_MS`)+ - When buffer reaches 5000 items (configurable via `FLUSH_MAX_BUFFER_SIZE`)++3. **Two-phase write:**+ - Traffic stats → `db.batchUpdateTrafficStats()`+ - Country stats → `db.batchUpdateCountryStats()`++### 3.6 WebSocket Server (`websocket.ts`)++**Connection Lifecycle:**++1. Client connects with optional token (URL param or cookie)+2. Server validates auth if enabled+3. Client sends `subscribe` message with options:+ - `backendId` - Which backend to subscribe to+ - `start/end` - Time range for historical data+ - `minPushIntervalMs` - Throttle push frequency+ - Various detail inclusion flags++**Broadcast Mechanism:**++```typescript+broadcastStats(changedBackendId?: number, force = false) {+ // Throttle: max 1 broadcast per second+ if (now - lastBroadcastTime < 1000 && !force) return;++ // Per-client push respecting minPushIntervalMs+ for each client:+ if client.minPushIntervalMs > 0 && elapsed < minPushIntervalMs:+ skip++ // Resolve backend+ if changedBackendId !== client.backendId:+ skip++ // Get stats and send+ stats = getStatsForBackend(...)+ ws.send(JSON.stringify({ type: 'stats', data: stats }))+}+```++**Summary Caching:**+- 2-second TTL for realtime ranges+- 5-minute TTL for historical ranges+- Cache key includes all query parameters for deduplication++### 3.7 Database Layer++#### 3.7.1 Schema Overview++**Primary Tables:**++| Table | Purpose | Primary Key |+|-------|---------|-------------|+| `domain_stats` | Per-domain aggregation | (backend_id, domain) |+| `ip_stats` | Per-IP aggregation | (backend_id, ip) |+| `proxy_stats` | Per-proxy aggregation | (backend_id, chain) |+| `rule_stats` | Per-rule aggregation | (backend_id, rule) |+| `country_stats` | Per-country aggregation | (backend_id, country) |+| `device_stats` | Per-source-IP aggregation | (backend_id, source_ip) |++**Fact Tables (for range queries):**++| Table | Granularity | Use Case |+|-------|-------------|----------|+| `minute_dim_stats` | Per-minute | Short-range queries (<6h) |+| `hourly_dim_stats` | Per-hour | Long-range queries (>6h) |+| `minute_country_stats` | Per-minute country | Short-range country queries |+| `hourly_country_stats` | Per-hour country | Long-range country queries |++**Cross-reference Tables:**++| Table | Purpose |+|-------|---------|+| `domain_proxy_stats` | Domain × Proxy traffic |+| `ip_proxy_stats` | IP × Proxy traffic |+| `device_domain_stats` | Device × Domain traffic |+| `device_ip_stats` | Device × IP traffic |+| `rule_chain_traffic` | Rule × Chain traffic |+| `rule_domain_traffic` | Rule × Domain traffic |+| `rule_ip_traffic` | Rule × IP traffic |++#### 3.7.2 Repository Pattern++The database uses a facade pattern (`StatsDatabase`) that delegates to specialized repositories:++```typescript+class StatsDatabase {+ public readonly repos = {+ auth: AuthRepository,+ surge: SurgeRepository,+ timeseries: TimeseriesRepository,+ country: CountryRepository,+ device: DeviceRepository,+ proxy: ProxyRepository,+ rule: RuleRepository,+ ip: IPRepository,+ config: ConfigRepository,+ trafficWriter: TrafficWriterRepository,+ domain: DomainRepository,+ backend: BackendRepository,+ };+}+```++Each repository extends `BaseRepository` which provides shared utilities:+- `parseMinuteRange()` - Time range parsing+- `splitChainParts()` - Proxy chain parsing+- `expandShortChainsForRules()` - Rule chain expansion+- `aggregateProxyStatsByFirstHop()` - Proxy aggregation++#### 3.7.3 Query Cache++Database-level range query cache:+- Default TTL: 8 seconds for realtime, 5 minutes for historical+- Maximum 1024 entries+- Configurable via environment variables++### 3.8 GeoIP Service (`geo-service.ts`)++**Dual Provider Support:**++1. **Online API** (default)+ - Endpoint: `https://api.ipinfo.es/ipinfo?ip=xxx`+ - Rate-limited: 100ms minimum between requests+ - 8-second timeout per request++2. **Local MMDB** (optional)+ - Requires: `GeoLite2-City.mmdb`, `GeoLite2-ASN.mmdb`+ - Optional: `GeoLite2-Country.mmdb`+ - No rate limiting+ - Hot-reload when files change++**Lookup Flow:**++```+getGeoLocation(ip):+ 1. Check if private IP → return "LOCAL"+ 2. Check database cache → return cached+ 3. Check failed IPs cooldown → return null+ 4. Check pending queries → return existing promise+ 5. Queue new lookup+ 6. Process queue with rate limiting+ 7. Save to database cache+```++**Failure Handling:**+- Failed IPs are cached for 30 minutes+- Queue overflow (100 items) drops new lookups+- Cleanup runs every 10 minutes++---++## 4. Frontend Deep Dive++### 4.1 Next.js Configuration (`next.config.ts`)++**Key Features:**++1. **API Proxying:**+ ```typescript+ // Rewrites /api/* to collector service+ rewrites: [{+ source: "/api/:path*",+ destination: `${API_URL}/api/:path*`+ }]+ ```++2. **PWA Support:**+ - Enabled only in production+ - Service worker with skip waiting+ - Cache-Control headers for sw.js++3. **Environment Variables:**+ - `NEXT_PUBLIC_APP_VERSION` - From package.json+ - `NEXT_PUBLIC_WS_PORT` - WebSocket port (default 3002)++### 4.2 Dashboard Page Structure++```+DashboardPage+├── Sidebar (navigation, backend status)+├── Header (controls, time range, refresh)+├── Content (tab-based content)+│ ├── Overview Tab (summary cards, top items)+│ ├── Domains Tab (paginated domain list)+│ ├── IPs Tab (paginated IP list with GeoIP)+│ ├── Proxies Tab (proxy traffic breakdown)+│ ├── Rules Tab (rule chain flow visualization)+│ ├── Devices Tab (per-device traffic)+│ └── Regions Tab (world map, country stats)+└── BackendConfigDialog (gateway setup)+```++### 4.3 Data Fetching Patterns++**HTTP API (React Query):**+```typescript+// Hook pattern+export function useTrafficTrend(backendId, params) {+ return useQuery({+ queryKey: ['trafficTrend', backendId, params],+ queryFn: () => api.getTrafficTrendAggregated(backendId, params),+ staleTime: 5000,+ });+}+```++**WebSocket (Custom Hook):**+```typescript+// Real-time subscription+useStatsWebSocket({+ backendId,+ timeRange,+ onMessage: (data) => {+ // Update local state+ }+});+```++### 4.4 Internationalization++- Uses `next-intl` package+- Two locales: `zh` (default), `en`+- Translation files: `apps/web/messages/{zh,en}.json`+- Hook: `useTranslations('namespace')`++---++## 5. Multi-Backend Architecture++### 5.1 Backend Isolation++All data is isolated by `backend_id`:++- **Database:** All tables have `backend_id` in primary key+- **RealtimeStore:** Maps are keyed by backend ID+- **WebSocket:** Clients subscribe to specific backend++### 5.2 Backend Management++```typescript+// Backend management loop (every 5s)+manageBackends():+ 1. Get all backend configs from DB+ 2. For each backend:+ - If config changed → restart collector+ - If enabled/listening changed → start/stop collector+ 3. Stop collectors for deleted backends+```++### 5.3 Backend Types++| Type | Protocol | Data Source |+|------|----------|-------------|+| `clash` | WebSocket | `/connections` endpoint |+| `surge` | HTTP Polling | `/v1/requests/recent` endpoint |++---++## 6. Authentication & Security++### 6.1 Cookie-Based Auth++1. User sets token in settings UI+2. Server hashes token and stores in `auth_config` table+3. Login endpoint sets `HttpOnly` cookie `neko-session`+4. Subsequent requests include cookie automatically++### 6.2 Auth Middleware++```typescript+// Fastify hook+app.addHook('onRequest', async (request, reply) => {+ // Skip public routes+ if (isPublicRoute(request.url)) return;++ // Check if auth required+ if (!authService.isAuthRequired()) return;++ // Validate cookie or Bearer token+ const token = request.cookies['neko-session']+ || extractBearerToken(request);+ if (!valid) return 401;+});+```++### 6.3 Emergency Access++- `FORCE_ACCESS_CONTROL_OFF=true` bypasses auth+- Used for password recovery+- Should NOT be enabled in production++---++## 7. Performance Optimizations++### 7.1 Database Level++| Optimization | Setting |+|--------------|---------|+| WAL Mode | `journal_mode = WAL` |+| Sync Mode | `synchronous = NORMAL` |+| Cache Size | 64MB (`cache_size = -65536`) |+| Temp Store | Memory (`temp_store = MEMORY`) |+| Busy Timeout | 5 seconds |++### 7.2 Application Level++| Strategy | Implementation |+|----------|----------------|+| Batch Writes | 30s interval or 5000 items |+| Query Cache | 8s realtime / 5min historical |+| RealtimeStore | In-memory delta aggregation |+| WebSocket Throttle | 1s minimum between broadcasts |+| GeoIP Rate Limit | 100ms between online API calls |++### 7.3 Memory Management++- RealtimeStore entry limits (50K domains, 50K IPs)+- Eviction removes bottom 25% by traffic+- Stale connection cleanup (5-minute timeout)++---++## 8. Data Retention++### 8.1 Configuration++Stored in `app_config` table:+- `retention.connection_logs_days` - Minute-level data (default: 7)+- `retention.hourly_stats_days` - Hourly aggregations (default: 30)+- `retention.auto_cleanup` - Enable automatic cleanup (default: true)++### 8.2 Cleanup Process++```typescript+// Runs every 6 hours+runAutoCleanup():+ 1. Get retention config+ 2. Calculate cutoff dates+ 3. Delete old minute_stats records+ 4. Delete old hourly_stats records+```++---++## 9. Environment Variables++### 9.1 Core Variables++| Variable | Default | Description |+|----------|---------|-------------|+| `NODE_ENV` | - | Environment mode |+| `DB_PATH` | `./stats.db` | SQLite database path |+| `COOKIE_SECRET` | auto-generated | Cookie signing key |+| `API_PORT` | 3001 | Fastify API port |+| `COLLECTOR_WS_PORT` | 3002 | WebSocket server port |+| `WEB_PORT` | 3000 | Next.js port |++### 9.2 GeoIP Variables++| Variable | Default | Description |+|----------|---------|-------------|+| `GEOIP_LOOKUP_PROVIDER` | `online` | `online` or `local` |+| `GEOIP_ONLINE_API_URL` | `https://api.ipinfo.es/ipinfo` | Online API endpoint |++### 9.3 Tuning Variables++| Variable | Default | Description |+|----------|---------|-------------|+| `FLUSH_INTERVAL_MS` | 30000 | Batch flush interval |+| `FLUSH_MAX_BUFFER_SIZE` | 5000 | Max buffer size before flush |+| `REALTIME_MAX_MINUTES` | 180 | RealtimeStore time window |+| `DB_RANGE_QUERY_CACHE_TTL_MS` | 8000 | Query cache TTL (realtime) |+| `DB_HISTORICAL_QUERY_CACHE_TTL_MS` | 300000 | Query cache TTL (historical) |++---++## 10. Testing++### 10.1 Test Framework++- **Framework:** Vitest+- **Location:** `apps/collector/src/**/*.test.ts`+- **Patterns:**+ - Unit tests for repositories+ - Service tests with mock databases+ - Integration tests for collectors++### 10.2 Running Tests++```bash+# All tests+pnpm --filter @neko-master/collector test++# Single file+pnpm --filter @neko-master/collector test -- src/modules/auth/auth.service.test.ts++# Watch mode+pnpm --filter @neko-master/collector test:watch+```++---++## 11. Docker Deployment++### 11.1 Container Architecture++Single container runs both services:+1. Next.js (port 3000) - Web UI+2. Fastify (port 3001) - API+3. WebSocket (port 3002) - Real-time++### 11.2 Docker Compose (Recommended)++```yaml+services:+ neko-master:+ image: foru17/neko-master:latest+ ports:+ - "3000:3000"+ - "3002:3002"+ volumes:+ - ./data:/app/data+ - ./geoip:/app/data/geoip:ro+ environment:+ - COOKIE_SECRET=your-secret+```++### 11.3 Reverse Proxy Setup++Nginx example:+```nginx+location / {+ proxy_pass http://localhost:3000;+}++location /_cm_ws {+ proxy_pass http://localhost:3002;+ proxy_http_version 1.1;+ proxy_set_header Upgrade $http_upgrade;+ proxy_set_header Connection "upgrade";+}+```++---++## 12. Key Design Patterns++### 12.1 Repository Pattern++Separates data access logic from business logic. Each repository handles a specific domain:+- `DomainRepository` - Domain statistics+- `IPRepository` - IP statistics + GeoIP+- `RuleRepository` - Rule statistics + chain flow+- etc.++### 12.2 Facade Pattern++`StatsDatabase` class acts as a facade, providing a simplified interface to the complex repository subsystem.++### 12.3 Observer Pattern++WebSocket server broadcasts updates to subscribed clients when new traffic data arrives.++### 12.4 Strategy Pattern++GeoIP service uses strategy pattern for provider selection (online vs local MMDB).++### 12.5 Batch Aggregation Pattern++Traffic updates are buffered and aggregated before database writes, reducing I/O by ~90%.++---++## 13. Potential Improvements++### 13.1 Identified Areas++1. **Type Safety:**+ - Some `any` types remain in test files (allowed by ESLint config)+ - GeoIP API responses could use stricter typing++2. **Error Handling:**+ - Some error paths log but don't propagate errors+ - Could benefit from structured error types++3. **Performance:**+ - RealtimeStore could use LRU cache instead of eviction+ - WebSocket message serialization could be optimized++4. **Testing:**+ - Frontend lacks test coverage+ - Could add E2E tests++5. **Documentation:**+ - API documentation could be generated (OpenAPI/Swagger)+ - Inline code comments could be expanded++---++## 14. Conclusion++Neko Master is a well-architected traffic analytics platform with:++- **Clean separation of concerns** between data collection, storage, and visualization+- **Efficient data pipeline** with batch aggregation and realtime caching+- **Flexible multi-backend support** for different gateway types+- **Production-ready deployment** with Docker and reverse proxy support+- **Good performance characteristics** through careful optimization++The codebase demonstrates strong TypeScript practices, modular design, and attention to operational concerns like data retention and graceful shutdown.diff --git a/docs/research-glm5.zh.md b/docs/research-glm5.zh.mdnew file mode 100644index 0000000..60ece90--- /dev/null+++ b/docs/research-glm5.zh.md@@ -0,0 +1,361 @@+# Neko Master 代码库综合研究报告(中文)++## 执行摘要++**Neko Master** 是一个面向边缘网关的现代网络流量分析平台,提供实时监控、流量审计与多网关管理能力。系统从 Clash/Mihomo 与 Surge 采集流量,聚合到 SQLite,并通过 Next.js 前端可视化。++---++## 1. 项目总览++### 1.1 目标与边界++Neko Master 是“流量可观测层”,它不提供代理接入或路由能力,而是聚焦:++- 从网关采集并聚合流量统计+- 向客户端推送实时 WebSocket 更新+- 提供历史流量分析与可视化+- 同时管理多个后端网关++### 1.2 技术栈++| 层 | 技术 |+|---|---|+| 前端 | Next.js 16、React 19、TypeScript、Tailwind CSS v4 |+| UI | shadcn/ui(new-york)、lucide-react |+| 图表 | Recharts、D3.js、react-simple-maps |+| 状态管理 | TanStack React Query、WebSocket hooks |+| 后端 | Fastify 5、ws、better-sqlite3 |+| GeoIP | maxmind(本地 MMDB)+ 在线回退 |+| i18n | next-intl(zh/en) |+| 工程化 | Turbo monorepo、pnpm workspaces |+| 部署 | Docker、Next.js standalone |++### 1.3 仓库结构++```text+neko-master/+├── apps/+│ ├── collector/ # 后端采集服务+│ │ └── src/+│ │ ├── index.ts # 入口,编排全部服务+│ │ ├── app.ts # Fastify API+│ │ ├── websocket.ts # WebSocket 广播+│ │ ├── collector.ts # Clash WS 采集+│ │ ├── surge-collector.ts # Surge 轮询采集+│ │ ├── realtime.ts # 内存实时聚合+│ │ ├── batch-buffer.ts # 批量写入优化+│ │ ├── geo-service.ts # GeoIP 服务+│ │ ├── db.ts # 数据库门面+│ │ └── database/ # schema + repositories+│ └── web/ # Next.js 前端+├── packages/+│ └── shared/ # 共享类型与工具+└── docs/+```++---++## 2. 核心架构++### 2.1 高层数据流++```text+Clash/Mihomo --WebSocket--> Collector+Surge --HTTP Poll--> Collector++Collector -> SQLite (持久化)+Collector -> RealtimeStore (低延迟)+Collector -> WS Server (实时推送)++Web UI 通过 HTTP + WebSocket 消费数据+```++### 2.2 三层模型++1. 前端层(Next.js):HTTP 拉取 + WS 实时更新。+2. 采集层(Fastify + ws):API、广播、网关采集。+3. 存储层(SQLite + 内存):持久化 + 低延迟查询。++---++## 3. 后端深度分析++### 3.1 入口编排(`index.ts`)++启动流程包含:++1. 初始化数据库+2. 初始化 GeoIP+3. 启动 WS 服务(3002)+4. 启动 API(3001)+5. 每 5 秒执行后端管理循环+6. 每 6 小时执行自动清理++### 3.2 Clash 采集器(`collector.ts`)++- 连接 `ws://.../connections`+- 支持 Bearer 认证+- 断线后 5 秒重连+- 对每条连接计算流量增量并写入 batch + realtime+- 清理超时未出现连接(默认 5 分钟)++### 3.3 Surge 采集器(`surge-collector.ts`)++- 轮询 `GET /v1/requests/recent`+- 默认 2 秒,错误时指数退避+- 解析 `notes` 中策略路径并转换为统一链路格式+- 用 `recentlyCompleted`(5 分钟 TTL)防止重复计数++### 3.4 RealtimeStore(`realtime.ts`)++维护多个按 `backendId` 分片的实时 map:summary、minute、domain、ip、proxy、device、rule、country。++核心能力:++- `recordTraffic()` 写入实时增量+- `merge*()` 与数据库结果合并+- `clearTraffic()` 在 flush 后清空对应增量+- 超限时按流量淘汰低价值条目(底部 25%)++### 3.5 BatchBuffer(`batch-buffer.ts`)++- 复合键聚合:`backendId + minute + domain + ip + chain + rule + sourceIP...`+- flush 条件:30 秒或 5000 条+- 分两阶段写入:流量统计 + 国家统计++### 3.6 WebSocket 服务(`websocket.ts`)++连接与订阅流程:++1. 客户端连接(可通过 query token/cookie)+2. 服务端鉴权+3. 客户端发送 `subscribe`(后端、时间范围、最小推送间隔等)+4. 服务端按订阅推送 `stats`++机制细节:++- 全局广播节流(最小 1 秒)+- 客户端级最小推送间隔+- 摘要缓存:实时范围短 TTL,历史范围长 TTL++### 3.7 数据库层++#### 核心表++| 表 | 作用 |+|---|---|+| `domain_stats` | 域名聚合 |+| `ip_stats` | IP 聚合 |+| `proxy_stats` | 代理链路聚合 |+| `rule_stats` | 规则聚合 |+| `country_stats` | 国家聚合 |+| `device_stats` | 设备(源 IP)聚合 |++#### 事实表(范围查询)++| 表 | 粒度 | 场景 |+|---|---|---|+| `minute_dim_stats` | 分钟 | 短时间范围(<6h) |+| `hourly_dim_stats` | 小时 | 长时间范围(>6h) |++#### 设计模式++- `StatsDatabase` 门面 + 多仓库分域实现+- `BaseRepository` 提供通用解析和聚合工具+- DB 范围查询缓存:实时约 8 秒、历史约 5 分钟++### 3.8 GeoIP 服务(`geo-service.ts`)++双提供者:++1. 在线 API(默认)+2. 本地 MMDB(可选)++查找流程:私网短路 -> DB 缓存 -> 失败冷却 -> 并发去重 -> 队列限流查询 -> 回写缓存。++容错策略:++- 失败 IP 30 分钟冷却+- 队列溢出丢弃新请求+- 周期清理失败记录++---++## 4. 前端深度分析++### 4.1 Next 配置(`next.config.ts`)++- `/api/*` rewrite 到 collector+- PWA 仅生产启用+- 注入版本号与 WS 端口等环境变量++### 4.2 Dashboard 结构++`DashboardPage` 由 `Sidebar`、`Header`、`Content` 组成,`Content` 按 tab 切换:Overview、Domains、IPs、Proxies、Rules、Devices、Regions。++### 4.3 数据获取模式++- HTTP:React Query 拉取历史和分页数据+- WS:自定义 hook 订阅实时数据并更新本地状态/缓存++### 4.4 国际化++- `next-intl`+- 语言:`zh`(默认)、`en`+- 文案:`apps/web/messages/{zh,en}.json`++---++## 5. 多后端架构++### 5.1 后端隔离++- 数据库:主键包含 `backend_id`+- RealtimeStore:按后端分片+- WebSocket:客户端订阅特定后端++### 5.2 后端管理循环++每 5 秒执行:读取配置 -> 判断是否变更 -> 重启/启停采集器 -> 清理已删除后端实例。++### 5.3 后端类型++| 类型 | 协议 | 数据源 |+|---|---|---|+| `clash` | WebSocket | `/connections` |+| `surge` | HTTP 轮询 | `/v1/requests/recent` |++---++## 6. 认证与安全++### 6.1 Cookie 鉴权++1. 用户在设置中配置 token+2. 服务端 hash 后存入 `auth_config`+3. 登录后签发 `HttpOnly` cookie `neko-session`+4. 后续请求自动携带 cookie++### 6.2 鉴权中间件++- 公共路由跳过+- 当开启鉴权时,验证 cookie 或 Bearer token+- 校验失败返回 401++### 6.3 紧急访问模式++- `FORCE_ACCESS_CONTROL_OFF=true` 可临时关闭鉴权+- 仅用于恢复场景,不建议生产启用++---++## 7. 性能优化++### 7.1 DB 级++- WAL 模式+- `synchronous=NORMAL`+- 增大 cache+- 内存临时表+- busy timeout++### 7.2 应用级++- 批量写入(30 秒或 5000 条)+- 查询缓存(实时短 TTL / 历史长 TTL)+- 内存 RealtimeStore 增量聚合+- WS 广播节流+- GeoIP 查询限速++### 7.3 内存治理++- domain/ip 上限 50K+- 超限淘汰低流量项+- 连接陈旧清理++---++## 8. 数据保留++配置存储于 `app_config`:++- `retention.connection_logs_days`+- `retention.hourly_stats_days`+- `retention.auto_cleanup`++自动清理每 6 小时执行一次,按阈值删除过期 minute/hourly 数据。++---++## 9. 环境变量++### 9.1 核心变量++| 变量 | 默认值 | 说明 |+|---|---|---|+| `DB_PATH` | `./stats.db` | SQLite 路径 |+| `COOKIE_SECRET` | 自动生成 | Cookie 签名密钥 |+| `API_PORT` | 3001 | API 端口 |+| `COLLECTOR_WS_PORT` | 3002 | WS 端口 |+| `WEB_PORT` | 3000 | Web 端口 |++### 9.2 调优变量++| 变量 | 默认值 | 说明 |+|---|---|---|+| `FLUSH_INTERVAL_MS` | 30000 | flush 间隔 |+| `FLUSH_MAX_BUFFER_SIZE` | 5000 | 缓冲上限 |+| `REALTIME_MAX_MINUTES` | 180 | 实时窗口 |+| `DB_RANGE_QUERY_CACHE_TTL_MS` | 8000 | 实时查询缓存 TTL |+| `DB_HISTORICAL_QUERY_CACHE_TTL_MS` | 300000 | 历史查询缓存 TTL |++---++## 10. 测试++- 框架:Vitest+- 位置:`apps/collector/src/**/*.test.ts`+- 以仓库与服务层测试为主,前端自动化测试仍偏少++常用命令:++```bash+pnpm --filter @neko-master/collector test+pnpm --filter @neko-master/collector test -- src/modules/auth/auth.service.test.ts+pnpm --filter @neko-master/collector test:watch+```++---++## 11. Docker 部署++- 单容器运行:Web(3000)+ API(3001)+ WS(3002)+- 推荐使用 `docker-compose` 挂载 `data` 与可选 `geoip` 目录+- 反向代理需正确转发 `/_cm_ws` 并设置 Upgrade 头++---++## 12. 关键设计模式++1. Repository Pattern:分离数据访问与业务逻辑+2. Facade Pattern:`StatsDatabase` 统一入口+3. Observer Pattern:WS 广播订阅更新+4. Strategy Pattern:GeoIP 在线/本地策略切换+5. Batch Aggregation Pattern:批量聚合写入,显著降低 I/O++---++## 13. 可改进方向++1. 类型安全:GeoIP 返回类型可再收紧+2. 错误体系:可引入更统一的结构化错误模型+3. 性能:RealtimeStore 可探索 LRU 替代当前淘汰策略+4. 测试:补齐前端与端到端覆盖+5. 文档:可生成 OpenAPI/Swagger 文档++---++## 14. 结论++Neko Master 在架构上实现了采集、存储、展示的清晰分层,并通过批量聚合、实时缓存和多后端隔离获得了较好的性能与可维护性。整体已具备生产可用形态,后续可在测试深度、契约治理和观测性上进一步加强。diff --git a/docs/research-kimi2.5.en.md b/docs/research-kimi2.5.en.mdnew file mode 100644index 0000000..1936b87--- /dev/null+++ b/docs/research-kimi2.5.en.md@@ -0,0 +1,211 @@+# Neko Master Deep Research Report (English)++## Executive Summary++**Neko Master** is a network traffic analytics and visualization platform for Clash, Mihomo, and Surge gateways. Built as a TypeScript monorepo, it provides real-time monitoring, historical analysis, and multi-backend management.++---++## Project Overview++### What It Does++Neko Master focuses on:++- **Collection**: ingesting live connection and traffic data from gateways+- **Analysis**: aggregating by domain, IP, proxy chain, rule, device, and region+- **Visualization**: showing trends and distributions in a web dashboard+- **Management**: operating multiple gateway backends in one place+- **Auditing**: supporting historical time-series queries and trace-back++### Typical Use Cases++- Network admins monitoring performance and anomalies+- Privacy-conscious users auditing outbound traffic behavior+- Developers debugging gateway rules and policy routing+- Teams managing multiple gateway instances++---++## Architecture Overview++### Repository Layout++```text+neko-master/+├── apps/+│ ├── web/ # Next.js 16 frontend+│ └── collector/ # Fastify backend collector+├── packages/+│ └── shared/ # Shared types and utilities+├── docs/+├── docker-compose.yml+└── Dockerfile+```++### Tech Stack++| Layer | Technologies |+|---|---|+| Frontend | Next.js 16, React 19, TypeScript, Tailwind CSS v4, shadcn/ui, Recharts, TanStack Query, next-intl |+| Backend | Node.js 22, Fastify 5, ws, SQLite (better-sqlite3), MaxMind |+| Tooling | pnpm, Turborepo, Docker |++---++## Core Components++### 1) Frontend (`apps/web`)++- App Router with `[locale]` i18n routing (`zh` / `en`)+- Main views include domains, ips, proxies, rules, devices, and settings+- React Query for server state and WebSocket for live updates+- Responsive UI built with shadcn/ui + Tailwind v4+- Basic PWA support (manifest + service worker)++### 2) Backend (`apps/collector`)++Key modules:++- `index.ts`: runtime orchestration entry+- `app.ts`: Fastify API registration+- `collector.ts`: Clash WebSocket collector+- `surge-collector.ts`: Surge HTTP polling collector+- `websocket.ts`: real-time broadcast server+- `realtime.ts`: in-memory real-time aggregation+- `db.ts` + `database/repositories/*`: DB facade and repository layer++Data pipeline:++1. Gateway traffic enters collectors (WS or HTTP)+2. Deltas are calculated and written to RealtimeStore+3. Records enter BatchBuffer (flush at 30s / 5000 entries)+4. Aggregated writes are persisted to SQLite+5. WebSocket pushes updates to frontend clients++### 3) Shared Package (`packages/shared`)++- Shared contracts and helpers across frontend and backend+- Side-effect-free and low-coupling by design+- Typical files: `index.ts`, `gateway-utils.ts`, `geo-ip-utils.ts`++---++## Data Model++### SQLite Design++WAL mode is enabled. Core tables include:++- `domain_stats`, `ip_stats`, `proxy_stats`, `rule_stats`, `country_stats`, `device_stats`+- Cross tables: `domain_proxy_stats`, `ip_proxy_stats`, `rule_domain_traffic`, `rule_ip_traffic`+- Time-series tables: `minute_*`, `hourly_*`, `daily_*`+- Config/cache tables: `backend_configs`, `auth_config`, `geoip_cache`, `asn_cache`, `surge_policy_cache`++### RealtimeStore++Realtime in-memory deltas are partitioned by `backendId` (domain/IP/proxy/rule/device/country). Query responses merge DB history with fresh in-memory increments.++---++## API and Communication++### REST API (high level)++- Backend management: `/api/backends/*`+- Stats queries: `/api/stats/*`, `/api/domains/*`, `/api/ips/*`, `/api/proxies/*`, `/api/rules/*`, `/api/devices/*`+- Gateway proxy APIs: `/api/gateway/:backendId/*`+- Authentication: `/api/auth/*`++### WebSocket++- Endpoint: `/_cm_ws`+- Auth: cookie-based (token-compatible variants are supported by deployment pattern)+- Subscription model: client sends `subscribe`, server pushes `stats` / `connections`+- Supports reconnect, heartbeat, and backend-scoped streams++---++## Key Design Patterns++1. **Repository Pattern**: query and aggregation logic is isolated in repositories.+2. **Dual Write Pattern**: write to in-memory realtime + batch persistence.+3. **Delta Merge Pattern**: merge historical DB data with realtime deltas on read.+4. **Fastify Plugin Pattern**: modular controllers with injected services.++---++## Configuration and Runtime++Common environment variables:++- `WEB_PORT`, `API_PORT`, `COLLECTOR_WS_PORT`+- `DB_PATH`, `COOKIE_SECRET`+- `GEOIP_LOOKUP_PROVIDER`, `MAXMIND_LICENSE_KEY`+- `FORCE_ACCESS_CONTROL_OFF`, `SHOWCASE_MODE`+- `FLUSH_INTERVAL_MS`, `FLUSH_MAX_BUFFER_SIZE`++Backend config includes `id`, `name`, `type(clash/mihomo/surge)`, `url`, `secret`, `pollingInterval`, and `enabled`.++---++## Development Workflow++```bash+pnpm install+pnpm dev+pnpm build+pnpm lint+pnpm --filter @neko-master/collector test+```++Collector tests can be filtered by file name or test title.++---++## Deployment and Security++### Docker++- Multi-stage build for production images+- `docker-compose.yml` exposes 3000 (web), 3001 (api), 3002 (ws)+- Mount `data` for database and runtime persistence++### Security++- Cookie session + token-based login flow+- bcrypt password hashing+- CORS, rate limits, request validation+- SQL parameterized queries+- Optional read-only showcase mode++---++## Performance Strategy++- DB: WAL + indexes + batch writes+- App: RealtimeStore hot data + incremental WS push + debounced write path+- GeoIP: cache + failed-IP cooldown + paced queue++---++## Risks and Suggested Next Steps++### Current Risks++- Startup orchestration and WS protocol paths can use deeper test coverage+- Periodic backend management loop may overlap under timing pressure+- Legacy/parallel paths increase maintenance complexity++### Suggested Improvements++1. Expand tests for collector orchestration and WS integration.+2. Publish detailed HTTP/WS endpoint schema documentation.+3. Add finer-grained alerting and anomaly detection.+4. Improve horizontal scaling and cache coordination for multi-instance deployment.++---++## Conclusion++Neko Master strikes a practical balance across real-time responsiveness, durable storage, and multi-backend isolation. It is production-capable today and has a clear path for scaling and hardening.diff --git a/docs/research-kimi2.5.zh.md b/docs/research-kimi2.5.zh.mdnew file mode 100644index 0000000..30fad4c--- /dev/null+++ b/docs/research-kimi2.5.zh.md@@ -0,0 +1,211 @@+# Neko Master 深度研究报告(中文整理版)++## 执行摘要++**Neko Master**(ねこマスター)是一个网络流量分析与可视化平台,面向 Clash、Mihomo、Surge 等代理网关。项目采用 TypeScript monorepo,提供实时监控、历史分析和多后端管理能力。++---++## 项目概述++### 功能定位++Neko Master 主要能力:++- **收集**:从网关实时采集连接与流量数据+- **分析**:按域名、IP、代理链、规则、设备、地区聚合+- **可视化**:通过仪表盘和图表展示趋势与分布+- **管理**:统一管理多个后端网关+- **审计**:支持历史时序查询与回溯++### 典型场景++- 网络管理员做性能与异常观测+- 隐私关注用户审计流量去向+- 开发者调试网关策略/规则匹配+- 团队统一管理多实例网关++---++## 架构总览++### 仓库结构++```text+neko-master/+├── apps/+│ ├── web/ # Next.js 16 前端+│ └── collector/ # Fastify 后端采集服务+├── packages/+│ └── shared/ # 共享类型与工具+├── docs/+├── docker-compose.yml+└── Dockerfile+```++### 技术栈++| 层级 | 技术 |+|---|---|+| 前端 | Next.js 16、React 19、TypeScript、Tailwind CSS v4、shadcn/ui、Recharts、TanStack Query、next-intl |+| 后端 | Node.js 22、Fastify 5、ws、SQLite(better-sqlite3)、MaxMind |+| 工具 | pnpm、Turborepo、Docker |++---++## 核心组件++### 1)前端(`apps/web`)++- App Router + `[locale]` 多语言路由(`zh`/`en`)+- 主页面包含 domains、ips、proxies、rules、devices、settings 等模块+- React Query 管理服务端状态,WebSocket 负责实时刷新+- 使用 shadcn/ui 与 Tailwind v4 构建响应式界面+- 具备 PWA 基础能力(manifest + service worker)++### 2)后端(`apps/collector`)++关键模块:++- `index.ts`:入口编排+- `app.ts`:Fastify API 注册+- `collector.ts`:Clash WS 采集+- `surge-collector.ts`:Surge HTTP 轮询+- `websocket.ts`:实时广播+- `realtime.ts`:内存实时聚合+- `db.ts` + `database/repositories/*`:数据库门面与仓库层++数据管道:++1. 网关数据进入采集器(WS 或 HTTP)+2. 计算增量并写入 RealtimeStore+3. 数据进入 BatchBuffer(30 秒/5000 条触发 flush)+4. 批量写入 SQLite 聚合表+5. WebSocket 向前端推送更新++### 3)共享包(`packages/shared`)++- 提供前后端共享的类型契约与工具函数+- 保持无副作用、低耦合+- 典型文件:`index.ts`、`gateway-utils.ts`、`geo-ip-utils.ts`++---++## 数据模型++### SQLite 设计++数据库启用 WAL 模式,核心表包括:++- `domain_stats`、`ip_stats`、`proxy_stats`、`rule_stats`、`country_stats`、`device_stats`+- 交叉表:`domain_proxy_stats`、`ip_proxy_stats`、`rule_domain_traffic`、`rule_ip_traffic`+- 时序表:`minute_*`、`hourly_*`、`daily_*`+- 配置表:`backend_configs`、`auth_config`、`geoip_cache`、`asn_cache`、`surge_policy_cache`++### RealtimeStore++按 `backendId` 分片维护域名、IP、代理、规则、设备等实时增量。查询时与 DB 数据合并,兼顾实时性与准确性。++---++## API 与通信++### REST API(摘要)++- 后端管理:`/api/backends/*`+- 统计查询:`/api/stats/*`、`/api/domains/*`、`/api/ips/*`、`/api/proxies/*`、`/api/rules/*`、`/api/devices/*`+- 网关代理:`/api/gateway/:backendId/*`+- 认证:`/api/auth/*`++### WebSocket++- 端点:`/_cm_ws`+- 认证:cookie(可配 token 方案)+- 订阅模型:客户端发送 `subscribe`,服务端推送 `stats` / `connections`+- 支持重连、心跳、按后端维度订阅++---++## 关键设计模式++1. **Repository Pattern**:仓库层封装查询和聚合逻辑。+2. **Dual Write Pattern**:实时写内存 + 批量落盘。+3. **Delta Merge Pattern**:查询时合并历史与实时增量。+4. **Fastify Plugin Pattern**:控制器模块化注册,服务通过装饰注入。++---++## 配置与运行++常用环境变量:++- `WEB_PORT`、`API_PORT`、`COLLECTOR_WS_PORT`+- `DB_PATH`、`COOKIE_SECRET`+- `GEOIP_LOOKUP_PROVIDER`、`MAXMIND_LICENSE_KEY`+- `FORCE_ACCESS_CONTROL_OFF`、`SHOWCASE_MODE`+- `FLUSH_INTERVAL_MS`、`FLUSH_MAX_BUFFER_SIZE`++后端配置包含:`id`、`name`、`type(clash/mihomo/surge)`、`url`、`secret`、`pollingInterval`、`enabled`。++---++## 开发流程++```bash+pnpm install+pnpm dev+pnpm build+pnpm lint+pnpm --filter @neko-master/collector test+```++单测可按文件或用例名过滤执行。++---++## 部署与安全++### Docker++- 多阶段构建,支持生产化镜像+- `docker-compose.yml` 暴露 3000(web)/3001(api)/3002(ws)+- 建议挂载 `data` 目录持久化数据库与关键配置++### 安全++- Cookie 会话 + token 登录+- bcrypt 密码哈希+- CORS、限流、参数校验+- SQL 参数化查询+- 可选只读展示模式++---++## 性能策略++- DB:WAL + 索引 + 批量写入+- 应用:RealtimeStore 热数据、WS 增量推送、写入去抖+- GeoIP:缓存 + 失败冷却 + 队列限速++---++## 风险与后续建议++### 当前风险++- 启动编排和 WS 协议路径的自动化覆盖仍可加强+- 定时后端管理循环存在潜在重叠执行风险+- 部分历史路径会提升理解与维护成本++### 建议方向++1. 补齐 collector 编排与 WS 协议集成测试。+2. 细化端点协议文档(HTTP/WS 请求响应 schema)。+3. 评估更细粒度的告警与异常检测能力。+4. 在多实例部署下增强横向扩展与缓存协同。++---++## 结论++Neko Master 在“实时性 + 持久化 + 多后端隔离”之间做了较好的工程平衡。它已经具备面向生产的核心能力,并在架构上保留了进一步演进到更高规模部署的空间。