Skip to content

A modern and elegant dashboard for network traffic visualization and analysis.

docs: update README and architecture docs with ClickHouse support

luolei4a6fc0e69d59
diff --git a/README.en.md b/README.en.md
index 857cffe..4fe2948 100644
--- a/README.en.md
+++ b/README.en.md
@@ -71,6 +71,7 @@ It is a lightweight analytics dashboard designed for modern gateway environments
- [๐Ÿ“– First Use](#-first-use)
- [๐Ÿ”ง Port Conflict Resolution](#-port-conflict-resolution)
- [๐Ÿณ Docker Configuration](#-docker-configuration)
+- [๐Ÿ—„๏ธ ClickHouse (Optional)](#-clickhouse-optional)
- [๐ŸŒ Reverse Proxy & Tunnel](#-reverse-proxy--tunnel)
- [๐Ÿ” Authentication & Security](#-authentication--security)
- [โ“ FAQ](#-faq)
@@ -429,6 +430,158 @@ Additional recommendations:
> Advanced Agent details (install, config, release, compatibility) are maintained under `docs/agent/*`.
+## ๐Ÿ—„๏ธ ClickHouse (Optional)
+
+SQLite is Neko Master's default storage engine and works well for most users.
+Consider enabling ClickHouse if you need:
+
+- Very large datasets (hundreds of thousands of domain/IP entries)
+- Fast aggregation queries over long time ranges (โ‰ฅ 7 days)
+- Separation of historical stats from configuration/metadata storage
+
+> ClickHouse is entirely optional. SQLite remains as the configuration and metadata store regardless of whether ClickHouse is enabled.
+
+### Architecture Overview
+
+When ClickHouse is enabled, the system enters **dual-write mode**:
+
+```
+BatchBuffer.flush()
+ โ”‚
+ โ”œโ”€โ”€โ†’ SQLite (config / metadata, always written)
+ โ””โ”€โ”€โ†’ ClickHouse (stats traffic data, dual-write)
+ โ””โ”€โ”€ Buffer tables โ†’ SummingMergeTree async merge
+```
+
+Read source is controlled by `STATS_QUERY_SOURCE` (default: `sqlite`).
+
+### Enabling ClickHouse (Docker)
+
+#### Step 1: Start the ClickHouse container
+
+The `docker-compose.yml` includes a ClickHouse service gated by `profiles: [clickhouse]`:
+
+```bash
+docker compose --profile clickhouse up -d
+```
+
+> ClickHouse data is persisted to `./data/clickhouse`, separate from the main app data directory.
+
+#### Step 2: Configure environment variables
+
+Add to your `.env` (same directory as `docker-compose.yml`):
+
+```env
+# Enable ClickHouse connection
+CH_ENABLED=1
+
+# Enable dual-write
+CH_WRITE_ENABLED=1
+
+# Read source: sqlite (default) / auto (smart routing) / clickhouse (force)
+STATS_QUERY_SOURCE=auto
+
+# ClickHouse connection (defaults match docker-compose.yml, no change needed)
+CH_HOST=clickhouse
+CH_PORT=8123
+CH_DATABASE=neko_master
+CH_USER=neko
+CH_PASSWORD=neko_master
+```
+
+Restart:
+
+```bash
+docker compose --profile clickhouse up -d
+```
+
+### ClickHouse Environment Variables
+
+| Variable | Default | Description |
+| :-- | :-- | :-- |
+| `CH_ENABLED` | `0` | Enable ClickHouse connection (`1` to enable) |
+| `CH_WRITE_ENABLED` | `0` | Enable dual-write (requires `CH_ENABLED=1`) |
+| `CH_ONLY_MODE` | `0` | When CH is healthy, skip SQLite stats writes (CH-only mode) |
+| `CH_HOST` | `clickhouse` | ClickHouse host address |
+| `CH_PORT` | `8123` | ClickHouse HTTP port |
+| `CH_DATABASE` | `neko_master` | Database name |
+| `CH_USER` | `neko` | Username |
+| `CH_PASSWORD` | `neko_master` | Password |
+| `CH_SECURE` | `0` | Use HTTPS connection |
+| `CH_REQUIRED` | `0` | Refuse to start if CH is unavailable |
+| `CH_AUTO_CREATE_TABLES` | `1` | Auto-create tables on first start |
+| `CH_WRITE_MAX_PENDING_BATCHES` | `200` | Max pending write batches |
+| `CH_UNHEALTHY_THRESHOLD` | `5` | Consecutive failures before marking unhealthy (auto-fallback to SQLite) |
+| `STATS_QUERY_SOURCE` | `sqlite` | Read source: `sqlite` / `auto` / `clickhouse` |
+| `CH_COMPARE_ENABLED` | `0` | Enable SQLite โ†” ClickHouse consistency check |
+| `CH_EXTERNAL_HTTP_PORT` | `8123` | ClickHouse HTTP external port (Compose mapping) |
+| `CH_EXTERNAL_NATIVE_PORT` | `9000` | ClickHouse Native external port (Compose mapping) |
+
+> **Health & Fallback**: After `CH_UNHEALTHY_THRESHOLD` consecutive write failures, the system automatically marks ClickHouse as unhealthy and resumes SQLite writesโ€”even when `CH_ONLY_MODE=1`. Once ClickHouse recovers, it is re-marked healthy and logged.
+
+### Migration Guide for Existing Users
+
+> Upgrading from a SQLite-only version? **Your data is safe.**
+> The SQLite file (`./data/stats.db`) is fully preserved. Here is the recommended gradual migration path:
+
+#### Phase 1: Dual-write (observation period, recommended starting point)
+
+```env
+CH_ENABLED=1
+CH_WRITE_ENABLED=1
+STATS_QUERY_SOURCE=sqlite # Keep reading from SQLite while CH accumulates data
+```
+
+Start and watch `[ClickHouse Writer]` logs to confirm successful writes.
+
+#### Phase 2: Switch read source
+
+```env
+STATS_QUERY_SOURCE=auto # Smart routing: recent data from CH, historical from SQLite
+# or
+STATS_QUERY_SOURCE=clickhouse # Force all reads to ClickHouse
+```
+
+#### Phase 3 (optional): Migrate historical data
+
+To move historical SQLite stats into ClickHouse:
+
+```bash
+# Standard migration (truncate CH then re-import, with consistency check)
+./scripts/ch-migrate-docker.sh
+
+# Append mode (keep existing CH data, incremental import)
+./scripts/ch-migrate-docker.sh --append
+
+# Specific time window
+./scripts/ch-migrate-docker.sh --from 2026-02-01T00:00:00Z --to 2026-02-20T00:00:00Z
+```
+
+#### Phase 4 (optional): CH-only mode
+
+Once ClickHouse is running stably, stop SQLite stats writes:
+
+```env
+CH_ONLY_MODE=1
+```
+
+> Even with `CH_ONLY_MODE=1`, if ClickHouse becomes unhealthy the system automatically falls back to SQLite writesโ€”no data loss.
+
+### Reverting to SQLite-only
+
+You can always roll back completely:
+
+```env
+CH_ENABLED=0
+CH_WRITE_ENABLED=0
+CH_ONLY_MODE=0
+STATS_QUERY_SOURCE=sqlite
+```
+
+Restart and everything returns to pure SQLite mode. Historical data remains intact.
+
+---
+
## ๐ŸŒ Reverse Proxy & Tunnel
Recommended approach: keep Web and WS under the same domain, with path routing:
@@ -598,35 +751,6 @@ If you forgot the token, temporarily set `FORCE_ACCESS_CONTROL_OFF=true` to ente
3. Reset token, then remove this flag and restart normally.
-### ClickHouse Migration (Docker Users)
-
-If you want to migrate stats from SQLite to ClickHouse (while keeping control/config in SQLite), use the one-command helper:
-
-```bash
-./scripts/ch-migrate-docker.sh
-```
-
-The script automatically:
-
-1. Starts services with the `clickhouse` profile
-2. Migrates SQLite stats into ClickHouse
-3. Verifies SQLite vs ClickHouse consistency (default threshold: 1%)
-
-Common options:
-
-```bash
-# Append import without truncating CH tables
-./scripts/ch-migrate-docker.sh --append
-
-# Migrate a specific time window
-./scripts/ch-migrate-docker.sh --from 2026-02-01T00:00:00Z --to 2026-02-20T00:00:00Z
-```
-
-After migration, set `STATS_QUERY_SOURCE=auto` and restart:
-
-```bash
-docker compose up -d
-```
## โ“ FAQ
@@ -705,8 +829,9 @@ If you want to quickly understand the system design depth, read in this order:
Chinese: [`docs/architecture.md`](./docs/architecture.md)
English: [`docs/architecture.en.md`](./docs/architecture.en.md)
2. **Data Flow**: Clash / Surge collection pipelines and aggregation
-3. **Data Model & Storage**: SQLite schema, cache tables, retention
+3. **Data Model & Storage**: SQLite schema, ClickHouse Buffer tables, retention policy
4. **Realtime Channel Design**: `RealtimeStore` merge strategy and WS push
+5. **ClickHouse Module**: dual-write architecture, health fallback, read routing
> This documentation covers the core design of collection, aggregation, caching, realtime push, and multi-backend management.
@@ -749,7 +874,7 @@ neko-master/
- **Charts**: [Recharts](https://recharts.org/)
- **i18n**: [next-intl](https://next-intl-docs.vercel.app/)
- **Backend**: [Node.js](https://nodejs.org/) + [Fastify](https://www.fastify.io/) + WebSocket
-- **Database**: [SQLite](https://www.sqlite.org/) ([better-sqlite3](https://github.com/WiseLibs/better-sqlite3))
+- **Database**: [SQLite](https://www.sqlite.org/) ([better-sqlite3](https://github.com/WiseLibs/better-sqlite3)) + [ClickHouse](https://clickhouse.com/) (optional)
- **Build**: [pnpm](https://pnpm.io/) + [Turborepo](https://turbo.build/)
## ๐Ÿค Contributing
diff --git a/README.md b/README.md
index b166741..4f2a193 100644
--- a/README.md
+++ b/README.md
@@ -76,6 +76,7 @@ Neko Master ไธ“ๆณจไบŽๅฏน็ฝ‘็ปœๆต้‡่ฟ›่กŒ่ฝป้‡ใ€็ฒพ็กฎ็š„ๅˆ†ๆžไธŽๅฏ่ง†ๅŒ–
- [๐Ÿ“– ้ฆ–ๆฌกไฝฟ็”จ](#-้ฆ–ๆฌกไฝฟ็”จ)
- [๐Ÿ”ง ็ซฏๅฃๅ†ฒ็ช่งฃๅ†ณ](#-็ซฏๅฃๅ†ฒ็ช่งฃๅ†ณ)
- [๐Ÿณ Docker ้…็ฝฎ](#-docker-้…็ฝฎ)
+- [๐Ÿ—„๏ธ ClickHouse๏ผˆๅฏ้€‰๏ผ‰](#-clickhouse-ๅฏ้€‰)
- [๐ŸŒ ๅๅ‘ไปฃ็†ไธŽ Tunnel](#-ๅๅ‘ไปฃ็†ไธŽ-tunnel)
- [๐Ÿ” ่ฎค่ฏไธŽๅฎ‰ๅ…จ](#-่ฎค่ฏไธŽๅฎ‰ๅ…จ)
- [โ“ ๅธธ่ง้—ฎ้ข˜](#-ๅธธ่ง้—ฎ้ข˜)
@@ -420,6 +421,158 @@ COOKIE_SECRET=<่‡ณๅฐ‘32ๅญ—่Š‚้šๆœบๅญ—็ฌฆไธฒ>
> ่ฟ›้˜ถ่ฏดๆ˜Ž๏ผšAgent ๆจกๅผ็š„ๅฎ‰่ฃ…ใ€ๅ‚ๆ•ฐใ€ๅ‘ๅธƒไธŽๅ…ผๅฎน็ญ–็•ฅๅทฒ่ฟ็งปๅˆฐ `docs/agent/*`๏ผŒ่ฏทไปฅๆ–‡ๆกฃไธบๅ‡†ใ€‚
+## ๐Ÿ—„๏ธ ClickHouse๏ผˆๅฏ้€‰๏ผ‰
+
+SQLite ๆ˜ฏ Neko Master ็š„้ป˜่ฎคๅญ˜ๅ‚จๅผ•ๆ“Ž๏ผŒๅฏนๅคงๅคšๆ•ฐ็”จๆˆทๅทฒๅฎŒๅ…จๅคŸ็”จใ€‚
+ๅฆ‚ๆžœไฝ ๆœ‰ไปฅไธ‹้œ€ๆฑ‚๏ผŒๅฏไปฅ้ขๅค–ๅฏ็”จ ClickHouse๏ผš
+
+- ๆ•ฐๆฎ้‡ๅพˆๅคง๏ผˆๅŸŸๅ / IP ๆก็›ฎๆ•ฐๅไธ‡ไปฅไธŠ๏ผ‰
+- ้œ€่ฆๅฟซ้€Ÿ็š„้•ฟๆ—ถ้—ด่Œƒๅ›ด๏ผˆโ‰ฅ 7 ๅคฉ๏ผ‰่šๅˆๆŸฅ่ฏข
+- ๅธŒๆœ›ๅฐ†ๅކๅฒ็ปŸ่ฎกๆ•ฐๆฎไธŽ้…็ฝฎๆ•ฐๆฎๅˆ†ๅฑ‚ๅญ˜ๅ‚จ
+
+> ClickHouse ๅฎŒๅ…จๅฏ้€‰ใ€‚SQLite ไฝœไธบ้…็ฝฎๅ’Œๅ…ƒๆ•ฐๆฎๅญ˜ๅ‚จ๏ผŒๆ— ่ฎบๆ˜ฏๅฆๅฏ็”จ ClickHouse ้ƒฝไธไผš่ขซ็งป้™คใ€‚
+
+### ๆžถๆž„่ฏดๆ˜Ž
+
+ๅฏ็”จ ClickHouse ๅŽ๏ผŒ็ณป็ปŸ่ฟ›ๅ…ฅ**ๅŒๅ†™ๆจกๅผ**๏ผš
+
+```
+BatchBuffer.flush()
+ โ”‚
+ โ”œโ”€โ”€โ†’ SQLite๏ผˆ้…็ฝฎ / ๅ…ƒๆ•ฐๆฎ๏ผŒๅง‹็ปˆๅ†™ๅ…ฅ๏ผ‰
+ โ””โ”€โ”€โ†’ ClickHouse๏ผˆ็ปŸ่ฎกๆต้‡ๆ•ฐๆฎ๏ผŒๅŒๅ†™๏ผ‰
+ โ””โ”€โ”€ Buffer ่กจ โ†’ SummingMergeTree ๅผ‚ๆญฅๅˆๅนถ
+```
+
+่ฏปๅ–ๆฅๆบ็”ฑ `STATS_QUERY_SOURCE` ๆŽงๅˆถ๏ผŒ้ป˜่ฎคไปไธบ `sqlite`ใ€‚
+
+### ๅฏ็”จ ClickHouse๏ผˆDocker๏ผ‰
+
+#### ๆญฅ้ชคไธ€๏ผšๅฏๅŠจ ClickHouse ๅฎนๅ™จ
+
+`docker-compose.yml` ๅ†…็ฝฎไบ† ClickHouse ๆœๅŠก๏ผŒ้€š่ฟ‡ `profiles: [clickhouse]` ้š”็ฆปใ€‚ๅฏๅŠจๆ—ถๅŠ ไธŠ profile๏ผš
+
+```bash
+docker compose --profile clickhouse up -d
+```
+
+> ClickHouse ๆ•ฐๆฎๆŒไน…ๅŒ–ๅˆฐ `./data/clickhouse`๏ผŒไธŽไธปๅบ”็”จๆ•ฐๆฎๅˆ†็›ฎๅฝ•ๅญ˜ๅ‚จใ€‚
+
+#### ๆญฅ้ชคไบŒ๏ผš้…็ฝฎ็Žฏๅขƒๅ˜้‡
+
+ๅœจ `.env` ไธญๆทปๅŠ ๏ผˆไธŽ `docker-compose.yml` ๅŒ็›ฎๅฝ•๏ผ‰๏ผš
+
+```env
+# ๅผ€ๅฏ ClickHouse ่ฟžๆŽฅ
+CH_ENABLED=1
+
+# ๅผ€ๅฏๅŒๅ†™
+CH_WRITE_ENABLED=1
+
+# ่ฏปๅ–ๆฅๆบ๏ผšsqlite๏ผˆ้ป˜่ฎค๏ผ‰/ auto๏ผˆ่‡ชๅŠจ้€‰ๆ‹ฉ๏ผ‰/ clickhouse๏ผˆๅผบๅˆถ๏ผ‰
+STATS_QUERY_SOURCE=auto
+
+# ClickHouse ่ฟžๆŽฅไฟกๆฏ๏ผˆไฝฟ็”จ docker-compose.yml ้ป˜่ฎคๅ€ผๆ—ถ๏ผŒไปฅไธ‹ๆ— ้œ€ไฟฎๆ”น๏ผ‰
+CH_HOST=clickhouse
+CH_PORT=8123
+CH_DATABASE=neko_master
+CH_USER=neko
+CH_PASSWORD=neko_master
+```
+
+้‡ๅฏ๏ผš
+
+```bash
+docker compose --profile clickhouse up -d
+```
+
+### ClickHouse ็Žฏๅขƒๅ˜้‡่ฏดๆ˜Ž
+
+| ๅ˜้‡ๅ | ้ป˜่ฎคๅ€ผ | ่ฏดๆ˜Ž |
+| :-- | :-- | :-- |
+| `CH_ENABLED` | `0` | ๆ˜ฏๅฆๅฏ็”จ ClickHouse ่ฟžๆŽฅ๏ผˆ`1` ๅผ€ๅฏ๏ผ‰ |
+| `CH_WRITE_ENABLED` | `0` | ๆ˜ฏๅฆๅผ€ๅฏๅŒๅ†™๏ผˆ้กปๅ…ˆๅผ€ `CH_ENABLED`๏ผ‰ |
+| `CH_ONLY_MODE` | `0` | CH ๅฅๅบทๆ—ถ่ทณ่ฟ‡ SQLite ็ปŸ่ฎกๅ†™ๅ…ฅ๏ผˆ็บฏ CH ๆจกๅผ๏ผ‰ |
+| `CH_HOST` | `clickhouse` | ClickHouse ไธปๆœบๅœฐๅ€ |
+| `CH_PORT` | `8123` | ClickHouse HTTP ็ซฏๅฃ |
+| `CH_DATABASE` | `neko_master` | ๆ•ฐๆฎๅบ“ๅ็งฐ |
+| `CH_USER` | `neko` | ็”จๆˆทๅ |
+| `CH_PASSWORD` | `neko_master` | ๅฏ†็  |
+| `CH_SECURE` | `0` | ๆ˜ฏๅฆไฝฟ็”จ HTTPS ่ฟžๆŽฅ |
+| `CH_REQUIRED` | `0` | CH ไธๅฏ็”จๆ—ถๆ‹’็ปๅฏๅŠจ๏ผˆ`1` ๅผ€ๅฏ๏ผ‰ |
+| `CH_AUTO_CREATE_TABLES` | `1` | ้ฆ–ๆฌกๅฏๅŠจๆ—ถ่‡ชๅŠจๅปบ่กจ |
+| `CH_WRITE_MAX_PENDING_BATCHES` | `200` | ๆœ€ๅคงๅ†™ๅ…ฅ็งฏๅŽ‹ๆ‰นๆฌกๆ•ฐ |
+| `CH_UNHEALTHY_THRESHOLD` | `5` | ่ฟž็ปญๅคฑ่ดฅๆฌกๆ•ฐ่ถ…่ฟ‡ๆญคๅ€ผๅŽๆ ‡่ฎฐไธบไธๅฅๅบท๏ผŒ่‡ชๅŠจๅ›ž้€€ SQLite |
+| `STATS_QUERY_SOURCE` | `sqlite` | ่ฏปๅ–ๆฅๆบ๏ผš`sqlite` / `auto` / `clickhouse` |
+| `CH_COMPARE_ENABLED` | `0` | ๅผ€ๅฏ SQLite โ†” ClickHouse ๆ•ฐๆฎๅฏน่ดฆ |
+| `CH_EXTERNAL_HTTP_PORT` | `8123` | ClickHouse HTTP ๅค–้ƒจ็ซฏๅฃ๏ผˆCompose ๆ˜ ๅฐ„๏ผ‰ |
+| `CH_EXTERNAL_NATIVE_PORT` | `9000` | ClickHouse Native ๅค–้ƒจ็ซฏๅฃ๏ผˆCompose ๆ˜ ๅฐ„๏ผ‰ |
+
+> **ๅฅๅบทๅ›ž้€€ๆœบๅˆถ**๏ผšClickHouse ่ฟž็ปญๅ†™ๅ…ฅๅคฑ่ดฅ่ถ…่ฟ‡ `CH_UNHEALTHY_THRESHOLD` ๆฌกๅŽ๏ผŒ็ณป็ปŸ่‡ชๅŠจๆ ‡่ฎฐไธบไธๅฅๅบท๏ผŒSQLite ๅ†™ๅ…ฅๆขๅค๏ผˆๅณไฝฟๅผ€ๅฏไบ† `CH_ONLY_MODE`๏ผ‰ใ€‚ClickHouse ๆขๅคๆญฃๅธธๅŽ่‡ชๅŠจ้‡ๆ–ฐๆ ‡่ฎฐไธบๅฅๅบทใ€‚
+
+### ่€็”จๆˆท่ฟ็งปๆŒ‡ๅผ•
+
+> ไปŽ็บฏ SQLite ็‰ˆๆœฌๅ‡็บงๆ—ถ๏ผŒ**ไฝ ็š„ๆ•ฐๆฎไธไผšไธขๅคฑ**ใ€‚
+> SQLite ๆ–‡ไปถ๏ผˆ`./data/stats.db`๏ผ‰ๅฎŒๆ•ดไฟ็•™๏ผŒไปฅไธ‹ๆ˜ฏๆŽจ่็š„ๆธ่ฟ›่ฟ็งป่ทฏๅพ„๏ผš
+
+#### ็ฌฌไธ€้˜ถๆฎต๏ผšๅŒๅ†™๏ผˆ่ง‚ๅฏŸๆœŸ๏ผŒๆŽจ่่ตท็‚น๏ผ‰
+
+```env
+CH_ENABLED=1
+CH_WRITE_ENABLED=1
+STATS_QUERY_SOURCE=sqlite # ็ปง็ปญ่ฏป SQLite๏ผŒCH ๅœจๅŽๅฐ็งฏ็ดฏๆ•ฐๆฎ
+```
+
+ๅฏๅŠจๅนถ่ง‚ๅฏŸ `[ClickHouse Writer]` ๆ—ฅๅฟ—็กฎ่ฎคๅ†™ๅ…ฅๆญฃๅธธใ€‚
+
+#### ็ฌฌไบŒ้˜ถๆฎต๏ผšๅˆ‡ๆข่ฏปๅ–ๆฅๆบ
+
+```env
+STATS_QUERY_SOURCE=auto # ่‡ชๅŠจ้€‰ๆ‹ฉ๏ผšๆœ€่ฟ‘ๆ•ฐๆฎ่ตฐ CH๏ผŒๅކๅฒ่ตฐ SQLite
+# ๆˆ–
+STATS_QUERY_SOURCE=clickhouse # ๅผบๅˆถ่ฏป CH
+```
+
+#### ็ฌฌไธ‰้˜ถๆฎต๏ผˆๅฏ้€‰๏ผ‰๏ผš่ฟ็งปๅކๅฒๆ•ฐๆฎ
+
+ๅฆ‚้œ€ๅฐ† SQLite ๅކๅฒ็ปŸ่ฎกๆ•ฐๆฎๆฌๅ…ฅ ClickHouse๏ผš
+
+```bash
+# ๆ ‡ๅ‡†่ฟ็งป๏ผˆๆธ…็ฉบ CH ๅŽ้‡ๆ–ฐๅฏผๅ…ฅ๏ผŒ้™„ๅธฆๅฏน่ดฆ้ชŒ่ฏ๏ผ‰
+./scripts/ch-migrate-docker.sh
+
+# ่ฟฝๅŠ ๆจกๅผ๏ผˆไฟ็•™ CH ็Žฐๆœ‰ๆ•ฐๆฎ๏ผŒๅขž้‡ๅฏผๅ…ฅ๏ผ‰
+./scripts/ch-migrate-docker.sh --append
+
+# ๆŒ‡ๅฎšๆ—ถ้—ด็ช—ๅฃ
+./scripts/ch-migrate-docker.sh --from 2026-02-01T00:00:00Z --to 2026-02-20T00:00:00Z
+```
+
+#### ็ฌฌๅ››้˜ถๆฎต๏ผˆๅฏ้€‰๏ผ‰๏ผš็บฏ ClickHouse ๆจกๅผ
+
+ๅฝ“ ClickHouse ็จณๅฎš่ฟ่กŒๅŽ๏ผŒๅฏๅœๆญข SQLite ็ปŸ่ฎกๅ†™ๅ…ฅ๏ผš
+
+```env
+CH_ONLY_MODE=1
+```
+
+> ๅณไฝฟๅœจ `CH_ONLY_MODE=1` ไธ‹๏ผŒ่‹ฅ ClickHouse ไธๅฅๅบท๏ผŒ็ณป็ปŸไผš่‡ชๅŠจๅ›ž้€€ๅˆฐ SQLite ๅ†™ๅ…ฅ๏ผŒๆ•ฐๆฎไธไผšไธขๅคฑใ€‚
+
+### ๅ›ž้€€ๅˆฐ็บฏ SQLite
+
+้šๆ—ถๅฏไปฅๅฎŒๅ…จๅ›ž้€€๏ผš
+
+```env
+CH_ENABLED=0
+CH_WRITE_ENABLED=0
+CH_ONLY_MODE=0
+STATS_QUERY_SOURCE=sqlite
+```
+
+้‡ๅฏๅŽๆขๅค็บฏ SQLite ๆจกๅผ๏ผŒๅކๅฒๆ•ฐๆฎๅฎŒๆ•ดไฟ็•™ใ€‚
+
+---
+
## ๐ŸŒ ๅๅ‘ไปฃ็†ไธŽ Tunnel
ๆŽจ่ๅฐ† Web ้กต้ขไธŽ WS ้ƒฝๆ”พๅœจๅŒไธ€ไธชๅŸŸๅไธ‹๏ผŒ้€š่ฟ‡ไธๅŒ่ทฏๅพ„่ฝฌๅ‘๏ผš`/` โ†’ `3000`๏ผŒ`/_cm_ws` โ†’ `3002`ใ€‚
@@ -590,35 +743,6 @@ COOKIE_SECRET=<่‡ณๅฐ‘32ๅญ—่Š‚้šๆœบๅญ—็ฌฆไธฒ>
3. ้‡็ฝฎๅฏ†็ ๅŽ๏ผŒๅ†ๆฌกๅœๆญขๅฎนๅ™จ๏ผŒๅŽป้™ค่ฏฅๅ‚ๆ•ฐๅนถ้‡ๅฏ๏ผŒๆขๅคๆญฃๅธธไฟๆŠคๆจกๅผใ€‚
-### ClickHouse ่ฟ็งป๏ผˆDocker ็”จๆˆท๏ผ‰
-
-ๅฆ‚ๆžœไฝ ๅ‡†ๅค‡ไปŽ SQLite ็ปŸ่ฎก่ฟ็งปๅˆฐ ClickHouse๏ผˆไฟ็•™้…็ฝฎไปๅœจ SQLite๏ผ‰๏ผŒๆŽจ่็›ดๆŽฅไฝฟ็”จไธ€้”ฎ่„šๆœฌ๏ผš
-
-```bash
-./scripts/ch-migrate-docker.sh
-```
-
-่„šๆœฌไผš่‡ชๅŠจๆ‰ง่กŒ๏ผš
-
-1. ๅฏๅŠจ `clickhouse` profile ๅฎนๅ™จ
-2. ่ฟ็งป SQLite ๆ•ฐๆฎๅˆฐ ClickHouse
-3. ๅฏน่ดฆ้ชŒ่ฏ๏ผˆ้ป˜่ฎค้˜ˆๅ€ผ 1%๏ผ‰
-
-ๅธธ็”จๅ‚ๆ•ฐ๏ผš
-
-```bash
-# ไธๆธ…็ฉบ CH ๅކๅฒๆ•ฐๆฎ๏ผŒๅš่ฟฝๅŠ ๅฏผๅ…ฅ
-./scripts/ch-migrate-docker.sh --append
-
-# ๆŒ‡ๅฎšๆ—ถ้—ด็ช—ๅฃ่ฟ็งป
-./scripts/ch-migrate-docker.sh --from 2026-02-01T00:00:00Z --to 2026-02-20T00:00:00Z
-```
-
-่ฟ็งปๅฎŒๆˆๅŽ๏ผŒๅปบ่ฎฎๅฐ† `STATS_QUERY_SOURCE=auto` ๅนถ้‡ๅฏ๏ผš
-
-```bash
-docker compose up -d
-```
## โ“ ๅธธ่ง้—ฎ้ข˜
@@ -696,8 +820,9 @@ docker compose up -d
ไธญๆ–‡๏ผš[`docs/architecture.md`](./docs/architecture.md)
English: [`docs/architecture.en.md`](./docs/architecture.en.md)
2. **ๆ•ฐๆฎๆต่ฏฆ่งฃ**๏ผšClash / Surge ไธคๆก้‡‡้›†้“พ่ทฏไธŽ่šๅˆ่ฟ‡็จ‹
-3. **ๆ•ฐๆฎๆจกๅž‹ไธŽๅญ˜ๅ‚จ**๏ผšSQLite ่กจ็ป“ๆž„ใ€็ผ“ๅญ˜่กจใ€ไฟ็•™็ญ–็•ฅ
+3. **ๆ•ฐๆฎๆจกๅž‹ไธŽๅญ˜ๅ‚จ**๏ผšSQLite ่กจ็ป“ๆž„ใ€ClickHouse Buffer ่กจใ€ไฟ็•™็ญ–็•ฅ
4. **ๅฎžๆ—ถ้€š้“่ฎพ่ฎก**๏ผš`RealtimeStore` ๅˆๅนถ็ญ–็•ฅไธŽ WS ๆŽจ้€ๆœบๅˆถ
+5. **ClickHouse ๆจกๅ—**๏ผšๅŒๅ†™ๆžถๆž„ใ€ๅฅๅบทๅ›ž้€€ใ€่ฏปๅ–่ทฏ็”ฑ
> ่ฏฅๆ–‡ๆกฃ่ฆ†็›–้‡‡้›†ใ€่šๅˆใ€็ผ“ๅญ˜ใ€ๅฎžๆ—ถๆŽจ้€ไธŽๅคšๅŽ็ซฏ็ฎก็†็š„ๆ ธๅฟƒ่ฎพ่ฎกใ€‚
@@ -737,7 +862,7 @@ neko-master/
- **ๅ‰็ซฏ**: Next.js 16 + React 19 + TypeScript + Tailwind CSS
- **UI ็ป„ไปถ**: shadcn/ui
-- **ๆ•ฐๆฎๆ”ถ้›†**: Node.js + Fastify + WebSocket + SQLite
+- **ๆ•ฐๆฎๆ”ถ้›†**: Node.js + Fastify + WebSocket + SQLite๏ผˆ+ ClickHouse ๅฏ้€‰๏ผ‰
- **ๅฏ่ง†ๅŒ–**: Recharts + D3.js
- **ๅ›ฝ้™…ๅŒ–**: next-intl๏ผˆไธญ/่‹ฑ๏ผ‰
- **้ƒจ็ฝฒ**: Docker + Docker Compose
diff --git a/docs/architecture.en.md b/docs/architecture.en.md
index 3a70765..12d39ea 100644
--- a/docs/architecture.en.md
+++ b/docs/architecture.en.md
@@ -132,13 +132,14 @@
โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
โ”‚
- โ”‚ SQLite (better-sqlite3)
+ โ”œโ”€โ”€โ†’ SQLite (better-sqlite3) [always written]
+ โ””โ”€โ”€โ†’ ClickHouse HTTP API [optional dual-write]
โ–ผ
โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
โ”‚ Storage Layer โ”‚
โ”‚ โ”‚
โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚
-โ”‚ โ”‚ SQLite Database (WAL Mode) โ”‚ โ”‚
+โ”‚ โ”‚ SQLite Database (WAL Mode) [always enabled] โ”‚ โ”‚
โ”‚ โ”‚ โ”‚ โ”‚
โ”‚ โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ โ”‚
โ”‚ โ”‚ โ”‚ Statistics Tables (partitioned by backend_id) โ”‚ โ”‚ โ”‚
@@ -171,6 +172,28 @@
โ”‚ โ”‚ โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ โ”‚ โ”‚
โ”‚ โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ โ”‚
โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚
+โ”‚ โ”‚
+โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚
+โ”‚ โ”‚ ClickHouse Database [optional, enabled with CH_ENABLED=1] โ”‚ โ”‚
+โ”‚ โ”‚ โ”‚ โ”‚
+โ”‚ โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ โ”‚
+โ”‚ โ”‚ โ”‚ Buffer Tables (async receive, ~5min merge) โ”‚ โ”‚ โ”‚
+โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚
+โ”‚ โ”‚ โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ โ”‚ โ”‚
+โ”‚ โ”‚ โ”‚ โ”‚ traffic_detail_bufferโ”‚ โ”‚ traffic_agg_buffer โ”‚ โ”‚ โ”‚ โ”‚
+โ”‚ โ”‚ โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ โ”‚ โ”‚
+โ”‚ โ”‚ โ”‚ โ”‚ merge โ”‚ merge โ”‚ โ”‚ โ”‚
+โ”‚ โ”‚ โ”‚ โ–ผ โ–ผ โ”‚ โ”‚ โ”‚
+โ”‚ โ”‚ โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ โ”‚ โ”‚
+โ”‚ โ”‚ โ”‚ โ”‚ traffic_detail โ”‚ โ”‚ traffic_agg โ”‚ โ”‚ โ”‚ โ”‚
+โ”‚ โ”‚ โ”‚ โ”‚ (SummingMergeTree) โ”‚ โ”‚ (SummingMergeTree) โ”‚ โ”‚ โ”‚ โ”‚
+โ”‚ โ”‚ โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ โ”‚ โ”‚
+โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚
+โ”‚ โ”‚ โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ โ”‚ โ”‚
+โ”‚ โ”‚ โ”‚ โ”‚ country_buffer โ”‚ โ†’ country_stats (SummingMergeTree) โ”‚ โ”‚ โ”‚
+โ”‚ โ”‚ โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ โ”‚ โ”‚
+โ”‚ โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ โ”‚
+โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
โ–ฒ
โ”‚
@@ -236,7 +259,8 @@ ClashCollector
โ”‚ - Batch cache optimization
โ”‚
โ”œโ”€โ”€ 4. Data Write
- โ”‚ โ”œโ”€โ”€โ†’ SQLite (Persistence)
+ โ”‚ โ”œโ”€โ”€โ†’ SQLite (Persistence, always written)
+ โ”‚ โ”œโ”€โ”€โ†’ ClickHouse (optional dual-write, when CH_WRITE_ENABLED=1)
โ”‚ โ””โ”€โ”€โ†’ RealtimeStore (In-Memory)
โ”‚
โ””โ”€โ”€ 5. Trigger Broadcast
@@ -271,7 +295,8 @@ SurgeCollector
โ”‚ - 30s flush
โ”‚
โ”œโ”€โ”€ 5. Data Write
- โ”‚ โ”œโ”€โ”€โ†’ SQLite (Persistence)
+ โ”‚ โ”œโ”€โ”€โ†’ SQLite (Persistence, always written)
+ โ”‚ โ”œโ”€โ”€โ†’ ClickHouse (optional dual-write, when CH_WRITE_ENABLED=1)
โ”‚ โ””โ”€โ”€โ†’ RealtimeStore (In-Memory)
โ”‚
โ””โ”€โ”€ 6. Trigger Broadcast
@@ -288,7 +313,7 @@ WebSocketServer
โ”‚ - Establish ClientInfo (backend, range, interval)
โ”‚
โ”œโ”€โ”€ Data Preparation
- โ”‚ - Query base data from SQLite
+ โ”‚ - Query base data from SQLite or ClickHouse (controlled by STATS_QUERY_SOURCE)
โ”‚ - RealtimeStore.merge*() merge real-time deltas
โ”‚ - Summary cache (2s TTL)
โ”‚
@@ -354,16 +379,29 @@ BaseRepository (Abstract Base Class)
โ””โ”€โ”€ ... 13 shared utility methods total
```
-### Dual Write Pattern
+### Triple Write Pattern
```
Collector receives traffic data
โ”‚
- โ”œโ”€โ”€โ†’ SQLite (Persistent Storage)
- โ”‚ โ””โ”€ For historical queries, aggregate statistics
+ โ”œโ”€โ”€โ†’ SQLite (Persistent Storage, always written)
+ โ”‚ โ””โ”€ Config / metadata / historical stats (auto-fallback when CH is down)
+ โ”‚
+ โ”œโ”€โ”€โ†’ ClickHouse (optional, when CH_WRITE_ENABLED=1)
+ โ”‚ โ””โ”€ Stats dual-write โ†’ Buffer tables โ†’ SummingMergeTree async merge
+ โ”‚ โ””โ”€ Health fallback: after CH_UNHEALTHY_THRESHOLD consecutive failures,
+ โ”‚ automatically falls back to SQLite writes
โ”‚
โ””โ”€โ”€โ†’ RealtimeStore (In-Memory Real-time)
- โ””โ”€ For real-time display, low-latency push
+ โ””โ”€ For real-time display, low-latency push (compensates Buffer delay)
+```
+
+**Read Routing (STATS_QUERY_SOURCE)**
+
+```
+STATS_QUERY_SOURCE=sqlite โ†’ all reads from SQLite (default)
+STATS_QUERY_SOURCE=clickhouse โ†’ all reads from ClickHouse
+STATS_QUERY_SOURCE=auto โ†’ smart routing (recent โ†’ CH, historical โ†’ SQLite)
```
### Delta Merge Pattern
@@ -418,6 +456,73 @@ SurgeCollector
---
+## ClickHouse Module Design
+
+### Writer (ClickHouseWriter)
+
+```
+ClickHouseWriter
+ โ”‚
+ โ”œโ”€โ”€ Health Tracking
+ โ”‚ consecutiveFailures โ”€โ”€ consecutive failure counter
+ โ”‚ isHealthy() โ”€โ”€ true if < CH_UNHEALTHY_THRESHOLD
+ โ”‚
+ โ”œโ”€โ”€ insertRows()
+ โ”‚ โ”œโ”€โ”€ Normal path โ†’ HTTP POST โ†’ ClickHouse Buffer table
+ โ”‚ โ”‚ success: reset consecutiveFailures, log recovery
+ โ”‚ โ””โ”€โ”€ On error: consecutiveFailures++
+ โ”‚ reaching threshold โ†’ mark unhealthy + warn log
+ โ”‚
+ โ””โ”€โ”€ Backpressure protection
+ pendingBatches โ‰ฅ CH_WRITE_MAX_PENDING_BATCHES โ†’ drop incoming batch
+```
+
+### Reader (ClickHouseReader)
+
+```
+ClickHouseReader
+ โ”‚
+ โ”œโ”€โ”€ query(sql)
+ โ”‚ HTTP POST body carries SQL (avoids URL length limits)
+ โ”‚ โ†’ FORMAT JSON โ†’ parse result set
+ โ”‚
+ โ””โ”€โ”€ toDateTime(value)
+ valid date โ†’ format as ClickHouse datetime string
+ invalid date โ†’ clamp to current time (prevents epoch โ†’ full table scan)
+```
+
+### Dual-Write Dispatcher (BatchBuffer)
+
+```
+BatchBuffer.flush()
+ โ”‚
+ โ”œโ”€โ”€ 1. Evaluate write mode
+ โ”‚ clickHouseWriter.isHealthy() โ†’ true
+ โ”‚ โ†’ shouldSkipSqliteStatsWrites(true)
+ โ”‚ โ†’ CH_ONLY_MODE=1: skip SQLite stats writes
+ โ”‚ clickHouseWriter.isHealthy() โ†’ false
+ โ”‚ โ†’ force SQLite write (regardless of CH_ONLY_MODE)
+ โ”‚
+ โ”œโ”€โ”€ 2. SQLite write (conditional on mode)
+ โ”‚
+ โ””โ”€โ”€ 3. ClickHouse write (parallel, when CH_WRITE_ENABLED=1)
+ failure โ†’ does not affect SQLite; tracked internally by Writer
+```
+
+### Read Routing (StatsService)
+
+```
+STATS_QUERY_SOURCE env var
+ โ”‚
+ โ”œโ”€โ”€ sqlite โ†’ all queries via SQLite Repository (default)
+ โ”œโ”€โ”€ clickhouse โ†’ all queries via ClickHouseReader
+ โ””โ”€โ”€ auto โ†’ shouldUseClickHouse()
+ โ”œโ”€โ”€ valid time range โ†’ ClickHouseReader
+ โ””โ”€โ”€ invalid time range โ†’ SQLite (fallback)
+```
+
+---
+
## Authentication Flow (Cookie-Based)
```
@@ -440,16 +545,18 @@ Subsequent requests automatically carry Cookie
## Performance Optimization
-| Layer | Optimization Technique | Effect |
-| ------------ | ------------------------------------- | --------------------------- |
-| **Collect** | Batch Write (Batch Buffer) | Reduce 90% DB writes |
-| **Collect** | GeoIP Batch Query + Cache | Reduce 80% external requests|
-| **Collect** | Policy Cache Sync | Reduce 95% API calls |
-| **Collect** | Exponential Backoff Retry | Improve connection stability|
-| **Query** | RealtimeStore Delta Merge | Real-time data < 100ms |
-| **Query** | WebSocket Summary Cache (2s TTL) | Reduce 70% DB queries |
-| **Storage** | SQLite WAL Mode | Concurrent read/write |
-| **Storage** | Data Retention Policy (Auto-cleanup) | Control storage growth |
+| Layer | Optimization Technique | Effect |
+| ------------ | ------------------------------------- | -------------------------------- |
+| **Collect** | Batch Write (Batch Buffer) | Reduce 90% DB writes |
+| **Collect** | GeoIP Batch Query + Cache | Reduce 80% external requests |
+| **Collect** | Policy Cache Sync | Reduce 95% API calls |
+| **Collect** | Exponential Backoff Retry | Improve connection stability |
+| **Query** | RealtimeStore Delta Merge | Real-time data < 100ms |
+| **Query** | WebSocket Summary Cache (2s TTL) | Reduce 70% DB queries |
+| **Query** | ClickHouse columnar storage (optional)| 10x+ speedup for wide time ranges|
+| **Storage** | SQLite WAL Mode | Concurrent read/write |
+| **Storage** | ClickHouse SummingMergeTree | Auto-dedup aggregation, less I/O |
+| **Storage** | Data Retention Policy (Auto-cleanup) | Control storage growth |
---
@@ -524,7 +631,15 @@ neko-master/
โ”‚ โ”‚ โ”œโ”€โ”€ modules/ # Business modules
โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ auth/ # Authentication
โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ backend/ # Backend management
-โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ stats/ # Stats service
+โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ clickhouse/ # ClickHouse module (optional)
+โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ clickhouse.config.ts # Config loader
+โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ clickhouse.writer.ts # Writer (dual-write + health tracking)
+โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ clickhouse.reader.ts # Reader (POST-body SQL queries)
+โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ collector/ # Collector core
+โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ batch-buffer.ts # Batch buffer + dual-write dispatcher
+โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ stats/ # Stats service (read routing)
+โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ stats.service.ts # Read routing (STATS_QUERY_SOURCE)
+โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ stats-write-mode.ts # Write mode decision
โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ surge/ # Surge service
โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ realtime/ # Real-time data
โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ websocket/ # WebSocket
diff --git a/docs/architecture.md b/docs/architecture.md
index d7b94ae..24e31b3 100644
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -132,13 +132,14 @@
โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
โ”‚
- โ”‚ SQLite (better-sqlite3)
+ โ”œโ”€โ”€โ†’ SQLite (better-sqlite3) [ๅง‹็ปˆๅ†™ๅ…ฅ]
+ โ””โ”€โ”€โ†’ ClickHouse HTTP API [ๅฏ้€‰ๅŒๅ†™]
โ–ผ
โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
โ”‚ ๆ•ฐๆฎๅญ˜ๅ‚จๅฑ‚ (Storage) โ”‚
โ”‚ โ”‚
โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚
-โ”‚ โ”‚ SQLite Database (WAL Mode) โ”‚ โ”‚
+โ”‚ โ”‚ SQLite Database (WAL Mode) [ๅง‹็ปˆๅฏ็”จ] โ”‚ โ”‚
โ”‚ โ”‚ โ”‚ โ”‚
โ”‚ โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ โ”‚
โ”‚ โ”‚ โ”‚ ็ปŸ่ฎก่กจ (ๆŒ‰ backend_id ๅˆ†ๅŒบ) โ”‚ โ”‚ โ”‚
@@ -171,6 +172,29 @@
โ”‚ โ”‚ โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ โ”‚ โ”‚
โ”‚ โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ โ”‚
โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚
+โ”‚ โ”‚
+โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚
+โ”‚ โ”‚ ClickHouse Database [ๅฏ้€‰๏ผŒCH_ENABLED=1 ๆ—ถๅฏ็”จ] โ”‚ โ”‚
+โ”‚ โ”‚ โ”‚ โ”‚
+โ”‚ โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ โ”‚
+โ”‚ โ”‚ โ”‚ Buffer ่กจ๏ผˆๅผ‚ๆญฅๆŽฅๆ”ถๅ†™ๅ…ฅ๏ผŒ็บฆ 5min ๅˆๅนถ๏ผ‰ โ”‚ โ”‚ โ”‚
+โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚
+โ”‚ โ”‚ โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ โ”‚ โ”‚
+โ”‚ โ”‚ โ”‚ โ”‚ traffic_detail_bufferโ”‚ โ”‚ traffic_agg_buffer โ”‚ โ”‚ โ”‚ โ”‚
+โ”‚ โ”‚ โ”‚ โ”‚ (่ฟžๆŽฅ่ฏฆๆƒ…) โ”‚ โ”‚ (่šๅˆ็ปŸ่ฎก) โ”‚ โ”‚ โ”‚ โ”‚
+โ”‚ โ”‚ โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ โ”‚ โ”‚
+โ”‚ โ”‚ โ”‚ โ”‚ merge โ”‚ merge โ”‚ โ”‚ โ”‚
+โ”‚ โ”‚ โ”‚ โ–ผ โ–ผ โ”‚ โ”‚ โ”‚
+โ”‚ โ”‚ โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ โ”‚ โ”‚
+โ”‚ โ”‚ โ”‚ โ”‚ traffic_detail โ”‚ โ”‚ traffic_agg โ”‚ โ”‚ โ”‚ โ”‚
+โ”‚ โ”‚ โ”‚ โ”‚ (SummingMergeTree) โ”‚ โ”‚ (SummingMergeTree) โ”‚ โ”‚ โ”‚ โ”‚
+โ”‚ โ”‚ โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ โ”‚ โ”‚
+โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚
+โ”‚ โ”‚ โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ โ”‚ โ”‚
+โ”‚ โ”‚ โ”‚ โ”‚ country_buffer โ”‚ โ†’ country_stats (SummingMergeTree) โ”‚ โ”‚ โ”‚
+โ”‚ โ”‚ โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ โ”‚ โ”‚
+โ”‚ โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ โ”‚
+โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
โ–ฒ
โ”‚
@@ -236,7 +260,8 @@ ClashCollector
โ”‚ - ๆ‰น้‡็ผ“ๅญ˜ไผ˜ๅŒ–
โ”‚
โ”œโ”€โ”€ 4. ๆ•ฐๆฎๅ†™ๅ…ฅ
- โ”‚ โ”œโ”€โ”€โ†’ SQLite (ๆŒไน…ๅŒ–)
+ โ”‚ โ”œโ”€โ”€โ†’ SQLite (ๆŒไน…ๅŒ–๏ผŒๅง‹็ปˆๅ†™ๅ…ฅ)
+ โ”‚ โ”œโ”€โ”€โ†’ ClickHouse (ๅฏ้€‰ๅŒๅ†™๏ผŒCH_WRITE_ENABLED=1 ๆ—ถ)
โ”‚ โ””โ”€โ”€โ†’ RealtimeStore (ๅ†…ๅญ˜ๅฎžๆ—ถ)
โ”‚
โ””โ”€โ”€ 5. ่งฆๅ‘ๅนฟๆ’ญ
@@ -271,7 +296,8 @@ SurgeCollector
โ”‚ - 30s ๅฎšๆ—ถ flush
โ”‚
โ”œโ”€โ”€ 5. ๆ•ฐๆฎๅ†™ๅ…ฅ
- โ”‚ โ”œโ”€โ”€โ†’ SQLite (ๆŒไน…ๅŒ–)
+ โ”‚ โ”œโ”€โ”€โ†’ SQLite (ๆŒไน…ๅŒ–๏ผŒๅง‹็ปˆๅ†™ๅ…ฅ)
+ โ”‚ โ”œโ”€โ”€โ†’ ClickHouse (ๅฏ้€‰ๅŒๅ†™๏ผŒCH_WRITE_ENABLED=1 ๆ—ถ)
โ”‚ โ””โ”€โ”€โ†’ RealtimeStore (ๅ†…ๅญ˜ๅฎžๆ—ถ)
โ”‚
โ””โ”€โ”€ 6. ่งฆๅ‘ๅนฟๆ’ญ
@@ -288,7 +314,7 @@ WebSocketServer
โ”‚ - ๅปบ็ซ‹ ClientInfo (backend, range, interval)
โ”‚
โ”œโ”€โ”€ ๆ•ฐๆฎๅ‡†ๅค‡
- โ”‚ - ไปŽ SQLite ๆŸฅ่ฏขๅŸบ็ก€ๆ•ฐๆฎ
+ โ”‚ - ไปŽ SQLite ๆˆ– ClickHouse ๆŸฅ่ฏขๅŸบ็ก€ๆ•ฐๆฎ๏ผˆ็”ฑ STATS_QUERY_SOURCE ๅ†ณๅฎš๏ผ‰
โ”‚ - RealtimeStore.merge*() ๅˆๅนถๅฎžๆ—ถๅขž้‡
โ”‚ - ๆ‘˜่ฆ็ผ“ๅญ˜ (2s TTL)
โ”‚
@@ -354,16 +380,28 @@ BaseRepository (ๆŠฝ่ฑกๅŸบ็ฑป)
โ””โ”€โ”€ ... ๅ…ฑ 13 ไธชๅ…ฑไบซๅทฅๅ…ทๆ–นๆณ•
```
-### ๅŒๅ†™ๆจกๅผ (Dual Write)
+### ไธ‰ๅ†™ๆจกๅผ (Triple Write)
```
Collector ๆ”ถๅˆฐๆต้‡ๆ•ฐๆฎ
โ”‚
- โ”œโ”€โ”€โ†’ SQLite (ๆŒไน…ๅŒ–ๅญ˜ๅ‚จ)
- โ”‚ โ””โ”€ ็”จไบŽๅކๅฒๆŸฅ่ฏขใ€่šๅˆ็ปŸ่ฎก
+ โ”œโ”€โ”€โ†’ SQLite (ๆŒไน…ๅŒ–ๅญ˜ๅ‚จ๏ผŒๅง‹็ปˆๅ†™ๅ…ฅ)
+ โ”‚ โ””โ”€ ้…็ฝฎ / ๅ…ƒๆ•ฐๆฎ / ็ปŸ่ฎกๅކๅฒ๏ผˆCH ไธๅฏ็”จๆ—ถ่‡ชๅŠจๅ›ž้€€๏ผ‰
+ โ”‚
+ โ”œโ”€โ”€โ†’ ClickHouse (ๅฏ้€‰๏ผŒCH_WRITE_ENABLED=1)
+ โ”‚ โ””โ”€ ๆต้‡็ปŸ่ฎกๅŒๅ†™ โ†’ Buffer ่กจ โ†’ SummingMergeTree ๅผ‚ๆญฅๅˆๅนถ
+ โ”‚ โ””โ”€ ๅฅๅบทๅ›ž้€€๏ผš่ฟž็ปญๅคฑ่ดฅ CH_UNHEALTHY_THRESHOLD ๆฌกๅŽ่‡ชๅŠจๅ›ž้€€ SQLite
โ”‚
โ””โ”€โ”€โ†’ RealtimeStore (ๅ†…ๅญ˜ๅฎžๆ—ถ)
- โ””โ”€ ็”จไบŽๅฎžๆ—ถๅฑ•็คบใ€ไฝŽๅปถ่ฟŸๆŽจ้€
+ โ””โ”€ ็”จไบŽๅฎžๆ—ถๅฑ•็คบใ€ไฝŽๅปถ่ฟŸๆŽจ้€๏ผˆ่กฅๅฟ Buffer ๅปถ่ฟŸ๏ผ‰
+```
+
+**่ฏปๅ–่ทฏ็”ฑ๏ผˆSTATS_QUERY_SOURCE๏ผ‰**
+
+```
+STATS_QUERY_SOURCE=sqlite โ†’ ๅ…จ้ƒจ่ฏป SQLite๏ผˆ้ป˜่ฎค๏ผ‰
+STATS_QUERY_SOURCE=clickhouse โ†’ ๅ…จ้ƒจ่ฏป ClickHouse
+STATS_QUERY_SOURCE=auto โ†’ ๆ™บ่ƒฝ่ทฏ็”ฑ๏ผˆ่ฟ‘ๆœŸ่ตฐ CH๏ผŒๅކๅฒ่ตฐ SQLite๏ผ‰
```
### ๅขž้‡ๅˆๅนถๆจกๅผ (Delta Merge)
@@ -418,6 +456,72 @@ SurgeCollector
---
+## ClickHouse ๆจกๅ—่ฎพ่ฎก
+
+### ๅ†™ๅ…ฅๅ™จ (ClickHouseWriter)
+
+```
+ClickHouseWriter
+ โ”‚
+ โ”œโ”€โ”€ ๅฅๅบท็Šถๆ€่ฟฝ่ธช
+ โ”‚ consecutiveFailures โ”€โ”€ ่ฟž็ปญๅคฑ่ดฅ่ฎกๆ•ฐ
+ โ”‚ isHealthy() โ”€โ”€ ๅˆคๆ–ญๆ˜ฏๅฆๅฅๅบท๏ผˆ< CH_UNHEALTHY_THRESHOLD๏ผ‰
+ โ”‚
+ โ”œโ”€โ”€ insertRows()
+ โ”‚ โ”œโ”€โ”€ ๆญฃๅธธๅ†™ๅ…ฅ โ†’ HTTP POST โ†’ ClickHouse Buffer ่กจ
+ โ”‚ โ”‚ ๆˆๅŠŸๆ—ถ: consecutiveFailures ๆธ…้›ถ๏ผŒๆ—ฅๅฟ—่ฎฐๅฝ•ๆขๅค
+ โ”‚ โ””โ”€โ”€ ๅคฑ่ดฅๆ—ถ: consecutiveFailures++
+ โ”‚ ่พพๅˆฐ้˜ˆๅ€ผ โ†’ ๆ ‡่ฎฐไธๅฅๅบท + ๅ‘Š่ญฆๆ—ฅๅฟ—
+ โ”‚
+ โ””โ”€โ”€ ๅ†™ๅ…ฅ็งฏๅŽ‹ไฟๆŠค
+ pendingBatches โ‰ฅ CH_WRITE_MAX_PENDING_BATCHES โ†’ ไธขๅผƒๆ–ฐๆ‰นๆฌก
+```
+
+### ่ฏปๅ–ๅ™จ (ClickHouseReader)
+
+```
+ClickHouseReader
+ โ”‚
+ โ”œโ”€โ”€ query(sql)
+ โ”‚ HTTP POST body ไผ ้€ SQL๏ผˆ้ฟๅ… URL ้•ฟๅบฆ้™ๅˆถ๏ผ‰
+ โ”‚ โ†’ FORMAT JSON โ†’ ่งฃๆž็ป“ๆžœ้›†
+ โ”‚
+ โ””โ”€โ”€ toDateTime(value)
+ ๆญฃๅธธๆ—ฅๆœŸ โ†’ ๆ ผๅผๅŒ–ไธบ ClickHouse datetime ๅญ—็ฌฆไธฒ
+ ๆ— ๆ•ˆๆ—ฅๆœŸ โ†’ ๅ–ๅฝ“ๅ‰ๆ—ถ้—ด๏ผˆ้˜ฒๆญข epoch ๅฏผ่‡ดๅ…จ่กจๆ‰ซๆ๏ผ‰
+```
+
+### ๅŒๅ†™่ฐƒๅบฆ (BatchBuffer)
+
+```
+BatchBuffer.flush()
+ โ”‚
+ โ”œโ”€โ”€ 1. ่ฏ„ไผฐๅ†™ๅ…ฅๆจกๅผ
+ โ”‚ clickHouseWriter.isHealthy() โ†’ true
+ โ”‚ โ†’ shouldSkipSqliteStatsWrites(true) โ†’ CH_ONLY_MODE=1 ๆ—ถ่ทณ่ฟ‡ SQLite ็ปŸ่ฎกๅ†™ๅ…ฅ
+ โ”‚ clickHouseWriter.isHealthy() โ†’ false
+ โ”‚ โ†’ ๅผบๅˆถๅ†™ SQLite๏ผˆๆ— ่ฎบ CH_ONLY_MODE ๅฆ‚ไฝ•๏ผ‰
+ โ”‚
+ โ”œโ”€โ”€ 2. SQLite ๅ†™ๅ…ฅ๏ผˆๆŒ‰ๆจกๅผๅ†ณๅฎšๆ˜ฏๅฆๆ‰ง่กŒ๏ผ‰
+ โ”‚
+ โ””โ”€โ”€ 3. ClickHouse ๅ†™ๅ…ฅ๏ผˆCH_WRITE_ENABLED=1 ๆ—ถๅนถ่กŒๆ‰ง่กŒ๏ผ‰
+ ๅ†™ๅ…ฅๅคฑ่ดฅ โ†’ ไธๅฝฑๅ“ SQLite๏ผ›้”™่ฏฏ็”ฑ Writer ๅ†…้ƒจ่ฟฝ่ธช
+```
+
+### ่ฏปๅ–่ทฏ็”ฑ (StatsService)
+
+```
+STATS_QUERY_SOURCE ็Žฏๅขƒๅ˜้‡
+ โ”‚
+ โ”œโ”€โ”€ sqlite โ†’ ๅ…จ้ƒจ่ตฐ SQLite Repository๏ผˆ้ป˜่ฎค๏ผ‰
+ โ”œโ”€โ”€ clickhouse โ†’ ๅ…จ้ƒจ่ตฐ ClickHouseReader
+ โ””โ”€โ”€ auto โ†’ shouldUseClickHouse()
+ โ”œโ”€โ”€ ๆ—ถ้—ด่Œƒๅ›ดๆœ‰ๆ•ˆ โ†’ ClickHouseReader
+ โ””โ”€โ”€ ๆ—ถ้—ด่Œƒๅ›ดๆ— ๆ•ˆ โ†’ SQLite๏ผˆๅ…œๅบ•๏ผ‰
+```
+
+---
+
## ่ฎค่ฏๆต็จ‹ (Cookie-Based)
```
@@ -448,7 +552,9 @@ POST /api/auth/login
| **้‡‡้›†** | ๆŒ‡ๆ•ฐ้€€้ฟ้‡่ฏ• | ๆ้ซ˜่ฟžๆŽฅ็จณๅฎšๆ€ง |
| **ๆŸฅ่ฏข** | RealtimeStore ๅขž้‡ๅˆๅนถ | ๅฎžๆ—ถๆ•ฐๆฎ < 100ms |
| **ๆŸฅ่ฏข** | WebSocket ๆ‘˜่ฆ็ผ“ๅญ˜ (2s TTL) | ๅ‡ๅฐ‘ 70% DB ๆŸฅ่ฏข |
+| **ๆŸฅ่ฏข** | ClickHouse ๅˆ—ๅผๅญ˜ๅ‚จ๏ผˆๅฏ้€‰๏ผ‰ | ๅคงๆ—ถ้—ด็ช—ๅฃ่šๅˆๆ้€Ÿ 10x+ |
| **ๅญ˜ๅ‚จ** | SQLite WAL Mode | ๅนถๅ‘่ฏปๅ†™ไผ˜ๅŒ– |
+| **ๅญ˜ๅ‚จ** | ClickHouse SummingMergeTree | ่‡ชๅŠจๅŽป้‡่šๅˆ๏ผŒ่Š‚็œๅญ˜ๅ‚จ |
| **ๅญ˜ๅ‚จ** | ๆ•ฐๆฎไฟ็•™็ญ–็•ฅ (่‡ชๅŠจๆธ…็†) | ๆŽงๅˆถๅญ˜ๅ‚จๅขž้•ฟ |
---
@@ -524,7 +630,15 @@ neko-master/
โ”‚ โ”‚ โ”œโ”€โ”€ modules/ # ไธšๅŠกๆจกๅ—
โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ auth/ # ่ฎค่ฏ
โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ backend/ # ๅŽ็ซฏ็ฎก็†
-โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ stats/ # ็ปŸ่ฎกๆœๅŠก
+โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ clickhouse/ # ClickHouse ๆจกๅ—๏ผˆๅฏ้€‰๏ผ‰
+โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ clickhouse.config.ts # ้…็ฝฎๅŠ ่ฝฝ
+โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ clickhouse.writer.ts # ๅ†™ๅ…ฅๅ™จ๏ผˆๅŒๅ†™ + ๅฅๅบท่ฟฝ่ธช๏ผ‰
+โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ clickhouse.reader.ts # ่ฏปๅ–ๅ™จ๏ผˆPOST ๆŸฅ่ฏข๏ผ‰
+โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ collector/ # ้‡‡้›†ๆ ธๅฟƒ
+โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ batch-buffer.ts # ๆ‰น้‡็ผ“ๅ†ฒ + ๅŒๅ†™่ฐƒๅบฆ
+โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ stats/ # ็ปŸ่ฎกๆœๅŠก๏ผˆ่ฏปๅ–่ทฏ็”ฑ๏ผ‰
+โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ stats.service.ts # ่ฏปๅ–่ทฏ็”ฑ๏ผˆSTATS_QUERY_SOURCE๏ผ‰
+โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ stats-write-mode.ts # ๅ†™ๅ…ฅๆจกๅผๅ†ณ็ญ–
โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ surge/ # Surge ๆœๅŠก
โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ realtime/ # ๅฎžๆ—ถๆ•ฐๆฎ
โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ websocket/ # WebSocket