Memos is an open-source, self-hosted note-taking app that brings your everyday thoughts together in a personal timeline. Your data stays yours, and you choose what to keep private or share.
Go dockerfossgomarkdownmemomicroblognote-takingnotesopen-sourceown-your-datareactself-hostedsqlitetimeline
docs: create AGENTS.md for AI agents and remove outdated CLAUDE.md
Steven9f5a7bb889ee
diff --git a/AGENTS.md b/AGENTS.mdnew file mode 100644index 000000000..fe20cce0d--- /dev/null+++ b/AGENTS.md@@ -0,0 +1,579 @@+# Memos Codebase Guide for AI Agents++This document provides comprehensive guidance for AI agents working with the Memos codebase. It covers architecture, workflows, conventions, and key patterns.++## Project Overview++Memos is a self-hosted knowledge management platform built with:+- **Backend:** Go 1.25 with gRPC + Connect RPC+- **Frontend:** React 18.3 + TypeScript + Vite 7+- **Databases:** SQLite (default), MySQL, PostgreSQL+- **Protocol:** Protocol Buffers (v2) with buf for code generation+- **API Layer:** Dual protocol - Connect RPC (browsers) + gRPC-Gateway (REST)++## Architecture++### Backend Architecture++```+cmd/memos/ # Entry point+└── main.go # Cobra CLI, profile setup, server initialization++server/+├── server.go # Echo HTTP server, healthz, background runners+├── auth/ # Authentication (JWT, PAT, session)+├── router/+│ ├── api/v1/ # gRPC service implementations+│ │ ├── v1.go # Service registration, gateway & Connect setup+│ │ ├── acl_config.go # Public endpoints whitelist+│ │ ├── connect_services.go # Connect RPC handlers+│ │ ├── connect_interceptors.go # Auth, logging, recovery+│ │ └── *_service.go # Individual services (memo, user, etc.)+│ ├── frontend/ # Static file serving (SPA)+│ ├── fileserver/ # Native HTTP file serving for media+│ └── rss/ # RSS feed generation+└── runner/+ ├── memopayload/ # Memo payload processing (tags, links, tasks)+ └── s3presign/ # S3 presigned URL management++store/ # Data layer with caching+├── driver.go # Driver interface (database operations)+├── store.go # Store wrapper with cache layer+├── cache.go # In-memory caching (instance settings, users)+├── migrator.go # Database migrations+├── db/+│ ├── db.go # Driver factory+│ ├── sqlite/ # SQLite implementation+│ ├── mysql/ # MySQL implementation+│ └── postgres/ # PostgreSQL implementation+└── migration/ # SQL migration files (embedded)++proto/ # Protocol Buffer definitions+├── api/v1/ # API v1 service definitions+└── gen/ # Generated Go & TypeScript code+```++### Frontend Architecture++```+web/+├── src/+│ ├── components/ # React components+│ ├── contexts/ # React Context (client state)+│ │ ├── AuthContext.tsx # Current user, auth state+│ │ ├── ViewContext.tsx # Layout, sort order+│ │ └── MemoFilterContext.tsx # Filters, shortcuts+│ ├── hooks/ # React Query hooks (server state)+│ │ ├── useMemoQueries.ts # Memo CRUD, pagination+│ │ ├── useUserQueries.ts # User operations+│ │ ├── useAttachmentQueries.ts # Attachment operations+│ │ └── ...+│ ├── lib/ # Utilities+│ │ ├── query-client.ts # React Query v5 client+│ │ └── connect.ts # Connect RPC client setup+│ ├── pages/ # Page components+│ └── types/proto/ # Generated TypeScript from .proto+├── package.json # Dependencies+└── vite.config.mts # Vite config with dev proxy++plugin/ # Backend plugins+├── scheduler/ # Cron jobs+├── email/ # Email delivery+├── filter/ # CEL filter expressions+├── webhook/ # Webhook dispatch+├── markdown/ # Markdown parsing & rendering+├── httpgetter/ # HTTP fetching (metadata, images)+└── storage/s3/ # S3 storage backend+```++## Key Architectural Patterns++### 1. API Layer: Dual Protocol++**Connect RPC (Browser Clients):**+- Protocol: `connectrpc.com/connect`+- Base path: `/memos.api.v1.*`+- Interceptor chain: Metadata → Logging → Recovery → Auth+- Returns type-safe responses to React frontend+- See: `server/router/api/v1/connect_interceptors.go:177-227`++**gRPC-Gateway (REST API):**+- Protocol: Standard HTTP/JSON+- Base path: `/api/v1/*`+- Uses same service implementations as Connect+- Useful for external tools, CLI clients+- See: `server/router/api/v1/v1.go:52-96`++**Authentication:**+- JWT Access Tokens (V2): Stateless, 15-min expiration, verified via `AuthenticateByAccessTokenV2`+- Personal Access Tokens (PAT): Stateful, long-lived, validated against database+- Both use `Authorization: Bearer <token>` header+- See: `server/auth/authenticator.go:17-166`++### 2. Store Layer: Interface Pattern++All database operations go through the `Driver` interface:+```go+type Driver interface {+ GetDB() *sql.DB+ Close() error++ IsInitialized(ctx context.Context) (bool, error)++ CreateMemo(ctx context.Context, create *Memo) (*Memo, error)+ ListMemos(ctx context.Context, find *FindMemo) ([]*Memo, error)+ UpdateMemo(ctx context.Context, update *UpdateMemo) error+ DeleteMemo(ctx context.Context, delete *DeleteMemo) error++ // ... similar methods for all resources+}+```++**Three Implementations:**+- `store/db/sqlite/` - SQLite (modernc.org/sqlite)+- `store/db/mysql/` - MySQL (go-sql-driver/mysql)+- `store/db/postgres/` - PostgreSQL (lib/pq)++**Caching Strategy:**+- Store wrapper maintains in-memory caches for:+ - Instance settings (`instanceSettingCache`)+ - Users (`userCache`)+ - User settings (`userSettingCache`)+- Config: Default TTL 10 min, cleanup interval 5 min, max 1000 items+- See: `store/store.go:10-57`++### 3. Frontend State Management++**React Query v5 (Server State):**+- All API calls go through custom hooks in `web/src/hooks/`+- Query keys organized by resource: `memoKeys`, `userKeys`, `attachmentKeys`+- Default staleTime: 30s, gcTime: 5min+- Automatic refetch on window focus, reconnect+- See: `web/src/lib/query-client.ts`++**React Context (Client State):**+- `AuthContext`: Current user, auth initialization, logout+- `ViewContext`: Layout mode (LIST/MASONRY), sort order+- `MemoFilterContext`: Active filters, shortcut selection, URL sync++### 4. Database Migration System++**Migration Flow:**+1. `preMigrate`: Check if DB exists. If not, apply `LATEST.sql`+2. `checkMinimumUpgradeVersion`: Reject pre-0.22 installations+3. `applyMigrations`: Apply incremental migrations in single transaction+4. Demo mode: Seed with demo data++**Schema Versioning:**+- Stored in `instance_setting` table (key: `bb.general.version`)+- Format: `major.minor.patch`+- Migration files: `store/migration/{driver}/{version}/NN__description.sql`+- See: `store/migrator.go:21-414`++### 5. Protocol Buffer Code Generation++**Definition Location:** `proto/api/v1/*.proto`++**Regeneration:**+```bash+cd proto && buf generate+```++**Generated Outputs:**+- Go: `proto/gen/api/v1/` (used by backend services)+- TypeScript: `web/src/types/proto/api/v1/` (used by frontend)++**Linting:** `proto/buf.yaml` - BASIC lint rules, FILE breaking changes++## Development Commands++### Backend++```bash+# Start dev server+go run ./cmd/memos --mode dev --port 8081++# Run all tests+go test ./...++# Run tests for specific package+go test ./store/...+go test ./server/router/api/v1/test/...++# Lint (golangci-lint)+golangci-lint run++# Format imports+goimports -w .++# Run with MySQL/Postgres+DRIVER=mysql go run ./cmd/memos+DRIVER=postgres go run ./cmd/memos+```++### Frontend++```bash+# Install dependencies+cd web && pnpm install++# Start dev server (proxies API to localhost:8081)+pnpm dev++# Type checking+pnpm lint++# Auto-fix lint issues+pnpm lint:fix++# Format code+pnpm format++# Build for production+pnpm build++# Build and copy to backend+pnpm release+```++### Protocol Buffers++```bash+# Regenerate Go and TypeScript from .proto files+cd proto && buf generate++# Lint proto files+cd proto && buf lint++# Check for breaking changes+cd proto && buf breaking --against .git#main+```++## Key Workflows++### Adding a New API Endpoint++1. **Define in Protocol Buffer:**+ - Edit `proto/api/v1/*_service.proto`+ - Add request/response messages+ - Add RPC method to service++2. **Regenerate Code:**+ ```bash+ cd proto && buf generate+ ```++3. **Implement Service (Backend):**+ - Add method to `server/router/api/v1/*_service.go`+ - Follow existing patterns: fetch user, validate, call store+ - Add Connect wrapper to `server/router/api/v1/connect_services.go` (optional, same implementation)++4. **If Public Endpoint:**+ - Add to `server/router/api/v1/acl_config.go:11-34`++5. **Create Frontend Hook (if needed):**+ - Add query/mutation to `web/src/hooks/use*Queries.ts`+ - Use existing query key factories++### Database Schema Changes++1. **Create Migration Files:**+ ```+ store/migration/sqlite/0.28/1__add_new_column.sql+ store/migration/mysql/0.28/1__add_new_column.sql+ store/migration/postgres/0.28/1__add_new_column.sql+ ```++2. **Update LATEST.sql:**+ - Add change to `store/migration/{driver}/LATEST.sql`++3. **Update Store Interface (if new table/model):**+ - Add methods to `store/driver.go:8-71`+ - Implement in `store/db/{driver}/*.go`++4. **Test Migration:**+ - Run `go test ./store/test/...` to verify++### Adding a New Frontend Page++1. **Create Page Component:**+ - Add to `web/src/pages/NewPage.tsx`+ - Use existing hooks for data fetching++2. **Add Route:**+ - Edit `web/src/App.tsx` (or router configuration)++3. **Use React Query:**+ ```typescript+ import { useMemos } from "@/hooks/useMemoQueries";+ const { data, isLoading } = useMemos({ filter: "..." });+ ```++4. **Use Context for Client State:**+ ```typescript+ import { useView } from "@/contexts/ViewContext";+ const { layout, toggleSortOrder } = useView();+ ```++## Testing++### Backend Tests++**Test Pattern:**+```go+func TestMemoCreation(t *testing.T) {+ ctx := context.Background()+ store := test.NewTestingStore(ctx, t)++ // Create test user+ user, _ := createTestUser(ctx, store, t)++ // Execute operation+ memo, err := store.CreateMemo(ctx, &store.Memo{+ CreatorID: user.ID,+ Content: "Test memo",+ // ...+ })+ require.NoError(t, err)+ assert.NotNil(t, memo)+}+```++**Test Utilities:**+- `store/test/store.go:22-35` - `NewTestingStore()` creates isolated DB+- `store/test/store.go:37-77` - `resetTestingDB()` cleans tables+- Test DB determined by `DRIVER` env var (default: sqlite)++**Running Tests:**+```bash+# All tests+go test ./...++# Specific package+go test ./store/...+go test ./server/router/api/v1/test/...++# With coverage+go test -cover ./...+```++### Frontend Testing++**TypeScript Checking:**+```bash+cd web && pnpm lint+```++**No Automated Tests:**+- Frontend relies on TypeScript checking and manual validation+- React Query DevTools available in dev mode (bottom-left)++## Code Conventions++### Go++**Error Handling:**+- Use `github.com/pkg/errors` for wrapping: `errors.Wrap(err, "context")`+- Return structured gRPC errors: `status.Errorf(codes.NotFound, "message")`++**Naming:**+- Package names: lowercase, single word (e.g., `store`, `server`)+- Interfaces: `Driver`, `Store`, `Service`+- Methods: PascalCase for exported, camelCase for internal++**Comments:**+- Public exported functions must have comments (godot enforces)+- Use `//` for single-line, `/* */` for multi-line++**Imports:**+- Grouped: stdlib, third-party, local+- Sorted alphabetically within groups+- Use `goimports -w .` to format++### TypeScript/React++**Components:**+- Functional components with hooks+- Use `useMemo`, `useCallback` for optimization+- Props interfaces: `interface Props { ... }`++**State Management:**+- Server state: React Query hooks+- Client state: React Context+- Avoid direct useState for server data++**Styling:**+- Tailwind CSS v4 via `@tailwindcss/vite`+- Use `clsx` and `tailwind-merge` for conditional classes++**Imports:**+- Absolute imports with `@/` alias+- Group: React, third-party, local+- Auto-organized by Biome++## Important Files Reference++### Backend Entry Points++| File | Purpose |+|------|---------|+| `cmd/memos/main.go` | Server entry point, CLI setup |+| `server/server.go` | Echo server initialization, background runners |+| `store/store.go` | Store wrapper with caching |+| `store/driver.go` | Database driver interface |++### API Layer++| File | Purpose |+|------|---------|+| `server/router/api/v1/v1.go` | Service registration, gateway setup |+| `server/router/api/v1/acl_config.go` | Public endpoints whitelist |+| `server/router/api/v1/connect_interceptors.go` | Connect interceptors |+| `server/auth/authenticator.go` | Authentication logic |++### Frontend Core++| File | Purpose |+|------|---------|+| `web/src/lib/query-client.ts` | React Query client configuration |+| `web/src/contexts/AuthContext.tsx` | User authentication state |+| `web/src/contexts/ViewContext.tsx` | UI preferences |+| `web/src/contexts/MemoFilterContext.tsx` | Filter state |+| `web/src/hooks/useMemoQueries.ts` | Memo queries/mutations |++### Data Layer++| File | Purpose |+|------|---------|+| `store/memo.go` | Memo model definitions, store methods |+| `store/user.go` | User model definitions |+| `store/attachment.go` | Attachment model definitions |+| `store/migrator.go` | Migration logic |+| `store/db/db.go` | Driver factory |+| `store/db/sqlite/sqlite.go` | SQLite driver implementation |++## Configuration++### Backend Environment Variables++| Variable | Default | Description |+|----------|----------|-------------|+| `MEMOS_MODE` | `dev` | Mode: `dev`, `prod`, `demo` |+| `MEMOS_PORT` | `8081` | HTTP port |+| `MEMOS_ADDR` | `` | Bind address (empty = all) |+| `MEMOS_DATA` | `~/.memos` | Data directory |+| `MEMOS_DRIVER` | `sqlite` | Database: `sqlite`, `mysql`, `postgres` |+| `MEMOS_DSN` | `` | Database connection string |+| `MEMOS_INSTANCE_URL` | `` | Instance base URL |++### Frontend Environment Variables++| Variable | Default | Description |+|----------|----------|-------------|+| `DEV_PROXY_SERVER` | `http://localhost:8081` | Backend proxy target |++## CI/CD++### GitHub Workflows++**Backend Tests** (`.github/workflows/backend-tests.yml`):+- Runs on `go.mod`, `go.sum`, `**.go` changes+- Steps: verify `go mod tidy`, golangci-lint, all tests++**Frontend Tests** (`.github/workflows/frontend-tests.yml`):+- Runs on `web/**` changes+- Steps: pnpm install, lint, build++**Proto Lint** (`.github/workflows/proto-linter.yml`):+- Runs on `.proto` changes+- Steps: buf lint, buf breaking check++### Linting Configuration++**Go** (`.golangci.yaml`):+- Linters: revive, govet, staticcheck, misspell, gocritic, etc.+- Formatter: goimports+- Forbidden: `fmt.Errorf`, `ioutil.ReadDir`++**TypeScript** (`web/biome.json`):+- Linting: Biome (ESLint replacement)+- Formatting: Biome (Prettier replacement)+- Line width: 140 characters+- Semicolons: always++## Common Tasks++### Debugging Database Issues++1. Check connection string in logs+2. Verify `store/db/{driver}/migration/` files exist+3. Check schema version: `SELECT * FROM instance_setting WHERE key = 'bb.general.version'`+4. Test migration: `go test ./store/test/... -v`++### Debugging API Issues++1. Check Connect interceptor logs: `server/router/api/v1/connect_interceptors.go:79-105`+2. Verify endpoint is in `acl_config.go` if public+3. Check authentication via `auth/authenticator.go:133-165`+4. Test with curl: `curl -H "Authorization: Bearer <token>" http://localhost:8081/api/v1/...`++### Debugging Frontend State++1. Open React Query DevTools (bottom-left in dev)+2. Inspect query cache, mutations, refetch behavior+3. Check Context state via React DevTools+4. Verify filter state in MemoFilterContext++### Running Tests Against Multiple Databases++```bash+# SQLite (default)+DRIVER=sqlite go test ./...++# MySQL (requires running MySQL server)+DRIVER=mysql DSN="user:pass@tcp(localhost:3306)/memos" go test ./...++# PostgreSQL (requires running PostgreSQL server)+DRIVER=postgres DSN="postgres://user:pass@localhost:5432/memos" go test ./...+```++## Plugin System++Backend supports pluggable components in `plugin/`:++| Plugin | Purpose |+|--------|----------|+| `scheduler` | Cron-based job scheduling |+| `email` | SMTP email delivery |+| `filter` | CEL expression filtering |+| `webhook` | HTTP webhook dispatch |+| `markdown` | Markdown parsing (goldmark) |+| `httpgetter` | HTTP content fetching |+| `storage/s3` | S3-compatible storage |++Each plugin has its own README with usage examples.++## Performance Considerations++### Backend++- Database queries use pagination (`limit`, `offset`)+- In-memory caching reduces DB hits for frequently accessed data+- WAL journal mode for SQLite (reduces locking)+- Thumbnail generation limited to 3 concurrent operations++### Frontend++- React Query reduces redundant API calls+- Infinite queries for large lists (pagination)+- Manual chunks: `utils-vendor`, `mermaid-vendor`, `leaflet-vendor`+- Lazy loading for heavy components++## Security Notes++- JWT secrets must be kept secret (`MEMOS_MODE=prod` generates random secret)+- Personal Access Tokens stored as SHA-256 hashes in database+- CSRF protection via SameSite cookies+- CORS enabled for all origins (configure for production)+- Input validation at service layer+- SQL injection prevention via parameterized queriesdiff --git a/CLAUDE.md b/CLAUDE.mddeleted file mode 100644index a46a17ba6..000000000--- a/CLAUDE.md+++ /dev/null@@ -1,126 +0,0 @@-# Memos Codebase Guide--## Project Overview--Memos is a self-hosted knowledge management platform with a Go backend and React frontend. The architecture uses gRPC for internal communication with REST access via gRPC-Gateway.--## Architecture Decision Context--**Why gRPC + gRPC-Gateway?**-- Native gRPC for performance, REST API for compatibility-- Both protocols served on same port via `cmux` connection multiplexer-- Frontend uses gRPC-Web (`nice-grpc-web`) for type-safe API calls--**Why multi-database support?**-- Store interface (`store/driver.go`) abstracts persistence-- Three implementations: SQLite (default), MySQL, PostgreSQL-- Each driver has its own migration files in `store/db/{driver}/migration/`-- Schema version tracked in `instance_setting` table (key: `bb.general.version`)--**Why React Query + Context for frontend state?**-- **Server state** (memos, users, attachments) managed by React Query (TanStack Query v5)- - Automatic caching, deduplication, and background refetching- - Hooks in `web/src/hooks/useMemoQueries.ts`, `useUserQueries.ts`, `useAttachmentQueries.ts`-- **Client state** (UI preferences, filters) managed by React Context- - ViewContext (`web/src/contexts/ViewContext.tsx`) - layout, sort order- - MemoFilterContext (`web/src/contexts/MemoFilterContext.tsx`) - filter state-- **Legacy MobX** still present in some components (gradual migration in progress)- - Stores in `web/src/store/` used by unmigrated components- - Both systems coexist during transition period--## Critical Development Commands--**Backend:**-```bash-go run ./cmd/memos --mode dev --port 8081 # Start dev server-go test ./... # Run tests-golangci-lint run # Lint-```--**Frontend:**-```bash-cd web && pnpm dev # Start dev server (React Query devtools at bottom-left)-cd web && pnpm lint:fix # Lint and fix-cd web && pnpm release # Build and copy to backend-```--**Protocol Buffers:**-```bash-cd proto && buf generate # Regenerate Go + TypeScript from .proto-```--## Frontend State Management--**Using React Query (Server State):**-```typescript-// Fetch memos-import { useMemos, useMemo } from "@/hooks/useMemoQueries";-const { data: memos, isLoading } = useMemos({ filter });-const { data: memo } = useMemo(memoName);--// Mutations-import { useCreateMemo, useUpdateMemo } from "@/hooks/useMemoQueries";-const { mutate: createMemo } = useCreateMemo();-const { mutate: updateMemo } = useUpdateMemo();-```--**Using Context (Client State):**-```typescript-// View preferences-import { useView } from "@/contexts/ViewContext";-const { layout, setLayout, orderByTimeAsc, toggleSortOrder } = useView();--// Filters-import { useMemoFilter } from "@/contexts/MemoFilterContext";-const { filter, updateFilter } = useMemoFilter();-```--**React Query DevTools:**-- Available in dev mode at bottom-left corner-- Inspect query cache, mutations, and refetch behavior-- Query keys organized by resource: `memoKeys`, `userKeys`, `attachmentKeys`--**Migration Status:**-- ✅ Migrated: Home, MemoDetail, UserProfile, Inboxes pages-- 🔄 In Progress: Remaining pages and components (gradual migration)-- See `web/scripts/migration-guide.md` for migration patterns--## Key Workflows--**Modifying APIs:**-1. Edit `.proto` files in `proto/api/v1/`-2. Run `buf generate` to regenerate code-3. Implement in `server/router/api/v1/`-4. Frontend types auto-update in `web/src/types/proto/`--**Database Schema Changes:**-1. Create migration files: `store/migration/{sqlite,mysql,postgres}/{version}/NN__description.sql`-2. Update `LATEST.sql` in each driver directory-3. Schema version auto-determined from migration files-4. If adding new tables/models, also update `store/driver.go` interface and implementations--**Authentication Flow:**-- Interceptor runs on all gRPC methods (`server/router/api/v1/acl.go`)-- Public endpoints listed in `acl_config.go`-- Supports both session cookies and JWT bearer tokens--## Critical Path Components--**Entry point:** `cmd/memos/` starts the server-**API layer:** `server/router/api/v1/` implements gRPC services-**Data layer:** `store/` handles all persistence-**Frontend:** `web/src/` React app with React Query + Context (migrating from MobX)--## Testing Expectations--Go tests are required for store and API changes. Frontend relies on TypeScript checking and manual validation.--Run `go test ./store/...` and `go test ./server/router/api/v1/test/...` before committing backend changes.--## Configuration--Backend accepts flags or `MEMOS_*` environment variables:-- `--mode` / `MEMOS_MODE`: `dev`, `prod`, `demo`-- `--port` / `MEMOS_PORT`: HTTP/gRPC port (default: 5230)-- `--data` / `MEMOS_DATA`: Data directory (default: ~/.memos)-- `--driver` / `MEMOS_DRIVER`: `sqlite`, `mysql`, `postgres`