Memos is an open-source, self-hosted note-taking app that brings your everyday thoughts together in a personal timeline. Your data stays yours, and you choose what to keep private or share.
Go dockerfossgomarkdownmemomicroblognote-takingnotesopen-sourceown-your-datareactself-hostedsqlitetimeline
feat(d1): add bridge access mode (#6333)
Johnny5ddf4745a20d
diff --git a/docs/design/cloudflare-d1.md b/docs/design/cloudflare-d1.mdindex 415b15a45..fcb2cf260 100644--- a/docs/design/cloudflare-d1.md+++ b/docs/design/cloudflare-d1.md@@ -11,16 +11,19 @@ Related: [Multi-Spaces Design](multi-spaces.md)## SummaryMemos gains a fourth database engine, Cloudflare D1, selected with-`--driver d1`. D1 runs SQLite, but it is reached over Cloudflare's REST API-rather than a local file or socket, it exposes no interactive transactions,-and it accepts no custom SQL functions. Those three facts make it a different+`--driver d1`. D1 runs SQLite, but it is reached over HTTP rather than a+local file or socket, either through Cloudflare's REST API or through a+bridge Worker the deployment provides, it exposes no interactive+transactions, and it accepts no custom SQL functions. Those three facts make it a differentengine from the SQLite driver rather than a variant of it, so it lives in itsown package, `store/db/d1`, with its own migration directory, its own filterdialect, and its own test harness, exactly like MySQL and PostgreSQL.## Goals-- Run an unmodified Memos binary against a D1 database from any host.+- Run an unmodified Memos binary against a D1 database from any host, with+ a Cloudflare API token or with access the deployment already holds through+ a Worker binding.- Keep every store invariant the other drivers enforce: no partial writes,no stale-read overwrites, the same error values for the same conditions.- Share no Go code with the SQLite driver so either can change freely.@@ -40,26 +43,176 @@ dialect, and its own test harness, exactly like MySQL and PostgreSQL.## Configuration+Two access modes exist. Either one leaves the driver's behaviour, the schema,+and the data untouched; switching is a DSN change.++**REST mode** talks to the Cloudflare REST API with an API token that has the+D1 Edit permission:+```--driver d1 --dsn 'd1://<account_id>/<database_id>?token=<api_token>'```The token may instead come from `CLOUDFLARE_API_TOKEN` so it stays out ofprocess listings. An `endpoint` query parameter overrides the API base URL and-exists for the test emulator. The token needs the D1 Edit permission.+exists for the test emulator.++**Bridge mode** talks to a Worker the deployment provides. The Worker holds the+D1 binding and implements the bridge protocol below, so Memos needs no+Cloudflare API token. The bridge is reached in one of two ways.++*Private path.* When Memos runs as a Cloudflare Container, its Worker can+serve the bridge on a virtual hostname through an `outboundByHost` handler.+Cloudflare intercepts the container's plain-HTTP requests to that hostname+and runs the handler in the Workers runtime, on the same machine, reachable+only from the container's own egress. No credential is involved:++```+--driver d1 --dsn 'd1-bridge://d1.internal/d1?private=true'+```++`private=true` states that the platform keeps the path private; the driver+then uses plain HTTP and requires no secret. This is the recommended+deployment for Containers because there is nothing to provision or rotate.++*Public HTTPS endpoint.* A bridge exposed on a public route must be+protected by a shared secret. Generate one, for example with+`openssl rand -hex 32`, and configure the same value on both sides: Memos+reads `MEMOS_D1_BRIDGE_TOKEN` and sends it as `Authorization: Bearer`; the+Worker reads `BRIDGE_SECRET` and checks it before executing any SQL. Keep the+secret out of the DSN so it stays out of process listings:++```+MEMOS_D1_BRIDGE_TOKEN=<secret> memos --driver d1 --dsn 'd1-bridge://worker.example.com/d1'+```++The driver refuses a public bridge DSN without a secret. This secret is a+random credential the operator generates, not a Cloudflare API token.## Transport-The driver holds one HTTP client for the database. Reads and single writes-go to the `/raw` endpoint, which returns rows as arrays with a column list.-Bound parameters travel as JSON numbers, strings, and nulls; booleans become-integers. Binary values cannot travel in JSON, so blob columns are written-with `unhex(?)` bound to hex text and read back through `hex(blob)`.+The driver holds one HTTP transport for the database, selected by the access+mode. The driver methods see only its four operations: run one statement, run+a batch atomically, run a parameter-free script, and report the database size.++In REST mode reads and single writes go to the `/raw` endpoint, which returns+rows as arrays with a column list, batches use the documented `batch` body,+and scripts are sent as one semicolon-separated text. In bridge mode every+operation is one bridge request; a script is divided into its statements+locally because a bridge entry carries exactly one statement. That division+is a layout rule, not a SQL parser: the D1 scripts keep one statement per+paragraph, so a blank line is the boundary (see Schema and migrations).++Bound parameters travel as JSON numbers, strings, and nulls in both modes;+booleans become integers. Binary values cannot travel in JSON, so blob columns+are written with `unhex(?)` bound to hex text and read back through `hex(blob)`.++A thin `database/sql` adapter wraps the same transport so `Driver.GetDB()`+keeps working for the migrator and for tests. Its transactions buffer writes+and send them as one atomic request on commit. A read after a buffered write+is refused, because the write is not visible until the batch runs.++## Bridge protocol++Memos owns this protocol; the deployment repository owns the Worker that+implements it, its D1 binding, and its Wrangler configuration. The protocol+is one endpoint.-A thin `database/sql` adapter wraps the same client so `Driver.GetDB()` keeps-working for the migrator and for tests. Its transactions buffer writes and-send them as one atomic request on commit. A read after a buffered write is-refused, because the write is not visible until the batch runs.+**Request.** `POST <bridge-url>` with `Content-Type: application/json` and,+when a secret is configured, `Authorization: Bearer <secret>`. The body is:++```json+{"statements": [{"sql": "INSERT INTO memo (uid) VALUES (?)", "params": ["m1"]},+ {"sql": "SELECT id, uid FROM memo WHERE uid = ?", "params": ["m1"]}]}+```++- `statements` holds one or more entries, each exactly one SQL statement.+ `params` is optional and binds positionally; values are JSON strings,+ numbers, or `null`. Memos never sends more than 100 parameters per entry.+- The Worker must run the entries in order as one transaction, which is what+ the binding's `batch()` provides: if any statement fails, none commits.++**Response.** `200 OK` with one result per statement, in order:++```json+{"results": [+ {"columns": [], "rows": [], "meta": {"changes": 1, "last_row_id": 7, "size_after": 8192}},+ {"columns": ["id", "uid"], "rows": [[7, "m1"]], "meta": {"changes": 0, "last_row_id": 7, "size_after": 8192}}+]}+```++- `columns` lists the result columns in order and `rows` holds one array per+ row in that order. A statement without a result set returns both empty.+ Cell values are JSON numbers, strings, or `null`; a blob is an array of byte+ values, though Memos reads blobs through `hex()` and never relies on it.+- `meta.changes` is the number of rows the statement modified,+ `meta.last_row_id` the last inserted rowid, and `meta.size_after` the+ database size in bytes after the batch committed. Memos reports the size+ through `size_after`, so a Worker that omits it makes the size unavailable+ rather than wrong.++**Errors.** Any failure is a non-200 status with a body of+`{"error": {"message": "<text>"}}`. The message must carry the D1 error text+verbatim, for example `D1_ERROR: UNIQUE constraint failed: space.uid`: the+driver recognises constraint failures and guard aborts by that text. A+missing or wrong bearer secret should answer `401`.++Memos never issues a query whose result has two columns of the same name, so+a Worker may build `columns` from the keys of the first row object that+`batch()` returns and `rows` from each row's values in that order.++A Worker that meets the contract serves the private path from the+Container's outbound handler and, if a public route is wanted as well, checks+the shared secret on `fetch`. A public route with no secret configured is+refused rather than left open:++```js+import { Container, ContainerProxy } from "@cloudflare/containers";+export { ContainerProxy };++async function runBatch(env, request) {+ const { statements } = await request.json();+ const prepared = statements.map((s) => env.DB.prepare(s.sql).bind(...(s.params ?? [])));+ try {+ const results = (await env.DB.batch(prepared)).map(({ results, meta }) => {+ const columns = results.length ? Object.keys(results[0]) : [];+ return {+ columns,+ rows: results.map((row) => columns.map((c) => row[c])),+ meta: { changes: meta.changes, last_row_id: meta.last_row_id, size_after: meta.size_after },+ };+ });+ return Response.json({ results });+ } catch (err) {+ return error(400, err.message);+ }+}++const error = (status, message) => Response.json({ error: { message } }, { status });++export class MemosContainer extends Container {+ defaultPort = 5230;+}++// Private path: plain HTTP from the container to http://d1.internal/d1 lands+// here, never on a public route, so no credential is needed.+MemosContainer.outboundByHost = {+ "d1.internal": (request, env) => runBatch(env, request),+};++// Public route: only for deployments that must reach the bridge from outside+// the container. It is closed until BRIDGE_SECRET is configured.+export default {+ async fetch(request, env) {+ if (request.method !== "POST") return error(405, "method not allowed");+ if (!env.BRIDGE_SECRET) return error(503, "bridge secret is not configured");+ if (request.headers.get("Authorization") !== `Bearer ${env.BRIDGE_SECRET}`) {+ return error(401, "unauthorized");+ }+ return runBatch(env, request);+ },+};+```## Atomicity without transactions@@ -96,6 +249,13 @@ files under `store/migration/d1/<version>/` alongside the other drivers'.They must avoid the `PRAGMA foreign_keys` toggles the SQLite migrations use,which D1 ignores.+D1 scripts follow one layout rule: **one statement per paragraph**. A blank+line separates statements, and nothing inside a statement, including a table+or trigger body, a string literal, or a comment, may contain a blank line.+The bridge transport divides scripts at blank lines to send them as a batch,+and the test emulator executes them the same way, so a script that breaks the+rule fails the store suite's fresh install rather than a deployment.+## Filter dialect`filter.DialectD1` renders like SQLite except that case folding uses the@@ -105,9 +265,11 @@ patterns with optional anchors.## Testing-`store/db/d1/d1test` emulates the D1 REST API in-process on a private SQLite-database: the same endpoints, request bodies, response shapes, implicit-transaction per request, atomic batches, foreign-key enforcement, and the-100-parameter limit, with no custom functions registered. `DRIVER=d1` runs-the whole store suite against it. Setting `D1_DSN` points the suite at a real+`store/db/d1/d1test` emulates D1 in-process on a private SQLite database and+speaks both protocols: the REST API endpoints and the bridge endpoint, with+the same implicit transaction per request, atomic batches, foreign-key+enforcement, and 100-parameter limit, and no custom functions registered.+`DRIVER=d1` runs the whole store suite against it over REST, and+`D1_ACCESS=bridge` runs it over the bridge protocol; the all-drivers loop+runs both. Setting `D1_DSN` points the suite at a realdatabase for spot checks.diff --git a/store/db/d1/bridge.go b/store/db/d1/bridge.gonew file mode 100644index 000000000..bc6c9fec3--- /dev/null+++ b/store/db/d1/bridge.go@@ -0,0 +1,146 @@+package d1++import (+ "context"+ "encoding/json"+ "net/http"++ "github.com/pkg/errors"++ "github.com/usememos/memos/store/db/d1/sqlsplit"+)++// bridgeTransport talks to a deployment-provided Worker that holds the D1+// binding and speaks the Memos bridge protocol documented in+// docs/design/cloudflare-d1.md. Every request is one atomic batch; the+// Worker runs it through the binding's batch() and returns one result per+// statement.+type bridgeTransport struct {+ httpClient *http.Client+ config *Config+}++type bridgeRequest struct {+ Statements []requestStatement `json:"statements"`+}++type bridgeMeta struct {+ Changes json.Number `json:"changes"`+ LastRowID json.Number `json:"last_row_id"`+ SizeAfter json.Number `json:"size_after"`+}++type bridgeResult struct {+ Columns []string `json:"columns"`+ Rows [][]json.RawMessage `json:"rows"`+ Meta bridgeMeta `json:"meta"`+}++type bridgeResponse struct {+ Results []bridgeResult `json:"results"`+ Error *struct {+ Code int `json:"code"`+ Message string `json:"message"`+ } `json:"error"`+}++func (t *bridgeTransport) exec(ctx context.Context, stmt statement) (*result, error) {+ results, err := t.run(ctx, []statement{stmt})+ if err != nil {+ return nil, err+ }+ return results[0], nil+}++func (t *bridgeTransport) batch(ctx context.Context, stmts []statement) ([]*result, error) {+ if len(stmts) == 0 {+ return nil, nil+ }+ return t.run(ctx, stmts)+}++// script splits the text into statements locally, since the bridge carries+// exactly one statement per entry, and runs them as one atomic batch.+func (t *bridgeTransport) script(ctx context.Context, sql string) error {+ parts := sqlsplit.Split(sql)+ if len(parts) == 0 {+ return nil+ }+ stmts := make([]statement, 0, len(parts))+ for _, part := range parts {+ stmts = append(stmts, statement{SQL: part})+ }+ _, err := t.run(ctx, stmts)+ return err+}++// databaseSize reads the size the binding reports after a trivial statement.+// A bridge that omits size_after yields -1.+func (t *bridgeTransport) databaseSize(ctx context.Context) (int64, error) {+ res, err := t.exec(ctx, statement{SQL: "SELECT 1"})+ if err != nil {+ return -1, err+ }+ if res.SizeAfter <= 0 {+ return -1, nil+ }+ return res.SizeAfter, nil+}++func (t *bridgeTransport) run(ctx context.Context, stmts []statement) ([]*result, error) {+ body, err := encodeStatements(stmts)+ if err != nil {+ return nil, err+ }+ payload, err := json.Marshal(bridgeRequest{Statements: body})+ if err != nil {+ return nil, errors.Wrap(err, "failed to encode d1 bridge request")+ }+ responseBody, status, err := httpDo(ctx, t.httpClient, http.MethodPost, t.config.BridgeURL, t.config.Token, payload)+ if err != nil {+ return nil, err+ }+ var response bridgeResponse+ if err := decodeJSON(responseBody, &response); err != nil {+ if status != http.StatusOK {+ return nil, &Error{Status: status, Message: http.StatusText(status)}+ }+ return nil, errors.Wrap(err, "failed to decode d1 bridge response")+ }+ if response.Error != nil {+ return nil, &Error{Status: status, Code: response.Error.Code, Message: response.Error.Message}+ }+ if status != http.StatusOK {+ return nil, &Error{Status: status, Message: http.StatusText(status)}+ }+ if len(response.Results) != len(stmts) {+ return nil, errors.Errorf("d1: bridge returned %d results for %d statements", len(response.Results), len(stmts))+ }+ results := make([]*result, 0, len(response.Results))+ for _, item := range response.Results {+ res, err := decodeBridgeResult(item)+ if err != nil {+ return nil, err+ }+ results = append(results, res)+ }+ return results, nil+}++func decodeBridgeResult(item bridgeResult) (*result, error) {+ res := &result{Columns: item.Columns}+ var err error+ if res.Changes, err = decodeNumber(item.Meta.Changes, "change count"); err != nil {+ return nil, err+ }+ if res.LastRowID, err = decodeNumber(item.Meta.LastRowID, "last row id"); err != nil {+ return nil, err+ }+ if res.SizeAfter, err = decodeNumber(item.Meta.SizeAfter, "database size"); err != nil {+ return nil, err+ }+ if res.Rows, err = decodeRows(item.Columns, item.Rows); err != nil {+ return nil, err+ }+ return res, nil+}diff --git a/store/db/d1/client.go b/store/db/d1/client.godeleted file mode 100644index dce511851..000000000--- a/store/db/d1/client.go+++ /dev/null@@ -1,462 +0,0 @@-package d1--import (- "bytes"- "context"- "database/sql/driver"- "encoding/json"- "fmt"- "io"- "net"- "net/http"- "net/url"- "os"- "strings"- "time"-- "github.com/pkg/errors"-)--// DefaultEndpoint is the Cloudflare REST API base URL.-const DefaultEndpoint = "https://api.cloudflare.com/client/v4"--// maxBoundParameters is D1's documented per-statement bind limit.-const maxBoundParameters = 100--// Config is the parsed form of a D1 DSN.-//-// DSN grammar:-//-// d1://<account_id>/<database_id>?token=<api_token>[&endpoint=<base_url>]-//-// The token may be omitted from the DSN and supplied through the-// CLOUDFLARE_API_TOKEN environment variable instead so it stays out of-// process listings and logs.-type Config struct {- AccountID string- DatabaseID string- Token string- Endpoint string-}--// ParseDSN parses a d1:// DSN.-func ParseDSN(dsn string) (*Config, error) {- parsed, err := url.Parse(dsn)- if err != nil {- return nil, errors.Wrap(err, "failed to parse d1 dsn")- }- if parsed.Scheme != "d1" {- return nil, errors.Errorf("d1 dsn must use the d1:// scheme, got %q", parsed.Scheme)- }- config := &Config{- AccountID: parsed.Host,- DatabaseID: strings.Trim(parsed.Path, "/"),- Token: parsed.Query().Get("token"),- Endpoint: strings.TrimRight(parsed.Query().Get("endpoint"), "/"),- }- if config.AccountID == "" {- return nil, errors.New("d1 dsn is missing the account id: d1://<account_id>/<database_id>")- }- if config.DatabaseID == "" || strings.Contains(config.DatabaseID, "/") {- return nil, errors.New("d1 dsn is missing the database id: d1://<account_id>/<database_id>")- }- if config.Token == "" {- config.Token = os.Getenv("CLOUDFLARE_API_TOKEN")- }- if config.Token == "" {- return nil, errors.New("d1 api token is required: add ?token=<api_token> to the dsn or set CLOUDFLARE_API_TOKEN")- }- if config.Endpoint == "" {- config.Endpoint = DefaultEndpoint- }- if err := validateEndpoint(config.Endpoint); err != nil {- return nil, err- }- return config, nil-}--// validateEndpoint rejects endpoints that would send the bearer token in the-// clear. Plain HTTP is allowed only towards loopback, where the test-// emulator listens.-func validateEndpoint(endpoint string) error {- parsed, err := url.Parse(endpoint)- if err != nil {- return errors.Wrap(err, "invalid d1 endpoint")- }- if parsed.Host == "" {- return errors.Errorf("d1 endpoint %q has no host", endpoint)- }- switch parsed.Scheme {- case "https":- return nil- case "http":- if isLoopbackHost(parsed.Hostname()) {- return nil- }- return errors.Errorf("d1 endpoint %q must use https unless it points at loopback", endpoint)- default:- return errors.Errorf("d1 endpoint %q must use https", endpoint)- }-}--func isLoopbackHost(host string) bool {- if host == "localhost" {- return true- }- ip := net.ParseIP(host)- return ip != nil && ip.IsLoopback()-}--// Error is a failure reported by the D1 REST API.-type Error struct {- // Status is the HTTP status code of the response.- Status int- // Code is the Cloudflare API error code, when the response carried one.- Code int- // Message is the human readable error text.- Message string-}--func (e *Error) Error() string {- if e.Code != 0 {- return fmt.Sprintf("d1: %s (code %d, http %d)", e.Message, e.Code, e.Status)- }- return fmt.Sprintf("d1: %s (http %d)", e.Message, e.Status)-}--// statement is one parameterized SQL statement.-type statement struct {- SQL string- Args []any-}--// result is the outcome of one statement.-type result struct {- Columns []string- Rows [][]any- Changes int64- LastRowID int64-}--// client talks to one D1 database through the Cloudflare REST API.-type client struct {- httpClient *http.Client- config *Config-}--func newClient(config *Config) *client {- return &client{- httpClient: &http.Client{Timeout: 60 * time.Second},- config: config,- }-}--type requestStatement struct {- SQL string `json:"sql"`- Params []any `json:"params,omitempty"`-}--type responseInfo struct {- Code int `json:"code"`- Message string `json:"message"`-}--type responseMeta struct {- Changes json.Number `json:"changes"`- LastRowID json.Number `json:"last_row_id"`-}--type rawResults struct {- Columns []string `json:"columns"`- Rows [][]json.RawMessage `json:"rows"`-}--type queryResult struct {- Success bool `json:"success"`- Meta responseMeta `json:"meta"`- Results json.RawMessage `json:"results"`-}--type envelope struct {- Success bool `json:"success"`- Errors []responseInfo `json:"errors"`- Result []queryResult `json:"result"`-}--// exec runs one statement and returns its rows and change counters.-func (c *client) exec(ctx context.Context, stmt statement) (*result, error) {- params, err := encodeArgs(stmt.Args)- if err != nil {- return nil, err- }- results, err := c.raw(ctx, map[string]any{"sql": stmt.SQL, "params": params})- if err != nil {- return nil, err- }- if len(results) != 1 {- return nil, errors.Errorf("d1: expected 1 result, got %d", len(results))- }- return results[0], nil-}--// script runs a parameter-free, possibly multi-statement SQL text as one-// atomic batch. It backs schema migrations.-func (c *client) script(ctx context.Context, sql string) error {- _, err := c.raw(ctx, map[string]any{"sql": sql})- return err-}--// batch runs the statements in order as one transaction: either every-// statement commits or none does.-func (c *client) batch(ctx context.Context, stmts []statement) ([]*result, error) {- if len(stmts) == 0 {- return nil, nil- }- if len(stmts) == 1 {- res, err := c.exec(ctx, stmts[0])- if err != nil {- return nil, err- }- return []*result{res}, nil- }- body := make([]requestStatement, 0, len(stmts))- for _, stmt := range stmts {- params, err := encodeArgs(stmt.Args)- if err != nil {- return nil, err- }- body = append(body, requestStatement{SQL: stmt.SQL, Params: params})- }- results, err := c.raw(ctx, map[string]any{"batch": body})- if err != nil {- return nil, err- }- if len(results) != len(stmts) {- return nil, errors.Errorf("d1: expected %d results, got %d", len(stmts), len(results))- }- return results, nil-}--func (c *client) raw(ctx context.Context, body map[string]any) ([]*result, error) {- payload, err := json.Marshal(body)- if err != nil {- return nil, errors.Wrap(err, "failed to encode d1 request")- }- responseBody, status, err := c.do(ctx, http.MethodPost, c.databaseURL()+"/raw", payload)- if err != nil {- return nil, err- }- var env envelope- if err := decodeJSON(responseBody, &env); err != nil {- return nil, errors.Wrap(err, "failed to decode d1 response")- }- if status != http.StatusOK || !env.Success {- return nil, envelopeError(status, env.Errors)- }- results := make([]*result, 0, len(env.Result))- for _, item := range env.Result {- if !item.Success {- return nil, &Error{Status: status, Message: "statement reported failure"}- }- res, err := decodeResult(item)- if err != nil {- return nil, err- }- results = append(results, res)- }- return results, nil-}--// databaseSize returns the database file size reported by the D1 metadata endpoint.-func (c *client) databaseSize(ctx context.Context) (int64, error) {- responseBody, status, err := c.do(ctx, http.MethodGet, c.databaseURL()+"?fields=file_size", nil)- if err != nil {- return -1, err- }- var env struct {- Success bool `json:"success"`- Errors []responseInfo `json:"errors"`- Result struct {- FileSize json.Number `json:"file_size"`- } `json:"result"`- }- if err := decodeJSON(responseBody, &env); err != nil {- return -1, errors.Wrap(err, "failed to decode d1 database metadata")- }- if status != http.StatusOK || !env.Success {- return -1, envelopeError(status, env.Errors)- }- size, err := env.Result.FileSize.Int64()- if err != nil {- return -1, errors.Wrap(err, "failed to parse d1 database size")- }- return size, nil-}--func (c *client) databaseURL() string {- return fmt.Sprintf("%s/accounts/%s/d1/database/%s", c.config.Endpoint, url.PathEscape(c.config.AccountID), url.PathEscape(c.config.DatabaseID))-}--func (c *client) do(ctx context.Context, method, target string, payload []byte) ([]byte, int, error) {- var reader io.Reader- if payload != nil {- reader = bytes.NewReader(payload)- }- request, err := http.NewRequestWithContext(ctx, method, target, reader)- if err != nil {- return nil, 0, errors.Wrap(err, "failed to build d1 request")- }- request.Header.Set("Authorization", "Bearer "+c.config.Token)- request.Header.Set("Accept", "application/json")- if payload != nil {- request.Header.Set("Content-Type", "application/json")- }- response, err := c.httpClient.Do(request)- if err != nil {- return nil, 0, errors.Wrap(err, "d1 request failed")- }- defer response.Body.Close()- responseBody, err := io.ReadAll(response.Body)- if err != nil {- return nil, response.StatusCode, errors.Wrap(err, "failed to read d1 response")- }- return responseBody, response.StatusCode, nil-}--func envelopeError(status int, infos []responseInfo) error {- if len(infos) == 0 {- return &Error{Status: status, Message: http.StatusText(status)}- }- messages := make([]string, 0, len(infos))- for _, info := range infos {- messages = append(messages, info.Message)- }- return &Error{Status: status, Code: infos[0].Code, Message: strings.Join(messages, "; ")}-}--func decodeJSON(data []byte, target any) error {- decoder := json.NewDecoder(bytes.NewReader(data))- decoder.UseNumber()- return decoder.Decode(target)-}--func decodeResult(item queryResult) (*result, error) {- res := &result{}- if item.Meta.Changes != "" {- changes, err := item.Meta.Changes.Int64()- if err != nil {- return nil, errors.Wrap(err, "failed to parse d1 change count")- }- res.Changes = changes- }- if item.Meta.LastRowID != "" {- lastRowID, err := item.Meta.LastRowID.Int64()- if err != nil {- return nil, errors.Wrap(err, "failed to parse d1 last row id")- }- res.LastRowID = lastRowID- }- if len(item.Results) == 0 || string(item.Results) == "null" {- return res, nil- }- var raw rawResults- if err := decodeJSON(item.Results, &raw); err != nil {- return nil, errors.Wrap(err, "failed to decode d1 rows")- }- res.Columns = raw.Columns- res.Rows = make([][]any, 0, len(raw.Rows))- for _, rawRow := range raw.Rows {- row := make([]any, 0, len(rawRow))- for _, rawValue := range rawRow {- value, err := decodeValue(rawValue)- if err != nil {- return nil, err- }- row = append(row, value)- }- res.Rows = append(res.Rows, row)- }- return res, nil-}--// decodeValue maps one JSON cell onto the driver.Value domain.-func decodeValue(raw json.RawMessage) (any, error) {- if len(raw) == 0 {- return nil, nil- }- switch raw[0] {- case 'n':- return nil, nil- case 't':- return int64(1), nil- case 'f':- return int64(0), nil- case '"':- var text string- if err := json.Unmarshal(raw, &text); err != nil {- return nil, errors.Wrap(err, "failed to decode d1 text cell")- }- return text, nil- case '[':- // Blobs arrive as arrays of byte values.- var bytesValue []byte- var numbers []json.Number- if err := decodeJSON(raw, &numbers); err != nil {- return nil, errors.Wrap(err, "failed to decode d1 blob cell")- }- bytesValue = make([]byte, 0, len(numbers))- for _, number := range numbers {- b, err := number.Int64()- if err != nil || b < 0 || b > 255 {- return nil, errors.Errorf("d1: invalid blob byte %q", number)- }- bytesValue = append(bytesValue, byte(b))- }- return bytesValue, nil- case '{':- return string(raw), nil- default:- number := json.Number(bytes.TrimSpace(raw))- if integer, err := number.Int64(); err == nil {- return integer, nil- }- float, err := number.Float64()- if err != nil {- return nil, errors.Wrapf(err, "d1: invalid numeric cell %q", string(raw))- }- return float, nil- }-}--// encodeArgs converts Go arguments to the JSON parameter list D1 accepts.-// Binary values are rejected: blob columns are written with unhex(?) and a-// hex-encoded string because JSON cannot carry raw bytes.-func encodeArgs(args []any) ([]any, error) {- if len(args) > maxBoundParameters {- return nil, errors.Errorf("d1: statement binds %d parameters, limit is %d", len(args), maxBoundParameters)- }- params := make([]any, 0, len(args))- for _, arg := range args {- value, err := driver.DefaultParameterConverter.ConvertValue(arg)- if err != nil {- return nil, errors.Wrap(err, "d1: unsupported parameter")- }- switch v := value.(type) {- case nil, int64, float64, string:- params = append(params, v)- case bool:- if v {- params = append(params, int64(1))- } else {- params = append(params, int64(0))- }- case []byte:- return nil, errors.New("d1: binary parameters are not supported; bind hex text through unhex(?)")- case time.Time:- params = append(params, v.UTC().Format(time.RFC3339Nano))- default:- return nil, errors.Errorf("d1: unsupported parameter type %T", value)- }- }- return params, nil-}diff --git a/store/db/d1/config.go b/store/db/d1/config.gonew file mode 100644index 000000000..fc57c2cf3--- /dev/null+++ b/store/db/d1/config.go@@ -0,0 +1,165 @@+package d1++import (+ "net"+ "net/url"+ "os"+ "strings"++ "github.com/pkg/errors"+)++// DefaultEndpoint is the Cloudflare REST API base URL.+const DefaultEndpoint = "https://api.cloudflare.com/client/v4"++// AccessMode selects how the driver reaches the database.+type AccessMode string++const (+ // AccessModeREST talks to the Cloudflare REST API with an API token.+ AccessModeREST AccessMode = "rest"+ // AccessModeBridge talks to a deployment-provided Worker that holds the+ // D1 binding, so Memos needs no Cloudflare API token.+ AccessModeBridge AccessMode = "bridge"+)++// Config is the parsed form of a D1 DSN.+//+// DSN grammar:+//+// d1://<account_id>/<database_id>?token=<api_token>[&endpoint=<base_url>]+// d1-bridge://<host>[:<port>][/<path>][?token=<secret>][&private=true]+//+// The first form selects REST access. Its token may be omitted from the DSN+// and supplied through the CLOUDFLARE_API_TOKEN environment variable instead+// so it stays out of process listings and logs.+//+// The second form selects bridge access: the host and path name the Worker+// endpoint that implements the bridge protocol (see docs/design/cloudflare-d1.md).+// A public bridge is reached over HTTPS and requires a shared secret, sent as+// a bearer credential; the secret is best supplied through+// MEMOS_D1_BRIDGE_TOKEN rather than the DSN. private=true names a bridge on+// a platform-private path, such as the virtual host a Cloudflare Containers+// outbound handler serves or the loopback test emulator: it is reached over+// plain HTTP and the secret is optional.+type Config struct {+ Mode AccessMode++ // Private marks a bridge on a platform-private path (bridge access only).+ Private bool++ // REST access.+ AccountID string+ DatabaseID string+ Endpoint string++ // Bridge access.+ BridgeURL string++ // Token is the API token (REST) or the bridge bearer secret (bridge).+ Token string+}++// ParseDSN parses a d1:// or d1-bridge:// DSN.+func ParseDSN(dsn string) (*Config, error) {+ parsed, err := url.Parse(dsn)+ if err != nil {+ return nil, errors.Wrap(err, "failed to parse d1 dsn")+ }+ switch parsed.Scheme {+ case "d1":+ return parseRESTDSN(parsed)+ case "d1-bridge":+ return parseBridgeDSN(parsed)+ default:+ return nil, errors.Errorf("d1 dsn must use the d1:// or d1-bridge:// scheme, got %q", parsed.Scheme)+ }+}++func parseRESTDSN(parsed *url.URL) (*Config, error) {+ config := &Config{+ Mode: AccessModeREST,+ AccountID: parsed.Host,+ DatabaseID: strings.Trim(parsed.Path, "/"),+ Token: parsed.Query().Get("token"),+ Endpoint: strings.TrimRight(parsed.Query().Get("endpoint"), "/"),+ }+ if config.AccountID == "" {+ return nil, errors.New("d1 dsn is missing the account id: d1://<account_id>/<database_id>")+ }+ if config.DatabaseID == "" || strings.Contains(config.DatabaseID, "/") {+ return nil, errors.New("d1 dsn is missing the database id: d1://<account_id>/<database_id>")+ }+ if config.Token == "" {+ config.Token = os.Getenv("CLOUDFLARE_API_TOKEN")+ }+ if config.Token == "" {+ return nil, errors.New("d1 api token is required: add ?token=<api_token> to the dsn or set CLOUDFLARE_API_TOKEN")+ }+ if config.Endpoint == "" {+ config.Endpoint = DefaultEndpoint+ }+ if err := validateEndpoint(config.Endpoint); err != nil {+ return nil, err+ }+ return config, nil+}++func parseBridgeDSN(parsed *url.URL) (*Config, error) {+ if parsed.Host == "" {+ return nil, errors.New("d1 bridge dsn is missing the host: d1-bridge://<host>[/<path>]")+ }+ query := parsed.Query()+ config := &Config{+ Mode: AccessModeBridge,+ Private: query.Get("private") == "true",+ Token: query.Get("token"),+ }+ if config.Token == "" {+ config.Token = os.Getenv("MEMOS_D1_BRIDGE_TOKEN")+ }+ path := strings.TrimRight(parsed.Path, "/")+ if config.Private {+ // The platform keeps the path private, so plain HTTP carries no risk+ // and a shared secret is not needed.+ config.BridgeURL = "http://" + parsed.Host + path+ return config, nil+ }+ config.BridgeURL = "https://" + parsed.Host + path+ if config.Token == "" {+ return nil, errors.New("a public d1 bridge requires a shared secret: set MEMOS_D1_BRIDGE_TOKEN (or add ?token=) or mark a platform-private bridge with ?private=true")+ }+ return config, nil+}++// validateEndpoint rejects endpoints that would send a bearer credential in+// the clear. Plain HTTP is allowed only towards loopback, where the test+// emulator listens.+func validateEndpoint(endpoint string) error {+ parsed, err := url.Parse(endpoint)+ if err != nil {+ return errors.Wrap(err, "invalid d1 endpoint")+ }+ if parsed.Host == "" {+ return errors.Errorf("d1 endpoint %q has no host", endpoint)+ }+ switch parsed.Scheme {+ case "https":+ return nil+ case "http":+ if isLoopbackHost(parsed.Hostname()) {+ return nil+ }+ return errors.Errorf("d1 endpoint %q must use https unless it points at loopback", endpoint)+ default:+ return errors.Errorf("d1 endpoint %q must use https", endpoint)+ }+}++func isLoopbackHost(host string) bool {+ if host == "localhost" {+ return true+ }+ ip := net.ParseIP(host)+ return ip != nil && ip.IsLoopback()+}diff --git a/store/db/d1/d1.go b/store/db/d1/d1.goindex 8487f43c1..77c587282 100644--- a/store/db/d1/d1.go+++ b/store/db/d1/d1.go@@ -1,7 +1,8 @@// Package d1 implements the store driver for Cloudflare D1.//-// D1 is reached through the Cloudflare REST API rather than a socket, and it-// exposes no interactive transactions. Each driver method therefore performs+// D1 is reached over HTTP rather than a socket, either through the Cloudflare+// REST API or through a deployment-provided bridge Worker, and it exposes no+// interactive transactions. Each driver method therefore performs// its validation reads first and then commits its writes as one atomic batch.// Preconditions that must hold at commit time are re-checked inside the batch// with guard statements (see guard.go) so a stale read never yields a partial@@ -20,9 +21,9 @@ import (// DB is the D1 store driver.type DB struct {- client *client- db *sql.DB- profile *profile.Profile+ transport transport+ db *sql.DB+ profile *profile.Profile}// NewDB opens a D1 database described by the profile DSN.@@ -31,11 +32,14 @@ func NewDB(profile *profile.Profile) (store.Driver, error) {if err != nil {return nil, err}- c := newClient(config)- return &DB{client: c, db: openSQLDB(c), profile: profile}, nil+ t, err := newTransport(config)+ if err != nil {+ return nil, err+ }+ return &DB{transport: t, db: openSQLDB(t), profile: profile}, nil}-// GetDB exposes a database/sql handle over the REST client for the migrator+// GetDB exposes a database/sql handle over the transport for the migrator// and tests. See sqldriver.go for its transaction semantics.func (d *DB) GetDB() *sql.DB {return d.db@@ -56,9 +60,10 @@ func (d *DB) IsInitialized(ctx context.Context) (bool, error) {return exists, nil}-// GetDatabaseSize returns the database size reported by the D1 metadata endpoint.+// GetDatabaseSize returns the database size the transport reports, or -1+// when it cannot report one.func (d *DB) GetDatabaseSize(ctx context.Context) (int64, error) {- size, err := d.client.databaseSize(ctx)+ size, err := d.transport.databaseSize(ctx)if err != nil {return -1, errors.Wrap(err, "failed to read d1 database size")}diff --git a/store/db/d1/d1_test.go b/store/db/d1/d1_test.goindex ec709c541..0c687077a 100644--- a/store/db/d1/d1_test.go+++ b/store/db/d1/d1_test.go@@ -3,6 +3,9 @@ package d1import ("context""errors"+ "net/http"+ "net/http/httptest"+ "strings""testing""github.com/stretchr/testify/require"@@ -13,10 +16,17 @@ import (var errConflict = errors.New("conflict")-func newTestDB(t *testing.T) *DB {+// accessModes lists the DSN of each access mode against the emulator; every+// transport test runs under both.+var accessModes = map[string]func(*d1test.Server) string{+ "rest": (*d1test.Server).DSN,+ "bridge": (*d1test.Server).BridgeDSN,+}++func newTestDB(t *testing.T, dsn func(*d1test.Server) string) *DB {t.Helper()server := d1test.New(t)- driver, err := NewDB(&profile.Profile{Driver: "d1", DSN: server.DSN()})+ driver, err := NewDB(&profile.Profile{Driver: "d1", DSN: dsn(server)})require.NoError(t, err)db, ok := driver.(*DB)require.True(t, ok)@@ -42,7 +52,7 @@ func TestParseDSN(t *testing.T) {}_, err = ParseDSN("sqlite://x")- require.ErrorContains(t, err, "d1:// scheme")+ require.ErrorContains(t, err, "d1:// or d1-bridge:// scheme")_, err = ParseDSN("d1:///dbid?token=t")require.ErrorContains(t, err, "account id")t.Setenv("CLOUDFLARE_API_TOKEN", "")@@ -53,11 +63,54 @@ func TestParseDSN(t *testing.T) {require.NoError(t, err)require.Equal(t, "from-env", config.Token)require.Equal(t, DefaultEndpoint, config.Endpoint)+ require.Equal(t, AccessModeREST, config.Mode)+}++func TestParseBridgeDSN(t *testing.T) {+ t.Setenv("MEMOS_D1_BRIDGE_TOKEN", "")++ // A public bridge is HTTPS and needs a shared secret.+ config, err := ParseDSN("d1-bridge://worker.example.com/d1?token=secret")+ require.NoError(t, err)+ require.Equal(t, AccessModeBridge, config.Mode)+ require.False(t, config.Private)+ require.Equal(t, "https://worker.example.com/d1", config.BridgeURL)+ require.Equal(t, "secret", config.Token)+ _, err = ParseDSN("d1-bridge://worker.example.com/d1")+ require.ErrorContains(t, err, "requires a shared secret")++ // A platform-private bridge is plain HTTP and needs no secret: the+ // Cloudflare Containers outbound handler and the loopback emulator.+ config, err = ParseDSN("d1-bridge://d1.internal/d1?private=true")+ require.NoError(t, err)+ require.True(t, config.Private)+ require.Equal(t, "http://d1.internal/d1", config.BridgeURL)+ require.Empty(t, config.Token)+ config, err = ParseDSN("d1-bridge://127.0.0.1:8787/bridge/?private=true&token=t")+ require.NoError(t, err)+ require.Equal(t, "http://127.0.0.1:8787/bridge", config.BridgeURL)+ require.Equal(t, "t", config.Token)++ t.Setenv("MEMOS_D1_BRIDGE_TOKEN", "from-env")+ config, err = ParseDSN("d1-bridge://worker.example.com")+ require.NoError(t, err)+ require.Equal(t, "https://worker.example.com", config.BridgeURL)+ require.Equal(t, "from-env", config.Token)++ _, err = ParseDSN("d1-bridge:///d1")+ require.ErrorContains(t, err, "missing the host")}func TestTransportRoundTrip(t *testing.T) {+ for mode, dsn := range accessModes {+ t.Run(mode, func(t *testing.T) {+ runTransportRoundTrip(t, newTestDB(t, dsn))+ })+ }+}++func runTransportRoundTrip(t *testing.T, db *DB) {ctx := context.Background()- db := newTestDB(t)initialized, err := db.IsInitialized(ctx)require.NoError(t, err)@@ -66,9 +119,11 @@ func TestTransportRoundTrip(t *testing.T) {// Multi-statement script through the sql adapter transaction, as the migrator does.tx, err := db.GetDB().Begin()require.NoError(t, err)+ // One statement per paragraph, as the D1 scripts are laid out._, err = tx.ExecContext(ctx, `-- schemaCREATE TABLE d1_guard (ok INTEGER NOT NULL CONSTRAINT d1_guard_ok CHECK (ok = 1));+CREATE TABLE memo (id INTEGER PRIMARY KEY AUTOINCREMENT, uid TEXT NOT NULL UNIQUE, content TEXT NOT NULL DEFAULT 'a;b', blob BLOB);`)require.NoError(t, err)@@ -157,8 +212,15 @@ func TestTransportRoundTrip(t *testing.T) {}func TestNullAndTypedScans(t *testing.T) {+ for mode, dsn := range accessModes {+ t.Run(mode, func(t *testing.T) {+ runNullAndTypedScans(t, newTestDB(t, dsn))+ })+ }+}++func runNullAndTypedScans(t *testing.T, db *DB) {ctx := context.Background()- db := newTestDB(t)_, err := db.execOne(ctx, "CREATE TABLE t (id INTEGER PRIMARY KEY, n INTEGER, f REAL, s TEXT, ts BIGINT NOT NULL DEFAULT (strftime('%s', 'now')))")require.NoError(t, err)var nilInt *int32@@ -174,3 +236,24 @@ func TestNullAndTypedScans(t *testing.T) {require.Equal(t, "x", s)require.Greater(t, ts, int64(0))}++func TestRedirectsAreNotFollowed(t *testing.T) {+ redirected := false+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ if r.URL.Path == "/moved" {+ redirected = true+ return+ }+ http.Redirect(w, r, "/moved", http.StatusTemporaryRedirect)+ }))+ t.Cleanup(server.Close)++ driver, err := NewDB(&profile.Profile{Driver: "d1", DSN: "d1-bridge://" + strings.TrimPrefix(server.URL, "http://") + "/bridge?private=true&token=t"})+ require.NoError(t, err)+ _, err = driver.GetDB().ExecContext(context.Background(), "SELECT 1")+ require.Error(t, err)+ var d1Err *Error+ require.ErrorAs(t, err, &d1Err)+ require.Equal(t, http.StatusTemporaryRedirect, d1Err.Status)+ require.False(t, redirected, "the redirect target must not receive the credential")+}diff --git a/store/db/d1/d1test/server.go b/store/db/d1/d1test/server.goindex 7a15b91df..6e6f4d457 100644--- a/store/db/d1/d1test/server.go+++ b/store/db/d1/d1test/server.go@@ -1,10 +1,11 @@-// Package d1test runs an in-process emulation of the Cloudflare D1 REST API-// for tests. It speaks the same JSON protocol the driver uses (the /raw and-// /query endpoints, batch bodies, the database metadata endpoint) on top of a-// private SQLite database, and enforces the D1 constraints that matter to the-// driver: foreign keys are on, statements are wrapped in an implicit-// transaction, a batch is atomic, and a statement may bind at most 100-// parameters. It registers no custom SQL functions, exactly like D1.+// Package d1test runs an in-process emulation of Cloudflare D1 for tests. It+// speaks both protocols the driver uses, the Cloudflare REST API (the /raw+// and /query endpoints, batch bodies, the database metadata endpoint) and the+// Memos bridge protocol, on top of a private SQLite database. It enforces the+// D1 constraints that matter to the driver: foreign keys are on, statements+// are wrapped in an implicit transaction, a batch is atomic, and a statement+// may bind at most 100 parameters. It registers no custom SQL functions,+// exactly like D1.package d1testimport (@@ -21,12 +22,16 @@ import (// SQLite engine behind the emulated service._ "modernc.org/sqlite"++ "github.com/usememos/memos/store/db/d1/sqlsplit")const (- accountID = "test-account"- databaseID = "test-database"- token = "test-token"+ accountID = "test-account"+ databaseID = "test-database"+ token = "test-token"+ bridgeToken = "bridge-secret"+ bridgePath = "/bridge/d1"maxBoundParameters = 100)@@ -57,11 +62,16 @@ func New(t testing.TB) *Server {return s}-// DSN returns the driver DSN pointing at this server.+// DSN returns the REST-mode driver DSN pointing at this server.func (s *Server) DSN() string {return fmt.Sprintf("d1://%s/%s?token=%s&endpoint=%s", accountID, databaseID, token, s.http.URL)}+// BridgeDSN returns the bridge-mode driver DSN pointing at this server.+func (s *Server) BridgeDSN() string {+ return fmt.Sprintf("d1-bridge://%s%s?token=%s&private=true", strings.TrimPrefix(s.http.URL, "http://"), bridgePath, bridgeToken)+}+// Close stops the server and discards the database.func (s *Server) Close() {s.http.Close()@@ -74,18 +84,36 @@ type requestStatement struct {Params []any `json:"params"`}-type requestBody struct {+type restRequest struct {SQL *string `json:"sql"`Params []any `json:"params"`Batch []requestStatement `json:"batch"`}+type bridgeRequest struct {+ Statements []requestStatement `json:"statements"`+}+type apiError struct {Code int `json:"code"`Message string `json:"message"`}+// statementResult is the outcome of one executed statement before it is+// rendered in either protocol's shape.+type statementResult struct {+ columns []string+ arrayRows [][]any+ objectRows []map[string]any+ changes int64+ lastRowID int64+}+func (s *Server) handle(w http.ResponseWriter, r *http.Request) {+ if r.URL.Path == bridgePath {+ s.handleBridge(w, r)+ return+ }if r.Header.Get("Authorization") != "Bearer "+token {writeErrors(w, http.StatusForbidden, 10000, "Authentication error")return@@ -108,14 +136,8 @@ func (s *Server) handle(w http.ResponseWriter, r *http.Request) {}func (s *Server) handleMetadata(w http.ResponseWriter) {- s.mu.Lock()- defer s.mu.Unlock()- var pageCount, pageSize int64- if err := s.conn.QueryRowContext(context.Background(), "PRAGMA page_count").Scan(&pageCount); err != nil {- writeErrors(w, http.StatusInternalServerError, 7500, err.Error())- return- }- if err := s.conn.QueryRowContext(context.Background(), "PRAGMA page_size").Scan(&pageSize); err != nil {+ size, err := s.databaseSize(context.Background())+ if err != nil {writeErrors(w, http.StatusInternalServerError, 7500, err.Error())return}@@ -125,16 +147,28 @@ func (s *Server) handleMetadata(w http.ResponseWriter) {"result": map[string]any{"uuid": databaseID,"name": "memos-test",- "file_size": pageCount * pageSize,+ "file_size": size,},})}+func (s *Server) databaseSize(ctx context.Context) (int64, error) {+ s.mu.Lock()+ defer s.mu.Unlock()+ var pageCount, pageSize int64+ if err := s.conn.QueryRowContext(ctx, "PRAGMA page_count").Scan(&pageCount); err != nil {+ return 0, err+ }+ if err := s.conn.QueryRowContext(ctx, "PRAGMA page_size").Scan(&pageSize); err != nil {+ return 0, err+ }+ return pageCount * pageSize, nil+}++// handleQuery serves the REST /raw and /query endpoints.func (s *Server) handleQuery(w http.ResponseWriter, r *http.Request, rawRows bool) {- var body requestBody- decoder := json.NewDecoder(r.Body)- decoder.UseNumber()- if err := decoder.Decode(&body); err != nil {+ var body restRequest+ if err := decodeBody(r, &body); err != nil {writeErrors(w, http.StatusBadRequest, 7400, "invalid request body: "+err.Error())return}@@ -149,44 +183,113 @@ func (s *Server) handleQuery(w http.ResponseWriter, r *http.Request, rawRows booreturn}- // Expand every entry into single statements; a parameterized entry must- // hold exactly one statement because D1 documents no binding rule for- // several.+ // A parameterized entry is one statement; a parameter-free entry is a+ // script, executed statement by statement the way D1 does.var stmts []requestStatementfor _, entry := range entries {- parts := splitStatements(entry.SQL)- if len(parts) == 0 {- writeErrors(w, http.StatusBadRequest, 7400, "empty sql")+ if len(entry.Params) > maxBoundParameters {+ writeErrors(w, http.StatusBadRequest, 7400, fmt.Sprintf("too many bound parameters: %d", len(entry.Params)))return}- if len(parts) > 1 && len(entry.Params) > 0 {- writeErrors(w, http.StatusBadRequest, 7400, "params cannot be bound to multiple statements")- return+ if len(entry.Params) > 0 {+ stmts = append(stmts, entry)+ continue}- if len(entry.Params) > maxBoundParameters {- writeErrors(w, http.StatusBadRequest, 7400, fmt.Sprintf("too many bound parameters: %d", len(entry.Params)))+ parts := sqlsplit.Split(entry.SQL)+ if len(parts) == 0 {+ writeErrors(w, http.StatusBadRequest, 7400, "empty sql")return}for _, part := range parts {- stmts = append(stmts, requestStatement{SQL: part, Params: entry.Params})+ stmts = append(stmts, requestStatement{SQL: part})}}- results, err := s.execute(r.Context(), stmts, rawRows)+ results, err := s.execute(r.Context(), stmts)if err != nil {writeErrors(w, http.StatusBadRequest, 7500, "D1_ERROR: "+err.Error())return}+ rendered := make([]map[string]any, 0, len(results))+ for _, res := range results {+ var rows any+ if rawRows {+ rows = map[string]any{"columns": res.columns, "rows": res.arrayRows}+ } else {+ rows = res.objectRows+ }+ rendered = append(rendered, map[string]any{+ "success": true,+ "meta": map[string]any{+ "changed_db": res.changes > 0,+ "changes": res.changes,+ "last_row_id": res.lastRowID,+ "duration": 0,+ },+ "results": rows,+ })+ }writeJSON(w, http.StatusOK, map[string]any{"success": true,"errors": []any{},- "result": results,+ "result": rendered,})}+// handleBridge serves the Memos bridge protocol: one atomic batch per+// request, one statement per entry, rows as arrays with a column list.+func (s *Server) handleBridge(w http.ResponseWriter, r *http.Request) {+ if r.Method != http.MethodPost {+ writeBridgeError(w, http.StatusMethodNotAllowed, "method not allowed")+ return+ }+ if r.Header.Get("Authorization") != "Bearer "+bridgeToken {+ writeBridgeError(w, http.StatusUnauthorized, "unauthorized")+ return+ }+ var body bridgeRequest+ if err := decodeBody(r, &body); err != nil {+ writeBridgeError(w, http.StatusBadRequest, "invalid request body: "+err.Error())+ return+ }+ if len(body.Statements) == 0 {+ writeBridgeError(w, http.StatusBadRequest, "request carries no statements")+ return+ }+ for _, entry := range body.Statements {+ if len(entry.Params) > maxBoundParameters {+ writeBridgeError(w, http.StatusBadRequest, fmt.Sprintf("too many bound parameters: %d", len(entry.Params)))+ return+ }+ }+ results, err := s.execute(r.Context(), body.Statements)+ if err != nil {+ writeBridgeError(w, http.StatusBadRequest, "D1_ERROR: "+err.Error())+ return+ }+ size, err := s.databaseSize(r.Context())+ if err != nil {+ writeBridgeError(w, http.StatusInternalServerError, err.Error())+ return+ }+ rendered := make([]map[string]any, 0, len(results))+ for _, res := range results {+ rendered = append(rendered, map[string]any{+ "columns": res.columns,+ "rows": res.arrayRows,+ "meta": map[string]any{+ "changes": res.changes,+ "last_row_id": res.lastRowID,+ "size_after": size,+ },+ })+ }+ writeJSON(w, http.StatusOK, map[string]any{"results": rendered})+}+// execute runs stmts inside one transaction, mirroring D1's implicit// transaction per request and its atomic batches.-func (s *Server) execute(ctx context.Context, stmts []requestStatement, rawRows bool) ([]map[string]any, error) {+func (s *Server) execute(ctx context.Context, stmts []requestStatement) ([]*statementResult, error) {s.mu.Lock()defer s.mu.Unlock()@@ -194,9 +297,9 @@ func (s *Server) execute(ctx context.Context, stmts []requestStatement, rawRowsif err != nil {return nil, err}- results := make([]map[string]any, 0, len(stmts))+ results := make([]*statementResult, 0, len(stmts))for _, stmt := range stmts {- res, err := runStatement(ctx, tx, stmt, rawRows)+ res, err := runStatement(ctx, tx, stmt)if err != nil {_ = tx.Rollback()return nil, err@@ -209,7 +312,7 @@ func (s *Server) execute(ctx context.Context, stmts []requestStatement, rawRowsreturn results, nil}-func runStatement(ctx context.Context, tx *sql.Tx, stmt requestStatement, rawRows bool) (map[string]any, error) {+func runStatement(ctx context.Context, tx *sql.Tx, stmt requestStatement) (*statementResult, error) {args := make([]any, 0, len(stmt.Params))for _, param := range stmt.Params {args = append(args, decodeParam(param))@@ -220,47 +323,31 @@ func runStatement(ctx context.Context, tx *sql.Tx, stmt requestStatement, rawRowif err := tx.QueryRowContext(ctx, "SELECT total_changes()").Scan(&before); err != nil {return nil, err}- columns, arrayRows, objectRows, err := collectRows(ctx, tx, stmt.SQL, args, rawRows)+ res, err := collectRows(ctx, tx, stmt.SQL, args)if err != nil {return nil, err}- var after, lastRowID int64- if err := tx.QueryRowContext(ctx, "SELECT total_changes(), last_insert_rowid()").Scan(&after, &lastRowID); err != nil {+ var after int64+ if err := tx.QueryRowContext(ctx, "SELECT total_changes(), last_insert_rowid()").Scan(&after, &res.lastRowID); err != nil {return nil, err}- changes := after - before- var results any- if rawRows {- results = map[string]any{"columns": columns, "rows": arrayRows}- } else {- results = objectRows- }- return map[string]any{- "success": true,- "meta": map[string]any{- "changed_db": changes > 0,- "changes": changes,- "last_row_id": lastRowID,- "duration": 0,- },- "results": results,- }, nil+ res.changes = after - before+ return res, nil}-// collectRows executes one statement and drains its result set in the-// requested shape. Statements without a result set yield empty slices.-func collectRows(ctx context.Context, tx *sql.Tx, query string, args []any, rawRows bool) ([]string, [][]any, []map[string]any, error) {+// collectRows executes one statement and drains its result set in both+// shapes. Statements without a result set yield empty slices.+func collectRows(ctx context.Context, tx *sql.Tx, query string, args []any) (*statementResult, error) {rows, err := tx.QueryContext(ctx, query, args...)if err != nil {- return nil, nil, nil, err+ return nil, err}defer rows.Close()columns, err := rows.Columns()if err != nil {- return nil, nil, nil, err+ return nil, err}- arrayRows := [][]any{}- objectRows := []map[string]any{}+ res := &statementResult{columns: columns, arrayRows: [][]any{}, objectRows: []map[string]any{}}for rows.Next() {cells := make([]any, len(columns))targets := make([]any, len(columns))@@ -268,29 +355,30 @@ func collectRows(ctx context.Context, tx *sql.Tx, query string, args []any, rawRtargets[i] = &cells[i]}if err := rows.Scan(targets...); err != nil {- return nil, nil, nil, err+ return nil, err}encoded := make([]any, len(columns))+ object := make(map[string]any, len(columns))for i, cell := range cells {encoded[i] = encodeCell(cell)+ object[columns[i]] = encoded[i]}- if rawRows {- arrayRows = append(arrayRows, encoded)- continue- }- object := make(map[string]any, len(columns))- for i, column := range columns {- object[column] = encoded[i]- }- objectRows = append(objectRows, object)+ res.arrayRows = append(res.arrayRows, encoded)+ res.objectRows = append(res.objectRows, object)}if err := rows.Err(); err != nil {- return nil, nil, nil, err+ return nil, err}if err := rows.Close(); err != nil {- return nil, nil, nil, err+ return nil, err}- return columns, arrayRows, objectRows, nil+ return res, nil+}++func decodeBody(r *http.Request, target any) error {+ decoder := json.NewDecoder(r.Body)+ decoder.UseNumber()+ return decoder.Decode(target)}// decodeParam maps a JSON parameter onto a SQLite bind value.@@ -331,73 +419,6 @@ func encodeCell(cell any) any {}}-// splitStatements divides SQL text at top-level semicolons, honoring string-// literals, quoted identifiers, and comments.-func splitStatements(text string) []string {- var statements []string- var current strings.Builder- flush := func() {- stmt := strings.TrimSpace(current.String())- current.Reset()- if stmt != "" && !isCommentOnly(stmt) {- statements = append(statements, stmt)- }- }- for i := 0; i < len(text); i++ {- c := text[i]- switch {- case c == '-' && i+1 < len(text) && text[i+1] == '-':- end := strings.IndexByte(text[i:], '\n')- if end < 0 {- i = len(text)- } else {- i += end- }- case c == '/' && i+1 < len(text) && text[i+1] == '*':- end := strings.Index(text[i+2:], "*/")- if end < 0 {- i = len(text)- } else {- i += end + 3- }- case c == '\'' || c == '"' || c == '`':- end := closingQuote(text, i, c)- current.WriteString(text[i : end+1])- i = end- case c == ';':- flush()- default:- current.WriteByte(c)- }- }- flush()- return statements-}--func closingQuote(text string, start int, quote byte) int {- for i := start + 1; i < len(text); i++ {- if text[i] != quote {- continue- }- if i+1 < len(text) && text[i+1] == quote {- i++- continue- }- return i- }- return len(text) - 1-}--func isCommentOnly(stmt string) bool {- for _, line := range strings.Split(stmt, "\n") {- line = strings.TrimSpace(line)- if line != "" && !strings.HasPrefix(line, "--") {- return false- }- }- return true-}-func writeErrors(w http.ResponseWriter, status, code int, message string) {writeJSON(w, status, map[string]any{"success": false,@@ -406,6 +427,10 @@ func writeErrors(w http.ResponseWriter, status, code int, message string) {})}+func writeBridgeError(w http.ResponseWriter, status int, message string) {+ writeJSON(w, status, map[string]any{"error": apiError{Message: message}})+}+func writeJSON(w http.ResponseWriter, status int, body any) {w.Header().Set("Content-Type", "application/json")w.WriteHeader(status)diff --git a/store/db/d1/executor.go b/store/db/d1/executor.goindex c42f3e785..87842fed2 100644--- a/store/db/d1/executor.go+++ b/store/db/d1/executor.go@@ -50,12 +50,12 @@ func (b *batch) guard(condition string, args ...any) {// commit executes the batch atomically and returns one result per statement.func (b *batch) commit(ctx context.Context, d *DB) ([]*result, error) {- return d.client.batch(ctx, b.stmts)+ return d.transport.batch(ctx, b.stmts)}// execOne runs a single write outside any batch.func (d *DB) execOne(ctx context.Context, sql string, args ...any) (*result, error) {- return d.client.exec(ctx, statement{SQL: sql, Args: args})+ return d.transport.exec(ctx, statement{SQL: sql, Args: args})}// placeholders returns n comma separated ? markers.diff --git a/store/db/d1/rest.go b/store/db/d1/rest.gonew file mode 100644index 000000000..68a97bfef--- /dev/null+++ b/store/db/d1/rest.go@@ -0,0 +1,191 @@+package d1++import (+ "context"+ "encoding/json"+ "fmt"+ "net/http"+ "net/url"+ "strings"++ "github.com/pkg/errors"+)++// restTransport talks to one D1 database through the Cloudflare REST API+// with an API token.+type restTransport struct {+ httpClient *http.Client+ config *Config+}++type restInfo struct {+ Code int `json:"code"`+ Message string `json:"message"`+}++type restMeta struct {+ Changes json.Number `json:"changes"`+ LastRowID json.Number `json:"last_row_id"`+ SizeAfter json.Number `json:"size_after"`+}++type restRows struct {+ Columns []string `json:"columns"`+ Rows [][]json.RawMessage `json:"rows"`+}++type restResult struct {+ Success bool `json:"success"`+ Meta restMeta `json:"meta"`+ Results json.RawMessage `json:"results"`+}++type restEnvelope struct {+ Success bool `json:"success"`+ Errors []restInfo `json:"errors"`+ Result []restResult `json:"result"`+}++func (t *restTransport) exec(ctx context.Context, stmt statement) (*result, error) {+ params, err := encodeArgs(stmt.Args)+ if err != nil {+ return nil, err+ }+ results, err := t.raw(ctx, map[string]any{"sql": stmt.SQL, "params": params})+ if err != nil {+ return nil, err+ }+ if len(results) != 1 {+ return nil, errors.Errorf("d1: expected 1 result, got %d", len(results))+ }+ return results[0], nil+}++// script sends the text as one request; the REST API executes the+// semicolon-separated statements as a batch.+func (t *restTransport) script(ctx context.Context, sql string) error {+ _, err := t.raw(ctx, map[string]any{"sql": sql})+ return err+}++func (t *restTransport) batch(ctx context.Context, stmts []statement) ([]*result, error) {+ if len(stmts) == 0 {+ return nil, nil+ }+ if len(stmts) == 1 {+ res, err := t.exec(ctx, stmts[0])+ if err != nil {+ return nil, err+ }+ return []*result{res}, nil+ }+ body, err := encodeStatements(stmts)+ if err != nil {+ return nil, err+ }+ results, err := t.raw(ctx, map[string]any{"batch": body})+ if err != nil {+ return nil, err+ }+ if len(results) != len(stmts) {+ return nil, errors.Errorf("d1: expected %d results, got %d", len(stmts), len(results))+ }+ return results, nil+}++func (t *restTransport) raw(ctx context.Context, body map[string]any) ([]*result, error) {+ payload, err := json.Marshal(body)+ if err != nil {+ return nil, errors.Wrap(err, "failed to encode d1 request")+ }+ responseBody, status, err := httpDo(ctx, t.httpClient, http.MethodPost, t.databaseURL()+"/raw", t.config.Token, payload)+ if err != nil {+ return nil, err+ }+ var env restEnvelope+ if err := decodeJSON(responseBody, &env); err != nil {+ return nil, errors.Wrap(err, "failed to decode d1 response")+ }+ if status != http.StatusOK || !env.Success {+ return nil, restError(status, env.Errors)+ }+ results := make([]*result, 0, len(env.Result))+ for _, item := range env.Result {+ if !item.Success {+ return nil, &Error{Status: status, Message: "statement reported failure"}+ }+ res, err := decodeRESTResult(item)+ if err != nil {+ return nil, err+ }+ results = append(results, res)+ }+ return results, nil+}++// databaseSize returns the file size reported by the D1 metadata endpoint.+func (t *restTransport) databaseSize(ctx context.Context) (int64, error) {+ responseBody, status, err := httpDo(ctx, t.httpClient, http.MethodGet, t.databaseURL()+"?fields=file_size", t.config.Token, nil)+ if err != nil {+ return -1, err+ }+ var env struct {+ Success bool `json:"success"`+ Errors []restInfo `json:"errors"`+ Result struct {+ FileSize json.Number `json:"file_size"`+ } `json:"result"`+ }+ if err := decodeJSON(responseBody, &env); err != nil {+ return -1, errors.Wrap(err, "failed to decode d1 database metadata")+ }+ if status != http.StatusOK || !env.Success {+ return -1, restError(status, env.Errors)+ }+ size, err := env.Result.FileSize.Int64()+ if err != nil {+ return -1, errors.Wrap(err, "failed to parse d1 database size")+ }+ return size, nil+}++func (t *restTransport) databaseURL() string {+ return fmt.Sprintf("%s/accounts/%s/d1/database/%s", t.config.Endpoint, url.PathEscape(t.config.AccountID), url.PathEscape(t.config.DatabaseID))+}++func restError(status int, infos []restInfo) error {+ if len(infos) == 0 {+ return &Error{Status: status, Message: http.StatusText(status)}+ }+ messages := make([]string, 0, len(infos))+ for _, info := range infos {+ messages = append(messages, info.Message)+ }+ return &Error{Status: status, Code: infos[0].Code, Message: strings.Join(messages, "; ")}+}++func decodeRESTResult(item restResult) (*result, error) {+ res := &result{}+ var err error+ if res.Changes, err = decodeNumber(item.Meta.Changes, "change count"); err != nil {+ return nil, err+ }+ if res.LastRowID, err = decodeNumber(item.Meta.LastRowID, "last row id"); err != nil {+ return nil, err+ }+ if res.SizeAfter, err = decodeNumber(item.Meta.SizeAfter, "database size"); err != nil {+ return nil, err+ }+ if len(item.Results) == 0 || string(item.Results) == "null" {+ return res, nil+ }+ var raw restRows+ if err := decodeJSON(item.Results, &raw); err != nil {+ return nil, errors.Wrap(err, "failed to decode d1 rows")+ }+ res.Columns = raw.Columns+ if res.Rows, err = decodeRows(raw.Columns, raw.Rows); err != nil {+ return nil, err+ }+ return res, nil+}diff --git a/store/db/d1/sqldriver.go b/store/db/d1/sqldriver.goindex 3266569fd..33f36b4fc 100644--- a/store/db/d1/sqldriver.go+++ b/store/db/d1/sqldriver.go@@ -10,7 +10,7 @@ import ("github.com/pkg/errors")-// The types below adapt the REST client to database/sql so that the migrator+// The types below adapt the transport to database/sql so that the migrator// and tests can use GetDB(). D1 has no interactive transactions: a Tx buffers// its writes and commits them as one atomic batch, and refuses reads once a// write has been buffered because the write is not yet visible.@@ -24,19 +24,19 @@ func (sqlDriver) Open(string) (driver.Conn, error) {}type connector struct {- client *client+ transport transport}func (c *connector) Connect(context.Context) (driver.Conn, error) {- return &conn{client: c.client}, nil+ return &conn{transport: c.transport}, nil}func (*connector) Driver() driver.Driver {return sqlDriver{}}-func openSQLDB(c *client) *sql.DB {- db := sql.OpenDB(&connector{client: c})+func openSQLDB(t transport) *sql.DB {+ db := sql.OpenDB(&connector{transport: t})// Every connection is a stateless HTTP session; keep the pool small so// concurrent callers do not fan out into the REST rate limit.db.SetMaxOpenConns(8)@@ -44,8 +44,8 @@ func openSQLDB(c *client) *sql.DB {}type conn struct {- client *client- tx *tx+ transport transport+ tx *tx}func (c *conn) Prepare(query string) (driver.Stmt, error) {@@ -69,7 +69,7 @@ func (c *conn) BeginTx(context.Context, driver.TxOptions) (driver.Tx, error) {}func (c *conn) Ping(ctx context.Context) error {- _, err := c.client.exec(ctx, statement{SQL: "SELECT 1"})+ _, err := c.transport.exec(ctx, statement{SQL: "SELECT 1"})return err}@@ -79,7 +79,7 @@ func (c *conn) ExecContext(ctx context.Context, query string, args []driver.Namec.tx.pending = append(c.tx.pending, stmt)return deferredResult{}, nil}- res, err := c.client.exec(ctx, stmt)+ res, err := c.transport.exec(ctx, stmt)if err != nil {return nil, err}@@ -90,7 +90,7 @@ func (c *conn) QueryContext(ctx context.Context, query string, args []driver.Namif c.tx != nil && len(c.tx.pending) > 0 {return nil, errUnsupportedQueryInTx}- res, err := c.client.exec(ctx, statement{SQL: query, Args: namedValues(args)})+ res, err := c.transport.exec(ctx, statement{SQL: query, Args: namedValues(args)})if err != nil {return nil, err}@@ -165,9 +165,9 @@ func (t *tx) Commit() error {for _, stmt := range t.pending {sqls = append(sqls, strings.TrimSpace(stmt.SQL))}- return t.conn.client.script(ctx, strings.Join(sqls, "\n;\n"))+ return t.conn.transport.script(ctx, strings.Join(sqls, "\n;\n"))}- _, err := t.conn.client.batch(ctx, t.pending)+ _, err := t.conn.transport.batch(ctx, t.pending)return err}diff --git a/store/db/d1/sqlsplit/sqlsplit.go b/store/db/d1/sqlsplit/sqlsplit.gonew file mode 100644index 000000000..6f26a82db--- /dev/null+++ b/store/db/d1/sqlsplit/sqlsplit.go@@ -0,0 +1,53 @@+// Package sqlsplit divides a Memos SQL script into its statements. The D1+// bridge transport needs it because a bridge request carries one statement+// per entry, and the test emulator uses it to execute a script statement by+// statement the way D1 does.+//+// The split is a file-layout convention rather than a SQL parser: the scripts+// under store/migration/d1 keep exactly one statement per paragraph, so a+// blank line is the statement boundary. Nothing inside a statement, be it a+// CREATE TABLE body, a trigger body, a string literal, or a comment, may+// contain a blank line. The convention is enforced by the store test suite,+// which applies every script through the emulator.+package sqlsplit++import "strings"++// Split divides text at blank lines. Paragraphs made only of line comments+// are dropped; a statement keeps the comment lines and the trailing+// semicolon it was written with.+func Split(text string) []string {+ var statements []string+ var paragraph []string+ flush := func() {+ if stmt := paragraphStatement(paragraph); stmt != "" {+ statements = append(statements, stmt)+ }+ paragraph = paragraph[:0]+ }+ for _, line := range strings.Split(strings.ReplaceAll(text, "\r\n", "\n"), "\n") {+ if strings.TrimSpace(line) == "" {+ flush()+ continue+ }+ paragraph = append(paragraph, line)+ }+ flush()+ return statements+}++// paragraphStatement returns the paragraph as one statement, or "" when it+// holds no SQL.+func paragraphStatement(lines []string) string {+ hasSQL := false+ for _, line := range lines {+ if !strings.HasPrefix(strings.TrimSpace(line), "--") {+ hasSQL = true+ break+ }+ }+ if !hasSQL {+ return ""+ }+ return strings.TrimSpace(strings.Join(lines, "\n"))+}diff --git a/store/db/d1/sqlsplit/sqlsplit_test.go b/store/db/d1/sqlsplit/sqlsplit_test.gonew file mode 100644index 000000000..c18bc8c8f--- /dev/null+++ b/store/db/d1/sqlsplit/sqlsplit_test.go@@ -0,0 +1,24 @@+package sqlsplit++import (+ "testing"++ "github.com/stretchr/testify/require"+)++func TestSplitOneStatementPerParagraph(t *testing.T) {+ statements := Split("-- header comment\n\n-- user\nCREATE TABLE user (\n id INTEGER PRIMARY KEY,\n name TEXT DEFAULT 'a;b'\n);\n\n\nCREATE INDEX idx ON user(name);\r\n\r\nSELECT 1\n")+ require.Equal(t, []string{+ "-- user\nCREATE TABLE user (\n id INTEGER PRIMARY KEY,\n name TEXT DEFAULT 'a;b'\n);",+ "CREATE INDEX idx ON user(name);",+ "SELECT 1",+ }, statements)++ require.Empty(t, Split("-- only a comment\n\n -- another\n"))+ require.Empty(t, Split("\n\n"))+}++func TestSplitKeepsCompoundStatementsTogether(t *testing.T) {+ trigger := "CREATE TRIGGER t AFTER UPDATE ON memo\nBEGIN\n UPDATE memo SET updated_ts = 1 WHERE id = NEW.id;\n UPDATE memo SET pinned = CASE WHEN NEW.pinned = 1 THEN 1 ELSE 0 END WHERE id = NEW.id;\nEND;"+ require.Equal(t, []string{"DROP TRIGGER IF EXISTS t;", trigger}, Split("DROP TRIGGER IF EXISTS t;\n\n"+trigger))+}diff --git a/store/db/d1/transport.go b/store/db/d1/transport.gonew file mode 100644index 000000000..ac5e12d55--- /dev/null+++ b/store/db/d1/transport.go@@ -0,0 +1,256 @@+package d1++import (+ "bytes"+ "context"+ "database/sql/driver"+ "encoding/json"+ "fmt"+ "io"+ "net/http"+ "time"++ "github.com/pkg/errors"+)++// maxBoundParameters is D1's documented per-statement bind limit.+const maxBoundParameters = 100++// transport executes statements against one D1 database. The REST API and+// the deployment bridge implement it; the driver methods never see which.+type transport interface {+ // exec runs one statement and returns its rows and change counters.+ exec(ctx context.Context, stmt statement) (*result, error)+ // batch runs the statements in order as one transaction: either every+ // statement commits or none does.+ batch(ctx context.Context, stmts []statement) ([]*result, error)+ // script runs a parameter-free, possibly multi-statement SQL text as+ // one atomic unit. It backs schema migrations.+ script(ctx context.Context, sql string) error+ // databaseSize returns the database size in bytes, or -1 when the+ // transport cannot report one.+ databaseSize(ctx context.Context) (int64, error)+}++// newTransport selects the transport for the configured access mode.+func newTransport(config *Config) (transport, error) {+ httpClient := &http.Client{+ Timeout: 60 * time.Second,+ // Neither endpoint redirects, and following one could carry the+ // bearer credential onto a plaintext hop, so redirects surface as+ // errors instead.+ CheckRedirect: func(*http.Request, []*http.Request) error {+ return http.ErrUseLastResponse+ },+ }+ switch config.Mode {+ case AccessModeREST:+ return &restTransport{httpClient: httpClient, config: config}, nil+ case AccessModeBridge:+ return &bridgeTransport{httpClient: httpClient, config: config}, nil+ default:+ return nil, errors.Errorf("d1: unsupported access mode %q", config.Mode)+ }+}++// Error is a failure reported by the D1 service or the bridge.+type Error struct {+ // Status is the HTTP status code of the response.+ Status int+ // Code is the service error code, when the response carried one.+ Code int+ // Message is the human readable error text.+ Message string+}++func (e *Error) Error() string {+ if e.Code != 0 {+ return fmt.Sprintf("d1: %s (code %d, http %d)", e.Message, e.Code, e.Status)+ }+ return fmt.Sprintf("d1: %s (http %d)", e.Message, e.Status)+}++// statement is one parameterized SQL statement.+type statement struct {+ SQL string+ Args []any+}++// result is the outcome of one statement.+type result struct {+ Columns []string+ Rows [][]any+ Changes int64+ LastRowID int64+ // SizeAfter is the database size after the statement committed, or 0+ // when the transport did not report it.+ SizeAfter int64+}++// requestStatement is the wire form of a statement shared by both transports.+type requestStatement struct {+ SQL string `json:"sql"`+ Params []any `json:"params,omitempty"`+}++// encodeStatements converts statements to their wire form.+func encodeStatements(stmts []statement) ([]requestStatement, error) {+ body := make([]requestStatement, 0, len(stmts))+ for _, stmt := range stmts {+ params, err := encodeArgs(stmt.Args)+ if err != nil {+ return nil, err+ }+ body = append(body, requestStatement{SQL: stmt.SQL, Params: params})+ }+ return body, nil+}++// httpDo posts or gets JSON with an optional bearer credential and returns+// the raw body with the status code.+func httpDo(ctx context.Context, httpClient *http.Client, method, target, token string, payload []byte) ([]byte, int, error) {+ var reader io.Reader+ if payload != nil {+ reader = bytes.NewReader(payload)+ }+ request, err := http.NewRequestWithContext(ctx, method, target, reader)+ if err != nil {+ return nil, 0, errors.Wrap(err, "failed to build d1 request")+ }+ if token != "" {+ request.Header.Set("Authorization", "Bearer "+token)+ }+ request.Header.Set("Accept", "application/json")+ if payload != nil {+ request.Header.Set("Content-Type", "application/json")+ }+ response, err := httpClient.Do(request)+ if err != nil {+ return nil, 0, errors.Wrap(err, "d1 request failed")+ }+ defer response.Body.Close()+ responseBody, err := io.ReadAll(response.Body)+ if err != nil {+ return nil, response.StatusCode, errors.Wrap(err, "failed to read d1 response")+ }+ return responseBody, response.StatusCode, nil+}++func decodeJSON(data []byte, target any) error {+ decoder := json.NewDecoder(bytes.NewReader(data))+ decoder.UseNumber()+ return decoder.Decode(target)+}++// decodeRows maps a column list and JSON row arrays onto driver values.+func decodeRows(columns []string, rawRows [][]json.RawMessage) ([][]any, error) {+ rows := make([][]any, 0, len(rawRows))+ for _, rawRow := range rawRows {+ row := make([]any, 0, len(rawRow))+ for _, rawValue := range rawRow {+ value, err := decodeValue(rawValue)+ if err != nil {+ return nil, err+ }+ row = append(row, value)+ }+ if len(row) != len(columns) {+ return nil, errors.Errorf("d1: row has %d cells for %d columns", len(row), len(columns))+ }+ rows = append(rows, row)+ }+ return rows, nil+}++// decodeNumber parses an optional JSON number, treating absence as zero.+func decodeNumber(number json.Number, what string) (int64, error) {+ if number == "" {+ return 0, nil+ }+ value, err := number.Int64()+ if err != nil {+ return 0, errors.Wrapf(err, "failed to parse d1 %s", what)+ }+ return value, nil+}++// decodeValue maps one JSON cell onto the driver.Value domain.+func decodeValue(raw json.RawMessage) (any, error) {+ if len(raw) == 0 {+ return nil, nil+ }+ switch raw[0] {+ case 'n':+ return nil, nil+ case 't':+ return int64(1), nil+ case 'f':+ return int64(0), nil+ case '"':+ var text string+ if err := json.Unmarshal(raw, &text); err != nil {+ return nil, errors.Wrap(err, "failed to decode d1 text cell")+ }+ return text, nil+ case '[':+ // Blobs arrive as arrays of byte values.+ var numbers []json.Number+ if err := decodeJSON(raw, &numbers); err != nil {+ return nil, errors.Wrap(err, "failed to decode d1 blob cell")+ }+ bytesValue := make([]byte, 0, len(numbers))+ for _, number := range numbers {+ b, err := number.Int64()+ if err != nil || b < 0 || b > 255 {+ return nil, errors.Errorf("d1: invalid blob byte %q", number)+ }+ bytesValue = append(bytesValue, byte(b))+ }+ return bytesValue, nil+ case '{':+ return string(raw), nil+ default:+ number := json.Number(bytes.TrimSpace(raw))+ if integer, err := number.Int64(); err == nil {+ return integer, nil+ }+ float, err := number.Float64()+ if err != nil {+ return nil, errors.Wrapf(err, "d1: invalid numeric cell %q", string(raw))+ }+ return float, nil+ }+}++// encodeArgs converts Go arguments to the JSON parameter list D1 accepts.+// Binary values are rejected: blob columns are written with unhex(?) and a+// hex-encoded string because JSON cannot carry raw bytes.+func encodeArgs(args []any) ([]any, error) {+ if len(args) > maxBoundParameters {+ return nil, errors.Errorf("d1: statement binds %d parameters, limit is %d", len(args), maxBoundParameters)+ }+ params := make([]any, 0, len(args))+ for _, arg := range args {+ value, err := driver.DefaultParameterConverter.ConvertValue(arg)+ if err != nil {+ return nil, errors.Wrap(err, "d1: unsupported parameter")+ }+ switch v := value.(type) {+ case nil, int64, float64, string:+ params = append(params, v)+ case bool:+ if v {+ params = append(params, int64(1))+ } else {+ params = append(params, int64(0))+ }+ case []byte:+ return nil, errors.New("d1: binary parameters are not supported; bind hex text through unhex(?)")+ case time.Time:+ params = append(params, v.UTC().Format(time.RFC3339Nano))+ default:+ return nil, errors.Errorf("d1: unsupported parameter type %T", value)+ }+ }+ return params, nil+}diff --git a/store/test/README.md b/store/test/README.mdindex 48f16825f..e8c811f56 100644--- a/store/test/README.md+++ b/store/test/README.md@@ -14,14 +14,17 @@ DRIVER=mysql DSN=root@/memos_test go test -v ./test/store/...## How to test store with Cloudflare D1?-`DRIVER=d1` runs the suite against an in-process emulation of the D1 REST API-(`store/db/d1/d1test`), so no Cloudflare account is needed:+`DRIVER=d1` runs the suite against an in-process emulation of D1+(`store/db/d1/d1test`), so no Cloudflare account is needed. It speaks the REST+protocol by default and the bridge protocol with `D1_ACCESS=bridge`:```goDRIVER=d1 go test -v ./store/test/...+DRIVER=d1 D1_ACCESS=bridge go test -v ./store/test/...```To run against a real database instead, set `D1_DSN` to-`d1://<account_id>/<database_id>?token=<api_token>`. That database is shared+`d1://<account_id>/<database_id>?token=<api_token>` or to a+`d1-bridge://<worker-host>/<path>?token=<secret>` bridge endpoint. That database is sharedby every test and is not reset between them, so run one test at a time with`-run` and expect to clean it up afterwards.diff --git a/store/test/main_test.go b/store/test/main_test.goindex 444c8c59d..10df1693e 100644--- a/store/test/main_test.go+++ b/store/test/main_test.go@@ -24,26 +24,37 @@ func TestMain(m *testing.M) {}func runAllDrivers() {- drivers := []string{"sqlite", "mysql", "postgres", "d1"}+ // Each run names a driver and the extra environment it needs; D1 runs+ // once per access mode.+ runs := []struct {+ label string+ env []string+ }{+ {"sqlite", []string{"DRIVER=sqlite"}},+ {"mysql", []string{"DRIVER=mysql"}},+ {"postgres", []string{"DRIVER=postgres"}},+ {"d1 (rest)", []string{"DRIVER=d1", "D1_ACCESS=rest"}},+ {"d1 (bridge)", []string{"DRIVER=d1", "D1_ACCESS=bridge"}},+ }_, currentFile, _, _ := runtime.Caller(0)projectRoot := filepath.Dir(filepath.Dir(filepath.Dir(currentFile)))var failed []string- for _, driver := range drivers {- fmt.Printf("\n==================== %s ====================\n\n", driver)+ for _, run := range runs {+ fmt.Printf("\n==================== %s ====================\n\n", run.label)cmd := exec.Command("go", "test", "-v", "-count=1", "./store/test/...")cmd.Dir = projectRoot// A D1_DSN names one shared database; the loop's parallel tests need// the per-test emulator instead, so it is dropped from the child.env := slices.DeleteFunc(os.Environ(), func(entry string) bool { return strings.HasPrefix(entry, "D1_DSN=") })- env = append(env, "DRIVER="+driver)+ env = append(env, run.env...)cmd.Env = envcmd.Stdout = os.Stdoutcmd.Stderr = os.Stderrif err := cmd.Run(); err != nil {- failed = append(failed, driver)+ failed = append(failed, run.label)}}diff --git a/store/test/store.go b/store/test/store.goindex 5b9a973a1..7e99b4800 100644--- a/store/test/store.go+++ b/store/test/store.go@@ -23,9 +23,10 @@ import (// Each test gets its own isolated database:// - SQLite: new temp file per test// - MySQL/PostgreSQL: new database per test in shared container-// - D1: new emulated database per test (see store/db/d1/d1test), or the-// database named by D1_DSN when set; that database is shared, so run one-// test at a time against it+// - D1: new emulated database per test (see store/db/d1/d1test), reached+// over the REST protocol or, with D1_ACCESS=bridge, the bridge protocol;+// or the database named by D1_DSN when set. That database is shared, so+// run one test at a time against itfunc NewTestingStore(ctx context.Context, t *testing.T) *store.Store {driver := getDriverFromEnv()profile := getTestingProfileForDriver(t, driver)@@ -112,7 +113,12 @@ func getTestingProfileForDriver(t *testing.T, driver string) *profile.Profile {case "d1":dsn = os.Getenv("D1_DSN")if dsn == "" {- dsn = d1test.New(t).DSN()+ server := d1test.New(t)+ if os.Getenv("D1_ACCESS") == "bridge" {+ dsn = server.BridgeDSN()+ } else {+ dsn = server.DSN()+ }}default:t.Fatalf("unsupported driver: %s", driver)