Skip to content

armgit / memos

Public Imported from GitHub
0

Memos is an open-source, self-hosted note-taking app that brings your everyday thoughts together in a personal timeline. Your data stays yours, and you choose what to keep private or share.

feat(d1): add bridge access mode (#6333)

Johnny5ddf4745a20d
diff --git a/docs/design/cloudflare-d1.md b/docs/design/cloudflare-d1.md
index 415b15a45..fcb2cf260 100644
--- a/docs/design/cloudflare-d1.md
+++ b/docs/design/cloudflare-d1.md
@@ -11,16 +11,19 @@ Related: [Multi-Spaces Design](multi-spaces.md)
## Summary
Memos gains a fourth database engine, Cloudflare D1, selected with
-`--driver d1`. D1 runs SQLite, but it is reached over Cloudflare's REST API
-rather than a local file or socket, it exposes no interactive transactions,
-and it accepts no custom SQL functions. Those three facts make it a different
+`--driver d1`. D1 runs SQLite, but it is reached over HTTP rather than a
+local file or socket, either through Cloudflare's REST API or through a
+bridge Worker the deployment provides, it exposes no interactive
+transactions, and it accepts no custom SQL functions. Those three facts make it a different
engine from the SQLite driver rather than a variant of it, so it lives in its
own package, `store/db/d1`, with its own migration directory, its own filter
dialect, and its own test harness, exactly like MySQL and PostgreSQL.
## Goals
-- Run an unmodified Memos binary against a D1 database from any host.
+- Run an unmodified Memos binary against a D1 database from any host, with
+ a Cloudflare API token or with access the deployment already holds through
+ a Worker binding.
- Keep every store invariant the other drivers enforce: no partial writes,
no stale-read overwrites, the same error values for the same conditions.
- Share no Go code with the SQLite driver so either can change freely.
@@ -40,26 +43,176 @@ dialect, and its own test harness, exactly like MySQL and PostgreSQL.
## Configuration
+Two access modes exist. Either one leaves the driver's behaviour, the schema,
+and the data untouched; switching is a DSN change.
+
+**REST mode** talks to the Cloudflare REST API with an API token that has the
+D1 Edit permission:
+
```
--driver d1 --dsn 'd1://<account_id>/<database_id>?token=<api_token>'
```
The token may instead come from `CLOUDFLARE_API_TOKEN` so it stays out of
process listings. An `endpoint` query parameter overrides the API base URL and
-exists for the test emulator. The token needs the D1 Edit permission.
+exists for the test emulator.
+
+**Bridge mode** talks to a Worker the deployment provides. The Worker holds the
+D1 binding and implements the bridge protocol below, so Memos needs no
+Cloudflare API token. The bridge is reached in one of two ways.
+
+*Private path.* When Memos runs as a Cloudflare Container, its Worker can
+serve the bridge on a virtual hostname through an `outboundByHost` handler.
+Cloudflare intercepts the container's plain-HTTP requests to that hostname
+and runs the handler in the Workers runtime, on the same machine, reachable
+only from the container's own egress. No credential is involved:
+
+```
+--driver d1 --dsn 'd1-bridge://d1.internal/d1?private=true'
+```
+
+`private=true` states that the platform keeps the path private; the driver
+then uses plain HTTP and requires no secret. This is the recommended
+deployment for Containers because there is nothing to provision or rotate.
+
+*Public HTTPS endpoint.* A bridge exposed on a public route must be
+protected by a shared secret. Generate one, for example with
+`openssl rand -hex 32`, and configure the same value on both sides: Memos
+reads `MEMOS_D1_BRIDGE_TOKEN` and sends it as `Authorization: Bearer`; the
+Worker reads `BRIDGE_SECRET` and checks it before executing any SQL. Keep the
+secret out of the DSN so it stays out of process listings:
+
+```
+MEMOS_D1_BRIDGE_TOKEN=<secret> memos --driver d1 --dsn 'd1-bridge://worker.example.com/d1'
+```
+
+The driver refuses a public bridge DSN without a secret. This secret is a
+random credential the operator generates, not a Cloudflare API token.
## Transport
-The driver holds one HTTP client for the database. Reads and single writes
-go to the `/raw` endpoint, which returns rows as arrays with a column list.
-Bound parameters travel as JSON numbers, strings, and nulls; booleans become
-integers. Binary values cannot travel in JSON, so blob columns are written
-with `unhex(?)` bound to hex text and read back through `hex(blob)`.
+The driver holds one HTTP transport for the database, selected by the access
+mode. The driver methods see only its four operations: run one statement, run
+a batch atomically, run a parameter-free script, and report the database size.
+
+In REST mode reads and single writes go to the `/raw` endpoint, which returns
+rows as arrays with a column list, batches use the documented `batch` body,
+and scripts are sent as one semicolon-separated text. In bridge mode every
+operation is one bridge request; a script is divided into its statements
+locally because a bridge entry carries exactly one statement. That division
+is a layout rule, not a SQL parser: the D1 scripts keep one statement per
+paragraph, so a blank line is the boundary (see Schema and migrations).
+
+Bound parameters travel as JSON numbers, strings, and nulls in both modes;
+booleans become integers. Binary values cannot travel in JSON, so blob columns
+are written with `unhex(?)` bound to hex text and read back through `hex(blob)`.
+
+A thin `database/sql` adapter wraps the same transport so `Driver.GetDB()`
+keeps working for the migrator and for tests. Its transactions buffer writes
+and send them as one atomic request on commit. A read after a buffered write
+is refused, because the write is not visible until the batch runs.
+
+## Bridge protocol
+
+Memos owns this protocol; the deployment repository owns the Worker that
+implements it, its D1 binding, and its Wrangler configuration. The protocol
+is one endpoint.
-A thin `database/sql` adapter wraps the same client so `Driver.GetDB()` keeps
-working for the migrator and for tests. Its transactions buffer writes and
-send them as one atomic request on commit. A read after a buffered write is
-refused, because the write is not visible until the batch runs.
+**Request.** `POST <bridge-url>` with `Content-Type: application/json` and,
+when a secret is configured, `Authorization: Bearer <secret>`. The body is:
+
+```json
+{"statements": [{"sql": "INSERT INTO memo (uid) VALUES (?)", "params": ["m1"]},
+ {"sql": "SELECT id, uid FROM memo WHERE uid = ?", "params": ["m1"]}]}
+```
+
+- `statements` holds one or more entries, each exactly one SQL statement.
+ `params` is optional and binds positionally; values are JSON strings,
+ numbers, or `null`. Memos never sends more than 100 parameters per entry.
+- The Worker must run the entries in order as one transaction, which is what
+ the binding's `batch()` provides: if any statement fails, none commits.
+
+**Response.** `200 OK` with one result per statement, in order:
+
+```json
+{"results": [
+ {"columns": [], "rows": [], "meta": {"changes": 1, "last_row_id": 7, "size_after": 8192}},
+ {"columns": ["id", "uid"], "rows": [[7, "m1"]], "meta": {"changes": 0, "last_row_id": 7, "size_after": 8192}}
+]}
+```
+
+- `columns` lists the result columns in order and `rows` holds one array per
+ row in that order. A statement without a result set returns both empty.
+ Cell values are JSON numbers, strings, or `null`; a blob is an array of byte
+ values, though Memos reads blobs through `hex()` and never relies on it.
+- `meta.changes` is the number of rows the statement modified,
+ `meta.last_row_id` the last inserted rowid, and `meta.size_after` the
+ database size in bytes after the batch committed. Memos reports the size
+ through `size_after`, so a Worker that omits it makes the size unavailable
+ rather than wrong.
+
+**Errors.** Any failure is a non-200 status with a body of
+`{"error": {"message": "<text>"}}`. The message must carry the D1 error text
+verbatim, for example `D1_ERROR: UNIQUE constraint failed: space.uid`: the
+driver recognises constraint failures and guard aborts by that text. A
+missing or wrong bearer secret should answer `401`.
+
+Memos never issues a query whose result has two columns of the same name, so
+a Worker may build `columns` from the keys of the first row object that
+`batch()` returns and `rows` from each row's values in that order.
+
+A Worker that meets the contract serves the private path from the
+Container's outbound handler and, if a public route is wanted as well, checks
+the shared secret on `fetch`. A public route with no secret configured is
+refused rather than left open:
+
+```js
+import { Container, ContainerProxy } from "@cloudflare/containers";
+export { ContainerProxy };
+
+async function runBatch(env, request) {
+ const { statements } = await request.json();
+ const prepared = statements.map((s) => env.DB.prepare(s.sql).bind(...(s.params ?? [])));
+ try {
+ const results = (await env.DB.batch(prepared)).map(({ results, meta }) => {
+ const columns = results.length ? Object.keys(results[0]) : [];
+ return {
+ columns,
+ rows: results.map((row) => columns.map((c) => row[c])),
+ meta: { changes: meta.changes, last_row_id: meta.last_row_id, size_after: meta.size_after },
+ };
+ });
+ return Response.json({ results });
+ } catch (err) {
+ return error(400, err.message);
+ }
+}
+
+const error = (status, message) => Response.json({ error: { message } }, { status });
+
+export class MemosContainer extends Container {
+ defaultPort = 5230;
+}
+
+// Private path: plain HTTP from the container to http://d1.internal/d1 lands
+// here, never on a public route, so no credential is needed.
+MemosContainer.outboundByHost = {
+ "d1.internal": (request, env) => runBatch(env, request),
+};
+
+// Public route: only for deployments that must reach the bridge from outside
+// the container. It is closed until BRIDGE_SECRET is configured.
+export default {
+ async fetch(request, env) {
+ if (request.method !== "POST") return error(405, "method not allowed");
+ if (!env.BRIDGE_SECRET) return error(503, "bridge secret is not configured");
+ if (request.headers.get("Authorization") !== `Bearer ${env.BRIDGE_SECRET}`) {
+ return error(401, "unauthorized");
+ }
+ return runBatch(env, request);
+ },
+};
+```
## Atomicity without transactions
@@ -96,6 +249,13 @@ files under `store/migration/d1/<version>/` alongside the other drivers'.
They must avoid the `PRAGMA foreign_keys` toggles the SQLite migrations use,
which D1 ignores.
+D1 scripts follow one layout rule: **one statement per paragraph**. A blank
+line separates statements, and nothing inside a statement, including a table
+or trigger body, a string literal, or a comment, may contain a blank line.
+The bridge transport divides scripts at blank lines to send them as a batch,
+and the test emulator executes them the same way, so a script that breaks the
+rule fails the store suite's fresh install rather than a deployment.
+
## Filter dialect
`filter.DialectD1` renders like SQLite except that case folding uses the
@@ -105,9 +265,11 @@ patterns with optional anchors.
## Testing
-`store/db/d1/d1test` emulates the D1 REST API in-process on a private SQLite
-database: the same endpoints, request bodies, response shapes, implicit
-transaction per request, atomic batches, foreign-key enforcement, and the
-100-parameter limit, with no custom functions registered. `DRIVER=d1` runs
-the whole store suite against it. Setting `D1_DSN` points the suite at a real
+`store/db/d1/d1test` emulates D1 in-process on a private SQLite database and
+speaks both protocols: the REST API endpoints and the bridge endpoint, with
+the same implicit transaction per request, atomic batches, foreign-key
+enforcement, and 100-parameter limit, and no custom functions registered.
+`DRIVER=d1` runs the whole store suite against it over REST, and
+`D1_ACCESS=bridge` runs it over the bridge protocol; the all-drivers loop
+runs both. Setting `D1_DSN` points the suite at a real
database for spot checks.
diff --git a/store/db/d1/bridge.go b/store/db/d1/bridge.go
new file mode 100644
index 000000000..bc6c9fec3
--- /dev/null
+++ b/store/db/d1/bridge.go
@@ -0,0 +1,146 @@
+package d1
+
+import (
+ "context"
+ "encoding/json"
+ "net/http"
+
+ "github.com/pkg/errors"
+
+ "github.com/usememos/memos/store/db/d1/sqlsplit"
+)
+
+// bridgeTransport talks to a deployment-provided Worker that holds the D1
+// binding and speaks the Memos bridge protocol documented in
+// docs/design/cloudflare-d1.md. Every request is one atomic batch; the
+// Worker runs it through the binding's batch() and returns one result per
+// statement.
+type bridgeTransport struct {
+ httpClient *http.Client
+ config *Config
+}
+
+type bridgeRequest struct {
+ Statements []requestStatement `json:"statements"`
+}
+
+type bridgeMeta struct {
+ Changes json.Number `json:"changes"`
+ LastRowID json.Number `json:"last_row_id"`
+ SizeAfter json.Number `json:"size_after"`
+}
+
+type bridgeResult struct {
+ Columns []string `json:"columns"`
+ Rows [][]json.RawMessage `json:"rows"`
+ Meta bridgeMeta `json:"meta"`
+}
+
+type bridgeResponse struct {
+ Results []bridgeResult `json:"results"`
+ Error *struct {
+ Code int `json:"code"`
+ Message string `json:"message"`
+ } `json:"error"`
+}
+
+func (t *bridgeTransport) exec(ctx context.Context, stmt statement) (*result, error) {
+ results, err := t.run(ctx, []statement{stmt})
+ if err != nil {
+ return nil, err
+ }
+ return results[0], nil
+}
+
+func (t *bridgeTransport) batch(ctx context.Context, stmts []statement) ([]*result, error) {
+ if len(stmts) == 0 {
+ return nil, nil
+ }
+ return t.run(ctx, stmts)
+}
+
+// script splits the text into statements locally, since the bridge carries
+// exactly one statement per entry, and runs them as one atomic batch.
+func (t *bridgeTransport) script(ctx context.Context, sql string) error {
+ parts := sqlsplit.Split(sql)
+ if len(parts) == 0 {
+ return nil
+ }
+ stmts := make([]statement, 0, len(parts))
+ for _, part := range parts {
+ stmts = append(stmts, statement{SQL: part})
+ }
+ _, err := t.run(ctx, stmts)
+ return err
+}
+
+// databaseSize reads the size the binding reports after a trivial statement.
+// A bridge that omits size_after yields -1.
+func (t *bridgeTransport) databaseSize(ctx context.Context) (int64, error) {
+ res, err := t.exec(ctx, statement{SQL: "SELECT 1"})
+ if err != nil {
+ return -1, err
+ }
+ if res.SizeAfter <= 0 {
+ return -1, nil
+ }
+ return res.SizeAfter, nil
+}
+
+func (t *bridgeTransport) run(ctx context.Context, stmts []statement) ([]*result, error) {
+ body, err := encodeStatements(stmts)
+ if err != nil {
+ return nil, err
+ }
+ payload, err := json.Marshal(bridgeRequest{Statements: body})
+ if err != nil {
+ return nil, errors.Wrap(err, "failed to encode d1 bridge request")
+ }
+ responseBody, status, err := httpDo(ctx, t.httpClient, http.MethodPost, t.config.BridgeURL, t.config.Token, payload)
+ if err != nil {
+ return nil, err
+ }
+ var response bridgeResponse
+ if err := decodeJSON(responseBody, &response); err != nil {
+ if status != http.StatusOK {
+ return nil, &Error{Status: status, Message: http.StatusText(status)}
+ }
+ return nil, errors.Wrap(err, "failed to decode d1 bridge response")
+ }
+ if response.Error != nil {
+ return nil, &Error{Status: status, Code: response.Error.Code, Message: response.Error.Message}
+ }
+ if status != http.StatusOK {
+ return nil, &Error{Status: status, Message: http.StatusText(status)}
+ }
+ if len(response.Results) != len(stmts) {
+ return nil, errors.Errorf("d1: bridge returned %d results for %d statements", len(response.Results), len(stmts))
+ }
+ results := make([]*result, 0, len(response.Results))
+ for _, item := range response.Results {
+ res, err := decodeBridgeResult(item)
+ if err != nil {
+ return nil, err
+ }
+ results = append(results, res)
+ }
+ return results, nil
+}
+
+func decodeBridgeResult(item bridgeResult) (*result, error) {
+ res := &result{Columns: item.Columns}
+ var err error
+ if res.Changes, err = decodeNumber(item.Meta.Changes, "change count"); err != nil {
+ return nil, err
+ }
+ if res.LastRowID, err = decodeNumber(item.Meta.LastRowID, "last row id"); err != nil {
+ return nil, err
+ }
+ if res.SizeAfter, err = decodeNumber(item.Meta.SizeAfter, "database size"); err != nil {
+ return nil, err
+ }
+ if res.Rows, err = decodeRows(item.Columns, item.Rows); err != nil {
+ return nil, err
+ }
+ return res, nil
+}
diff --git a/store/db/d1/client.go b/store/db/d1/client.go
deleted file mode 100644
index dce511851..000000000
--- a/store/db/d1/client.go
+++ /dev/null
@@ -1,462 +0,0 @@
-package d1
-
-import (
- "bytes"
- "context"
- "database/sql/driver"
- "encoding/json"
- "fmt"
- "io"
- "net"
- "net/http"
- "net/url"
- "os"
- "strings"
- "time"
-
- "github.com/pkg/errors"
-)
-
-// DefaultEndpoint is the Cloudflare REST API base URL.
-const DefaultEndpoint = "https://api.cloudflare.com/client/v4"
-
-// maxBoundParameters is D1's documented per-statement bind limit.
-const maxBoundParameters = 100
-
-// Config is the parsed form of a D1 DSN.
-//
-// DSN grammar:
-//
-// d1://<account_id>/<database_id>?token=<api_token>[&endpoint=<base_url>]
-//
-// The token may be omitted from the DSN and supplied through the
-// CLOUDFLARE_API_TOKEN environment variable instead so it stays out of
-// process listings and logs.
-type Config struct {
- AccountID string
- DatabaseID string
- Token string
- Endpoint string
-}
-
-// ParseDSN parses a d1:// DSN.
-func ParseDSN(dsn string) (*Config, error) {
- parsed, err := url.Parse(dsn)
- if err != nil {
- return nil, errors.Wrap(err, "failed to parse d1 dsn")
- }
- if parsed.Scheme != "d1" {
- return nil, errors.Errorf("d1 dsn must use the d1:// scheme, got %q", parsed.Scheme)
- }
- config := &Config{
- AccountID: parsed.Host,
- DatabaseID: strings.Trim(parsed.Path, "/"),
- Token: parsed.Query().Get("token"),
- Endpoint: strings.TrimRight(parsed.Query().Get("endpoint"), "/"),
- }
- if config.AccountID == "" {
- return nil, errors.New("d1 dsn is missing the account id: d1://<account_id>/<database_id>")
- }
- if config.DatabaseID == "" || strings.Contains(config.DatabaseID, "/") {
- return nil, errors.New("d1 dsn is missing the database id: d1://<account_id>/<database_id>")
- }
- if config.Token == "" {
- config.Token = os.Getenv("CLOUDFLARE_API_TOKEN")
- }
- if config.Token == "" {
- return nil, errors.New("d1 api token is required: add ?token=<api_token> to the dsn or set CLOUDFLARE_API_TOKEN")
- }
- if config.Endpoint == "" {
- config.Endpoint = DefaultEndpoint
- }
- if err := validateEndpoint(config.Endpoint); err != nil {
- return nil, err
- }
- return config, nil
-}
-
-// validateEndpoint rejects endpoints that would send the bearer token in the
-// clear. Plain HTTP is allowed only towards loopback, where the test
-// emulator listens.
-func validateEndpoint(endpoint string) error {
- parsed, err := url.Parse(endpoint)
- if err != nil {
- return errors.Wrap(err, "invalid d1 endpoint")
- }
- if parsed.Host == "" {
- return errors.Errorf("d1 endpoint %q has no host", endpoint)
- }
- switch parsed.Scheme {
- case "https":
- return nil
- case "http":
- if isLoopbackHost(parsed.Hostname()) {
- return nil
- }
- return errors.Errorf("d1 endpoint %q must use https unless it points at loopback", endpoint)
- default:
- return errors.Errorf("d1 endpoint %q must use https", endpoint)
- }
-}
-
-func isLoopbackHost(host string) bool {
- if host == "localhost" {
- return true
- }
- ip := net.ParseIP(host)
- return ip != nil && ip.IsLoopback()
-}
-
-// Error is a failure reported by the D1 REST API.
-type Error struct {
- // Status is the HTTP status code of the response.
- Status int
- // Code is the Cloudflare API error code, when the response carried one.
- Code int
- // Message is the human readable error text.
- Message string
-}
-
-func (e *Error) Error() string {
- if e.Code != 0 {
- return fmt.Sprintf("d1: %s (code %d, http %d)", e.Message, e.Code, e.Status)
- }
- return fmt.Sprintf("d1: %s (http %d)", e.Message, e.Status)
-}
-
-// statement is one parameterized SQL statement.
-type statement struct {
- SQL string
- Args []any
-}
-
-// result is the outcome of one statement.
-type result struct {
- Columns []string
- Rows [][]any
- Changes int64
- LastRowID int64
-}
-
-// client talks to one D1 database through the Cloudflare REST API.
-type client struct {
- httpClient *http.Client
- config *Config
-}
-
-func newClient(config *Config) *client {
- return &client{
- httpClient: &http.Client{Timeout: 60 * time.Second},
- config: config,
- }
-}
-
-type requestStatement struct {
- SQL string `json:"sql"`
- Params []any `json:"params,omitempty"`
-}
-
-type responseInfo struct {
- Code int `json:"code"`
- Message string `json:"message"`
-}
-
-type responseMeta struct {
- Changes json.Number `json:"changes"`
- LastRowID json.Number `json:"last_row_id"`
-}
-
-type rawResults struct {
- Columns []string `json:"columns"`
- Rows [][]json.RawMessage `json:"rows"`
-}
-
-type queryResult struct {
- Success bool `json:"success"`
- Meta responseMeta `json:"meta"`
- Results json.RawMessage `json:"results"`
-}
-
-type envelope struct {
- Success bool `json:"success"`
- Errors []responseInfo `json:"errors"`
- Result []queryResult `json:"result"`
-}
-
-// exec runs one statement and returns its rows and change counters.
-func (c *client) exec(ctx context.Context, stmt statement) (*result, error) {
- params, err := encodeArgs(stmt.Args)
- if err != nil {
- return nil, err
- }
- results, err := c.raw(ctx, map[string]any{"sql": stmt.SQL, "params": params})
- if err != nil {
- return nil, err
- }
- if len(results) != 1 {
- return nil, errors.Errorf("d1: expected 1 result, got %d", len(results))
- }
- return results[0], nil
-}
-
-// script runs a parameter-free, possibly multi-statement SQL text as one
-// atomic batch. It backs schema migrations.
-func (c *client) script(ctx context.Context, sql string) error {
- _, err := c.raw(ctx, map[string]any{"sql": sql})
- return err
-}
-
-// batch runs the statements in order as one transaction: either every
-// statement commits or none does.
-func (c *client) batch(ctx context.Context, stmts []statement) ([]*result, error) {
- if len(stmts) == 0 {
- return nil, nil
- }
- if len(stmts) == 1 {
- res, err := c.exec(ctx, stmts[0])
- if err != nil {
- return nil, err
- }
- return []*result{res}, nil
- }
- body := make([]requestStatement, 0, len(stmts))
- for _, stmt := range stmts {
- params, err := encodeArgs(stmt.Args)
- if err != nil {
- return nil, err
- }
- body = append(body, requestStatement{SQL: stmt.SQL, Params: params})
- }
- results, err := c.raw(ctx, map[string]any{"batch": body})
- if err != nil {
- return nil, err
- }
- if len(results) != len(stmts) {
- return nil, errors.Errorf("d1: expected %d results, got %d", len(stmts), len(results))
- }
- return results, nil
-}
-
-func (c *client) raw(ctx context.Context, body map[string]any) ([]*result, error) {
- payload, err := json.Marshal(body)
- if err != nil {
- return nil, errors.Wrap(err, "failed to encode d1 request")
- }
- responseBody, status, err := c.do(ctx, http.MethodPost, c.databaseURL()+"/raw", payload)
- if err != nil {
- return nil, err
- }
- var env envelope
- if err := decodeJSON(responseBody, &env); err != nil {
- return nil, errors.Wrap(err, "failed to decode d1 response")
- }
- if status != http.StatusOK || !env.Success {
- return nil, envelopeError(status, env.Errors)
- }
- results := make([]*result, 0, len(env.Result))
- for _, item := range env.Result {
- if !item.Success {
- return nil, &Error{Status: status, Message: "statement reported failure"}
- }
- res, err := decodeResult(item)
- if err != nil {
- return nil, err
- }
- results = append(results, res)
- }
- return results, nil
-}
-
-// databaseSize returns the database file size reported by the D1 metadata endpoint.
-func (c *client) databaseSize(ctx context.Context) (int64, error) {
- responseBody, status, err := c.do(ctx, http.MethodGet, c.databaseURL()+"?fields=file_size", nil)
- if err != nil {
- return -1, err
- }
- var env struct {
- Success bool `json:"success"`
- Errors []responseInfo `json:"errors"`
- Result struct {
- FileSize json.Number `json:"file_size"`
- } `json:"result"`
- }
- if err := decodeJSON(responseBody, &env); err != nil {
- return -1, errors.Wrap(err, "failed to decode d1 database metadata")
- }
- if status != http.StatusOK || !env.Success {
- return -1, envelopeError(status, env.Errors)
- }
- size, err := env.Result.FileSize.Int64()
- if err != nil {
- return -1, errors.Wrap(err, "failed to parse d1 database size")
- }
- return size, nil
-}
-
-func (c *client) databaseURL() string {
- return fmt.Sprintf("%s/accounts/%s/d1/database/%s", c.config.Endpoint, url.PathEscape(c.config.AccountID), url.PathEscape(c.config.DatabaseID))
-}
-
-func (c *client) do(ctx context.Context, method, target string, payload []byte) ([]byte, int, error) {
- var reader io.Reader
- if payload != nil {
- reader = bytes.NewReader(payload)
- }
- request, err := http.NewRequestWithContext(ctx, method, target, reader)
- if err != nil {
- return nil, 0, errors.Wrap(err, "failed to build d1 request")
- }
- request.Header.Set("Authorization", "Bearer "+c.config.Token)
- request.Header.Set("Accept", "application/json")
- if payload != nil {
- request.Header.Set("Content-Type", "application/json")
- }
- response, err := c.httpClient.Do(request)
- if err != nil {
- return nil, 0, errors.Wrap(err, "d1 request failed")
- }
- defer response.Body.Close()
- responseBody, err := io.ReadAll(response.Body)
- if err != nil {
- return nil, response.StatusCode, errors.Wrap(err, "failed to read d1 response")
- }
- return responseBody, response.StatusCode, nil
-}
-
-func envelopeError(status int, infos []responseInfo) error {
- if len(infos) == 0 {
- return &Error{Status: status, Message: http.StatusText(status)}
- }
- messages := make([]string, 0, len(infos))
- for _, info := range infos {
- messages = append(messages, info.Message)
- }
- return &Error{Status: status, Code: infos[0].Code, Message: strings.Join(messages, "; ")}
-}
-
-func decodeJSON(data []byte, target any) error {
- decoder := json.NewDecoder(bytes.NewReader(data))
- decoder.UseNumber()
- return decoder.Decode(target)
-}
-
-func decodeResult(item queryResult) (*result, error) {
- res := &result{}
- if item.Meta.Changes != "" {
- changes, err := item.Meta.Changes.Int64()
- if err != nil {
- return nil, errors.Wrap(err, "failed to parse d1 change count")
- }
- res.Changes = changes
- }
- if item.Meta.LastRowID != "" {
- lastRowID, err := item.Meta.LastRowID.Int64()
- if err != nil {
- return nil, errors.Wrap(err, "failed to parse d1 last row id")
- }
- res.LastRowID = lastRowID
- }
- if len(item.Results) == 0 || string(item.Results) == "null" {
- return res, nil
- }
- var raw rawResults
- if err := decodeJSON(item.Results, &raw); err != nil {
- return nil, errors.Wrap(err, "failed to decode d1 rows")
- }
- res.Columns = raw.Columns
- res.Rows = make([][]any, 0, len(raw.Rows))
- for _, rawRow := range raw.Rows {
- row := make([]any, 0, len(rawRow))
- for _, rawValue := range rawRow {
- value, err := decodeValue(rawValue)
- if err != nil {
- return nil, err
- }
- row = append(row, value)
- }
- res.Rows = append(res.Rows, row)
- }
- return res, nil
-}
-
-// decodeValue maps one JSON cell onto the driver.Value domain.
-func decodeValue(raw json.RawMessage) (any, error) {
- if len(raw) == 0 {
- return nil, nil
- }
- switch raw[0] {
- case 'n':
- return nil, nil
- case 't':
- return int64(1), nil
- case 'f':
- return int64(0), nil
- case '"':
- var text string
- if err := json.Unmarshal(raw, &text); err != nil {
- return nil, errors.Wrap(err, "failed to decode d1 text cell")
- }
- return text, nil
- case '[':
- // Blobs arrive as arrays of byte values.
- var bytesValue []byte
- var numbers []json.Number
- if err := decodeJSON(raw, &numbers); err != nil {
- return nil, errors.Wrap(err, "failed to decode d1 blob cell")
- }
- bytesValue = make([]byte, 0, len(numbers))
- for _, number := range numbers {
- b, err := number.Int64()
- if err != nil || b < 0 || b > 255 {
- return nil, errors.Errorf("d1: invalid blob byte %q", number)
- }
- bytesValue = append(bytesValue, byte(b))
- }
- return bytesValue, nil
- case '{':
- return string(raw), nil
- default:
- number := json.Number(bytes.TrimSpace(raw))
- if integer, err := number.Int64(); err == nil {
- return integer, nil
- }
- float, err := number.Float64()
- if err != nil {
- return nil, errors.Wrapf(err, "d1: invalid numeric cell %q", string(raw))
- }
- return float, nil
- }
-}
-
-// encodeArgs converts Go arguments to the JSON parameter list D1 accepts.
-// Binary values are rejected: blob columns are written with unhex(?) and a
-// hex-encoded string because JSON cannot carry raw bytes.
-func encodeArgs(args []any) ([]any, error) {
- if len(args) > maxBoundParameters {
- return nil, errors.Errorf("d1: statement binds %d parameters, limit is %d", len(args), maxBoundParameters)
- }
- params := make([]any, 0, len(args))
- for _, arg := range args {
- value, err := driver.DefaultParameterConverter.ConvertValue(arg)
- if err != nil {
- return nil, errors.Wrap(err, "d1: unsupported parameter")
- }
- switch v := value.(type) {
- case nil, int64, float64, string:
- params = append(params, v)
- case bool:
- if v {
- params = append(params, int64(1))
- } else {
- params = append(params, int64(0))
- }
- case []byte:
- return nil, errors.New("d1: binary parameters are not supported; bind hex text through unhex(?)")
- case time.Time:
- params = append(params, v.UTC().Format(time.RFC3339Nano))
- default:
- return nil, errors.Errorf("d1: unsupported parameter type %T", value)
- }
- }
- return params, nil
-}
diff --git a/store/db/d1/config.go b/store/db/d1/config.go
new file mode 100644
index 000000000..fc57c2cf3
--- /dev/null
+++ b/store/db/d1/config.go
@@ -0,0 +1,165 @@
+package d1
+
+import (
+ "net"
+ "net/url"
+ "os"
+ "strings"
+
+ "github.com/pkg/errors"
+)
+
+// DefaultEndpoint is the Cloudflare REST API base URL.
+const DefaultEndpoint = "https://api.cloudflare.com/client/v4"
+
+// AccessMode selects how the driver reaches the database.
+type AccessMode string
+
+const (
+ // AccessModeREST talks to the Cloudflare REST API with an API token.
+ AccessModeREST AccessMode = "rest"
+ // AccessModeBridge talks to a deployment-provided Worker that holds the
+ // D1 binding, so Memos needs no Cloudflare API token.
+ AccessModeBridge AccessMode = "bridge"
+)
+
+// Config is the parsed form of a D1 DSN.
+//
+// DSN grammar:
+//
+// d1://<account_id>/<database_id>?token=<api_token>[&endpoint=<base_url>]
+// d1-bridge://<host>[:<port>][/<path>][?token=<secret>][&private=true]
+//
+// The first form selects REST access. Its token may be omitted from the DSN
+// and supplied through the CLOUDFLARE_API_TOKEN environment variable instead
+// so it stays out of process listings and logs.
+//
+// The second form selects bridge access: the host and path name the Worker
+// endpoint that implements the bridge protocol (see docs/design/cloudflare-d1.md).
+// A public bridge is reached over HTTPS and requires a shared secret, sent as
+// a bearer credential; the secret is best supplied through
+// MEMOS_D1_BRIDGE_TOKEN rather than the DSN. private=true names a bridge on
+// a platform-private path, such as the virtual host a Cloudflare Containers
+// outbound handler serves or the loopback test emulator: it is reached over
+// plain HTTP and the secret is optional.
+type Config struct {
+ Mode AccessMode
+
+ // Private marks a bridge on a platform-private path (bridge access only).
+ Private bool
+
+ // REST access.
+ AccountID string
+ DatabaseID string
+ Endpoint string
+
+ // Bridge access.
+ BridgeURL string
+
+ // Token is the API token (REST) or the bridge bearer secret (bridge).
+ Token string
+}
+
+// ParseDSN parses a d1:// or d1-bridge:// DSN.
+func ParseDSN(dsn string) (*Config, error) {
+ parsed, err := url.Parse(dsn)
+ if err != nil {
+ return nil, errors.Wrap(err, "failed to parse d1 dsn")
+ }
+ switch parsed.Scheme {
+ case "d1":
+ return parseRESTDSN(parsed)
+ case "d1-bridge":
+ return parseBridgeDSN(parsed)
+ default:
+ return nil, errors.Errorf("d1 dsn must use the d1:// or d1-bridge:// scheme, got %q", parsed.Scheme)
+ }
+}
+
+func parseRESTDSN(parsed *url.URL) (*Config, error) {
+ config := &Config{
+ Mode: AccessModeREST,
+ AccountID: parsed.Host,
+ DatabaseID: strings.Trim(parsed.Path, "/"),
+ Token: parsed.Query().Get("token"),
+ Endpoint: strings.TrimRight(parsed.Query().Get("endpoint"), "/"),
+ }
+ if config.AccountID == "" {
+ return nil, errors.New("d1 dsn is missing the account id: d1://<account_id>/<database_id>")
+ }
+ if config.DatabaseID == "" || strings.Contains(config.DatabaseID, "/") {
+ return nil, errors.New("d1 dsn is missing the database id: d1://<account_id>/<database_id>")
+ }
+ if config.Token == "" {
+ config.Token = os.Getenv("CLOUDFLARE_API_TOKEN")
+ }
+ if config.Token == "" {
+ return nil, errors.New("d1 api token is required: add ?token=<api_token> to the dsn or set CLOUDFLARE_API_TOKEN")
+ }
+ if config.Endpoint == "" {
+ config.Endpoint = DefaultEndpoint
+ }
+ if err := validateEndpoint(config.Endpoint); err != nil {
+ return nil, err
+ }
+ return config, nil
+}
+
+func parseBridgeDSN(parsed *url.URL) (*Config, error) {
+ if parsed.Host == "" {
+ return nil, errors.New("d1 bridge dsn is missing the host: d1-bridge://<host>[/<path>]")
+ }
+ query := parsed.Query()
+ config := &Config{
+ Mode: AccessModeBridge,
+ Private: query.Get("private") == "true",
+ Token: query.Get("token"),
+ }
+ if config.Token == "" {
+ config.Token = os.Getenv("MEMOS_D1_BRIDGE_TOKEN")
+ }
+ path := strings.TrimRight(parsed.Path, "/")
+ if config.Private {
+ // The platform keeps the path private, so plain HTTP carries no risk
+ // and a shared secret is not needed.
+ config.BridgeURL = "http://" + parsed.Host + path
+ return config, nil
+ }
+ config.BridgeURL = "https://" + parsed.Host + path
+ if config.Token == "" {
+ return nil, errors.New("a public d1 bridge requires a shared secret: set MEMOS_D1_BRIDGE_TOKEN (or add ?token=) or mark a platform-private bridge with ?private=true")
+ }
+ return config, nil
+}
+
+// validateEndpoint rejects endpoints that would send a bearer credential in
+// the clear. Plain HTTP is allowed only towards loopback, where the test
+// emulator listens.
+func validateEndpoint(endpoint string) error {
+ parsed, err := url.Parse(endpoint)
+ if err != nil {
+ return errors.Wrap(err, "invalid d1 endpoint")
+ }
+ if parsed.Host == "" {
+ return errors.Errorf("d1 endpoint %q has no host", endpoint)
+ }
+ switch parsed.Scheme {
+ case "https":
+ return nil
+ case "http":
+ if isLoopbackHost(parsed.Hostname()) {
+ return nil
+ }
+ return errors.Errorf("d1 endpoint %q must use https unless it points at loopback", endpoint)
+ default:
+ return errors.Errorf("d1 endpoint %q must use https", endpoint)
+ }
+}
+
+func isLoopbackHost(host string) bool {
+ if host == "localhost" {
+ return true
+ }
+ ip := net.ParseIP(host)
+ return ip != nil && ip.IsLoopback()
+}
diff --git a/store/db/d1/d1.go b/store/db/d1/d1.go
index 8487f43c1..77c587282 100644
--- a/store/db/d1/d1.go
+++ b/store/db/d1/d1.go
@@ -1,7 +1,8 @@
// Package d1 implements the store driver for Cloudflare D1.
//
-// D1 is reached through the Cloudflare REST API rather than a socket, and it
-// exposes no interactive transactions. Each driver method therefore performs
+// D1 is reached over HTTP rather than a socket, either through the Cloudflare
+// REST API or through a deployment-provided bridge Worker, and it exposes no
+// interactive transactions. Each driver method therefore performs
// its validation reads first and then commits its writes as one atomic batch.
// Preconditions that must hold at commit time are re-checked inside the batch
// with guard statements (see guard.go) so a stale read never yields a partial
@@ -20,9 +21,9 @@ import (
// DB is the D1 store driver.
type DB struct {
- client *client
- db *sql.DB
- profile *profile.Profile
+ transport transport
+ db *sql.DB
+ profile *profile.Profile
}
// NewDB opens a D1 database described by the profile DSN.
@@ -31,11 +32,14 @@ func NewDB(profile *profile.Profile) (store.Driver, error) {
if err != nil {
return nil, err
}
- c := newClient(config)
- return &DB{client: c, db: openSQLDB(c), profile: profile}, nil
+ t, err := newTransport(config)
+ if err != nil {
+ return nil, err
+ }
+ return &DB{transport: t, db: openSQLDB(t), profile: profile}, nil
}
-// GetDB exposes a database/sql handle over the REST client for the migrator
+// GetDB exposes a database/sql handle over the transport for the migrator
// and tests. See sqldriver.go for its transaction semantics.
func (d *DB) GetDB() *sql.DB {
return d.db
@@ -56,9 +60,10 @@ func (d *DB) IsInitialized(ctx context.Context) (bool, error) {
return exists, nil
}
-// GetDatabaseSize returns the database size reported by the D1 metadata endpoint.
+// GetDatabaseSize returns the database size the transport reports, or -1
+// when it cannot report one.
func (d *DB) GetDatabaseSize(ctx context.Context) (int64, error) {
- size, err := d.client.databaseSize(ctx)
+ size, err := d.transport.databaseSize(ctx)
if err != nil {
return -1, errors.Wrap(err, "failed to read d1 database size")
}
diff --git a/store/db/d1/d1_test.go b/store/db/d1/d1_test.go
index ec709c541..0c687077a 100644
--- a/store/db/d1/d1_test.go
+++ b/store/db/d1/d1_test.go
@@ -3,6 +3,9 @@ package d1
import (
"context"
"errors"
+ "net/http"
+ "net/http/httptest"
+ "strings"
"testing"
"github.com/stretchr/testify/require"
@@ -13,10 +16,17 @@ import (
var errConflict = errors.New("conflict")
-func newTestDB(t *testing.T) *DB {
+// accessModes lists the DSN of each access mode against the emulator; every
+// transport test runs under both.
+var accessModes = map[string]func(*d1test.Server) string{
+ "rest": (*d1test.Server).DSN,
+ "bridge": (*d1test.Server).BridgeDSN,
+}
+
+func newTestDB(t *testing.T, dsn func(*d1test.Server) string) *DB {
t.Helper()
server := d1test.New(t)
- driver, err := NewDB(&profile.Profile{Driver: "d1", DSN: server.DSN()})
+ driver, err := NewDB(&profile.Profile{Driver: "d1", DSN: dsn(server)})
require.NoError(t, err)
db, ok := driver.(*DB)
require.True(t, ok)
@@ -42,7 +52,7 @@ func TestParseDSN(t *testing.T) {
}
_, err = ParseDSN("sqlite://x")
- require.ErrorContains(t, err, "d1:// scheme")
+ require.ErrorContains(t, err, "d1:// or d1-bridge:// scheme")
_, err = ParseDSN("d1:///dbid?token=t")
require.ErrorContains(t, err, "account id")
t.Setenv("CLOUDFLARE_API_TOKEN", "")
@@ -53,11 +63,54 @@ func TestParseDSN(t *testing.T) {
require.NoError(t, err)
require.Equal(t, "from-env", config.Token)
require.Equal(t, DefaultEndpoint, config.Endpoint)
+ require.Equal(t, AccessModeREST, config.Mode)
+}
+
+func TestParseBridgeDSN(t *testing.T) {
+ t.Setenv("MEMOS_D1_BRIDGE_TOKEN", "")
+
+ // A public bridge is HTTPS and needs a shared secret.
+ config, err := ParseDSN("d1-bridge://worker.example.com/d1?token=secret")
+ require.NoError(t, err)
+ require.Equal(t, AccessModeBridge, config.Mode)
+ require.False(t, config.Private)
+ require.Equal(t, "https://worker.example.com/d1", config.BridgeURL)
+ require.Equal(t, "secret", config.Token)
+ _, err = ParseDSN("d1-bridge://worker.example.com/d1")
+ require.ErrorContains(t, err, "requires a shared secret")
+
+ // A platform-private bridge is plain HTTP and needs no secret: the
+ // Cloudflare Containers outbound handler and the loopback emulator.
+ config, err = ParseDSN("d1-bridge://d1.internal/d1?private=true")
+ require.NoError(t, err)
+ require.True(t, config.Private)
+ require.Equal(t, "http://d1.internal/d1", config.BridgeURL)
+ require.Empty(t, config.Token)
+ config, err = ParseDSN("d1-bridge://127.0.0.1:8787/bridge/?private=true&token=t")
+ require.NoError(t, err)
+ require.Equal(t, "http://127.0.0.1:8787/bridge", config.BridgeURL)
+ require.Equal(t, "t", config.Token)
+
+ t.Setenv("MEMOS_D1_BRIDGE_TOKEN", "from-env")
+ config, err = ParseDSN("d1-bridge://worker.example.com")
+ require.NoError(t, err)
+ require.Equal(t, "https://worker.example.com", config.BridgeURL)
+ require.Equal(t, "from-env", config.Token)
+
+ _, err = ParseDSN("d1-bridge:///d1")
+ require.ErrorContains(t, err, "missing the host")
}
func TestTransportRoundTrip(t *testing.T) {
+ for mode, dsn := range accessModes {
+ t.Run(mode, func(t *testing.T) {
+ runTransportRoundTrip(t, newTestDB(t, dsn))
+ })
+ }
+}
+
+func runTransportRoundTrip(t *testing.T, db *DB) {
ctx := context.Background()
- db := newTestDB(t)
initialized, err := db.IsInitialized(ctx)
require.NoError(t, err)
@@ -66,9 +119,11 @@ func TestTransportRoundTrip(t *testing.T) {
// Multi-statement script through the sql adapter transaction, as the migrator does.
tx, err := db.GetDB().Begin()
require.NoError(t, err)
+ // One statement per paragraph, as the D1 scripts are laid out.
_, err = tx.ExecContext(ctx, `
-- schema
CREATE TABLE d1_guard (ok INTEGER NOT NULL CONSTRAINT d1_guard_ok CHECK (ok = 1));
+
CREATE TABLE memo (id INTEGER PRIMARY KEY AUTOINCREMENT, uid TEXT NOT NULL UNIQUE, content TEXT NOT NULL DEFAULT 'a;b', blob BLOB);
`)
require.NoError(t, err)
@@ -157,8 +212,15 @@ func TestTransportRoundTrip(t *testing.T) {
}
func TestNullAndTypedScans(t *testing.T) {
+ for mode, dsn := range accessModes {
+ t.Run(mode, func(t *testing.T) {
+ runNullAndTypedScans(t, newTestDB(t, dsn))
+ })
+ }
+}
+
+func runNullAndTypedScans(t *testing.T, db *DB) {
ctx := context.Background()
- db := newTestDB(t)
_, err := db.execOne(ctx, "CREATE TABLE t (id INTEGER PRIMARY KEY, n INTEGER, f REAL, s TEXT, ts BIGINT NOT NULL DEFAULT (strftime('%s', 'now')))")
require.NoError(t, err)
var nilInt *int32
@@ -174,3 +236,24 @@ func TestNullAndTypedScans(t *testing.T) {
require.Equal(t, "x", s)
require.Greater(t, ts, int64(0))
}
+
+func TestRedirectsAreNotFollowed(t *testing.T) {
+ redirected := false
+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ if r.URL.Path == "/moved" {
+ redirected = true
+ return
+ }
+ http.Redirect(w, r, "/moved", http.StatusTemporaryRedirect)
+ }))
+ t.Cleanup(server.Close)
+
+ driver, err := NewDB(&profile.Profile{Driver: "d1", DSN: "d1-bridge://" + strings.TrimPrefix(server.URL, "http://") + "/bridge?private=true&token=t"})
+ require.NoError(t, err)
+ _, err = driver.GetDB().ExecContext(context.Background(), "SELECT 1")
+ require.Error(t, err)
+ var d1Err *Error
+ require.ErrorAs(t, err, &d1Err)
+ require.Equal(t, http.StatusTemporaryRedirect, d1Err.Status)
+ require.False(t, redirected, "the redirect target must not receive the credential")
+}
diff --git a/store/db/d1/d1test/server.go b/store/db/d1/d1test/server.go
index 7a15b91df..6e6f4d457 100644
--- a/store/db/d1/d1test/server.go
+++ b/store/db/d1/d1test/server.go
@@ -1,10 +1,11 @@
-// Package d1test runs an in-process emulation of the Cloudflare D1 REST API
-// for tests. It speaks the same JSON protocol the driver uses (the /raw and
-// /query endpoints, batch bodies, the database metadata endpoint) on top of a
-// private SQLite database, and enforces the D1 constraints that matter to the
-// driver: foreign keys are on, statements are wrapped in an implicit
-// transaction, a batch is atomic, and a statement may bind at most 100
-// parameters. It registers no custom SQL functions, exactly like D1.
+// Package d1test runs an in-process emulation of Cloudflare D1 for tests. It
+// speaks both protocols the driver uses, the Cloudflare REST API (the /raw
+// and /query endpoints, batch bodies, the database metadata endpoint) and the
+// Memos bridge protocol, on top of a private SQLite database. It enforces the
+// D1 constraints that matter to the driver: foreign keys are on, statements
+// are wrapped in an implicit transaction, a batch is atomic, and a statement
+// may bind at most 100 parameters. It registers no custom SQL functions,
+// exactly like D1.
package d1test
import (
@@ -21,12 +22,16 @@ import (
// SQLite engine behind the emulated service.
_ "modernc.org/sqlite"
+
+ "github.com/usememos/memos/store/db/d1/sqlsplit"
)
const (
- accountID = "test-account"
- databaseID = "test-database"
- token = "test-token"
+ accountID = "test-account"
+ databaseID = "test-database"
+ token = "test-token"
+ bridgeToken = "bridge-secret"
+ bridgePath = "/bridge/d1"
maxBoundParameters = 100
)
@@ -57,11 +62,16 @@ func New(t testing.TB) *Server {
return s
}
-// DSN returns the driver DSN pointing at this server.
+// DSN returns the REST-mode driver DSN pointing at this server.
func (s *Server) DSN() string {
return fmt.Sprintf("d1://%s/%s?token=%s&endpoint=%s", accountID, databaseID, token, s.http.URL)
}
+// BridgeDSN returns the bridge-mode driver DSN pointing at this server.
+func (s *Server) BridgeDSN() string {
+ return fmt.Sprintf("d1-bridge://%s%s?token=%s&private=true", strings.TrimPrefix(s.http.URL, "http://"), bridgePath, bridgeToken)
+}
+
// Close stops the server and discards the database.
func (s *Server) Close() {
s.http.Close()
@@ -74,18 +84,36 @@ type requestStatement struct {
Params []any `json:"params"`
}
-type requestBody struct {
+type restRequest struct {
SQL *string `json:"sql"`
Params []any `json:"params"`
Batch []requestStatement `json:"batch"`
}
+type bridgeRequest struct {
+ Statements []requestStatement `json:"statements"`
+}
+
type apiError struct {
Code int `json:"code"`
Message string `json:"message"`
}
+// statementResult is the outcome of one executed statement before it is
+// rendered in either protocol's shape.
+type statementResult struct {
+ columns []string
+ arrayRows [][]any
+ objectRows []map[string]any
+ changes int64
+ lastRowID int64
+}
+
func (s *Server) handle(w http.ResponseWriter, r *http.Request) {
+ if r.URL.Path == bridgePath {
+ s.handleBridge(w, r)
+ return
+ }
if r.Header.Get("Authorization") != "Bearer "+token {
writeErrors(w, http.StatusForbidden, 10000, "Authentication error")
return
@@ -108,14 +136,8 @@ func (s *Server) handle(w http.ResponseWriter, r *http.Request) {
}
func (s *Server) handleMetadata(w http.ResponseWriter) {
- s.mu.Lock()
- defer s.mu.Unlock()
- var pageCount, pageSize int64
- if err := s.conn.QueryRowContext(context.Background(), "PRAGMA page_count").Scan(&pageCount); err != nil {
- writeErrors(w, http.StatusInternalServerError, 7500, err.Error())
- return
- }
- if err := s.conn.QueryRowContext(context.Background(), "PRAGMA page_size").Scan(&pageSize); err != nil {
+ size, err := s.databaseSize(context.Background())
+ if err != nil {
writeErrors(w, http.StatusInternalServerError, 7500, err.Error())
return
}
@@ -125,16 +147,28 @@ func (s *Server) handleMetadata(w http.ResponseWriter) {
"result": map[string]any{
"uuid": databaseID,
"name": "memos-test",
- "file_size": pageCount * pageSize,
+ "file_size": size,
},
})
}
+func (s *Server) databaseSize(ctx context.Context) (int64, error) {
+ s.mu.Lock()
+ defer s.mu.Unlock()
+ var pageCount, pageSize int64
+ if err := s.conn.QueryRowContext(ctx, "PRAGMA page_count").Scan(&pageCount); err != nil {
+ return 0, err
+ }
+ if err := s.conn.QueryRowContext(ctx, "PRAGMA page_size").Scan(&pageSize); err != nil {
+ return 0, err
+ }
+ return pageCount * pageSize, nil
+}
+
+// handleQuery serves the REST /raw and /query endpoints.
func (s *Server) handleQuery(w http.ResponseWriter, r *http.Request, rawRows bool) {
- var body requestBody
- decoder := json.NewDecoder(r.Body)
- decoder.UseNumber()
- if err := decoder.Decode(&body); err != nil {
+ var body restRequest
+ if err := decodeBody(r, &body); err != nil {
writeErrors(w, http.StatusBadRequest, 7400, "invalid request body: "+err.Error())
return
}
@@ -149,44 +183,113 @@ func (s *Server) handleQuery(w http.ResponseWriter, r *http.Request, rawRows boo
return
}
- // Expand every entry into single statements; a parameterized entry must
- // hold exactly one statement because D1 documents no binding rule for
- // several.
+ // A parameterized entry is one statement; a parameter-free entry is a
+ // script, executed statement by statement the way D1 does.
var stmts []requestStatement
for _, entry := range entries {
- parts := splitStatements(entry.SQL)
- if len(parts) == 0 {
- writeErrors(w, http.StatusBadRequest, 7400, "empty sql")
+ if len(entry.Params) > maxBoundParameters {
+ writeErrors(w, http.StatusBadRequest, 7400, fmt.Sprintf("too many bound parameters: %d", len(entry.Params)))
return
}
- if len(parts) > 1 && len(entry.Params) > 0 {
- writeErrors(w, http.StatusBadRequest, 7400, "params cannot be bound to multiple statements")
- return
+ if len(entry.Params) > 0 {
+ stmts = append(stmts, entry)
+ continue
}
- if len(entry.Params) > maxBoundParameters {
- writeErrors(w, http.StatusBadRequest, 7400, fmt.Sprintf("too many bound parameters: %d", len(entry.Params)))
+ parts := sqlsplit.Split(entry.SQL)
+ if len(parts) == 0 {
+ writeErrors(w, http.StatusBadRequest, 7400, "empty sql")
return
}
for _, part := range parts {
- stmts = append(stmts, requestStatement{SQL: part, Params: entry.Params})
+ stmts = append(stmts, requestStatement{SQL: part})
}
}
- results, err := s.execute(r.Context(), stmts, rawRows)
+ results, err := s.execute(r.Context(), stmts)
if err != nil {
writeErrors(w, http.StatusBadRequest, 7500, "D1_ERROR: "+err.Error())
return
}
+ rendered := make([]map[string]any, 0, len(results))
+ for _, res := range results {
+ var rows any
+ if rawRows {
+ rows = map[string]any{"columns": res.columns, "rows": res.arrayRows}
+ } else {
+ rows = res.objectRows
+ }
+ rendered = append(rendered, map[string]any{
+ "success": true,
+ "meta": map[string]any{
+ "changed_db": res.changes > 0,
+ "changes": res.changes,
+ "last_row_id": res.lastRowID,
+ "duration": 0,
+ },
+ "results": rows,
+ })
+ }
writeJSON(w, http.StatusOK, map[string]any{
"success": true,
"errors": []any{},
- "result": results,
+ "result": rendered,
})
}
+// handleBridge serves the Memos bridge protocol: one atomic batch per
+// request, one statement per entry, rows as arrays with a column list.
+func (s *Server) handleBridge(w http.ResponseWriter, r *http.Request) {
+ if r.Method != http.MethodPost {
+ writeBridgeError(w, http.StatusMethodNotAllowed, "method not allowed")
+ return
+ }
+ if r.Header.Get("Authorization") != "Bearer "+bridgeToken {
+ writeBridgeError(w, http.StatusUnauthorized, "unauthorized")
+ return
+ }
+ var body bridgeRequest
+ if err := decodeBody(r, &body); err != nil {
+ writeBridgeError(w, http.StatusBadRequest, "invalid request body: "+err.Error())
+ return
+ }
+ if len(body.Statements) == 0 {
+ writeBridgeError(w, http.StatusBadRequest, "request carries no statements")
+ return
+ }
+ for _, entry := range body.Statements {
+ if len(entry.Params) > maxBoundParameters {
+ writeBridgeError(w, http.StatusBadRequest, fmt.Sprintf("too many bound parameters: %d", len(entry.Params)))
+ return
+ }
+ }
+ results, err := s.execute(r.Context(), body.Statements)
+ if err != nil {
+ writeBridgeError(w, http.StatusBadRequest, "D1_ERROR: "+err.Error())
+ return
+ }
+ size, err := s.databaseSize(r.Context())
+ if err != nil {
+ writeBridgeError(w, http.StatusInternalServerError, err.Error())
+ return
+ }
+ rendered := make([]map[string]any, 0, len(results))
+ for _, res := range results {
+ rendered = append(rendered, map[string]any{
+ "columns": res.columns,
+ "rows": res.arrayRows,
+ "meta": map[string]any{
+ "changes": res.changes,
+ "last_row_id": res.lastRowID,
+ "size_after": size,
+ },
+ })
+ }
+ writeJSON(w, http.StatusOK, map[string]any{"results": rendered})
+}
+
// execute runs stmts inside one transaction, mirroring D1's implicit
// transaction per request and its atomic batches.
-func (s *Server) execute(ctx context.Context, stmts []requestStatement, rawRows bool) ([]map[string]any, error) {
+func (s *Server) execute(ctx context.Context, stmts []requestStatement) ([]*statementResult, error) {
s.mu.Lock()
defer s.mu.Unlock()
@@ -194,9 +297,9 @@ func (s *Server) execute(ctx context.Context, stmts []requestStatement, rawRows
if err != nil {
return nil, err
}
- results := make([]map[string]any, 0, len(stmts))
+ results := make([]*statementResult, 0, len(stmts))
for _, stmt := range stmts {
- res, err := runStatement(ctx, tx, stmt, rawRows)
+ res, err := runStatement(ctx, tx, stmt)
if err != nil {
_ = tx.Rollback()
return nil, err
@@ -209,7 +312,7 @@ func (s *Server) execute(ctx context.Context, stmts []requestStatement, rawRows
return results, nil
}
-func runStatement(ctx context.Context, tx *sql.Tx, stmt requestStatement, rawRows bool) (map[string]any, error) {
+func runStatement(ctx context.Context, tx *sql.Tx, stmt requestStatement) (*statementResult, error) {
args := make([]any, 0, len(stmt.Params))
for _, param := range stmt.Params {
args = append(args, decodeParam(param))
@@ -220,47 +323,31 @@ func runStatement(ctx context.Context, tx *sql.Tx, stmt requestStatement, rawRow
if err := tx.QueryRowContext(ctx, "SELECT total_changes()").Scan(&before); err != nil {
return nil, err
}
- columns, arrayRows, objectRows, err := collectRows(ctx, tx, stmt.SQL, args, rawRows)
+ res, err := collectRows(ctx, tx, stmt.SQL, args)
if err != nil {
return nil, err
}
- var after, lastRowID int64
- if err := tx.QueryRowContext(ctx, "SELECT total_changes(), last_insert_rowid()").Scan(&after, &lastRowID); err != nil {
+ var after int64
+ if err := tx.QueryRowContext(ctx, "SELECT total_changes(), last_insert_rowid()").Scan(&after, &res.lastRowID); err != nil {
return nil, err
}
- changes := after - before
- var results any
- if rawRows {
- results = map[string]any{"columns": columns, "rows": arrayRows}
- } else {
- results = objectRows
- }
- return map[string]any{
- "success": true,
- "meta": map[string]any{
- "changed_db": changes > 0,
- "changes": changes,
- "last_row_id": lastRowID,
- "duration": 0,
- },
- "results": results,
- }, nil
+ res.changes = after - before
+ return res, nil
}
-// collectRows executes one statement and drains its result set in the
-// requested shape. Statements without a result set yield empty slices.
-func collectRows(ctx context.Context, tx *sql.Tx, query string, args []any, rawRows bool) ([]string, [][]any, []map[string]any, error) {
+// collectRows executes one statement and drains its result set in both
+// shapes. Statements without a result set yield empty slices.
+func collectRows(ctx context.Context, tx *sql.Tx, query string, args []any) (*statementResult, error) {
rows, err := tx.QueryContext(ctx, query, args...)
if err != nil {
- return nil, nil, nil, err
+ return nil, err
}
defer rows.Close()
columns, err := rows.Columns()
if err != nil {
- return nil, nil, nil, err
+ return nil, err
}
- arrayRows := [][]any{}
- objectRows := []map[string]any{}
+ res := &statementResult{columns: columns, arrayRows: [][]any{}, objectRows: []map[string]any{}}
for rows.Next() {
cells := make([]any, len(columns))
targets := make([]any, len(columns))
@@ -268,29 +355,30 @@ func collectRows(ctx context.Context, tx *sql.Tx, query string, args []any, rawR
targets[i] = &cells[i]
}
if err := rows.Scan(targets...); err != nil {
- return nil, nil, nil, err
+ return nil, err
}
encoded := make([]any, len(columns))
+ object := make(map[string]any, len(columns))
for i, cell := range cells {
encoded[i] = encodeCell(cell)
+ object[columns[i]] = encoded[i]
}
- if rawRows {
- arrayRows = append(arrayRows, encoded)
- continue
- }
- object := make(map[string]any, len(columns))
- for i, column := range columns {
- object[column] = encoded[i]
- }
- objectRows = append(objectRows, object)
+ res.arrayRows = append(res.arrayRows, encoded)
+ res.objectRows = append(res.objectRows, object)
}
if err := rows.Err(); err != nil {
- return nil, nil, nil, err
+ return nil, err
}
if err := rows.Close(); err != nil {
- return nil, nil, nil, err
+ return nil, err
}
- return columns, arrayRows, objectRows, nil
+ return res, nil
+}
+
+func decodeBody(r *http.Request, target any) error {
+ decoder := json.NewDecoder(r.Body)
+ decoder.UseNumber()
+ return decoder.Decode(target)
}
// decodeParam maps a JSON parameter onto a SQLite bind value.
@@ -331,73 +419,6 @@ func encodeCell(cell any) any {
}
}
-// splitStatements divides SQL text at top-level semicolons, honoring string
-// literals, quoted identifiers, and comments.
-func splitStatements(text string) []string {
- var statements []string
- var current strings.Builder
- flush := func() {
- stmt := strings.TrimSpace(current.String())
- current.Reset()
- if stmt != "" && !isCommentOnly(stmt) {
- statements = append(statements, stmt)
- }
- }
- for i := 0; i < len(text); i++ {
- c := text[i]
- switch {
- case c == '-' && i+1 < len(text) && text[i+1] == '-':
- end := strings.IndexByte(text[i:], '\n')
- if end < 0 {
- i = len(text)
- } else {
- i += end
- }
- case c == '/' && i+1 < len(text) && text[i+1] == '*':
- end := strings.Index(text[i+2:], "*/")
- if end < 0 {
- i = len(text)
- } else {
- i += end + 3
- }
- case c == '\'' || c == '"' || c == '`':
- end := closingQuote(text, i, c)
- current.WriteString(text[i : end+1])
- i = end
- case c == ';':
- flush()
- default:
- current.WriteByte(c)
- }
- }
- flush()
- return statements
-}
-
-func closingQuote(text string, start int, quote byte) int {
- for i := start + 1; i < len(text); i++ {
- if text[i] != quote {
- continue
- }
- if i+1 < len(text) && text[i+1] == quote {
- i++
- continue
- }
- return i
- }
- return len(text) - 1
-}
-
-func isCommentOnly(stmt string) bool {
- for _, line := range strings.Split(stmt, "\n") {
- line = strings.TrimSpace(line)
- if line != "" && !strings.HasPrefix(line, "--") {
- return false
- }
- }
- return true
-}
-
func writeErrors(w http.ResponseWriter, status, code int, message string) {
writeJSON(w, status, map[string]any{
"success": false,
@@ -406,6 +427,10 @@ func writeErrors(w http.ResponseWriter, status, code int, message string) {
})
}
+func writeBridgeError(w http.ResponseWriter, status int, message string) {
+ writeJSON(w, status, map[string]any{"error": apiError{Message: message}})
+}
+
func writeJSON(w http.ResponseWriter, status int, body any) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(status)
diff --git a/store/db/d1/executor.go b/store/db/d1/executor.go
index c42f3e785..87842fed2 100644
--- a/store/db/d1/executor.go
+++ b/store/db/d1/executor.go
@@ -50,12 +50,12 @@ func (b *batch) guard(condition string, args ...any) {
// commit executes the batch atomically and returns one result per statement.
func (b *batch) commit(ctx context.Context, d *DB) ([]*result, error) {
- return d.client.batch(ctx, b.stmts)
+ return d.transport.batch(ctx, b.stmts)
}
// execOne runs a single write outside any batch.
func (d *DB) execOne(ctx context.Context, sql string, args ...any) (*result, error) {
- return d.client.exec(ctx, statement{SQL: sql, Args: args})
+ return d.transport.exec(ctx, statement{SQL: sql, Args: args})
}
// placeholders returns n comma separated ? markers.
diff --git a/store/db/d1/rest.go b/store/db/d1/rest.go
new file mode 100644
index 000000000..68a97bfef
--- /dev/null
+++ b/store/db/d1/rest.go
@@ -0,0 +1,191 @@
+package d1
+
+import (
+ "context"
+ "encoding/json"
+ "fmt"
+ "net/http"
+ "net/url"
+ "strings"
+
+ "github.com/pkg/errors"
+)
+
+// restTransport talks to one D1 database through the Cloudflare REST API
+// with an API token.
+type restTransport struct {
+ httpClient *http.Client
+ config *Config
+}
+
+type restInfo struct {
+ Code int `json:"code"`
+ Message string `json:"message"`
+}
+
+type restMeta struct {
+ Changes json.Number `json:"changes"`
+ LastRowID json.Number `json:"last_row_id"`
+ SizeAfter json.Number `json:"size_after"`
+}
+
+type restRows struct {
+ Columns []string `json:"columns"`
+ Rows [][]json.RawMessage `json:"rows"`
+}
+
+type restResult struct {
+ Success bool `json:"success"`
+ Meta restMeta `json:"meta"`
+ Results json.RawMessage `json:"results"`
+}
+
+type restEnvelope struct {
+ Success bool `json:"success"`
+ Errors []restInfo `json:"errors"`
+ Result []restResult `json:"result"`
+}
+
+func (t *restTransport) exec(ctx context.Context, stmt statement) (*result, error) {
+ params, err := encodeArgs(stmt.Args)
+ if err != nil {
+ return nil, err
+ }
+ results, err := t.raw(ctx, map[string]any{"sql": stmt.SQL, "params": params})
+ if err != nil {
+ return nil, err
+ }
+ if len(results) != 1 {
+ return nil, errors.Errorf("d1: expected 1 result, got %d", len(results))
+ }
+ return results[0], nil
+}
+
+// script sends the text as one request; the REST API executes the
+// semicolon-separated statements as a batch.
+func (t *restTransport) script(ctx context.Context, sql string) error {
+ _, err := t.raw(ctx, map[string]any{"sql": sql})
+ return err
+}
+
+func (t *restTransport) batch(ctx context.Context, stmts []statement) ([]*result, error) {
+ if len(stmts) == 0 {
+ return nil, nil
+ }
+ if len(stmts) == 1 {
+ res, err := t.exec(ctx, stmts[0])
+ if err != nil {
+ return nil, err
+ }
+ return []*result{res}, nil
+ }
+ body, err := encodeStatements(stmts)
+ if err != nil {
+ return nil, err
+ }
+ results, err := t.raw(ctx, map[string]any{"batch": body})
+ if err != nil {
+ return nil, err
+ }
+ if len(results) != len(stmts) {
+ return nil, errors.Errorf("d1: expected %d results, got %d", len(stmts), len(results))
+ }
+ return results, nil
+}
+
+func (t *restTransport) raw(ctx context.Context, body map[string]any) ([]*result, error) {
+ payload, err := json.Marshal(body)
+ if err != nil {
+ return nil, errors.Wrap(err, "failed to encode d1 request")
+ }
+ responseBody, status, err := httpDo(ctx, t.httpClient, http.MethodPost, t.databaseURL()+"/raw", t.config.Token, payload)
+ if err != nil {
+ return nil, err
+ }
+ var env restEnvelope
+ if err := decodeJSON(responseBody, &env); err != nil {
+ return nil, errors.Wrap(err, "failed to decode d1 response")
+ }
+ if status != http.StatusOK || !env.Success {
+ return nil, restError(status, env.Errors)
+ }
+ results := make([]*result, 0, len(env.Result))
+ for _, item := range env.Result {
+ if !item.Success {
+ return nil, &Error{Status: status, Message: "statement reported failure"}
+ }
+ res, err := decodeRESTResult(item)
+ if err != nil {
+ return nil, err
+ }
+ results = append(results, res)
+ }
+ return results, nil
+}
+
+// databaseSize returns the file size reported by the D1 metadata endpoint.
+func (t *restTransport) databaseSize(ctx context.Context) (int64, error) {
+ responseBody, status, err := httpDo(ctx, t.httpClient, http.MethodGet, t.databaseURL()+"?fields=file_size", t.config.Token, nil)
+ if err != nil {
+ return -1, err
+ }
+ var env struct {
+ Success bool `json:"success"`
+ Errors []restInfo `json:"errors"`
+ Result struct {
+ FileSize json.Number `json:"file_size"`
+ } `json:"result"`
+ }
+ if err := decodeJSON(responseBody, &env); err != nil {
+ return -1, errors.Wrap(err, "failed to decode d1 database metadata")
+ }
+ if status != http.StatusOK || !env.Success {
+ return -1, restError(status, env.Errors)
+ }
+ size, err := env.Result.FileSize.Int64()
+ if err != nil {
+ return -1, errors.Wrap(err, "failed to parse d1 database size")
+ }
+ return size, nil
+}
+
+func (t *restTransport) databaseURL() string {
+ return fmt.Sprintf("%s/accounts/%s/d1/database/%s", t.config.Endpoint, url.PathEscape(t.config.AccountID), url.PathEscape(t.config.DatabaseID))
+}
+
+func restError(status int, infos []restInfo) error {
+ if len(infos) == 0 {
+ return &Error{Status: status, Message: http.StatusText(status)}
+ }
+ messages := make([]string, 0, len(infos))
+ for _, info := range infos {
+ messages = append(messages, info.Message)
+ }
+ return &Error{Status: status, Code: infos[0].Code, Message: strings.Join(messages, "; ")}
+}
+
+func decodeRESTResult(item restResult) (*result, error) {
+ res := &result{}
+ var err error
+ if res.Changes, err = decodeNumber(item.Meta.Changes, "change count"); err != nil {
+ return nil, err
+ }
+ if res.LastRowID, err = decodeNumber(item.Meta.LastRowID, "last row id"); err != nil {
+ return nil, err
+ }
+ if res.SizeAfter, err = decodeNumber(item.Meta.SizeAfter, "database size"); err != nil {
+ return nil, err
+ }
+ if len(item.Results) == 0 || string(item.Results) == "null" {
+ return res, nil
+ }
+ var raw restRows
+ if err := decodeJSON(item.Results, &raw); err != nil {
+ return nil, errors.Wrap(err, "failed to decode d1 rows")
+ }
+ res.Columns = raw.Columns
+ if res.Rows, err = decodeRows(raw.Columns, raw.Rows); err != nil {
+ return nil, err
+ }
+ return res, nil
+}
diff --git a/store/db/d1/sqldriver.go b/store/db/d1/sqldriver.go
index 3266569fd..33f36b4fc 100644
--- a/store/db/d1/sqldriver.go
+++ b/store/db/d1/sqldriver.go
@@ -10,7 +10,7 @@ import (
"github.com/pkg/errors"
)
-// The types below adapt the REST client to database/sql so that the migrator
+// The types below adapt the transport to database/sql so that the migrator
// and tests can use GetDB(). D1 has no interactive transactions: a Tx buffers
// its writes and commits them as one atomic batch, and refuses reads once a
// write has been buffered because the write is not yet visible.
@@ -24,19 +24,19 @@ func (sqlDriver) Open(string) (driver.Conn, error) {
}
type connector struct {
- client *client
+ transport transport
}
func (c *connector) Connect(context.Context) (driver.Conn, error) {
- return &conn{client: c.client}, nil
+ return &conn{transport: c.transport}, nil
}
func (*connector) Driver() driver.Driver {
return sqlDriver{}
}
-func openSQLDB(c *client) *sql.DB {
- db := sql.OpenDB(&connector{client: c})
+func openSQLDB(t transport) *sql.DB {
+ db := sql.OpenDB(&connector{transport: t})
// Every connection is a stateless HTTP session; keep the pool small so
// concurrent callers do not fan out into the REST rate limit.
db.SetMaxOpenConns(8)
@@ -44,8 +44,8 @@ func openSQLDB(c *client) *sql.DB {
}
type conn struct {
- client *client
- tx *tx
+ transport transport
+ tx *tx
}
func (c *conn) Prepare(query string) (driver.Stmt, error) {
@@ -69,7 +69,7 @@ func (c *conn) BeginTx(context.Context, driver.TxOptions) (driver.Tx, error) {
}
func (c *conn) Ping(ctx context.Context) error {
- _, err := c.client.exec(ctx, statement{SQL: "SELECT 1"})
+ _, err := c.transport.exec(ctx, statement{SQL: "SELECT 1"})
return err
}
@@ -79,7 +79,7 @@ func (c *conn) ExecContext(ctx context.Context, query string, args []driver.Name
c.tx.pending = append(c.tx.pending, stmt)
return deferredResult{}, nil
}
- res, err := c.client.exec(ctx, stmt)
+ res, err := c.transport.exec(ctx, stmt)
if err != nil {
return nil, err
}
@@ -90,7 +90,7 @@ func (c *conn) QueryContext(ctx context.Context, query string, args []driver.Nam
if c.tx != nil && len(c.tx.pending) > 0 {
return nil, errUnsupportedQueryInTx
}
- res, err := c.client.exec(ctx, statement{SQL: query, Args: namedValues(args)})
+ res, err := c.transport.exec(ctx, statement{SQL: query, Args: namedValues(args)})
if err != nil {
return nil, err
}
@@ -165,9 +165,9 @@ func (t *tx) Commit() error {
for _, stmt := range t.pending {
sqls = append(sqls, strings.TrimSpace(stmt.SQL))
}
- return t.conn.client.script(ctx, strings.Join(sqls, "\n;\n"))
+ return t.conn.transport.script(ctx, strings.Join(sqls, "\n;\n"))
}
- _, err := t.conn.client.batch(ctx, t.pending)
+ _, err := t.conn.transport.batch(ctx, t.pending)
return err
}
diff --git a/store/db/d1/sqlsplit/sqlsplit.go b/store/db/d1/sqlsplit/sqlsplit.go
new file mode 100644
index 000000000..6f26a82db
--- /dev/null
+++ b/store/db/d1/sqlsplit/sqlsplit.go
@@ -0,0 +1,53 @@
+// Package sqlsplit divides a Memos SQL script into its statements. The D1
+// bridge transport needs it because a bridge request carries one statement
+// per entry, and the test emulator uses it to execute a script statement by
+// statement the way D1 does.
+//
+// The split is a file-layout convention rather than a SQL parser: the scripts
+// under store/migration/d1 keep exactly one statement per paragraph, so a
+// blank line is the statement boundary. Nothing inside a statement, be it a
+// CREATE TABLE body, a trigger body, a string literal, or a comment, may
+// contain a blank line. The convention is enforced by the store test suite,
+// which applies every script through the emulator.
+package sqlsplit
+
+import "strings"
+
+// Split divides text at blank lines. Paragraphs made only of line comments
+// are dropped; a statement keeps the comment lines and the trailing
+// semicolon it was written with.
+func Split(text string) []string {
+ var statements []string
+ var paragraph []string
+ flush := func() {
+ if stmt := paragraphStatement(paragraph); stmt != "" {
+ statements = append(statements, stmt)
+ }
+ paragraph = paragraph[:0]
+ }
+ for _, line := range strings.Split(strings.ReplaceAll(text, "\r\n", "\n"), "\n") {
+ if strings.TrimSpace(line) == "" {
+ flush()
+ continue
+ }
+ paragraph = append(paragraph, line)
+ }
+ flush()
+ return statements
+}
+
+// paragraphStatement returns the paragraph as one statement, or "" when it
+// holds no SQL.
+func paragraphStatement(lines []string) string {
+ hasSQL := false
+ for _, line := range lines {
+ if !strings.HasPrefix(strings.TrimSpace(line), "--") {
+ hasSQL = true
+ break
+ }
+ }
+ if !hasSQL {
+ return ""
+ }
+ return strings.TrimSpace(strings.Join(lines, "\n"))
+}
diff --git a/store/db/d1/sqlsplit/sqlsplit_test.go b/store/db/d1/sqlsplit/sqlsplit_test.go
new file mode 100644
index 000000000..c18bc8c8f
--- /dev/null
+++ b/store/db/d1/sqlsplit/sqlsplit_test.go
@@ -0,0 +1,24 @@
+package sqlsplit
+
+import (
+ "testing"
+
+ "github.com/stretchr/testify/require"
+)
+
+func TestSplitOneStatementPerParagraph(t *testing.T) {
+ statements := Split("-- header comment\n\n-- user\nCREATE TABLE user (\n id INTEGER PRIMARY KEY,\n name TEXT DEFAULT 'a;b'\n);\n\n\nCREATE INDEX idx ON user(name);\r\n\r\nSELECT 1\n")
+ require.Equal(t, []string{
+ "-- user\nCREATE TABLE user (\n id INTEGER PRIMARY KEY,\n name TEXT DEFAULT 'a;b'\n);",
+ "CREATE INDEX idx ON user(name);",
+ "SELECT 1",
+ }, statements)
+
+ require.Empty(t, Split("-- only a comment\n\n -- another\n"))
+ require.Empty(t, Split("\n\n"))
+}
+
+func TestSplitKeepsCompoundStatementsTogether(t *testing.T) {
+ trigger := "CREATE TRIGGER t AFTER UPDATE ON memo\nBEGIN\n UPDATE memo SET updated_ts = 1 WHERE id = NEW.id;\n UPDATE memo SET pinned = CASE WHEN NEW.pinned = 1 THEN 1 ELSE 0 END WHERE id = NEW.id;\nEND;"
+ require.Equal(t, []string{"DROP TRIGGER IF EXISTS t;", trigger}, Split("DROP TRIGGER IF EXISTS t;\n\n"+trigger))
+}
diff --git a/store/db/d1/transport.go b/store/db/d1/transport.go
new file mode 100644
index 000000000..ac5e12d55
--- /dev/null
+++ b/store/db/d1/transport.go
@@ -0,0 +1,256 @@
+package d1
+
+import (
+ "bytes"
+ "context"
+ "database/sql/driver"
+ "encoding/json"
+ "fmt"
+ "io"
+ "net/http"
+ "time"
+
+ "github.com/pkg/errors"
+)
+
+// maxBoundParameters is D1's documented per-statement bind limit.
+const maxBoundParameters = 100
+
+// transport executes statements against one D1 database. The REST API and
+// the deployment bridge implement it; the driver methods never see which.
+type transport interface {
+ // exec runs one statement and returns its rows and change counters.
+ exec(ctx context.Context, stmt statement) (*result, error)
+ // batch runs the statements in order as one transaction: either every
+ // statement commits or none does.
+ batch(ctx context.Context, stmts []statement) ([]*result, error)
+ // script runs a parameter-free, possibly multi-statement SQL text as
+ // one atomic unit. It backs schema migrations.
+ script(ctx context.Context, sql string) error
+ // databaseSize returns the database size in bytes, or -1 when the
+ // transport cannot report one.
+ databaseSize(ctx context.Context) (int64, error)
+}
+
+// newTransport selects the transport for the configured access mode.
+func newTransport(config *Config) (transport, error) {
+ httpClient := &http.Client{
+ Timeout: 60 * time.Second,
+ // Neither endpoint redirects, and following one could carry the
+ // bearer credential onto a plaintext hop, so redirects surface as
+ // errors instead.
+ CheckRedirect: func(*http.Request, []*http.Request) error {
+ return http.ErrUseLastResponse
+ },
+ }
+ switch config.Mode {
+ case AccessModeREST:
+ return &restTransport{httpClient: httpClient, config: config}, nil
+ case AccessModeBridge:
+ return &bridgeTransport{httpClient: httpClient, config: config}, nil
+ default:
+ return nil, errors.Errorf("d1: unsupported access mode %q", config.Mode)
+ }
+}
+
+// Error is a failure reported by the D1 service or the bridge.
+type Error struct {
+ // Status is the HTTP status code of the response.
+ Status int
+ // Code is the service error code, when the response carried one.
+ Code int
+ // Message is the human readable error text.
+ Message string
+}
+
+func (e *Error) Error() string {
+ if e.Code != 0 {
+ return fmt.Sprintf("d1: %s (code %d, http %d)", e.Message, e.Code, e.Status)
+ }
+ return fmt.Sprintf("d1: %s (http %d)", e.Message, e.Status)
+}
+
+// statement is one parameterized SQL statement.
+type statement struct {
+ SQL string
+ Args []any
+}
+
+// result is the outcome of one statement.
+type result struct {
+ Columns []string
+ Rows [][]any
+ Changes int64
+ LastRowID int64
+ // SizeAfter is the database size after the statement committed, or 0
+ // when the transport did not report it.
+ SizeAfter int64
+}
+
+// requestStatement is the wire form of a statement shared by both transports.
+type requestStatement struct {
+ SQL string `json:"sql"`
+ Params []any `json:"params,omitempty"`
+}
+
+// encodeStatements converts statements to their wire form.
+func encodeStatements(stmts []statement) ([]requestStatement, error) {
+ body := make([]requestStatement, 0, len(stmts))
+ for _, stmt := range stmts {
+ params, err := encodeArgs(stmt.Args)
+ if err != nil {
+ return nil, err
+ }
+ body = append(body, requestStatement{SQL: stmt.SQL, Params: params})
+ }
+ return body, nil
+}
+
+// httpDo posts or gets JSON with an optional bearer credential and returns
+// the raw body with the status code.
+func httpDo(ctx context.Context, httpClient *http.Client, method, target, token string, payload []byte) ([]byte, int, error) {
+ var reader io.Reader
+ if payload != nil {
+ reader = bytes.NewReader(payload)
+ }
+ request, err := http.NewRequestWithContext(ctx, method, target, reader)
+ if err != nil {
+ return nil, 0, errors.Wrap(err, "failed to build d1 request")
+ }
+ if token != "" {
+ request.Header.Set("Authorization", "Bearer "+token)
+ }
+ request.Header.Set("Accept", "application/json")
+ if payload != nil {
+ request.Header.Set("Content-Type", "application/json")
+ }
+ response, err := httpClient.Do(request)
+ if err != nil {
+ return nil, 0, errors.Wrap(err, "d1 request failed")
+ }
+ defer response.Body.Close()
+ responseBody, err := io.ReadAll(response.Body)
+ if err != nil {
+ return nil, response.StatusCode, errors.Wrap(err, "failed to read d1 response")
+ }
+ return responseBody, response.StatusCode, nil
+}
+
+func decodeJSON(data []byte, target any) error {
+ decoder := json.NewDecoder(bytes.NewReader(data))
+ decoder.UseNumber()
+ return decoder.Decode(target)
+}
+
+// decodeRows maps a column list and JSON row arrays onto driver values.
+func decodeRows(columns []string, rawRows [][]json.RawMessage) ([][]any, error) {
+ rows := make([][]any, 0, len(rawRows))
+ for _, rawRow := range rawRows {
+ row := make([]any, 0, len(rawRow))
+ for _, rawValue := range rawRow {
+ value, err := decodeValue(rawValue)
+ if err != nil {
+ return nil, err
+ }
+ row = append(row, value)
+ }
+ if len(row) != len(columns) {
+ return nil, errors.Errorf("d1: row has %d cells for %d columns", len(row), len(columns))
+ }
+ rows = append(rows, row)
+ }
+ return rows, nil
+}
+
+// decodeNumber parses an optional JSON number, treating absence as zero.
+func decodeNumber(number json.Number, what string) (int64, error) {
+ if number == "" {
+ return 0, nil
+ }
+ value, err := number.Int64()
+ if err != nil {
+ return 0, errors.Wrapf(err, "failed to parse d1 %s", what)
+ }
+ return value, nil
+}
+
+// decodeValue maps one JSON cell onto the driver.Value domain.
+func decodeValue(raw json.RawMessage) (any, error) {
+ if len(raw) == 0 {
+ return nil, nil
+ }
+ switch raw[0] {
+ case 'n':
+ return nil, nil
+ case 't':
+ return int64(1), nil
+ case 'f':
+ return int64(0), nil
+ case '"':
+ var text string
+ if err := json.Unmarshal(raw, &text); err != nil {
+ return nil, errors.Wrap(err, "failed to decode d1 text cell")
+ }
+ return text, nil
+ case '[':
+ // Blobs arrive as arrays of byte values.
+ var numbers []json.Number
+ if err := decodeJSON(raw, &numbers); err != nil {
+ return nil, errors.Wrap(err, "failed to decode d1 blob cell")
+ }
+ bytesValue := make([]byte, 0, len(numbers))
+ for _, number := range numbers {
+ b, err := number.Int64()
+ if err != nil || b < 0 || b > 255 {
+ return nil, errors.Errorf("d1: invalid blob byte %q", number)
+ }
+ bytesValue = append(bytesValue, byte(b))
+ }
+ return bytesValue, nil
+ case '{':
+ return string(raw), nil
+ default:
+ number := json.Number(bytes.TrimSpace(raw))
+ if integer, err := number.Int64(); err == nil {
+ return integer, nil
+ }
+ float, err := number.Float64()
+ if err != nil {
+ return nil, errors.Wrapf(err, "d1: invalid numeric cell %q", string(raw))
+ }
+ return float, nil
+ }
+}
+
+// encodeArgs converts Go arguments to the JSON parameter list D1 accepts.
+// Binary values are rejected: blob columns are written with unhex(?) and a
+// hex-encoded string because JSON cannot carry raw bytes.
+func encodeArgs(args []any) ([]any, error) {
+ if len(args) > maxBoundParameters {
+ return nil, errors.Errorf("d1: statement binds %d parameters, limit is %d", len(args), maxBoundParameters)
+ }
+ params := make([]any, 0, len(args))
+ for _, arg := range args {
+ value, err := driver.DefaultParameterConverter.ConvertValue(arg)
+ if err != nil {
+ return nil, errors.Wrap(err, "d1: unsupported parameter")
+ }
+ switch v := value.(type) {
+ case nil, int64, float64, string:
+ params = append(params, v)
+ case bool:
+ if v {
+ params = append(params, int64(1))
+ } else {
+ params = append(params, int64(0))
+ }
+ case []byte:
+ return nil, errors.New("d1: binary parameters are not supported; bind hex text through unhex(?)")
+ case time.Time:
+ params = append(params, v.UTC().Format(time.RFC3339Nano))
+ default:
+ return nil, errors.Errorf("d1: unsupported parameter type %T", value)
+ }
+ }
+ return params, nil
+}
diff --git a/store/test/README.md b/store/test/README.md
index 48f16825f..e8c811f56 100644
--- a/store/test/README.md
+++ b/store/test/README.md
@@ -14,14 +14,17 @@ DRIVER=mysql DSN=root@/memos_test go test -v ./test/store/...
## How to test store with Cloudflare D1?
-`DRIVER=d1` runs the suite against an in-process emulation of the D1 REST API
-(`store/db/d1/d1test`), so no Cloudflare account is needed:
+`DRIVER=d1` runs the suite against an in-process emulation of D1
+(`store/db/d1/d1test`), so no Cloudflare account is needed. It speaks the REST
+protocol by default and the bridge protocol with `D1_ACCESS=bridge`:
```go
DRIVER=d1 go test -v ./store/test/...
+DRIVER=d1 D1_ACCESS=bridge go test -v ./store/test/...
```
To run against a real database instead, set `D1_DSN` to
-`d1://<account_id>/<database_id>?token=<api_token>`. That database is shared
+`d1://<account_id>/<database_id>?token=<api_token>` or to a
+`d1-bridge://<worker-host>/<path>?token=<secret>` bridge endpoint. That database is shared
by every test and is not reset between them, so run one test at a time with
`-run` and expect to clean it up afterwards.
diff --git a/store/test/main_test.go b/store/test/main_test.go
index 444c8c59d..10df1693e 100644
--- a/store/test/main_test.go
+++ b/store/test/main_test.go
@@ -24,26 +24,37 @@ func TestMain(m *testing.M) {
}
func runAllDrivers() {
- drivers := []string{"sqlite", "mysql", "postgres", "d1"}
+ // Each run names a driver and the extra environment it needs; D1 runs
+ // once per access mode.
+ runs := []struct {
+ label string
+ env []string
+ }{
+ {"sqlite", []string{"DRIVER=sqlite"}},
+ {"mysql", []string{"DRIVER=mysql"}},
+ {"postgres", []string{"DRIVER=postgres"}},
+ {"d1 (rest)", []string{"DRIVER=d1", "D1_ACCESS=rest"}},
+ {"d1 (bridge)", []string{"DRIVER=d1", "D1_ACCESS=bridge"}},
+ }
_, currentFile, _, _ := runtime.Caller(0)
projectRoot := filepath.Dir(filepath.Dir(filepath.Dir(currentFile)))
var failed []string
- for _, driver := range drivers {
- fmt.Printf("\n==================== %s ====================\n\n", driver)
+ for _, run := range runs {
+ fmt.Printf("\n==================== %s ====================\n\n", run.label)
cmd := exec.Command("go", "test", "-v", "-count=1", "./store/test/...")
cmd.Dir = projectRoot
// A D1_DSN names one shared database; the loop's parallel tests need
// the per-test emulator instead, so it is dropped from the child.
env := slices.DeleteFunc(os.Environ(), func(entry string) bool { return strings.HasPrefix(entry, "D1_DSN=") })
- env = append(env, "DRIVER="+driver)
+ env = append(env, run.env...)
cmd.Env = env
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
if err := cmd.Run(); err != nil {
- failed = append(failed, driver)
+ failed = append(failed, run.label)
}
}
diff --git a/store/test/store.go b/store/test/store.go
index 5b9a973a1..7e99b4800 100644
--- a/store/test/store.go
+++ b/store/test/store.go
@@ -23,9 +23,10 @@ import (
// Each test gets its own isolated database:
// - SQLite: new temp file per test
// - MySQL/PostgreSQL: new database per test in shared container
-// - D1: new emulated database per test (see store/db/d1/d1test), or the
-// database named by D1_DSN when set; that database is shared, so run one
-// test at a time against it
+// - D1: new emulated database per test (see store/db/d1/d1test), reached
+// over the REST protocol or, with D1_ACCESS=bridge, the bridge protocol;
+// or the database named by D1_DSN when set. That database is shared, so
+// run one test at a time against it
func NewTestingStore(ctx context.Context, t *testing.T) *store.Store {
driver := getDriverFromEnv()
profile := getTestingProfileForDriver(t, driver)
@@ -112,7 +113,12 @@ func getTestingProfileForDriver(t *testing.T, driver string) *profile.Profile {
case "d1":
dsn = os.Getenv("D1_DSN")
if dsn == "" {
- dsn = d1test.New(t).DSN()
+ server := d1test.New(t)
+ if os.Getenv("D1_ACCESS") == "bridge" {
+ dsn = server.BridgeDSN()
+ } else {
+ dsn = server.DSN()
+ }
}
default:
t.Fatalf("unsupported driver: %s", driver)