Skip to content

armgit / memos

Public Imported from GitHub
0

Memos is an open-source, self-hosted note-taking app that brings your everyday thoughts together in a personal timeline. Your data stays yours, and you choose what to keep private or share.

fix(d1): close guard gaps, chunk-free list bindings, and statement layout

bluedbird9ca12926c7c1
diff --git a/docs/design/cloudflare-d1.md b/docs/design/cloudflare-d1.md
index fcb2cf260..4d41ea049 100644
--- a/docs/design/cloudflare-d1.md
+++ b/docs/design/cloudflare-d1.md
@@ -174,7 +174,7 @@ async function runBatch(env, request) {
const { statements } = await request.json();
const prepared = statements.map((s) => env.DB.prepare(s.sql).bind(...(s.params ?? [])));
try {
- const results = (await env.DB.batch(prepared)).map(({ results, meta }) => {
+ const results = (await env.DB.batch(prepared)).map(({ results = [], meta }) => {
const columns = results.length ? Object.keys(results[0]) : [];
return {
columns,
@@ -230,11 +230,22 @@ statements run in one transaction. Every driver method is shaped around them.
batch, and the driver maps it to the conflict error the caller expects.
`d1_guard` never holds rows.
-A batch cannot read its own writes, so an id generated by an `INSERT` cannot
-feed a later statement. Methods that need it use the unique natural key
-instead: a membership created with a new space references
-`(SELECT id FROM space WHERE uid = ?)`, and the space is read back by uid
-after the commit.
+A batch runs its statements in order inside one transaction, so a later
+statement sees the rows an earlier one inserted, but the generated ids are
+only reported after the commit. Methods that need a new id inside the batch
+use the unique natural key instead: a membership created with a new space
+references `(SELECT id FROM space WHERE uid = ?)`, and the space is read back
+by uid after the commit. Statements that return generated values through
+`RETURNING` get them from the batch result, so a policy-checked insert can
+sit behind its guards and still report its id.
+
+Guards re-assert every fact a validation read established, not only the
+row's existence: a memo's creator, lifecycle, audience, and placement, the
+actor's memberships, and, where the caller computed the mutation from the
+memo text, the content itself. A set of rows read before the batch, such as
+the attachments a memo deletion returns for storage cleanup, is guarded on
+its count together with its highest id, because ids only grow and a row
+swapped for a newer one keeps the count equal.
## Schema and migrations
@@ -253,8 +264,27 @@ D1 scripts follow one layout rule: **one statement per paragraph**. A blank
line separates statements, and nothing inside a statement, including a table
or trigger body, a string literal, or a comment, may contain a blank line.
The bridge transport divides scripts at blank lines to send them as a batch,
-and the test emulator executes them the same way, so a script that breaks the
-rule fails the store suite's fresh install rather than a deployment.
+and D1 refuses a bridge entry that holds more than one statement. The test
+emulator refuses it too, judging statement boundaries with SQLite's own
+`sqlite3_complete`, so a paragraph with two statements fails the store
+suite's bridge run rather than a deployment.
+
+## Operational limits
+
+The emulator enforces the bind limit and the one-statement rule, but not the
+service's size and rate limits, which a deployment must keep in mind:
+
+- A bound string or blob may be at most 2 MB. Blobs travel as hex text, so an
+ attachment stored in the database is limited to about 1 MB; larger
+ attachments belong in local or S3 storage.
+- A batch may hold on the order of a thousand statements. Deleting a user or
+ a space removes its memos in chunks of ninety, so an account with many
+ thousands of memos exceeds the batch and must be trimmed first.
+- The REST API is subject to Cloudflare's account-wide rate limit of 1200
+ requests per five minutes, after which every request is refused with a 429
+ for the rest of the window. The driver does not retry a 429, since an
+ immediate retry only extends the block. The bridge is a Worker request and
+ is not subject to that limit.
## Filter dialect
@@ -268,7 +298,11 @@ patterns with optional anchors.
`store/db/d1/d1test` emulates D1 in-process on a private SQLite database and
speaks both protocols: the REST API endpoints and the bridge endpoint, with
the same implicit transaction per request, atomic batches, foreign-key
-enforcement, and 100-parameter limit, and no custom functions registered.
+enforcement, 100-parameter limit, and one-statement rule for prepared
+entries, and no custom functions registered. Driver tests in
+`store/db/d1/race_test.go` interpose on the transport to change the database
+between a method's validation reads and its batch, which exercises the guards
+the way a concurrent writer would.
`DRIVER=d1` runs the whole store suite against it over REST, and
`D1_ACCESS=bridge` runs it over the bridge protocol; the all-drivers loop
runs both. Setting `D1_DSN` points the suite at a real
diff --git a/go.mod b/go.mod
index 60316fef0..53bdde078 100644
--- a/go.mod
+++ b/go.mod
@@ -46,6 +46,7 @@ require (
google.golang.org/grpc v1.83.0
google.golang.org/protobuf v1.36.12
gopkg.in/yaml.v3 v3.0.1
+ modernc.org/libc v1.75.3
modernc.org/sqlite v1.56.0
)
@@ -148,7 +149,6 @@ require (
golang.org/x/time v0.15.0 // indirect
golang.org/x/tools v0.49.0 // indirect
google.golang.org/api v0.293.0 // indirect
- modernc.org/libc v1.75.3 // indirect
modernc.org/mathutil v1.7.1 // indirect
modernc.org/memory v1.12.0 // indirect
)
diff --git a/store/db/d1/attachment.go b/store/db/d1/attachment.go
index 71ab852f3..e9349f63b 100644
--- a/store/db/d1/attachment.go
+++ b/store/db/d1/attachment.go
@@ -31,19 +31,23 @@ func (d *DB) CreateAttachment(ctx context.Context, create *store.Attachment) (*s
if create.MemoID == nil || create.CreatorID != create.Policy.ActorUserID {
return nil, store.ErrMemoPermissionDenied
}
- if err := validateMemoWritePolicy(ctx, d.db, *create.MemoID, create.Policy, nil); err != nil {
+ state, err := validateMemoWritePolicy(ctx, d.db, *create.MemoID, create.Policy, nil)
+ if err != nil {
return nil, err
}
- // Without a transaction the validated memo can vanish before the insert;
- // selecting the values from the memo row makes the insert conditional on
- // its continued existence, so a vanished memo yields no row instead of an
- // orphaned attachment.
- args = append(args, *create.MemoID)
- stmt := "INSERT INTO attachment (" + strings.Join(columns, ", ") + ") SELECT " + strings.Join(values, ", ") + " FROM memo WHERE memo.id = ? RETURNING id, created_ts, updated_ts"
- if err := d.db.QueryRowContext(ctx, stmt, args...).Scan(&create.ID, &create.CreatedTs, &create.UpdatedTs); err != nil {
- if errors.Is(err, sql.ErrNoRows) {
- return nil, store.ErrMemoMutationConflict
- }
+ // The guards re-assert the validated memo row and the actor's membership
+ // at commit time, so a memo that vanished, was archived, or moved to a
+ // Space the actor left yields a conflict instead of an orphaned or
+ // unauthorized attachment.
+ b := newBatch()
+ b.guard(activeUserCondition, create.Policy.ActorUserID)
+ memoGuardWritePolicy(b, state, create.Policy, nil)
+ index := b.add("INSERT INTO attachment ("+strings.Join(columns, ", ")+") VALUES ("+strings.Join(values, ", ")+") RETURNING id, created_ts, updated_ts", args...)
+ results, err := b.commit(ctx, d)
+ if err != nil {
+ return nil, guardError(err, store.ErrMemoMutationConflict)
+ }
+ if err := scanResultRow(results[index], &create.ID, &create.CreatedTs, &create.UpdatedTs); err != nil {
return nil, err
}
return create, nil
@@ -98,8 +102,11 @@ func (d *DB) ListAttachments(ctx context.Context, find *store.FindAttachment) ([
where, args = append(where, "attachment.memo_id = ?"), append(args, *v)
}
if len(find.MemoIDList) > 0 {
- clause, memoArgs := inClause(find.MemoIDList)
- where, args = append(where, "attachment.memo_id IN "+clause), append(args, memoArgs...)
+ clause, memoArg, err := jsonList(find.MemoIDList)
+ if err != nil {
+ return nil, err
+ }
+ where, args = append(where, "attachment.memo_id IN "+clause), append(args, memoArg)
}
if find.HasRelatedMemo {
where = append(where, "attachment.memo_id IS NOT NULL")
@@ -238,12 +245,12 @@ func (d *DB) UpdateAttachment(ctx context.Context, update *store.UpdateAttachmen
if err != nil {
return err
}
- if err := authorizeAttachmentMutation(ctx, d.db, update.Policy.ActorUserID, memoIDs, nil); err != nil {
+ states, err := authorizeAttachmentMutation(ctx, d.db, update.Policy.ActorUserID, memoIDs, nil)
+ if err != nil {
return err
}
- // The actor was validated above; the guard re-asserts it at commit time.
b := newBatch()
- b.guard(activeUserCondition, update.Policy.ActorUserID)
+ guardAttachmentMutation(b, update.Policy.ActorUserID, states, attachments, nil)
b.add(stmt, args...)
if _, err := b.commit(ctx, d); err != nil {
return errors.Wrap(guardError(err, store.ErrMemoPermissionDenied), "failed to update attachment")
diff --git a/store/db/d1/attachment_delete.go b/store/db/d1/attachment_delete.go
index 348bcdb00..2c93f3d73 100644
--- a/store/db/d1/attachment_delete.go
+++ b/store/db/d1/attachment_delete.go
@@ -9,10 +9,10 @@ import (
)
// DeleteAttachmentsWithPolicy authorizes the actor against every linked memo
-// and then deletes the rows in one atomic batch. Each delete is preceded by a
-// guard on the row's existence so a row that vanished after the validation
-// reads aborts the whole batch as a conflict, mirroring the per-row
-// RowsAffected check a transaction would perform.
+// and then deletes the rows in one atomic batch. The batch re-asserts the
+// memo states and bindings the authorization read, including the memo
+// content the caller expects, so a row or memo that changed after the
+// validation reads aborts the whole batch as a conflict.
func (d *DB) DeleteAttachmentsWithPolicy(ctx context.Context, policy *store.AttachmentDeletionPolicy, attachmentIDs []int32) error {
if policy == nil || policy.ActorUserID <= 0 {
return store.ErrMemoPermissionDenied
@@ -28,15 +28,15 @@ func (d *DB) DeleteAttachmentsWithPolicy(ctx context.Context, policy *store.Atta
if err := store.ValidateAttachmentDeletionMemoSnapshots(memoIDs, policy.ExpectedMemoContents); err != nil {
return err
}
- if err := authorizeAttachmentMutation(ctx, d.db, policy.ActorUserID, memoIDs, policy.ExpectedMemoContents); err != nil {
+ states, err := authorizeAttachmentMutation(ctx, d.db, policy.ActorUserID, memoIDs, policy.ExpectedMemoContents)
+ if err != nil {
return err
}
b := newBatch()
- b.guard(activeUserCondition, policy.ActorUserID)
- for _, attachmentID := range attachmentIDs {
- b.guard("EXISTS (SELECT 1 FROM attachment WHERE id = ?)", attachmentID)
- b.add("DELETE FROM attachment WHERE id = ?", attachmentID)
+ guardAttachmentMutation(b, policy.ActorUserID, states, attachments, policy.ExpectedMemoContents)
+ for _, attachment := range attachments {
+ b.add("DELETE FROM attachment WHERE id = ?", attachment.ID)
}
if _, err := b.commit(ctx, d); err != nil {
return guardError(err, store.ErrMemoMutationConflict)
diff --git a/store/db/d1/d1_test.go b/store/db/d1/d1_test.go
index 0c687077a..b8eb8cb44 100644
--- a/store/db/d1/d1_test.go
+++ b/store/db/d1/d1_test.go
@@ -5,6 +5,8 @@ import (
"errors"
"net/http"
"net/http/httptest"
+ "os"
+ "path/filepath"
"strings"
"testing"
@@ -211,6 +213,77 @@ func runTransportRoundTrip(t *testing.T, db *DB) {
require.Greater(t, size, int64(0))
}
+// TestSchemaScriptsFollowTheParagraphRule applies the real schema file over
+// both transports. The bridge sends one paragraph per prepared statement and
+// the emulator, like D1, refuses a prepared statement holding two, so a
+// paragraph with two statements fails here rather than on a deployment.
+func TestSchemaScriptsFollowTheParagraphRule(t *testing.T) {
+ schema, err := os.ReadFile(filepath.Join("..", "..", "migration", "d1", "LATEST.sql"))
+ require.NoError(t, err)
+ for mode, dsn := range accessModes {
+ t.Run(mode, func(t *testing.T) {
+ db := newTestDB(t, dsn)
+ ctx := context.Background()
+ tx, err := db.GetDB().Begin()
+ require.NoError(t, err)
+ _, err = tx.ExecContext(ctx, string(schema))
+ require.NoError(t, err)
+ require.NoError(t, tx.Commit())
+ initialized, err := db.IsInitialized(ctx)
+ require.NoError(t, err)
+ require.True(t, initialized)
+ })
+ }
+}
+
+// TestTransactionJoinsScriptsAtParagraphs buffers two scripts in one
+// transaction, as the migrator does when several migration files apply, and
+// checks they commit over both transports.
+func TestTransactionJoinsScriptsAtParagraphs(t *testing.T) {
+ for mode, dsn := range accessModes {
+ t.Run(mode, func(t *testing.T) {
+ db := newTestDB(t, dsn)
+ ctx := context.Background()
+ tx, err := db.GetDB().Begin()
+ require.NoError(t, err)
+ _, err = tx.ExecContext(ctx, "CREATE TABLE a (id INTEGER);\n")
+ require.NoError(t, err)
+ _, err = tx.ExecContext(ctx, "-- second file\nCREATE TABLE b (id INTEGER);\n\nCREATE INDEX b_id ON b(id);\n")
+ require.NoError(t, err)
+ require.NoError(t, tx.Commit())
+ var count int
+ require.NoError(t, db.GetDB().QueryRowContext(ctx, "SELECT COUNT(*) FROM sqlite_master WHERE name IN ('a', 'b', 'b_id')").Scan(&count))
+ require.Equal(t, 3, count)
+ })
+ }
+}
+
+// TestPreparedEntryHoldsOneStatement pins the emulator to D1's rule: a
+// bridge entry and a parameterized REST entry are prepared statements and
+// hold exactly one statement, while a parameter-free REST entry is a script.
+func TestPreparedEntryHoldsOneStatement(t *testing.T) {
+ ctx := context.Background()
+ script := "CREATE TABLE a (id INTEGER);\nCREATE TABLE b (id INTEGER);"
+
+ bridge := newTestDB(t, accessModes["bridge"])
+ _, err := bridge.execOne(ctx, script)
+ require.ErrorContains(t, err, "only one statement")
+ _, err = bridge.execOne(ctx, "CREATE TRIGGER t AFTER INSERT ON a BEGIN UPDATE a SET id = 1; UPDATE a SET id = 2; END;")
+ require.ErrorContains(t, err, "no such table", "a trigger body is one statement and reaches the database")
+
+ rest := newTestDB(t, accessModes["rest"])
+ require.NoError(t, rest.transport.script(ctx, script), "a parameter-free REST entry is a script")
+ _, err = rest.execOne(ctx, "INSERT INTO a (id) VALUES (?);\nINSERT INTO b (id) VALUES (?);", 1, 2)
+ require.ErrorContains(t, err, "only one statement")
+}
+
+func TestRateLimitIsNotRetried(t *testing.T) {
+ require.False(t, isRetryable(&Error{Status: http.StatusTooManyRequests, Message: "rate limited"}))
+ require.True(t, isRetryable(&Error{Status: http.StatusServiceUnavailable, Message: "unavailable"}))
+ require.True(t, isRetryable(&Error{Status: http.StatusBadRequest, Message: "D1_ERROR: database is locked: SQLITE_BUSY"}))
+ require.False(t, isRetryable(&Error{Status: http.StatusBadRequest, Message: "D1_ERROR: UNIQUE constraint failed: memo.uid"}))
+}
+
func TestNullAndTypedScans(t *testing.T) {
for mode, dsn := range accessModes {
t.Run(mode, func(t *testing.T) {
diff --git a/store/db/d1/d1test/server.go b/store/db/d1/d1test/server.go
index 6e6f4d457..d0221b1b9 100644
--- a/store/db/d1/d1test/server.go
+++ b/store/db/d1/d1test/server.go
@@ -3,9 +3,11 @@
// and /query endpoints, batch bodies, the database metadata endpoint) and the
// Memos bridge protocol, on top of a private SQLite database. It enforces the
// D1 constraints that matter to the driver: foreign keys are on, statements
-// are wrapped in an implicit transaction, a batch is atomic, and a statement
-// may bind at most 100 parameters. It registers no custom SQL functions,
-// exactly like D1.
+// are wrapped in an implicit transaction, a batch is atomic, a statement may
+// bind at most 100 parameters, and a prepared entry (a bridge statement or a
+// parameterized REST entry) holds exactly one statement while a parameter-free
+// REST entry is a script. It registers no custom SQL functions, exactly like
+// D1.
package d1test
import (
@@ -20,10 +22,10 @@ import (
"testing"
"time"
+ "modernc.org/libc"
// SQLite engine behind the emulated service.
_ "modernc.org/sqlite"
-
- "github.com/usememos/memos/store/db/d1/sqlsplit"
+ sqlite3 "modernc.org/sqlite/lib"
)
const (
@@ -183,23 +185,28 @@ func (s *Server) handleQuery(w http.ResponseWriter, r *http.Request, rawRows boo
return
}
- // A parameterized entry is one statement; a parameter-free entry is a
- // script, executed statement by statement the way D1 does.
+ // A parameterized entry is prepared and must be one statement; a
+ // parameter-free entry is a script, executed statement by statement the
+ // way D1 does.
var stmts []requestStatement
for _, entry := range entries {
if len(entry.Params) > maxBoundParameters {
writeErrors(w, http.StatusBadRequest, 7400, fmt.Sprintf("too many bound parameters: %d", len(entry.Params)))
return
}
- if len(entry.Params) > 0 {
- stmts = append(stmts, entry)
- continue
- }
- parts := sqlsplit.Split(entry.SQL)
+ parts := splitStatements(entry.SQL)
if len(parts) == 0 {
writeErrors(w, http.StatusBadRequest, 7400, "empty sql")
return
}
+ if len(entry.Params) > 0 {
+ if len(parts) > 1 {
+ writeErrors(w, http.StatusBadRequest, 7500, errMultipleStatements)
+ return
+ }
+ stmts = append(stmts, entry)
+ continue
+ }
for _, part := range parts {
stmts = append(stmts, requestStatement{SQL: part})
}
@@ -261,6 +268,12 @@ func (s *Server) handleBridge(w http.ResponseWriter, r *http.Request) {
writeBridgeError(w, http.StatusBadRequest, fmt.Sprintf("too many bound parameters: %d", len(entry.Params)))
return
}
+ // Every bridge entry goes through prepare(), which takes exactly one
+ // statement.
+ if len(splitStatements(entry.SQL)) != 1 {
+ writeBridgeError(w, http.StatusBadRequest, "D1_ERROR: "+errMultipleStatements)
+ return
+ }
}
results, err := s.execute(r.Context(), body.Statements)
if err != nil {
@@ -375,6 +388,70 @@ func collectRows(ctx context.Context, tx *sql.Tx, query string, args []any) (*st
return res, nil
}
+// errMultipleStatements is the text D1 returns for a prepared statement that
+// holds more than one statement.
+const errMultipleStatements = "A prepared SQL statement must contain only one statement."
+
+// splitStatements divides a script into statements the way SQLite's own
+// parser would: sqlite3_complete reports where each statement ends, so
+// string literals, comments, and trigger bodies never split a statement, and
+// text that is only whitespace or comments yields nothing.
+func splitStatements(text string) []string {
+ statements := []string{}
+ start := 0
+ for i := 0; i < len(text); i++ {
+ if text[i] != ';' || !sqlComplete(text[start:i+1]) {
+ continue
+ }
+ if part := strings.TrimSpace(text[start : i+1]); !sqlBlank(part) {
+ statements = append(statements, part)
+ }
+ start = i + 1
+ }
+ if tail := strings.TrimSpace(text[start:]); !sqlBlank(tail) {
+ statements = append(statements, tail)
+ }
+ return statements
+}
+
+// sqlComplete reports whether text ends with a complete SQL statement.
+func sqlComplete(text string) bool {
+ tls := libc.NewTLS()
+ defer tls.Close()
+ cText, err := libc.CString(text)
+ if err != nil {
+ return false
+ }
+ defer libc.Xfree(tls, cText)
+ return sqlite3.Xsqlite3_complete(tls, cText) != 0
+}
+
+// sqlBlank reports whether text holds no SQL: only whitespace, comments, and
+// empty statements.
+func sqlBlank(text string) bool {
+ for {
+ text = strings.TrimLeft(text, " \t\r\n;")
+ switch {
+ case text == "":
+ return true
+ case strings.HasPrefix(text, "--"):
+ if _, rest, found := strings.Cut(text, "\n"); found {
+ text = rest
+ } else {
+ return true
+ }
+ case strings.HasPrefix(text, "/*"):
+ if _, rest, found := strings.Cut(text[2:], "*/"); found {
+ text = rest
+ } else {
+ return true
+ }
+ default:
+ return false
+ }
+ }
+}
+
func decodeBody(r *http.Request, target any) error {
decoder := json.NewDecoder(r.Body)
decoder.UseNumber()
diff --git a/store/db/d1/errors.go b/store/db/d1/errors.go
index 27e8db1d4..070b5fbac 100644
--- a/store/db/d1/errors.go
+++ b/store/db/d1/errors.go
@@ -19,14 +19,16 @@ func isUniqueViolation(err error) bool {
strings.Contains(message, "SQLITE_CONSTRAINT_PRIMARYKEY")
}
-// isRetryable reports whether err is a transient D1 failure worth retrying:
-// rate limiting, a busy or overloaded database, or a server-side error.
+// isRetryable reports whether err is a transient D1 failure worth retrying
+// right away: a busy or overloaded database, or a server-side error. A 429
+// is deliberately not retried; the Cloudflare API rate limit that produces it
+// blocks the caller for minutes, so an immediate retry only deepens the block.
func isRetryable(err error) bool {
var d1Err *Error
if !errors.As(err, &d1Err) {
return false
}
- if d1Err.Status == http.StatusTooManyRequests || d1Err.Status >= http.StatusInternalServerError {
+ if d1Err.Status >= http.StatusInternalServerError {
return true
}
message := d1Err.Message
diff --git a/store/db/d1/executor.go b/store/db/d1/executor.go
index 87842fed2..a385eec03 100644
--- a/store/db/d1/executor.go
+++ b/store/db/d1/executor.go
@@ -3,6 +3,7 @@ package d1
import (
"context"
"database/sql"
+ "encoding/json"
"fmt"
"strconv"
"strings"
@@ -75,6 +76,68 @@ func inClause[T any](values []T) (string, []any) {
return "(" + placeholders(len(values)) + ")", args
}
+// jsonList renders a membership test whose values travel as one JSON array
+// binding, so a list of any length costs a single bound parameter and the
+// statement keeps its LIMIT and ORDER BY semantics.
+func jsonList[T any](values []T) (string, any, error) {
+ encoded, err := json.Marshal(values)
+ if err != nil {
+ return "", nil, errors.Wrap(err, "failed to encode list binding")
+ }
+ return "(SELECT value FROM json_each(?))", string(encoded), nil
+}
+
+// guardIDSet asserts that the rows selected by "FROM <from>" are still
+// exactly the ids read before the batch. The count alone would accept a row
+// deleted and another inserted in the window; ids only grow, so the highest
+// id catches the swap.
+func (b *batch) guardIDSet(from string, args []any, ids []int32) {
+ var maxID int32
+ for _, id := range ids {
+ maxID = max(maxID, id)
+ }
+ condition := "(SELECT COUNT(*) FROM " + from + ") = ? AND COALESCE((SELECT MAX(id) FROM " + from + "), 0) = ?"
+ guardArgs := append(append(append([]any{}, args...), len(ids)), append(append([]any{}, args...), maxID)...)
+ b.guard(condition, guardArgs...)
+}
+
+// scanResultRow copies the first row of a batch result into dest, for
+// statements that RETURNING generated values.
+func scanResultRow(res *result, dest ...any) error {
+ if res == nil || len(res.Rows) == 0 {
+ return errors.New("d1: statement returned no row")
+ }
+ row := res.Rows[0]
+ if len(row) != len(dest) {
+ return errors.Errorf("d1: statement returned %d columns for %d targets", len(row), len(dest))
+ }
+ for i, target := range dest {
+ switch t := target.(type) {
+ case *int32:
+ value, ok := row[i].(int64)
+ if !ok {
+ return errors.Errorf("d1: column %d is %T, not an integer", i, row[i])
+ }
+ *t = int32(value)
+ case *int64:
+ value, ok := row[i].(int64)
+ if !ok {
+ return errors.Errorf("d1: column %d is %T, not an integer", i, row[i])
+ }
+ *t = value
+ case *string:
+ value, ok := row[i].(string)
+ if !ok {
+ return errors.Errorf("d1: column %d is %T, not text", i, row[i])
+ }
+ *t = value
+ default:
+ return errors.Errorf("d1: unsupported scan target %T", target)
+ }
+ }
+ return nil
+}
+
// intList renders trusted integer ids as a literal IN list. Inlining them
// keeps long id lists under D1's 100 bound parameter limit.
func intList(ids []int32) string {
diff --git a/store/db/d1/memo.go b/store/db/d1/memo.go
index eabd262ea..03b2c4f11 100644
--- a/store/db/d1/memo.go
+++ b/store/db/d1/memo.go
@@ -25,28 +25,6 @@ type memoState struct {
content string
}
-// memoInlineUIDThreshold is the uid list length above which ListMemos resolves
-// uids to ids before querying, leaving bind capacity for the other filters.
-const memoInlineUIDThreshold = 40
-
-// memoResolveUIDs maps uids to memo ids in chunks that respect the bind limit.
-// Unknown uids are dropped; -1 keeps the IN list valid when none resolve.
-func memoResolveUIDs(ctx context.Context, q querier, uids []string) ([]int32, error) {
- ids := make([]int32, 0, len(uids))
- for _, batch := range chunk(uids, inClauseBatchSize) {
- clause, args := inClause(batch)
- resolved, err := listMemoIDs(ctx, q, "SELECT id FROM memo WHERE uid IN "+clause, args...)
- if err != nil {
- return nil, err
- }
- ids = append(ids, resolved...)
- }
- if len(ids) == 0 {
- ids = append(ids, -1)
- }
- return ids, nil
-}
-
// loadMemoState reads the memo row; a missing memo yields sql.ErrNoRows.
func loadMemoState(ctx context.Context, q querier, memoID int32) (*memoState, error) {
state := &memoState{}
@@ -165,7 +143,7 @@ func memoLoadCreated(ctx context.Context, q querier, create *store.Memo) error {
// ListMemos returns the memos matching find.
func (d *DB) ListMemos(ctx context.Context, find *store.FindMemo) ([]*store.Memo, error) {
- where, args, err := memoListConditions(ctx, d.db, find)
+ where, args, err := memoListConditions(ctx, find)
if err != nil {
return nil, err
}
@@ -218,9 +196,8 @@ func (d *DB) ListMemos(ctx context.Context, find *store.FindMemo) ([]*store.Memo
return list, nil
}
-// memoListConditions renders the WHERE clauses for find. q serves the uid
-// resolution a long uid list needs.
-func memoListConditions(ctx context.Context, q querier, find *store.FindMemo) ([]string, []any, error) {
+// memoListConditions renders the WHERE clauses for find.
+func memoListConditions(ctx context.Context, find *store.FindMemo) ([]string, []any, error) {
where, args := []string{"1 = 1"}, []any{}
engine, err := filter.DefaultEngine()
@@ -240,18 +217,11 @@ func memoListConditions(ctx context.Context, q querier, find *store.FindMemo) ([
where, args = append(where, "memo.uid = ?"), append(args, *v)
}
if len(find.UIDList) > 0 {
- // A long uid list would exhaust D1's bind limit together with the
- // other filters, so it is resolved to ids first and inlined.
- if len(find.UIDList) > memoInlineUIDThreshold {
- ids, err := memoResolveUIDs(ctx, q, find.UIDList)
- if err != nil {
- return nil, nil, err
- }
- where = append(where, "memo.id IN "+intList(ids))
- } else {
- clause, uidArgs := inClause(find.UIDList)
- where, args = append(where, "memo.uid IN "+clause), append(args, uidArgs...)
+ clause, uidArg, err := jsonList(find.UIDList)
+ if err != nil {
+ return nil, nil, err
}
+ where, args = append(where, "memo.uid IN "+clause), append(args, uidArg)
}
if v := find.CreatorID; v != nil {
where, args = append(where, "memo.creator_id = ?"), append(args, *v)
@@ -342,20 +312,12 @@ func memoScanRow(rows *sql.Rows, excludeContent bool) (*store.Memo, error) {
func (d *DB) UpdateMemo(ctx context.Context, update *store.UpdateMemo) error {
b := newBatch()
if update.Policy != nil {
- if err := validateMemoWritePolicy(ctx, d.db, update.ID, update.Policy, update); err != nil {
- return err
- }
- state, err := loadMemoState(ctx, d.db, update.ID)
+ state, err := validateMemoWritePolicy(ctx, d.db, update.ID, update.Policy, update)
if err != nil {
- if errors.Is(err, sql.ErrNoRows) {
- return store.ErrMemoMutationConflict
- }
- return errors.Wrap(err, "failed to read memo")
+ return err
}
b.guard(activeUserCondition, update.Policy.ActorUserID)
- condition, args := state.unchangedCondition()
- b.guard(condition, args...)
- memoGuardPolicyPlacement(b, state, update.Policy, update)
+ memoGuardWritePolicy(b, state, update.Policy, update)
}
if err := memoAddUpdate(b, update); err != nil {
return err
diff --git a/store/db/d1/memo_delete_policy.go b/store/db/d1/memo_delete_policy.go
index 88e52502e..0e59d3991 100644
--- a/store/db/d1/memo_delete_policy.go
+++ b/store/db/d1/memo_delete_policy.go
@@ -45,7 +45,7 @@ func (d *DB) DeleteMemoWithPolicy(ctx context.Context, delete *store.DeleteMemoW
// The attachments were listed before the batch and are returned for
// storage cleanup; abort if the bound set changed so none is orphaned in
// the database or missed by the caller.
- b.guard("(SELECT COUNT(*) FROM attachment WHERE memo_id = ?) = ?", delete.MemoID, len(attachments))
+ b.guardIDSet("attachment WHERE memo_id = ?", []any{delete.MemoID}, attachmentIDs(attachments))
addMemoSetDeletes(b, memoIDs, attachmentIDs(attachments))
if _, err := b.commit(ctx, d); err != nil {
return nil, guardError(err, errors.Wrap(store.ErrMemoMutationConflict, "memo changed while deleting"))
diff --git a/store/db/d1/memo_mutation.go b/store/db/d1/memo_mutation.go
index 6fc0d3ca1..17ed4061f 100644
--- a/store/db/d1/memo_mutation.go
+++ b/store/db/d1/memo_mutation.go
@@ -79,13 +79,16 @@ func (d *DB) planMemoMutation(ctx context.Context, mutation *store.MemoMutation)
if plan.policy == nil && mutation.MemoUpdate != nil {
plan.policy = mutation.MemoUpdate.Policy
}
+ var validated *memoState
if plan.policy != nil {
- if err := validateMemoWritePolicy(ctx, d.db, mutation.MemoID, plan.policy, mutation.MemoUpdate); err != nil {
+ state, err := validateMemoWritePolicy(ctx, d.db, mutation.MemoID, plan.policy, mutation.MemoUpdate)
+ if err != nil {
return nil, err
}
+ validated = state
}
if mutation.MemoCreate == nil {
- if err := plan.planExistingMemo(ctx, d.db); err != nil {
+ if err := plan.planExistingMemo(ctx, d.db, validated); err != nil {
return nil, err
}
}
@@ -135,14 +138,20 @@ func (p *memoMutationPlan) planCreate(ctx context.Context, q querier, create *st
}
// planExistingMemo checks the memo still matches what the caller computed
-// the mutation from.
-func (p *memoMutationPlan) planExistingMemo(ctx context.Context, q querier) error {
- state, err := loadMemoState(ctx, q, p.mutation.MemoID)
- if err != nil {
- if errors.Is(err, sql.ErrNoRows) {
- return errors.Wrap(store.ErrMemoMutationConflict, "memo no longer exists")
+// the mutation from. validated is the state the write policy was checked
+// against, when there was one; the guards are built from that same read so
+// nothing validated is re-read from a different moment.
+func (p *memoMutationPlan) planExistingMemo(ctx context.Context, q querier, validated *memoState) error {
+ state := validated
+ if state == nil {
+ loaded, err := loadMemoState(ctx, q, p.mutation.MemoID)
+ if err != nil {
+ if errors.Is(err, sql.ErrNoRows) {
+ return errors.Wrap(store.ErrMemoMutationConflict, "memo no longer exists")
+ }
+ return errors.Wrap(err, "failed to read memo")
}
- return errors.Wrap(err, "failed to read memo")
+ state = loaded
}
if state.creatorID != p.mutation.MemoCreatorID || state.content != p.mutation.ExpectedMemoContent {
return errors.Wrap(store.ErrMemoMutationConflict, "memo changed while applying mutation")
diff --git a/store/db/d1/memo_share.go b/store/db/d1/memo_share.go
index 16d2ed18a..a90b6254d 100644
--- a/store/db/d1/memo_share.go
+++ b/store/db/d1/memo_share.go
@@ -25,19 +25,22 @@ func (d *DB) CreateMemoShare(ctx context.Context, create *store.MemoShare) (*sto
}
return create, nil
}
- if err := validateMemoWritePolicy(ctx, d.db, create.MemoID, create.Policy, nil); err != nil {
+ state, err := validateMemoWritePolicy(ctx, d.db, create.MemoID, create.Policy, nil)
+ if err != nil {
return nil, err
}
- // The validated memo and actor are re-asserted by the insert itself:
- // selecting from the memo row and requiring the actor to still be active
- // makes a stale validation yield no row instead of a share.
- args = append(args, create.MemoID, create.Policy.ActorUserID)
- stmt := "INSERT INTO memo_share (" + strings.Join(columns, ", ") + ") SELECT " + strings.Join(values, ", ") +
- " FROM memo WHERE memo.id = ? AND " + activeUserCondition + " RETURNING id, created_ts"
- if err := d.db.QueryRowContext(ctx, stmt, args...).Scan(&create.ID, &create.CreatedTs); err != nil {
- if errors.Is(err, sql.ErrNoRows) {
- return nil, store.ErrMemoMutationConflict
- }
+ // The memo's audience, lifecycle, and placement were all part of the
+ // validation, so the batch re-asserts the whole row: a memo moved to the
+ // SPACE audience meanwhile must not gain a share.
+ b := newBatch()
+ b.guard(activeUserCondition, create.Policy.ActorUserID)
+ memoGuardWritePolicy(b, state, create.Policy, nil)
+ index := b.add("INSERT INTO memo_share ("+strings.Join(columns, ", ")+") VALUES ("+strings.Join(values, ", ")+") RETURNING id, created_ts", args...)
+ results, err := b.commit(ctx, d)
+ if err != nil {
+ return nil, guardError(err, store.ErrMemoMutationConflict)
+ }
+ if err := scanResultRow(results[index], &create.ID, &create.CreatedTs); err != nil {
return nil, err
}
return create, nil
@@ -131,7 +134,7 @@ func (d *DB) DeleteMemoShare(ctx context.Context, delete *store.DeleteMemoShare)
_, err := d.execOne(ctx, "DELETE FROM memo_share WHERE "+strings.Join(where, " AND "), args...)
return err
}
- if err := validateMemoWritePolicy(ctx, d.db, *delete.MemoID, delete.Policy, nil); err != nil {
+ if _, err := validateMemoWritePolicy(ctx, d.db, *delete.MemoID, delete.Policy, nil); err != nil {
return err
}
// One conditional statement: the share must still belong to the validated
diff --git a/store/db/d1/race_test.go b/store/db/d1/race_test.go
new file mode 100644
index 000000000..dcae1d0fd
--- /dev/null
+++ b/store/db/d1/race_test.go
@@ -0,0 +1,316 @@
+package d1
+
+import (
+ "context"
+ "os"
+ "path/filepath"
+ "strings"
+ "testing"
+
+ "github.com/stretchr/testify/require"
+
+ "github.com/usememos/memos/internal/profile"
+ "github.com/usememos/memos/store"
+ "github.com/usememos/memos/store/db/d1/d1test"
+)
+
+// raceTransport wraps a transport and runs action once, right before the
+// first request whose SQL contains trigger reaches the database. Driver
+// methods validate with reads and then commit a batch, so an action fired on
+// the batch changes the database in exactly the window a concurrent writer
+// would, and the guards must catch it.
+type raceTransport struct {
+ transport
+ trigger string
+ action func()
+ fired bool
+}
+
+func (t *raceTransport) exec(ctx context.Context, stmt statement) (*result, error) {
+ t.fire(stmt.SQL)
+ return t.transport.exec(ctx, stmt)
+}
+
+func (t *raceTransport) batch(ctx context.Context, stmts []statement) ([]*result, error) {
+ sqls := make([]string, 0, len(stmts))
+ for _, stmt := range stmts {
+ sqls = append(sqls, stmt.SQL)
+ }
+ t.fire(strings.Join(sqls, "\n"))
+ return t.transport.batch(ctx, stmts)
+}
+
+func (t *raceTransport) fire(sql string) {
+ if t.fired || !strings.Contains(sql, t.trigger) {
+ return
+ }
+ t.fired = true
+ t.action()
+}
+
+// raceDB is a schema-initialized driver whose transport can be interposed,
+// plus a second, independent handle on the same database for the
+// interposing action to use.
+type raceDB struct {
+ *DB
+ other *DB
+ ctx context.Context
+}
+
+func newRaceDB(t *testing.T, dsn func(*d1test.Server) string) *raceDB {
+ t.Helper()
+ server := d1test.New(t)
+ open := func() *DB {
+ driver, err := NewDB(&profile.Profile{Driver: "d1", DSN: dsn(server)})
+ require.NoError(t, err)
+ db, ok := driver.(*DB)
+ require.True(t, ok)
+ t.Cleanup(func() { require.NoError(t, db.Close()) })
+ return db
+ }
+ db := &raceDB{DB: open(), other: open(), ctx: context.Background()}
+
+ schema, err := os.ReadFile(filepath.Join("..", "..", "migration", "d1", "LATEST.sql"))
+ require.NoError(t, err)
+ tx, err := db.GetDB().Begin()
+ require.NoError(t, err)
+ _, err = tx.ExecContext(db.ctx, string(schema))
+ require.NoError(t, err)
+ require.NoError(t, tx.Commit())
+ return db
+}
+
+// interpose installs action to run before the first request containing
+// trigger. Requests made by other bypass it.
+func (db *raceDB) interpose(t *testing.T, trigger string, action func()) {
+ t.Helper()
+ require.NoError(t, db.db.Close())
+ wrapped := &raceTransport{transport: db.transport, trigger: trigger, action: action}
+ db.transport = wrapped
+ db.db = openSQLDB(wrapped)
+ t.Cleanup(func() { require.True(t, wrapped.fired, "the interposed action never ran") })
+}
+
+func (db *raceDB) user(t *testing.T, username string) *store.User {
+ t.Helper()
+ user, err := db.other.CreateUser(db.ctx, &store.User{Username: username, Role: store.RoleUser, PasswordHash: "hash"})
+ require.NoError(t, err)
+ return user
+}
+
+func (db *raceDB) memo(t *testing.T, uid string, creator *store.User) *store.Memo {
+ t.Helper()
+ memo, err := db.other.CreateMemo(db.ctx, &store.Memo{UID: uid, CreatorID: creator.ID, Content: "original", Visibility: store.Private})
+ require.NoError(t, err)
+ return memo
+}
+
+func (db *raceDB) attachment(t *testing.T, uid string, creator *store.User, memo *store.Memo) *store.Attachment {
+ t.Helper()
+ attachment, err := db.other.CreateAttachment(db.ctx, &store.Attachment{UID: uid, CreatorID: creator.ID, Filename: uid + ".png", Type: "image/png", MemoID: &memo.ID})
+ require.NoError(t, err)
+ return attachment
+}
+
+func (db *raceDB) archive(t *testing.T, memo *store.Memo) {
+ t.Helper()
+ archived := store.Archived
+ require.NoError(t, db.other.UpdateMemo(db.ctx, &store.UpdateMemo{ID: memo.ID, RowStatus: &archived}))
+}
+
+func (db *raceDB) memoContent(t *testing.T, memo *store.Memo) string {
+ t.Helper()
+ var content string
+ require.NoError(t, db.other.GetDB().QueryRowContext(db.ctx, "SELECT content FROM memo WHERE id = ?", memo.ID).Scan(&content))
+ return content
+}
+
+func (db *raceDB) count(t *testing.T, query string, args ...any) int {
+ t.Helper()
+ var n int
+ require.NoError(t, db.other.GetDB().QueryRowContext(db.ctx, query, args...).Scan(&n))
+ return n
+}
+
+func forEachAccessMode(t *testing.T, run func(t *testing.T, db *raceDB)) {
+ t.Helper()
+ for mode, dsn := range accessModes {
+ t.Run(mode, func(t *testing.T) {
+ run(t, newRaceDB(t, dsn))
+ })
+ }
+}
+
+func TestPolicyInsertsReportGeneratedValues(t *testing.T) {
+ forEachAccessMode(t, func(t *testing.T, db *raceDB) {
+ user := db.user(t, "alice")
+ memo := db.memo(t, "m1", user)
+ policy := &store.MemoWritePolicy{ActorUserID: user.ID, CreatingShare: true}
+
+ share, err := db.CreateMemoShare(db.ctx, &store.MemoShare{UID: "s1", MemoID: memo.ID, CreatorID: user.ID, Policy: policy})
+ require.NoError(t, err)
+ require.NotZero(t, share.ID)
+ require.NotZero(t, share.CreatedTs)
+
+ attachment, err := db.CreateAttachment(db.ctx, &store.Attachment{UID: "a1", CreatorID: user.ID, MemoID: &memo.ID, Policy: &store.MemoWritePolicy{ActorUserID: user.ID}})
+ require.NoError(t, err)
+ require.NotZero(t, attachment.ID)
+ require.NotZero(t, attachment.CreatedTs)
+ require.NotZero(t, attachment.UpdatedTs)
+ })
+}
+
+func TestShareCreateAbortsWhenMemoJoinsSpaceAudience(t *testing.T) {
+ forEachAccessMode(t, func(t *testing.T, db *raceDB) {
+ user := db.user(t, "alice")
+ space, err := db.other.CreateSpace(db.ctx, &store.Space{UID: "space", Title: "Space"}, user.ID)
+ require.NoError(t, err)
+ memo := db.memo(t, "m1", user)
+
+ // The memo is moved to the SPACE audience after the share was
+ // validated against a private memo; the insert must not land.
+ db.interpose(t, "INSERT INTO memo_share", func() {
+ visibility := store.SpaceAudience
+ require.NoError(t, db.other.UpdateMemo(db.ctx, &store.UpdateMemo{ID: memo.ID, Visibility: &visibility, SpaceID: &space.ID}))
+ })
+ _, err = db.CreateMemoShare(db.ctx, &store.MemoShare{UID: "s1", MemoID: memo.ID, CreatorID: user.ID, Policy: &store.MemoWritePolicy{ActorUserID: user.ID, CreatingShare: true}})
+ require.ErrorIs(t, err, store.ErrMemoMutationConflict)
+ require.Equal(t, 0, db.count(t, "SELECT COUNT(*) FROM memo_share"))
+ })
+}
+
+func TestUpdateMemoAbortsWhenMemoChanges(t *testing.T) {
+ forEachAccessMode(t, func(t *testing.T, db *raceDB) {
+ user := db.user(t, "alice")
+ memo := db.memo(t, "m1", user)
+
+ db.interpose(t, "UPDATE memo SET", func() { db.archive(t, memo) })
+ content := "edited"
+ err := db.UpdateMemo(db.ctx, &store.UpdateMemo{ID: memo.ID, Content: &content, Policy: &store.MemoWritePolicy{ActorUserID: user.ID}})
+ require.ErrorIs(t, err, store.ErrMemoMutationConflict)
+ require.Equal(t, "original", db.memoContent(t, memo))
+ })
+}
+
+func TestMemoMutationAbortsWhenMemoChanges(t *testing.T) {
+ forEachAccessMode(t, func(t *testing.T, db *raceDB) {
+ user := db.user(t, "alice")
+ memo := db.memo(t, "m1", user)
+
+ db.interpose(t, "UPDATE memo SET", func() { db.archive(t, memo) })
+ content := "edited"
+ err := db.ApplyMemoMutation(db.ctx, &store.MemoMutation{
+ MemoID: memo.ID,
+ MemoCreatorID: user.ID,
+ ExpectedMemoContent: "original",
+ MemoUpdate: &store.UpdateMemo{ID: memo.ID, Content: &content},
+ Policy: &store.MemoWritePolicy{ActorUserID: user.ID},
+ })
+ require.ErrorIs(t, err, store.ErrMemoMutationConflict)
+ require.Equal(t, "original", db.memoContent(t, memo))
+ })
+}
+
+func TestAttachmentCreateAbortsWhenMemoChanges(t *testing.T) {
+ forEachAccessMode(t, func(t *testing.T, db *raceDB) {
+ user := db.user(t, "alice")
+ memo := db.memo(t, "m1", user)
+
+ db.interpose(t, "INSERT INTO attachment", func() { db.archive(t, memo) })
+ _, err := db.CreateAttachment(db.ctx, &store.Attachment{UID: "a1", CreatorID: user.ID, MemoID: &memo.ID, Policy: &store.MemoWritePolicy{ActorUserID: user.ID}})
+ require.ErrorIs(t, err, store.ErrMemoMutationConflict)
+ require.Equal(t, 0, db.count(t, "SELECT COUNT(*) FROM attachment"))
+ })
+}
+
+func TestAttachmentUpdateAbortsWhenBindingChanges(t *testing.T) {
+ forEachAccessMode(t, func(t *testing.T, db *raceDB) {
+ user := db.user(t, "alice")
+ memo := db.memo(t, "m1", user)
+ attachment := db.attachment(t, "a1", user, memo)
+
+ // The attachment is unbound from the memo the actor was authorized
+ // against before the update commits.
+ db.interpose(t, "UPDATE attachment SET", func() {
+ _, err := db.other.GetDB().ExecContext(db.ctx, "UPDATE attachment SET memo_id = NULL WHERE id = ?", attachment.ID)
+ require.NoError(t, err)
+ })
+ filename := "renamed.png"
+ err := db.UpdateAttachment(db.ctx, &store.UpdateAttachment{ID: attachment.ID, Filename: &filename, Policy: &store.MemoWritePolicy{ActorUserID: user.ID}})
+ require.ErrorIs(t, err, store.ErrMemoPermissionDenied)
+ require.Equal(t, 0, db.count(t, "SELECT COUNT(*) FROM attachment WHERE filename = ?", filename))
+ })
+}
+
+func TestAttachmentDeleteAbortsWhenMemoContentChanges(t *testing.T) {
+ forEachAccessMode(t, func(t *testing.T, db *raceDB) {
+ user := db.user(t, "alice")
+ memo := db.memo(t, "m1", user)
+ attachment := db.attachment(t, "a1", user, memo)
+
+ db.interpose(t, "DELETE FROM attachment", func() {
+ content := "edited"
+ require.NoError(t, db.other.UpdateMemo(db.ctx, &store.UpdateMemo{ID: memo.ID, Content: &content}))
+ })
+ policy := &store.AttachmentDeletionPolicy{ActorUserID: user.ID, ExpectedMemoContents: map[int32]string{memo.ID: "original"}}
+ err := db.DeleteAttachmentsWithPolicy(db.ctx, policy, []int32{attachment.ID})
+ require.ErrorIs(t, err, store.ErrMemoMutationConflict)
+ require.Equal(t, 1, db.count(t, "SELECT COUNT(*) FROM attachment WHERE id = ?", attachment.ID))
+ })
+}
+
+func TestMemoDeleteAbortsWhenAttachmentSetIsSwapped(t *testing.T) {
+ forEachAccessMode(t, func(t *testing.T, db *raceDB) {
+ user := db.user(t, "alice")
+ memo := db.memo(t, "m1", user)
+ attachment := db.attachment(t, "a1", user, memo)
+
+ // One attachment is replaced by another: the count the deletion read
+ // still matches, the set does not.
+ db.interpose(t, "DELETE FROM memo WHERE", func() {
+ require.NoError(t, db.other.DeleteAttachment(db.ctx, &store.DeleteAttachment{ID: attachment.ID}))
+ db.attachment(t, "a2", user, memo)
+ })
+ _, err := db.DeleteMemoWithPolicy(db.ctx, &store.DeleteMemoWithPolicy{MemoID: memo.ID, ActorUserID: user.ID})
+ require.ErrorIs(t, err, store.ErrMemoMutationConflict)
+ require.Equal(t, 1, db.count(t, "SELECT COUNT(*) FROM memo WHERE id = ?", memo.ID))
+ require.Equal(t, 1, db.count(t, "SELECT COUNT(*) FROM attachment WHERE memo_id = ? AND uid = 'a2'", memo.ID))
+ })
+}
+
+func TestUserDeleteAbortsWhenMemoSetIsSwapped(t *testing.T) {
+ forEachAccessMode(t, func(t *testing.T, db *raceDB) {
+ user := db.user(t, "bob")
+ memo := db.memo(t, "m1", user)
+
+ db.interpose(t, "DELETE FROM user WHERE", func() {
+ require.NoError(t, db.other.DeleteMemo(db.ctx, &store.DeleteMemo{ID: memo.ID}))
+ db.memo(t, "m2", user)
+ })
+ _, err := db.DeleteUser(db.ctx, &store.DeleteUser{ID: user.ID})
+ require.ErrorIs(t, err, errUserDeleteConflict)
+ require.Equal(t, 1, db.count(t, "SELECT COUNT(*) FROM user WHERE id = ?", user.ID))
+ require.Equal(t, 1, db.count(t, "SELECT COUNT(*) FROM memo WHERE creator_id = ?", user.ID))
+ })
+}
+
+func TestSpaceDeleteAbortsWhenMemoSetIsSwapped(t *testing.T) {
+ forEachAccessMode(t, func(t *testing.T, db *raceDB) {
+ user := db.user(t, "alice")
+ space, err := db.other.CreateSpace(db.ctx, &store.Space{UID: "space", Title: "Space"}, user.ID)
+ require.NoError(t, err)
+ visibility := store.SpaceAudience
+ memo, err := db.other.CreateMemo(db.ctx, &store.Memo{UID: "m1", CreatorID: user.ID, Content: "original", Visibility: visibility, SpaceID: &space.ID})
+ require.NoError(t, err)
+
+ db.interpose(t, "DELETE FROM space WHERE", func() {
+ require.NoError(t, db.other.DeleteMemo(db.ctx, &store.DeleteMemo{ID: memo.ID}))
+ _, err := db.other.CreateMemo(db.ctx, &store.Memo{UID: "m2", CreatorID: user.ID, Content: "later", Visibility: visibility, SpaceID: &space.ID})
+ require.NoError(t, err)
+ })
+ _, err = db.DeleteSpace(db.ctx, &store.DeleteSpace{ID: space.ID, ActorUserID: user.ID})
+ require.ErrorIs(t, err, errSpaceDeleteConflict)
+ require.Equal(t, 1, db.count(t, "SELECT COUNT(*) FROM space WHERE id = ?", space.ID))
+ require.Equal(t, 1, db.count(t, "SELECT COUNT(*) FROM memo WHERE space_id = ?", space.ID))
+ })
+}
diff --git a/store/db/d1/reaction.go b/store/db/d1/reaction.go
index 9272979be..6927a8499 100644
--- a/store/db/d1/reaction.go
+++ b/store/db/d1/reaction.go
@@ -3,7 +3,6 @@ package d1
import (
"context"
"database/sql"
- "slices"
"strings"
"github.com/pkg/errors"
@@ -105,25 +104,14 @@ func (d *DB) ListReactions(ctx context.Context, find *store.FindReaction) ([]*st
if find.MemoID != nil {
where, args = append(where, "memo_id = ?"), append(args, *find.MemoID)
}
- if len(find.MemoIDList) == 0 {
- return reactionQuery(ctx, d.db, where, args)
- }
-
- // A memo id list can exceed D1's bind limit, so it is queried in chunks
- // and the union re-sorted to preserve the id order of a single query.
- list := []*store.Reaction{}
- for _, ids := range chunk(find.MemoIDList, inClauseBatchSize) {
- clause, memoArgs := inClause(ids)
- chunkWhere := append(append([]string{}, where...), "memo_id IN "+clause)
- chunkArgs := append(append([]any{}, args...), memoArgs...)
- reactions, err := reactionQuery(ctx, d.db, chunkWhere, chunkArgs)
+ if len(find.MemoIDList) > 0 {
+ clause, memoArg, err := jsonList(find.MemoIDList)
if err != nil {
return nil, err
}
- list = append(list, reactions...)
+ where, args = append(where, "memo_id IN "+clause), append(args, memoArg)
}
- slices.SortFunc(list, func(a, b *store.Reaction) int { return int(a.ID) - int(b.ID) })
- return list, nil
+ return reactionQuery(ctx, d.db, where, args)
}
// reactionQuery lists the reactions matching where, ordered by id.
diff --git a/store/db/d1/shared.go b/store/db/d1/shared.go
index 056c933cc..0716a209b 100644
--- a/store/db/d1/shared.go
+++ b/store/db/d1/shared.go
@@ -75,27 +75,29 @@ func spaceState(ctx context.Context, q querier, spaceID, userID int32) (exists b
}
// validateMemoWritePolicy checks that the actor may write memoID under policy,
-// optionally for the pending update.
-func validateMemoWritePolicy(ctx context.Context, q querier, memoID int32, policy *store.MemoWritePolicy, update *store.UpdateMemo) error {
+// optionally for the pending update, and returns the memo state the check was
+// made against so the batch can guard on exactly that state.
+func validateMemoWritePolicy(ctx context.Context, q querier, memoID int32, policy *store.MemoWritePolicy, update *store.UpdateMemo) (*memoState, error) {
if err := requireActiveUser(ctx, q, policy.ActorUserID, store.ErrMemoSpaceMembershipRequired); err != nil {
- return err
+ return nil, err
}
-
- snapshot := new(store.MemoWriteSnapshot)
- var spaceID sql.NullInt64
- if err := q.QueryRowContext(ctx, `SELECT creator_id, row_status, space_id, visibility FROM memo WHERE id = ?`, memoID).Scan(
- &snapshot.CreatorID, &snapshot.RowStatus, &spaceID, &snapshot.Visibility,
- ); err != nil {
+ state, err := loadMemoState(ctx, q, memoID)
+ if err != nil {
if errors.Is(err, sql.ErrNoRows) {
- return store.ErrMemoMutationConflict
+ return nil, store.ErrMemoMutationConflict
}
- return err
+ return nil, err
+ }
+ snapshot := &store.MemoWriteSnapshot{
+ CreatorID: state.creatorID,
+ RowStatus: state.rowStatus,
+ SpaceID: state.spaceID,
+ Visibility: state.visibility,
}
- snapshot.SpaceID = store.NullInt32Pointer(spaceID)
if snapshot.SpaceID != nil {
exists, member, err := spaceState(ctx, q, *snapshot.SpaceID, policy.ActorUserID)
if err != nil {
- return err
+ return nil, err
}
snapshot.SourceSpaceExists = exists
snapshot.SourceMemberActive = member
@@ -103,7 +105,7 @@ func validateMemoWritePolicy(ctx context.Context, q querier, memoID int32, polic
if update != nil && update.SpaceID != nil {
exists, member, err := spaceState(ctx, q, *update.SpaceID, policy.ActorUserID)
if err != nil {
- return err
+ return nil, err
}
snapshot.TargetSpaceExists = exists
snapshot.TargetMemberActive = member
@@ -115,10 +117,24 @@ func validateMemoWritePolicy(ctx context.Context, q querier, memoID int32, polic
LIMIT 1`, memoID).Scan(&shareID)
snapshot.HasActiveShare = err == nil
if err != nil && !errors.Is(err, sql.ErrNoRows) {
- return err
+ return nil, err
}
}
- return store.ValidateMemoWriteSnapshot(policy, update, snapshot)
+ if err := store.ValidateMemoWriteSnapshot(policy, update, snapshot); err != nil {
+ return nil, err
+ }
+ return state, nil
+}
+
+// memoGuardWritePolicy re-asserts inside the batch everything
+// validateMemoWritePolicy accepted: the memo row is unchanged, its Space
+// still exists, the actor keeps the memberships the policy needs, and no
+// active share blocks a move to the SPACE audience. The caller guards the
+// actor's own row once.
+func memoGuardWritePolicy(b *batch, state *memoState, policy *store.MemoWritePolicy, update *store.UpdateMemo) {
+ condition, args := state.unchangedCondition()
+ b.guard(condition, args...)
+ memoGuardPolicyPlacement(b, state, policy, update)
}
// loadMemoParticipation resolves the actor, Space, membership, and memo state
@@ -267,40 +283,61 @@ func listMemoSetAttachments(ctx context.Context, q querier, memoIDs []int32) ([]
}
// authorizeAttachmentMutation checks the actor may mutate attachments bound to
-// memoIDs, optionally requiring the memos' current contents to match.
-func authorizeAttachmentMutation(ctx context.Context, q querier, actorUserID int32, memoIDs []int32, expectedMemoContents map[int32]string) error {
+// memoIDs, optionally requiring the memos' current contents to match, and
+// returns the memo states the checks were made against.
+func authorizeAttachmentMutation(ctx context.Context, q querier, actorUserID int32, memoIDs []int32, expectedMemoContents map[int32]string) ([]*memoState, error) {
if err := requireActiveUser(ctx, q, actorUserID, store.ErrMemoPermissionDenied); err != nil {
- return err
+ return nil, err
}
+ states := make([]*memoState, 0, len(memoIDs))
for _, memoID := range memoIDs {
- snapshot := &store.MemoWriteSnapshot{}
- var currentSpace sql.NullInt64
- var content string
- if err := q.QueryRowContext(ctx, `SELECT creator_id, row_status, space_id, visibility, content FROM memo WHERE id = ?`, memoID).Scan(
- &snapshot.CreatorID, &snapshot.RowStatus, &currentSpace, &snapshot.Visibility, &content,
- ); err != nil {
+ state, err := loadMemoState(ctx, q, memoID)
+ if err != nil {
if errors.Is(err, sql.ErrNoRows) {
- return store.ErrMemoMutationConflict
+ return nil, store.ErrMemoMutationConflict
}
- return errors.Wrap(err, "failed to read attachment memo")
+ return nil, errors.Wrap(err, "failed to read attachment memo")
+ }
+ if expectedMemoContents != nil && state.content != expectedMemoContents[memoID] {
+ return nil, store.ErrMemoMutationConflict
}
- snapshot.SpaceID = store.NullInt32Pointer(currentSpace)
- if expectedMemoContents != nil && content != expectedMemoContents[memoID] {
- return store.ErrMemoMutationConflict
+ snapshot := &store.MemoWriteSnapshot{
+ CreatorID: state.creatorID,
+ RowStatus: state.rowStatus,
+ SpaceID: state.spaceID,
+ Visibility: state.visibility,
}
if snapshot.SpaceID != nil {
exists, member, err := spaceState(ctx, q, *snapshot.SpaceID, actorUserID)
if err != nil {
- return errors.Wrap(err, "failed to read attachment memo space")
+ return nil, errors.Wrap(err, "failed to read attachment memo space")
}
snapshot.SourceSpaceExists = exists
snapshot.SourceMemberActive = member
}
if err := store.ValidateMemoWriteSnapshot(&store.MemoWritePolicy{ActorUserID: actorUserID}, nil, snapshot); err != nil {
- return err
+ return nil, err
}
+ states = append(states, state)
+ }
+ return states, nil
+}
+
+// guardAttachmentMutation re-asserts what authorizeAttachmentMutation
+// accepted for every linked memo and pins each attachment to the binding it
+// was read with. With expectedMemoContents set, the memo content the caller
+// computed the mutation from must still be current at commit time.
+func guardAttachmentMutation(b *batch, actorUserID int32, states []*memoState, attachments []*store.Attachment, expectedMemoContents map[int32]string) {
+ b.guard(activeUserCondition, actorUserID)
+ for _, state := range states {
+ memoGuardWritePolicy(b, state, &store.MemoWritePolicy{ActorUserID: actorUserID}, nil)
+ if expectedMemoContents != nil {
+ b.guard("EXISTS (SELECT 1 FROM memo WHERE id = ? AND content = ?)", state.id, expectedMemoContents[state.id])
+ }
+ }
+ for _, attachment := range attachments {
+ b.guard("EXISTS (SELECT 1 FROM attachment WHERE id = ? AND memo_id IS ?)", attachment.ID, attachment.MemoID)
}
- return nil
}
// attachmentIDs extracts the ids of attachments.
diff --git a/store/db/d1/space.go b/store/db/d1/space.go
index 4110f4e4c..afef05abb 100644
--- a/store/db/d1/space.go
+++ b/store/db/d1/space.go
@@ -135,8 +135,9 @@ func (d *DB) CreateSpace(ctx context.Context, create *store.Space, creatorID int
if err != nil {
return nil, err
}
- // The membership row cannot use the id generated by the first insert
- // because a batch never reads its own writes; the unique uid resolves it.
+ // The batch runs in order inside one transaction, so the membership row
+ // can find the space inserted just before it; it resolves the row through
+ // the unique uid because the generated id is only reported after commit.
b := newBatch()
b.guard(activeUserCondition, creatorID)
b.add("INSERT INTO space (uid, title, description, payload) VALUES (?, ?, ?, ?)", create.UID, create.Title, create.Description, payload)
@@ -170,8 +171,11 @@ func (d *DB) ListSpaces(ctx context.Context, find *store.FindSpace) ([]*store.Sp
where, args = append(where, "space.id = ?"), append(args, *find.ID)
}
if len(find.IDList) > 0 {
- clause, ids := inClause(find.IDList)
- where, args = append(where, "space.id IN "+clause), append(args, ids...)
+ clause, ids, err := jsonList(find.IDList)
+ if err != nil {
+ return nil, err
+ }
+ where, args = append(where, "space.id IN "+clause), append(args, ids)
}
if find.UID != nil {
where, args = append(where, "space.uid = ?"), append(args, *find.UID)
diff --git a/store/db/d1/space_delete.go b/store/db/d1/space_delete.go
index 050c07a21..488c95bd3 100644
--- a/store/db/d1/space_delete.go
+++ b/store/db/d1/space_delete.go
@@ -63,8 +63,8 @@ func (d *DB) DeleteSpace(ctx context.Context, delete *store.DeleteSpace) (*store
b.guard("EXISTS (SELECT 1 FROM space_member WHERE space_id = ? AND user_id = ? AND status = 'ACTIVE' AND role = 'ADMIN')", delete.ID, delete.ActorUserID)
// The memo set was read outside the batch; abort when it changed so no
// memo or attachment is left pointing at a deleted space.
- b.guard("(SELECT COUNT(*) FROM memo WHERE space_id = ?) = ?", delete.ID, len(memoIDs))
- b.guard("(SELECT COUNT(*) FROM attachment WHERE memo_id IN (SELECT id FROM memo WHERE space_id = ?)) = ?", delete.ID, len(attachments))
+ b.guardIDSet("memo WHERE space_id = ?", []any{delete.ID}, memoIDs)
+ b.guardIDSet("attachment WHERE memo_id IN (SELECT id FROM memo WHERE space_id = ?)", []any{delete.ID}, attachmentIDs(attachments))
addMemoSetDeletes(b, memoIDs, attachmentIDs(attachments))
b.add("DELETE FROM space_member WHERE space_id = ?", delete.ID)
b.add("DELETE FROM space WHERE id = ?", delete.ID)
diff --git a/store/db/d1/sqldriver.go b/store/db/d1/sqldriver.go
index 33f36b4fc..1ea8023ab 100644
--- a/store/db/d1/sqldriver.go
+++ b/store/db/d1/sqldriver.go
@@ -161,11 +161,14 @@ func (t *tx) Commit() error {
}
ctx := context.Background()
if parameterFree(t.pending) {
+ // Each buffered text is a script that already follows the one
+ // statement per paragraph rule, so a blank line joins them into one
+ // script that still does.
sqls := make([]string, 0, len(t.pending))
for _, stmt := range t.pending {
sqls = append(sqls, strings.TrimSpace(stmt.SQL))
}
- return t.conn.transport.script(ctx, strings.Join(sqls, "\n;\n"))
+ return t.conn.transport.script(ctx, strings.Join(sqls, "\n\n"))
}
_, err := t.conn.transport.batch(ctx, t.pending)
return err
diff --git a/store/db/d1/user.go b/store/db/d1/user.go
index 69ef0f669..71a321e8b 100644
--- a/store/db/d1/user.go
+++ b/store/db/d1/user.go
@@ -168,12 +168,18 @@ func (d *DB) ListUsers(ctx context.Context, find *store.FindUser) ([]*store.User
where, args = append(where, "id = ?"), append(args, *v)
}
if len(find.IDList) > 0 {
- clause, ids := inClause(find.IDList)
- where, args = append(where, "id IN "+clause), append(args, ids...)
+ clause, ids, err := jsonList(find.IDList)
+ if err != nil {
+ return nil, err
+ }
+ where, args = append(where, "id IN "+clause), append(args, ids)
}
if len(find.UsernameList) > 0 {
- clause, usernames := inClause(find.UsernameList)
- where, args = append(where, "username IN "+clause), append(args, usernames...)
+ clause, usernames, err := jsonList(find.UsernameList)
+ if err != nil {
+ return nil, err
+ }
+ where, args = append(where, "username IN "+clause), append(args, usernames)
}
if v := find.RowStatus; v != nil {
where, args = append(where, "row_status = ?"), append(args, *v)
diff --git a/store/db/d1/user_delete.go b/store/db/d1/user_delete.go
index fe2cf6423..7a0d66c6f 100644
--- a/store/db/d1/user_delete.go
+++ b/store/db/d1/user_delete.go
@@ -108,8 +108,8 @@ func (d *DB) DeleteUser(ctx context.Context, delete *store.DeleteUser) (*store.D
b.guard("NOT EXISTS (SELECT 1 FROM space_member WHERE user_id = ? AND status <> 'INVITED')", delete.ID)
// The memo and attachment sets were read outside the batch; abort when
// they changed so nothing authored by the user survives its deletion.
- b.guard("(SELECT COUNT(*) FROM memo WHERE creator_id = ?) = ?", delete.ID, len(targets.memoIDs))
- b.guard("(SELECT COUNT(*) FROM attachment WHERE creator_id = ? OR memo_id IN (SELECT id FROM memo WHERE creator_id = ?)) = ?", delete.ID, delete.ID, len(targets.attachments))
+ b.guardIDSet("memo WHERE creator_id = ?", []any{delete.ID}, targets.memoIDs)
+ b.guardIDSet("attachment WHERE creator_id = ? OR memo_id IN (SELECT id FROM memo WHERE creator_id = ?)", []any{delete.ID, delete.ID}, attachmentIDs(targets.attachments))
b.add("DELETE FROM space_member WHERE user_id = ? AND status = 'INVITED'", delete.ID)
addMemoSetDeletes(b, targets.memoIDs, attachmentIDs(targets.attachments))
b.add("DELETE FROM reaction WHERE creator_id = ?", delete.ID)
diff --git a/store/db/d1/user_identity.go b/store/db/d1/user_identity.go
index 98d5301ee..bcdc298e8 100644
--- a/store/db/d1/user_identity.go
+++ b/store/db/d1/user_identity.go
@@ -20,10 +20,11 @@ func (d *DB) CreateUserIdentity(ctx context.Context, create *store.UserIdentity)
return create, nil
}
-// CreateUserWithIdentity inserts both rows in one batch. The identity cannot
-// use the id generated by the user insert, because a batch never reads its
-// own writes, so it resolves the user through the unique username and both
-// rows are read back after the commit.
+// CreateUserWithIdentity inserts both rows in one batch. The batch runs in
+// order inside one transaction, so the identity insert can find the user row
+// inserted just before it; it resolves the user through the unique username
+// because the generated id is only reported after commit, and both rows are
+// read back afterwards.
func (d *DB) CreateUserWithIdentity(ctx context.Context, createUser *store.User, createIdentity *store.UserIdentity) (*store.User, error) {
b := newBatch()
b.add("INSERT INTO user (username, role, email, nickname, password_hash, avatar_url) VALUES (?, ?, ?, ?, ?, ?)",
diff --git a/store/migration/d1/LATEST.sql b/store/migration/d1/LATEST.sql
index 6c8b718fa..88401e7ab 100644
--- a/store/migration/d1/LATEST.sql
+++ b/store/migration/d1/LATEST.sql
@@ -76,6 +76,7 @@ CREATE TABLE memo (
);
CREATE INDEX idx_memo_creator_id ON memo(creator_id);
+
CREATE INDEX idx_memo_space_id ON memo(space_id, row_status, created_ts DESC, id DESC);
-- memo_relation
diff --git a/store/test/list_id_list_test.go b/store/test/list_id_list_test.go
new file mode 100644
index 000000000..8da96b831
--- /dev/null
+++ b/store/test/list_id_list_test.go
@@ -0,0 +1,79 @@
+package test
+
+import (
+ "context"
+ "fmt"
+ "testing"
+
+ "github.com/stretchr/testify/require"
+
+ "github.com/usememos/memos/store"
+)
+
+// TestListsAcceptLongIDLists lists by id lists far longer than the hundred
+// bound parameters Cloudflare D1 allows per statement. Every driver must
+// accept them: the API hands page-sized lists, up to a thousand entries, to
+// these lookups.
+func TestListsAcceptLongIDLists(t *testing.T) {
+ ctx := context.Background()
+ ts := NewTestingStore(ctx, t)
+ user, err := createTestingHostUser(ctx, ts)
+ require.NoError(t, err)
+ space, err := ts.CreateSpace(ctx, &store.Space{UID: "long-lists", Title: "Long lists"}, user.ID)
+ require.NoError(t, err)
+ memo, err := ts.CreateMemo(ctx, &store.Memo{UID: "long-lists-memo", CreatorID: user.ID, Content: "listed", Visibility: store.Private})
+ require.NoError(t, err)
+ attachment, err := ts.CreateAttachment(ctx, &store.Attachment{UID: "long-lists-attachment", CreatorID: user.ID, Filename: "listed.png", Type: "image/png", MemoID: &memo.ID})
+ require.NoError(t, err)
+ reaction, err := ts.UpsertReaction(ctx, &store.Reaction{CreatorID: user.ID, MemoID: memo.ID, ReactionType: "👍"})
+ require.NoError(t, err)
+
+ const length = 250
+ ids := make([]int32, 0, length)
+ uids := make([]string, 0, length)
+ usernames := make([]string, 0, length)
+ for i := 1; i <= length; i++ {
+ ids = append(ids, int32(i))
+ uids = append(uids, fmt.Sprintf("absent-%d", i))
+ usernames = append(usernames, fmt.Sprintf("absent-%d", i))
+ }
+ uids[0] = memo.UID
+ usernames[0] = user.Username
+
+ memos, err := ts.ListMemos(ctx, &store.FindMemo{UIDList: uids})
+ require.NoError(t, err)
+ require.Len(t, memos, 1)
+ require.Equal(t, memo.ID, memos[0].ID)
+
+ memos, err = ts.ListMemos(ctx, &store.FindMemo{IDList: ids})
+ require.NoError(t, err)
+ require.Len(t, memos, 1)
+
+ attachments, err := ts.ListAttachments(ctx, &store.FindAttachment{MemoIDList: ids})
+ require.NoError(t, err)
+ require.Len(t, attachments, 1)
+ require.Equal(t, attachment.ID, attachments[0].ID)
+
+ users, err := ts.ListUsers(ctx, &store.FindUser{IDList: ids})
+ require.NoError(t, err)
+ require.Len(t, users, 1)
+ require.Equal(t, user.ID, users[0].ID)
+
+ users, err = ts.ListUsers(ctx, &store.FindUser{UsernameList: usernames})
+ require.NoError(t, err)
+ require.Len(t, users, 1)
+
+ spaces, err := ts.ListSpaces(ctx, &store.FindSpace{IDList: ids})
+ require.NoError(t, err)
+ require.Len(t, spaces, 1)
+ require.Equal(t, space.ID, spaces[0].ID)
+
+ reactions, err := ts.ListReactions(ctx, &store.FindReaction{MemoIDList: ids})
+ require.NoError(t, err)
+ require.Len(t, reactions, 1)
+ require.Equal(t, reaction.ID, reactions[0].ID)
+
+ relations, err := ts.ListMemoRelations(ctx, &store.FindMemoRelation{MemoIDList: ids})
+ require.NoError(t, err)
+ require.Empty(t, relations)
+}