Memos is an open-source, self-hosted note-taking app that brings your everyday thoughts together in a personal timeline. Your data stays yours, and you choose what to keep private or share.
Go dockerfossgomarkdownmemomicroblognote-takingnotesopen-sourceown-your-datareactself-hostedsqlitetimeline
fix(d1): close guard gaps, chunk-free list bindings, and statement layout
bluedbird9ca12926c7c1
diff --git a/docs/design/cloudflare-d1.md b/docs/design/cloudflare-d1.mdindex fcb2cf260..4d41ea049 100644--- a/docs/design/cloudflare-d1.md+++ b/docs/design/cloudflare-d1.md@@ -174,7 +174,7 @@ async function runBatch(env, request) {const { statements } = await request.json();const prepared = statements.map((s) => env.DB.prepare(s.sql).bind(...(s.params ?? [])));try {- const results = (await env.DB.batch(prepared)).map(({ results, meta }) => {+ const results = (await env.DB.batch(prepared)).map(({ results = [], meta }) => {const columns = results.length ? Object.keys(results[0]) : [];return {columns,@@ -230,11 +230,22 @@ statements run in one transaction. Every driver method is shaped around them.batch, and the driver maps it to the conflict error the caller expects.`d1_guard` never holds rows.-A batch cannot read its own writes, so an id generated by an `INSERT` cannot-feed a later statement. Methods that need it use the unique natural key-instead: a membership created with a new space references-`(SELECT id FROM space WHERE uid = ?)`, and the space is read back by uid-after the commit.+A batch runs its statements in order inside one transaction, so a later+statement sees the rows an earlier one inserted, but the generated ids are+only reported after the commit. Methods that need a new id inside the batch+use the unique natural key instead: a membership created with a new space+references `(SELECT id FROM space WHERE uid = ?)`, and the space is read back+by uid after the commit. Statements that return generated values through+`RETURNING` get them from the batch result, so a policy-checked insert can+sit behind its guards and still report its id.++Guards re-assert every fact a validation read established, not only the+row's existence: a memo's creator, lifecycle, audience, and placement, the+actor's memberships, and, where the caller computed the mutation from the+memo text, the content itself. A set of rows read before the batch, such as+the attachments a memo deletion returns for storage cleanup, is guarded on+its count together with its highest id, because ids only grow and a row+swapped for a newer one keeps the count equal.## Schema and migrations@@ -253,8 +264,27 @@ D1 scripts follow one layout rule: **one statement per paragraph**. A blankline separates statements, and nothing inside a statement, including a tableor trigger body, a string literal, or a comment, may contain a blank line.The bridge transport divides scripts at blank lines to send them as a batch,-and the test emulator executes them the same way, so a script that breaks the-rule fails the store suite's fresh install rather than a deployment.+and D1 refuses a bridge entry that holds more than one statement. The test+emulator refuses it too, judging statement boundaries with SQLite's own+`sqlite3_complete`, so a paragraph with two statements fails the store+suite's bridge run rather than a deployment.++## Operational limits++The emulator enforces the bind limit and the one-statement rule, but not the+service's size and rate limits, which a deployment must keep in mind:++- A bound string or blob may be at most 2 MB. Blobs travel as hex text, so an+ attachment stored in the database is limited to about 1 MB; larger+ attachments belong in local or S3 storage.+- A batch may hold on the order of a thousand statements. Deleting a user or+ a space removes its memos in chunks of ninety, so an account with many+ thousands of memos exceeds the batch and must be trimmed first.+- The REST API is subject to Cloudflare's account-wide rate limit of 1200+ requests per five minutes, after which every request is refused with a 429+ for the rest of the window. The driver does not retry a 429, since an+ immediate retry only extends the block. The bridge is a Worker request and+ is not subject to that limit.## Filter dialect@@ -268,7 +298,11 @@ patterns with optional anchors.`store/db/d1/d1test` emulates D1 in-process on a private SQLite database andspeaks both protocols: the REST API endpoints and the bridge endpoint, withthe same implicit transaction per request, atomic batches, foreign-key-enforcement, and 100-parameter limit, and no custom functions registered.+enforcement, 100-parameter limit, and one-statement rule for prepared+entries, and no custom functions registered. Driver tests in+`store/db/d1/race_test.go` interpose on the transport to change the database+between a method's validation reads and its batch, which exercises the guards+the way a concurrent writer would.`DRIVER=d1` runs the whole store suite against it over REST, and`D1_ACCESS=bridge` runs it over the bridge protocol; the all-drivers loopruns both. Setting `D1_DSN` points the suite at a realdiff --git a/go.mod b/go.modindex 60316fef0..53bdde078 100644--- a/go.mod+++ b/go.mod@@ -46,6 +46,7 @@ require (google.golang.org/grpc v1.83.0google.golang.org/protobuf v1.36.12gopkg.in/yaml.v3 v3.0.1+ modernc.org/libc v1.75.3modernc.org/sqlite v1.56.0)@@ -148,7 +149,6 @@ require (golang.org/x/time v0.15.0 // indirectgolang.org/x/tools v0.49.0 // indirectgoogle.golang.org/api v0.293.0 // indirect- modernc.org/libc v1.75.3 // indirectmodernc.org/mathutil v1.7.1 // indirectmodernc.org/memory v1.12.0 // indirect)diff --git a/store/db/d1/attachment.go b/store/db/d1/attachment.goindex 71ab852f3..e9349f63b 100644--- a/store/db/d1/attachment.go+++ b/store/db/d1/attachment.go@@ -31,19 +31,23 @@ func (d *DB) CreateAttachment(ctx context.Context, create *store.Attachment) (*sif create.MemoID == nil || create.CreatorID != create.Policy.ActorUserID {return nil, store.ErrMemoPermissionDenied}- if err := validateMemoWritePolicy(ctx, d.db, *create.MemoID, create.Policy, nil); err != nil {+ state, err := validateMemoWritePolicy(ctx, d.db, *create.MemoID, create.Policy, nil)+ if err != nil {return nil, err}- // Without a transaction the validated memo can vanish before the insert;- // selecting the values from the memo row makes the insert conditional on- // its continued existence, so a vanished memo yields no row instead of an- // orphaned attachment.- args = append(args, *create.MemoID)- stmt := "INSERT INTO attachment (" + strings.Join(columns, ", ") + ") SELECT " + strings.Join(values, ", ") + " FROM memo WHERE memo.id = ? RETURNING id, created_ts, updated_ts"- if err := d.db.QueryRowContext(ctx, stmt, args...).Scan(&create.ID, &create.CreatedTs, &create.UpdatedTs); err != nil {- if errors.Is(err, sql.ErrNoRows) {- return nil, store.ErrMemoMutationConflict- }+ // The guards re-assert the validated memo row and the actor's membership+ // at commit time, so a memo that vanished, was archived, or moved to a+ // Space the actor left yields a conflict instead of an orphaned or+ // unauthorized attachment.+ b := newBatch()+ b.guard(activeUserCondition, create.Policy.ActorUserID)+ memoGuardWritePolicy(b, state, create.Policy, nil)+ index := b.add("INSERT INTO attachment ("+strings.Join(columns, ", ")+") VALUES ("+strings.Join(values, ", ")+") RETURNING id, created_ts, updated_ts", args...)+ results, err := b.commit(ctx, d)+ if err != nil {+ return nil, guardError(err, store.ErrMemoMutationConflict)+ }+ if err := scanResultRow(results[index], &create.ID, &create.CreatedTs, &create.UpdatedTs); err != nil {return nil, err}return create, nil@@ -98,8 +102,11 @@ func (d *DB) ListAttachments(ctx context.Context, find *store.FindAttachment) ([where, args = append(where, "attachment.memo_id = ?"), append(args, *v)}if len(find.MemoIDList) > 0 {- clause, memoArgs := inClause(find.MemoIDList)- where, args = append(where, "attachment.memo_id IN "+clause), append(args, memoArgs...)+ clause, memoArg, err := jsonList(find.MemoIDList)+ if err != nil {+ return nil, err+ }+ where, args = append(where, "attachment.memo_id IN "+clause), append(args, memoArg)}if find.HasRelatedMemo {where = append(where, "attachment.memo_id IS NOT NULL")@@ -238,12 +245,12 @@ func (d *DB) UpdateAttachment(ctx context.Context, update *store.UpdateAttachmenif err != nil {return err}- if err := authorizeAttachmentMutation(ctx, d.db, update.Policy.ActorUserID, memoIDs, nil); err != nil {+ states, err := authorizeAttachmentMutation(ctx, d.db, update.Policy.ActorUserID, memoIDs, nil)+ if err != nil {return err}- // The actor was validated above; the guard re-asserts it at commit time.b := newBatch()- b.guard(activeUserCondition, update.Policy.ActorUserID)+ guardAttachmentMutation(b, update.Policy.ActorUserID, states, attachments, nil)b.add(stmt, args...)if _, err := b.commit(ctx, d); err != nil {return errors.Wrap(guardError(err, store.ErrMemoPermissionDenied), "failed to update attachment")diff --git a/store/db/d1/attachment_delete.go b/store/db/d1/attachment_delete.goindex 348bcdb00..2c93f3d73 100644--- a/store/db/d1/attachment_delete.go+++ b/store/db/d1/attachment_delete.go@@ -9,10 +9,10 @@ import ()// DeleteAttachmentsWithPolicy authorizes the actor against every linked memo-// and then deletes the rows in one atomic batch. Each delete is preceded by a-// guard on the row's existence so a row that vanished after the validation-// reads aborts the whole batch as a conflict, mirroring the per-row-// RowsAffected check a transaction would perform.+// and then deletes the rows in one atomic batch. The batch re-asserts the+// memo states and bindings the authorization read, including the memo+// content the caller expects, so a row or memo that changed after the+// validation reads aborts the whole batch as a conflict.func (d *DB) DeleteAttachmentsWithPolicy(ctx context.Context, policy *store.AttachmentDeletionPolicy, attachmentIDs []int32) error {if policy == nil || policy.ActorUserID <= 0 {return store.ErrMemoPermissionDenied@@ -28,15 +28,15 @@ func (d *DB) DeleteAttachmentsWithPolicy(ctx context.Context, policy *store.Attaif err := store.ValidateAttachmentDeletionMemoSnapshots(memoIDs, policy.ExpectedMemoContents); err != nil {return err}- if err := authorizeAttachmentMutation(ctx, d.db, policy.ActorUserID, memoIDs, policy.ExpectedMemoContents); err != nil {+ states, err := authorizeAttachmentMutation(ctx, d.db, policy.ActorUserID, memoIDs, policy.ExpectedMemoContents)+ if err != nil {return err}b := newBatch()- b.guard(activeUserCondition, policy.ActorUserID)- for _, attachmentID := range attachmentIDs {- b.guard("EXISTS (SELECT 1 FROM attachment WHERE id = ?)", attachmentID)- b.add("DELETE FROM attachment WHERE id = ?", attachmentID)+ guardAttachmentMutation(b, policy.ActorUserID, states, attachments, policy.ExpectedMemoContents)+ for _, attachment := range attachments {+ b.add("DELETE FROM attachment WHERE id = ?", attachment.ID)}if _, err := b.commit(ctx, d); err != nil {return guardError(err, store.ErrMemoMutationConflict)diff --git a/store/db/d1/d1_test.go b/store/db/d1/d1_test.goindex 0c687077a..b8eb8cb44 100644--- a/store/db/d1/d1_test.go+++ b/store/db/d1/d1_test.go@@ -5,6 +5,8 @@ import ("errors""net/http""net/http/httptest"+ "os"+ "path/filepath""strings""testing"@@ -211,6 +213,77 @@ func runTransportRoundTrip(t *testing.T, db *DB) {require.Greater(t, size, int64(0))}+// TestSchemaScriptsFollowTheParagraphRule applies the real schema file over+// both transports. The bridge sends one paragraph per prepared statement and+// the emulator, like D1, refuses a prepared statement holding two, so a+// paragraph with two statements fails here rather than on a deployment.+func TestSchemaScriptsFollowTheParagraphRule(t *testing.T) {+ schema, err := os.ReadFile(filepath.Join("..", "..", "migration", "d1", "LATEST.sql"))+ require.NoError(t, err)+ for mode, dsn := range accessModes {+ t.Run(mode, func(t *testing.T) {+ db := newTestDB(t, dsn)+ ctx := context.Background()+ tx, err := db.GetDB().Begin()+ require.NoError(t, err)+ _, err = tx.ExecContext(ctx, string(schema))+ require.NoError(t, err)+ require.NoError(t, tx.Commit())+ initialized, err := db.IsInitialized(ctx)+ require.NoError(t, err)+ require.True(t, initialized)+ })+ }+}++// TestTransactionJoinsScriptsAtParagraphs buffers two scripts in one+// transaction, as the migrator does when several migration files apply, and+// checks they commit over both transports.+func TestTransactionJoinsScriptsAtParagraphs(t *testing.T) {+ for mode, dsn := range accessModes {+ t.Run(mode, func(t *testing.T) {+ db := newTestDB(t, dsn)+ ctx := context.Background()+ tx, err := db.GetDB().Begin()+ require.NoError(t, err)+ _, err = tx.ExecContext(ctx, "CREATE TABLE a (id INTEGER);\n")+ require.NoError(t, err)+ _, err = tx.ExecContext(ctx, "-- second file\nCREATE TABLE b (id INTEGER);\n\nCREATE INDEX b_id ON b(id);\n")+ require.NoError(t, err)+ require.NoError(t, tx.Commit())+ var count int+ require.NoError(t, db.GetDB().QueryRowContext(ctx, "SELECT COUNT(*) FROM sqlite_master WHERE name IN ('a', 'b', 'b_id')").Scan(&count))+ require.Equal(t, 3, count)+ })+ }+}++// TestPreparedEntryHoldsOneStatement pins the emulator to D1's rule: a+// bridge entry and a parameterized REST entry are prepared statements and+// hold exactly one statement, while a parameter-free REST entry is a script.+func TestPreparedEntryHoldsOneStatement(t *testing.T) {+ ctx := context.Background()+ script := "CREATE TABLE a (id INTEGER);\nCREATE TABLE b (id INTEGER);"++ bridge := newTestDB(t, accessModes["bridge"])+ _, err := bridge.execOne(ctx, script)+ require.ErrorContains(t, err, "only one statement")+ _, err = bridge.execOne(ctx, "CREATE TRIGGER t AFTER INSERT ON a BEGIN UPDATE a SET id = 1; UPDATE a SET id = 2; END;")+ require.ErrorContains(t, err, "no such table", "a trigger body is one statement and reaches the database")++ rest := newTestDB(t, accessModes["rest"])+ require.NoError(t, rest.transport.script(ctx, script), "a parameter-free REST entry is a script")+ _, err = rest.execOne(ctx, "INSERT INTO a (id) VALUES (?);\nINSERT INTO b (id) VALUES (?);", 1, 2)+ require.ErrorContains(t, err, "only one statement")+}++func TestRateLimitIsNotRetried(t *testing.T) {+ require.False(t, isRetryable(&Error{Status: http.StatusTooManyRequests, Message: "rate limited"}))+ require.True(t, isRetryable(&Error{Status: http.StatusServiceUnavailable, Message: "unavailable"}))+ require.True(t, isRetryable(&Error{Status: http.StatusBadRequest, Message: "D1_ERROR: database is locked: SQLITE_BUSY"}))+ require.False(t, isRetryable(&Error{Status: http.StatusBadRequest, Message: "D1_ERROR: UNIQUE constraint failed: memo.uid"}))+}+func TestNullAndTypedScans(t *testing.T) {for mode, dsn := range accessModes {t.Run(mode, func(t *testing.T) {diff --git a/store/db/d1/d1test/server.go b/store/db/d1/d1test/server.goindex 6e6f4d457..d0221b1b9 100644--- a/store/db/d1/d1test/server.go+++ b/store/db/d1/d1test/server.go@@ -3,9 +3,11 @@// and /query endpoints, batch bodies, the database metadata endpoint) and the// Memos bridge protocol, on top of a private SQLite database. It enforces the// D1 constraints that matter to the driver: foreign keys are on, statements-// are wrapped in an implicit transaction, a batch is atomic, and a statement-// may bind at most 100 parameters. It registers no custom SQL functions,-// exactly like D1.+// are wrapped in an implicit transaction, a batch is atomic, a statement may+// bind at most 100 parameters, and a prepared entry (a bridge statement or a+// parameterized REST entry) holds exactly one statement while a parameter-free+// REST entry is a script. It registers no custom SQL functions, exactly like+// D1.package d1testimport (@@ -20,10 +22,10 @@ import ("testing""time"+ "modernc.org/libc"// SQLite engine behind the emulated service._ "modernc.org/sqlite"-- "github.com/usememos/memos/store/db/d1/sqlsplit"+ sqlite3 "modernc.org/sqlite/lib")const (@@ -183,23 +185,28 @@ func (s *Server) handleQuery(w http.ResponseWriter, r *http.Request, rawRows booreturn}- // A parameterized entry is one statement; a parameter-free entry is a- // script, executed statement by statement the way D1 does.+ // A parameterized entry is prepared and must be one statement; a+ // parameter-free entry is a script, executed statement by statement the+ // way D1 does.var stmts []requestStatementfor _, entry := range entries {if len(entry.Params) > maxBoundParameters {writeErrors(w, http.StatusBadRequest, 7400, fmt.Sprintf("too many bound parameters: %d", len(entry.Params)))return}- if len(entry.Params) > 0 {- stmts = append(stmts, entry)- continue- }- parts := sqlsplit.Split(entry.SQL)+ parts := splitStatements(entry.SQL)if len(parts) == 0 {writeErrors(w, http.StatusBadRequest, 7400, "empty sql")return}+ if len(entry.Params) > 0 {+ if len(parts) > 1 {+ writeErrors(w, http.StatusBadRequest, 7500, errMultipleStatements)+ return+ }+ stmts = append(stmts, entry)+ continue+ }for _, part := range parts {stmts = append(stmts, requestStatement{SQL: part})}@@ -261,6 +268,12 @@ func (s *Server) handleBridge(w http.ResponseWriter, r *http.Request) {writeBridgeError(w, http.StatusBadRequest, fmt.Sprintf("too many bound parameters: %d", len(entry.Params)))return}+ // Every bridge entry goes through prepare(), which takes exactly one+ // statement.+ if len(splitStatements(entry.SQL)) != 1 {+ writeBridgeError(w, http.StatusBadRequest, "D1_ERROR: "+errMultipleStatements)+ return+ }}results, err := s.execute(r.Context(), body.Statements)if err != nil {@@ -375,6 +388,70 @@ func collectRows(ctx context.Context, tx *sql.Tx, query string, args []any) (*streturn res, nil}+// errMultipleStatements is the text D1 returns for a prepared statement that+// holds more than one statement.+const errMultipleStatements = "A prepared SQL statement must contain only one statement."++// splitStatements divides a script into statements the way SQLite's own+// parser would: sqlite3_complete reports where each statement ends, so+// string literals, comments, and trigger bodies never split a statement, and+// text that is only whitespace or comments yields nothing.+func splitStatements(text string) []string {+ statements := []string{}+ start := 0+ for i := 0; i < len(text); i++ {+ if text[i] != ';' || !sqlComplete(text[start:i+1]) {+ continue+ }+ if part := strings.TrimSpace(text[start : i+1]); !sqlBlank(part) {+ statements = append(statements, part)+ }+ start = i + 1+ }+ if tail := strings.TrimSpace(text[start:]); !sqlBlank(tail) {+ statements = append(statements, tail)+ }+ return statements+}++// sqlComplete reports whether text ends with a complete SQL statement.+func sqlComplete(text string) bool {+ tls := libc.NewTLS()+ defer tls.Close()+ cText, err := libc.CString(text)+ if err != nil {+ return false+ }+ defer libc.Xfree(tls, cText)+ return sqlite3.Xsqlite3_complete(tls, cText) != 0+}++// sqlBlank reports whether text holds no SQL: only whitespace, comments, and+// empty statements.+func sqlBlank(text string) bool {+ for {+ text = strings.TrimLeft(text, " \t\r\n;")+ switch {+ case text == "":+ return true+ case strings.HasPrefix(text, "--"):+ if _, rest, found := strings.Cut(text, "\n"); found {+ text = rest+ } else {+ return true+ }+ case strings.HasPrefix(text, "/*"):+ if _, rest, found := strings.Cut(text[2:], "*/"); found {+ text = rest+ } else {+ return true+ }+ default:+ return false+ }+ }+}+func decodeBody(r *http.Request, target any) error {decoder := json.NewDecoder(r.Body)decoder.UseNumber()diff --git a/store/db/d1/errors.go b/store/db/d1/errors.goindex 27e8db1d4..070b5fbac 100644--- a/store/db/d1/errors.go+++ b/store/db/d1/errors.go@@ -19,14 +19,16 @@ func isUniqueViolation(err error) bool {strings.Contains(message, "SQLITE_CONSTRAINT_PRIMARYKEY")}-// isRetryable reports whether err is a transient D1 failure worth retrying:-// rate limiting, a busy or overloaded database, or a server-side error.+// isRetryable reports whether err is a transient D1 failure worth retrying+// right away: a busy or overloaded database, or a server-side error. A 429+// is deliberately not retried; the Cloudflare API rate limit that produces it+// blocks the caller for minutes, so an immediate retry only deepens the block.func isRetryable(err error) bool {var d1Err *Errorif !errors.As(err, &d1Err) {return false}- if d1Err.Status == http.StatusTooManyRequests || d1Err.Status >= http.StatusInternalServerError {+ if d1Err.Status >= http.StatusInternalServerError {return true}message := d1Err.Messagediff --git a/store/db/d1/executor.go b/store/db/d1/executor.goindex 87842fed2..a385eec03 100644--- a/store/db/d1/executor.go+++ b/store/db/d1/executor.go@@ -3,6 +3,7 @@ package d1import ("context""database/sql"+ "encoding/json""fmt""strconv""strings"@@ -75,6 +76,68 @@ func inClause[T any](values []T) (string, []any) {return "(" + placeholders(len(values)) + ")", args}+// jsonList renders a membership test whose values travel as one JSON array+// binding, so a list of any length costs a single bound parameter and the+// statement keeps its LIMIT and ORDER BY semantics.+func jsonList[T any](values []T) (string, any, error) {+ encoded, err := json.Marshal(values)+ if err != nil {+ return "", nil, errors.Wrap(err, "failed to encode list binding")+ }+ return "(SELECT value FROM json_each(?))", string(encoded), nil+}++// guardIDSet asserts that the rows selected by "FROM <from>" are still+// exactly the ids read before the batch. The count alone would accept a row+// deleted and another inserted in the window; ids only grow, so the highest+// id catches the swap.+func (b *batch) guardIDSet(from string, args []any, ids []int32) {+ var maxID int32+ for _, id := range ids {+ maxID = max(maxID, id)+ }+ condition := "(SELECT COUNT(*) FROM " + from + ") = ? AND COALESCE((SELECT MAX(id) FROM " + from + "), 0) = ?"+ guardArgs := append(append(append([]any{}, args...), len(ids)), append(append([]any{}, args...), maxID)...)+ b.guard(condition, guardArgs...)+}++// scanResultRow copies the first row of a batch result into dest, for+// statements that RETURNING generated values.+func scanResultRow(res *result, dest ...any) error {+ if res == nil || len(res.Rows) == 0 {+ return errors.New("d1: statement returned no row")+ }+ row := res.Rows[0]+ if len(row) != len(dest) {+ return errors.Errorf("d1: statement returned %d columns for %d targets", len(row), len(dest))+ }+ for i, target := range dest {+ switch t := target.(type) {+ case *int32:+ value, ok := row[i].(int64)+ if !ok {+ return errors.Errorf("d1: column %d is %T, not an integer", i, row[i])+ }+ *t = int32(value)+ case *int64:+ value, ok := row[i].(int64)+ if !ok {+ return errors.Errorf("d1: column %d is %T, not an integer", i, row[i])+ }+ *t = value+ case *string:+ value, ok := row[i].(string)+ if !ok {+ return errors.Errorf("d1: column %d is %T, not text", i, row[i])+ }+ *t = value+ default:+ return errors.Errorf("d1: unsupported scan target %T", target)+ }+ }+ return nil+}+// intList renders trusted integer ids as a literal IN list. Inlining them// keeps long id lists under D1's 100 bound parameter limit.func intList(ids []int32) string {diff --git a/store/db/d1/memo.go b/store/db/d1/memo.goindex eabd262ea..03b2c4f11 100644--- a/store/db/d1/memo.go+++ b/store/db/d1/memo.go@@ -25,28 +25,6 @@ type memoState struct {content string}-// memoInlineUIDThreshold is the uid list length above which ListMemos resolves-// uids to ids before querying, leaving bind capacity for the other filters.-const memoInlineUIDThreshold = 40--// memoResolveUIDs maps uids to memo ids in chunks that respect the bind limit.-// Unknown uids are dropped; -1 keeps the IN list valid when none resolve.-func memoResolveUIDs(ctx context.Context, q querier, uids []string) ([]int32, error) {- ids := make([]int32, 0, len(uids))- for _, batch := range chunk(uids, inClauseBatchSize) {- clause, args := inClause(batch)- resolved, err := listMemoIDs(ctx, q, "SELECT id FROM memo WHERE uid IN "+clause, args...)- if err != nil {- return nil, err- }- ids = append(ids, resolved...)- }- if len(ids) == 0 {- ids = append(ids, -1)- }- return ids, nil-}-// loadMemoState reads the memo row; a missing memo yields sql.ErrNoRows.func loadMemoState(ctx context.Context, q querier, memoID int32) (*memoState, error) {state := &memoState{}@@ -165,7 +143,7 @@ func memoLoadCreated(ctx context.Context, q querier, create *store.Memo) error {// ListMemos returns the memos matching find.func (d *DB) ListMemos(ctx context.Context, find *store.FindMemo) ([]*store.Memo, error) {- where, args, err := memoListConditions(ctx, d.db, find)+ where, args, err := memoListConditions(ctx, find)if err != nil {return nil, err}@@ -218,9 +196,8 @@ func (d *DB) ListMemos(ctx context.Context, find *store.FindMemo) ([]*store.Memoreturn list, nil}-// memoListConditions renders the WHERE clauses for find. q serves the uid-// resolution a long uid list needs.-func memoListConditions(ctx context.Context, q querier, find *store.FindMemo) ([]string, []any, error) {+// memoListConditions renders the WHERE clauses for find.+func memoListConditions(ctx context.Context, find *store.FindMemo) ([]string, []any, error) {where, args := []string{"1 = 1"}, []any{}engine, err := filter.DefaultEngine()@@ -240,18 +217,11 @@ func memoListConditions(ctx context.Context, q querier, find *store.FindMemo) ([where, args = append(where, "memo.uid = ?"), append(args, *v)}if len(find.UIDList) > 0 {- // A long uid list would exhaust D1's bind limit together with the- // other filters, so it is resolved to ids first and inlined.- if len(find.UIDList) > memoInlineUIDThreshold {- ids, err := memoResolveUIDs(ctx, q, find.UIDList)- if err != nil {- return nil, nil, err- }- where = append(where, "memo.id IN "+intList(ids))- } else {- clause, uidArgs := inClause(find.UIDList)- where, args = append(where, "memo.uid IN "+clause), append(args, uidArgs...)+ clause, uidArg, err := jsonList(find.UIDList)+ if err != nil {+ return nil, nil, err}+ where, args = append(where, "memo.uid IN "+clause), append(args, uidArg)}if v := find.CreatorID; v != nil {where, args = append(where, "memo.creator_id = ?"), append(args, *v)@@ -342,20 +312,12 @@ func memoScanRow(rows *sql.Rows, excludeContent bool) (*store.Memo, error) {func (d *DB) UpdateMemo(ctx context.Context, update *store.UpdateMemo) error {b := newBatch()if update.Policy != nil {- if err := validateMemoWritePolicy(ctx, d.db, update.ID, update.Policy, update); err != nil {- return err- }- state, err := loadMemoState(ctx, d.db, update.ID)+ state, err := validateMemoWritePolicy(ctx, d.db, update.ID, update.Policy, update)if err != nil {- if errors.Is(err, sql.ErrNoRows) {- return store.ErrMemoMutationConflict- }- return errors.Wrap(err, "failed to read memo")+ return err}b.guard(activeUserCondition, update.Policy.ActorUserID)- condition, args := state.unchangedCondition()- b.guard(condition, args...)- memoGuardPolicyPlacement(b, state, update.Policy, update)+ memoGuardWritePolicy(b, state, update.Policy, update)}if err := memoAddUpdate(b, update); err != nil {return errdiff --git a/store/db/d1/memo_delete_policy.go b/store/db/d1/memo_delete_policy.goindex 88e52502e..0e59d3991 100644--- a/store/db/d1/memo_delete_policy.go+++ b/store/db/d1/memo_delete_policy.go@@ -45,7 +45,7 @@ func (d *DB) DeleteMemoWithPolicy(ctx context.Context, delete *store.DeleteMemoW// The attachments were listed before the batch and are returned for// storage cleanup; abort if the bound set changed so none is orphaned in// the database or missed by the caller.- b.guard("(SELECT COUNT(*) FROM attachment WHERE memo_id = ?) = ?", delete.MemoID, len(attachments))+ b.guardIDSet("attachment WHERE memo_id = ?", []any{delete.MemoID}, attachmentIDs(attachments))addMemoSetDeletes(b, memoIDs, attachmentIDs(attachments))if _, err := b.commit(ctx, d); err != nil {return nil, guardError(err, errors.Wrap(store.ErrMemoMutationConflict, "memo changed while deleting"))diff --git a/store/db/d1/memo_mutation.go b/store/db/d1/memo_mutation.goindex 6fc0d3ca1..17ed4061f 100644--- a/store/db/d1/memo_mutation.go+++ b/store/db/d1/memo_mutation.go@@ -79,13 +79,16 @@ func (d *DB) planMemoMutation(ctx context.Context, mutation *store.MemoMutation)if plan.policy == nil && mutation.MemoUpdate != nil {plan.policy = mutation.MemoUpdate.Policy}+ var validated *memoStateif plan.policy != nil {- if err := validateMemoWritePolicy(ctx, d.db, mutation.MemoID, plan.policy, mutation.MemoUpdate); err != nil {+ state, err := validateMemoWritePolicy(ctx, d.db, mutation.MemoID, plan.policy, mutation.MemoUpdate)+ if err != nil {return nil, err}+ validated = state}if mutation.MemoCreate == nil {- if err := plan.planExistingMemo(ctx, d.db); err != nil {+ if err := plan.planExistingMemo(ctx, d.db, validated); err != nil {return nil, err}}@@ -135,14 +138,20 @@ func (p *memoMutationPlan) planCreate(ctx context.Context, q querier, create *st}// planExistingMemo checks the memo still matches what the caller computed-// the mutation from.-func (p *memoMutationPlan) planExistingMemo(ctx context.Context, q querier) error {- state, err := loadMemoState(ctx, q, p.mutation.MemoID)- if err != nil {- if errors.Is(err, sql.ErrNoRows) {- return errors.Wrap(store.ErrMemoMutationConflict, "memo no longer exists")+// the mutation from. validated is the state the write policy was checked+// against, when there was one; the guards are built from that same read so+// nothing validated is re-read from a different moment.+func (p *memoMutationPlan) planExistingMemo(ctx context.Context, q querier, validated *memoState) error {+ state := validated+ if state == nil {+ loaded, err := loadMemoState(ctx, q, p.mutation.MemoID)+ if err != nil {+ if errors.Is(err, sql.ErrNoRows) {+ return errors.Wrap(store.ErrMemoMutationConflict, "memo no longer exists")+ }+ return errors.Wrap(err, "failed to read memo")}- return errors.Wrap(err, "failed to read memo")+ state = loaded}if state.creatorID != p.mutation.MemoCreatorID || state.content != p.mutation.ExpectedMemoContent {return errors.Wrap(store.ErrMemoMutationConflict, "memo changed while applying mutation")diff --git a/store/db/d1/memo_share.go b/store/db/d1/memo_share.goindex 16d2ed18a..a90b6254d 100644--- a/store/db/d1/memo_share.go+++ b/store/db/d1/memo_share.go@@ -25,19 +25,22 @@ func (d *DB) CreateMemoShare(ctx context.Context, create *store.MemoShare) (*sto}return create, nil}- if err := validateMemoWritePolicy(ctx, d.db, create.MemoID, create.Policy, nil); err != nil {+ state, err := validateMemoWritePolicy(ctx, d.db, create.MemoID, create.Policy, nil)+ if err != nil {return nil, err}- // The validated memo and actor are re-asserted by the insert itself:- // selecting from the memo row and requiring the actor to still be active- // makes a stale validation yield no row instead of a share.- args = append(args, create.MemoID, create.Policy.ActorUserID)- stmt := "INSERT INTO memo_share (" + strings.Join(columns, ", ") + ") SELECT " + strings.Join(values, ", ") +- " FROM memo WHERE memo.id = ? AND " + activeUserCondition + " RETURNING id, created_ts"- if err := d.db.QueryRowContext(ctx, stmt, args...).Scan(&create.ID, &create.CreatedTs); err != nil {- if errors.Is(err, sql.ErrNoRows) {- return nil, store.ErrMemoMutationConflict- }+ // The memo's audience, lifecycle, and placement were all part of the+ // validation, so the batch re-asserts the whole row: a memo moved to the+ // SPACE audience meanwhile must not gain a share.+ b := newBatch()+ b.guard(activeUserCondition, create.Policy.ActorUserID)+ memoGuardWritePolicy(b, state, create.Policy, nil)+ index := b.add("INSERT INTO memo_share ("+strings.Join(columns, ", ")+") VALUES ("+strings.Join(values, ", ")+") RETURNING id, created_ts", args...)+ results, err := b.commit(ctx, d)+ if err != nil {+ return nil, guardError(err, store.ErrMemoMutationConflict)+ }+ if err := scanResultRow(results[index], &create.ID, &create.CreatedTs); err != nil {return nil, err}return create, nil@@ -131,7 +134,7 @@ func (d *DB) DeleteMemoShare(ctx context.Context, delete *store.DeleteMemoShare)_, err := d.execOne(ctx, "DELETE FROM memo_share WHERE "+strings.Join(where, " AND "), args...)return err}- if err := validateMemoWritePolicy(ctx, d.db, *delete.MemoID, delete.Policy, nil); err != nil {+ if _, err := validateMemoWritePolicy(ctx, d.db, *delete.MemoID, delete.Policy, nil); err != nil {return err}// One conditional statement: the share must still belong to the validateddiff --git a/store/db/d1/race_test.go b/store/db/d1/race_test.gonew file mode 100644index 000000000..dcae1d0fd--- /dev/null+++ b/store/db/d1/race_test.go@@ -0,0 +1,316 @@+package d1++import (+ "context"+ "os"+ "path/filepath"+ "strings"+ "testing"++ "github.com/stretchr/testify/require"++ "github.com/usememos/memos/internal/profile"+ "github.com/usememos/memos/store"+ "github.com/usememos/memos/store/db/d1/d1test"+)++// raceTransport wraps a transport and runs action once, right before the+// first request whose SQL contains trigger reaches the database. Driver+// methods validate with reads and then commit a batch, so an action fired on+// the batch changes the database in exactly the window a concurrent writer+// would, and the guards must catch it.+type raceTransport struct {+ transport+ trigger string+ action func()+ fired bool+}++func (t *raceTransport) exec(ctx context.Context, stmt statement) (*result, error) {+ t.fire(stmt.SQL)+ return t.transport.exec(ctx, stmt)+}++func (t *raceTransport) batch(ctx context.Context, stmts []statement) ([]*result, error) {+ sqls := make([]string, 0, len(stmts))+ for _, stmt := range stmts {+ sqls = append(sqls, stmt.SQL)+ }+ t.fire(strings.Join(sqls, "\n"))+ return t.transport.batch(ctx, stmts)+}++func (t *raceTransport) fire(sql string) {+ if t.fired || !strings.Contains(sql, t.trigger) {+ return+ }+ t.fired = true+ t.action()+}++// raceDB is a schema-initialized driver whose transport can be interposed,+// plus a second, independent handle on the same database for the+// interposing action to use.+type raceDB struct {+ *DB+ other *DB+ ctx context.Context+}++func newRaceDB(t *testing.T, dsn func(*d1test.Server) string) *raceDB {+ t.Helper()+ server := d1test.New(t)+ open := func() *DB {+ driver, err := NewDB(&profile.Profile{Driver: "d1", DSN: dsn(server)})+ require.NoError(t, err)+ db, ok := driver.(*DB)+ require.True(t, ok)+ t.Cleanup(func() { require.NoError(t, db.Close()) })+ return db+ }+ db := &raceDB{DB: open(), other: open(), ctx: context.Background()}++ schema, err := os.ReadFile(filepath.Join("..", "..", "migration", "d1", "LATEST.sql"))+ require.NoError(t, err)+ tx, err := db.GetDB().Begin()+ require.NoError(t, err)+ _, err = tx.ExecContext(db.ctx, string(schema))+ require.NoError(t, err)+ require.NoError(t, tx.Commit())+ return db+}++// interpose installs action to run before the first request containing+// trigger. Requests made by other bypass it.+func (db *raceDB) interpose(t *testing.T, trigger string, action func()) {+ t.Helper()+ require.NoError(t, db.db.Close())+ wrapped := &raceTransport{transport: db.transport, trigger: trigger, action: action}+ db.transport = wrapped+ db.db = openSQLDB(wrapped)+ t.Cleanup(func() { require.True(t, wrapped.fired, "the interposed action never ran") })+}++func (db *raceDB) user(t *testing.T, username string) *store.User {+ t.Helper()+ user, err := db.other.CreateUser(db.ctx, &store.User{Username: username, Role: store.RoleUser, PasswordHash: "hash"})+ require.NoError(t, err)+ return user+}++func (db *raceDB) memo(t *testing.T, uid string, creator *store.User) *store.Memo {+ t.Helper()+ memo, err := db.other.CreateMemo(db.ctx, &store.Memo{UID: uid, CreatorID: creator.ID, Content: "original", Visibility: store.Private})+ require.NoError(t, err)+ return memo+}++func (db *raceDB) attachment(t *testing.T, uid string, creator *store.User, memo *store.Memo) *store.Attachment {+ t.Helper()+ attachment, err := db.other.CreateAttachment(db.ctx, &store.Attachment{UID: uid, CreatorID: creator.ID, Filename: uid + ".png", Type: "image/png", MemoID: &memo.ID})+ require.NoError(t, err)+ return attachment+}++func (db *raceDB) archive(t *testing.T, memo *store.Memo) {+ t.Helper()+ archived := store.Archived+ require.NoError(t, db.other.UpdateMemo(db.ctx, &store.UpdateMemo{ID: memo.ID, RowStatus: &archived}))+}++func (db *raceDB) memoContent(t *testing.T, memo *store.Memo) string {+ t.Helper()+ var content string+ require.NoError(t, db.other.GetDB().QueryRowContext(db.ctx, "SELECT content FROM memo WHERE id = ?", memo.ID).Scan(&content))+ return content+}++func (db *raceDB) count(t *testing.T, query string, args ...any) int {+ t.Helper()+ var n int+ require.NoError(t, db.other.GetDB().QueryRowContext(db.ctx, query, args...).Scan(&n))+ return n+}++func forEachAccessMode(t *testing.T, run func(t *testing.T, db *raceDB)) {+ t.Helper()+ for mode, dsn := range accessModes {+ t.Run(mode, func(t *testing.T) {+ run(t, newRaceDB(t, dsn))+ })+ }+}++func TestPolicyInsertsReportGeneratedValues(t *testing.T) {+ forEachAccessMode(t, func(t *testing.T, db *raceDB) {+ user := db.user(t, "alice")+ memo := db.memo(t, "m1", user)+ policy := &store.MemoWritePolicy{ActorUserID: user.ID, CreatingShare: true}++ share, err := db.CreateMemoShare(db.ctx, &store.MemoShare{UID: "s1", MemoID: memo.ID, CreatorID: user.ID, Policy: policy})+ require.NoError(t, err)+ require.NotZero(t, share.ID)+ require.NotZero(t, share.CreatedTs)++ attachment, err := db.CreateAttachment(db.ctx, &store.Attachment{UID: "a1", CreatorID: user.ID, MemoID: &memo.ID, Policy: &store.MemoWritePolicy{ActorUserID: user.ID}})+ require.NoError(t, err)+ require.NotZero(t, attachment.ID)+ require.NotZero(t, attachment.CreatedTs)+ require.NotZero(t, attachment.UpdatedTs)+ })+}++func TestShareCreateAbortsWhenMemoJoinsSpaceAudience(t *testing.T) {+ forEachAccessMode(t, func(t *testing.T, db *raceDB) {+ user := db.user(t, "alice")+ space, err := db.other.CreateSpace(db.ctx, &store.Space{UID: "space", Title: "Space"}, user.ID)+ require.NoError(t, err)+ memo := db.memo(t, "m1", user)++ // The memo is moved to the SPACE audience after the share was+ // validated against a private memo; the insert must not land.+ db.interpose(t, "INSERT INTO memo_share", func() {+ visibility := store.SpaceAudience+ require.NoError(t, db.other.UpdateMemo(db.ctx, &store.UpdateMemo{ID: memo.ID, Visibility: &visibility, SpaceID: &space.ID}))+ })+ _, err = db.CreateMemoShare(db.ctx, &store.MemoShare{UID: "s1", MemoID: memo.ID, CreatorID: user.ID, Policy: &store.MemoWritePolicy{ActorUserID: user.ID, CreatingShare: true}})+ require.ErrorIs(t, err, store.ErrMemoMutationConflict)+ require.Equal(t, 0, db.count(t, "SELECT COUNT(*) FROM memo_share"))+ })+}++func TestUpdateMemoAbortsWhenMemoChanges(t *testing.T) {+ forEachAccessMode(t, func(t *testing.T, db *raceDB) {+ user := db.user(t, "alice")+ memo := db.memo(t, "m1", user)++ db.interpose(t, "UPDATE memo SET", func() { db.archive(t, memo) })+ content := "edited"+ err := db.UpdateMemo(db.ctx, &store.UpdateMemo{ID: memo.ID, Content: &content, Policy: &store.MemoWritePolicy{ActorUserID: user.ID}})+ require.ErrorIs(t, err, store.ErrMemoMutationConflict)+ require.Equal(t, "original", db.memoContent(t, memo))+ })+}++func TestMemoMutationAbortsWhenMemoChanges(t *testing.T) {+ forEachAccessMode(t, func(t *testing.T, db *raceDB) {+ user := db.user(t, "alice")+ memo := db.memo(t, "m1", user)++ db.interpose(t, "UPDATE memo SET", func() { db.archive(t, memo) })+ content := "edited"+ err := db.ApplyMemoMutation(db.ctx, &store.MemoMutation{+ MemoID: memo.ID,+ MemoCreatorID: user.ID,+ ExpectedMemoContent: "original",+ MemoUpdate: &store.UpdateMemo{ID: memo.ID, Content: &content},+ Policy: &store.MemoWritePolicy{ActorUserID: user.ID},+ })+ require.ErrorIs(t, err, store.ErrMemoMutationConflict)+ require.Equal(t, "original", db.memoContent(t, memo))+ })+}++func TestAttachmentCreateAbortsWhenMemoChanges(t *testing.T) {+ forEachAccessMode(t, func(t *testing.T, db *raceDB) {+ user := db.user(t, "alice")+ memo := db.memo(t, "m1", user)++ db.interpose(t, "INSERT INTO attachment", func() { db.archive(t, memo) })+ _, err := db.CreateAttachment(db.ctx, &store.Attachment{UID: "a1", CreatorID: user.ID, MemoID: &memo.ID, Policy: &store.MemoWritePolicy{ActorUserID: user.ID}})+ require.ErrorIs(t, err, store.ErrMemoMutationConflict)+ require.Equal(t, 0, db.count(t, "SELECT COUNT(*) FROM attachment"))+ })+}++func TestAttachmentUpdateAbortsWhenBindingChanges(t *testing.T) {+ forEachAccessMode(t, func(t *testing.T, db *raceDB) {+ user := db.user(t, "alice")+ memo := db.memo(t, "m1", user)+ attachment := db.attachment(t, "a1", user, memo)++ // The attachment is unbound from the memo the actor was authorized+ // against before the update commits.+ db.interpose(t, "UPDATE attachment SET", func() {+ _, err := db.other.GetDB().ExecContext(db.ctx, "UPDATE attachment SET memo_id = NULL WHERE id = ?", attachment.ID)+ require.NoError(t, err)+ })+ filename := "renamed.png"+ err := db.UpdateAttachment(db.ctx, &store.UpdateAttachment{ID: attachment.ID, Filename: &filename, Policy: &store.MemoWritePolicy{ActorUserID: user.ID}})+ require.ErrorIs(t, err, store.ErrMemoPermissionDenied)+ require.Equal(t, 0, db.count(t, "SELECT COUNT(*) FROM attachment WHERE filename = ?", filename))+ })+}++func TestAttachmentDeleteAbortsWhenMemoContentChanges(t *testing.T) {+ forEachAccessMode(t, func(t *testing.T, db *raceDB) {+ user := db.user(t, "alice")+ memo := db.memo(t, "m1", user)+ attachment := db.attachment(t, "a1", user, memo)++ db.interpose(t, "DELETE FROM attachment", func() {+ content := "edited"+ require.NoError(t, db.other.UpdateMemo(db.ctx, &store.UpdateMemo{ID: memo.ID, Content: &content}))+ })+ policy := &store.AttachmentDeletionPolicy{ActorUserID: user.ID, ExpectedMemoContents: map[int32]string{memo.ID: "original"}}+ err := db.DeleteAttachmentsWithPolicy(db.ctx, policy, []int32{attachment.ID})+ require.ErrorIs(t, err, store.ErrMemoMutationConflict)+ require.Equal(t, 1, db.count(t, "SELECT COUNT(*) FROM attachment WHERE id = ?", attachment.ID))+ })+}++func TestMemoDeleteAbortsWhenAttachmentSetIsSwapped(t *testing.T) {+ forEachAccessMode(t, func(t *testing.T, db *raceDB) {+ user := db.user(t, "alice")+ memo := db.memo(t, "m1", user)+ attachment := db.attachment(t, "a1", user, memo)++ // One attachment is replaced by another: the count the deletion read+ // still matches, the set does not.+ db.interpose(t, "DELETE FROM memo WHERE", func() {+ require.NoError(t, db.other.DeleteAttachment(db.ctx, &store.DeleteAttachment{ID: attachment.ID}))+ db.attachment(t, "a2", user, memo)+ })+ _, err := db.DeleteMemoWithPolicy(db.ctx, &store.DeleteMemoWithPolicy{MemoID: memo.ID, ActorUserID: user.ID})+ require.ErrorIs(t, err, store.ErrMemoMutationConflict)+ require.Equal(t, 1, db.count(t, "SELECT COUNT(*) FROM memo WHERE id = ?", memo.ID))+ require.Equal(t, 1, db.count(t, "SELECT COUNT(*) FROM attachment WHERE memo_id = ? AND uid = 'a2'", memo.ID))+ })+}++func TestUserDeleteAbortsWhenMemoSetIsSwapped(t *testing.T) {+ forEachAccessMode(t, func(t *testing.T, db *raceDB) {+ user := db.user(t, "bob")+ memo := db.memo(t, "m1", user)++ db.interpose(t, "DELETE FROM user WHERE", func() {+ require.NoError(t, db.other.DeleteMemo(db.ctx, &store.DeleteMemo{ID: memo.ID}))+ db.memo(t, "m2", user)+ })+ _, err := db.DeleteUser(db.ctx, &store.DeleteUser{ID: user.ID})+ require.ErrorIs(t, err, errUserDeleteConflict)+ require.Equal(t, 1, db.count(t, "SELECT COUNT(*) FROM user WHERE id = ?", user.ID))+ require.Equal(t, 1, db.count(t, "SELECT COUNT(*) FROM memo WHERE creator_id = ?", user.ID))+ })+}++func TestSpaceDeleteAbortsWhenMemoSetIsSwapped(t *testing.T) {+ forEachAccessMode(t, func(t *testing.T, db *raceDB) {+ user := db.user(t, "alice")+ space, err := db.other.CreateSpace(db.ctx, &store.Space{UID: "space", Title: "Space"}, user.ID)+ require.NoError(t, err)+ visibility := store.SpaceAudience+ memo, err := db.other.CreateMemo(db.ctx, &store.Memo{UID: "m1", CreatorID: user.ID, Content: "original", Visibility: visibility, SpaceID: &space.ID})+ require.NoError(t, err)++ db.interpose(t, "DELETE FROM space WHERE", func() {+ require.NoError(t, db.other.DeleteMemo(db.ctx, &store.DeleteMemo{ID: memo.ID}))+ _, err := db.other.CreateMemo(db.ctx, &store.Memo{UID: "m2", CreatorID: user.ID, Content: "later", Visibility: visibility, SpaceID: &space.ID})+ require.NoError(t, err)+ })+ _, err = db.DeleteSpace(db.ctx, &store.DeleteSpace{ID: space.ID, ActorUserID: user.ID})+ require.ErrorIs(t, err, errSpaceDeleteConflict)+ require.Equal(t, 1, db.count(t, "SELECT COUNT(*) FROM space WHERE id = ?", space.ID))+ require.Equal(t, 1, db.count(t, "SELECT COUNT(*) FROM memo WHERE space_id = ?", space.ID))+ })+}diff --git a/store/db/d1/reaction.go b/store/db/d1/reaction.goindex 9272979be..6927a8499 100644--- a/store/db/d1/reaction.go+++ b/store/db/d1/reaction.go@@ -3,7 +3,6 @@ package d1import ("context""database/sql"- "slices""strings""github.com/pkg/errors"@@ -105,25 +104,14 @@ func (d *DB) ListReactions(ctx context.Context, find *store.FindReaction) ([]*stif find.MemoID != nil {where, args = append(where, "memo_id = ?"), append(args, *find.MemoID)}- if len(find.MemoIDList) == 0 {- return reactionQuery(ctx, d.db, where, args)- }-- // A memo id list can exceed D1's bind limit, so it is queried in chunks- // and the union re-sorted to preserve the id order of a single query.- list := []*store.Reaction{}- for _, ids := range chunk(find.MemoIDList, inClauseBatchSize) {- clause, memoArgs := inClause(ids)- chunkWhere := append(append([]string{}, where...), "memo_id IN "+clause)- chunkArgs := append(append([]any{}, args...), memoArgs...)- reactions, err := reactionQuery(ctx, d.db, chunkWhere, chunkArgs)+ if len(find.MemoIDList) > 0 {+ clause, memoArg, err := jsonList(find.MemoIDList)if err != nil {return nil, err}- list = append(list, reactions...)+ where, args = append(where, "memo_id IN "+clause), append(args, memoArg)}- slices.SortFunc(list, func(a, b *store.Reaction) int { return int(a.ID) - int(b.ID) })- return list, nil+ return reactionQuery(ctx, d.db, where, args)}// reactionQuery lists the reactions matching where, ordered by id.diff --git a/store/db/d1/shared.go b/store/db/d1/shared.goindex 056c933cc..0716a209b 100644--- a/store/db/d1/shared.go+++ b/store/db/d1/shared.go@@ -75,27 +75,29 @@ func spaceState(ctx context.Context, q querier, spaceID, userID int32) (exists b}// validateMemoWritePolicy checks that the actor may write memoID under policy,-// optionally for the pending update.-func validateMemoWritePolicy(ctx context.Context, q querier, memoID int32, policy *store.MemoWritePolicy, update *store.UpdateMemo) error {+// optionally for the pending update, and returns the memo state the check was+// made against so the batch can guard on exactly that state.+func validateMemoWritePolicy(ctx context.Context, q querier, memoID int32, policy *store.MemoWritePolicy, update *store.UpdateMemo) (*memoState, error) {if err := requireActiveUser(ctx, q, policy.ActorUserID, store.ErrMemoSpaceMembershipRequired); err != nil {- return err+ return nil, err}-- snapshot := new(store.MemoWriteSnapshot)- var spaceID sql.NullInt64- if err := q.QueryRowContext(ctx, `SELECT creator_id, row_status, space_id, visibility FROM memo WHERE id = ?`, memoID).Scan(- &snapshot.CreatorID, &snapshot.RowStatus, &spaceID, &snapshot.Visibility,- ); err != nil {+ state, err := loadMemoState(ctx, q, memoID)+ if err != nil {if errors.Is(err, sql.ErrNoRows) {- return store.ErrMemoMutationConflict+ return nil, store.ErrMemoMutationConflict}- return err+ return nil, err+ }+ snapshot := &store.MemoWriteSnapshot{+ CreatorID: state.creatorID,+ RowStatus: state.rowStatus,+ SpaceID: state.spaceID,+ Visibility: state.visibility,}- snapshot.SpaceID = store.NullInt32Pointer(spaceID)if snapshot.SpaceID != nil {exists, member, err := spaceState(ctx, q, *snapshot.SpaceID, policy.ActorUserID)if err != nil {- return err+ return nil, err}snapshot.SourceSpaceExists = existssnapshot.SourceMemberActive = member@@ -103,7 +105,7 @@ func validateMemoWritePolicy(ctx context.Context, q querier, memoID int32, policif update != nil && update.SpaceID != nil {exists, member, err := spaceState(ctx, q, *update.SpaceID, policy.ActorUserID)if err != nil {- return err+ return nil, err}snapshot.TargetSpaceExists = existssnapshot.TargetMemberActive = member@@ -115,10 +117,24 @@ func validateMemoWritePolicy(ctx context.Context, q querier, memoID int32, policLIMIT 1`, memoID).Scan(&shareID)snapshot.HasActiveShare = err == nilif err != nil && !errors.Is(err, sql.ErrNoRows) {- return err+ return nil, err}}- return store.ValidateMemoWriteSnapshot(policy, update, snapshot)+ if err := store.ValidateMemoWriteSnapshot(policy, update, snapshot); err != nil {+ return nil, err+ }+ return state, nil+}++// memoGuardWritePolicy re-asserts inside the batch everything+// validateMemoWritePolicy accepted: the memo row is unchanged, its Space+// still exists, the actor keeps the memberships the policy needs, and no+// active share blocks a move to the SPACE audience. The caller guards the+// actor's own row once.+func memoGuardWritePolicy(b *batch, state *memoState, policy *store.MemoWritePolicy, update *store.UpdateMemo) {+ condition, args := state.unchangedCondition()+ b.guard(condition, args...)+ memoGuardPolicyPlacement(b, state, policy, update)}// loadMemoParticipation resolves the actor, Space, membership, and memo state@@ -267,40 +283,61 @@ func listMemoSetAttachments(ctx context.Context, q querier, memoIDs []int32) ([]}// authorizeAttachmentMutation checks the actor may mutate attachments bound to-// memoIDs, optionally requiring the memos' current contents to match.-func authorizeAttachmentMutation(ctx context.Context, q querier, actorUserID int32, memoIDs []int32, expectedMemoContents map[int32]string) error {+// memoIDs, optionally requiring the memos' current contents to match, and+// returns the memo states the checks were made against.+func authorizeAttachmentMutation(ctx context.Context, q querier, actorUserID int32, memoIDs []int32, expectedMemoContents map[int32]string) ([]*memoState, error) {if err := requireActiveUser(ctx, q, actorUserID, store.ErrMemoPermissionDenied); err != nil {- return err+ return nil, err}+ states := make([]*memoState, 0, len(memoIDs))for _, memoID := range memoIDs {- snapshot := &store.MemoWriteSnapshot{}- var currentSpace sql.NullInt64- var content string- if err := q.QueryRowContext(ctx, `SELECT creator_id, row_status, space_id, visibility, content FROM memo WHERE id = ?`, memoID).Scan(- &snapshot.CreatorID, &snapshot.RowStatus, ¤tSpace, &snapshot.Visibility, &content,- ); err != nil {+ state, err := loadMemoState(ctx, q, memoID)+ if err != nil {if errors.Is(err, sql.ErrNoRows) {- return store.ErrMemoMutationConflict+ return nil, store.ErrMemoMutationConflict}- return errors.Wrap(err, "failed to read attachment memo")+ return nil, errors.Wrap(err, "failed to read attachment memo")+ }+ if expectedMemoContents != nil && state.content != expectedMemoContents[memoID] {+ return nil, store.ErrMemoMutationConflict}- snapshot.SpaceID = store.NullInt32Pointer(currentSpace)- if expectedMemoContents != nil && content != expectedMemoContents[memoID] {- return store.ErrMemoMutationConflict+ snapshot := &store.MemoWriteSnapshot{+ CreatorID: state.creatorID,+ RowStatus: state.rowStatus,+ SpaceID: state.spaceID,+ Visibility: state.visibility,}if snapshot.SpaceID != nil {exists, member, err := spaceState(ctx, q, *snapshot.SpaceID, actorUserID)if err != nil {- return errors.Wrap(err, "failed to read attachment memo space")+ return nil, errors.Wrap(err, "failed to read attachment memo space")}snapshot.SourceSpaceExists = existssnapshot.SourceMemberActive = member}if err := store.ValidateMemoWriteSnapshot(&store.MemoWritePolicy{ActorUserID: actorUserID}, nil, snapshot); err != nil {- return err+ return nil, err}+ states = append(states, state)+ }+ return states, nil+}++// guardAttachmentMutation re-asserts what authorizeAttachmentMutation+// accepted for every linked memo and pins each attachment to the binding it+// was read with. With expectedMemoContents set, the memo content the caller+// computed the mutation from must still be current at commit time.+func guardAttachmentMutation(b *batch, actorUserID int32, states []*memoState, attachments []*store.Attachment, expectedMemoContents map[int32]string) {+ b.guard(activeUserCondition, actorUserID)+ for _, state := range states {+ memoGuardWritePolicy(b, state, &store.MemoWritePolicy{ActorUserID: actorUserID}, nil)+ if expectedMemoContents != nil {+ b.guard("EXISTS (SELECT 1 FROM memo WHERE id = ? AND content = ?)", state.id, expectedMemoContents[state.id])+ }+ }+ for _, attachment := range attachments {+ b.guard("EXISTS (SELECT 1 FROM attachment WHERE id = ? AND memo_id IS ?)", attachment.ID, attachment.MemoID)}- return nil}// attachmentIDs extracts the ids of attachments.diff --git a/store/db/d1/space.go b/store/db/d1/space.goindex 4110f4e4c..afef05abb 100644--- a/store/db/d1/space.go+++ b/store/db/d1/space.go@@ -135,8 +135,9 @@ func (d *DB) CreateSpace(ctx context.Context, create *store.Space, creatorID intif err != nil {return nil, err}- // The membership row cannot use the id generated by the first insert- // because a batch never reads its own writes; the unique uid resolves it.+ // The batch runs in order inside one transaction, so the membership row+ // can find the space inserted just before it; it resolves the row through+ // the unique uid because the generated id is only reported after commit.b := newBatch()b.guard(activeUserCondition, creatorID)b.add("INSERT INTO space (uid, title, description, payload) VALUES (?, ?, ?, ?)", create.UID, create.Title, create.Description, payload)@@ -170,8 +171,11 @@ func (d *DB) ListSpaces(ctx context.Context, find *store.FindSpace) ([]*store.Spwhere, args = append(where, "space.id = ?"), append(args, *find.ID)}if len(find.IDList) > 0 {- clause, ids := inClause(find.IDList)- where, args = append(where, "space.id IN "+clause), append(args, ids...)+ clause, ids, err := jsonList(find.IDList)+ if err != nil {+ return nil, err+ }+ where, args = append(where, "space.id IN "+clause), append(args, ids)}if find.UID != nil {where, args = append(where, "space.uid = ?"), append(args, *find.UID)diff --git a/store/db/d1/space_delete.go b/store/db/d1/space_delete.goindex 050c07a21..488c95bd3 100644--- a/store/db/d1/space_delete.go+++ b/store/db/d1/space_delete.go@@ -63,8 +63,8 @@ func (d *DB) DeleteSpace(ctx context.Context, delete *store.DeleteSpace) (*storeb.guard("EXISTS (SELECT 1 FROM space_member WHERE space_id = ? AND user_id = ? AND status = 'ACTIVE' AND role = 'ADMIN')", delete.ID, delete.ActorUserID)// The memo set was read outside the batch; abort when it changed so no// memo or attachment is left pointing at a deleted space.- b.guard("(SELECT COUNT(*) FROM memo WHERE space_id = ?) = ?", delete.ID, len(memoIDs))- b.guard("(SELECT COUNT(*) FROM attachment WHERE memo_id IN (SELECT id FROM memo WHERE space_id = ?)) = ?", delete.ID, len(attachments))+ b.guardIDSet("memo WHERE space_id = ?", []any{delete.ID}, memoIDs)+ b.guardIDSet("attachment WHERE memo_id IN (SELECT id FROM memo WHERE space_id = ?)", []any{delete.ID}, attachmentIDs(attachments))addMemoSetDeletes(b, memoIDs, attachmentIDs(attachments))b.add("DELETE FROM space_member WHERE space_id = ?", delete.ID)b.add("DELETE FROM space WHERE id = ?", delete.ID)diff --git a/store/db/d1/sqldriver.go b/store/db/d1/sqldriver.goindex 33f36b4fc..1ea8023ab 100644--- a/store/db/d1/sqldriver.go+++ b/store/db/d1/sqldriver.go@@ -161,11 +161,14 @@ func (t *tx) Commit() error {}ctx := context.Background()if parameterFree(t.pending) {+ // Each buffered text is a script that already follows the one+ // statement per paragraph rule, so a blank line joins them into one+ // script that still does.sqls := make([]string, 0, len(t.pending))for _, stmt := range t.pending {sqls = append(sqls, strings.TrimSpace(stmt.SQL))}- return t.conn.transport.script(ctx, strings.Join(sqls, "\n;\n"))+ return t.conn.transport.script(ctx, strings.Join(sqls, "\n\n"))}_, err := t.conn.transport.batch(ctx, t.pending)return errdiff --git a/store/db/d1/user.go b/store/db/d1/user.goindex 69ef0f669..71a321e8b 100644--- a/store/db/d1/user.go+++ b/store/db/d1/user.go@@ -168,12 +168,18 @@ func (d *DB) ListUsers(ctx context.Context, find *store.FindUser) ([]*store.Userwhere, args = append(where, "id = ?"), append(args, *v)}if len(find.IDList) > 0 {- clause, ids := inClause(find.IDList)- where, args = append(where, "id IN "+clause), append(args, ids...)+ clause, ids, err := jsonList(find.IDList)+ if err != nil {+ return nil, err+ }+ where, args = append(where, "id IN "+clause), append(args, ids)}if len(find.UsernameList) > 0 {- clause, usernames := inClause(find.UsernameList)- where, args = append(where, "username IN "+clause), append(args, usernames...)+ clause, usernames, err := jsonList(find.UsernameList)+ if err != nil {+ return nil, err+ }+ where, args = append(where, "username IN "+clause), append(args, usernames)}if v := find.RowStatus; v != nil {where, args = append(where, "row_status = ?"), append(args, *v)diff --git a/store/db/d1/user_delete.go b/store/db/d1/user_delete.goindex fe2cf6423..7a0d66c6f 100644--- a/store/db/d1/user_delete.go+++ b/store/db/d1/user_delete.go@@ -108,8 +108,8 @@ func (d *DB) DeleteUser(ctx context.Context, delete *store.DeleteUser) (*store.Db.guard("NOT EXISTS (SELECT 1 FROM space_member WHERE user_id = ? AND status <> 'INVITED')", delete.ID)// The memo and attachment sets were read outside the batch; abort when// they changed so nothing authored by the user survives its deletion.- b.guard("(SELECT COUNT(*) FROM memo WHERE creator_id = ?) = ?", delete.ID, len(targets.memoIDs))- b.guard("(SELECT COUNT(*) FROM attachment WHERE creator_id = ? OR memo_id IN (SELECT id FROM memo WHERE creator_id = ?)) = ?", delete.ID, delete.ID, len(targets.attachments))+ b.guardIDSet("memo WHERE creator_id = ?", []any{delete.ID}, targets.memoIDs)+ b.guardIDSet("attachment WHERE creator_id = ? OR memo_id IN (SELECT id FROM memo WHERE creator_id = ?)", []any{delete.ID, delete.ID}, attachmentIDs(targets.attachments))b.add("DELETE FROM space_member WHERE user_id = ? AND status = 'INVITED'", delete.ID)addMemoSetDeletes(b, targets.memoIDs, attachmentIDs(targets.attachments))b.add("DELETE FROM reaction WHERE creator_id = ?", delete.ID)diff --git a/store/db/d1/user_identity.go b/store/db/d1/user_identity.goindex 98d5301ee..bcdc298e8 100644--- a/store/db/d1/user_identity.go+++ b/store/db/d1/user_identity.go@@ -20,10 +20,11 @@ func (d *DB) CreateUserIdentity(ctx context.Context, create *store.UserIdentity)return create, nil}-// CreateUserWithIdentity inserts both rows in one batch. The identity cannot-// use the id generated by the user insert, because a batch never reads its-// own writes, so it resolves the user through the unique username and both-// rows are read back after the commit.+// CreateUserWithIdentity inserts both rows in one batch. The batch runs in+// order inside one transaction, so the identity insert can find the user row+// inserted just before it; it resolves the user through the unique username+// because the generated id is only reported after commit, and both rows are+// read back afterwards.func (d *DB) CreateUserWithIdentity(ctx context.Context, createUser *store.User, createIdentity *store.UserIdentity) (*store.User, error) {b := newBatch()b.add("INSERT INTO user (username, role, email, nickname, password_hash, avatar_url) VALUES (?, ?, ?, ?, ?, ?)",diff --git a/store/migration/d1/LATEST.sql b/store/migration/d1/LATEST.sqlindex 6c8b718fa..88401e7ab 100644--- a/store/migration/d1/LATEST.sql+++ b/store/migration/d1/LATEST.sql@@ -76,6 +76,7 @@ CREATE TABLE memo ();CREATE INDEX idx_memo_creator_id ON memo(creator_id);+CREATE INDEX idx_memo_space_id ON memo(space_id, row_status, created_ts DESC, id DESC);-- memo_relationdiff --git a/store/test/list_id_list_test.go b/store/test/list_id_list_test.gonew file mode 100644index 000000000..8da96b831--- /dev/null+++ b/store/test/list_id_list_test.go@@ -0,0 +1,79 @@+package test++import (+ "context"+ "fmt"+ "testing"++ "github.com/stretchr/testify/require"++ "github.com/usememos/memos/store"+)++// TestListsAcceptLongIDLists lists by id lists far longer than the hundred+// bound parameters Cloudflare D1 allows per statement. Every driver must+// accept them: the API hands page-sized lists, up to a thousand entries, to+// these lookups.+func TestListsAcceptLongIDLists(t *testing.T) {+ ctx := context.Background()+ ts := NewTestingStore(ctx, t)+ user, err := createTestingHostUser(ctx, ts)+ require.NoError(t, err)+ space, err := ts.CreateSpace(ctx, &store.Space{UID: "long-lists", Title: "Long lists"}, user.ID)+ require.NoError(t, err)+ memo, err := ts.CreateMemo(ctx, &store.Memo{UID: "long-lists-memo", CreatorID: user.ID, Content: "listed", Visibility: store.Private})+ require.NoError(t, err)+ attachment, err := ts.CreateAttachment(ctx, &store.Attachment{UID: "long-lists-attachment", CreatorID: user.ID, Filename: "listed.png", Type: "image/png", MemoID: &memo.ID})+ require.NoError(t, err)+ reaction, err := ts.UpsertReaction(ctx, &store.Reaction{CreatorID: user.ID, MemoID: memo.ID, ReactionType: "👍"})+ require.NoError(t, err)++ const length = 250+ ids := make([]int32, 0, length)+ uids := make([]string, 0, length)+ usernames := make([]string, 0, length)+ for i := 1; i <= length; i++ {+ ids = append(ids, int32(i))+ uids = append(uids, fmt.Sprintf("absent-%d", i))+ usernames = append(usernames, fmt.Sprintf("absent-%d", i))+ }+ uids[0] = memo.UID+ usernames[0] = user.Username++ memos, err := ts.ListMemos(ctx, &store.FindMemo{UIDList: uids})+ require.NoError(t, err)+ require.Len(t, memos, 1)+ require.Equal(t, memo.ID, memos[0].ID)++ memos, err = ts.ListMemos(ctx, &store.FindMemo{IDList: ids})+ require.NoError(t, err)+ require.Len(t, memos, 1)++ attachments, err := ts.ListAttachments(ctx, &store.FindAttachment{MemoIDList: ids})+ require.NoError(t, err)+ require.Len(t, attachments, 1)+ require.Equal(t, attachment.ID, attachments[0].ID)++ users, err := ts.ListUsers(ctx, &store.FindUser{IDList: ids})+ require.NoError(t, err)+ require.Len(t, users, 1)+ require.Equal(t, user.ID, users[0].ID)++ users, err = ts.ListUsers(ctx, &store.FindUser{UsernameList: usernames})+ require.NoError(t, err)+ require.Len(t, users, 1)++ spaces, err := ts.ListSpaces(ctx, &store.FindSpace{IDList: ids})+ require.NoError(t, err)+ require.Len(t, spaces, 1)+ require.Equal(t, space.ID, spaces[0].ID)++ reactions, err := ts.ListReactions(ctx, &store.FindReaction{MemoIDList: ids})+ require.NoError(t, err)+ require.Len(t, reactions, 1)+ require.Equal(t, reaction.ID, reactions[0].ID)++ relations, err := ts.ListMemoRelations(ctx, &store.FindMemoRelation{MemoIDList: ids})+ require.NoError(t, err)+ require.Empty(t, relations)+}