An open-source, privacy-first, self-hosted knowledge workspace where humans and AI agents work together 开源、隐私优先、自托管的知识工作空间,让人与智能体在此协作
TypeScript agentic-aiai-agentknowledge-baselocal-firstmarkdownmcpmindmapnote-takingself-hostedsiyuanworkspace
:zap: Improve incremental indexing and database template evaluation https://github.com/siyuan-note/siyuan/issues/20692
Daniel7888f0868c4f
diff --git a/docs/PERFORMANCE.md b/docs/PERFORMANCE.mdindex 2e757411fd..ef56246e95 100644--- a/docs/PERFORMANCE.md+++ b/docs/PERFORMANCE.md@@ -20,7 +20,9 @@ Records start with `siyuan-performance ` followed by JSON. `duration_us` and `st| `document.load` | Transaction barrier, load/parse tree, locate blocks, render response; block count || `document.save` | Prepare/serialize, encrypt, compare, write, notify; prepared and written bytes || `index.flush` | Lock wait, take queue, begin/execute/commit, notify, persist queue completion; batch size and oldest enqueue age |+| `index.update` | Full tree projection, block comparison, auxiliary-index comparison, writes; total and changed blocks, planned auxiliary deletions and insertions || `database.view`, `database.fields`, `database.rows` | Load and sync wait, field calculation, filter, sort, aggregation, pagination, conditional colors; row counts before filtering, after filtering and after pagination |+| `database.templates` | Template evaluation; evaluated values, including attempts that report template errors; `database.fields` also counts required and deferred template fields when a plan is used || `search`, `search.fts`, `search.path`, `search.regexp`, `search.document` | Query, result processing, exact count, grouping/references; method, returned results, matches and cancellation || `editor.content` | Parse/normalize, DOM insertion, synchronous render dispatch, global state; render roots and loaded top-level blocks || `editor.databases` | Elapsed time until serial database initialization finishes; candidate root count |@@ -29,6 +31,10 @@ The initial stage is named `prepare`. Repeated stages accumulate time. A canceleNested operations overlap: do not add a parent duration to its children. `editor.content` measures synchronous work and dispatch, excluding later math/chart script loading, embedded queries and browser layout. `editor.databases` includes asynchronous database waits and does not represent all components becoming visible. Global heading numbering, fold state, readonly state, focus and plugin events retain their existing scope.+Auxiliary-index updates compare complete business-field snapshots in the existing transaction. Equal per-block multisets retain their stored IDs; changed blocks replace only their affected table records. Full projection remains necessary for reference and asset metadata that can change independently of a block hash. Counters describe planned row writes, including operations in transactions that subsequently fail; they are not committed-row totals.++Ordinary table, list and gallery views can evaluate templates needed by filters, sorting, calculations and conditional colors, plus their transitive dependencies, before pagination. Other templates run on the current page. Both passes share original raw-value snapshots and preserve required display output. Search, grouped layouts, unresolved/cyclic dependencies, opaque template expressions and dependencies that would change the original evaluation order retain full evaluation. Existing views eligible for deferring every template keep that behavior.+Export kernel and browser captures separately as UTF-8 files. From `app/`, summarize each run with:```text@@ -45,6 +51,8 @@ Run from `kernel/` using the same Go toolchain, machine, power settings and othego test ./filesys -run '^$' -bench '^BenchmarkDocumentJSON$' -benchmem -benchtime=200ms -count=3go test ./model -run '^$' -bench '^BenchmarkDatabaseDisplayPage$' -benchmem -benchtime=200ms -count=3go test ./model -run '^$' -bench '^BenchmarkDocumentSearch$' -benchmem -benchtime=200ms -count=3+go test -tags fts5 ./sql -run '^$' -bench '^BenchmarkUpsertTreeAuxiliaryIndexes$' -benchmem -benchtime=200ms -count=3+go test ./model -run '^$' -bench '^BenchmarkDatabaseTemplateDependencies$' -benchmem -benchtime=200ms -count=3```| Benchmark | Fixed workload | Included work |@@ -52,6 +60,8 @@ go test ./model -run '^$' -bench '^BenchmarkDocumentSearch$' -benchmem -benchtim| `BenchmarkDocumentJSON` | 1,000 / 5,000 / 20,000 paragraph blocks, 112 ASCII content bytes per block, no references/assets, flat structure | Document preparation/JSON serialization or JSON parsing/validation; preparation uses a temporary directory; no product binary or workspace is replaced || `BenchmarkDatabaseDisplayPage` | 1,000 / 10,000 / 50,000 detached rows, block/text/display-template fields, page size 50, no filter/group/sort | Render fields using the existing display-template deferral, paginate and evaluate the current page's template || `BenchmarkDocumentSearch` | 10,000 / 100,000 / 1,000,000 indexed blocks, 100 blocks per document, two keywords in separate paragraphs, every document matches, first 50 documents | Warm in-memory SQLite document query, exact count and result-row decoding; no FTS, disk I/O, full API formatting or frontend |+| `BenchmarkUpsertTreeAuxiliaryIndexes` | 1,000 / 5,000 / 20,000 paragraphs with a tag and custom attribute per block, one document tag; alternate one paragraph's ordinary text | Complete auxiliary replacement versus incremental comparison and writes, including identical full projections, block/FTS updates and transaction commit; SQLite `total_changes()` delta is reported as `sqlite_changes/op` and includes internal FTS writes |+| `BenchmarkDatabaseTemplateDependencies` | 1,000 / 10,000 / 50,000 detached rows, two dependent templates required by filter/sort and five display templates, page size 50 | Complete versus dependency-based evaluation, including plan creation, field rendering, filter/sort and pagination; each iteration clears the rendered-collection cache |Use sub-benchmark filters for smaller runs, for example `-bench '^BenchmarkDocumentSearch$/^10000$'`. Database setup and fixture generation are outside measured loops. The database benchmark reuses the attribute view across iterations; it is not a cold-open measurement. `-benchmem` measures Go allocations, not SQLite native allocations or total RSS. Benchmark results are baselines for those paths, not application latency or a general speedup claim.@@ -81,6 +91,11 @@ Focused checks from `kernel/`:go test -tags fts5 ./model -run 'TestFullTextSearchRejectsCanceledRequest|TestSearchContextPreservesResultsCountsAndPagination' -count=1go test ./sql -run 'TestParameterizedSearchQueriesCancelAndReleaseConnection|TestSearchQueriesRejectCanceledContextBeforeReading|TestBlockQueriesBeyondVariableLimit' -count=1go test ./util -run Performance -count=1+go test -tags fts5 ./sql -run 'TestAuxiliary|TestUpsertTree|TestStaticTemplateKeyDependencies' -count=1+go test ./model -run 'TestAttributeViewTemplatePlan|TestRenderAttributeViewUsesTemplateDependencyPlan|TestShouldDeferAttributeViewTemplateValues|TestRenderAttributeViewDefersDisplayOnlyTemplatesUntilAfterPagination' -count=1+go test -tags 'fts5 sqlcipher' ./sql -run '^TestAuxiliaryIndexEncryptedUpdateAndReopen$' -count=1```Cancellation tests cover running SQLite work and connection reuse in global and notebook-routed databases. The scoped in-memory database fixture tests routing, not SQLCipher format compatibility. Rendering tests use real browser DOM for insertion roots, whole replacement, selection identity, prepend scroll compensation and stale/removed roots, with component renderers replaced by spies. Component behavior and real device IME/touch interactions require their own regression checks. Semantic search retains serial scheduling because its backend cancellation is a separate change.++Auxiliary-index tests compare all five tables against complete replacement, including duplicates, removals, changed metadata, replay, failure rollback, NULL handling and batched deletion. A 20,000-block regression forbids any auxiliary-table write during an ordinary text edit. The SQLCipher test uses an actual encrypted database and verifies index migration/reopen, wrong-key rejection and isolation from the global database. Template-plan tests compare page values, order, counts, conditional colors and calculations against complete evaluation for table/list/gallery layouts, including original raw values and stale display state. These kernel changes are shared by desktop and mobile and do not change their frontend interactions.diff --git a/docs/PERFORMANCE.zh-CN.md b/docs/PERFORMANCE.zh-CN.mdindex 2f61e16e16..265107f8dc 100644--- a/docs/PERFORMANCE.zh-CN.md+++ b/docs/PERFORMANCE.zh-CN.md@@ -20,7 +20,9 @@| `document.load` | 事务屏障、加载/解析树、定位块、渲染响应;块数 || `document.save` | 准备/序列化、加密、比对、写入、通知;待保存字节数和实际写入字节数 || `index.flush` | 等锁、提取队列、开始/执行/提交、通知、持久化队列完成状态;批次大小和最老入队任务的等待时间 |+| `index.update` | 完整树投影、块比较、附属索引比较、写入;总块数和变化块数、计划删除及插入的附属记录数 || `database.view`、`database.fields`、`database.rows` | 加载和同步等待、字段计算、筛选、排序、汇总、分页、条件色;筛选前、筛选后及分页后的条目数 |+| `database.templates` | 模板求值;求值次数,包括产生模板错误的尝试;使用计算计划时,`database.fields` 还记录必需和延迟计算的模板字段数 || `search`、`search.fts`、`search.path`、`search.regexp`、`search.document` | 查询、结果加工、精确计数、分组/引用;方法、返回结果数、命中数和取消状态 || `editor.content` | 解析/规范化、DOM 插入、同步渲染调度、全局状态;渲染根节点数和已加载的顶层块数 || `editor.databases` | 串行数据库初始化完成前的经过时间;候选根节点数 |@@ -29,6 +31,10 @@嵌套操作耗时有重叠,不能将父操作与子操作相加。`editor.content` 测量同步工作和调度,不含后续公式/图表脚本加载、嵌入查询及浏览器布局。`editor.databases` 包含数据库异步等待,不代表所有组件已经可见。全局标题编号、折叠状态、只读状态、焦点和插件事件保持原有范围。+附属索引更新在原有事务内比较完整业务字段快照。各块记录的多重集相等时保留原有记录 ID;有变化时只替换该块在对应表中的记录。引用和附件元数据可能独立于块哈希变化,因此仍保留完整投影。计数表示计划写入行数,包含之后失败的事务中的操作,不能视为已提交行数。++普通表格、列表和画廊视图可在分页前计算筛选、排序、汇总及条件着色所需的模板和传递依赖,其余模板只计算当前页。两次计算共享原始值快照,并保留必需字段已生成的显示结果。搜索、分组布局、无法解析或循环的依赖、不透明模板表达式,以及会改变原有求值顺序的依赖继续完整计算。原本可以延迟全部模板的视图保持已有行为。+将内核与浏览器采样分别导出为 UTF-8 文件。在 `app/` 下汇总每次运行:```text@@ -45,6 +51,8 @@ node scripts/performance.mjs before.log after.loggo test ./filesys -run '^$' -bench '^BenchmarkDocumentJSON$' -benchmem -benchtime=200ms -count=3go test ./model -run '^$' -bench '^BenchmarkDatabaseDisplayPage$' -benchmem -benchtime=200ms -count=3go test ./model -run '^$' -bench '^BenchmarkDocumentSearch$' -benchmem -benchtime=200ms -count=3+go test -tags fts5 ./sql -run '^$' -bench '^BenchmarkUpsertTreeAuxiliaryIndexes$' -benchmem -benchtime=200ms -count=3+go test ./model -run '^$' -bench '^BenchmarkDatabaseTemplateDependencies$' -benchmem -benchtime=200ms -count=3```| 基准 | 固定工作负载 | 包含的工作 |@@ -52,6 +60,8 @@ go test ./model -run '^$' -bench '^BenchmarkDocumentSearch$' -benchmem -benchtim| `BenchmarkDocumentJSON` | 1 千 / 5 千 / 2 万个段落块,每块 112 字节 ASCII 正文,无引用/资源,扁平结构 | 文档准备/JSON 序列化,或 JSON 解析/校验;准备过程使用临时目录,不替换产品内核或工作区 || `BenchmarkDatabaseDisplayPage` | 1 千 / 1 万 / 5 万个游离条目,块/文本/显示模板字段,每页 50 条,无筛选/分组/排序 | 通过已有显示模板延迟计算生成字段、分页并求值当前页模板 || `BenchmarkDocumentSearch` | 1 万 / 10 万 / 100 万个索引块,每篇 100 块,两个词分别位于不同段落,每篇均命中,取前 50 篇 | 热内存 SQLite 文档查询、精确计数及结果行解码;不含 FTS、磁盘 I/O、完整 API 加工或前端 |+| `BenchmarkUpsertTreeAuxiliaryIndexes` | 1 千 / 5 千 / 2 万个段落,每块一个标签和自定义属性,文档另有一个标签;交替修改一个段落的普通正文 | 完整替换附属记录与增量比较、写入对照,两者均包含相同的完整投影、块/FTS 更新和事务提交;`sqlite_changes/op` 来自 SQLite `total_changes()` 差值,包含 FTS 内部写入 |+| `BenchmarkDatabaseTemplateDependencies` | 1 千 / 1 万 / 5 万个游离条目,筛选/排序需要两个有依赖关系的模板,另有五个显示模板,每页 50 条 | 完整计算与按依赖计算对照,包含计划构建、字段渲染、筛选/排序和分页;每次迭代清除已渲染集合缓存 |可通过子基准筛选缩小规模,例如 `-bench '^BenchmarkDocumentSearch$/^10000$'`。数据库建立和夹具生成不在计时循环内。数据库视图基准在迭代间复用属性视图,不代表冷启动打开。`-benchmem` 测量 Go 分配,不是 SQLite 原生分配或总 RSS。基准结果是对应路径的基线,不是应用延迟或整体提速结论。@@ -81,6 +91,11 @@ pnpm run lintgo test -tags fts5 ./model -run 'TestFullTextSearchRejectsCanceledRequest|TestSearchContextPreservesResultsCountsAndPagination' -count=1go test ./sql -run 'TestParameterizedSearchQueriesCancelAndReleaseConnection|TestSearchQueriesRejectCanceledContextBeforeReading|TestBlockQueriesBeyondVariableLimit' -count=1go test ./util -run Performance -count=1+go test -tags fts5 ./sql -run 'TestAuxiliary|TestUpsertTree|TestStaticTemplateKeyDependencies' -count=1+go test ./model -run 'TestAttributeViewTemplatePlan|TestRenderAttributeViewUsesTemplateDependencyPlan|TestShouldDeferAttributeViewTemplateValues|TestRenderAttributeViewDefersDisplayOnlyTemplatesUntilAfterPagination' -count=1+go test -tags 'fts5 sqlcipher' ./sql -run '^TestAuxiliaryIndexEncryptedUpdateAndReopen$' -count=1```取消测试覆盖全局及笔记本路由数据库中运行的 SQLite 工作和连接复用。使用独立内存库的夹具验证路由,不验证 SQLCipher 格式兼容性。渲染测试使用真实浏览器 DOM,覆盖插入范围、全量替换、选区节点身份、前插滚动补偿以及过期/已移除节点;组件渲染器以调用记录器替代。组件行为及真机输入法/触摸交互还需各自的回归检查。语义搜索继续保持串行调度,其内核取消另行实施。++附属索引测试将五类表与完整替换结果对照,覆盖重复记录、删除、元数据变化、重放、失败回滚、NULL 区分和分批删除。两万块回归在普通正文编辑时禁止任何附属表写入。SQLCipher 测试使用真实加密库,验证索引迁移/重开、错误密钥拒绝和全局库隔离。模板计划测试对表格、列表和画廊的分页值、顺序、总数、条件着色及汇总结果做完整计算对照,同时覆盖原始值和过期显示状态。这些内核改动由桌面端和移动端共用,不改变前端交互。diff --git a/kernel/model/attribute_view_conditional_color_fts5_test.go b/kernel/model/attribute_view_conditional_color_fts5_test.goindex c60b961c80..b6d330fb35 100644--- a/kernel/model/attribute_view_conditional_color_fts5_test.go+++ b/kernel/model/attribute_view_conditional_color_fts5_test.go@@ -37,6 +37,7 @@ func TestAttributeViewConditionalColorsUndoAndEncryptedReplay(t *testing.T) {boxID = ast.NewNodeID()markRuntimeEncryptedBox(boxID)setDEKForTest(boxID, bytes.Repeat([]byte{0x63}, 32))+ setEncryptedBoxState(boxID, EncryptedBoxStateUnlocked)av.SetAVBoxID(original.ID, boxID)t.Cleanup(func() {av.SetAVBoxID(original.ID, "")diff --git a/kernel/model/attribute_view_render.go b/kernel/model/attribute_view_render.goindex ba89a51fbc..233c1f3f0d 100644--- a/kernel/model/attribute_view_render.go+++ b/kernel/model/attribute_view_render.go@@ -559,10 +559,14 @@ func renderAttributeView(attrView *av.AttributeView, nodeID, viewID, carrierViewrenderContext := sql.NewAttributeViewRenderContext()renderContext.ReadOnly = !writabledefer renderContext.PushTemplateErrors()- deferTemplateValues := shouldDeferAttributeViewTemplateValues(attrView, view, query, ignoreRows)+ templatePlan := planAttributeViewTemplates(attrView, view, query, ignoreRows)+ deferTemplateValues := templatePlan != niltiming.Stage("fields")if deferTemplateValues {viewable = sql.RenderViewWithDeferredTemplatesContext(attrView, view, query, ignoreRows, renderContext)+ timing.Count("required_templates", int64(len(templatePlan.required)))+ timing.Count("deferred_templates", int64(len(templatePlan.deferred)))+ sql.FillAttributeViewTemplateValuesForKeysWithContext(attrView, view, viewable.(av.Collection), templatePlan.required, renderContext)} else {viewable = sql.RenderViewWithContext(attrView, view, query, ignoreRows, renderContext)}@@ -586,7 +590,7 @@ func renderAttributeView(attrView *av.AttributeView, nodeID, viewID, carrierViewreturn}if deferTemplateValues {- sql.FillAttributeViewTemplateValuesWithContext(attrView, view, viewable.(av.Collection), renderContext)+ sql.FillAttributeViewTemplateValuesForKeysWithContext(attrView, view, viewable.(av.Collection), templatePlan.deferred, renderContext)}if nil != target && targetIndex >= 0 && !view.IsGroupView() && view.LayoutType != av.LayoutTypeKanban {setAttributeViewRenderTarget(target, "", targetIndex, targetOffset, view.PageSize)diff --git a/kernel/model/attribute_view_template_plan.go b/kernel/model/attribute_view_template_plan.gonew file mode 100644index 0000000000..6b9412a63a--- /dev/null+++ b/kernel/model/attribute_view_template_plan.go@@ -0,0 +1,138 @@+package model++import (+ "strings"++ "github.com/siyuan-note/siyuan/kernel/av"+ "github.com/siyuan-note/siyuan/kernel/sql"+)++type attributeViewTemplatePlan struct {+ required map[string]bool+ deferred map[string]bool+}++func planAttributeViewTemplates(attrView *av.AttributeView, view *av.View, query string, ignoreRows bool) *attributeViewTemplatePlan {+ if attrView == nil || view == nil || ignoreRows {+ return nil+ }+ keys := map[string]*av.Key{}+ for _, values := range attrView.KeyValues {+ if values != nil && values.Key != nil && (values.Key.Type == av.KeyTypeTemplate || strings.TrimSpace(values.Key.RenderTemplate) != "") {+ keys[values.Key.ID] = values.Key+ }+ }+ if len(keys) == 0 {+ return nil+ }+ plan := &attributeViewTemplatePlan{required: map[string]bool{}, deferred: map[string]bool{}}+ if shouldDeferAttributeViewTemplateValues(attrView, view, query, ignoreRows) {+ for id := range keys {+ plan.deferred[id] = true+ }+ return plan+ }+ if strings.TrimSpace(query) != "" || view.IsGroupView() {+ return nil+ }+ switch view.LayoutType {+ case av.LayoutTypeTable, av.LayoutTypeList, av.LayoutTypeGallery:+ default:+ return nil+ }+ ordered, resolved := sql.GetTemplateKeysByResolutionOrder(attrView)+ if !resolved {+ return nil+ }+ positions := map[string]int{}+ for i, key := range ordered {+ positions[key.ID] = i+ }+ dependencies := map[string][]*av.Key{}+ for id, key := range keys {+ values, known := sql.StaticTemplateKeyDependencies(attrView, key)+ if !known {+ return nil+ }+ for _, dependency := range values {+ if dependency.ID != id && keys[dependency.ID] != nil && positions[dependency.ID] > positions[id] {+ // 不提前暴露完整求值顺序中尚未生成的显示结果。+ return nil+ }+ }+ dependencies[id] = values+ }+ require := func(id string, rendered bool) {+ if key := keys[id]; key != nil && (key.Type == av.KeyTypeTemplate || rendered) {+ plan.required[id] = true+ }+ }+ var filters func([]*av.ViewFilter)+ filters = func(values []*av.ViewFilter) {+ for _, filter := range values {+ if filter == nil {+ continue+ }+ if filter.IsGroup() {+ filters(filter.Filters)+ } else {+ require(filter.Column, filter.ValueSource == av.ValueSourceRendered)+ }+ }+ }+ filters(view.Filters)+ for _, sort := range view.Sorts {+ if sort != nil {+ require(sort.Column, sort.ValueSource == av.ValueSourceRendered)+ }+ }+ for _, rule := range view.EffectiveConditionalColors() {+ if rule != nil {+ filters([]*av.ViewFilter{rule.Filter})+ }+ }+ if view.GroupCalc != nil {+ require(view.GroupCalc.Field, true)+ }+ calc := func(id string, value *av.FieldCalc) {+ if value != nil && value.Operator != av.CalcOperatorNone {+ require(id, true)+ }+ }+ if view.LayoutType == av.LayoutTypeGallery {+ for _, field := range view.Gallery.CardFields {+ if field != nil && field.BaseField != nil {+ calc(field.ID, field.Calc)+ }+ }+ } else {+ for _, field := range view.GetTableLayout().Columns {+ if field != nil && field.BaseField != nil {+ calc(field.ID, field.Calc)+ }+ }+ }+ var pending []string+ for id := range plan.required {+ pending = append(pending, id)+ }+ for len(pending) > 0 {+ id := pending[len(pending)-1]+ pending = pending[:len(pending)-1]+ for _, dependency := range dependencies[id] {+ if keys[dependency.ID] != nil && !plan.required[dependency.ID] {+ plan.required[dependency.ID] = true+ pending = append(pending, dependency.ID)+ }+ }+ }+ for id := range keys {+ if !plan.required[id] {+ plan.deferred[id] = true+ }+ }+ if len(plan.deferred) == 0 {+ return nil+ }+ return plan+}diff --git a/kernel/model/attribute_view_template_plan_benchmark_test.go b/kernel/model/attribute_view_template_plan_benchmark_test.gonew file mode 100644index 0000000000..5940bc2c76--- /dev/null+++ b/kernel/model/attribute_view_template_plan_benchmark_test.go@@ -0,0 +1,32 @@+package model++import (+ "fmt"+ "testing"++ "github.com/siyuan-note/siyuan/kernel/av"+)++func BenchmarkDatabaseTemplateDependencies(b *testing.B) {+ for _, size := range []int{1000, 10000, 50000} {+ for _, selective := range []bool{false, true} {+ name := "complete"+ if selective {+ name = "selective"+ }+ b.Run(fmt.Sprintf("rows_%d/%s", size, name), func(b *testing.B) {+ database, view := newTemplatePlanTestView(size, 5, av.LayoutTypeTable)+ b.ReportAllocs()+ b.ResetTimer()+ for range b.N {+ // 两条路径均重建集合,确保测量包含完整渲染和分页过程。+ clear(database.RenderedViewables)+ rendered, err := renderTemplatePlanTestView(database, view, selective, 50)+ if err != nil || rendered.(av.Collection).CountItems() != 50 {+ b.Fatalf("unexpected page: %v", err)+ }+ }+ })+ }+ }+}diff --git a/kernel/model/attribute_view_template_plan_test.go b/kernel/model/attribute_view_template_plan_test.gonew file mode 100644index 0000000000..92c10d1cfe--- /dev/null+++ b/kernel/model/attribute_view_template_plan_test.go@@ -0,0 +1,298 @@+package model++import (+ "encoding/json"+ "fmt"+ "strings"+ "testing"++ "github.com/siyuan-note/siyuan/kernel/av"+ "github.com/siyuan-note/siyuan/kernel/sql"+)++func newTemplatePlanTestView(size, displays int, layout av.LayoutType) (*av.AttributeView, *av.View) {+ keys := []*av.Key{av.NewKey("block", "Block", "", av.KeyTypeBlock), av.NewKey("text", "Text", "", av.KeyTypeText),+ av.NewKey("source", "Source", "", av.KeyTypeTemplate), av.NewKey("required", "Required", "", av.KeyTypeTemplate)}+ keys[2].Template = `.action{index . "Text"}`+ keys[3].Template = `.action{printf "%s eligible" (index . "Source")}`+ for i := range displays {+ key := av.NewKey(fmt.Sprintf("display%d", i), fmt.Sprintf("Display%d", i), "", av.KeyTypeTemplate)+ key.Template = fmt.Sprintf(`.action{printf "display%d:%%s" (index . "Required")}`, i)+ keys = append(keys, key)+ }+ view := &av.View{ID: "view", LayoutType: layout, PageSize: 1, Table: av.NewLayoutTable(),+ Filters: []*av.ViewFilter{{Column: "required", Operator: av.FilterOperatorContains,+ Value: &av.Value{Type: av.KeyTypeTemplate, Template: &av.ValueTemplate{Content: "eligible"}}}},+ Sorts: []*av.ViewSort{{Column: "required", Order: av.SortOrderDesc}}}+ database := &av.AttributeView{ID: "av", Views: []*av.View{view}, RenderedViewables: map[string]av.Viewable{}}+ for _, key := range keys {+ values := &av.KeyValues{Key: key}+ for i := range size {+ id := fmt.Sprintf("20261019000000-%07d", i)+ value := &av.Value{ID: id + "-" + key.ID, KeyID: key.ID, BlockID: id, Type: key.Type}+ content := fmt.Sprintf("row%07d", i)+ switch key.Type {+ case av.KeyTypeBlock:+ value.IsDetached, value.Block = true, &av.ValueBlock{Content: content}+ view.ItemIDs = append(view.ItemIDs, id)+ case av.KeyTypeText:+ value.Text = &av.ValueText{Content: content}+ case av.KeyTypeTemplate:+ value.Template = &av.ValueTemplate{Content: key.Template}+ }+ values.Values = append(values.Values, value)+ }+ database.KeyValues = append(database.KeyValues, values)+ view.Table.Columns = append(view.Table.Columns, &av.ViewTableColumn{BaseField: &av.BaseField{ID: key.ID}})+ }+ if layout == av.LayoutTypeList {+ view.List, view.Table = view.Table, nil+ } else if layout == av.LayoutTypeGallery {+ view.Gallery = av.NewLayoutGallery()+ view.Gallery.CoverFrom = av.CoverFromNone+ for _, column := range view.Table.Columns {+ view.Gallery.CardFields = append(view.Gallery.CardFields, &av.ViewGalleryCardField{BaseField: column.BaseField})+ }+ view.Table = nil+ }+ return database, view+}++func renderTemplatePlanTestView(database *av.AttributeView, view *av.View, selective bool, pageSize int) (av.Viewable, error) {+ context := sql.NewAttributeViewRenderContext()+ context.ReadOnly = true+ var rendered av.Viewable+ var plan *attributeViewTemplatePlan+ if selective {+ plan = planAttributeViewTemplates(database, view, "", false)+ }+ if plan == nil {+ rendered = sql.RenderViewWithContext(database, view, "", false, context)+ } else {+ rendered = sql.RenderViewWithDeferredTemplatesContext(database, view, "", false, context)+ sql.FillAttributeViewTemplateValuesForKeysWithContext(database, view, rendered.(av.Collection), plan.required, context)+ }+ if _, _, err := renderViewableInstance(rendered, view, database, 1, pageSize, false, "", context); err != nil {+ return nil, err+ }+ if plan != nil {+ sql.FillAttributeViewTemplateValuesForKeysWithContext(database, view, rendered.(av.Collection), plan.deferred, context)+ }+ return rendered, nil+}++func templatePlanSnapshot(t *testing.T, rendered av.Viewable) string {+ t.Helper()+ collection := rendered.(av.Collection)+ var rows []any+ for _, item := range collection.GetItems() {+ var values []any+ for _, value := range item.GetValues() {+ values = append(values, []any{value.KeyID, value.String(true), value.RenderedContent, value.HasRenderTemplate})+ }+ var colors *av.ItemConditionalColors+ switch row := item.(type) {+ case *av.TableRow:+ colors = row.ConditionalColors+ case *av.GalleryCard:+ colors = row.ConditionalColors+ }+ rows = append(rows, []any{item.GetID(), values, colors})+ }+ var calculations []any+ for _, field := range collection.GetFields() {+ if calc := field.GetCalc(); calc != nil && calc.Result != nil {+ calculations = append(calculations, []string{field.GetID(), calc.Result.String(true)})+ }+ }+ count := 0+ if gallery, ok := rendered.(*av.Gallery); ok {+ count = gallery.CardCount+ } else {+ count = av.TableFromViewable(rendered).RowCount+ }+ data, err := json.Marshal([]any{count, rows, calculations, rendered.GetGroupCalc()})+ if err != nil {+ t.Fatal(err)+ }+ return string(data)+}++func TestAttributeViewTemplatePlanMatchesCompleteEvaluation(t *testing.T) {+ for _, layout := range []av.LayoutType{av.LayoutTypeTable, av.LayoutTypeList, av.LayoutTypeGallery} {+ for _, scenario := range []string{"dependencies", "rendered filter", "raw dependency", "conditional color", "calculation"} {+ t.Run(string(layout)+"/"+scenario, func(t *testing.T) {+ makeView := func() (*av.AttributeView, *av.View) {+ database, view := newTemplatePlanTestView(3, 2, layout)+ switch scenario {+ case "rendered filter":+ source, _ := database.GetKey("source")+ source.Template = `.action{index . "Block"}`+ text, _ := database.GetKey("text")+ text.RenderTemplate = `.action{printf "shown:%s" .Text_raw.Text.Content}`+ view.Filters = append(view.Filters, &av.ViewFilter{Column: "text", ValueSource: av.ValueSourceRendered,+ Operator: av.FilterOperatorContains, Value: &av.Value{Type: av.KeyTypeText, Text: &av.ValueText{Content: "shown:"}}})+ view.Sorts = []*av.ViewSort{{Column: "text", Order: av.SortOrderDesc, ValueSource: av.ValueSourceRendered}}+ case "raw dependency":+ key, _ := database.GetKey("display0")+ key.Template = `.action{.Source_raw.Template.Content}`+ case "conditional color":+ view.ConditionalColors = []*av.ConditionalColorRule{{ID: "color", Target: "item", Color: &av.ValueSelect{Color: "3"},+ Filter: &av.ViewFilter{Column: "display0", Operator: av.FilterOperatorContains,+ Value: &av.Value{Type: av.KeyTypeTemplate, Template: &av.ValueTemplate{Content: "eligible"}}}}}+ case "calculation":+ if layout == av.LayoutTypeGallery {+ view.Gallery.CardFields[4].Calc = &av.FieldCalc{Operator: av.CalcOperatorCountValues}+ } else {+ view.GetTableLayout().Columns[4].Calc = &av.FieldCalc{Operator: av.CalcOperatorCountValues}+ }+ view.GroupCalc = &av.GroupCalc{Field: "required", FieldCalc: &av.FieldCalc{Operator: av.CalcOperatorCountValues}}+ }+ return database, view+ }+ full, fullView := makeView()+ partial, view := makeView()+ plan := planAttributeViewTemplates(partial, view, "", false)+ if plan == nil || !plan.required["source"] || !plan.required["required"] || !plan.deferred["display1"] {+ t.Fatalf("missing transitive dependency or deferred display: %+v", plan)+ }+ want, err := renderTemplatePlanTestView(full, fullView, false, 1)+ if err != nil {+ t.Fatal(err)+ }+ got, err := renderTemplatePlanTestView(partial, view, true, 1)+ if err != nil {+ t.Fatal(err)+ }+ if actual, expected := templatePlanSnapshot(t, got), templatePlanSnapshot(t, want); actual != expected {+ t.Fatalf("selective result differs:\ngot %s\nwant %s", actual, expected)+ }+ items := got.(av.Collection).GetItems()+ if len(items) != 1 || items[0].GetID() != view.ItemIDs[2] || items[0].GetValue("required").Template.Content != "row0000002 eligible" {+ t.Fatalf("filter, sort, or dependency output is incorrect: %+v", items)+ }+ values, _ := partial.GetKeyValues("display1")+ for _, value := range values.Values {+ if value.BlockID != items[0].GetID() && value.Template.Content != values.Key.Template {+ t.Fatal("display template outside current page was evaluated")+ }+ }+ if scenario == "rendered filter" {+ value := items[0].GetValue("text")+ if !value.HasRenderTemplate || value.RenderedContent != "shown:row0000002" {+ t.Fatalf("late evaluation cleared required display output: %+v", value)+ }+ }+ if scenario == "conditional color" {+ var colors *av.ItemConditionalColors+ switch item := items[0].(type) {+ case *av.TableRow:+ colors = item.ConditionalColors+ case *av.GalleryCard:+ colors = item.ConditionalColors+ }+ if colors == nil || colors.Background == nil || colors.Background.Color != "3" {+ t.Fatalf("conditional color dependency was not evaluated: %+v", colors)+ }+ }+ if scenario == "calculation" {+ field, _ := got.(av.Collection).GetField("display0")+ if calc := field.GetCalc(); calc == nil || calc.Result == nil || calc.Result.Number == nil || calc.Result.Number.Content != 3 {+ t.Fatalf("field calculation did not include off-page rows: %+v", calc)+ }+ if calc := got.GetGroupCalc(); calc == nil || calc.FieldCalc == nil || calc.FieldCalc.Result == nil || calc.FieldCalc.Result.Number == nil || calc.FieldCalc.Result.Number.Content != 3 {+ t.Fatalf("group calculation did not include off-page rows: %+v", calc)+ }+ }+ })+ }+ }+}++func TestAttributeViewTemplatePlanEligibility(t *testing.T) {+ for _, test := range []struct {+ name string+ configure func(*av.AttributeView, *av.View)+ full bool+ }{+ {"nested filter", func(_ *av.AttributeView, view *av.View) { view.Filters = []*av.ViewFilter{{Filters: view.Filters}} }, false},+ {"query-dependent group", func(_ *av.AttributeView, view *av.View) { view.Group = &av.ViewGroup{Field: "text"} }, true},+ {"kanban", func(_ *av.AttributeView, view *av.View) { view.LayoutType = av.LayoutTypeKanban }, true},+ {"calendar", func(_ *av.AttributeView, view *av.View) { view.LayoutType = av.LayoutTypeCalendar }, true},+ {"branch", func(database *av.AttributeView, _ *av.View) {+ database.KeyValues[4].Key.Template = `.action{if .Required}.action{.Required}.action{end}`+ }, true},+ {"dynamic key", func(database *av.AttributeView, _ *av.View) {+ database.KeyValues[4].Key.Template = `.action{index . (print "Required")}`+ }, true},+ {"parse error", func(database *av.AttributeView, _ *av.View) { database.KeyValues[4].Key.Template = `.action{` }, true},+ {"cycle", func(database *av.AttributeView, _ *av.View) {+ database.KeyValues[2].Key.Template = `.action{.Required}`+ }, true},+ {"later rendered dependency", func(database *av.AttributeView, _ *av.View) { database.KeyValues[1].Key.RenderTemplate = "display" }, true},+ } {+ t.Run(test.name, func(t *testing.T) {+ database, view := newTemplatePlanTestView(3, 2, av.LayoutTypeTable)+ test.configure(database, view)+ if got := planAttributeViewTemplates(database, view, "", false); (got == nil) != test.full {+ t.Fatalf("unexpected plan: %+v", got)+ }+ })+ }+ database, view := newTemplatePlanTestView(3, 2, av.LayoutTypeTable)+ for _, query := range []string{"search", " search "} {+ if planAttributeViewTemplates(database, view, query, false) != nil {+ t.Fatal("search path must evaluate all templates")+ }+ }+ if planAttributeViewTemplates(database, view, "", true) != nil || planAttributeViewTemplates(nil, view, "", false) != nil {+ t.Fatal("empty rendering must not create a template plan")+ }+ view.Filters, view.Sorts = nil, nil+ database.KeyValues[4].Key.Template = `.action{if .Required}display.action{end}`+ if plan := planAttributeViewTemplates(database, view, "", false); plan == nil || len(plan.required) != 0 || len(plan.deferred) != 4 {+ t.Fatal("existing all-deferred eligibility regressed")+ }+}++func TestAttributeViewTemplatePlanClearsPreviousDisplayState(t *testing.T) {+ database, view := newTemplatePlanTestView(3, 2, av.LayoutTypeTable)+ text, _ := database.GetKey("text")+ text.RenderTemplate = `.action{printf "shown:%s" .Text_raw.Text.Content}`+ // 存储值筛选只使用原始值,不读取上次渲染的显示文本。+ view.Filters = []*av.ViewFilter{{Column: "text", Operator: av.FilterOperatorContains,+ Value: &av.Value{Type: av.KeyTypeText, Text: &av.ValueText{Content: "row"}}}}+ view.Sorts = nil+ values, _ := database.GetKeyValues("text")+ for _, value := range values.Values {+ value.HasRenderTemplate, value.RenderedContent = true, "stale"+ }+ rendered, err := renderTemplatePlanTestView(database, view, true, 1)+ if err != nil || rendered.(av.Collection).CountItems() != 1 {+ t.Fatalf("previous display state affected pagination: %v", err)+ }+ if value := rendered.(av.Collection).GetItems()[0].GetValue("text"); !strings.HasPrefix(value.RenderedContent, "shown:") {+ t.Fatalf("display was not recalculated: %+v", value)+ }+ if values.Values[1].RenderedContent != "" || values.Values[1].HasRenderTemplate {+ t.Fatal("off-page previous display state was retained")+ }+}++func TestRenderAttributeViewUsesTemplateDependencyPlan(t *testing.T) {+ database, view := newTemplatePlanTestView(3, 2, av.LayoutTypeTable)+ rendered, err := renderAttributeView(database, "", view.ID, "", "", 1, 1, nil, false, false, nil, "")+ if err != nil {+ t.Fatal(err)+ }+ items := rendered.(av.Collection).GetItems()+ if len(items) != 1 || items[0].GetValue("display0").Template.Content != "display0:row0000002 eligible" {+ t.Fatalf("render entry did not evaluate the visible dependency chain: %+v", items)+ }+ values, _ := database.GetKeyValues("display0")+ for _, value := range values.Values {+ if value.BlockID != items[0].GetID() && value.Template.Content != values.Key.Template {+ t.Fatal("render entry evaluated an off-page display template")+ }+ }+}diff --git a/kernel/sql/auxiliary_index.go b/kernel/sql/auxiliary_index.gonew file mode 100644index 0000000000..e15915fc7d--- /dev/null+++ b/kernel/sql/auxiliary_index.go@@ -0,0 +1,156 @@+package sql++import (+ "database/sql"+ "fmt"+ "strings"+)++// auxiliaryIndexKey 只比较业务字段,派生记录 ID 不参与相等性判断;NULL 与空字符串仍有区别。+type auxiliaryIndexKey struct {+ fields [11]string+ nulls uint16+}++func auxiliaryKey(fields ...string) (key auxiliaryIndexKey) {+ copy(key.fields[:], fields)+ return+}++type auxiliaryIndexTable struct {+ name, columns string // columns 的首列必须是记录所属块的 block_id。+ byPath bool+}++var (+ spanIndex = auxiliaryIndexTable{"spans", "block_id,root_id,box,path,content,markdown,type,ial", false}+ assetIndex = auxiliaryIndexTable{"assets", "block_id,root_id,box,docpath,path,name,title,hash", false}+ attributeIndex = auxiliaryIndexTable{"attributes", "block_id,root_id,box,path,name,value,type", false}+ refIndex = auxiliaryIndexTable{"refs", "block_id,root_id,box,path,def_block_id,def_block_parent_id,def_block_root_id,def_block_path,content,markdown,type", true}+ fileAnnotationIndex = auxiliaryIndexTable{"file_annotation_refs", "block_id,root_id,box,path,file_path,annotation_id,content,type", true}+)++type auxiliaryIndexChange[T any] struct {+ table auxiliaryIndexTable+ remove []int64+ insert []*T+}++// diffAuxiliaryIndex 在写入事务内比较完整快照,逐块保留相等的记录多重集。+func diffAuxiliaryIndex[T any](tx *sql.Tx, table auxiliaryIndexTable, rootID, box, path string,+ next []*T, keyOf func(*T) auxiliaryIndexKey) (change auxiliaryIndexChange[T], err error) {+ change.table = table+ remaining := make(map[auxiliaryIndexKey]int, len(next))+ for _, value := range next {+ remaining[keyOf(value)]+++ }+ condition, args := "root_id = ?", []any{rootID}+ if table.byPath {+ condition, args = "box = ? AND path = ?", []any{box, path}+ }+ rows, err := tx.Query("SELECT rowid,"+table.columns+" FROM "+table.name+" WHERE "+condition, args...)+ if err != nil {+ return change, fmt.Errorf("read %s index: %w", table.name, err)+ }+ defer rows.Close()+ type existingRow struct {+ rowID int64+ block string+ }+ var existing []existingRow+ changedBlocks := map[string]bool{}+ var rowID int64+ fields := make([]sql.NullString, strings.Count(table.columns, ",")+1)+ scan := make([]any, len(fields)+1)+ scan[0] = &rowID+ for i := range fields {+ scan[i+1] = &fields[i]+ }+ for rows.Next() {+ if err = rows.Scan(scan...); err != nil {+ return change, err+ }+ var key auxiliaryIndexKey+ for i, field := range fields {+ key.fields[i] = field.String+ if !field.Valid {+ key.nulls |= 1 << i+ }+ }+ existing = append(existing, existingRow{rowID, key.fields[0]})+ if remaining[key] > 0 {+ remaining[key]--+ } else {+ changedBlocks[key.fields[0]] = true+ }+ }+ if err = rows.Err(); err != nil {+ return change, err+ }+ if err = rows.Close(); err != nil {+ return change, err+ }+ for key, count := range remaining {+ if count > 0 {+ changedBlocks[key.fields[0]] = true+ }+ }+ for _, row := range existing {+ if changedBlocks[row.block] {+ change.remove = append(change.remove, row.rowID)+ }+ }+ for _, value := range next {+ if changedBlocks[keyOf(value).fields[0]] {+ change.insert = append(change.insert, value)+ }+ }+ return+}++func (change auxiliaryIndexChange[T]) delete(tx *sql.Tx) error {+ const batchSize = 512+ for start := 0; start < len(change.remove); start += batchSize {+ batch := change.remove[start:min(start+batchSize, len(change.remove))]+ args := make([]any, len(batch))+ for i, rowID := range batch {+ args[i] = rowID+ }+ if err := execStmtTx(tx, "DELETE FROM "+change.table.name+" WHERE rowid IN ("++ strings.Repeat("?,", len(batch)-1)+"?)", args...); err != nil {+ return err+ }+ }+ return nil+}++func spanIndexKey(value *Span) auxiliaryIndexKey {+ return auxiliaryKey(value.BlockID, value.RootID, value.Box, value.Path, value.Content, value.Markdown, value.Type, value.IAL)+}++func assetIndexKey(value *Asset) auxiliaryIndexKey {+ return auxiliaryKey(value.BlockID, value.RootID, value.Box, value.DocPath, value.Path, value.Name, value.Title, value.Hash)+}++func attributeIndexKey(value *Attribute) auxiliaryIndexKey {+ return auxiliaryKey(value.BlockID, value.RootID, value.Box, value.Path, value.Name, value.Value, value.Type)+}++func refIndexKey(value *Ref) auxiliaryIndexKey {+ return auxiliaryKey(value.BlockID, value.RootID, value.Box, value.Path, value.DefBlockID, value.DefBlockParentID,+ value.DefBlockRootID, value.DefBlockPath, value.Content, value.Markdown, value.Type)+}++func fileAnnotationIndexKey(value *FileAnnotationRef) auxiliaryIndexKey {+ return auxiliaryKey(value.BlockID, value.RootID, value.Box, value.Path, value.FilePath, value.AnnotationID, value.Content, value.Type)+}++// ensureAuxiliaryRootIndexes 为既有普通库与加密独立库补齐快照比较所需的根文档索引。+func ensureAuxiliaryRootIndexes(database *sql.DB) error {+ for _, table := range []auxiliaryIndexTable{spanIndex, assetIndex, attributeIndex} {+ if _, err := database.Exec("CREATE INDEX IF NOT EXISTS idx_" + table.name + "_root_id ON " + table.name + "(root_id)"); err != nil {+ return err+ }+ }+ return nil+}diff --git a/kernel/sql/auxiliary_index_fts5_test.go b/kernel/sql/auxiliary_index_fts5_test.gonew file mode 100644index 0000000000..31f4fb1d2a--- /dev/null+++ b/kernel/sql/auxiliary_index_fts5_test.go@@ -0,0 +1,268 @@+//go:build fts5++package sql++import (+ "database/sql"+ "encoding/json"+ "fmt"+ "reflect"+ "slices"+ "strings"+ "testing"+)++const auxiliaryTestRoot = "20261019000000-root001"++func auxiliaryTableSnapshot(t testing.TB, database *sql.DB, table auxiliaryIndexTable, withIDs bool) []string {+ t.Helper()+ columns := table.columns+ if withIDs {+ columns = "rowid,id," + columns+ }+ rows, err := database.Query("SELECT " + columns + " FROM " + table.name)+ if err != nil {+ t.Fatal(err)+ }+ defer rows.Close()+ var result []string+ for rows.Next() {+ values := make([]any, strings.Count(columns, ",")+1)+ args := make([]any, len(values))+ for i := range values {+ args[i] = &values[i]+ }+ if err = rows.Scan(args...); err != nil {+ t.Fatal(err)+ }+ for i, value := range values {+ if bytes, ok := value.([]byte); ok {+ values[i] = string(bytes)+ }+ }+ encoded, err := json.Marshal(values)+ if err != nil {+ t.Fatal(err)+ }+ result = append(result, string(encoded))+ }+ if err = rows.Err(); err != nil {+ t.Fatal(err)+ }+ slices.Sort(result)+ return result+}++func runAuxiliaryIndexCases[T any](t *testing.T, encrypted bool, table auxiliaryIndexTable,+ makeRow func(id, block, content string) *T, keyOf func(*T) auxiliaryIndexKey,+ insert func(*sql.Tx, []*T) error) {+ t.Helper()+ database := newDeletePathTestDB(t, encrypted)+ previousDisabled := cacheDisabled+ cacheDisabled = true+ t.Cleanup(func() { cacheDisabled = previousDisabled; ClearCache() })+ write := func(target *sql.DB, values []*T) {+ t.Helper()+ tx, err := target.Begin()+ if err != nil {+ t.Fatal(err)+ }+ defer closeTxPreparedStmts(tx)+ defer tx.Rollback()+ if err = insert(tx, values); err != nil {+ t.Fatal(err)+ }+ if err = tx.Commit(); err != nil {+ t.Fatal(err)+ }+ }+ old := []*T{makeRow("old-keep", "keep", "unchanged"), makeRow("old-one", "change", "one"),+ makeRow("old-duplicate", "change", "one"), makeRow("old-remove", "remove", "removed")}+ write(database, old)+ before := auxiliaryTableSnapshot(t, database, table, true)+ tx, err := database.Begin()+ if err != nil {+ t.Fatal(err)+ }+ same := []*T{makeRow("new-remove", "remove", "removed"), makeRow("new-one", "change", "one"),+ makeRow("new-keep", "keep", "unchanged"), makeRow("new-duplicate", "change", "one")}+ change, err := diffAuxiliaryIndex(tx, table, auxiliaryTestRoot, "box", "/doc.sy", same, keyOf)+ tx.Rollback()+ if err != nil || len(change.remove) != 0 || len(change.insert) != 0 {+ t.Fatalf("IDs and ordering must not cause rewrites: remove=%v insert=%v err=%v", change.remove, change.insert, err)+ }+ if after := auxiliaryTableSnapshot(t, database, table, true); !reflect.DeepEqual(before, after) {+ t.Fatal("comparing records changed the database")+ }++ next := []*T{makeRow("new-keep", "keep", "unchanged"), makeRow("new-changed", "change", "two"), makeRow("new-added", "added", "new")}+ tx, err = database.Begin()+ if err != nil {+ t.Fatal(err)+ }+ change, err = diffAuxiliaryIndex(tx, table, auxiliaryTestRoot, "box", "/doc.sy", next, keyOf)+ if err != nil || len(change.remove) != 3 || len(change.insert) != 2 {+ tx.Rollback()+ t.Fatalf("unexpected changed block multiset: deleted=%d inserted=%d err=%v", len(change.remove), len(change.insert), err)+ }+ if err = change.delete(tx); err == nil {+ err = insert(tx, change.insert)+ }+ if err != nil {+ tx.Rollback()+ t.Fatal(err)+ }+ if err = commitTx(tx); err != nil {+ t.Fatal(err)+ }+ var retained string+ if err = database.QueryRow("SELECT id FROM " + table.name + " WHERE block_id = 'keep'").Scan(&retained); err != nil || retained != "old-keep" {+ t.Fatalf("unchanged record was replaced: id=%s err=%v", retained, err)+ }+ oracle := newDeletePathTestDB(t, encrypted)+ write(oracle, next)+ if got, want := auxiliaryTableSnapshot(t, database, table, false), auxiliaryTableSnapshot(t, oracle, table, false); !reflect.DeepEqual(got, want) {+ t.Fatalf("incremental index differs from complete replacement: got=%v want=%v", got, want)+ }+ before = auxiliaryTableSnapshot(t, database, table, true)+ // 重放同一个最终快照必须保持所有记录 ID 和行号。+ tx, err = database.Begin()+ if err != nil {+ t.Fatal(err)+ }+ change, err = diffAuxiliaryIndex(tx, table, auxiliaryTestRoot, "box", "/doc.sy", next, keyOf)+ tx.Rollback()+ if err != nil || len(change.remove) != 0 || len(change.insert) != 0 {+ t.Fatalf("replayed snapshot rewrites records: %+v err=%v", change, err)+ }++ execDeletePathTestSQL(t, database, "CREATE TRIGGER injected_failure BEFORE INSERT ON "+table.name++ " WHEN new.id = 'fail' BEGIN SELECT RAISE(ABORT, 'injected auxiliary failure'); END")+ tx, err = database.Begin()+ if err != nil {+ t.Fatal(err)+ }+ failing := []*T{makeRow("fail", "keep", "changed before rollback"), next[1], next[2]}+ change, err = diffAuxiliaryIndex(tx, table, auxiliaryTestRoot, "box", "/doc.sy", failing, keyOf)+ if err == nil {+ err = change.delete(tx)+ }+ if err == nil {+ err = insert(tx, change.insert)+ }+ tx.Rollback()+ closeTxPreparedStmts(tx)+ if err == nil || !strings.Contains(err.Error(), "injected auxiliary failure") {+ t.Fatalf("expected the injected write failure, got %v", err)+ }+ if got := auxiliaryTableSnapshot(t, database, table, true); !reflect.DeepEqual(got, before) {+ t.Fatalf("rollback changed the committed index: got=%v want=%v", got, before)+ }+}++func TestAuxiliaryIndexChangesPreserveBusinessRecords(t *testing.T) {+ for _, encrypted := range []bool{false, true} {+ t.Run(fmt.Sprint(encrypted), func(t *testing.T) {+ t.Run("spans", func(t *testing.T) {+ runAuxiliaryIndexCases(t, encrypted, spanIndex, func(id, block, content string) *Span {+ return &Span{ID: id, BlockID: block, RootID: auxiliaryTestRoot, Box: "box", Path: "/doc.sy", Content: content, Type: "tag"}+ }, spanIndexKey, insertSpans)+ })+ t.Run("assets", func(t *testing.T) {+ runAuxiliaryIndexCases(t, encrypted, assetIndex, func(id, block, content string) *Asset {+ return &Asset{ID: id, BlockID: block, RootID: auxiliaryTestRoot, Box: "box", DocPath: "/doc.sy", Path: "assets/source.png", Hash: content}+ }, assetIndexKey, insertAssets)+ })+ t.Run("attributes", func(t *testing.T) {+ runAuxiliaryIndexCases(t, encrypted, attributeIndex, func(id, block, content string) *Attribute {+ return &Attribute{ID: id, BlockID: block, RootID: auxiliaryTestRoot, Box: "box", Path: "/doc.sy", Name: "custom-test", Value: content, Type: "b"}+ }, attributeIndexKey, insertAttributes)+ })+ t.Run("refs", func(t *testing.T) {+ runAuxiliaryIndexCases(t, encrypted, refIndex, func(id, block, content string) *Ref {+ return &Ref{ID: id, BlockID: block, RootID: auxiliaryTestRoot, Box: "box", Path: "/doc.sy", DefBlockID: "definition", DefBlockPath: content}+ }, refIndexKey, insertBlockRefs)+ })+ t.Run("file_annotations", func(t *testing.T) {+ runAuxiliaryIndexCases(t, encrypted, fileAnnotationIndex, func(id, block, content string) *FileAnnotationRef {+ return &FileAnnotationRef{ID: id, BlockID: block, RootID: auxiliaryTestRoot, Box: "box", Path: "/doc.sy", AnnotationID: content, FilePath: "assets/source.pdf"}+ }, fileAnnotationIndexKey, insertFileAnnotationRefs)+ })+ })+ }+}++func TestAuxiliaryIndexesPreserveExistingDataAndUseIndexedLookups(t *testing.T) {+ for _, encrypted := range []bool{false, true} {+ database := newDeletePathTestDB(t, encrypted)+ for _, table := range []auxiliaryIndexTable{spanIndex, assetIndex, attributeIndex} {+ execDeletePathTestSQL(t, database, "INSERT INTO "+table.name+" (id, root_id) VALUES ('existing', 'root')")+ execDeletePathTestSQL(t, database, "DROP INDEX idx_"+table.name+"_root_id")+ }+ for range 2 {+ if err := ensureAuxiliaryRootIndexes(database); err != nil {+ t.Fatal(err)+ }+ }+ for _, table := range []auxiliaryIndexTable{spanIndex, assetIndex, attributeIndex, refIndex, fileAnnotationIndex} {+ condition, args := "root_id = ?", []any{"root"}+ if table.byPath {+ condition, args = "box = ? AND path = ?", []any{"box", "/doc.sy"}+ }+ plan := queryPlanDetails(t, database, "SELECT rowid,"+table.columns+" FROM "+table.name+" WHERE "+condition, args...)+ if !strings.Contains(plan, "SEARCH "+table.name+" USING INDEX") {+ t.Fatalf("snapshot lookup scans %s: %s", table.name, plan)+ }+ plan = queryPlanDetails(t, database, "DELETE FROM "+table.name+" WHERE rowid IN (?, ?)", 1, 2)+ if !strings.Contains(plan, "INTEGER PRIMARY KEY") {+ t.Fatalf("changed rows are not located by rowid: %s", plan)+ }+ }+ for _, table := range []auxiliaryIndexTable{spanIndex, assetIndex, attributeIndex} {+ if ids := deletePathTestIDs(t, database, "SELECT id FROM "+table.name); !reflect.DeepEqual(ids, []string{"existing"}) {+ t.Fatalf("index upgrade changed existing rows: %s %v", table.name, ids)+ }+ }+ }+}++func TestAuxiliaryIndexPreservesNullDistinctionAndBatchesDeletes(t *testing.T) {+ database := newDeletePathTestDB(t, false)+ execDeletePathTestSQL(t, database, "INSERT INTO spans VALUES ('nullable', 'block', ?, '', '', '', '', '', NULL)", auxiliaryTestRoot)+ tx, err := database.Begin()+ if err != nil {+ t.Fatal(err)+ }+ change, err := diffAuxiliaryIndex(tx, spanIndex, auxiliaryTestRoot, "box", "/doc.sy",+ []*Span{{ID: "replacement", BlockID: "block", RootID: auxiliaryTestRoot}}, spanIndexKey)+ tx.Rollback()+ if err != nil || len(change.remove) != 1 || len(change.insert) != 1 {+ t.Fatalf("NULL was treated as an empty string: %+v err=%v", change, err)+ }+ execDeletePathTestSQL(t, database, "DELETE FROM spans")+ for i := range 1300 {+ execDeletePathTestSQL(t, database, "INSERT INTO spans (id, block_id, root_id, ial) VALUES (?, 'block', ?, NULL)", fmt.Sprint(i), auxiliaryTestRoot)+ }+ tx, err = database.Begin()+ if err != nil {+ t.Fatal(err)+ }+ defer tx.Rollback()+ change, err = diffAuxiliaryIndex(tx, spanIndex, auxiliaryTestRoot, "box", "/doc.sy",+ []*Span{{ID: "replacement", BlockID: "block", RootID: auxiliaryTestRoot}}, spanIndexKey)+ if err != nil || len(change.remove) != 1300 || len(change.insert) != 1 {+ t.Fatalf("NULL values or duplicate rows were treated as equal: removed=%d inserted=%d err=%v", len(change.remove), len(change.insert), err)+ }+ if err = change.delete(tx); err == nil {+ err = insertSpans(tx, change.insert)+ }+ if err != nil {+ t.Fatal(err)+ }+ if err = commitTx(tx); err != nil {+ t.Fatal(err)+ }+ if ids := deletePathTestIDs(t, database, "SELECT id FROM spans"); !reflect.DeepEqual(ids, []string{"replacement"}) {+ t.Fatalf("batched deletion left stale records: %v", ids)+ }+}diff --git a/kernel/sql/auxiliary_index_sqlcipher_test.go b/kernel/sql/auxiliary_index_sqlcipher_test.gonew file mode 100644index 0000000000..07d86bdfb6--- /dev/null+++ b/kernel/sql/auxiliary_index_sqlcipher_test.go@@ -0,0 +1,111 @@+//go:build fts5 && (sqlcipher || libsqlcipher)++package sql++import (+ "bytes"+ "os"+ "reflect"+ "testing"++ "github.com/siyuan-note/siyuan/kernel/util"+)++func TestAuxiliaryIndexEncryptedUpdateAndReopen(t *testing.T) {+ previousTemp, previousDB, previousEncrypted, previousDir := util.TempDir, db, IsEncryptedBoxFn, util.DataDir+ previousCached, previousIgnore, previousDisabled := IndexIgnoreCached, indexIgnore, cacheDisabled+ util.TempDir, util.DataDir = t.TempDir(), t.TempDir()+ IndexIgnoreCached, indexIgnore, cacheDisabled = true, nil, true+ tree := newUpsertTestTree(3)+ db = newDeletePathTestDB(t, false)+ IsEncryptedBoxFn = func(id string) bool { return id == tree.Box || previousEncrypted != nil && previousEncrypted(id) }+ t.Cleanup(func() {+ CloseEncryptedDB(tree.Box)+ util.TempDir, db, IsEncryptedBoxFn, util.DataDir = previousTemp, previousDB, previousEncrypted, previousDir+ IndexIgnoreCached, indexIgnore, cacheDisabled = previousCached, previousIgnore, previousDisabled+ ClearCache()+ })+ execDeletePathTestSQL(t, db, "INSERT INTO attributes VALUES ('decoy', 'custom-note', 'plaintext decoy', 'b', ?, ?, ?, ?)",+ tree.Root.FirstChild.ID, tree.ID, tree.Box, tree.Path)+ decoy := auxiliaryTableSnapshot(t, db, attributeIndex, true)+ key := bytes.Repeat([]byte{39}, 32)+ if err := OpenEncryptedDB(tree.Box, key); err != nil {+ t.Fatal(err)+ }+ database := GetEncryptedDB(tree.Box)+ var version string+ if err := database.QueryRow("PRAGMA cipher_version").Scan(&version); err != nil || version == "" {+ t.Fatalf("SQLCipher required: %q, %v", version, err)+ }+ write := func(full bool) {+ t.Helper()+ tx, err := beginTxForBox(tree.Box)+ if err != nil {+ t.Fatal(err)+ }+ defer closeTxPreparedStmts(tx)+ defer tx.Rollback()+ if full {+ err = indexTree(tx, tree, nil)+ } else {+ err = upsertTree(tx, tree, nil)+ }+ if err == nil {+ err = commitTx(tx)+ }+ if err != nil {+ t.Fatal(err)+ }+ }+ write(true)+ want := upsertTableSnapshots(t, database, true)+ tree.Root.FirstChild.FirstChild.Tokens = []byte("changed private paragraph content ")+ write(false)+ got := upsertTableSnapshots(t, database, true)+ for _, table := range upsertTestTables[:5] {+ if !reflect.DeepEqual(got[table.name], want[table.name]) {+ t.Fatalf("unchanged encrypted %s records were rewritten", table.name)+ }+ }+ // 重开旧版加密库时补建索引并保留已有数据。+ for _, table := range []auxiliaryIndexTable{spanIndex, assetIndex, attributeIndex} {+ execDeletePathTestSQL(t, database, "DROP INDEX idx_"+table.name+"_root_id")+ }+ CloseEncryptedDB(tree.Box)+ original, err := os.ReadFile(util.EncryptedDBPath(tree.Box))+ if err != nil || bytes.HasPrefix(original, []byte("SQLite format 3")) {+ t.Fatalf("expected encrypted database: %v", err)+ }+ if tx, err := beginTxForBox(tree.Box); err == nil {+ tx.Rollback()+ t.Fatal("locked update fell back to the global database")+ }+ if err = OpenEncryptedDB(tree.Box, bytes.Repeat([]byte{40}, 32)); err == nil {+ t.Fatal("wrong key accepted")+ }+ preserved, err := os.ReadFile(util.EncryptedDBPath(tree.Box))+ if err != nil || !bytes.Equal(original, preserved) || GetEncryptedDB(tree.Box) != nil {+ t.Fatal("failed authentication changed or registered the encrypted database")+ }+ for range 2 {+ if err = OpenEncryptedDB(tree.Box, key); err != nil {+ t.Fatal(err)+ }+ database = GetEncryptedDB(tree.Box)+ if reopened := upsertTableSnapshots(t, database, true); !reflect.DeepEqual(got, reopened) {+ t.Fatal("encrypted index migration changed existing records")+ }+ for _, table := range []auxiliaryIndexTable{spanIndex, assetIndex, attributeIndex} {+ rows, err := database.Query("SELECT rowid FROM "+table.name+" INDEXED BY idx_"+table.name+"_root_id WHERE root_id = ?", tree.ID)+ if err != nil {+ t.Fatal(err)+ }+ rows.Close()+ }+ write(false)+ CloseEncryptedDB(tree.Box)+ }+ if !reflect.DeepEqual(decoy, auxiliaryTableSnapshot(t, db, attributeIndex, true)) {+ t.Fatal("encrypted updates changed the global database")+ }+}diff --git a/kernel/sql/av.go b/kernel/sql/av.goindex 1c938ecebd..6c164beef8 100644--- a/kernel/sql/av.go+++ b/kernel/sql/av.go@@ -50,7 +50,8 @@ type AttributeViewRenderContext struct {templateErrors map[string]*attributeViewTemplateRenderErrorrenderingRelationTemplates boolrenderingRollupTargets map[rollupTargetID]bool- ReadOnly bool // 预览只允许修正内存副本,不保存缺失字段等订正结果。+ templateStoredValues map[av.Collection]map[*av.Value]*av.Value // 分页前后共享原始值副本,保持 _raw 访问语义。+ ReadOnly bool // 预览只允许修正内存副本,不保存缺失字段等订正结果。}type attributeViewTemplateRenderError struct {@@ -275,6 +276,14 @@ func RenderViewWithContext(attrView *av.AttributeView, view *av.View, query strifunc RenderViewWithDeferredTemplatesContext(attrView *av.AttributeView, view *av.View, query string, ignoreRows bool,renderContext *AttributeViewRenderContext) (ret av.Viewable) {ret = renderViewWithContext(attrView, view, query, ignoreRows, true, renderContext)+ if !ignoreRows && strings.TrimSpace(query) == "" {+ if collection, ok := ret.(av.Collection); ok {+ clearAttributeViewRenderedContents(collection, nil)+ if renderContext != nil {+ delete(renderContext.templateStoredValues, collection)+ }+ }+ }return}@@ -986,6 +995,16 @@ func GetFurtherCollectionsWithContext(attrView *av.AttributeView, cachedAttrView// FillAttributeViewTemplateValuesWithContext 计算集合中当前条目的模板字段值和显示模板结果。func FillAttributeViewTemplateValuesWithContext(attrView *av.AttributeView, view *av.View, collection av.Collection,renderContext *AttributeViewRenderContext) {+ FillAttributeViewTemplateValuesForKeysWithContext(attrView, view, collection, nil, renderContext)+}++// FillAttributeViewTemplateValuesForKeysWithContext 只求值指定字段,保留其他字段已计算的显示结果。+// keys 为 nil 时求值全部模板,非 nil 的空集合不执行模板。+func FillAttributeViewTemplateValuesForKeysWithContext(attrView *av.AttributeView, view *av.View, collection av.Collection,+ keys map[string]bool, renderContext *AttributeViewRenderContext) {+ if keys != nil && len(keys) == 0 {+ return+ }var ialIDs []stringfor _, item := range collection.GetItems() {blockValue := item.GetBlockValue()@@ -994,11 +1013,30 @@ func FillAttributeViewTemplateValuesWithContext(attrView *av.AttributeView, view}ialIDs = append(ialIDs, blockValue.Block.ID)}- fillAttributeViewTemplateValues(attrView, view, collection, BatchGetBlockAttrs(ialIDs), renderContext)+ fillAttributeViewTemplateValuesForKeys(attrView, view, collection, BatchGetBlockAttrs(ialIDs), renderContext, keys)}func fillAttributeViewTemplateValues(attrView *av.AttributeView, view *av.View, collection av.Collection,ials map[string]map[string]string, renderContext *AttributeViewRenderContext) {+ fillAttributeViewTemplateValuesForKeys(attrView, view, collection, ials, renderContext, nil)+}++func clearAttributeViewRenderedContents(collection av.Collection, keys map[string]bool) {+ for _, item := range collection.GetItems() {+ for _, value := range item.GetValues() {+ if value != nil && (keys == nil || keys[value.KeyID]) {+ value.RenderedContent = ""+ value.HasRenderTemplate = false+ }+ }+ }+}++func fillAttributeViewTemplateValuesForKeys(attrView *av.AttributeView, view *av.View, collection av.Collection,+ ials map[string]map[string]string, renderContext *AttributeViewRenderContext, keys map[string]bool) {+ timing := util.BeginPerformance("database.templates")+ var evaluated int64+ defer func() { timing.Count("values", evaluated); timing.Finish() }()items := generateAttrViewItems(attrView, view)existTemplateField := falsefor _, kVals := range attrView.KeyValues {@@ -1007,14 +1045,7 @@ func fillAttributeViewTemplateValues(attrView *av.AttributeView, view *av.View,break}}- for _, item := range collection.GetItems() {- for _, value := range item.GetValues() {- if nil != value {- value.RenderedContent = ""- value.HasRenderTemplate = false- }- }- }+ clearAttributeViewRenderedContents(collection, keys)if !existTemplateField {return}@@ -1024,7 +1055,21 @@ func fillAttributeViewTemplateValues(attrView *av.AttributeView, view *av.View,tplCache := map[string]*template.Template{}compileErrCache := map[string]error{}storedValueCache := map[*av.Value]*av.Value{}+ if keys != nil && renderContext != nil {+ if renderContext.templateStoredValues == nil {+ renderContext.templateStoredValues = map[av.Collection]map[*av.Value]*av.Value{}+ }+ storedValueCache = renderContext.templateStoredValues[collection]+ if storedValueCache == nil {+ storedValueCache = map[*av.Value]*av.Value{}+ renderContext.templateStoredValues[collection] = storedValueCache+ }+ }+ timing.Stage("evaluate")for _, templateKey := range templateKeys {+ if keys != nil && !keys[templateKey.ID] {+ continue+ }templateContent := templateKey.TemplateisRenderTemplate := av.KeyTypeTemplate != templateKey.Typeif isRenderTemplate {@@ -1035,6 +1080,7 @@ func fillAttributeViewTemplateValues(attrView *av.AttributeView, view *av.View,if nil == value || (!isRenderTemplate && nil == value.Template) {continue}+ evaluated++keyValues := items[item.GetID()]var ial map[string]stringdiff --git a/kernel/sql/av_template_plan.go b/kernel/sql/av_template_plan.gonew file mode 100644index 0000000000..03fd2a53e6--- /dev/null+++ b/kernel/sql/av_template_plan.go@@ -0,0 +1,87 @@+package sql++import (+ "text/template/parse"++ "github.com/siyuan-note/siyuan/kernel/av"+)++// StaticTemplateKeyDependencies 复用字段依赖解析,仅为可静态识别的表达式返回完整依赖。+// 分支、命名模板、变量别名或动态访问保留完整求值路径,避免将遗漏依赖当成显示字段。+func StaticTemplateKeyDependencies(attrView *av.AttributeView, key *av.Key) ([]*av.Key, bool) {+ if attrView == nil || key == nil {+ return nil, false+ }+ content := key.Template+ if key.Type != av.KeyTypeTemplate {+ content = key.RenderTemplate+ }+ tpl, err := compileTemplateField(content)+ if err != nil || tpl == nil || tpl.Tree == nil || !staticTemplateNode(tpl.Tree.Root) {+ return nil, false+ }+ return GetTemplateKeyRelevantKeys(attrView, key), true+}++func staticTemplateNode(node parse.Node) bool {+ if node == nil {+ return true+ }+ switch n := node.(type) {+ case *parse.ListNode:+ for _, child := range n.Nodes {+ if !staticTemplateNode(child) {+ return false+ }+ }+ return true+ case *parse.ActionNode:+ return staticTemplateNode(n.Pipe)+ case *parse.PipeNode:+ if len(n.Decl) != 0 {+ return false+ }+ for _, command := range n.Cmds {+ if !staticTemplateNode(command) {+ return false+ }+ }+ return true+ case *parse.CommandNode:+ if len(n.Args) >= 2 && n.Args[0].Type() == parse.NodeIdentifier && n.Args[0].String() == "index" &&+ isTemplateIDModField(n.Args[1]) && len(n.Args[1].(*parse.FieldNode).Ident) != 1 {+ return false+ }+ // 字面索引的根对象不代表依赖全部字段。+ if len(n.Args) >= 3 && n.Args[0].Type() == parse.NodeIdentifier && n.Args[0].String() == "index" &&+ n.Args[2].Type() == parse.NodeString {+ if n.Args[1].Type() == parse.NodeDot {+ name := n.Args[2].(*parse.StringNode).Text+ if name == "id_mod" || name == "id_mod_raw" {+ return false+ }+ }+ staticRoot := n.Args[1].Type() == parse.NodeDot || isTemplateIDModField(n.Args[1])+ if staticRoot {+ for i, argument := range n.Args {+ if i != 1 && !staticTemplateNode(argument) {+ return false+ }+ }+ return true+ }+ }+ for _, argument := range n.Args {+ if !staticTemplateNode(argument) {+ return false+ }+ }+ return true+ case *parse.FieldNode:+ return len(n.Ident) > 0 && !((n.Ident[0] == "id_mod" || n.Ident[0] == "id_mod_raw") && len(n.Ident) == 1)+ case *parse.TextNode, *parse.StringNode, *parse.NumberNode, *parse.BoolNode, *parse.NilNode, *parse.IdentifierNode:+ return true+ default:+ return false+ }+}diff --git a/kernel/sql/av_template_plan_test.go b/kernel/sql/av_template_plan_test.gonew file mode 100644index 0000000000..9cea89be65--- /dev/null+++ b/kernel/sql/av_template_plan_test.go@@ -0,0 +1,52 @@+package sql++import (+ "slices"+ "testing"++ "github.com/siyuan-note/siyuan/kernel/av"+)++func TestStaticTemplateKeyDependencies(t *testing.T) {+ text := av.NewKey("text", "Text", "", av.KeyTypeText)+ template := av.NewKey("template", "Template", "", av.KeyTypeTemplate)+ attrView := &av.AttributeView{KeyValues: []*av.KeyValues{{Key: text}, {Key: template}}}+ for _, test := range []struct {+ content string+ known bool+ }{+ {`.action{.Text}`, true}, {`.action{index . "Text"}`, true},+ {`.action{.Text_raw.Text.Content}`, true}, {`.action{.id_mod.text}`, true},+ {`.action{index .id_mod "text"}`, true}, {`.action{index .id_mod_raw "text"}`, true},+ {`.action{printf "%s" .Text}`, true},+ {`.action{if .Text}.action{.Text}.action{end}`, false},+ {`.action{range .}.action{.Text}.action{end}`, false},+ {`.action{with .}.action{.Text}.action{end}`, false},+ {`.action{$x := .}.action{index $x "Text"}`, false},+ {`.action{index . (print "Te" "xt")}`, false},+ {`.action{index .id_mod (print "text")}`, false},+ {`.action{index .id_mod.text "Text"}`, false},+ {`.action{index . "id_mod" "text"}`, false}, {`.action{index . "id_mod_raw"}`, false},+ {`.action{template "named" .}`, false}, {`.action{.id_mod}`, false}, {`.action{`, false},+ } {+ t.Run(test.content, func(t *testing.T) {+ template.Template = test.content+ dependencies, known := StaticTemplateKeyDependencies(attrView, template)+ if known != test.known {+ t.Fatalf("known=%t, want %t", known, test.known)+ }+ if known {+ var ids []string+ for _, key := range dependencies {+ ids = append(ids, key.ID)+ }+ if !slices.Equal(ids, []string{"text"}) {+ t.Fatalf("missing or extra dependency: %v", ids)+ }+ }+ })+ }+ if _, known := StaticTemplateKeyDependencies(attrView, nil); known {+ t.Fatal("nil template was treated as known")+ }+}diff --git a/kernel/sql/database.go b/kernel/sql/database.goindex 3385e7830f..e59fbe9a73 100644--- a/kernel/sql/database.go+++ b/kernel/sql/database.go@@ -128,6 +128,9 @@ func initDatabase(forceRebuild bool) {if err := ensureDocumentPathIndexes(db, true); err != nil {logging.LogFatalf(logging.ExitCodeUnavailableDatabase, "create document path indexes failed: %s", err)}+ if err := ensureAuxiliaryRootIndexes(db); err != nil {+ logging.LogFatalf(logging.ExitCodeUnavailableDatabase, "create auxiliary root indexes failed: %s", err)+ }if err := ensureTagSpansIndex(db); err != nil {logging.LogFatalf(logging.ExitCodeUnavailableDatabase, "create tag spans index failed: %s", err)}@@ -319,6 +322,9 @@ func initDBTables() {if err = ensureDocumentPathIndexes(db, true); err != nil {logging.LogFatalf(logging.ExitCodeUnavailableDatabase, "create document path indexes failed: %s", err)}+ if err = ensureAuxiliaryRootIndexes(db); err != nil {+ logging.LogFatalf(logging.ExitCodeUnavailableDatabase, "create auxiliary root indexes failed: %s", err)+ }}func initFTSBlocks() (err error) {@@ -2163,6 +2169,9 @@ func initEncryptedDBTables(boxDB *sql.DB) (err error) {if err = ensureDocumentPathIndexes(boxDB, false); err != nil {return}+ if err = ensureAuxiliaryRootIndexes(boxDB); err != nil {+ return+ }if err = ensureTagSpansIndex(boxDB); err != nil {return}diff --git a/kernel/sql/upsert.go b/kernel/sql/upsert.goindex f631a41e4f..58a193e7e5 100644--- a/kernel/sql/upsert.go+++ b/kernel/sql/upsert.go@@ -323,7 +323,7 @@ func insertSpans0(tx *sql.Tx, bulk []*Span) (err error) {return}-func insertBlockRefs(tx *sql.Tx, refs []*Ref) (err error) {+func validBlockRefs(refs []*Ref) []*Ref {validRefs := refs[:0]for _, ref := range refs {if nil == ref || "" == strings.TrimSpace(ref.DefBlockID) || "" == strings.TrimSpace(ref.BlockID) ||@@ -332,7 +332,11 @@ func insertBlockRefs(tx *sql.Tx, refs []*Ref) (err error) {}validRefs = append(validRefs, ref)}- refs = validRefs+ return validRefs+}++func insertBlockRefs(tx *sql.Tx, refs []*Ref) (err error) {+ refs = validBlockRefs(refs)if 1 > len(refs) {return}@@ -446,8 +450,13 @@ func upsertTree(tx *sql.Tx, tree *parse.Tree, context map[string]any) (err errorif isIndexIgnored(tree) {return}+ timing := util.BeginPerformance("index.update")+ defer timing.Finish()+ timing.Stage("project")oldBlockHashes := queryBlockHashes(tx, tree.ID)blocks, spans, assets, attributes := fromTree(tree.Root, tree)+ timing.Count("blocks_total", int64(len(blocks)))+ timing.Stage("compare_blocks")newBlockHashes := map[string]string{}for _, block := range blocks {newBlockHashes[block.ID] = block.Hash@@ -470,6 +479,7 @@ func upsertTree(tx *sql.Tx, tree *parse.Tree, context map[string]any) (err error}}blocks = tmp+ timing.Count("blocks_changed", int64(len(blocks)))for _, b := range blocks {toRemoves = append(toRemoves, b.ID)}@@ -478,24 +488,39 @@ func upsertTree(tx *sql.Tx, tree *parse.Tree, context map[string]any) (err errorreturn}- if err = deleteSpansByRootID(tx, tree.ID); err != nil {+ timing.Stage("compare_auxiliary")+ refs, fileAnnotationRefs := refsFromTree(tree)+ refs = validBlockRefs(refs)+ spanChange, err := diffAuxiliaryIndex(tx, spanIndex, tree.ID, tree.Box, tree.Path, spans, spanIndexKey)+ if err != nil {return}- if err = deleteAssetsByRootID(tx, tree.ID); err != nil {+ assetChange, err := diffAuxiliaryIndex(tx, assetIndex, tree.ID, tree.Box, tree.Path, assets, assetIndexKey)+ if err != nil {return}- if err = deleteAttributesByRootID(tx, tree.ID); err != nil {+ attributeChange, err := diffAuxiliaryIndex(tx, attributeIndex, tree.ID, tree.Box, tree.Path, attributes, attributeIndexKey)+ if err != nil {return}- if err = deleteRefsByPathTx(tx, tree.Box, tree.Path); err != nil {+ refChange, err := diffAuxiliaryIndex(tx, refIndex, tree.ID, tree.Box, tree.Path, refs, refIndexKey)+ if err != nil {return}- if err = deleteFileAnnotationRefsByPathTx(tx, tree.Box, tree.Path); err != nil {+ fileChange, err := diffAuxiliaryIndex(tx, fileAnnotationIndex, tree.ID, tree.Box, tree.Path, fileAnnotationRefs, fileAnnotationIndexKey)+ if err != nil {return}-- refs, fileAnnotationRefs := refsFromTree(tree)- if err = insertTree0(tx, tree, context, blocks, spans, assets, attributes, refs, fileAnnotationRefs); err != nil {+ timing.Count("auxiliary_deleted", int64(len(spanChange.remove)+len(assetChange.remove)+len(attributeChange.remove)+len(refChange.remove)+len(fileChange.remove)))+ timing.Count("auxiliary_inserted", int64(len(spanChange.insert)+len(assetChange.insert)+len(attributeChange.insert)+len(refChange.insert)+len(fileChange.insert)))+ timing.Stage("write")+ for _, remove := range []func(*sql.Tx) error{spanChange.delete, assetChange.delete, attributeChange.delete, refChange.delete, fileChange.delete} {+ if err = remove(tx); err != nil {+ return+ }+ }+ if err = insertTreeRows(tx, context, blocks, spanChange.insert, assetChange.insert, attributeChange.insert,+ refChange.insert, fileChange.insert); err != nil {return}return err@@ -515,6 +540,18 @@ func insertTree0(tx *sql.Tx, tree *parse.Tree, context map[string]any,if isIndexIgnored(tree) {return}+ if 0 < len(spans) {+ // 完整插入时移除已有文档标签,避免重复添加。+ if err = deleteSpansByRootID(tx, tree.Root.ID); err != nil {+ return+ }+ }+ return insertTreeRows(tx, context, blocks, spans, assets, attributes, refs, fileAnnotationRefs)+}++func insertTreeRows(tx *sql.Tx, context map[string]any,+ blocks []*Block, spans []*Span, assets []*Asset, attributes []*Attribute,+ refs []*Ref, fileAnnotationRefs []*FileAnnotationRef) (err error) {if err = insertBlocks(tx, blocks, context); err != nil {return@@ -528,10 +565,6 @@ func insertTree0(tx *sql.Tx, tree *parse.Tree, context map[string]any,}if 0 < len(spans) {- // 移除文档标签,否则会重复添加 https://github.com/siyuan-note/siyuan/issues/3723- if err = deleteSpansByRootID(tx, tree.Root.ID); err != nil {- return- }if err = insertSpans(tx, spans); err != nil {return}diff --git a/kernel/sql/upsert_benchmark_fts5_test.go b/kernel/sql/upsert_benchmark_fts5_test.gonew file mode 100644index 0000000000..8ae23fa38f--- /dev/null+++ b/kernel/sql/upsert_benchmark_fts5_test.go@@ -0,0 +1,129 @@+//go:build fts5++package sql++import (+ "database/sql"+ "fmt"+ "testing"++ "github.com/88250/lute/parse"+ "github.com/emirpasic/gods/sets/hashset"+ "github.com/siyuan-note/siyuan/kernel/util"+)++// upsertTreeWithFullAuxiliaryReplacement 以完整替换附属记录作为增量路径的基准对照。+func upsertTreeWithFullAuxiliaryReplacement(tx *sql.Tx, tree *parse.Tree, context map[string]any) (err error) {+ oldBlockHashes := queryBlockHashes(tx, tree.ID)+ blocks, spans, assets, attributes := fromTree(tree.Root, tree)+ newBlockHashes := map[string]string{}+ for _, block := range blocks {+ newBlockHashes[block.ID] = block.Hash+ }+ unchanged := hashset.New()+ var removed []string+ for id, hash := range oldBlockHashes {+ if next, ok := newBlockHashes[id]; ok {+ if next == hash {+ unchanged.Add(id)+ }+ } else {+ removed = append(removed, id)+ }+ }+ next := blocks[:0]+ for _, block := range blocks {+ if !unchanged.Contains(block.ID) {+ next = append(next, block)+ removed = append(removed, block.ID)+ }+ }+ if err = deleteBlocksByIDs(tx, removed); err != nil {+ return+ }+ for _, remove := range []func() error{+ func() error { return deleteSpansByRootID(tx, tree.ID) },+ func() error { return deleteAssetsByRootID(tx, tree.ID) },+ func() error { return deleteAttributesByRootID(tx, tree.ID) },+ func() error { return deleteRefsByPathTx(tx, tree.Box, tree.Path) },+ func() error { return deleteFileAnnotationRefsByPathTx(tx, tree.Box, tree.Path) },+ } {+ if err = remove(); err != nil {+ return+ }+ }+ refs, annotations := refsFromTree(tree)+ return insertTree0(tx, tree, context, next, spans, assets, attributes, refs, annotations)+}++func BenchmarkUpsertTreeAuxiliaryIndexes(b *testing.B) {+ previousCached, previousIgnore, previousDir, previousDisabled, previousDB := IndexIgnoreCached, indexIgnore, util.DataDir, cacheDisabled, db+ IndexIgnoreCached, indexIgnore, cacheDisabled, util.DataDir = true, nil, true, b.TempDir()+ b.Cleanup(func() {+ IndexIgnoreCached, indexIgnore, util.DataDir, cacheDisabled = previousCached, previousIgnore, previousDir, previousDisabled+ db = previousDB+ ClearCache()+ })+ for _, size := range []int{1000, 5000, 20000} {+ b.Run(fmt.Sprint(size), func(b *testing.B) {+ for _, mode := range []struct {+ name string+ update func(*sql.Tx, *parse.Tree, map[string]any) error+ }{{"complete", upsertTreeWithFullAuxiliaryReplacement}, {"incremental", upsertTree}} {+ b.Run(mode.name, func(b *testing.B) {+ database := newDeletePathTestDB(b, false)+ db = database+ tree := newUpsertTestTree(size)+ tx, err := database.Begin()+ if err != nil {+ b.Fatal(err)+ }+ if err = indexTree(tx, tree, nil); err == nil {+ err = commitTx(tx)+ }+ if err != nil {+ b.Fatal(err)+ }+ tx, err = database.Begin()+ if err != nil {+ b.Fatal(err)+ }+ if hashes := queryBlockHashes(tx, tree.ID); len(hashes) != size+1 {+ b.Fatalf("baseline must contain all existing block hashes: got %d", len(hashes))+ }+ tx.Rollback()+ var before, after int64+ if err = database.QueryRow("SELECT total_changes()").Scan(&before); err != nil {+ b.Fatal(err)+ }+ b.ReportAllocs()+ b.ResetTimer()+ for i := 0; i < b.N; i++ {+ content := "changed ordinary paragraph content "+ if i%2 == 1 {+ content = "original paragraph content "+ }+ tree.Root.FirstChild.FirstChild.Tokens = []byte(content)+ tx, err = database.Begin()+ if err != nil {+ b.Fatal(err)+ }+ if err = mode.update(tx, tree, nil); err == nil {+ err = commitTx(tx)+ }+ if err != nil {+ tx.Rollback()+ closeTxPreparedStmts(tx)+ b.Fatal(err)+ }+ }+ b.StopTimer()+ if err = database.QueryRow("SELECT total_changes()").Scan(&after); err != nil {+ b.Fatal(err)+ }+ b.ReportMetric(float64(after-before)/float64(b.N), "sqlite_changes/op")+ })+ }+ })+ }+}diff --git a/kernel/sql/upsert_fts5_test.go b/kernel/sql/upsert_fts5_test.gonew file mode 100644index 0000000000..86de3f58d9--- /dev/null+++ b/kernel/sql/upsert_fts5_test.go@@ -0,0 +1,240 @@+//go:build fts5++package sql++import (+ "database/sql"+ "fmt"+ "os"+ "path/filepath"+ "reflect"+ "strings"+ "testing"++ "github.com/88250/lute/ast"+ "github.com/88250/lute/parse"+ "github.com/siyuan-note/siyuan/kernel/cache"+ "github.com/siyuan-note/siyuan/kernel/treenode"+ "github.com/siyuan-note/siyuan/kernel/util"+)++var upsertTestTables = []auxiliaryIndexTable{spanIndex, assetIndex, attributeIndex, refIndex, fileAnnotationIndex,+ {name: "blocks", columns: "id,parent_id,root_id,hash,box,path,hpath,name,alias,memo,tag,content,fcontent,markdown,length,type,subtype,ial,sort,created,updated"}}++func newUpsertTestTree(size int) *parse.Tree {+ tree := treenode.NewTree("20261019000000-box0001", "/"+auxiliaryTestRoot+".sy", "/Index baseline", "Index baseline")+ tree.Root.FirstChild.Unlink()+ tree.Root.SetIALAttr("tags", "document-tag")+ for i := range size {+ paragraph := treenode.NewParagraph(fmt.Sprintf("20261019010000-%07d", i))+ paragraph.SetIALAttr("custom-note", "unchanged")+ paragraph.AppendChild(&ast.Node{Type: ast.NodeText, Tokens: []byte("original paragraph content ")})+ paragraph.AppendChild(&ast.Node{Type: ast.NodeTextMark, TextMarkType: "tag", TextMarkTextContent: "tag"})+ tree.Root.AppendChild(paragraph)+ }+ return tree+}++func upsertTableSnapshots(t testing.TB, database *sql.DB, withIDs bool) map[string][]string {+ t.Helper()+ result := map[string][]string{}+ for _, table := range upsertTestTables {+ result[table.name] = auxiliaryTableSnapshot(t, database, table, withIDs)+ }+ return result+}++func TestUpsertTreeLargeDocumentDoesNotRewriteUnchangedAuxiliaryRows(t *testing.T) {+ previousDB, previousDisabled := db, cacheDisabled+ previousCached, previousIgnore, previousDir := IndexIgnoreCached, indexIgnore, util.DataDir+ IndexIgnoreCached, indexIgnore, cacheDisabled, util.DataDir = true, nil, true, t.TempDir()+ t.Cleanup(func() {+ db, cacheDisabled = previousDB, previousDisabled+ IndexIgnoreCached, indexIgnore, util.DataDir = previousCached, previousIgnore, previousDir+ ClearCache()+ })+ database := newDeletePathTestDB(t, false)+ db = database+ tree := newUpsertTestTree(20000)+ write := func(full bool) {+ t.Helper()+ tx, err := database.Begin()+ if err != nil {+ t.Fatal(err)+ }+ defer closeTxPreparedStmts(tx)+ defer tx.Rollback()+ if full {+ err = indexTree(tx, tree, nil)+ } else {+ err = upsertTree(tx, tree, nil)+ }+ if err == nil {+ err = commitTx(tx)+ }+ if err != nil {+ t.Fatal(err)+ }+ }+ write(true)+ for _, table := range upsertTestTables[:5] {+ for _, operation := range []string{"INSERT", "DELETE", "UPDATE"} {+ execDeletePathTestSQL(t, database, "CREATE TRIGGER forbid_"+table.name+operation+" BEFORE "+operation++ " ON "+table.name+" BEGIN SELECT RAISE(ABORT, 'unchanged auxiliary row was written'); END")+ }+ }+ tree.Root.FirstChild.FirstChild.Tokens = []byte("changed ordinary paragraph content ")+ write(false)+ var content string+ if err := database.QueryRow("SELECT content FROM blocks WHERE id = ?", tree.Root.FirstChild.ID).Scan(&content); err != nil ||+ !strings.Contains(content, "changed ordinary") {+ t.Fatalf("block was not updated: %q, %v", content, err)+ }+ execDeletePathTestSQL(t, database, "INSERT INTO blocks_fts(blocks_fts, rank) VALUES('integrity-check', 1)")+}++func TestUpsertTreeMatchesCompleteIndexAndPreservesUnchangedRows(t *testing.T) {+ for _, scoped := range []bool{false, true} {+ t.Run(fmt.Sprint(scoped), func(t *testing.T) {+ previousDB, previousDisabled := db, cacheDisabled+ previousCached, previousIgnore, previousDir := IndexIgnoreCached, indexIgnore, util.DataDir+ IndexIgnoreCached, indexIgnore, cacheDisabled, util.DataDir = true, nil, true, t.TempDir()+ t.Cleanup(func() {+ db, cacheDisabled = previousDB, previousDisabled+ IndexIgnoreCached, indexIgnore, util.DataDir = previousCached, previousIgnore, previousDir+ ClearCache()+ })+ database := newDeletePathTestDB(t, scoped)+ db = database+ tree := newUpsertTestTree(3)+ var decoy *sql.DB+ if scoped {+ decoy = newDeletePathTestDB(t, false)+ db = decoy+ encryptedDBs.Store(tree.Box, database)+ t.Cleanup(func() { encryptedDBs.Delete(tree.Box) })+ execDeletePathTestSQL(t, decoy, "INSERT INTO attributes VALUES ('decoy', 'custom-note', 'private decoy', 'b', ?, ?, ?, ?)",+ tree.Root.FirstChild.ID, tree.ID, tree.Box, tree.Path)+ }+ asset := "assets/upsert-" + ast.NewNodeID() + ".png"+ if err := os.MkdirAll(filepath.Join(util.DataDir, "assets"), 0700); err != nil {+ t.Fatal(err)+ }+ if err := os.WriteFile(filepath.Join(util.DataDir, asset), []byte("fixture asset"), 0600); err != nil {+ t.Fatal(err)+ }+ cache.SetAssetHash("initial-asset-hash", asset)+ t.Cleanup(func() { cache.RemoveAssetHash("initial-asset-hash"); cache.RemoveAssetHash("changed-asset-hash") })+ attached := tree.Root.FirstChild.Next+ attached.AppendChild(&ast.Node{Type: ast.NodeTextMark, TextMarkType: "a", TextMarkAHref: asset, TextMarkTextContent: "asset"})+ attached.AppendChild(&ast.Node{Type: ast.NodeTextMark, TextMarkType: "block-ref", TextMarkBlockRefID: "20261019000000-target1", TextMarkBlockRefSubtype: "s", TextMarkTextContent: "reference"})+ annotation := &ast.Node{Type: ast.NodeTextMark, TextMarkType: "file-annotation-ref", TextMarkFileAnnotationRefID: "assets/test.pdf/20261019000000-annot01", TextMarkTextContent: "annotation"}+ attached.AppendChild(annotation)+ write := func(full bool) error {+ tx, err := beginTxForBox(tree.Box)+ if err != nil {+ return err+ }+ defer closeTxPreparedStmts(tx)+ defer tx.Rollback()+ if full {+ err = indexTree(tx, tree, nil)+ } else {+ err = upsertTree(tx, tree, nil)+ }+ if err != nil {+ return err+ }+ return commitTx(tx)+ }+ if err := write(true); err != nil {+ t.Fatal(err)+ }+ oracle := newDeletePathTestDB(t, scoped)+ compare := func() {+ t.Helper()+ tx, err := oracle.Begin()+ if err != nil {+ t.Fatal(err)+ }+ if err = deleteByRootID(tx, tree.ID, nil); err == nil {+ err = indexTree(tx, tree, nil)+ }+ if err == nil {+ err = commitTx(tx)+ }+ closeTxPreparedStmts(tx)+ tx.Rollback()+ if err != nil {+ t.Fatal(err)+ }+ if got, want := upsertTableSnapshots(t, database, false), upsertTableSnapshots(t, oracle, false); !reflect.DeepEqual(got, want) {+ for table, values := range want {+ if !reflect.DeepEqual(got[table], values) {+ t.Errorf("%s differs from complete index: got=%v want=%v", table, got[table], values)+ }+ }+ t.FailNow()+ }+ execDeletePathTestSQL(t, database, "INSERT INTO blocks_fts(blocks_fts, rank) VALUES('integrity-check', 1)")+ }+ before := upsertTableSnapshots(t, database, true)+ tree.Root.FirstChild.FirstChild.Tokens = []byte("changed ordinary paragraph content ")+ if err := write(false); err != nil {+ t.Fatal(err)+ }+ for _, table := range upsertTestTables[:5] {+ if got := auxiliaryTableSnapshot(t, database, table, true); !reflect.DeepEqual(got, before[table.name]) {+ t.Fatalf("ordinary text edit rewrote unchanged %s records", table.name)+ }+ }+ compare()+ // 源块哈希未变时,外部资源元数据仍需比较并更新。+ cache.SetAssetHash("changed-asset-hash", asset)+ if err := write(false); err != nil {+ t.Fatal(err)+ }+ compare()+ attached.SetIALAttr("custom-note", "changed")+ attached.FirstChild.Next.TextMarkTextContent = "changed-tag"+ annotation.TextMarkTextContent = "changed annotation anchor"+ if err := write(false); err != nil {+ t.Fatal(err)+ }+ compare()+ attached.Unlink()+ added := treenode.NewParagraph("20261019010000-added01")+ added.SetIALAttr("custom-note", "new")+ added.AppendChild(&ast.Node{Type: ast.NodeTextMark, TextMarkType: "tag", TextMarkTextContent: "new tag"})+ tree.Root.AppendChild(added)+ if err := write(false); err != nil {+ t.Fatal(err)+ }+ compare()+ before = upsertTableSnapshots(t, database, true)+ for range 2 {+ if err := write(false); err != nil {+ t.Fatal(err)+ }+ }+ if got := upsertTableSnapshots(t, database, true); !reflect.DeepEqual(got, before) {+ t.Fatal("repeated final snapshots rewrote committed rows")+ }+ execDeletePathTestSQL(t, database, "CREATE TRIGGER fail_attribute BEFORE INSERT ON attributes WHEN new.value = 'fail' BEGIN SELECT RAISE(ABORT, 'injected upsert failure'); END")+ tree.Root.FirstChild.FirstChild.Tokens = []byte("failed text update")+ tree.Root.FirstChild.SetIALAttr("custom-note", "fail")+ if err := write(false); err == nil || !strings.Contains(err.Error(), "injected upsert failure") {+ t.Fatalf("expected injected failure, got %v", err)+ }+ if got := upsertTableSnapshots(t, database, true); !reflect.DeepEqual(got, before) {+ t.Fatal("failed upsert left partial block or auxiliary updates")+ }+ execDeletePathTestSQL(t, database, "INSERT INTO blocks_fts(blocks_fts, rank) VALUES('integrity-check', 1)")+ if scoped {+ if got := deletePathTestIDs(t, decoy, "SELECT value FROM attributes"); !reflect.DeepEqual(got, []string{"private decoy"}) {+ t.Fatalf("scoped update changed global rows with the same document IDs: %v", got)+ }+ }+ })+ }+}