BrainAPI is a knowledge graph–powered AI memory layer that transforms unstructured data into structured knowledge, enabling intelligent search, recommendations, and contextual memory for AI agents and applications.
Python ai-agentsai-memoryknowledge-graphknowledge-managementknowledge-representationreccomendation-system
Document search architecture and benchmark results
Christian618f2ad43a81
diff --git a/benchmarks/REPORTS.json b/benchmarks/REPORTS.jsonindex 94a721a..858c569 100644--- a/benchmarks/REPORTS.json+++ b/benchmarks/REPORTS.json@@ -1,6 +1,6 @@{"schema_version": 2,- "updated_at": "2026-08-05T20:07:44.710028+00:00",+ "updated_at": "2026-08-21T07:53:39.332214+00:00","description": "BrainAPI benchmark results. Top published scores across suites. Updated when a suite evaluate/report completes successfully.","benchmarks": {"locomo": {@@ -803,6 +803,1790 @@"recorded_at": "2026-08-05T20:07:44.707984+00:00"}]+ },+ "search": {+ "name": "Search (hybrid BM25 + dense)",+ "leaderboard": [+ {+ "run_id": "search-20260818T111250Z",+ "brain": "searchbenchsmoke",+ "dataset": "search_toy.jsonl",+ "fusion": "rrf",+ "rerank": "none",+ "n_queries": 10,+ "ndcg@10": 1.0,+ "recall@10": 1.0,+ "mrr": 1.0,+ "p50_retrieve_ms": 25.1575,+ "p95_retrieve_ms": 27.67875,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-20260818T111250Z/report.json",+ "recorded_at": "2026-08-18T11:13:57.784280+00:00"+ },+ {+ "run_id": "search-toy-cc",+ "brain": "searchbenchsmoke",+ "dataset": "search_toy.jsonl",+ "fusion": "cc",+ "rerank": "none",+ "n_queries": 10,+ "ndcg@10": 1.0,+ "recall@10": 1.0,+ "mrr": 1.0,+ "p50_retrieve_ms": 24.2875,+ "p95_retrieve_ms": 41.931949999999986,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-toy-cc/report.json",+ "recorded_at": "2026-08-18T11:15:19.039999+00:00"+ },+ {+ "run_id": "search-esci-slice-allcols",+ "brain": "searchbenchesci20",+ "dataset": "search_esci_slice.jsonl",+ "fusion": "rrf",+ "rerank": "none",+ "channels": [+ "passages",+ "entities",+ "communities"+ ],+ "n_queries": 11,+ "ndcg@10": 0.6815199154995802,+ "recall@10": 0.4656565656565656,+ "mrr": 0.8484848484848485,+ "p50_retrieve_ms": 108.159,+ "p95_retrieve_ms": 129.32150000000001,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-slice-allcols/report.json",+ "recorded_at": "2026-08-18T13:02:36.140118+00:00"+ },+ {+ "run_id": "search-esci-slice-passages",+ "brain": "searchbenchesci20",+ "dataset": "search_esci_slice.jsonl",+ "fusion": "rrf",+ "rerank": "none",+ "channels": [+ "passages"+ ],+ "n_queries": 11,+ "ndcg@10": 0.7579469533596556,+ "recall@10": 0.5106060606060606,+ "mrr": 0.8484848484848485,+ "p50_retrieve_ms": 29.279,+ "p95_retrieve_ms": 153.97,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-slice-passages/report.json",+ "recorded_at": "2026-08-18T13:02:47.533040+00:00"+ },+ {+ "run_id": "search-esci-slice-entities",+ "brain": "searchbenchesci20",+ "dataset": "search_esci_slice.jsonl",+ "fusion": "rrf",+ "rerank": "none",+ "channels": [+ "entities"+ ],+ "n_queries": 11,+ "ndcg@10": 0.5323459350114446,+ "recall@10": 0.4151515151515151,+ "mrr": 0.6673553719008264,+ "p50_retrieve_ms": 52.75,+ "p95_retrieve_ms": 75.479,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-slice-entities/report.json",+ "recorded_at": "2026-08-18T13:31:39.862006+00:00"+ },+ {+ "run_id": "search-esci-slice-communities",+ "brain": "searchbenchesci20",+ "dataset": "search_esci_slice.jsonl",+ "fusion": "rrf",+ "rerank": "none",+ "channels": [+ "communities"+ ],+ "n_queries": 11,+ "ndcg@10": 0.0,+ "recall@10": 0.0,+ "mrr": 0.0,+ "p50_retrieve_ms": 51.536,+ "p95_retrieve_ms": 58.432500000000005,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-slice-communities/report.json",+ "recorded_at": "2026-08-18T13:31:42.972981+00:00"+ },+ {+ "run_id": "search-esci-slice-pe",+ "brain": "searchbenchesci20",+ "dataset": "search_esci_slice.jsonl",+ "fusion": "rrf",+ "rerank": "none",+ "channels": [+ "passages",+ "entities"+ ],+ "n_queries": 11,+ "ndcg@10": 0.6815199154995802,+ "recall@10": 0.4656565656565656,+ "mrr": 0.8484848484848485,+ "p50_retrieve_ms": 74.312,+ "p95_retrieve_ms": 91.006,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-slice-pe/report.json",+ "recorded_at": "2026-08-18T13:31:46.700938+00:00"+ },+ {+ "run_id": "search-esci-slice-entities-after",+ "brain": "searchbenchesci20",+ "dataset": "search_esci_slice.jsonl",+ "fusion": "rrf",+ "rerank": "none",+ "channels": [+ "entities"+ ],+ "n_queries": 11,+ "ndcg@10": 0.6549492067418874,+ "recall@10": 0.4818181818181818,+ "mrr": 0.7568181818181817,+ "p50_retrieve_ms": 51.071,+ "p95_retrieve_ms": 74.023,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-slice-entities-after/report.json",+ "recorded_at": "2026-08-18T13:36:54.694299+00:00"+ },+ {+ "run_id": "search-esci-slice-communities-after",+ "brain": "searchbenchesci20",+ "dataset": "search_esci_slice.jsonl",+ "fusion": "rrf",+ "rerank": "none",+ "channels": [+ "communities"+ ],+ "n_queries": 11,+ "ndcg@10": 0.4776363794553365,+ "recall@10": 0.35000000000000003,+ "mrr": 0.8484848484848485,+ "p50_retrieve_ms": 129.202,+ "p95_retrieve_ms": 152.889,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-slice-communities-after/report.json",+ "recorded_at": "2026-08-18T13:39:09.162227+00:00"+ },+ {+ "run_id": "search-esci-slice-passages-smoke",+ "brain": "searchbenchesci20",+ "dataset": "search_esci_slice.jsonl",+ "fusion": "rrf",+ "rerank": "none",+ "channels": [+ "passages"+ ],+ "n_queries": 11,+ "ndcg@10": 0.7579469533596556,+ "recall@10": 0.5106060606060606,+ "mrr": 0.8484848484848485,+ "p50_retrieve_ms": 27.735,+ "p95_retrieve_ms": 32.483000000000004,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-slice-passages-smoke/report.json",+ "recorded_at": "2026-08-18T13:37:03.038349+00:00"+ },+ {+ "run_id": "search-esci-slice-pec-after",+ "brain": "searchbenchesci20",+ "dataset": "search_esci_slice.jsonl",+ "fusion": "rrf",+ "rerank": "none",+ "channels": [+ "passages",+ "entities",+ "communities"+ ],+ "n_queries": 11,+ "ndcg@10": 0.6526549564168582,+ "recall@10": 0.496969696969697,+ "mrr": 0.816017316017316,+ "p50_retrieve_ms": 183.946,+ "p95_retrieve_ms": 216.89249999999998,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-slice-pec-after/report.json",+ "recorded_at": "2026-08-18T13:39:52.408131+00:00"+ },+ {+ "run_id": "search-esci-slice-passages-ce",+ "brain": "searchbenchesci20",+ "dataset": "search_esci_slice.jsonl",+ "fusion": "rrf",+ "rerank": "plugin:cross-encoder",+ "channels": [+ "passages"+ ],+ "rank_pool": false,+ "rank_pool_ce": false,+ "n_queries": 11,+ "ndcg@10": 0.7375861975343849,+ "ndcg@20": null,+ "ndcg": null,+ "recall@10": 0.5106060606060606,+ "recall@20": 0.8469696969696969,+ "mrr": 0.8030303030303031,+ "p50_retrieve_ms": 30.029,+ "p95_retrieve_ms": 76.32499999999999,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-slice-passages-ce/report.json",+ "recorded_at": "2026-08-18T14:20:28.184907+00:00"+ },+ {+ "run_id": "search-esci-slice-passages-pool",+ "brain": "searchbenchesci20",+ "dataset": "search_esci_slice.jsonl",+ "fusion": "rrf",+ "rerank": "none",+ "channels": [+ "passages"+ ],+ "rank_pool": true,+ "n_queries": 11,+ "ndcg@10": 0.7884517684883318,+ "ndcg@20": 0.8114424261982733,+ "ndcg": 0.8137804024940007,+ "recall@10": 0.5348484848484848,+ "recall@20": 0.8469696969696969,+ "mrr": 0.9090909090909091,+ "p50_retrieve_ms": 22.721,+ "p95_retrieve_ms": 75.3825,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-slice-passages-pool/report.json",+ "recorded_at": "2026-08-18T14:18:44.171034+00:00"+ },+ {+ "run_id": "search-esci-slice-ce-pool",+ "brain": "searchbenchesci20",+ "dataset": "search_esci_slice.jsonl",+ "fusion": "none",+ "rerank": "plugin:cross-encoder",+ "channels": [+ "rank-pool-ce"+ ],+ "rank_pool": true,+ "rank_pool_ce": true,+ "n_queries": 11,+ "ndcg@10": 0.5976277406374534,+ "ndcg@20": 0.668518640484444,+ "ndcg": 0.7862048091602294,+ "recall@10": 0.398637349324982,+ "recall@20": 0.6598423817505625,+ "mrr": 0.8333333333333333,+ "p50_retrieve_ms": 0.0,+ "p95_retrieve_ms": 0.0,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-slice-ce-pool/report.json",+ "recorded_at": "2026-08-18T14:20:28.034427+00:00"+ },+ {+ "run_id": "search-esci-74-ce-pool",+ "brain": "searchbenchesci74",+ "dataset": "search_esci_74.jsonl",+ "fusion": "none",+ "rerank": "plugin:cross-encoder",+ "channels": [+ "rank-pool-ce"+ ],+ "rank_pool": true,+ "rank_pool_ce": true,+ "n_queries": 74,+ "ndcg@10": 0.5492314171926193,+ "ndcg@20": 0.6311374744057928,+ "ndcg": 0.7174501167408996,+ "recall@10": 0.4907400145298616,+ "recall@20": 0.7577102510384188,+ "mrr": 0.8306570306570307,+ "p50_retrieve_ms": 0.0,+ "p95_retrieve_ms": 0.0,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-74-ce-pool/report.json",+ "recorded_at": "2026-08-18T14:29:17.114211+00:00"+ },+ {+ "run_id": "search-esci-74-ce-pool-ft",+ "brain": "searchbenchesci74",+ "dataset": "search_esci_74.jsonl",+ "fusion": "none",+ "rerank": "plugin:cross-encoder",+ "channels": [+ "rank-pool-ce"+ ],+ "rank_pool": true,+ "rank_pool_ce": true,+ "n_queries": 74,+ "ndcg@10": 0.5832632846066277,+ "ndcg@20": 0.6690889170924552,+ "ndcg": 0.7451408010463653,+ "recall@10": 0.4893954738298456,+ "recall@20": 0.7641337775411228,+ "mrr": 0.8932519057519057,+ "p50_retrieve_ms": 0.0,+ "p95_retrieve_ms": 0.0,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-74-ce-pool-ft/report.json",+ "recorded_at": "2026-08-18T14:32:48.672213+00:00"+ },+ {+ "run_id": "search-esci-74-passages",+ "brain": "searchbenchesci74",+ "dataset": "search_esci_74.jsonl",+ "fusion": "rrf",+ "rerank": "none",+ "channels": [+ "passages"+ ],+ "rank_pool": false,+ "rank_pool_ce": false,+ "n_queries": 74,+ "ndcg@10": 0.49528253069041633,+ "ndcg@20": 0.5490094202707876,+ "ndcg": 0.5490094202707876,+ "recall@10": 0.3769307112873572,+ "recall@20": 0.5920965629425753,+ "mrr": 0.7580598455598455,+ "p50_retrieve_ms": 70.9225,+ "p95_retrieve_ms": 94.17495,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-74-passages/report.json",+ "recorded_at": "2026-08-18T15:38:50.867049+00:00"+ },+ {+ "run_id": "search-esci-74-passages-pool",+ "brain": "searchbenchesci74",+ "dataset": "search_esci_74.jsonl",+ "fusion": "rrf",+ "rerank": "none",+ "channels": [+ "passages"+ ],+ "rank_pool": true,+ "rank_pool_ce": false,+ "n_queries": 74,+ "ndcg@10": 0.5458756376131827,+ "ndcg@20": 0.5785048208487761,+ "ndcg": 0.6013933046523523,+ "recall@10": 0.42064799804369324,+ "recall@20": 0.5920965629425753,+ "mrr": 0.8254504504504505,+ "p50_retrieve_ms": 65.447,+ "p95_retrieve_ms": 91.44999999999999,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-74-passages-pool/report.json",+ "recorded_at": "2026-08-18T15:41:37.970033+00:00"+ },+ {+ "run_id": "search-esci-slice-passages-gate",+ "brain": "searchbenchesci20",+ "dataset": "search_esci_slice.jsonl",+ "fusion": "rrf",+ "rerank": "none",+ "channels": [+ "passages"+ ],+ "rank_pool": false,+ "rank_pool_ce": false,+ "n_queries": 11,+ "ndcg@10": 0.7579469533596556,+ "ndcg@20": 0.7962544743266965,+ "ndcg": 0.7962544743266965,+ "recall@10": 0.5106060606060606,+ "recall@20": 0.8469696969696969,+ "mrr": 0.8484848484848485,+ "p50_retrieve_ms": 25.549,+ "p95_retrieve_ms": 41.532,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-slice-passages-gate/report.json",+ "recorded_at": "2026-08-18T16:47:11.911155+00:00"+ },+ {+ "run_id": "search-esci-slice-pec-lists",+ "brain": "searchbenchesci20",+ "dataset": "search_esci_slice.jsonl",+ "fusion": "rrf",+ "rerank": "none",+ "channels": [+ "passages",+ "entities",+ "communities"+ ],+ "rank_pool": false,+ "rank_pool_ce": false,+ "n_queries": 11,+ "ndcg@10": 0.6526549564168582,+ "ndcg@20": 0.6261389422887412,+ "ndcg": 0.683063799169122,+ "recall@10": 0.496969696969697,+ "recall@20": 0.6530303030303031,+ "mrr": 0.816017316017316,+ "p50_retrieve_ms": 160.231,+ "p95_retrieve_ms": 198.8395,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-slice-pec-lists/report.json",+ "recorded_at": "2026-08-18T16:47:26.669145+00:00"+ },+ {+ "run_id": "search-esci-slice-entities-neighbors",+ "brain": "searchbenchesci20",+ "dataset": "search_esci_slice.jsonl",+ "fusion": "rrf",+ "rerank": "none",+ "channels": [+ "entities"+ ],+ "rank_pool": false,+ "rank_pool_ce": false,+ "n_queries": 11,+ "ndcg@10": 0.6549492067418874,+ "ndcg@20": 0.7173885785786568,+ "ndcg": 0.7173885785786568,+ "recall@10": 0.4818181818181818,+ "recall@20": 0.8217171717171716,+ "mrr": 0.7568181818181817,+ "p50_retrieve_ms": 223.289,+ "p95_retrieve_ms": 271.031,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-slice-entities-neighbors/report.json",+ "recorded_at": "2026-08-18T16:48:04.631525+00:00"+ },+ {+ "run_id": "search-esci-74-passages-control",+ "brain": "searchbenchesci74",+ "dataset": "search_esci_74.jsonl",+ "fusion": "rrf",+ "rerank": "none",+ "channels": [+ "passages"+ ],+ "rank_pool": false,+ "rank_pool_ce": false,+ "n_queries": 74,+ "ndcg@10": 0.49528253069041633,+ "ndcg@20": 0.5490094202707876,+ "ndcg": 0.5490094202707876,+ "recall@10": 0.3769307112873572,+ "recall@20": 0.5920965629425753,+ "mrr": 0.7580598455598455,+ "p50_retrieve_ms": 61.7115,+ "p95_retrieve_ms": 91.52104999999995,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-74-passages-control/report.json",+ "recorded_at": "2026-08-18T17:13:00.463193+00:00"+ },+ {+ "run_id": "search-esci-74-ce-pool-l12-4class",+ "brain": "searchbenchesci74",+ "dataset": "search_esci_74.jsonl",+ "fusion": "none",+ "rerank": "4class-weighted-ce",+ "channels": [+ "rank-pool-4class"+ ],+ "rank_pool": true,+ "rank_pool_ce": true,+ "n_queries": 74,+ "ndcg@10": 0.5968432958924662,+ "ndcg@20": 0.6782580445181766,+ "ndcg": 0.7467109659912253,+ "recall@10": 0.5088965287461584,+ "recall@20": 0.7673275730894211,+ "mrr": 0.8351464398761695,+ "p50_retrieve_ms": 0.0,+ "p95_retrieve_ms": 0.0,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-74-ce-pool-l12-4class/report.json",+ "recorded_at": "2026-08-18T17:44:17.593025+00:00"+ },+ {+ "run_id": "search-esci-74-ce-pool-4class",+ "brain": "searchbenchesci74",+ "dataset": "search_esci_74.jsonl",+ "fusion": "none",+ "rerank": "4class-weighted-ce",+ "channels": [+ "rank-pool-4class"+ ],+ "rank_pool": true,+ "rank_pool_ce": true,+ "n_queries": 74,+ "ndcg@10": 0.5826375707951627,+ "ndcg@20": 0.6696127031036031,+ "ndcg": 0.7419804620261266,+ "recall@10": 0.49798997460620914,+ "recall@20": 0.7668403956654921,+ "mrr": 0.8317567567567568,+ "p50_retrieve_ms": 0.0,+ "p95_retrieve_ms": 0.0,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-74-ce-pool-4class/report.json",+ "recorded_at": "2026-08-18T17:54:14.073711+00:00"+ },+ {+ "run_id": "search-esci-74-ce-pool-4class-nowt",+ "brain": "searchbenchesci74",+ "dataset": "search_esci_74.jsonl",+ "fusion": "none",+ "rerank": "4class-weighted-ce",+ "channels": [+ "rank-pool-4class"+ ],+ "rank_pool": true,+ "rank_pool_ce": true,+ "n_queries": 74,+ "ndcg@10": 0.6096378704103984,+ "ndcg@20": 0.6881968860231006,+ "ndcg": 0.7594213054854562,+ "recall@10": 0.5249760000665886,+ "recall@20": 0.7631438693842529,+ "mrr": 0.8530036338859869,+ "p50_retrieve_ms": 0.0,+ "p95_retrieve_ms": 0.0,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-74-ce-pool-4class-nowt/report.json",+ "recorded_at": "2026-08-18T18:20:20.750483+00:00"+ },+ {+ "run_id": "search-esci-74-ce-pool-4class-e2",+ "brain": "searchbenchesci74",+ "dataset": "search_esci_74.jsonl",+ "fusion": "none",+ "rerank": "4class-weighted-ce",+ "channels": [+ "rank-pool-4class"+ ],+ "rank_pool": true,+ "rank_pool_ce": true,+ "n_queries": 74,+ "ndcg@10": 0.6169746541917066,+ "ndcg@20": 0.6945278030601721,+ "ndcg": 0.7652624576380807,+ "recall@10": 0.5214445444747297,+ "recall@20": 0.7719306369596903,+ "mrr": 0.8525418275418275,+ "p50_retrieve_ms": 0.0,+ "p95_retrieve_ms": 0.0,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-74-ce-pool-4class-e2/report.json",+ "recorded_at": "2026-08-18T18:48:00.939749+00:00"+ },+ {+ "run_id": "search-esci-74-passages-k50",+ "brain": "searchbenchesci74",+ "dataset": "search_esci_74.jsonl",+ "fusion": "rrf",+ "rerank": "none",+ "channels": [+ "passages"+ ],+ "rank_pool": false,+ "rank_pool_ce": false,+ "n_queries": 74,+ "k": 50,+ "ndcg@10": 0.4998984443226263,+ "ndcg@20": 0.5522842574567189,+ "ndcg@50": 0.6378451547901237,+ "ndcg": 0.6378451547901237,+ "recall@10": 0.378732513089159,+ "recall@20": 0.589865760711773,+ "recall@50": 0.833954370548067,+ "mrr": 0.7645849420849421,+ "pool_coverage": null,+ "p50_retrieve_ms": 60.091,+ "p95_retrieve_ms": 88.02255,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-74-passages-k50/report.json",+ "recorded_at": "2026-08-18T20:31:55.998500+00:00"+ },+ {+ "run_id": "search-esci-74-passages-k100",+ "brain": "searchbenchesci74",+ "dataset": "search_esci_74.jsonl",+ "fusion": "rrf",+ "rerank": "none",+ "channels": [+ "passages"+ ],+ "rank_pool": false,+ "rank_pool_ce": false,+ "n_queries": 74,+ "k": 100,+ "ndcg@10": 0.4998055248060819,+ "ndcg@20": 0.5523315287868084,+ "ndcg@50": 0.6375525242924566,+ "ndcg": 0.6564185982755066,+ "recall@10": 0.378732513089159,+ "recall@20": 0.5913672622132746,+ "recall@50": 0.830951367545064,+ "mrr": 0.76497727997728,+ "pool_coverage": null,+ "p50_retrieve_ms": 57.553,+ "p95_retrieve_ms": 81.81569999999999,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-74-passages-k100/report.json",+ "recorded_at": "2026-08-18T20:32:27.553496+00:00"+ },+ {+ "run_id": "search-esci-74-passages-k50-pool",+ "brain": "searchbenchesci74",+ "dataset": "search_esci_74.jsonl",+ "fusion": "rrf",+ "rerank": "none",+ "channels": [+ "passages"+ ],+ "rank_pool": true,+ "rank_pool_ce": false,+ "n_queries": 74,+ "k": 50,+ "ndcg@10": 0.5668024300832123,+ "ndcg@20": 0.624102711461205,+ "ndcg@50": 0.6826188564511254,+ "ndcg": 0.6947261855882534,+ "recall@10": 0.4507852781809733,+ "recall@20": 0.6686543312372134,+ "recall@50": 0.8324528690465655,+ "mrr": 0.8271396396396397,+ "pool_coverage": 0.778222315718655,+ "p50_retrieve_ms": 60.1315,+ "p95_retrieve_ms": 86.49954999999997,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-74-passages-k50-pool/report.json",+ "recorded_at": "2026-08-18T20:33:02.203247+00:00"+ },+ {+ "run_id": "search-esci-74-passages-k100-pool",+ "brain": "searchbenchesci74",+ "dataset": "search_esci_74.jsonl",+ "fusion": "rrf",+ "rerank": "none",+ "channels": [+ "passages"+ ],+ "rank_pool": true,+ "rank_pool_ce": false,+ "n_queries": 74,+ "k": 100,+ "ndcg@10": 0.5737527941225129,+ "ndcg@20": 0.6352462971440838,+ "ndcg@50": 0.7092403328584962,+ "ndcg": 0.7183777959320791,+ "recall@10": 0.46580029319598837,+ "recall@20": 0.6964864396614271,+ "recall@50": 0.9062083174986402,+ "mrr": 0.836599099099099,+ "pool_coverage": 0.8462618325454071,+ "p50_retrieve_ms": 58.5245,+ "p95_retrieve_ms": 82.5653,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-74-passages-k100-pool/report.json",+ "recorded_at": "2026-08-18T20:33:39.620042+00:00"+ },+ {+ "run_id": "search-esci-74-passages-k50-ce",+ "brain": "searchbenchesci74",+ "dataset": "search_esci_74.jsonl",+ "fusion": "rrf",+ "rerank": "harness:cross-encoder",+ "channels": [+ "passages"+ ],+ "rank_pool": false,+ "rank_pool_ce": false,+ "n_queries": 74,+ "k": 50,+ "ndcg@10": 0.4478964444706324,+ "ndcg@20": 0.5146313751051845,+ "ndcg@50": 0.6060742524325241,+ "ndcg": 0.6060742524325241,+ "recall@10": 0.331273452580599,+ "recall@20": 0.5538599992903688,+ "recall@50": 0.833954370548067,+ "mrr": 0.7197929141477528,+ "pool_coverage": null,+ "p50_retrieve_ms": 60.091,+ "p95_retrieve_ms": 88.02255,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-74-passages-k50-ce/report.json",+ "recorded_at": "2026-08-18T20:34:39.854467+00:00"+ },+ {+ "run_id": "search-esci-74-passages-k50-ce-l12",+ "brain": "searchbenchesci74",+ "dataset": "search_esci_74.jsonl",+ "fusion": "rrf",+ "rerank": "harness:cross-encoder",+ "channels": [+ "passages"+ ],+ "rank_pool": false,+ "rank_pool_ce": false,+ "n_queries": 74,+ "k": 50,+ "ndcg@10": 0.4674585131694284,+ "ndcg@20": 0.5084668214514952,+ "ndcg@50": 0.6140738833603318,+ "ndcg": 0.6140738833603318,+ "recall@10": 0.3412172585548375,+ "recall@20": 0.5191872363321448,+ "recall@50": 0.833954370548067,+ "mrr": 0.7063349063349063,+ "pool_coverage": null,+ "p50_retrieve_ms": 60.091,+ "p95_retrieve_ms": 88.02255,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-74-passages-k50-ce-l12/report.json",+ "recorded_at": "2026-08-18T20:35:44.462471+00:00"+ },+ {+ "run_id": "search-esci-74-passages-k50-qrewrite",+ "brain": "searchbenchesci74",+ "dataset": "search_esci_74_qrewrite.jsonl",+ "fusion": "rrf",+ "rerank": "none",+ "channels": [+ "passages"+ ],+ "rank_pool": false,+ "rank_pool_ce": false,+ "n_queries": 74,+ "k": 50,+ "ndcg@10": 0.4998984443226263,+ "ndcg@20": 0.5522842574567189,+ "ndcg@50": 0.6378451547901237,+ "ndcg": 0.6378451547901237,+ "recall@10": 0.378732513089159,+ "recall@20": 0.589865760711773,+ "recall@50": 0.833954370548067,+ "mrr": 0.7645849420849421,+ "pool_coverage": null,+ "p50_retrieve_ms": 59.432500000000005,+ "p95_retrieve_ms": 82.09889999999999,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-74-passages-k50-qrewrite/report.json",+ "recorded_at": "2026-08-18T21:18:51.308816+00:00"+ },+ {+ "run_id": "search-esci-74-dense-ance-k50",+ "brain": "harness-local",+ "dataset": "search_esci_74.jsonl",+ "fusion": "none",+ "rerank": "none",+ "channels": [+ "harness-dense"+ ],+ "rank_pool": false,+ "rank_pool_ce": false,+ "n_queries": 74,+ "k": 50,+ "ndcg@10": 0.3960504272587966,+ "ndcg@20": 0.445560860839057,+ "ndcg@50": 0.5370854731938006,+ "ndcg": 0.5370854731938006,+ "recall@10": 0.3015013697338196,+ "recall@20": 0.4872087291019431,+ "recall@50": 0.7503156876126188,+ "mrr": 0.6471417846417846,+ "pool_coverage": null,+ "p50_retrieve_ms": 74.9370945944778,+ "p95_retrieve_ms": 74.9370945944778,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-74-dense-ance-k50/report.json",+ "recorded_at": "2026-08-18T21:22:24.054560+00:00"+ },+ {+ "run_id": "search-esci-74-passages-k50-ce-retrieved",+ "brain": "searchbenchesci74",+ "dataset": "search_esci_74.jsonl",+ "fusion": "rrf",+ "rerank": "harness:cross-encoder",+ "channels": [+ "passages"+ ],+ "rank_pool": false,+ "rank_pool_ce": false,+ "n_queries": 74,+ "k": 50,+ "ndcg@10": 0.4648895994453083,+ "ndcg@20": 0.5245865140860362,+ "ndcg@50": 0.6126753191137789,+ "ndcg": 0.6126753191137789,+ "recall@10": 0.35140463046557896,+ "recall@20": 0.5410788442403298,+ "recall@50": 0.833954370548067,+ "mrr": 0.7120610588001892,+ "pool_coverage": null,+ "p50_retrieve_ms": 60.091,+ "p95_retrieve_ms": 88.02255,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-74-passages-k50-ce-retrieved/report.json",+ "recorded_at": "2026-08-18T22:38:31.141079+00:00"+ },+ {+ "run_id": "search-esci-74-colbert-k50",+ "brain": "harness-local-colbert",+ "dataset": "search_esci_74.jsonl",+ "fusion": "none",+ "rerank": "none",+ "channels": [+ "harness-colbert"+ ],+ "rank_pool": false,+ "rank_pool_ce": false,+ "n_queries": 74,+ "k": 50,+ "ndcg@10": 0.43412701768229023,+ "ndcg@20": 0.4815601734708754,+ "ndcg@50": 0.5523204205368764,+ "ndcg": 0.5523204205368764,+ "recall@10": 0.3110168068124024,+ "recall@20": 0.5023603484326227,+ "recall@50": 0.7138820979266567,+ "mrr": 0.6689285195333583,+ "pool_coverage": null,+ "p50_retrieve_ms": 10063.364791662614,+ "p95_retrieve_ms": 10063.364791662614,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-74-colbert-k50/report.json",+ "recorded_at": "2026-08-18T22:51:16.979167+00:00"+ },+ {+ "run_id": "search-esci-74-bge-base-k50",+ "brain": "harness-local",+ "dataset": "search_esci_74.jsonl",+ "fusion": "none",+ "rerank": "none",+ "channels": [+ "harness-dense"+ ],+ "rank_pool": false,+ "rank_pool_ce": false,+ "n_queries": 74,+ "k": 50,+ "ndcg@10": 0.44136708015333975,+ "ndcg@20": 0.5041842920107446,+ "ndcg@50": 0.5892299329783445,+ "ndcg": 0.5892299329783445,+ "recall@10": 0.32179328339461255,+ "recall@20": 0.5251414009467592,+ "recall@50": 0.7989247943561789,+ "mrr": 0.7471303646913403,+ "pool_coverage": null,+ "p50_retrieve_ms": 408.0482843509762,+ "p95_retrieve_ms": 408.04828435097625,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-74-bge-base-k50/report.json",+ "recorded_at": "2026-08-18T22:52:53.742680+00:00"+ },+ {+ "run_id": "search-esci-74-union-bge-k50",+ "brain": "harness-local-union",+ "dataset": "search_esci_74.jsonl",+ "fusion": "rrf",+ "rerank": "none",+ "channels": [+ "harness-union"+ ],+ "rank_pool": false,+ "rank_pool_ce": false,+ "n_queries": 74,+ "k": 50,+ "ndcg@10": 0.47918374395281466,+ "ndcg@20": 0.5381296158524957,+ "ndcg@50": 0.6141255447421892,+ "ndcg": 0.6141255447421892,+ "recall@10": 0.36314016383757297,+ "recall@20": 0.5774058852814317,+ "recall@50": 0.8271070974286027,+ "mrr": 0.7653163449499656,+ "pool_coverage": null,+ "p50_retrieve_ms": 0.0,+ "p95_retrieve_ms": 0.0,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-74-union-bge-k50/report.json",+ "recorded_at": "2026-08-19T08:10:06.928384+00:00"+ },+ {+ "run_id": "search-esci-74-exhaustive-ce",+ "brain": "harness-local-exhaustive",+ "dataset": "search_esci_74.jsonl",+ "fusion": "none",+ "rerank": "4class-weighted-ce",+ "channels": [+ "exhaustive-4class"+ ],+ "rank_pool": false,+ "rank_pool_ce": false,+ "n_queries": 74,+ "k": 50,+ "ndcg@10": 0.4162185585932262,+ "ndcg@20": 0.4392534244105597,+ "ndcg@50": 0.5042093175793891,+ "ndcg": 0.5042093175793891,+ "recall@10": 0.26585896457775815,+ "recall@20": 0.4060812399456508,+ "recall@50": 0.6029270278490679,+ "mrr": 0.6578079129936715,+ "pool_coverage": null,+ "p50_retrieve_ms": 94253.11211881155,+ "p95_retrieve_ms": 94253.11211881155,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-74-exhaustive-ce/report.json",+ "recorded_at": "2026-08-19T10:05:59.463503+00:00"+ },+ {+ "run_id": "search-esci-74-catalog-ce-k50",+ "brain": "searchbenchesci74",+ "dataset": "search_esci_74.jsonl",+ "fusion": "rrf",+ "rerank": "plugin:cross-encoder",+ "channels": [+ "passages"+ ],+ "rank_pool": false,+ "rank_pool_ce": false,+ "n_queries": 74,+ "k": 50,+ "ndcg@10": 0.4674585131694284,+ "ndcg@20": 0.5084668214514952,+ "ndcg@50": 0.6140738833603318,+ "ndcg": 0.6140738833603318,+ "recall@10": 0.3412172585548375,+ "recall@20": 0.5191872363321448,+ "recall@50": 0.833954370548067,+ "mrr": 0.7063349063349063,+ "pool_coverage": null,+ "p50_retrieve_ms": 59.308499999999995,+ "p95_retrieve_ms": 86.88884999999999,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-74-catalog-ce-k50/report.json",+ "recorded_at": "2026-08-19T10:17:45.568452+00:00"+ },+ {+ "run_id": "search-esci-74-cascade-tail-k50",+ "brain": "harness-local-cascade",+ "dataset": "search_esci_74.jsonl",+ "fusion": "none",+ "rerank": "none",+ "channels": [+ "harness-cascade"+ ],+ "rank_pool": false,+ "rank_pool_ce": false,+ "n_queries": 74,+ "k": 50,+ "ndcg@10": 0.4998984443226263,+ "ndcg@20": 0.6194622913116794,+ "ndcg@50": 0.6686100079902446,+ "ndcg": 0.6686100079902446,+ "recall@10": 0.378732513089159,+ "recall@20": 0.7504072036031975,+ "recall@50": 0.8885964311815054,+ "mrr": 0.7679931554931555,+ "pool_coverage": null,+ "p50_retrieve_ms": 0.0,+ "p95_retrieve_ms": 0.0,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-74-cascade-tail-k50/report.json",+ "recorded_at": "2026-08-19T10:32:17.738702+00:00"+ },+ {+ "run_id": "search-esci-74-ce-pool-deberta-base",+ "brain": "searchbenchesci74",+ "dataset": "search_esci_74.jsonl",+ "fusion": "none",+ "rerank": "4class-weighted-ce",+ "channels": [+ "rank-pool-4class"+ ],+ "rank_pool": true,+ "rank_pool_ce": true,+ "n_queries": 74,+ "k": 40,+ "ndcg@10": 0.6492728519674476,+ "ndcg@20": 0.7101117922903459,+ "ndcg@50": null,+ "ndcg": 0.7772540710194906,+ "recall@10": 0.5183145850778,+ "recall@20": 0.7604145345598889,+ "recall@50": null,+ "mrr": 0.8646712599298806,+ "pool_coverage": null,+ "p50_retrieve_ms": 0.0,+ "p95_retrieve_ms": 0.0,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-74-ce-pool-deberta-base/report.json",+ "recorded_at": "2026-08-19T14:00:25.594211+00:00"+ },+ {+ "run_id": "search-esci-74-passages-k50-ce-deberta",+ "brain": "searchbenchsmoke",+ "dataset": "search_esci_74.jsonl",+ "fusion": "rrf",+ "rerank": "harness:cross-encoder",+ "channels": [+ "passages"+ ],+ "rank_pool": false,+ "rank_pool_ce": false,+ "n_queries": 74,+ "k": 50,+ "ndcg@10": 0.5101053946025641,+ "ndcg@20": 0.5568741362838021,+ "ndcg@50": 0.6365973490191127,+ "ndcg": 0.6365973490191127,+ "recall@10": 0.36281504203381093,+ "recall@20": 0.5467544580912761,+ "recall@50": 0.833954370548067,+ "mrr": 0.7457139784725992,+ "pool_coverage": null,+ "p50_retrieve_ms": 60.091,+ "p95_retrieve_ms": 88.02255,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-74-passages-k50-ce-deberta/report.json",+ "recorded_at": "2026-08-19T14:09:19.540811+00:00"+ },+ {+ "run_id": "search-esci-es-passages-k50",+ "brain": "searchbenchescies",+ "dataset": "search_esci_es.jsonl",+ "fusion": "rrf",+ "rerank": "none",+ "channels": [+ "passages"+ ],+ "rank_pool": false,+ "rank_pool_ce": false,+ "n_queries": 62,+ "k": 50,+ "ndcg@10": 0.5774393479403104,+ "ndcg@20": 0.6303553323648456,+ "ndcg@50": 0.7327853549091549,+ "ndcg": 0.7327853549091549,+ "recall@10": 0.35328224222955834,+ "recall@20": 0.5758587932370467,+ "recall@50": 0.9141280526823616,+ "mrr": 0.8704493087557603,+ "pool_coverage": null,+ "p50_retrieve_ms": 84.13,+ "p95_retrieve_ms": 96.16295000000001,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-es-passages-k50/report.json",+ "recorded_at": "2026-08-19T16:34:42.965337+00:00"+ },+ {+ "run_id": "search-italian-smoke",+ "brain": "searchbenchitsmoke",+ "dataset": "search_italian_smoke.jsonl",+ "fusion": "rrf",+ "rerank": "none",+ "channels": [+ "passages"+ ],+ "rank_pool": false,+ "rank_pool_ce": false,+ "n_queries": 3,+ "k": 20,+ "ndcg@10": 1.0,+ "ndcg@20": 1.0,+ "ndcg@50": null,+ "ndcg": 1.0,+ "recall@10": 1.0,+ "recall@20": 1.0,+ "recall@50": null,+ "mrr": 1.0,+ "pool_coverage": null,+ "p50_retrieve_ms": 25.33,+ "p95_retrieve_ms": 29.3206,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-italian-smoke/report.json",+ "recorded_at": "2026-08-19T16:37:37.730780+00:00"+ },+ {+ "run_id": "search-italian-smoke-inflect",+ "brain": "searchbenchitsmoke",+ "dataset": "search_italian_smoke_inflect.jsonl",+ "fusion": "rrf",+ "rerank": "none",+ "channels": [+ "passages"+ ],+ "rank_pool": false,+ "rank_pool_ce": false,+ "n_queries": 3,+ "k": 20,+ "ndcg@10": 1.0,+ "ndcg@20": 1.0,+ "ndcg@50": null,+ "ndcg": 1.0,+ "recall@10": 1.0,+ "recall@20": 1.0,+ "recall@50": null,+ "mrr": 1.0,+ "pool_coverage": null,+ "p50_retrieve_ms": 21.494,+ "p95_retrieve_ms": 23.211199999999998,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-italian-smoke-inflect/report.json",+ "recorded_at": "2026-08-19T16:37:46.108861+00:00"+ },+ {+ "run_id": "search-italian-smoke-inflect-fts",+ "brain": "searchbenchitsmoke",+ "dataset": "search_italian_smoke_inflect.jsonl",+ "fusion": "rrf",+ "rerank": "none",+ "channels": [+ "passages"+ ],+ "rank_pool": false,+ "rank_pool_ce": false,+ "n_queries": 3,+ "k": 20,+ "ndcg@10": 1.0,+ "ndcg@20": 1.0,+ "ndcg@50": null,+ "ndcg": 1.0,+ "recall@10": 1.0,+ "recall@20": 1.0,+ "recall@50": null,+ "mrr": 1.0,+ "pool_coverage": null,+ "p50_retrieve_ms": 33.56,+ "p95_retrieve_ms": 36.8936,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-italian-smoke-inflect-fts/report.json",+ "recorded_at": "2026-08-19T16:44:05.699330+00:00"+ },+ {+ "run_id": "search-italian-minilm-pipeline",+ "brain": "searchbenchitmmini",+ "dataset": "search_italian_smoke.jsonl",+ "fusion": "none",+ "rerank": "none",+ "channels": [+ "harness-dense"+ ],+ "rank_pool": false,+ "rank_pool_ce": false,+ "n_queries": 3,+ "k": 10,+ "ndcg@10": 1.0,+ "ndcg@20": 1.0,+ "ndcg@50": null,+ "ndcg": 1.0,+ "recall@10": 1.0,+ "recall@20": null,+ "recall@50": null,+ "mrr": 1.0,+ "pool_coverage": null,+ "p50_retrieve_ms": 188.19916666689096,+ "p95_retrieve_ms": 188.19916666689096,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-italian-minilm-pipeline/report.json",+ "recorded_at": "2026-08-19T19:27:33.792776+00:00"+ },+ {+ "run_id": "search-esci-es-spell-k50",+ "brain": "searchbenchescies",+ "dataset": "search_esci_es_spell.jsonl",+ "fusion": "rrf",+ "rerank": "none",+ "channels": [+ "passages"+ ],+ "rank_pool": false,+ "rank_pool_ce": false,+ "n_queries": 62,+ "k": 50,+ "ndcg@10": 0.5950327313726855,+ "ndcg@20": 0.6464666665333806,+ "ndcg@50": 0.7424579237775919,+ "ndcg": 0.7424579237775919,+ "recall@10": 0.36220476611566177,+ "recall@20": 0.5832920760052671,+ "recall@50": 0.9143064382925891,+ "mrr": 0.8865783410138248,+ "pool_coverage": null,+ "p50_retrieve_ms": 92.655,+ "p95_retrieve_ms": 107.2923,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-es-spell-k50/report.json",+ "recorded_at": "2026-08-19T19:28:14.068481+00:00"+ },+ {+ "run_id": "search-italian-smoke-extras-locale",+ "brain": "searchbenchitsmoke",+ "dataset": "search_italian_smoke.jsonl",+ "fusion": "rrf",+ "rerank": "none",+ "channels": [+ "passages"+ ],+ "rank_pool": false,+ "rank_pool_ce": false,+ "n_queries": 3,+ "k": 10,+ "ndcg@10": 1.0,+ "ndcg@20": 1.0,+ "ndcg@50": null,+ "ndcg": 1.0,+ "recall@10": 1.0,+ "recall@20": null,+ "recall@50": null,+ "mrr": 1.0,+ "pool_coverage": null,+ "p50_retrieve_ms": 17.126,+ "p95_retrieve_ms": 23.894000000000002,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-italian-smoke-extras-locale/report.json",+ "recorded_at": "2026-08-19T19:30:47.530480+00:00"+ },+ {+ "run_id": "search-italian-smoke-extras-nope",+ "brain": "searchbenchitsmoke",+ "dataset": "search_italian_smoke.jsonl",+ "fusion": "rrf",+ "rerank": "none",+ "channels": [+ "passages"+ ],+ "rank_pool": false,+ "rank_pool_ce": false,+ "n_queries": 3,+ "k": 10,+ "ndcg@10": 0.0,+ "ndcg@20": 0.0,+ "ndcg@50": null,+ "ndcg": 0.0,+ "recall@10": 0.0,+ "recall@20": null,+ "recall@50": null,+ "mrr": 0.0,+ "pool_coverage": null,+ "p50_retrieve_ms": 22.721,+ "p95_retrieve_ms": 24.2753,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-italian-smoke-extras-nope/report.json",+ "recorded_at": "2026-08-19T19:29:31.676418+00:00"+ },+ {+ "run_id": "search-esci-74-passages-k50-extras-sanity",+ "brain": "searchbenchesci74",+ "dataset": "search_esci_74.jsonl",+ "fusion": "rrf",+ "rerank": "none",+ "channels": [+ "passages"+ ],+ "rank_pool": false,+ "rank_pool_ce": false,+ "n_queries": 74,+ "k": 50,+ "ndcg@10": 0.4998984443226263,+ "ndcg@20": 0.5522842574567189,+ "ndcg@50": 0.6378451547901237,+ "ndcg": 0.6378451547901237,+ "recall@10": 0.378732513089159,+ "recall@20": 0.589865760711773,+ "recall@50": 0.833954370548067,+ "mrr": 0.7645849420849421,+ "pool_coverage": null,+ "p50_retrieve_ms": 63.1325,+ "p95_retrieve_ms": 88.86579999999998,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-74-passages-k50-extras-sanity/report.json",+ "recorded_at": "2026-08-19T19:35:44.545077+00:00"+ },+ {+ "run_id": "search-italian-smoke-inflect-fts-or",+ "brain": "searchbenchitsmoke",+ "dataset": "search_italian_smoke_inflect.jsonl",+ "fusion": "rrf",+ "rerank": "none",+ "channels": [+ "passages"+ ],+ "rank_pool": false,+ "rank_pool_ce": false,+ "n_queries": 3,+ "k": 20,+ "ndcg@10": 1.0,+ "ndcg@20": 1.0,+ "ndcg@50": null,+ "ndcg": 1.0,+ "recall@10": 1.0,+ "recall@20": 1.0,+ "recall@50": null,+ "mrr": 1.0,+ "pool_coverage": null,+ "p50_retrieve_ms": 26.364,+ "p95_retrieve_ms": 42.744899999999994,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-italian-smoke-inflect-fts-or/report.json",+ "recorded_at": "2026-08-19T20:16:24.241357+00:00"+ },+ {+ "run_id": "search-esci-74-cc-a03",+ "brain": "searchbenchesci74",+ "dataset": "search_esci_74.jsonl",+ "fusion": "cc",+ "rerank": "none",+ "channels": [+ "passages"+ ],+ "rank_pool": false,+ "rank_pool_ce": false,+ "n_queries": 74,+ "k": 50,+ "ndcg@10": 0.48296913346675313,+ "ndcg@20": 0.5353434819101804,+ "ndcg@50": 0.6235227747628354,+ "ndcg": 0.6235227747628354,+ "recall@10": 0.37594464118364007,+ "recall@20": 0.5859902261910772,+ "recall@50": 0.8324528690465655,+ "mrr": 0.7492780202562811,+ "pool_coverage": null,+ "p50_retrieve_ms": 62.8885,+ "p95_retrieve_ms": 89.98204999999999,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-74-cc-a03/report.json",+ "recorded_at": "2026-08-19T20:18:18.361436+00:00"+ },+ {+ "run_id": "search-esci-74-cc-a05",+ "brain": "searchbenchesci74",+ "dataset": "search_esci_74.jsonl",+ "fusion": "cc",+ "rerank": "none",+ "channels": [+ "passages"+ ],+ "rank_pool": false,+ "rank_pool_ce": false,+ "n_queries": 74,+ "k": 50,+ "ndcg@10": 0.4861505982058926,+ "ndcg@20": 0.5412686017716648,+ "ndcg@50": 0.62608241624179,+ "ndcg": 0.62608241624179,+ "recall@10": 0.37673766109430706,+ "recall@20": 0.5919104321112831,+ "recall@50": 0.8324528690465655,+ "mrr": 0.7604021879021879,+ "pool_coverage": null,+ "p50_retrieve_ms": 61.432,+ "p95_retrieve_ms": 92.56864999999999,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-74-cc-a05/report.json",+ "recorded_at": "2026-08-19T20:18:41.196658+00:00"+ },+ {+ "run_id": "search-esci-74-cc-a07",+ "brain": "searchbenchesci74",+ "dataset": "search_esci_74.jsonl",+ "fusion": "cc",+ "rerank": "none",+ "channels": [+ "passages"+ ],+ "rank_pool": false,+ "rank_pool_ce": false,+ "n_queries": 74,+ "k": 50,+ "ndcg@10": 0.4925393217389386,+ "ndcg@20": 0.5491258060245199,+ "ndcg@50": 0.6321844176410597,+ "ndcg": 0.6321844176410597,+ "recall@10": 0.3780890124456584,+ "recall@20": 0.5964401922862047,+ "recall@50": 0.833954370548067,+ "mrr": 0.7630083655083656,+ "pool_coverage": null,+ "p50_retrieve_ms": 60.5155,+ "p95_retrieve_ms": 89.04719999999998,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-74-cc-a07/report.json",+ "recorded_at": "2026-08-19T20:19:05.271923+00:00"+ },+ {+ "run_id": "search-esci-74-literal-fill-k50",+ "brain": "searchbenchesci74",+ "dataset": "search_esci_74.jsonl",+ "fusion": "rrf",+ "rerank": "none",+ "channels": [+ "passages"+ ],+ "rank_pool": false,+ "rank_pool_ce": false,+ "n_queries": 74,+ "k": 50,+ "ndcg@10": 0.4998984443226263,+ "ndcg@20": 0.45027838324634667,+ "ndcg@50": 0.5436008239474691,+ "ndcg": 0.5436008239474691,+ "recall@10": 0.378732513089159,+ "recall@20": 0.3929861145019264,+ "recall@50": 0.6549969089908488,+ "mrr": 0.7639237451737452,+ "pool_coverage": null,+ "p50_retrieve_ms": 49.019999999999996,+ "p95_retrieve_ms": 74.27314999999999,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-74-literal-fill-k50/report.json",+ "recorded_at": "2026-08-19T20:20:15.131585+00:00"+ },+ {+ "run_id": "search-esci-74-ltr-head-k50",+ "brain": "harness-local-ltr",+ "dataset": "search_esci_74.jsonl",+ "fusion": "none",+ "rerank": "ltr-pairwise",+ "channels": [+ "harness-ltr-head"+ ],+ "rank_pool": false,+ "rank_pool_ce": false,+ "n_queries": 74,+ "k": 50,+ "ndcg@10": 0.5146015046757452,+ "ndcg@20": 0.5567887112159354,+ "ndcg@50": 0.6406685515140392,+ "ndcg": 0.6406685515140392,+ "recall@10": 0.384165323643092,+ "recall@20": 0.5830174861607277,+ "recall@50": 0.833954370548067,+ "mrr": 0.7525579150579151,+ "pool_coverage": null,+ "p50_retrieve_ms": 0.0,+ "p95_retrieve_ms": 0.0,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-74-ltr-head-k50/report.json",+ "recorded_at": "2026-08-19T21:09:47.411344+00:00"+ },+ {+ "run_id": "search-esci-74-ltr-cefeat-k50",+ "brain": "harness-local-ltr",+ "dataset": "search_esci_74.jsonl",+ "fusion": "none",+ "rerank": "ltr-pairwise",+ "channels": [+ "harness-ltr-head"+ ],+ "rank_pool": false,+ "rank_pool_ce": false,+ "n_queries": 74,+ "k": 50,+ "ndcg@10": 0.5242780297400812,+ "ndcg@20": 0.5655195290162985,+ "ndcg@50": 0.6456032400547804,+ "ndcg": 0.6456032400547804,+ "recall@10": 0.38343544674019964,+ "recall@20": 0.5687827599719494,+ "recall@50": 0.833954370548067,+ "mrr": 0.7792567567567568,+ "pool_coverage": null,+ "p50_retrieve_ms": 0.0,+ "p95_retrieve_ms": 0.0,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-74-ltr-cefeat-k50/report.json",+ "recorded_at": "2026-08-20T07:14:39.761450+00:00"+ },+ {+ "run_id": "search-esci-74-ltr-deberta-k50",+ "brain": "harness-local-ltr",+ "dataset": "search_esci_74.jsonl",+ "fusion": "none",+ "rerank": "ltr-pairwise",+ "channels": [+ "harness-ltr-head"+ ],+ "rank_pool": false,+ "rank_pool_ce": false,+ "n_queries": 74,+ "k": 50,+ "ndcg@10": 0.5420964444237363,+ "ndcg@20": 0.580013195834899,+ "ndcg@50": 0.6509103805030941,+ "ndcg": 0.6509103805030941,+ "recall@10": 0.3872531182148088,+ "recall@20": 0.5863055009035371,+ "recall@50": 0.833954370548067,+ "mrr": 0.7744787644787645,+ "pool_coverage": null,+ "p50_retrieve_ms": 0.0,+ "p95_retrieve_ms": 0.0,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-74-ltr-deberta-k50/report.json",+ "recorded_at": "2026-08-20T07:43:07.017767+00:00"+ },+ {+ "run_id": "search-esci-74-ltr-deberta-lgbm-k50",+ "brain": "harness-local-ltr",+ "dataset": "search_esci_74.jsonl",+ "fusion": "none",+ "rerank": "ltr-lightgbm",+ "channels": [+ "harness-ltr-head"+ ],+ "rank_pool": false,+ "rank_pool_ce": false,+ "n_queries": 74,+ "k": 50,+ "ndcg@10": 0.5159432058815566,+ "ndcg@20": 0.5658120484311862,+ "ndcg@50": 0.6415336394479464,+ "ndcg": 0.6415336394479464,+ "recall@10": 0.37878019077485103,+ "recall@20": 0.5918308018978374,+ "recall@50": 0.833954370548067,+ "mrr": 0.7098312215959275,+ "pool_coverage": null,+ "p50_retrieve_ms": 0.0,+ "p95_retrieve_ms": 0.0,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-74-ltr-deberta-lgbm-k50/report.json",+ "recorded_at": "2026-08-20T07:48:40.794647+00:00"+ },+ {+ "run_id": "search-esci-ltr200-passages-k50",+ "brain": "searchbenchesciltr2",+ "dataset": "search_esci_ltr200.jsonl",+ "fusion": "rrf",+ "rerank": "none",+ "channels": [+ "passages"+ ],+ "rank_pool": false,+ "rank_pool_ce": false,+ "n_queries": 170,+ "k": 50,+ "ndcg@10": 0.38688497520690734,+ "ndcg@20": 0.4428785087667844,+ "ndcg@50": 0.5191838961643084,+ "ndcg": 0.5191838961643084,+ "recall@10": 0.3668275379939918,+ "recall@20": 0.5506847180507305,+ "recall@50": 0.7435914662537475,+ "mrr": 0.6599425680048517,+ "pool_coverage": null,+ "p50_retrieve_ms": 76.07849999999999,+ "p95_retrieve_ms": 152.37659999999983,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-ltr200-passages-k50/report.json",+ "recorded_at": "2026-08-20T10:45:30.571169+00:00"+ },+ {+ "run_id": "search-esci-74-ltr-deberta-train200",+ "brain": "harness-local-ltr",+ "dataset": "search_esci_74.jsonl",+ "fusion": "none",+ "rerank": "ltr-pairwise",+ "channels": [+ "harness-ltr-head"+ ],+ "rank_pool": false,+ "rank_pool_ce": false,+ "n_queries": 74,+ "k": 50,+ "ndcg@10": 0.5436096369070966,+ "ndcg@20": 0.5825504636416924,+ "ndcg@50": 0.6525856561608427,+ "ndcg": 0.6525856561608427,+ "recall@10": 0.3911684808472767,+ "recall@20": 0.5850220289937974,+ "recall@50": 0.833954370548067,+ "mrr": 0.7621138996138996,+ "pool_coverage": null,+ "p50_retrieve_ms": 0.0,+ "p95_retrieve_ms": 0.0,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-74-ltr-deberta-train200/report.json",+ "recorded_at": "2026-08-20T10:57:34.082897+00:00"+ },+ {+ "run_id": "search-esci-74-ltr-deberta-lgbm-train200",+ "brain": "harness-local-ltr",+ "dataset": "search_esci_74.jsonl",+ "fusion": "none",+ "rerank": "ltr-lightgbm",+ "channels": [+ "harness-ltr-head"+ ],+ "rank_pool": false,+ "rank_pool_ce": false,+ "n_queries": 74,+ "k": 50,+ "ndcg@10": 0.5333470812074246,+ "ndcg@20": 0.5739148370361776,+ "ndcg@50": 0.6521013344978932,+ "ndcg": 0.6521013344978932,+ "recall@10": 0.38938710235182716,+ "recall@20": 0.5909398273461974,+ "recall@50": 0.833954370548067,+ "mrr": 0.7517797970384177,+ "pool_coverage": null,+ "p50_retrieve_ms": 0.0,+ "p95_retrieve_ms": 0.0,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-74-ltr-deberta-lgbm-train200/report.json",+ "recorded_at": "2026-08-20T11:33:18.933082+00:00"+ },+ {+ "run_id": "search-esci-74-ltr-deberta-hybrid170",+ "brain": "harness-local-ltr",+ "dataset": "search_esci_74.jsonl",+ "fusion": "none",+ "rerank": "ltr-pairwise",+ "channels": [+ "harness-ltr-head"+ ],+ "rank_pool": false,+ "rank_pool_ce": false,+ "n_queries": 74,+ "k": 50,+ "ndcg@10": 0.5304449022081842,+ "ndcg@20": 0.5753013243082012,+ "ndcg@50": 0.6452218148321592,+ "ndcg": 0.6452218148321592,+ "recall@10": 0.38518108116293304,+ "recall@20": 0.5882930387092911,+ "recall@50": 0.833954370548067,+ "mrr": 0.7646718146718147,+ "pool_coverage": null,+ "p50_retrieve_ms": 0.0,+ "p95_retrieve_ms": 0.0,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-esci-74-ltr-deberta-hybrid170/report.json",+ "recorded_at": "2026-08-20T11:57:34.067863+00:00"+ },+ {+ "run_id": "search-wands-passages-k50",+ "brain": "searchbenchwands",+ "dataset": "search_wands.jsonl",+ "fusion": "rrf",+ "rerank": "none",+ "channels": [+ "passages"+ ],+ "rank_pool": false,+ "rank_pool_ce": false,+ "n_queries": 66,+ "k": 50,+ "ndcg@10": 0.823176135584089,+ "ndcg@20": 0.8120944688355884,+ "ndcg@50": 0.8223686140560469,+ "ndcg": 0.8223686140560469,+ "recall@10": 0.26880378393536286,+ "recall@20": 0.46518737110842373,+ "recall@50": 0.8365552576078893,+ "mrr": 0.9245931537598204,+ "pool_coverage": null,+ "p50_retrieve_ms": 86.595,+ "p95_retrieve_ms": 96.93674999999999,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-wands-passages-k50/report.json",+ "recorded_at": "2026-08-20T13:17:40.353988+00:00"+ },+ {+ "run_id": "search-wandsgraph-communities-k50",+ "brain": "searchbenchwandsgraph",+ "dataset": "search_wands.jsonl",+ "fusion": "rrf",+ "rerank": "none",+ "channels": [+ "communities"+ ],+ "rank_pool": false,+ "rank_pool_ce": false,+ "n_queries": 66,+ "k": 50,+ "ndcg@10": 0.4094084735740818,+ "ndcg@20": 0.36455789387786824,+ "ndcg@50": 0.325224291157982,+ "ndcg": 0.4570937299953597,+ "recall@10": 0.14824773180036338,+ "recall@20": 0.20624528979792137,+ "recall@50": 0.28701452056715215,+ "mrr": 0.6737914862914863,+ "pool_coverage": null,+ "p50_retrieve_ms": 5609.237499999999,+ "p95_retrieve_ms": 11856.0285,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-wandsgraph-communities-k50/report.json",+ "recorded_at": "2026-08-21T03:12:00.249304+00:00",+ "protocol": "architecture demo: isolated catalog graph on searchbenchwandsgraph; not a quality-default claim vs frozen WANDS passages 0.823 or ESCI 0.500",+ "claim": "architecture-demo"+ },+ {+ "run_id": "search-wandsgraph-passages-k50",+ "brain": "searchbenchwandsgraph",+ "dataset": "search_wands.jsonl",+ "fusion": "rrf",+ "rerank": "none",+ "channels": [+ "passages"+ ],+ "rank_pool": false,+ "rank_pool_ce": false,+ "n_queries": 66,+ "k": 50,+ "ndcg@10": 0.8230998576010691,+ "ndcg@20": 0.812056754200934,+ "ndcg@50": 0.8223486981324697,+ "ndcg": 0.8223486981324697,+ "recall@10": 0.26880378393536286,+ "recall@20": 0.46518737110842373,+ "recall@50": 0.8365552576078893,+ "mrr": 0.9245931537598204,+ "pool_coverage": null,+ "p50_retrieve_ms": 98.604,+ "p95_retrieve_ms": 115.12925,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-wandsgraph-passages-k50/report.json",+ "recorded_at": "2026-08-21T03:12:36.455586+00:00",+ "protocol": "architecture demo: isolated catalog graph on searchbenchwandsgraph; not a quality-default claim vs frozen WANDS passages 0.823 or ESCI 0.500",+ "claim": "architecture-demo"+ },+ {+ "run_id": "search-wandsgraph-passages-nodeid-k50",+ "brain": "searchbenchwandsgraph",+ "dataset": "search_wands.jsonl",+ "fusion": "rrf",+ "rerank": "none",+ "channels": [+ "passages"+ ],+ "rank_pool": false,+ "rank_pool_ce": false,+ "n_queries": 66,+ "k": 50,+ "ndcg@10": 0.8230998576010691,+ "ndcg@20": 0.812056754200934,+ "ndcg@50": 0.8223486981324697,+ "ndcg": 0.8223486981324697,+ "recall@10": 0.26880378393536286,+ "recall@20": 0.46518737110842373,+ "recall@50": 0.8365552576078893,+ "mrr": 0.9245931537598204,+ "pool_coverage": null,+ "p50_retrieve_ms": 84.0175,+ "p95_retrieve_ms": 105.24199999999999,+ "git_sha": "f42c4a8eaadb3c0cc79f78c34b0e1a70534bee10",+ "report_path": "runs/search-wandsgraph-passages-nodeid-k50/report.json",+ "recorded_at": "2026-08-21T07:53:39.311033+00:00",+ "protocol": "architecture demo: isolated catalog graph on searchbenchwandsgraph; not a quality-default claim vs frozen WANDS passages 0.823 or ESCI 0.500",+ "claim": "architecture-demo"+ }+ ]}}}diff --git a/docs/decisions/007-three-product-surfaces-one-kb.md b/docs/decisions/007-three-product-surfaces-one-kb.mdnew file mode 100644index 0000000..ff4b88f--- /dev/null+++ b/docs/decisions/007-three-product-surfaces-one-kb.md@@ -0,0 +1,128 @@+# ADR-007: Three Product Surfaces on One Knowledge Base++## Status++Accepted — 2026-08-18. Does not supersede ADR-005 or ADR-006. Those remain the memory-path contract. This ADR adds **search** as a first-class product beside memory and recommendations, and states how they share the same brain without sharing request semantics.++Extended by [ADR-008](008-query-gated-search-personalization.md) (Accepted 2026-08-21): optional search `target` reranks retrieved hits. The omitted-`target` path and this ADR’s SLO are unchanged.++Locked §15 resolutions:++1. Routes stay under `/retrieve` as `GET|POST /retrieve/search` (not a new top-level prefix). Existing `GET /retrieve/` is unchanged.+2. Search is env-gated (`SEARCH_ENABLED`, also in TUI init/config). v1 lexical is Okapi BM25 over Postgres `tsvector` + GIN on the stock `pgvector/pgvector:pg16` image — not `ILIKE`, not `ts_rank_cd`, not `rum` / ParadeDB `pg_search`.+3. Dense quality at 3072-d: keep float32 storage; build HNSW on a `halfvec` expression (legal to 4000-d); exact float32 cosine rerank of the overfetch window. Do not truncate embedding dimensions.+4. `plugins/chatbot-memory` is out of scope for v1.+5. Search SLO: p50 **< 200 ms excluding embed RTT**. `profile_stages` splits `embed.query` from retrieve so the budget is measurable.+6. Operators switch dense / BM25 / both via `SEARCH_USE_DENSE` / `SEARCH_USE_BM25`; default is both fused with `SEARCH_FUSION=rrf`. `/retrieve/context` stays ILIKE unless `SEARCH_ENABLED=true` and `CONTEXT_PASSAGE_MODE` is `hybrid` / `bm25` / `dense`.++## Date++2026-08-18++## Context++`docs/research/00-scope-and-constraints.md` already names the product as a memory layer that is also “a substrate for agents, recommenders, and search engines.” In code those are not three products yet:++| Intended product | Shipped surface | What the caller actually gets |+| --- | --- | --- |+| Agent / conversational memory | `POST /retrieve/context` | Prompt-shaped blob: passages + event facts + topics. Budget **p50 < 1000 ms**, no LLM (ADR-006). Evaluated on LoCoMo / LongMemEval / BEAM. |+| Recommendations | `GET|POST /retrieve/recommend` | Ranked **nodes** (items) from graph walks, synergies, optional attribute prefs. Evaluated on `demorecsys` (`16-recsys-eval-protocol.md`). Heavy CF is a plugin (`plugins/recsys-gnn`). |+| Search | **Missing** | `GET /retrieve/` is entity-biased graph+data lookup. Passage “hybrid” in context is dense ANN ∪ **`ILIKE '%query%'`**, fused with RRF — not BM25, not a hit list API, not a search SLO. |++Default pgvector installs ship **3072-d** embeddings and skip HNSW (`vectors.py` returns no DDL above 2000 dims). Context p50 after dossier deletion is **~1.3–1.4 s**, residual dominated by **serial embedding HTTP**, not ANN. Passage evidence-session recall on LoCoMo conv-26 is already **~97%**; graph vs passages-only does not move judge accuracy above the 5–7 point noise floor.++If search is implemented by stuffing more ranking into `/retrieve/context`, two failures follow: (1) memory benchmarks and the sub-second budget become the search SLO, which they cannot be; (2) search users receive a concatenated context blob instead of scored hits with filters, facets, and stable document IDs.++The literature that actually transfers is first-stage IR (BM25, dense dual-encoders, hybrid fusion) and BEIR’s finding that **lexical and dense are complementary off-distribution** — not MS MARCO leaderboard recipes that assume a 200 ms p99 web index and a cross-encoder on every query. Analysis and task breakdown: `docs/research/17-search-surface-and-cross-features.md`.++## Decision++**1. One write path, three read products.**++A brain remains the unique source of truth. Ingest (`POST /ingest/`, structured ingest, observations) writes chunks, embeddings, event graph, topics, and (after this work) a real lexical index **once**. Read products are separate HTTP contracts:++| Product | Canonical API | Response shape | Default SLO | Benchmarks |+| --- | --- | --- | --- | --- |+| Memory | `POST /retrieve/context` | `text_context`, `triples`, passages, topics, paths | p50 < 1000 ms; no LLM | LoCoMo, LongMemEval, BEAM |+| Search | **`POST /retrieve/search`** (new) | Ranked **hits** (id, channel, score, snippet, extras) | p50 < 200 ms excluding embed RTT; p95 budget TBD from telemetry | New `benchmarks.search` (not LoCoMo judge) |+| Recommendations | `GET|POST /retrieve/recommend` | Ranked **nodes** + channel | Unchanged | `benchmarks.recsys` on `demorecsys` only |++MCP / deep REST remains the expensive escalation lane (ADR-006): agentic traversal, cross-encoder, sufficiency retry, HyDE-style query expansion.++**2. Do not overload `/retrieve/context` into a search engine.**++Context stays a **memory assembler** (evidence for an answering model). Search stays a **ranker of addressable objects** (chunks, entities, events, optional structured records). A search client may *call* context afterwards; the APIs do not merge.++**3. Share write-time indexes; isolate query-time defaults.**++Shared (one index, many readers):++- Vector tables + a **legal ANN index** (HNSW or `halfvec` HNSW at 3072-d; loud failure if skipped).+- Lexical postings (`tsvector` / BM25 or equivalent) over chunk text and entity names.+- Provenance, validity (`invalid_at`, `deprecated`), `happened_at`, topic memberships, hub-bridge table.++Isolated (per product, flags on the request, defaults frozen per surface):++- Fusion method and weights (RRF vs convex combination).+- Candidate fanout, channels enabled, graph PPR, fact diversification.+- Rerankers. Core: linear / small GBDT over already-computed scores. Plugins: SPLADE, ColBERT/PLAID, cross-encoder.++`/retrieve/context` **must not** change its default ranking because search shipped. Opt-in only (`lexical=bm25` on context, or a one-way cutover after the non-regression gate).++**4. Core vs plugin, same pattern as recsys.**++| Layer | Owns | Analog |+| --- | --- | --- |+| Core | BM25 (or `tsvector` rank), dense ANN, RRF/CC fusion, filters, cheap feature ranker, search API | `/retrieve/recommend` train-free graph ranker |+| Plugin (optional) | Learned sparse (SPLADE family), late interaction (ColBERT/PLAID), cross-encoder rerank, instruction-tuned embedders | `plugins/recsys-gnn`, `plugins/features-rec` |++A missing plugin degrades search to core hybrid, never 500s the memory path.++**5. Non-regression is a gate, not a hope.**++Any change that touches a **shared** index or the default context path:++1. Paired LoCoMo (or the current memory champion arm) McNemar vs the last accepted product/SOTA run: **no statistically significant accuracy drop**; point estimate not lower. Prefer retrieval-side **passage EvR held** (deterministic) over judge accuracy.+2. Context p50 **not worse** than the post-dossier baseline (~1.4 s) by more than measurement noise; the sub-second target remains ADR-006’s, not this ADR’s.+3. Graph EvR agreement gate from `00` (≥95% identical session sets) still applies if seeds/ANN change.+4. Recsys HitRate on `demorecsys` unchanged unless the change is explicitly a recs feature.+5. Search and recs eval **never** ingest into or wipe `locomoconv26*`, `beam*clean`, or other memory brains.++Search quality is **not** judged by LoCoMo LLM-as-judge. It uses ranking metrics (Recall@k, nDCG@k, MRR) on a dedicated query set.++**6. Cross-features are allowed when they are indexes or cheap scores, not when they are product-specific loops.**++Allowed to help more than one product: BM25, HNSW, batched embeddings, metadata filters, topic coarse-to-fine, recency/validity features, a linear blend of dense+lexical+PPR.++Forbidden on the memory hot path, allowed on search-deep or plugins: per-query LLM (HyDE), cross-encoder over large K, ANCE training loops, ColBERT serving.++## Alternatives considered++### Implement search by adding flags to `/retrieve/context`+- Pros: one endpoint; less API surface.+- Cons: prompt blob ≠ hit list; memory SLO and search SLO conflict; LoCoMo harness would pick up search ranking noise.+- Rejected.++### Separate search database (Elastic/Meilisearch) as source of truth+- Pros: mature BM25.+- Cons: two write paths, drift from the event graph, contradicts “unique source of truth.”+- Rejected as the system of record. An **adapter** that *projects* brain indexes into an external engine is a plugin, not a second brain.++### Put a distilled cross-encoder on every search and context query+- Pros: largest precision jump in the MS MARCO literature.+- Cons: ADR-006 already rejected this on context (50–200 ms, 5–20% of budget). Search can add it as an **optional** second stage with small K.+- Rejected as default; accepted as plugin / `rerank=cross_encoder` on search only.++### Train a BrainAPI-specific bi-encoder (DPR/ANCE) before shipping BM25+- Pros: best dense quality in-domain.+- Cons: needs relevance labels and an index-refresh loop; LoCoMo N cannot support it; BEIR shows off-the-shelf dense is weak zero-shot without lexical backup.+- Rejected as a prerequisite. Off-the-shelf embeddings + real BM25 first.++## Consequences++- New route family under `/retrieve/search*`. Existing `/retrieve/context` and `/retrieve/recommend` keep their contracts.+- Lexical storage and ANN legality become substrate work (`03`), not a search-plugin secret.+- `benchmarks/search/` and `benchmarks.search` in `REPORTS.json` are required before claiming search quality, mirroring recsys isolation.+- Memory SOTA work (`12`, `13`) may **opt into** BM25∪dense RRF on context after the gate — HyperMem (`2604.08256`) already uses that hybrid — but shipping search does not force that opt-in.+- Documentation in `00` gains workstream `17`. Implementation does not start from this ADR alone; tasks live in `docs/research/17-search-surface-and-cross-features.md`.+- Optional query-gated personalize on search (`target`) is [ADR-008](008-query-gated-search-personalization.md). It must not change omitted-`target` ranking, default `channels=["passages"]`, or this ADR’s 200 ms SLO.diff --git a/docs/decisions/008-query-gated-search-personalization.md b/docs/decisions/008-query-gated-search-personalization.mdnew file mode 100644index 0000000..fe17bd8--- /dev/null+++ b/docs/decisions/008-query-gated-search-personalization.md@@ -0,0 +1,97 @@+# ADR-008: Query-Gated Personalized Search on Retrieved Hits++## Status++Accepted — 2026-08-21. Does not supersede ADR-007. Search remains a separate product from recommendations. This ADR locks how identity and taste may touch `/retrieve/search` without sharing recs request semantics or becoming the default ranker.++Locked resolutions:++1. Identity on search is optional `target` (same name as `/retrieve/recommend`). The field is never `user_id`.+2. First-stage stays query → passage (omitted `channels=["passages"]`). Personalize reorders **already retrieved** hits; it does not call `EntityRecommendRetriever.recommend()`.+3. User-asserted `extras` remains a hard AND filter. Inferred taste is blend-only (I7 rejected).+4. HEM / TEM / ZAM training is deferred (I6). WANDS and ESCI cannot supply query×user labels.+5. Omitted-`target` ranking and latency stay the ADR-007 product path (p50 < 200 ms excluding embed). Personalized search is opt-in.++## Date++2026-08-21++## Context++ADR-007 shipped search as a ranked hit list with its own SLO and eval (`benchmarks.search`). Catalog work ([24-search-catalog-graph.md](../research/24-search-catalog-graph.md)) attached `SearchHit.node_id` so a client can walk the same KB. Recommendations already take `target` on `/retrieve/recommend` and are scored as next-item HitRate on `demorecsys` ([16-recsys-eval-protocol.md](../research/16-recsys-eval-protocol.md)).++Three failures follow if personalize is implemented as “recs on the search path”:++1. **Wrong first-stage.** Hub-walk / recommend candidate generation on `searchbenchwandsgraph` recovered Recall@50 **0.287** vs passages **0.837**. That is missing gold, not a ranking defect.+2. **Wrong training data.** HEM (SIGIR 2017, DOI `10.1145/3077136.3080813`) and TEM (arXiv `2005.08936`) need query×user×item logs. Frozen WANDS and ESCI have no users.+3. **Wrong filter.** Writing `extras` from inferred style AND-drops relevant hits the user did not exclude.++A fourth failure is **wrong identity**: a `user_id` field on `SearchRequestBody` forks the recommend contract and invites mixing HitRate with nDCG.++Literature that actually transfers is **query-dependent** personalization — ZAM (arXiv `1908.11322`) and TEM both record that personalization does not always help — not always-on user vectors and not replacing BM25∪dense.++Analysis: [25-personalized-search.md](../research/25-personalized-search.md).++## Decision++**1. Optional `target` on search; omit means anonymous.**++`SearchRequestBody.target` and `GET /retrieve/search?target=` accept a USER uuid or id. Empty / omitted → no personalize call; hit order equals the pre-ship retrieve order. Unknown user → no-op. Schema forbids `user_id`, `product_id`, `sku`, and `brand` on the search body.++**2. Rerank retrieved hits; do not generate candidates.**++Personalize runs after extras AND and `node_id` attach, before the final `k` cut. Catalog `mode` personalizes `k_ret`. Hits without `node_id` get pref 0 and stay in the list. All-zero prefs preserve order. Never drop an id because taste did not match.++Do not call `EntityRecommendRetriever.recommend()`. Score only the given node ids by ATTR-hub overlap with `user_pref_weights`.++**3. Query-gated blend, not always-on taste.**++λ is 0 when the query has digits / SKU-like tokens; otherwise it falls with token count (one token high, four+ low). Blend is min-max retrieve vs pref. This is a cheap stand-in for ZAM/TEM’s “personalize only when the query is under-specified,” not a trained attention model.++**4. Write taste as PREFERS; keep EVENT for dated behavior.**++Interaction `options` / `attributes` emit `USER -PREFERS-> ATTR` with the same hub uuid as catalog HAS. Options are not copied onto the product catalog blob. View / cart / purchase / favorite stay EVENT wrappers with `happened_at`. Architect persists any no-event direct triple, not only static HAS.++Favorite / wishlist weight **0.7**; unknown behaviors **0.2**. Search pref vectors combine write-time PREFERS with recency-decayed EVENT→ITEM→ATTR (14-day half-life) even when PREFERS exist.++**5. Inferred taste is not a filter.**++User-asserted `extras` stays AND. The ranker must not inject `extras={style: …}` from history.++**6. Eval isolation.**++Unit tests in `tests/test_search_personalize.py`. Optional skip-ingest smoke only on `searchbenchwandsgraph`. Never wipe frozen `searchbench*` quality brains, `locomoconv*`, `beam*`, or `demorecsys`. Do not quote vs frozen WANDS **0.823** / ESCI **0.500**. Do not score recsys HitRate as search nDCG. If ledgered: `claim: architecture-demo` only.++## Alternatives considered++### Add `user_id` on `SearchRequestBody`+- Pros: matches ingest JSONL and recsys datasets.+- Cons: two identity names across retrieve; invites mixing HitRate with nDCG.+- Rejected. Reuse `target`.++### Call `/retrieve/recommend` (or `EntityRecommendRetriever`) as search first-stage+- Pros: one ranking implementation.+- Cons: different task (next-item vs query); communities/graph first-stage already lost Recall@50 on the architecture-demo brain.+- Rejected.++### Infer `extras` from PREFERS / recent events+- Pros: “personalize” looks like faceted search.+- Cons: AND-drops relevant hits the caller did not filter; confuses asserted constraints with taste.+- Rejected (I7).++### Train HEM / TEM / ZAM before shipping a personalize knob+- Pros: literature SOTA for personalized product search.+- Cons: no query×user logs on WANDS/ESCI; would not be the 200 ms default; blocks the opt-in rerank that the graph already supports.+- Deferred (I6). Not a prerequisite.++### Always-on personalize whenever `target` is present+- Pros: simpler than λ.+- Cons: ZAM/TEM evidence that specific queries get worse; navigational / SKU queries must stay retrieve-order.+- Rejected.++## Consequences++- `SearchHitScores.personalize` is additive; omitted-`target` responses keep `personalize` unset.+- Core library: `src/core/search/personalize.py`. Recommend candidate generation is unchanged aside from I4 weights.+- Recommend `_attribute_pref_targets` still skips EVENT walks when PREFERS exist; search personalize does not. That split is intentional until recs explicitly adopt the combine.+- Research record: `docs/research/25-personalized-search.md`. ADR-007 SLO and default channels stay the omitted-`target` path.diff --git a/docs/research/00-scope-and-constraints.md b/docs/research/00-scope-and-constraints.mdindex 6dd8736..cc39012 100644--- a/docs/research/00-scope-and-constraints.md+++ b/docs/research/00-scope-and-constraints.md@@ -40,6 +40,15 @@ Anything expensive must be either moved to write time or exposed only on the dee| `13-locomo-93-research-levers.md` | Evidence-backed research levers for HyperMem-class ≥93% (harness-first) || `15-ecommerce-gnn-recsys-landscape.md` | External GNN/e-commerce RecSys curriculum & industry index; BrainAPI mapping || `16-recsys-eval-protocol.md` | Held-out next-item eval on demorecsys via structured ingest + /retrieve/recommend |+| `17-search-surface-and-cross-features.md` | Search as a third product surface on the same KB; shared indexes; non-regression vs LoCoMo/BEAM/recsys |+| `18-search-eval-protocol.md` | Hybrid BM25+dense search eval on `searchbench*`; ledger `benchmarks.search`; no LoCoMo judge |+| `19-search-esci-quality.md` | ESCI US first-stage / rerank quality track; frozen `searchbenchesci74` |+| `20-esci-sota-targets.md` | External ESCI / commerce-search SOTA numbers; what we may and may not quote |+| `21-esci-ce-graded-and-first-stage.md` | Graded CE and first-stage plugin arms; not the ADR-007 200 ms default |+| `22-multilingual-ecommerce-search.md` | Multilingual first-stage (ESCI ES/JP; Italian smoke, not ESCI) and production commerce plan; `/retrieve/search` only |+| `23-search-wands-quality.md` | Frozen WANDS passages control (nDCG@10 **0.823**); not a catalog-graph claim |+| `24-search-catalog-graph.md` | Opt-in catalog graph + `node_id` sidecar; architecture-demo on `searchbenchwandsgraph` |+| `25-personalized-search.md` | Query-gated rerank of already retrieved hits via optional `target`; ADR-008 |## Decisions already taken during researchdiff --git a/docs/research/16-recsys-eval-protocol.md b/docs/research/16-recsys-eval-protocol.mdindex 87f2270..d1cb27a 100644--- a/docs/research/16-recsys-eval-protocol.md+++ b/docs/research/16-recsys-eval-protocol.md@@ -14,7 +14,7 @@ Freshness: each structured ingest changes graph walks / attribute prefs immediat**Not** LoCoMo / LongMemEval / BEAM. Never write to or wipe `beam1m1clean`, `locomoconv26*`, or other memory-eval brains. Ledger upserts go only to `benchmarks.recsys` in [`benchmarks/REPORTS.json`](../../benchmarks/REPORTS.json).-Related: [ADR-002](../decisions/002-structured-ingestion-specific-processing.md), [15-ecommerce-gnn-recsys-landscape.md](15-ecommerce-gnn-recsys-landscape.md).+Related: [ADR-002](../decisions/002-structured-ingestion-specific-processing.md), [15-ecommerce-gnn-recsys-landscape.md](15-ecommerce-gnn-recsys-landscape.md), [ADR-008](../decisions/008-query-gated-search-personalization.md) (search `target` is not this protocol).---@@ -100,3 +100,4 @@ Requires `BRAINPAT_TOKEN` in `benchmarks/.env`. Default brain: `demorecsys`. Rep- Default harness backend is **graph** (`GET /retrieve/recommend`); use `--backend lightgcn` only for the offline comparator.- Do **not** mutate `benchmarks.locomo` / `beam` / `longmemeval` ledger rows.- Do **not** point `--brain` at memory-eval brains.+- Do **not** score recsys HitRate as search nDCG. Optional `/retrieve/search?target=` is [25-personalized-search.md](25-personalized-search.md) / ADR-008 on `searchbench*` only, never `demorecsys`.diff --git a/docs/research/17-search-surface-and-cross-features.md b/docs/research/17-search-surface-and-cross-features.mdnew file mode 100644index 0000000..3792cf7--- /dev/null+++ b/docs/research/17-search-surface-and-cross-features.md@@ -0,0 +1,595 @@+# 17 — Search surface on the shared knowledge base++Workstream: make BrainAPI a **unique source of truth** that already powers memory and recommendations, and can also power **search**, with shared write-time features and isolated query contracts. Binding: `00-scope-and-constraints.md`, ADR-005, ADR-006, **ADR-007** (Accepted). Recs isolation: `16-recsys-eval-protocol.md`. Memory SOTA: `08`, `12`, `13`. Substrate indexes: `03` §G.++**Plan mode.** This document does not change `src/` or harnesses. Tasks are sized for later implementers.++Skills applied: scientific-brainstorming (ideas labeled, no automatic winner), scientific-critical-thinking (severity, construct validity), planning-and-task-breakdown (vertical slices, gates), paper-lookup (arXiv 2026-08-18).++---++## 0. Scope of this session++**Focal question.** How can one brain serve (a) agent memory, (b) document/entity search, and (c) recommendations as *separate products* that share indexes and cheap scores, without making LoCoMo / LongMemEval / BEAM / recsys **worse**?++| | Record |+| --- | --- |+| Purpose | Architecture + literature + implementable plan |+| Audience | Maintainer; retrieval / substrate / eval owners |+| Decision owner | Maintainer (ADR-007 Accepted 2026-08-18; §15 locked below) |+| Time horizon | Core search API in weeks; plugins and LTR later |+| In scope | Read APIs, lexical+ANN indexes, fusion, cheap rankers, search eval, plugin boundary, cross-features that are indexes or scores |+| Out of scope | Replacing ingest; training a proprietary bi-encoder as a prerequisite; making `/retrieve/context` the search engine; MS MARCO as a product SLO; dual-use / biosafety (not implicated) |+| Real constraints | ADR-006: no LLM on context path; p50 < 1000 ms target unmet (~1.4 s); judge cannot resolve <5–7 LoCoMo points; 3072-d HNSW skipped; keyword path is ILIKE |+| Assumed | Event-leg supersession bug stays fixed going forward; existing unrepaired brains may still exist |+| Negotiable | Whether context *later* opts into BM25; search p95 number; Postgres FTS vs external inverted-index plugin |+| Unknown | Keyword vs paraphrase mix of future search tenants; corpus size per brain in production |+| Prohibited | LoCoMo judge as a search metric; wiping memory eval brains; silent default changes on `/retrieve/context` |++**Perspectives represented:** retrieval engineering, memory eval, recsys (existing plugin split), IR literature. **Missing:** a production search-traffic owner (no click log). Do not treat LoCoMo questions as a substitute for that mix.++---++## 1. Product model (decision, proposed)++Three **read** products, one **write** path. Analogous to recsys (`16`): BrainAPI is the KB; heavy models are plugins.++```text+ POST /ingest* (chunks, graph, vectors, topics)+ │+ ┌─────────────────┼─────────────────┐+ ▼ ▼ ▼+ /retrieve/context /retrieve/search /retrieve/recommend+ memory assembler ranked hits ranked items+ LoCoMo/BEAM benchmarks.search benchmarks.recsys+ ADR-006 SLO search SLO recs SLO+```++**What “search” is not.** It is not “call context and parse `text_context`.” Context exists to feed an answering model. Search exists to return **addressable hits** (chunk / entity / event / structured row) with scores, filters, and snippets.++**What “cross-feature” is.** A structure computed at write time, or a cheap score at read time, that more than one product can consume without inheriting the other’s loops (PPR over event hubs, cross-encoder, LightGCN train, sufficiency retry).++---++## 2. Current state (located evidence)++Anchors are the working tree as of 2026-08-18. Older line numbers in `02` / `07` may have drifted; behaviour below was re-checked in `retrieve.py`, `data.py`, `vectors.py`.++### 2.1 Surfaces that exist++| Surface | Role today | Search-engine analogue |+| --- | --- | --- |+| `POST /retrieve/context` | Memory: spaCy variants → dense node/rel seeds → event-centric facts + PPR + bridges + topics; passages via RRF(dense, keyword) | RAG retriever, not a SERP |+| `GET /retrieve/` | `retrieve_data`: text + preferred entity types | Internal lookup |+| `GET\|POST /retrieve/recommend` | Graph recommend (+ optional LightGCN plugin) | Recs, not lexical search |+| `GET /retrieve/entity/synergies` | Sibling / synergy scores | Related-entities widget |+| MCP `search_semantically` / `traverse_graph` / `search_memory` | Deep / agentic | Escalation, unbounded |++### 2.2 The “hybrid” passage channel is not BM25∪dense++```1680:1722:src/services/api/controllers/retrieve.py+def _retrieve_passages(...):+ ...+ vector_hits = vector_search.search_data(...)+ ...+ search_result = data_adapter.search(text, brain_id) # ILIKE+ fused = reciprocal_rank_fusion([vector_ids, keyword_ids])+```++`data.py` `search` is `WHERE text ILIKE %s` with `'%' + escaped_query + '%'`. No inverted index, no term weights, no `k1`/`b`. RRF (`fact_filter.py`, `k=60`) fuses ANN order with a **substring hit list**.++**Implication (interpretation):** ADR-005’s hybrid design is real as *architecture*; the sparse leg is not the IR method the SOTA brief and HyperMem call BM25. Keyword/navigational search is the first thing this stack cannot honestly sell.++### 2.3 Dense ANN is optional / silently off on default pgvector++`_vector_index_ddl` returns empty string when `dimension > 2000`. Deploy examples set all stores to **3072**. `search_vectors` still `ORDER BY embeddings <=> $1` and sets `hnsw.ef_search` (no-op without an index). Over-fetch + uuid tie-break (`ann_overfetch_k`, `stable_top_k_vectors`) is in place for **determinism**, not speed. Milvus uses `AUTOINDEX` + COSINE **if** that backend is selected.++On LoCoMo-sized brains this is not the latency bottleneck (`06`: passages p50 829 ms of which **682 ms embed**). On BEAM-scale or multi-million-chunk brains it becomes a sequential scan. **Two different products, two different bottlenecks.**++### 2.4 Memory path latency and quality (do not mix with search claims)++| Signal | Status | Use for search work |+| --- | --- | --- |+| Context p50 | ~1.3–1.4 s after dossier deletion; ADR-006 target 1 s | Shared embed batching helps both; do not put CE on this path |+| Passage EvR | ~97% conv-26 | LoCoMo **cannot** show BM25 recall gains in aggregate; need a keyword slice or a non-LoCoMo corpus |+| Graph vs passages-only judge acc | McNemar ns at n=152 | Do not justify search graph-channel by LoCoMo accuracy |+| Graph EvR | C3: 47.3% → 56.0% with reserved bridge slots + ANN seed stability | Graph as a **search channel** (return events/entities as hits) is still a product idea, not a LoCoMo win |+| SOTA compose arm | conv-26 95.4% judge (`13`); full-10 still open | **Do not touch default context ranking** while that claim is live unless the gate in §8 passes |+| Recsys | Isolated brain `demorecsys`; plugins for GNN / attribute prefs | Copy this isolation for search eval |++### 2.5 Ranking on context (no LTR, no CE)++Distance (min over event nodes) → optional PPR **replaces** distance → MMR diversify. LLM fact filter is a no-op without a deep-tier adapter. No LambdaMART, no linear blend of BM25+dense+PPR. Historical context still re-runs the same passage search (~91.5% duplicate; cost, not wall-clock). Passage collection still **serial** over 3–6 queries each calling `embed_text` though `embed_texts` exists.++### 2.6 Recs already show the split we want for search++`16`: core = train-free graph ranker on `/retrieve/recommend`; `plugins/features-rec` writes extra edges; `plugins/recsys-gnn` is optional LightGCN and **does not replace** ingest. Search should copy that: core BM25+dense+filters; plugins for SPLADE / ColBERT / CE.++---++## 3. Literature (located, 2026-08-18)++**Retrieval summary**++- Query: canonical IR + memory papers by arXiv ID, plus title search for hybrid/RRF/SPLADE/ANCE+- Scope: targeted lookup (not exhaustive)+- Databases: arXiv Atom API (`export.arxiv.org/api/query`) parsed with `paper-lookup/scripts/arxiv_atom.py`+- Access date: 2026-08-18+- Limits: first page of title-OR search (`total_results=59`, retrieved 8); ID lists complete (12 + 4). RRF’s original SIGIR 2009 paper is **not on arXiv** (Cormack, Clarke, Buettcher) — cited by name only.++**Warnings:** Absence from this bound is not “never studied.” UniIR (`2311.17136`) is **multimodal** instruction retrieval — weak fit for text-only BrainAPI v1. HyDE (`2212.10496`) uses an LLM at query time — incompatible with ADR-006 on context; plugin/deep only.++### 3.1 What actually supports a BrainAPI search product++| Paper | arXiv | Status for us | Takeaway (not a BrainAPI result) |+| --- | --- | --- | --- |+| BEIR | [2104.08663](https://arxiv.org/abs/2104.08663) | **support-located** (zero-shot IR) | BM25 is a strong OOD baseline; dense often loses zero-shot; hybrid / CE / late-interaction win on average at higher cost. **This is the argument for a real lexical index before a custom bi-encoder.** |+| DPR | [2004.04906](https://arxiv.org/abs/2004.04906) | support-located (mechanism) | Dual-encoder + ANN can beat BM25 **in-domain with labels**. We have hosted embeddings, not DPR training. |+| ANCE | [2007.00808](https://arxiv.org/abs/2007.00808) | support-located (mechanism) | Hard negatives from a live ANN index close the train/test gap. High engineering cost; not a v1 gate. |+| ColBERT / v2 / PLAID | [2004.12832](https://arxiv.org/abs/2004.12832), [2112.01488](https://arxiv.org/abs/2112.01488), [2205.09707](https://arxiv.org/abs/2205.09707) | mixed | Token-level matching approaches CE quality; PLAID claims tens of ms on GPU at 140M passages. **Index size and serving complexity → plugin**, not core. |+| SPLADE / v2 | [2107.05720](https://arxiv.org/abs/2107.05720), [2109.10086](https://arxiv.org/abs/2109.10086) | mixed | Learned sparse on inverted indexes; v2 reports large BEIR gains. Index growth and inference → plugin after BM25 exists. |+| M3-Embedding (BGE-M3) | [2402.03216](https://arxiv.org/abs/2402.03216) | **idea-adjacent** | One model, three modes: dense, sparse, multi-vector, long context (8k). Strong **cross-feature candidate** if we ever change the embedder; not a silent swap (would re-embed every brain). |+| Matryoshka / MRL | [2205.13147](https://arxiv.org/abs/2205.13147) | support-located | Truncatable embeddings; `03` already maps this to making HNSW legal. `text-embedding-3-large` supports dimension cut natively. |+| Fusion analysis | [2210.11934](https://arxiv.org/abs/2210.11934) | **challenge-located vs “just RRF”** | Convex combination of lexical+dense scores can beat RRF; RRF is **parameter-sensitive**. Our hardcoded `k=60` is a choice, not a law. Prefer exposing RRF **and** CC on the search API. |+| HyperMem | [2604.08256](https://arxiv.org/abs/2604.08256) | support-located (memory) | LoCoMo SOTA report uses **topic→episode→fact + BM25∪dense RRF**. We already have topic coarse-to-fine; we lack BM25. **Same hybrid is a memory cross-feature**, opt-in on context after the gate. |+| BEAM | [2510.27246](https://arxiv.org/abs/2510.27246) | support-located (memory scale) | Long dialogues; retrieval cost grows with N. HNSW legality matters here more than on conv-26. |+| Filtered ANN | [2602.11443](https://arxiv.org/abs/2602.11443) | support-located (substrate) | Metadata filters + ANN; Milvus recall more stable than pgvector plans in their study. Search **filters** (type, validity, time) are first-class, not a WHERE after LIMIT. |+| Contrastive embeddings | [2201.10005](https://arxiv.org/abs/2201.10005) | background | Off-the-shelf embeddings can be strong for semantic search. Justifies keeping a general embedder in v1. |+| HyDE | [2212.10496](https://arxiv.org/abs/2212.10496) | **veto on context** | LLM generates a hypothetical doc then embeds. Deep/search-plugin only. |++**Not treated as evidence for BrainAPI quality:** MS MARCO MRR tables in the user’s SOTA brief (analytical interpretation, different task, no reproduction here).++### 3.2 What would confirm / refute transferring these papers++- **Confirm BM25 value:** keyword-slice Recall@10 on a brain where ILIKE misses exact tokens that BM25 posts; latency not worse than ILIKE seqscan on a 100k-chunk fixture.+- **Refute BM25 value:** no Recall@k lift on that slice and on LoCoMo passage EvR (already saturated).+- **Confirm HNSW:** recall@k vs exact `ORDER BY <=>` within a declared drop (e.g. ≤2 points at k=24) **and** p95 vector stage drop on a large fixture.+- **Refute “search will lift LoCoMo judge”:** McNemar ns after BM25-on-context opt-in — expected; do not use that as a search-ship failure.++---++## 4. Independent idea register++Stage: `independent` then clustered. Origin: `AI-assisted` + `literature-inspired` (this session). These are **proposals**, not findings.++| ID | Statement | Origin | Predicted observation | Disconfirm |+| --- | --- | --- | --- | --- |+| S01 | Add `POST /retrieve/search` returning scored hits, not a prompt blob | product | Search clients stop scraping `text_context` | Nobody uses it; context remains the only caller |+| S02 | Replace ILIKE with Postgres BM25/`tsvector` (or `pg_trgm` for short codes) as **core** lexical | BEIR, HyperMem | Keyword Recall@k ↑; long-query ILIKE misses ↓ | No lift on keyword slice |+| S03 | Legalize ANN (`halfvec` HNSW or MRL truncate ≤2000) with loud skip | `03` G, MRL | BEAM/large-brain vector p95 ↓; seed stability held via overfetch | Recall@k vs exact collapses |+| S04 | Batch `embed_texts` once per request; concurrent dense+lexical; single query on context | ADR-006 leftover | Context p50 toward <1 s | Stage timings still show multi-RTT embed |+| S05 | Expose fusion `rrf` and `cc` (convex combination) on search | 2210.11934 | CC wins in-domain if a tiny labelled set exists | RRF ≈ CC on our mix |+| S06 | Cheap linear/GBDT ranker over {dense, lexical, PPR, recency, topic, validity} | ADR-006, LTR brief | Precision@10 on search set ↑; microseconds | Features collinear; no lift |+| S07 | Search channels: `passages` \| `entities` \| `events` \| `structured` | product | Entity-name queries don’t require passage hits | Graph channel noisy; users ignore non-passage |+| S08 | Metadata filters (label, validity, time range) applied **inside** retrieval | 2602.11443 | Filtered search correct; no post-LIMIT drop | Planner ignores filter; recall hole |+| S09 | Opt-in `lexical=bm25` on context after search BM25 is proven | HyperMem | Memory keyword misses ↓ without judge drop | EvR or McNemar regresses |+| S10 | Plugin `features-search` or `search-rerank`: CE on K≤10 | brief, ADR-006 | Search nDCG ↑; context untouched | p95 blows search SLO |+| S11 | Plugin late-interaction (ColBERT/PLAID) or SPLADE | ColBERT family, SPLADE | Hard queries ↑ | Index size / reindex cost dominates per-brain |+| S12 | Optional embedder swap to BGE-M3 for dense+sparse+colbert from one model | 2402.03216 | One write, three retrieval modes | Re-embed cost; LoCoMo seed shift |+| S13 | Recs cold-start: `/retrieve/search` on item text for new SKUs, then graph recs | recs×search | Cold items appear before walks exist | Duplicate ranking; worse HitRate |+| S14 | Do **not** train ANCE/DPR in v1 | negative control | — | If we later get labels and a refresh loop, reopen |+| S15 | HyDE / query LLM rewrite only on deep search | 2212.10496, ADR-006 | Hard paraphrases ↑ at high latency | Context accidentally calls it |+| S16 | Search eval harness on a **separate** brain; ledger `benchmarks.search` | `16` analog | Claims don’t contaminate LoCoMo | People still quote judge % for search |++**Minority / preserved:** S12 (embedder swap) and S11 (ColBERT core) are attractive and **wrong as v1** if they force re-ingest of memory brains. S07 graph-as-hits may look unused until a tenant searches people/events rather than documents — keep the channel even if default k focuses on passages.++**What became less obvious after evidence check:** “Ship hybrid” is already the *shape* of context; the gap is the **sparse implementation** and the **missing hit-list API**, not the absence of RRF. “Need IVF+PQ / 100M HNSW RAM” is the SOTA brief’s catalog assumption; BrainAPI shards **per brain**. “Cross-encoder on K=10 for 200 ms p99” fights ADR-006 on memory and is optional on search.++---++## 5. Clusters (by mechanism, not by slogan)++| Cluster | IDs | Shared mechanism |+| --- | --- | --- |+| C-API | S01, S07, S08 | New contract: hits, channels, filters |+| C-LEX | S02, S09 | Real inverted/lexical index |+| C-ANN | S03, S04 | Vectors actually indexed; embed RTT collapsed |+| C-FUSE | S05, S06 | How scores combine (RRF/CC/LTR) |+| C-PLUG | S10, S11, S12, S15 | Heavy IR off the core path |+| C-X | S09, S13 | Cross-product: memory opt-in BM25; recs cold-start via search |+| C-EVAL | S16 | Isolated measurement |+| C-NOT | S14 | Explicit non-goals for v1 |++Merges: S02 and “SPLADE in core” stay **split** (different index ops and failure modes). S04 is not the same as S03 (latency vs scale).++---++## 6. Criteria (declared before preference)++Direction: higher is better unless noted. Weights are decision aids, not a computed winner.++| Criterion | Weight | Anchors (1–5) | Evidence needed |+| --- | --- | --- | --- |+| Memory non-regression | **Gate** (noncompensatory) | Fail = LoCoMo/BEAM/recsys worse | Paired McNemar, passage EvR, recs HitRate, p50 |+| Search usefulness | 3 | 1 = still ILIKE+blob; 5 = hit API + BM25 + filters | Recall@k / nDCG on search set |+| Latency / SLO fit | 3 | 1 = extra LLM; 5 = index lookup + one embed | `profile_stages` |+| Cross-feature leverage | 2 | 1 = search-only fork; 5 = one index, three readers | Same DDL used by context/search/recs |+| Feasibility / reversibility | 2 | 1 = re-embed all brains; 5 = additive index + flag | Rollback to ILIKE/exact scan |+| Information gain if null | 1 | Null still teaches query mix | Pre-registered slices |++**Vetoes (not averaged):** LLM on `/retrieve/context`; default context ranker change without §8 gate; search eval writing memory brains; dual source of truth.++---++## 7. Adversarial review (short)++| Idea | Wrong if… | Alternative explanation | Hidden dependency | Harm |+| --- | --- | --- | --- | --- |+| S02 BM25 | LoCoMo-like questions are long paraphrases; BM25 adds posting I/O and zero recall | Dense already covers; ILIKE was enough on tiny N | Tokenization mismatch with embedder tokenizer | Over-rank boilerplate terms |+| S03 HNSW | Approx top-k reshuffles seeds → graph EvR jitter (C3 already fought this) | Latency win is still embed RTT | `halfvec` ops / pgvector version | Silent quality flake on memory |+| S01 new API | All “search” users were agents who wanted context anyway | Product confusion; two ways to do one thing | Client SDK + docs | Split brain of integrations |+| S06 LTR | No labels except LoCoMo sessions (biased to memory) | Overfit to session IDs | Click log does not exist | Ranking that looks good offline, worse UX |+| S12 M3 embedder | Different geometry vs current 3072-d OpenAI vectors | “One model three modes” marketing | Full re-embed | SOTA compose arm incomparable |++**Anchoring risk:** the user’s SOTA brief pushes hybrid→CE→LTR on a 200 ms p99 web SLO. We keep the **sequence of primitives** (lexical, ANN, optional CE) but **reject the SLO and the default CE**.++---++## 8. Non-regression protocol (how we refuse to make memory worse)++Copy recsys isolation (`16`, `benchmarks/AGENTS.md`).++### 8.1 What may change without a memory A/B++- New routes under `/retrieve/search*` that do not alter context defaults.+- New tables/indexes that context **does not read** yet (e.g. `tsvector` column unused by `_retrieve_passages`).+- Plugins that only register search rerank.++### 8.2 What requires the gate++- Context reading BM25 instead of ILIKE.+- Enabling HNSW / changing embedding dimensionality or metric.+- Changing RRF `k`, seed `k`, passage fanout, PPR default, fact diversify.+- Shared embed batching that changes variant set or seed order (stabilize with existing uuid/distance quantization).++### 8.3 Gate (all must hold)++1. **Memory champion arm** (document the exact `run_id` at implementation time; today the conv-26 SOTA pointer is `locomo-compose-sota-conv26-v4d` in `REPORTS.json`): paired McNemar **not** significantly worse; point estimate ≥ baseline. Prefer **product** profile for default-path changes; do not require SOTA compose (SC/gap-fill) to move.+2. Passage **full evidence-session recall** held to the digit on that brain (deterministic). Graph EvR: if ANN/seeds change, identical-config agreement ≥95% (`00`).+3. Context p50: not worse than the then-current post-dossier baseline beyond noise (record both numbers). Sub-second remains ADR-006, not a search ship blocker.+4. Recsys: `demorecsys` HitRate@K unchanged unless the PR is a recs PR.+5. Brains: search fixtures use IDs like `searchbench*`. Never wipe `locomoconv26*`, `beam*clean`, `demorecsys`.++**If BM25-on-context is null on LoCoMo:** keep ILIKE as context default; search still ships BM25. That is a successful split, not a failed hybrid.++---++## 9. Proposed search API (idea → later spec)++Vertical slice: one call that a UI or engine can use without parsing memory blobs.++```http+POST /retrieve/search+```++```json+{+ "query": "alice counseling license",+ "brain_id": "default",+ "k": 20,+ "channels": ["passages", "entities"],+ "lexical": "bm25",+ "dense": true,+ "fusion": "rrf",+ "filters": {+ "labels": ["PERSON", "EVENT"],+ "currently_valid": true,+ "happened_after": null+ },+ "rerank": "none",+ "profile_stages": false+}+```++```json+{+ "hits": [+ {+ "id": "chunk_…",+ "channel": "passages",+ "score": 0.031,+ "scores": { "bm25": 12.4, "dense": 0.22, "rrf": 0.031 },+ "snippet": "…",+ "source_session_ids": ["session_3"],+ "entity_uuid": null+ }+ ],+ "insufficient": false,+ "stage_timings": null+}+```++Optional later: `GET /retrieve/search/suggest` (prefix / trgm). Out of v1 if it expands scope.++`rerank`: `none` | `linear` | `plugin:<name>`. Unknown plugin → 400, not a silent fallback that looks like core quality.++---++## 10. Cross-feature map++| Feature | Memory | Search | Recs | When to share |+| --- | --- | --- | --- | --- |+| BM25 / tsvector | Opt-in passage channel (HyperMem-like) | **Default lexical** | Item-text cold start (S13) | Write once; read per product flag |+| HNSW / halfvec | Seed + passage ANN at scale | Same | Node similarity if used | Shared; gate on seed stability |+| Batched embed | Closes ~400 ms gap (`06`) | One RTT per search query | If recs embed queries | Shared helper, no rank change if vectors identical |+| Topic memberships | Coarse-to-fine already on context | Facet / boost | Weak | Already written |+| Validity / `happened_at` | Current-truth vs history (`05`) | Filters | Recency decay (already on recs) | Shared properties |+| PPR / hub bridges | Multi-hop coverage | `events` channel ranking | Not default | **Do not** turn search into a second context assembler |+| Linear LTR features | ADR-006 context ranker | Search precision | Could consume overlap scores | Train **per product** or freeze a tiny global blend with separate weights |+| Cross-encoder | Deep tier only | Search plugin, K≤10 | No | Plugin |+| LightGCN | No | No | Plugin | Unchanged |++**Prediction:** the highest cross-leverage items are **S02+S03+S04** (indexes + embed batch). They improve search immediately and *can* improve memory latency/keyword recall without touching judge prompts.++---++## 11. Architecture decisions for implementers++- **Lexical engine v1:** Postgres FTS (`tsvector` + `ts_rank_cd` or a BM25 extension available in the deployed image). Avoid requiring Elasticsearch as system of record (ADR-007). A plugin may *project* to an external engine later.+- **Do not change `EMBEDDING_*_DIMENSION` as a drive-by** in the search PR. HNSW legality is a dedicated slice with a recall-vs-latency measurement (`00`, `03`).+- **Context keeps ILIKE** until S09 gate. Implementation may write tsvector in the same migration.+- **RRF stays** as default fusion (already in code, HyperMem-like). Expose `fusion=cc` with `alpha` for search; do not retune context `k=60` in the same PR.+- **No ANCE/DPR training, no ColBERT core, no HyDE on context** in this workstream’s v1.++---++## 12. Search evaluation protocol++Mirror `16` so search cannot hide inside LoCoMo. Binding write-up: [`18-search-eval-protocol.md`](18-search-eval-protocol.md). Harness: `benchmarks/search/` + `./search.sh`.++| Item | Rule |+| --- | --- |+| Brain | `searchbench*` only |+| Harness | `benchmarks/search/` + `./search.sh`; HTTP only |+| Ledger | `benchmarks.search` in `REPORTS.json` only |+| Metrics | Recall@{5,10,20}, nDCG@10, MRR; p50/p95 of `/retrieve/search` **excluding** `embed.query`; optional keyword/paraphrase slice |+| Query mix | At least: (a) **keyword / name**, (b) **paraphrase / semantic**. Filter/time deferred until the search API accepts filters. Gold = chunk IDs via `DOCID` markers, not LLM judge |+| Cheap start | Reuse a **held-out** subset of memory questions as *retrieval* labels (gold session → chunks via provenance) **without** running the answerer. Report as `search-from-locomo-qrels`, never as LoCoMo accuracy. Not in harness v1 |+| Forbidden | Ingest into memory eval brains; quoting judge % as search quality |++Until a scored `evaluate` run is on the `benchmarks.search` leaderboard, do not claim “search is better than BM25-in-the-brief.” Toy `search_toy` rows are isolation checks, not BEIR.++---++## 13. Task list++Vertical slices. Each leaves the system working. No task is XL.++### Phase 0 — Contract and isolation++## Task 1: Freeze the three-surface contract++**Description:** ADR-007 accepted or explicitly deferred by the maintainer; this doc linked from `00`. No ranking code.++**Acceptance criteria:**+- [ ] Maintainer records accept / amend / reject on ADR-007+- [ ] `00` workstream row for `17` remains accurate++**Verification:** Human review of ADR-007.++**Dependencies:** None+**Files:** `docs/decisions/007-three-product-surfaces-one-kb.md`, `docs/research/00-scope-and-constraints.md`+**Scope:** S++## Task 2: Search eval skeleton++**Description:** Package `benchmarks/search/` with smoke: ingest N chunks into `searchbenchsmoke`, call a **stub** or interim `POST /retrieve/search` (may 501 until Task 4), ledger key documented in `benchmarks/AGENTS.md`.++**Acceptance criteria:**+- [ ] `./search.sh smoke` fails loudly if it would use a memory brain id+- [ ] `REPORTS.json` schema comment or empty `benchmarks.search` leaderboard allowed++**Verification:** `./search.sh smoke` documented; unit test on brain-id guard.++**Dependencies:** Task 1 (or proceed as Proposed)+**Files:** `benchmarks/search/**`, `benchmarks/AGENTS.md`+**Scope:** M++### Checkpoint A+- [ ] Product split is written; search eval cannot wipe LoCoMo/BEAM brains++---++### Phase 1 — Core search first stage (does not change context defaults)++## Task 3: Lexical index at write time++**Description:** Persist `tsvector` (or equivalent) on `data_text_chunks` (and entity name field if cheap). GIN index. Backfill existing brains lazily or via explicit job. Context **does not** query it yet.++**Acceptance criteria:**+- [ ] New chunks get lexical vectors on ingest+- [ ] ILIKE path still used by `_retrieve_passages`+- [ ] Unit test: token query matches a chunk ILIKE would miss or vice versa, documented++**Verification:** `pytest` on FTS helpers; explain/analyze on a fixture shows index use.++**Dependencies:** None (can parallel Task 2)+**Files:** `src/lib/postgresql/data.py`, ingest write path, tests+**Scope:** M++## Task 4: `POST /retrieve/search` core hybrid++**Description:** New controller: embed once, dense `search_data` + BM25/tsvector, fusion RRF default, return hits. No PPR, no fact assembler, no historical duplicate.++**Acceptance criteria:**+- [ ] OpenAPI/schema: query, k, channels⊇passages, lexical, dense, fusion+- [ ] Does not import dossier/synergy retrievers+- [ ] `profile_stages` works like context+- [ ] Context tests unchanged (no default behavior change)++**Verification:** `pytest` API tests; `./search.sh smoke` green against live API.++**Dependencies:** Task 3+**Files:** `src/services/api/routes/retrieve.py`, new controller module, `requests.py`+**Scope:** M++## Task 5: Filters + entity channel (optional second slice)++**Description:** `currently_valid`, label filter; optional `entities` channel via node ANN + name FTS.++**Acceptance criteria:**+- [ ] Filter is applied so that invalid entities are not in top-k when `currently_valid=true`+- [ ] Passage-only default remains if `channels` omitted++**Verification:** Tests with a superseded edge / deprecated node.++**Dependencies:** Task 4+**Files:** search controller, graph/data queries+**Scope:** M++### Checkpoint B+- [ ] Search API returns BM25∪dense hits+- [ ] `/retrieve/context` byte-level ranking defaults unchanged (diff + LoCoMo smoke optional)++---++### Phase 2 — Shared speed/scale (gated)++## Task 6: Batch embeddings on context **and** search++**Description:** One `embed_texts` per request; concurrent dense+lexical; collapse context passage variant loop toward ADR-006 (single query or batched variants sharing the batch). Preserve seed uuid/distance stabilization.++**Acceptance criteria:**+- [ ] Stage timings: a single `embed.query` (or one batched stage) on a profiled request+- [ ] Passage EvR held on a recorded memory arm **if** context code changed+- [ ] Search p50 excluding network policy: one embed RTT++**Verification:** `profile_stages` JSON; paired EvR if context touched.++**Dependencies:** Task 4 if sharing helper; else can start from context-only+**Files:** `retrieve.py`, embeddings adapter call sites+**Scope:** M++## Task 7: Legalize HNSW++**Description:** Measurement-first: `halfvec` vs truncate-to-2000 vs Milvus-only. Loud log/metric when index DDL skipped. Keep overfetch+uuid order.++**Acceptance criteria:**+- [ ] Default 3072 install either has an ANN index **or** fails closed with a visible error at store init+- [ ] Recall@k vs exact on a ≥10k vector fixture recorded+- [ ] Memory gate §8 if context seeds change++**Verification:** Fixture script + notes in run dir; no silent seqscan.++**Dependencies:** None vs search API; **gate** before flipping production default+**Files:** `src/lib/postgresql/vectors.py`, config, tests+**Scope:** M++### Checkpoint C+- [ ] Context p50 moved toward 1 s **or** numbers escalate per ADR-006+- [ ] ANN skip cannot be silent++---++### Phase 3 — Fusion / cheap LTR / memory opt-in++## Task 8: Search fusion knobs + linear rerank++**Description:** `fusion=rrf|cc` with documented `alpha`; optional `rerank=linear` using dense, lexical, recency, validity — **no LLM**.++**Acceptance criteria:**+- [ ] Default remains RRF k=60-equivalent+- [ ] Linear weights are config/request, not a trained black box in v1+- [ ] Search nDCG measured on `searchbench*` vs RRF-only++**Verification:** `./search.sh evaluate` two arms.++**Dependencies:** Task 2, Task 4+**Files:** search controller, `fact_filter.py` or shared fusion+**Scope:** S–M++## Task 9: Opt-in BM25 on context (cross-feature)++**Description:** `GetContextRequestBody.lexical: ilike | bm25` default **`ilike`**. Document HyperMem-style union as the reason to try `bm25`.++**Acceptance criteria:**+- [ ] Default path = today’s ILIKE RRF+- [ ] `lexical=bm25` uses the Task 3 index+- [ ] Gate §8 on a product LoCoMo arm before considering default flip++**Verification:** McNemar + passage EvR; do not use SOTA compose as the only arm.++**Dependencies:** Task 3, Task 6 recommended+**Files:** `retrieve.py` `_retrieve_passages`, `requests.py`+**Scope:** S++### Checkpoint D+- [ ] Search has fusion/linear knobs+- [ ] Memory default still ILIKE unless gate passed and maintainer flips++---++### Phase 4 — Plugins (after core is real)++## Task 10: Search rerank plugin hook++**Description:** Same pattern as `recsys-gnn`: optional package registers `rerank=plugin:cross-encoder` (or SPLADE document expansion). Core works if plugin absent.++**Acceptance criteria:**+- [x] Plugin yaml + route or entrypoint+- [x] Context path cannot call the plugin+- [x] K≤10 documented for CE++**Verification:** Load/unload plugin; 400 on unknown name.++**Dependencies:** Task 4+**Files:** `plugins/search-rerank/` or `features-search/`, plugin loader+**Scope:** M++## Task 11: Recs cold-start via search (optional)++**Description:** When graph walks return < k items, fill from `/retrieve/search` on item text / attributes. Flag on recommend request, default off.++**Acceptance criteria:**+- [ ] Default HitRate on `demorecsys` unchanged+- [ ] Flag-on arm recorded separately in `benchmarks.recsys`++**Verification:** `./recsys.sh evaluate` both flags.++**Dependencies:** Task 4+**Files:** `recommend.py`, recsys harness+**Scope:** S–M++### Checkpoint E+- [ ] Heavy IR is optional+- [ ] Recs default unchanged++---++## 14. Risks++| Risk | Impact | Mitigation |+| --- | --- | --- |+| HNSW changes memory seeds | High | Overfetch+uuid; EvR agreement gate; exact scan fallback flag |+| BM25 on context looks “free SOTA” | Med | Opt-in; HyperMem is not our judge protocol (`13`) |+| Search API unused; context still scraped | Med | SDKs/docs; chatbot plugin uses search for “find” and context for “answer” |+| FTS tokenization ≠ embed tokenizer | Med | Document analyzer; don’t expect identical hits |+| Scope bleed into ColBERT/ANCE | High | C-NOT / Task 14; plugins only |+| Judge-noise used to “prove” search | High | Separate metrics and brains |++---++## 15. Open questions (maintainer)++Locked 2026-08-18 (see ADR-007 Status):++1. **Routes.** Keep search under `/retrieve` as `GET|POST /retrieve/search`. Do not replace `GET /retrieve/`.+2. **Lexical engine.** Env-gated. When on, Okapi BM25 over Postgres `tsvector` + GIN on `pgvector/pgvector:pg16`. Do not require `pg_search` / `rum`. Do not ship `ILIKE` or `ts_rank_cd` as the search lexical ranker.+3. **3072-d ANN.** Keep float32 vectors. `halfvec` HNSW (≤4000-d) + exact float32 cosine rerank of overfetch. Do not truncate / MRL-cut dimensions.+4. **Chatbot plugin.** Out of scope for v1. No `plugins/chatbot-memory` client.+5. **SLO.** p50 **< 200 ms excluding embed RTT**. `profile_stages` must expose `embed.query` separately from retrieve.+6. **Context vs search ranking.** Search default is both channels fused (`SEARCH_FUSION=rrf`). Context stays today’s dense ∪ ILIKE unless `SEARCH_ENABLED=true` and `CONTEXT_PASSAGE_MODE` is left at `hybrid` (or set to `bm25` / `dense`). `ilike` freezes the memory eval path.++---++## 16. Overall assessment++BrainAPI is already a **knowledge base** with a strong memory assembler and a recs surface that knows how to keep plugins off the default path. It is **not** yet a search engine: there is no hit-list API, no BM25, and default vectors may be unindexed.++The SOTA search brief is useful as a **checklist of primitives** (lexical, dense ANN, optional CE, LTR). It is the wrong **product recipe** if pasted onto `/retrieve/context` (wrong SLO, wrong response shape, wrong eval).++Highest-leverage path that can help search *and* memory without betting LoCoMo: **write a real lexical index and a legal ANN index, batch embeddings, ship `/retrieve/search` that reads them, keep context defaults frozen until a paired gate.** Cross-encoders, SPLADE, ColBERT, HyDE, and ANCE stay plugins or deep-tier — the same split recsys already uses.++**Next action:** maintainer accept/amend ADR-007, then Task 2+3 in parallel (eval skeleton + tsvector), then Task 4 (search API). Label: **protocol development**, not a validated quality claim.++**Status note (2026-08-21):** ADR-007 accepted; `/retrieve/search` shipped. Subsequent workstreams: `18`–`24` (eval, ESCI, WANDS, catalog graph), `25` + [ADR-008](../decisions/008-query-gated-search-personalization.md) (query-gated personalize). §16 above is the 2026-08-18 assessment that motivated the ship. Default omitted `channels=["passages"]` and the 200 ms SLO are unchanged.diff --git a/docs/research/18-search-eval-protocol.md b/docs/research/18-search-eval-protocol.mdnew file mode 100644index 0000000..199401e--- /dev/null+++ b/docs/research/18-search-eval-protocol.md@@ -0,0 +1,189 @@+# Search eval protocol — hybrid BM25 + dense (`/retrieve/search`)++How to ingest a labeled toy corpus into BrainAPI and score ranked hits without contaminating memory eval.++**Role split (locked):**++| Component | Responsibility |+| --- | --- |+| BrainAPI | KB + **core hybrid search**: `POST /ingest/` writes chunks (tsvector + dense); `POST /retrieve/search` returns ranked hits (`fusion=rrf` default, `cc` override). `rerank=none` (default). Unknown `plugin:<name>` is **400**, never a silent fallback |+| `plugins/search-rerank` | Optional second-stage cross-encoder via `rerank=plugin:cross-encoder`. Default `mode=default` reranks at most `RERANK_MAX_K=10`. Opt-in `mode=catalog` retrieves `k_ret=min(200, max(k, 50))` and reranks at most `CATALOG_RERANK_MAX_K=50`, then cuts to request `k`. Health: `GET /search-rerank/health`. Must not run on `/retrieve/context`. Catalog+CE is **not** the ADR-007 200 ms default path |+| `plugins/search-splade` | Learned-sparse first stage. Own index: `POST /search-splade/index`. Retrieve via `channels=["plugin:splade"]` (optionally fused with `passages`) |+| `plugins/search-colbert` | Late-interaction first stage (MaxSim). Own index: `POST /search-colbert/index`. Retrieve via `channels=["plugin:colbert"]` |++**Not** LoCoMo / LongMemEval / BEAM / RecSys. Never write to or wipe `beam*`, `locomoconv*`, `demorecsys`, or LongMemEval brains. Ledger upserts go only to `benchmarks.search` in [`benchmarks/REPORTS.json`](../../benchmarks/REPORTS.json). Never quote LoCoMo judge % as a search score.++Related: [ADR-007](../decisions/007-three-product-surfaces-one-kb.md), [ADR-008](../decisions/008-query-gated-search-personalization.md), [17-search-surface-and-cross-features.md](17-search-surface-and-cross-features.md) §12.++---++## Server prerequisites++The API under test must have:++- `SEARCH_ENABLED=true`+- `DATA_DB=postgresql`+- `BRAIN_CREATION_ALLOWED=true`+- At least one of `SEARCH_USE_DENSE` / `SEARCH_USE_BM25` true (product default: both)++Harness auth: `BRAINPAT_TOKEN` and `BRAINAPI_URL` in `benchmarks/.env`.++A 404 on `/retrieve/search` is **search disabled**, not a ranking miss. The harness must fail loudly (`Set SEARCH_ENABLED=true`).++---++## Pipeline++```text+docs JSONL → POST /ingest/ (skip_enrichment=true → chunk + embed only)+ → optional POST /ingest/structured (`--ingest-graph`; entity uuid = catalog `doc_id`)+ → optional interaction JSONL (`--interactions`; EVENT + `happened_at` on `searchbench*`)+ → GET /retrieve/text-chunks (map DOCID markers → chunk ids)+ → POST /retrieve/search (profile_stages=true, fusion=rrf|cc;+ channels: passages | entities | events | communities;+ expand: none | neighbors;+ mode: default | catalog)+ → Recall@{5,10,20} / nDCG@10 / MRR+ → p50/p95 retrieve excluding embed.query+```++Gold matches **chunk ids** (via DOCID mapping) **or** `hit.id == doc_id` (graph entity uuid). Binary relevance is enough for the toy fixture. Catalog datasets carry graded `gold_grades` and nDCG@10 uses those gains.++Each ingested document embeds a unique marker `DOCID <doc_id>`. After ingest, paginate `GET /retrieve/text-chunks` and map `doc_id → {chunk_id, ...}` by substring. A hit matches if `hit.id` is in that gold set **or** equals the catalog `doc_id`.++Search ingest defaults to `skip_enrichment=true` (chunk + embedding, no Scout/Architect). Pass `--enrich` to restore full KG ingest. Catalog evals should stay on the skip path so OpenAI/DeepSeek extract quota is not in the loop. Embeddings still run. `--ingest-graph` writes generic HAS triples (entity + CLASS/TYPE/ATTR hubs) via product `catalog_graph`. Core search has no PRODUCT/CATEGORY/BRAND field names; those strings may appear only as ordinary node types in harness JSONL.++---++## Query mix++Bundled fixture: [`benchmarks/data/search_toy.jsonl`](../../benchmarks/data/search_toy.jsonl).++| Slice | Intent |+| --- | --- |+| `keyword` | Name / token overlap (BM25 should help) |+| `paraphrase` | Semantic rewrite (dense should help) |++Product-search corpora (download, not committed):++| Dataset | Command | Default slice | Gains |+| --- | --- | --- | --- |+| Amazon ESCI / Shopping Queries (KDD Cup 2022) | `./search.sh download --name esci` | US, Task 1 `small_version`, `test` | E=1, S=0.1, C=0.01, I=0 |+| WANDS (Wayfair) | `./search.sh download --name wands` | `query_class` | Exact=1, Partial=0.5, Irrelevant=0 |++Defaults cap the local JSONL at 80 queries / 2000 products / 40 candidates per query so ingest is feasible. Frozen WANDS quality slice is `data/search_wands.jsonl` (caps 80/2000/40; actual n may be lower because `_select_catalog` drops queries with no remaining gold once `max_docs` fills). `download --name wands` refuses to overwrite that file once it exists. Use `--brain searchbenchesci` / `searchbenchwands`. Do **not** reuse `demorecsys`. Structured recsys JSONL is not a search qrel. WANDS first-stage control is passages-only, k=50, `rerank=none` — see [23-search-wands-quality.md](23-search-wands-quality.md). Graph `--ingest-graph --channels passages,entities,communities` is not the quality control. Opt-in catalog graph (hub intersection + node hybrid) uses a **new** brain `searchbenchwandsgraph` and is an **architecture demo** — see [24-search-catalog-graph.md](24-search-catalog-graph.md). Do **not** mix those numbers with frozen WANDS 0.823 or ESCI 0.500. `--ingest-graph` / `--interactions` on `searchbenchwands`, `searchbenchesci74`, `searchbenchescies`, or `searchbenchesciltr2` is refused.++Filter / time slices are **deferred** until `/retrieve/search` accepts filters. Do not fake them with ILIKE on the harness side.++Graph channels (core, use-case agnostic): `entities` (node ANN ∪ name CONTAINS), `events` (EVENT nodes; empty-ok on catalog-only brains; recency from `happened_at` when present), `communities` (typed hub nodes, default labels `TYPE,CLASS,TOPIC` via `SEARCH_COMMUNITY_LABELS`), `expand=neighbors` (depth-1, fanout cap + degree IDF). Default omitted `channels` remains `passages`. Hits are a mixed list of chunk ids and node uuids until a join table exists. Communities are **not** Leiden / `kg_topic_sessions`. Do not score `/retrieve/recommend` HitRate as search.++Optional later arm: reuse held-out memory questions as *retrieval* labels (`search-from-locomo-qrels`) **without** running the answerer, and never as LoCoMo accuracy. Not in v1 of this harness.++---++## Metrics and SLO++| Metric | Definition |+| --- | --- |+| Recall@k | \|retrieved ∩ gold\| / \|gold\| at k ∈ {5,10,20} |+| nDCG@10 | Binary on toy gold; graded `gold_grades` on ESCI/WANDS. Hits collapse to `doc_id` (chunk map **or** node uuid) |+| MRR | Reciprocal rank of the first gold hit |+| `p50_retrieve_ms` / `p95_retrieve_ms` | Prefer `stage_timings` `search.retrieve.wall_ms`; else `client_wall − embed.query.wall_ms` |++ADR-007 SLO: **p50 < 200 ms excluding embed RTT**. Record `embed.query` separately so the ledger cannot mix the two. That SLO is the **default** path (`mode=default`, `rerank=none`). `mode=catalog` plus `rerank=plugin:cross-encoder` is an opt-in two-stage; do not cite its client wall as the 200 ms product claim.++Headline for the ledger: `ndcg@10`, `recall@10`, `p50_retrieve_ms`. Label `dataset` (`search_toy.jsonl` vs `search_esci.jsonl` / `search_wands.jsonl`) so a toy run is not mistaken for product search.++A passing **smoke** is a health check, not a published quality claim. Only `evaluate` with `status: ok` upserts `benchmarks.search`.++---++## Harness++HTTP only (`benchmarks/search/` must not import `src/`, except `search/mapping.py` which re-exports `src.core.search.catalog_graph` after putting the repo root on `sys.path`). Default brain: `searchbenchsmoke`. Any `--brain` **must** start with `searchbench`.++```bash+cd benchmarks+./search.sh dataset-stats+./search.sh smoke+./search.sh evaluate --fusion rrf+./search.sh evaluate --fusion cc --run search-toy-cc+./search.sh download --name esci+./search.sh download --name esci --locale es+./search.sh download --name wands+./search.sh --brain searchbenchitsmoke evaluate --dataset data/search_italian_smoke.jsonl --run search-italian-smoke+./search.sh evaluate --dataset data/search_esci.jsonl --brain searchbenchesci --run search-esci+./search.sh --brain searchbenchwands evaluate --dataset data/search_wands.jsonl --run search-wands-passages-k50 --channels passages --k 50 --ks 5,10,20,50+./search.sh --brain searchbenchwands evaluate --dataset data/search_wands.jsonl --run search-wands-passages-k50 --channels passages --k 50 --ks 5,10,20,50 --skip-ingest+./search.sh --brain searchbenchwandsgraph evaluate --dataset data/search_wands.jsonl --run search-wandsgraph-communities-k50 --channels communities --k 50 --ks 5,10,20,50 --ingest-graph+./search.sh evaluate --rerank plugin:cross-encoder --run search-toy-ce+./search.sh evaluate --mode catalog --rerank plugin:cross-encoder --k 50 --brain searchbenchesci74 --skip-ingest --dataset data/search_esci_74.jsonl --run search-esci-74-catalog-ce-k50+./search.sh rank-corpus --dataset data/search_esci_74.jsonl --run search-esci-74-exhaustive-ce+./search.sh list-overlap --passages-run search-esci-74-passages-k50 --against-runs search-esci-74-bge-base-k50,search-esci-74-colbert-k50+./search.sh union-lists --from-runs search-esci-74-passages-k50,search-esci-74-bge-base-k50,search-esci-74-colbert-k50 --run search-esci-74-union-bge-k50+./search.sh cascade-lists --passages-run search-esci-74-passages-k50 --from-runs search-esci-74-bge-base-k50,search-esci-74-colbert-k50 --run search-esci-74-cascade-tail-k50+./search.sh finetune-4class --dataset data/search_esci_74.jsonl --base microsoft/deberta-v3-base --epochs 2 --max-pairs 80000 --batch-size 8 --out data/models/esci-deberta-v3-base-4class+SEARCH_RERANK_MODEL=data/models/esci-deberta-v3-base-4class ./search.sh rerank-retrieved --from-run search-esci-74-passages-k50 --dataset data/search_esci_74.jsonl --run search-esci-74-passages-k50-ce-deberta --ks 5,10,20,50+./search.sh evaluate --channels passages,plugin:splade --run search-toy-splade+./search.sh smoke --interactions data/recsys_toy.jsonl --channels events --brain searchbenchevents+./search.sh report --run <run_id>+```++Requires `BRAINPAT_TOKEN` in `benchmarks/.env`.++**Live stack note:** TUI / `brainapi start` may run from `~/.brainapi/source`. Sync or restart from the intended checkout before measuring, or scores will reflect stale code (`SEARCH_ENABLED` must be on in **that** process).++---++## Guardrails++- Score only via `POST /retrieve/search`. Do **not** call `/retrieve/context` as the metric path.+- Do **not** call `/retrieve/recommend`. Optional search `target` is query-gated rerank of retrieved hits (ADR-008), not this recommend protocol.+- Do **not** mutate `benchmarks.locomo` / `beam` / `longmemeval` / `recsys` ledger rows.+- Do **not** point `--brain` at memory-eval or recsys brains (`searchbench*` only; never `demorecsys`).+- Failed / empty runs stay off the leaderboard.+- Cross-encoder, SPLADE, and ColBERT are plugins. Core hybrid works if they are absent. `--rerank plugin:<missing>` and `--channels plugin:<missing>` must 400, never look like a ranking miss.++---++## Multilingual catalog paths (2026-08-19)++**Located evidence:** `download --locale` already filtered ESCI parquet (`us` / `es` / `jp`) but defaulted to the same `search_esci.jsonl` as US. **Decision:** US stays `search_esci.jsonl`; ES/JP write `search_esci_{locale}.jsonl`; `--locale it` errors. ESCI has no Italian split (Reddy HTML; arXiv Italian product-search query empty). Fixture `data/search_italian_smoke.jsonl` is a pipeline smoke (`slice=italian-smoke`), not ESCI Task 1 and not an Italian quality claim. FTS analyzer stays `to_tsvector('english'`. Do not re-ingest `searchbenchesci74`. Plan: [22-multilingual-ecommerce-search.md](22-multilingual-ecommerce-search.md).++**ES first-stage (2026-08-19, located evidence):** `./search.sh download --name esci --locale es` wrote `data/search_esci_es.jsonl` (n=62 queries, 2000 docs, `slice=esci-es`) without changing `search_esci.jsonl` / `search_esci_74.jsonl`. Live `./search.sh --brain searchbenchescies evaluate --dataset data/search_esci_es.jsonl --run search-esci-es-passages-k50 --channels passages --k 50 --ks 5,10,20,50` → nDCG@10 **0.577**, Recall@10 **0.353**, Recall@50 **0.914**, p50 84 ms. Protocol: first-stage shared-index, passages, `rerank=none`. Do not mix with US n=74 0.500/0.379 or Reddy ES 0.849 (ranking-in-pool). Product default unchanged.++Italian smoke is pipeline-only (`./search.sh --brain searchbenchitsmoke evaluate --dataset data/search_italian_smoke.jsonl`). Inflected BM25 miss on english FTS; optional `SEARCH_FTS_BRAINS=searchbenchitsmoke` + `SEARCH_FTS_REGCONFIG=italian` adds `search_tsv_alt` on that brain only. `_SEARCH_DDL` stays english.++**Phase 2 (2026-08-19).** Italian product-search qrels: still **no direct evidence located** (arXiv 0; OpenAlex incidental hits are not qrels). ES spelling harness `search-esci-es-spell-k50` skip-ingest: nDCG@10 0.595 (win bar 0.597) / Recall@10 0.362 / Recall@50 0.914 — **null**, not live. Optional `SearchRequestBody.extras` equality filter + `SearchResponse.facets` on hit extras; live `searchbenchitsmoke` `locale=it` keeps 3 docs, `color=nope` empty. MiniLM local-dense `search-italian-minilm-pipeline` on labeled `searchbenchitmmini` retrieved all 3 golds at k=10 (pipeline only; not product embedder). Do not mix with Reddy 0.849 / US 0.500 / ES n=62 first-stage. Plan: [22-multilingual-ecommerce-search.md](22-multilingual-ecommerce-search.md).++**Production Italian + US n=74 arms (2026-08-19).** Isolation held: `searchbench*` only; `_SEARCH_DDL` stays `to_tsvector('english'`; locomo/beam/`demorecsys` never get `search_tsv_alt`. **Decision:** production Italian is a dedicated `searchbench*` brain + `SEARCH_FTS_BRAINS` + `SEARCH_FTS_REGCONFIG=italian` + optional `extras={"locale":"it"}`. Alt BM25 matches **any** query lexeme (OR); english `search_tsv` keeps `plainto_tsquery` AND. Inflect skip-ingest `search-italian-smoke-inflect-fts-or`: BM25 nonempty for `bollitori` / `divani` / `caffettiere`. Do not quote nDCG as Italian quality.++US n=74 skip-ingest, passages, k=50, `rerank=none` (control nDCG@10 **0.500** / Recall@10 **0.379** / Recall@50 **0.834** / p50 ~60 ms):++- Frozen-head cascade is in `hybrid.frozen_head_merge` and is wired only when extra plugin (or `SEARCH_LITERAL_FILL`) lists are present. Default omitted `channels=["passages"]` stays hybrid RRF. Replay of stored passages+BGE+ColBERT lists still Recall@50 **0.889** / nDCG@10 **0.500**. Live n=74 sidecar not run: ColBERT/SPLADE plugin indexes are empty for `searchbenchesci74`. Honest default Recall@50 stays **0.834**.+- `fusion=cc` alpha `{0.3,0.5,0.7}`: best nDCG@10 **0.493** (alpha 0.7) vs win **0.520**. **Null.** Leave RRF.+- Title-token literal residual (`SEARCH_LITERAL_FILL`, frozen-head): Recall@10 **0.379** held; Recall@50 **0.655** (below 0.834). **Null.** Flag stays false.+- Head LTR (`./search.sh ltr-head`, run `search-esci-74-ltr-head-k50`): query-grouped 5-fold CV reorders stored `search-esci-74-passages-k50` `hit_ids` (same 50; Recall@50 **0.834** held by construction). nDCG@10 **0.515**, Recall@10 **0.384** vs win **0.520** / **0.397**. Overlap-only diagnostic nDCG@10 **0.476** / Recall@10 **0.339**. **Null.** Not live. Do not train-on-all-74 as the quality number.+- Head LTR + 4-class `ce_gain` feature (`search-esci-74-ltr-cefeat-k50`): same CV, pair policy `other_query_neg`. nDCG@10 **0.524** (gate ≥ 0.520), Recall@10 **0.383** (hold ≥ 0.379), Recall@50 **0.834**. **Harness win on predeclared nDCG/hold.** Not live. Human review before `search.py`. Do not mix with Reddy 0.857 or CE-as-sole-ranker (those hurt Recall@10).+- Head LTR + DeBERTa-v3-base `ce_gain` (`search-esci-74-ltr-deberta-k50`): same lists/CV/`other_query_neg`; checkpoint `data/models/esci-deberta-v3-base-4class`. nDCG@10 **0.542** (must-beat this round: > **0.524**), Recall@10 **0.387** (≥ 0.379), Recall@50 **0.834**. **Harness win vs MiniLM blend.** Horizon nDCG@10 ≥ **0.70** missed (oracle on these 50 is 0.876). Gated LightGBM lambdarank (`search-esci-74-ltr-deberta-lgbm-k50`): nDCG@10 **0.516**, Recall@10 **0.379** (unrounded 0.3788, below hold), `ce_gain` gain-importance largest. **Null vs RankNet DeBERTa.** Not live. Do not mix with Reddy 0.857, pool nDCG@20 0.710, or DeBERTa-as-sole-sorter (0.510 / 0.363).+- I04 train-on-matched-hybrid (`search-esci-74-ltr-deberta-train200`): RankNet+DeBERTa fit on 170 US train queries' hybrid k=50 from new brain `searchbenchesciltr2` (JSONL cap 200q/4k docs, doc budget bound to **170** queries / **4000** docs; 0 qid overlap with the 74). Applied to frozen `search-esci-74-passages-k50`. nDCG@10 **0.544** (> **0.542**), Recall@10 **0.391** (≥ 0.379), Recall@50 **0.834**. **Harness win vs I01 CV.** Horizon 0.70 missed. Train-set first-stage diagnostic (different catalog): nDCG@10 **0.387** / Recall@50 **0.744**. Not live. Do not wipe `searchbenchesci74`.+- I-LGBM-APPLY (`search-esci-74-ltr-deberta-lgbm-train200`): LightGBM lambdarank on the same 170 hybrid lists / DeBERTa `ce_gain` / `other_query_neg`, applied to frozen 74. Frozen hypers (`n_estimators=100`, `max_depth=3`, `lr=0.05`). nDCG@10 **0.533**, Recall@10 **0.389**, Recall@50 **0.834**. **Null vs RankNet apply 0.544.** Horizon 0.70 missed. Stop-for-human bar 0.58 not reached. Not live.+- I-CE-HYB (`search-esci-74-ltr-deberta-hybrid170`): continue pool DeBERTa 4-class on 8500 hybrid-k50 rows from the 170 lists (`data/models/esci-deberta-v3-base-4class-hybrid170`; did not clobber the pool checkpoint). RankNet apply with new `ce_gain`. nDCG@10 **0.530**, Recall@10 **0.385**, Recall@50 **0.834**. **Null vs C6 0.544.** Horizon 0.70 missed. Not live.++Product default that survived: hybrid BM25+dense, passages, `rerank=none`, extras/facets as shipped, Italian via gated FTS+OR. Harness nDCG@10 **0.544** is not a live default. Do not mix with Reddy 0.857 or CE-on-pool 0.710.++## WANDS first-stage control (2026-08-20)++**Decision:** frozen `data/search_wands.jsonl` (caps 80q/2000d/40 cand; **actual 66 queries / 2000 docs**), brain `searchbenchwands`, passages, k=50, `rerank=none`, skip_enrichment, no graph. Run `search-wands-passages-k50`. Gains Exact=1 / Partial=0.5 / Irrelevant=0. Recall is binary Exact+Partial. Linear DCG. After first eval, `--skip-ingest` only. `download --name wands` refuses overwrite.++**Located evidence:** nDCG@10 **0.823**, Recall@10 **0.269**, Recall@50 **0.837**, MRR **0.925**, p50 **87 ms**. Gold median 40; Recall@10 ceiling **0.365**. Do **not** mix with ESCI 0.500 / C6 0.544, Reddy 0.857, or Chen et al. (DOI located; nDCG **not located**). Details: [23-search-wands-quality.md](23-search-wands-quality.md).++## Opt-in catalog graph (2026-08-20)++**Decision:** architecture-opt-in on new brain `searchbenchwandsgraph` only. Product mapper is [`src/core/search/catalog_graph.py`](../../src/core/search/catalog_graph.py) (harness re-exports it). Communities match hubs then intersect item neighbors when ≥2 hub kinds match, else union; hybrid-rank product nodes inside `graph_channels`. Live omitted `channels` stays `["passages"]`. Node BM25 is `SEARCH_ENABLED` english `kg_nodes.search_tsv`. Do not edit `search.py` fusion this round.++**Isolation:** reuse frozen `data/search_wands.jsonl`; never wipe `searchbenchwands` / `searchbenchesci74`. Ledger runs `search-wandsgraph-communities-k50` / `search-wandsgraph-passages-k50` are labeled **architecture-demo**. Do not quote them as beating 0.823 / 0.500. Details: [24-search-catalog-graph.md](24-search-catalog-graph.md).++## Query-gated personalize (2026-08-21)++**Decision:** optional `SearchRequestBody.target` reranks retrieved hits after extras AND. Omit `target` → anonymous ranking (this protocol’s default). Do not call `/retrieve/recommend`. Do not score HitRate as nDCG. If ledgered: `architecture-demo` only. Details: [25-personalized-search.md](25-personalized-search.md), [ADR-008](../decisions/008-query-gated-search-personalization.md).diff --git a/docs/research/19-search-esci-quality.md b/docs/research/19-search-esci-quality.mdnew file mode 100644index 0000000..e28e28e--- /dev/null+++ b/docs/research/19-search-esci-quality.md@@ -0,0 +1,1166 @@+# 19 — ESCI slice search quality: retrieval, fusion, and eval++Workstream: `/retrieve/search` ranking quality on `searchbench*` only. Ledger: `benchmarks.search` only. This document is a **proposal and experiment plan**, not a published finding.++**Focal question:** Which interventions could raise nDCG@10, Recall@10, and/or MRR on BrainAPI search for the ESCI product-search slice, and which of those are actually testable given the current eval?++Session purpose: decide the next search-quality experiments. Audience: product owner. Decision owner: the user. Horizon: days.++Every claim is labeled **idea**, **assumption**, **prediction**, **located evidence**, or **decision**.++---++## What this workstream does++Search quality here is labeled product ranking, not LoCoMo judge accuracy and not `/retrieve/recommend`.++### Write path (this slice)++1. Catalog JSONL docs are ingested as chunks with a `DOCID <doc_id>` marker (`benchmarks/search/catalog.py:234-256`, `dataset.py:9-12`). Default ingest is `skip_enrichment=true` (chunk + embed only; `evaluate.py:45-46`, `cli.py:46-47`).+2. Optional `--ingest-graph` writes deterministic HAS triples (`evaluate.py:315-320`). Entity uuid equals catalog `doc_id` (`mapping.py:39-40`). Hubs are `hub:{kind}:{value}` (`mapping.py:43-44`). Class/brand/color/locale/feature values become CLASS / ATTR / TYPE nodes (`mapping.py:180-251`). If no fields exist, a TYPE hub from `dataset` is used (`mapping.py:241-250`).+3. **Located evidence:** this slice’s 200 docs have brand/color/locale in the JSONL and **0/200 `class` fields**. CLASS hubs from mapping are therefore mostly absent; ATTR/TYPE hubs still exist.++### Read path++1. Default omitted `channels` is `["passages"]` (`src/services/api/constants/requests.py:478`; `hooks.py:81-89` treats core names as non-plugins).+2. Passages: dense ANN + BM25, fused with RRF or convex combination (`search.py:176-177, 257-265`; `hybrid.py:109-135`; RRF at `fact_filter.py:67-78`).+3. Graph channels (`graph_channels.py:18-28`):+ - `entities`: name CONTAINS via `graph.search_entities` plus optional node ANN (`graph_channels.py:280-392`). No default `node_labels` filter unless the request sets one (`search.py:111`).+ - `events`: same as entities with `EVENT` labels plus recency (`graph_channels.py:395-433`). Empty-ok on catalog-only brains.+ - `communities`: search TYPE/CLASS/TOPIC hubs (default `SEARCH_COMMUNITY_LABELS`, `config.py:701-709`; `graph_channels.py:30, 564-615`), then expand neighbors with degree-IDF. Member hits can be ATTR hubs because skip labels are only the community labels (`graph_channels.py:588-614`).+4. Each populated graph channel is appended as its own `extra_id_lists` entry and RRF-fused with passage id lists (`search.py:238-265`). IDs are mixed chunk ids and node uuids. A unit test locks this mixing in (`tests/test_search_graph.py:409-416`).+5. Optional `rerank=plugin:<name>` reranks at most `RERANK_MAX_K = 10` on `mode=default` (`hooks.py`; `search.py`). Opt-in `mode=catalog` retrieves `min(200, max(k, 50))` and reranks at most `CATALOG_RERANK_MAX_K = 50`, then cuts to request `k`. `rerank=none` on both ESCI arms unless stated.++### Eval path++1. After ingest (or `--skip-ingest` reuse), chunks are listed and mapped by DOCID substring (`evaluate.py:332-335`, `dataset.py:61-76`).+2. Gold is doc_id space: `gold_hit_ids` unions `gold_doc_ids` / positive `gold_grades` and maps `gold_chunk_ids` through `chunk_to_doc` (`evaluate.py:201-222`).+3. Hits are canonicalized chunk to doc and deduped; unknown ids (including `hub:*`) stay as themselves (`evaluate.py:149-161, 361-362`).+4. Metrics: Recall@{5,10,20}, graded nDCG@10, MRR (`metrics.py:7-67`). nDCG uses `gold_grades` when present (`metrics.py:42-56`).+5. Ledger upserts `benchmarks.search` and now records `channels` (`report.py:61, 157`).++### Located evidence — two arms on the same brain++Same dataset `benchmarks/data/search_esci_slice.jsonl` (200 docs, 11 queries, all product columns), brain `searchbenchesci20`, fusion `rrf`, rerank `none`, skip_enrichment true. Graph arm ingested triples; passages arm used `--skip-ingest` (chunks reused; `eval.json` `ingest.reused=true`).++| Arm | Channels | nDCG@10 | Recall@10 | Recall@20 | MRR | p50 retrieve |+| --- | --- | --- | --- | --- | --- | --- |+| Graph | passages+entities+communities | 0.6815 | 0.4657 | 0.5818 | 0.8485 | 108 ms |+| Passages | passages | 0.7579 | 0.5106 | 0.8470 | 0.8485 | 29 ms |++Sources: `benchmarks/runs/search-esci-slice-allcols/{report,eval}.json`, `benchmarks/runs/search-esci-slice-passages/{report,eval}.json`.++Do **not** claim graph-channel lift. Graph dropped nDCG@10 (−0.076) and Recall@10 (−0.045). MRR was identical on every query. Recall@20 dropped far more (−0.265).++Per-query first-relevant rank never moved. Five graph queries put `hub:attr:*` in the fused top-10. Graph unique hit counts after canonicalize were 12–18 vs 20 for passages. Offline drop of `hub:*` from the already fused graph lists only moved nDCG 0.682 to 0.691 and Recall@10 0.466 to 0.476 (Recall@20 unchanged). **Prediction that failed:** removing visible hubs from the scored list recovers passages. Residual gap is ASIN substitution and a shorter unique product pool created at fusion time.++One query (esci-18) had graph nDCG +0.057 with unchanged Recall@10 because rank-10 swapped a Substitute (gain 0.1) for an Exact (gain 1.0). That is a single-query graded accident, not a mechanism claim.++Both arms score 0 on esci-72 (`$100 things that are not electronics`; 6 Exact golds never retrieved).++Gold sizes: min 1 / median 20 / max 20. Gain histogram on the slice: E=1.0 ×127, S=0.1 ×37, C=0.01 ×2. Recall@10 is therefore capped near 0.5 on the typical query even if top-10 is all gold.++---++## Maintainer decision (2026-08-18)++**Decision (located):** make graph channels useful even if passages stays the default. Headline nDCG vs passages-only is **not** the go/no-go. Isolated-channel gold retrieval is: Recall@10 of gold ASINs and presence of product uuids in hits. Follow-up evals reuse brain `searchbenchesci20` with `--skip-ingest`. Default omitted `channels` remains `["passages"]`.++---++## Isolated channels then ranking change (same brain, n=11)++All arms: `search_esci_slice.jsonl`, `--skip-ingest`, fusion `rrf`, rerank `none`. No re-ingest. `eval.json` now stores per-hit `id`, `channel`, and canonical `doc_id` (`evaluate.py`).++### Before ranking change (measurement)++| Arm | Channels | nDCG@10 | Recall@10 | Recall@20 | MRR | p50 ms | Queries with gold in top-10 | `hub:*` in top-10 |+| --- | --- | --- | --- | --- | --- | --- | --- | --- |+| Passages | passages | 0.758 | 0.511 | 0.847 | 0.848 | 29 | 10/11 | 0 |+| Entities | entities | 0.532 | 0.415 | 0.709 | 0.667 | 53 | 10/11 | 6 queries (13 slots) |+| Communities | communities | 0.000 | 0.000 | 0.000 | 0.000 | 52 | 0/11 | empty lists |+| Passages+entities | passages,entities | 0.682 | 0.466 | 0.582 | 0.848 | 74 | 10/11 | 5 queries |++Sources: `runs/search-esci-slice-{passages,entities,communities,pe}/eval.json`.++**Call (located evidence):** entities-only already retrieved gold ASINs (Recall@10 0.415, 10/11 queries). ATTR hubs stole rank on 6 queries. Communities-only never fired (TYPE/CLASS/TOPIC hubs do not match these queries; this slice has 0/200 `class` fields). Passages+entities matched the original fused graph arm — communities added nothing. Peer RRF of entity lists with passages still shortened the product pool (Recall@20 0.582 vs 0.847).++Offline compacting of `hub:*` from the original fused `hit_ids` (promote later ranks into @10) yields nDCG@10 0.764 / Recall@10 0.476 / Recall@20 0.582 / MRR 0.848 (`python -m search.replay_fusion`). An earlier 0.691 nDCG figure did not refill @10 from ranks 11+. Compact nDCG matching passages is **not** a claim that live fusion is fixed; Recall@20 stays 0.582.++### After ranking change (same frozen corpus)++Code: `graph_channels.py` defaults the entities channel to `ENTITY` nodes, ranks by name overlap, drops `hub:*` / ATTR/TYPE/CLASS members from entity lists. Communities also seed ATTR hubs (brand/color/feature), expand at most 8 hubs, and emit only item ENTITY neighbors. `search.py` will not RRF hub ids into fused lists. Default omitted channels still `["passages"]`.++| Arm | Channels | nDCG@10 | Recall@10 | Recall@20 | MRR | p50 ms | Queries with gold in top-10 | `hub:*` in top-10 |+| --- | --- | --- | --- | --- | --- | --- | --- | --- |+| Passages smoke | passages | 0.758 | 0.511 | 0.847 | 0.848 | 28 | 10/11 | 0 |+| Entities | entities | 0.655 | 0.482 | 0.822 | 0.757 | 51 | 10/11 | 0 |+| Communities | communities | 0.478 | 0.350 | 0.391 | 0.848 | 129 | 10/11 | 0 |+| Fused | passages,entities,communities | 0.653 | 0.497 | 0.653 | 0.816 | 184 | — | 0 |++Sources: `runs/search-esci-slice-{passages-smoke,entities-after,communities-after,pec-after}/eval.json`.++**Useful, in numbers (go/no-go):** isolated entities Recall@10 0.415 → 0.482; zero hub ids in top-10; product uuids on all 11 queries; Recall@20 0.709 → 0.822. Isolated communities Recall@10 0 → 0.350 with gold in top-10 on 10/11 queries (esci-19 still 0). Passages-only smoke is unchanged at 0.758 / 0.511 / 0.848.++**Not a quality win for fused search.** p+e+c nDCG@10 0.653 is still below passages 0.758 (and below the old fused 0.682). MRR fell from 0.848 to 0.816. Do not put entities/communities in the default mix. Do not claim ESCI-wide lift. n=11.++Communities retrieve p50 129 ms is under the ~200 ms SLO; an uncapped hub expansion on this brain was 239 ms p50 and was capped to 8 hubs.++### Still not useful / unmeasured++- **events:** no interaction EVENT ingest on this catalog; still empty-ok.+- **expand=neighbors:** unit-tested; no isolated ESCI arm.+- **esci-72:** entities-only still 0 gold in top-10; passages-only still 0; communities-after had 1/6 gold in top-10 — n=1, not a negation fix.+- **Fused graph+passages:** still crowds Recall@20 vs passages (0.653 vs 0.847).++Reproduce (no re-ingest):++```bash+cd benchmarks+./search.sh --brain searchbenchesci20 evaluate \+ --dataset data/search_esci_slice.jsonl --skip-ingest --fusion rrf \+ --channels entities --run search-esci-slice-entities-after+./search.sh --brain searchbenchesci20 evaluate \+ --dataset data/search_esci_slice.jsonl --skip-ingest --fusion rrf \+ --channels communities --run search-esci-slice-communities-after+./search.sh --brain searchbenchesci20 evaluate \+ --dataset data/search_esci_slice.jsonl --skip-ingest --fusion rrf \+ --channels passages --run search-esci-slice-passages-smoke+```++---++## Passages CE, esci-72, ranking-in-pool (2026-08-18)++**Decision:** default `rerank=none` stays. Isolated graph channels stay off the default mix.++### Passages MiniLM (Task 1)++`runs/search-esci-slice-passages-ce` — `--channels passages --rerank plugin:cross-encoder`, `--skip-ingest`, n=11.++| Arm | nDCG@10 | Recall@10 | Recall@20 | MRR | p50 retrieve |+| --- | --- | --- | --- | --- | --- |+| Passages (control) | 0.758 | 0.511 | 0.847 | 0.848 | 28–29 ms |+| Passages + MiniLM k≤10 | **0.738** | 0.511 | 0.847 | 0.803 | 30 ms |++**Located evidence:** nDCG@10 −0.020 vs control (win rule was +0.02). Recall@20 unchanged. MRR down. p50 retrieve 30 ms passes the ~200 ms labeled gate. p95 client wall ~7.5 s is first-query model load, not retrieve. **Null result.** Do not change the product default.++### esci-72 dump (Tasks 2–3)++Query: `$100 things that are not electronics`. All 6 Exact golds are **in the JSONL**. None appear in passages top-20. Helper: `benchmarks/search/miss.py`.++| ASIN | Title (short) | Marker |+| --- | --- | --- |+| B00TREI0JI | Apple iPad Air 2 (Renewed) | electronics |+| B073R68TSH | Beam Electronics car phone holder | electronics |+| B07NSX2ZBS | FindKey RF key/phone tracker | electronics |+| B07QYLTT7X | Galaxy Watch Active TPU case | electronics |+| B07XG6Y847 | AirPods silicone case | electronics |+| B083S3ZXDF | Potaroma electric flopping fish toy | electronics (electric toy) |++Hits are mostly price tags, gift cards, Pokémon lots, batteries — lexical `$100` / “tags”, not a negation failure. **Decision: qrel mismatch.** Do not rewrite `not electronics` on this slice. No eval-only intervention.++### Ranking-in-pool (Tasks 4–7)++Catalog query rows now store `candidate_doc_ids` / `candidate_grades` (I=0 kept off `gold_grades`). Slice JSONL grew 200 → 365 docs so CE-on-pool has texts for I-labels. Those extra docs were **not** ingested into `searchbenchesci20`.++| Arm | Protocol | nDCG@10 | nDCG@20 | nDCG full | R@10 | MRR | notes |+| --- | --- | --- | --- | --- | --- | --- | --- |+| BrainAPI in-pool | search then keep pool ids | 0.788 | **0.811** | 0.814 | 0.535 | 0.909 | coverage 0.47; ~15.5 pool ids/query missing from brain |+| MiniLM CE-on-pool | harness CE on all labeled pairs | 0.598 | **0.669** | 0.786 | 0.399 | 0.833 | `cross-encoder/ms-marco-MiniLM-L-6-v2`; missing_text=0 |++**Cite Reddy EN CE 0.857 / KDD 0.9043 only against CE-on-pool.** That arm is n=11, zero-shot MS MARCO, not ESCI-finetuned, not the 4,477-query public test. Do not average these rows with slice first-stage 0.758.++Reproduce:++```bash+cd benchmarks+./search.sh --brain searchbenchesci20 evaluate \+ --dataset data/search_esci_slice.jsonl --skip-ingest \+ --channels passages --rerank plugin:cross-encoder \+ --run search-esci-slice-passages-ce+./search.sh --brain searchbenchesci20 evaluate \+ --dataset data/search_esci_slice.jsonl --skip-ingest \+ --channels passages --rank-pool \+ --run search-esci-slice-passages-pool+./search.sh --brain searchbenchesci20 rank-pool-ce \+ --dataset data/search_esci_slice.jsonl \+ --run search-esci-slice-ce-pool+```++`rank-pool-ce` uses the repo `.venv` (sentence-transformers). 74-query ingest is a later human checkpoint on a **new** `searchbench*` brain.++---++## Guarantees and where they break++**Stated guarantee (this workstream):** hybrid search returns a ranked list of catalog items such that graded nDCG@10, Recall@10, and MRR measure product relevance on ESCI-style qrels, with p50 retrieve (ex-embed) under ~200 ms (`18-search-eval-protocol.md` SLO).++Where that guarantee cannot be delivered today:++1. **Fusion pollution (high impact on nDCG and recall).** RRF treats graph node lists as peer ranked lists (`search.py:246-265`). Incompatible ids occupy slots; more importantly, extra lists reorder and crowd out passage ASINs. **Gap**, not a documented trade-off. Tests currently require mixed ids (`test_search_graph.py:409-416`).+2. **Community/entity channels unfiltered (high impact on recall@20).** Entities search all labels (`search.py:111`). Communities emit neighbors of TYPE/CLASS/TOPIC, including ATTR hubs (`graph_channels.py:588-614`). Those ids cannot match gold doc_ids unless they are product entity uuids.+3. **Eval cannot attribute a hit to a channel (high impact on diagnosis).** `eval.json` stores canonicalized `hit_ids` only (`evaluate.py:364-372`). We cannot yet falsify “graph never fired useful hits” vs “graph fired gold ASINs that RRF buried” without a new dump or isolated arms.+4. **n=11, no pre-registered control until this ablation (critical for inference).** Means have no interval. One query (esci-67) has a single gold; one (esci-72) is a total miss. Deliberate slice size for ingest cost; **not** a full-ESCI claim.+5. **MRR is saturated and insensitive here (important).** Mean 0.848 is “first Exact usually at rank 1.” Interventions that only clean ranks 2–10 will not move MRR. Eval artifact, not a ranking win.+6. **skip_enrichment + generic HAS triples (medium).** Graph is attribute hubs, not LLM entity resolution. Deliberate cost trade-off. Do not interpret this graph as GraphRAG.+7. **Title/class BM25 confound (medium, weaker than assumed on this file).** Session prior: class words in titles can inflate BM25. Located evidence: 0/200 docs have a `class` field; titles still contain type words (envelopes, pads, fence). Allcols text also includes brand and bullets. Do not attribute passages quality to graph CLASS hubs.+8. **k=10 vs gold size ~20 (important).** Recall@10 cannot exceed ~0.5–0.67 on most queries. Recall@20 is the more sensitive recall readout on this slice (located evidence: passages 0.847 vs graph 0.582).++---++## Open questions for the maintainer++1. Is the decision criterion “beat passages-only 0.758 / 0.511 / 0.848 on this slice,” or “make graph channels eventually useful even if passages stay the default”? **Answered:** useful isolated channels; passages stays default. Fused nDCG is not go/no-go.+2. May all follow-up arms reuse `searchbenchesci20` with `--skip-ingest` (no re-ingest, no wipe)? **Answered:** yes.+3. When isolated `entities` / `communities` arms show no gold ASINs, should those channels stay off the default search path (`channels=["passages"]`)? **Answered:** they stay off the default path even after they retrieve gold ASINs.+4. Is p50 retrieve ~200 ms ex-embed still binding for `rerank=plugin:cross-encoder` on this brain?+5. Should headline decisions use nDCG@10, or also treat Recall@20 as primary given gold size ≈20?+6. Is `benchmarks/data/search_esci.jsonl` (title-only, 2000 docs) in scope only as a later confirmation condition, not a substitute for this allcols slice?+7. Should `entities` default to `node_labels=["ENTITY"]` in product search, or remain unfiltered so ATTR/TYPE name matches can enter RRF?++---++## Independent ideas (generated before literature)++Origin: AI-assisted, stage `independent` unless marked `post-check`.++| ID | Statement | Cluster | Predicted metric move vs passages | Disconfirming observation |+| --- | --- | --- | --- | --- |+| I01 | Treat passages-only as the quality baseline; do not add graph fusion until pollution is diagnosed | eval / fusion | nDCG/recall stay; graph arm must not be the default | Isolated graph arms beat passages on nDCG and recall without hub ids |+| I02 | Dump per-hit `channel` + run isolated `entities`, `communities`, `passages+entities` on this brain | eval | none (measurement) | Isolated arms empty and fused lists contain no graph-origin gold ASINs |+| I03 | Never emit `hub:*` into `extra_id_lists`; only product ENTITY uuids | fusion pollution | +nDCG, +recall@10/@20; MRR flat | Offline/online filter leaves Recall@20 much less than 0.85 (already partly located) |+| I04 | Force `node_labels=["ENTITY"]` on the entities channel | ingest/graph | +recall if entities retrieve gold ASINs; else null | Entities-only with ENTITY labels has Recall@10 ≈ 0 |+| I05 | Downweight or gate graph lists in RRF (not peer k=60 lists) | fusion | +nDCG if entities have some gold | Weighted RRF still loses Recall@20 to passages |+| I06 | Cross-encoder rerank on passages-only head (k≤10) | precision-at-top | +nDCG; MRR maybe; recall@10 unchanged unless head misses gold | nDCG flat after rerank; or p50 retrieve >200 ms |+| I07 | Handle exclusionary queries (`without` / `not`) | recall | +nDCG/recall on esci-72 and similar | Rewrites still miss the 6 gold ASINs |+| I08 | Fielded / title-weighted BM25 | precision | +nDCG if body tokens pollute | Title-only BM25 ≤ allcols passages |+| I09 | SPLADE / ColBERT first-stage plugins | recall | +recall@10/@20 vs passages | Plugin index + latency cost, no metric lift |+| I10 | Enlarge n + paired tests; keep this slice as a smoke | eval artifact | none (inference) | Larger ESCI still shows the same fusion drop |+| I11 | Passages+entities without communities | fusion | smaller drop than full graph if communities are the pollutant | p+e equals or worse than p+e+c |+| I12 | Raise retrieve `k` then cut @10 | recall (gaming) | +recall@10 by stuffing | nDCG@10 falls; SLO missed — do not use as a quality claim |+| I13 | LLM enrichment / richer triples | ingest | unknown | Cost/latency; skip_enrichment was deliberate |+| I14 | Use graph as expansion after passages, not as RRF peers | fusion | recover passages nDCG; optional recall if expansion adds gold | Expansion still injects non-gold ASINs into @10 |+| I31 | Offline RRF replay from stored per-channel lists (`post-check`) | fusion | distinguishes pollution vs never-fired | Cannot replay until I02 dumps raw lists |++### Criteria (declared before scoring)++Scale 1–5, higher better. Weights (decision aid only; **decision stays null**):++| Criterion | Weight | 1 | 3 | 5 |+| --- | --- | --- | --- | --- |+| information_gain | 3 | No mechanism split | Splits one pair of stories | Splits pollution vs never-fired vs passages headroom |+| relevance_metrics | 3 | Will not move the three metrics vs 0.758/0.511/0.848 | Moves one metric on this slice | Directly targets a measured gap |+| originality | 1 | Standard IR | Known method, new here | No nearby literature located |+| feasibility | 2 | New models / re-ingest | 3–5 files + one eval | Eval-only on `searchbenchesci20` |+| rigor | 2 | Easy to game | Needs care | Clean ablation, hard to HARKing |+| value_if_null | 2 | Null is noise | Null slightly useful | Null decides the next fork |+| latency_slo | 2 | Likely blow ~200 ms p50 retrieve | Borderline | No extra retrieve cost |++Weighted score = 100 × Σ p_j (x−1)/4, p_j = w_j / Σ w. Not a winner picker.++| ID | IG | Rel | Orig | Feas | Rig | Null | Lat | Score | Uncertainty |+| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |+| I02 | 5 | 4 | 3 | 5 | 5 | 5 | 5 | 92 | Low |+| I01 | 5 | 4 | 2 | 5 | 5 | 5 | 5 | 90 | Low |+| I04 | 4 | 4 | 2 | 5 | 4 | 5 | 5 | 82 | Med: depends on search_entities quality |+| I03 | 4 | 5 | 3 | 4 | 4 | 4 | 5 | 82 | Low–med: post-hoc hub drop already weak |+| I10 | 4 | 3 | 2 | 4 | 5 | 5 | 5 | 77 | Low |+| I14 | 4 | 4 | 3 | 3 | 4 | 4 | 3 | 67 | Med |+| I05 | 3 | 4 | 2 | 3 | 3 | 4 | 5 | 63 | Med |+| I07 | 4 | 4 | 3 | 2 | 3 | 4 | 3 | 60 | High transfer |+| I08 | 3 | 3 | 2 | 3 | 3 | 3 | 5 | 55 | Med |+| I06 | 3 | 4 | 2 | 3 | 3 | 3 | 2 | 50 | Lat veto possible |+| I09 | 3 | 4 | 2 | 2 | 3 | 3 | 2 | 47 | Fashionable here |+| I12 | 2 | 3 | 1 | 5 | 1 | 2 | 4 | 42 | Rigor veto |+| I13 | 2 | 2 | 2 | 2 | 2 | 3 | 4 | 40 | Out of first wave |++---++## Adversarial review (shortlist)++**I02 isolated arms.** Falsifier: all isolated graph arms return gold ASINs in top-10 and fused p+e+c still loses — then the bug is specifically three-list RRF, not empty channels. Alternative: `--skip-ingest` passages arm hit a warmer cache (esci-67 passages retrieve 266 ms vs graph 108 ms). Residual: n=11.++**I03 hub filter.** Falsifier: already located — dropping `hub:*` after canonicalize does not recover Recall@20. Alternative: hubs in the scored list are a symptom; the cause is extra RRF lists changing passage ranks. Revise I03 to “do not append non-product lists,” not “strip hubs after the fact.”++**I04 ENTITY-only.** Falsifier: entities-only Recall@10 ≈ 0. Alternative: `search_entities` CONTAINS on long titles already returns products; label filter changes little. Selection: we only observed fused lists, not raw entity lists.++**I06 rerank.** Falsifier: nDCG flat because first-stage already puts Exact at rank 1 (MRR 0.85). Alternative: any nDCG lift is E vs S reordering on n=11. Latency: CE on 10 docs can blow the retrieve budget; call it out. Metric gaming: rerank cannot raise Recall@10 if gold is outside the head.++**I07 negation.** Falsifier: gold products for esci-72 are not matchable in this 200-doc pool. Alternative: conceptual query, not syntactic negation.++**I12 raise k.** Reject as a quality intervention. It games Recall@k without fixing ranking.++---++## Frontier techniques++Search date: **2026-08-18**. Tools: arXiv MCP `search_papers` / `get_abstract`. Queries: `"Shopping Queries Dataset" OR ESCI "product search"`; `"reciprocal rank fusion"`; `ti:"reciprocal rank fusion"`; `"knowledge graph" retrieval product search OR "entity-oriented search"`; `ti:"passage re-ranking" BERT OR ti:SPLADE OR ti:ColBERT`; `query negation retrieval OR exclusionary queries`; plus direct ids `2206.06588`, `1805.07591`, `2404.16130`, `1901.04085`, `2004.12832`, `2107.05720`. Limits: one session, cs.IR-heavy; absence from search is not novelty. Cormack et al. SIGIR 2009 RRF was not located on arXiv in this pass (`search-incomplete` for that specific paper).++### Reciprocal rank fusion of heterogeneous lists++- **Mechanism:** Unweighted RRF sums 1/(k+rank) across lists (`fact_filter.py:67-78`, k=60). BrainAPI appends each graph channel as an extra list (`search.py:246-265`).+- **arXiv:** Canonical 2009 paper not retrieved here. **2503.20698** (Samuel et al., MMMORRF) reports weighted, modality-aware RRF and +37% nDCG@20 vs single-modality on video benchmarks — different domain.+- **Cost:** Unweighted extra lists are cheap; they still change the top-k composition.+- **Fit:** We already fuse incompatible id spaces. Weighted/gated RRF is an adapt if isolated arms show useful product ids. Peer unweighted fusion of hubs is challenge-located by our ablation (graph worse than passages).+- **Verdict:** **adapt** gating/weighting after I02; do not add more unweighted lists.+- **Literature status:** `mixed` (method exists; our setting violated the “compatible lists” assumption).++### ESCI / Shopping Queries Dataset++- **Mechanism:** Difficult Amazon queries; E/S/C/I labels; Task 1 is ranking with gains typically E=1, S=0.1, C=0.01, I=0 (`2206.06588`, Reddy et al.).+- **Reported gain:** Paper is a dataset plus baselines, not a BrainAPI method. About 130k queries / 2.6M judgments, multilingual.+- **Cost:** Full ESCI ingest is large; this slice is 11 queries / 200 docs by design (`catalog.py` caps).+- **Fit:** Our grades match Task 1. Our n does not. Title-only 2000-doc file is a different condition.+- **Verdict:** **adopt** as the label protocol; **reject** treating this slice as a full-ESCI result.+- **Literature status:** `support-located` for metric definitions; `challenge-located` for generalizing n=11.++### Entity-oriented / KG-in-the-ranker (not hub-in-RRF)++- **Mechanism:** EDRM (`1805.07591`, Liu et al.) represents queries/docs with words and entity annotations; KG semantics live in embeddings; ranking is interaction-based, end-to-end.+- **Reported gain:** Commercial search log; paper claims better generalization. Not ESCI.+- **Cost:** Needs entity linking plus a learned ranker, not `hub:attr` uuids in RRF.+- **Fit:** Current mapping emits generic HAS hubs (`mapping.py:180-251`) and fuses raw node ids. That is not EDRM.+- **Verdict:** **reject** as a next experiment; **adapt** later only if entities-only retrieves gold products and we add a ranker that consumes annotations.+- **Literature status:** `mixed`.++### GraphRAG community summaries++- **Mechanism:** `2404.16130` (Edge et al.): LLM-extracted entity graph, community summaries, map-reduce answers for global questions.+- **Reported gain:** Comprehensiveness/diversity on query-focused summarization, ~1M-token corpora — not nDCG of product ASINs.+- **Cost:** LLM index plus per-query summary tokens. Violates skip_enrichment and the search retrieve SLO.+- **Fit:** BrainAPI `communities` are typed catalog hubs, explicitly not Leiden / `kg_topic_sessions` (`18-search-eval-protocol.md`).+- **Verdict:** **reject**. Assumption BrainAPI violates: search is item ranking, not corpus QFS; skip_enrichment forbids the LLM graph.+- **Literature status:** `challenge-located` (wrong task).++### Cross-encoder passage rerank++- **Mechanism:** `1901.04085` (Nogueira and Cho): BERT scores query–passage pairs and reorders a first-stage head. Protocol already has `plugins/search-rerank` via `rerank=plugin:cross-encoder` (`18-search-eval-protocol.md`).+- **Reported gain:** +27% relative MRR@10 on MS MARCO passage vs prior SOTA (2019). Not ESCI; our MRR is already 0.85.+- **Cost:** Second-stage model on ≤10 candidates (`hooks.py:7`). Likely material retrieve latency. Call out if p50 approaches 200 ms.+- **Fit:** Best tested on **passages-only**. Reranking a polluted graph list first is the wrong order.+- **Verdict:** **adapt** as a passages-only arm after I02, with an explicit latency gate.+- **Literature status:** `support-located` for method; `mixed` for expected MRR lift here.++### Learned sparse / late interaction++- **Mechanism:** SPLADE (`2107.05720`, Formal et al.) learns sparse term weights for first-stage ranking. ColBERT (`2004.12832`, Khattab and Zaharia) late-interacts token vectors (MaxSim). Both exist as optional plugins in the protocol.+- **Reported gain:** Competitive with dense/sparse SOTA on MS MARCO-class passage tasks. Extra index and FLOPs.+- **Cost:** Own index (`POST /search-splade/index` / ColBERT index). Can blow the retrieve budget. Fashionable relative to this repo’s measured gap.+- **Fit:** Worth a later passages-only bake-off, not a graph-fusion fix.+- **Verdict:** **reject** for the next few days; **adapt** after passages-only headroom is measured on a larger n.+- **Literature status:** `support-located` as IR methods; low expected gain per implementation cost here.++### Exclusionary / negation queries++- **Mechanism:** ExcluIR (`2404.17288`, Zhang et al.) shows first-stage retrievers fail on “what I do not want” queries; a dedicated train set helps but stays below humans; generative retrieval is claimed stronger.+- **Reported gain:** Architecture-wide struggle on 3,452 annotated exclusionary queries; training helps, gap remains.+- **Cost:** New query rewrite or trained retriever. Latency depends on implementation.+- **Fit:** Several slice queries are `without` / `not` (esci-12/13/18/19/2/34/42/60/67/72). Only esci-72 is a total miss. Do not treat the whole slice as an ExcluIR replica.+- **Verdict:** **adapt** as a targeted error analysis plus optional rewrite **after** fusion diagnosis. Status: `support-located` for the esci-72 failure mode.++### Instruction-following retrievers / metadata-as-text++- **Mechanism:** InF-IR (`2505.21439`) trains instruction-aware embeddings. Metadata-aware RAG (`2601.11863`) finds prefixing metadata in the chunk helps.+- **Fit:** Allcols ingest already prefixes Title/Brand/bullets (`catalog.py:234-256`). InF-style models are a new index.+- **Verdict:** **reject** for this sprint (`mixed` / not needed to explain graph vs passages). Reopen after I02.++### Post-check ideation (after literature)++- I31 (offline RRF replay) added. Weighted RRF (I05) upgraded from “fashionable” to “adapt if I02 shows useful entity ASINs” because **2503.20698** is support-located for *weighted* fusion, not for unweighted extra lists.+- GraphRAG and EDRM were not added as winners. Absence of a “graph always helps product nDCG” paper in this search is not a gap we should fill with more channels.++---++## Critique of the current eval as a study++### Summary++Two arms on one frozen 200-doc / 11-query ESCI-US slice, same brain, same qrels, fusion RRF, no rerank, skip_enrichment. The graph arm added entities+communities and structured HAS triples. The passages arm reused chunks (`--skip-ingest`). Headline: passages 0.758 / 0.511 / 0.848 vs graph 0.682 / 0.466 / 0.848. This is a paired within-corpus contrast, not an RCT and not full ESCI.++### Strengths++- Same brain and corpus; passages arm did not re-ingest (reduces write-path drift).+- Graded nDCG uses ESCI-style gains; gold collapse to doc_id is implemented (`evaluate.py:149-161, 201-222`).+- Latency split retrieve vs embed is recorded; graph p50 108 ms and passages p50 29 ms both sit under the ~200 ms retrieve SLO.+- Ledger now records `channels`, so the two arms cannot be silently compared as if they were the same system.+- Per-query lists exist in `eval.json` even though `report.json` is aggregate-only.++### Concerns++**Critical**++- n=11. No confidence interval, no multiplicity control, no pre-registered primary metric among {nDCG, Recall@10, Recall@20, MRR}. One total-miss query and one single-gold query dominate means.+- No channel-attributed hits. The study cannot yet say whether graph lists contained gold ASINs.+- Graph vs passages is confounded with extra RRF lists, mixed id spaces, structured ingest (graph arm only), and different retrieve-path work. The passages ablation removes the “no control” problem but does **not** isolate a single mechanism.++**Important**++- Recall@10 is a poor recall target when |gold|≈20; Recall@20 tells the real crowding story (0.847 vs 0.582).+- MRR identical on all 11 queries: the study is silent on precision-at-1 differences.+- Title-class BM25 confound remains a plausible passages explanation; this file has no `class` field, so the confound is title/body type words, not mapped CLASS hubs.+- skip_enrichment plus generic HAS triples means a graph loss is not a loss for “KG retrieval” in the EDRM/GraphRAG sense.+- esci-67 passages retrieve 266 ms vs graph 108 ms: cache or load difference; do not over-read p95.++**Minor**++- `report.json` omits per-query rows (they live in `eval.json`).+- Default evaluate `k=20` then metrics cut at 10 (`cli.py:321`, `evaluate.py:277-279`).+- Mixed chunk/node ids are tested as desired behavior (`test_search_graph.py:409-416`).++### Recommendations++1. Keep passages-only as the baseline for quality claims.+2. Next measurements: isolated channels + per-hit channel dump + Recall@20 in the headline table.+3. Do not ship more graph fusion until I02/I11 falsify or confirm pollution.+4. Enlarge n before any “ESCI improved” sentence. Keep title-only 2000-doc as a separate condition.+5. Preregister: primary = nDCG@10 vs passages; secondary = Recall@20; MRR expected flat; latency gate p50 retrieve <200 ms.++### Overall assessment++The numbers **can** support: on this 11-query allcols slice, adding entities+communities via unweighted RRF **hurt** nDCG@10 and recall relative to passages-only, while MRR did not change, and retrieve got slower (108 vs 29 ms p50). They **cannot** support: graph-aware search is worse on ESCI; passages is SOTA; class-word BM25 is proven; GraphRAG failed; or any intervention is validated. Offline hub-drop **can** support that visible `hub:*` ids are not the whole gap. It **cannot** support that graph never produced a useful product id.++---++## Implementation plan++**Architecture decisions**++- Measurement and ablations before ranking-code changes.+- Reuse `searchbenchesci20` with `--skip-ingest`. Never wipe or score `locomoconv*`, `beam*`, `demorecsys`.+- Quality claims compare to **passages-only** 0.758 / 0.511 / 0.848, not to the graph arm.+- Core channel names stay `passages|entities|events|communities`. Default omitted channels stay `["passages"]`.+- No edit to `~/.cursor/plans/graph-aware_search_channels_e0ddd720.plan.md`.++Passages-only on this slice/brain is **done** (located evidence above). It is Task 0, not Task 1.++### Phase 1: Measurement (fail fast)++### Task 1: Persist per-hit channel and raw id before canonicalize++**Description:** Write each hit’s `id`, `channel`, and pre-canonical id into `eval.json` queries so later arms can replay fusion and count graph-origin gold ASINs.++**Acceptance criteria:**++- [x] `eval.json` `queries[i].hits` includes `id`, `channel`, and canonical `doc_id`+- [x] Existing metrics still match a recompute from those hits+- [x] Ledger still upserts only `benchmarks.search`++**Verification:**++- [ ] `cd benchmarks && .venv/bin/python -c "import json; json.load(open('runs/search-esci-slice-passages/eval.json'))"`+- [ ] `pytest tests/test_search_graph.py -q` (no harness import of `src/` from `benchmarks/search`)++**Dependencies:** None++**Files likely touched:** `benchmarks/search/evaluate.py`, `benchmarks/search/report.py`++**Estimated scope:** S++### Task 2: Isolated channel arms on the frozen brain++**Description:** Three `--skip-ingest` evaluates on `searchbenchesci20` / `search_esci_slice.jsonl`: `entities`; `communities`; `passages,entities`. Do **not** pass `--ingest-graph`. Compare to passages-only.++**Acceptance criteria:**++- [x] Three new `runs/search-esci-slice-*` reports with `channels` recorded+- [x] Table of nDCG@10, Recall@10, Recall@20, MRR, p50 retrieve vs passages+- [x] Written call: useful-hits (gold ASINs in isolated lists) vs pollution (fused p+e or p+e+c worse than passages)++**Verification:**++```bash+cd benchmarks+./search.sh --brain searchbenchesci20 evaluate \+ --dataset data/search_esci_slice.jsonl --skip-ingest \+ --channels entities --run search-esci-slice-entities+./search.sh --brain searchbenchesci20 evaluate \+ --dataset data/search_esci_slice.jsonl --skip-ingest \+ --channels communities --run search-esci-slice-communities+./search.sh --brain searchbenchesci20 evaluate \+ --dataset data/search_esci_slice.jsonl --skip-ingest \+ --channels passages,entities --run search-esci-slice-pe+```++**Dependencies:** Task 1 preferred (channel dump); can run without it++**Files likely touched:** none in core (eval only); new run dirs++**Estimated scope:** S++### Task 3: Offline fusion-replay / hub-drop notebook in-repo++**Description:** Script that reads two `eval.json` files (or Task 1 dumps) and recomputes metrics after (a) dropping `hub:*`, (b) intersecting with product ASINs, (c) if raw lists exist, RRF replay without graph lists. Check in under `benchmarks/search/` or a research helper, not repo root.++**Acceptance criteria:**++- [x] Reproduces compact hub-drop Recall@10 0.476 / MRR 0.848 from `search-esci-slice-allcols` (nDCG 0.764 when ranks 11+ refill @10; earlier 0.691 did not)+- [x] Prints per-query deltas vs passages+- [x] Exits 0 on the two existing eval files++**Verification:**++```bash+cd benchmarks && .venv/bin/python -m search.replay_fusion \+ --graph runs/search-esci-slice-allcols/eval.json \+ --passages runs/search-esci-slice-passages/eval.json+```++**Dependencies:** None for (a)(b); Task 1 for (c)++**Files likely touched:** `benchmarks/search/replay_fusion.py` (new), maybe `tests` under `benchmarks/` if present++**Estimated scope:** S++### Checkpoint: After Tasks 1–3++- [x] We can say, in one sentence, whether entities-only retrieved gold ASINs+- [x] We can say whether communities-only retrieved gold ASINs+- [x] We can say whether passages+entities is closer to passages or to p+e+c+- [x] Human reviews before any fusion code change+- [x] If both isolated graph arms have Recall@10 = 0 and no gold ASINs: **stop graph fusion work**; go to Phase 3 (passages headroom)++Entities-only Recall@10 was 0.415 **before** the ranking change (gold ASINs present; hubs stealing slots). Communities-only was 0. Passages+entities equaled p+e+c. Ranking/filter work proceeded; default channels were not changed.++### Phase 2: Fusion hygiene (only if Task 2 shows useful product ids)++### Task 4: Fuse only product ENTITY uuids++**Description:** Before `extra_id_lists.append`, keep ids that are product entities (uuid equals a catalog doc_id / label ENTITY). Do not append ATTR/TYPE/CLASS hub ids. Default omitted channels remain passages.++**Acceptance criteria:**++- [x] Unit test: hub id does not appear in fused output when passages+entities+communities run on a fixture+- [x] Mixed chunk+ENTITY uuid still allowed (existing test can stay for ENTITY ids)+- [x] `--skip-ingest` p+e+c eval: Recall@10 moved toward passages (0.466 → 0.497); nDCG@10 did not (0.682 → 0.653). Default path unchanged.++**Verification:**++- [ ] `pytest tests/test_search_graph.py -q`+- [ ] Repeat Task 2 p+e+c command after the change; compare to passages 0.758 / 0.511 / 0.847++**Dependencies:** Task 2 (go/no-go), Task 1++**Files likely touched:** `src/services/api/controllers/search.py`, `src/core/search/graph_channels.py`, `tests/test_search_graph.py`++**Estimated scope:** M++### Task 5: Optional ENTITY label default on entities channel++**Description:** If Task 2 entities-only is dominated by ATTR hubs, pass or default `node_labels=["ENTITY"]` for catalog search without adding recsys field names to the API.++**Acceptance criteria:**++- [ ] Entities-only eval with `--node-labels ENTITY` recorded+- [ ] If Recall@10 still ≈ 0, document I04 as null and do not change product default++**Verification:** same `./search.sh --brain searchbenchesci20 evaluate --skip-ingest --channels entities --node-labels ENTITY --run search-esci-slice-entities-label`++**Dependencies:** Task 2++**Files likely touched:** none, or `search.py` only if default changes++**Estimated scope:** S++### Checkpoint: After Tasks 4–5++- [ ] Graph fusion is either gated to product ids or explicitly left off the default path+- [ ] No claim of graph lift unless nDCG and Recall@20 both beat passages on this slice+- [ ] p50 retrieve still <200 ms++### Phase 3: Passages headroom (higher leverage than more graph)++### Task 6: Passages-only cross-encoder arm (latency-gated)++**Description:** `--channels passages --rerank plugin:cross-encoder` on the frozen brain. Primary: nDCG@10 vs 0.758. Hard gate: p50 retrieve. If plugin missing, expect HTTP 400, not a fake ranking miss (`hooks.py:103-113`).++**Acceptance criteria:**++- [x] Report written even if status failed (plugin absent)+- [x] If ok: nDCG@10, MRR, Recall@10, p50/p95 retrieve vs passages+- [x] No graph channels on this arm++**Verification:**++```bash+cd benchmarks+./search.sh --brain searchbenchesci20 evaluate \+ --dataset data/search_esci_slice.jsonl --skip-ingest \+ --channels passages --rerank plugin:cross-encoder \+ --run search-esci-slice-passages-ce+```++**Dependencies:** Checkpoint after Phase 1; open question 4++**Files likely touched:** none if plugin already wired++**Estimated scope:** S++### Task 7: Exclusionary miss analysis (esci-72)++**Description:** For esci-72 (and optionally other `without`/`not` queries), dump gold titles vs top-20 passages. Decide whether the miss is pool coverage, lexical negation, or conceptual intent. No production rewrite until the dump exists.++**Acceptance criteria:**++- [x] One page or eval artifact listing the 6 gold ASINs and why they missed+- [x] Binary decision: rewrite experiment vs “not testable on this pool”++**Verification:** python over `eval.json` + `search_esci_slice.jsonl` (no new ingest)++**Dependencies:** None++**Files likely touched:** optional helper under `benchmarks/search/`++**Estimated scope:** S++### Task 8: Protocol — Recall@20 in report + paired note++**Description:** Surface Recall@20 in `report.json` / printed table; add a one-line paired-delta note (graph minus passages) in `report.md`. Do not invent p-values on n=11.++**Acceptance criteria:**++- [ ] `report.json` includes `recall@20`+- [ ] Markdown table shows channels++**Verification:** `./search.sh report --run search-esci-slice-passages`++**Dependencies:** None++**Files likely touched:** `benchmarks/search/report.py`++**Estimated scope:** S++### Checkpoint: Complete++- [ ] Passages-only remains the cited baseline+- [ ] I02 falsification path executed+- [ ] No ranking change shipped without Task 2 go/no-go+- [ ] Human approved before any larger ESCI ingest++---++## Risks++| Risk | Impact | Detection |+| --- | --- | --- |+| HARKing a graph win from esci-18 nDCG | High | Require Recall@20 and ≥2 queries |+| Treating n=11 as ESCI | High | Dataset label + n_queries in every sentence |+| Re-ingest drift on searchbenchesci20 | High | `--skip-ingest` only; never wipe |+| Ledger pollution of other suites | High | Upsert `benchmarks.search` only |+| CE / SPLADE blow 200 ms SLO | Med | Record p50 retrieve; abort claim if ≥200 ms |+| Metric gaming via larger k | Med | Reject I12 as quality |+| Cache confound on latency | Low | esci-67 already odd; compare medians |+| Changing default channels in core | High | Default stays `["passages"]`; test `test_search_graph.py:108` |++---++## Decision log (proposal only)++- Candidates considered: I01–I14, I31, plus rejected GraphRAG/EDRM-now/I12/I13.+- Criteria/weights: table above, set before scoring.+- Literature and review date: 2026-08-18.+- **Decision: useful isolated graph channels; passages remains default.** Accountable human: isolated Recall@10 / gold-ASIN presence is go/no-go; fused nDCG is not.+- Revisit trigger: larger-n ESCI, or an events/expand arm.++---++## Graph fusion hygiene (2026-08-18) — G01–G08++Workstream: optional graph channels in `/retrieve/search` without shrinking unique-product recall or nDCG vs passages-only. Brain `searchbenchesci20`, dataset `benchmarks/data/search_esci_slice.jsonl`, n=11, `--skip-ingest`, `--fusion rrf`, `--rerank none`. Default omitted `channels` stayed `["passages"]`. No production `RERANK_MAX_K` change. Ledger: `benchmarks.search` only.++Predeclared primary: Recall@20 then nDCG@10. Live fusion change only if an arm is not worse than passages on both. Null → G08 (graph as side channel).++### Task 1 — per-channel lists and unique-doc counts (located evidence)++`eval.json` now stores `dense_ids`, `bm25_ids`, `entity_ids`, `community_ids`, `n_unique_docs_raw`, `n_unique_docs_canonical`, and `gold_grades`. Search responses expose `channel_lists` when the API from this checkout is running. Ranking of hits is unchanged.++| Arm | Run | unique-doc@20 raw | unique-doc@20 canonical |+| --- | --- | --- | --- |+| Passages | `search-esci-slice-passages-gate` | 20.0 | **20.0** |+| Fused p+e+c | `search-esci-slice-pec-lists` | 20.0 | **15.3** |++**Located evidence:** passages-only unique-doc@20 stays 20 on every query. Fused p+e+c still drops unique products (mean 15.3) because chunk UUIDs and catalog `doc_id`s occupy separate RRF slots (P1).++### Tasks 2–3 — offline replay (located evidence)++Replay: `python -m search.replay_fusion --mode all` on the Task 1 dumps (`pec-lists` + `passages-gate`), stitching isolated `entities-after` / `communities-after` when a list is missing. Collapse maps every id to `doc_id` **before** RRF. Gated arms used the same dumps; expansion N=10 and graph weights `{0.1, 0.25, 0.5}` were predeclared.++| Arm | nDCG@10 | Recall@10 | Recall@20 | MRR | unique@20 |+| --- | --- | --- | --- | --- | --- |+| Passages | 0.758 | 0.511 | 0.847 | 0.848 | 20.00 |+| G01 collapse-rrf | 0.723 | 0.498 | 0.839 | 0.847 | 20.00 |+| G04 weighted-0.1 | 0.740 | 0.511 | 0.847 | 0.852 | 20.00 |+| G04 weighted-0.25 | 0.725 | 0.498 | 0.847 | 0.852 | 20.00 |+| G04 weighted-0.5 | 0.726 | 0.498 | 0.852 | 0.852 | 20.00 |+| G02 expansion N=10 | 0.758 | 0.511 | 0.650 | 0.848 | 20.00 |+| G05 confirmation | 0.704 | 0.502 | 0.847 | 0.795 | 20.00 |++Same protocol on the earlier isolated-channel dumps (no live `channel_lists`) gave collapse R@20 0.827 / nDCG 0.686 and expansion R@20 0.815 / nDCG 0.758. Winner rule was identical.++**P1 survived as the unique-doc / Recall@20 hole.** Collapse restores unique@20 to 20 and Recall@20 to 0.839 (not ≪ 0.847). **P2/P3 survived as the nDCG hole.** Collapse and every weighted/confirmation mix stay below passages nDCG 0.758. Expansion keeps nDCG but loses Recall@20 (novel graph ASINs displace gold in ranks 11–20).++No arm is not-worse than passages on **both** nDCG@10 and Recall@20.++### Checkpoint — G08 (decision)++**Decision:** G08. Keep graph off the ranking mix; expose it as a side channel. Do not implement G01/G02/G04/G05 in `fuse_passage_lists` / live RRF. Default omitted channels remain `["passages"]`.++This is not a claim that graph lists contain no gold ASINs (isolated entities Recall@10 0.482 still holds). It is a claim that **unweighted or gated fusion does not beat passages on this n=11 slice**.++### Task 5 — live passages gate (located evidence)++`runs/search-esci-slice-passages-gate` — `--channels passages`, `--skip-ingest`.++| Arm | nDCG@10 | Recall@10 | Recall@20 | MRR | p50 retrieve |+| --- | --- | --- | --- | --- | --- |+| Passages (frozen control) | 0.758 | 0.511 | 0.847 | 0.848 | 28–29 ms |+| Passages gate (this run) | 0.758 | 0.511 | 0.847 | 0.848 | 26 ms |++Within noise of the frozen control. p50 retrieve 26 ms passes the 200 ms gate. Fused p+e+c `pec-lists` is unchanged at 0.653 / 0.497 / 0.653, p50 160 ms (still under 200 ms, still worse than passages). Do not treat 0.857 as a number from this workstream (that figure is CE-on-pool only).++### Task 6 — `expand=neighbors` (located evidence)++`runs/search-esci-slice-entities-neighbors` — `--channels entities --expand neighbors`.++| Arm | nDCG@10 | Recall@10 | Recall@20 | MRR | unique@20 | p50 retrieve |+| --- | --- | --- | --- | --- | --- | --- |+| Entities-only (prior) | 0.655 | 0.482 | 0.822 | 0.757 | 20 | 51 ms |+| Entities + neighbors | 0.655 | 0.482 | 0.822 | 0.757 | 20 | **223 ms** |++Recall@10 matches entities-only 0.482. `neighbor_ids` were empty on all 11 queries (adjacent items already in the entity seed set, or dropped by the item filter). p50 223 ms **fails** the 200 ms gate. Stop. Do not fuse neighbors into default search.++### What this does not support++- Product claim of “graph precision” or ESCI-wide lift.+- Changing default search to graph.+- Shipping ID-collapse, weighted RRF, expansion-after, or confirmation as a ranking change.+- n=11 as a published ESCI result.++---++## CE-on-pool / first-stage n=74 (2026-08-18, separate from fusion)++Fusion files were not edited here. Protocol: CE-on-pool vs Reddy **0.857**; first-stage vs **0.495** nDCG@10. n=74 ≠ ~4477.++**Located evidence**++- Live `--skip-ingest --channels passages` on `searchbenchesci74`: nDCG@10 **0.495**, nDCG@20 0.549, R@10 0.377, R@20 0.592, MRR 0.758, p50 62 ms (matches the prior first-stage row).+- Local fielded BM25 / RM3 / SPLADE on the same 2043-doc JSONL: best nDCG@10 **0.465** (SPLADE) / **0.458** (title-boost BM25). All **below** 0.495. First-stage win rule (≥0.515) **failed**.+- 4-class MiniLM-L-12 CE-on-pool (40k US Task-1 train, test qids held out, catalog text, weighted ESCI gains): nDCG@20 **0.678**, full-list nDCG **0.747**. Exceeds our FT **0.669**. Does **not** beat Reddy **0.857**.++Do not mix these rows with slice n=11 0.758. Do not change product default rerank.++---++## Deeper k, then harness CE on retrieved hits (2026-08-18)++**Decision:** Goal 1 is more gold in the retrieved list (Recall@k, pool coverage). Goal 2 is nDCG of that same list. Graph stays off the ranking mix (G08). Production default `k` and `RERANK_MAX_K = 10` were not changed. Do **not** cite Reddy **0.857** on these arms (that figure is CE-on-pool only).++**Control (located evidence):** `searchbenchesci74`, `search_esci_74.jsonl`, `--skip-ingest --channels passages --fusion rrf --k 20`: nDCG@10 **0.495**, nDCG@20 0.549, Recall@10 0.377, Recall@20 0.592 (`search-esci-74-passages-control`). With `--rank-pool`: nDCG@20 0.579, **pool_coverage 0.517**, `missing_from_brain` 0 (`search-esci-74-passages-pool`).++**Predeclared Phase 1 win:** pool_coverage **≥ 0.57** or Recall@50 **≥ 0.65**. nDCG@10 may stay flat. **Phase 2** only after that win: nDCG@10 **≥ 0.515** on the same retrieve-k list, Recall@50 not down vs Phase 1.++### Phase 1 — deeper first-stage retrieve (located evidence)++Same brain, `--skip-ingest`, passages only, `rerank=none`. Mean `n_hits` at k=50 is 50; unique-doc@50 is 50. k=100 unique-doc@100 is 100. p50 retrieve 58–60 ms (under 200 ms).++| Arm | k | Recall@10 | Recall@20 | Recall@50 | Recall@100 | nDCG@10 | nDCG@20 | p50 ms |+| --- | --- | --- | --- | --- | --- | --- | --- | --- |+| Control | 20 | 0.377 | 0.592 | — | — | **0.495** | 0.549 | 62 |+| `search-esci-74-passages-k50` | 50 | 0.379 | 0.590 | **0.834** | — | 0.500 | 0.552 | 60 |+| `search-esci-74-passages-k100` | 100 | 0.379 | 0.591 | 0.831 | **0.906** | 0.500 | 0.552 | 58 |++`--rank-pool` (coverage of the labeled candidate list, not Reddy Task 1):++| Arm | k | pool_coverage | missing_from_brain | Recall@20 | Recall@50 | nDCG@10 | nDCG@20 |+| --- | --- | --- | --- | --- | --- | --- | --- |+| `search-esci-74-passages-pool` | 20 | **0.517** | 0 | 0.592 | — | 0.546 | 0.579 |+| `search-esci-74-passages-k50-pool` | 50 | **0.778** | 0 | 0.669 | 0.832 | 0.567 | 0.624 |+| `search-esci-74-passages-k100-pool` | 100 | **0.846** | 0 | 0.696 | 0.906 | 0.574 | 0.635 |++**Phase 1 call:** win. Recall@50 **0.834 ≥ 0.65**; pool_coverage **0.778 ≥ 0.57**. Extra golds sit in ranks 21–50 (Recall@20 is flat vs k=20). nDCG@10 is unchanged (~0.500). k=100 does not raise Recall@50 vs k=50; Recall@100 is 0.906. Do not mix with CE-on-pool **0.695** or Reddy **0.857**.++### Phase 2 — harness CE over stored `hit_ids` (located evidence)++Sibling of `rank_pool.py`: `benchmarks/search/rerank_retrieved.py` reorders the k=50 retrieved list. It does not call production rerank and does not score the full labeled pool. Default API `rerank=none` and `RERANK_MAX_K = 10` unchanged.++| Arm | model | nDCG@10 | nDCG@20 | Recall@10 | Recall@50 | MRR |+| --- | --- | --- | --- | --- | --- | --- |+| k=50 first-stage | none | **0.500** | 0.552 | 0.379 | **0.834** | 0.765 |+| `search-esci-74-passages-k50-ce` | MS MARCO MiniLM-L-6 | 0.448 | 0.515 | 0.331 | 0.834 | 0.720 |+| `search-esci-74-passages-k50-ce-l12` | 4-class MiniLM-L-12 e2 | 0.467 | 0.508 | 0.341 | 0.834 | 0.706 |++**Phase 2 call:** null. nDCG@10 did not reach **0.515** (needed +0.02 vs the 0.495 k=20 control). Both CEs **hurt** nDCG@10 vs first-stage k=50. Recall@50 held at 0.834 (same hit set). These rows are first-stage retrieve-then-rerank, not CE-on-pool 0.695 / Reddy 0.857.++**Decision:** keep production search at default k / `RERANK_MAX_K=10`. Deeper retrieve finds more gold; zero-shot and local 4-class CE over that retrieved list do not promote it into the nDCG@10 head.++---++## Gold in small and large k (2026-08-18)++**Focal question:** raise gold in both Recall@10 and Recall@50 on the same 74 queries. Graph off. Production `k` / `RERANK_MAX_K` unchanged. Do **not** cite Reddy **0.857** or CE-on-pool **0.695**.++**Predeclared dual win:** Recall@10 **≥ 0.397** and Recall@50 **≥ 0.854** (both +0.02 vs k=20 / k=50 passages). Secondary nDCG@10 ≥ 0.515.++### Task 1 — miss taxonomy (located evidence)++`miss-strata` on `search-esci-74-passages-k50` / `search_esci_74.jsonl`. Reconciles **74** queries / **1270** golds.++| Stratum | Queries | Gold items |+| --- | --- | --- |+| head-ok (gold in top-10) | 67 | 424 in top-10 |+| rank-too-low (gold in 11–50 only) | 2 | 728 in ranks 11–50 |+| total-miss | 5 | 118 missed |++Total-miss qids: `esci-113`, `esci-177`, `esci-267`, `esci-393`, `esci-72`. Dual-k gap is mostly **rank-too-low golds** (728), not the five total misses. Macro Recall@50 0.834 vs micro miss 118/1270 (9%).++### Task 2 — query-side on pathological total-miss only (located evidence)++Rewrote **only** `esci-113` (HTML unescape) and `esci-267` (strip leading `- *`). Did **not** rewrite `esci-72` (prior qrel mismatch). SQL-ish `esci-177` left unchanged. Live `--skip-ingest` `search-esci-74-passages-k50-qrewrite`:++| Arm | Recall@10 | Recall@50 | nDCG@10 |+| --- | --- | --- | --- |+| k=50 passages | 0.379 | 0.834 | 0.500 |+| qrewrite (2 qids) | 0.379 | 0.834 | 0.500 |++Per-qid Recall@10/@50 for `esci-113` and `esci-267` stayed **0**. Stratum-only null. Global dual win not moved.++### Task 3 — local ANCE dual encoder (located evidence)++Harness MiniLM-L-6 dual encoder, US Task-1 train, test qids held out, hard negatives from stored k=50 non-gold `hit_ids` (bank 1387), 6557 triples, 1 epoch. Encode 2043 JSONL docs locally. Not BrainAPI embeddings. Run `search-esci-74-dense-ance-k50`.++| Arm | Recall@10 | Recall@50 | nDCG@10 |+| --- | --- | --- | --- |+| Passages k=20 control | **0.377** | — | **0.495** |+| Passages k=50 | 0.379 | **0.834** | 0.500 |+| Local ANCE MiniLM | 0.302 | 0.750 | 0.396 |++**Dual-k call:** null. Both recalls moved **down**. Do not fuse this retriever into passages. Do not raise `RERANK_MAX_K`. ColBERT and BGE sidecars were run next (below); both also null.++**Decision:** BrainAPI hybrid passages remain the first-stage control. Matching interventions tried here (query rewrite on 2 qids; MiniLM ANCE on the 2043-doc JSONL) did not put more gold in both small and large k.++---++## Retrieved-neg CE, then ColBERT, then BGE (2026-08-18)++**Focal question:** (1) promote Exact already in hybrid ranks 11–50 via a 4-class CE trained on BM25 retrieved negatives; (2–3) try two first-stage sidecars that are not MiniLM dual-encoder. Graph off. Frozen `searchbenchesci74` / `--skip-ingest`. `RERANK_MAX_K` stayed 10. Do **not** mix these rows with CE-on-pool **0.695** or Reddy **0.857**. n=74, no p-values. All three arms were run even though arm 1 was null.++**Predeclared wins:** arm 1 nDCG@10 **≥ 0.515** and Recall@50 not below **0.834**. Arms 2–3 Recall@10 **≥ 0.397** **and** Recall@50 **≥ 0.854**. Secondary nDCG@10 ≥ 0.515.++### Arm 1 — retrieved-negative 4-class CE (located evidence)++Train lists: US Task-1 small-train, test qids held out (`n_holdout_qids=148`), BM25 top-50 over the train product catalog, unlabeled hits = class **I**. 6000 queries, 300000 pairs, all lists had ≥1 I. Checkpoint `benchmarks/data/models/esci-minilm-l12-retrieved` (`source=retrieved-bm25`, 80k pairs, 1 epoch, unweighted). Eval: harness `rerank-retrieved` on stored k=50 `hit_ids` (`search-esci-74-passages-k50-ce-retrieved`). Not production rerank.++| Arm | nDCG@10 | Recall@10 | Recall@50 | MRR |+| --- | --- | --- | --- | --- |+| k=50 passages | **0.500** | 0.379 | **0.834** | 0.765 |+| retrieved-neg 4-class L-12 | 0.465 | 0.351 | 0.834 | 0.712 |++**Arm 1 call:** null. nDCG@10 **0.465** did not reach **0.515** and is below first-stage 0.500 (same direction as pool-trained / MS MARCO CE on these hits). Recall@50 held at **0.834** (same hit set).++### Arm 2 — ColBERT MaxSim sidecar (located evidence)++In-process `plugins/search-colbert` (`colbert-ir/colbertv2.0`) over the 2043 JSONL docs. Brain id `harness-local-colbert`. Did **not** `POST /search-colbert/index` onto `searchbenchesci74`. Run `search-esci-74-colbert-k50`. Channels `harness-colbert`. Not fused with passages.++| Arm | Recall@10 | Recall@50 | nDCG@10 | p50 encode+retrieve |+| --- | --- | --- | --- | --- |+| Passages k=20 / k=50 | **0.377** / 0.379 | **0.834** | **0.495** / 0.500 | ~60 ms retrieve |+| ColBERT MaxSim | 0.311 | 0.714 | 0.434 | 10063 ms |++**Arm 2 call:** dual-k null. Recall@10 **0.311** < 0.397; Recall@50 **0.714** < 0.854. p50 ~10 s is **labeled**, not a 200 ms product claim, not a stop.++### Arm 3 — zero-shot BGE-base (located evidence)++`BAAI/bge-base-en-v1.5` via `local-dense` on the same 2043 JSONL. **Not MiniLM.** No `query:` prefixes. Run `search-esci-74-bge-base-k50`. No new production brain.++| Arm | Recall@10 | Recall@50 | nDCG@10 |+| --- | --- | --- | --- |+| Passages k=20 / k=50 | **0.377** / 0.379 | **0.834** | **0.495** / 0.500 |+| MiniLM ANCE (prior, not this arm) | 0.302 | 0.750 | 0.396 |+| BGE-base (this arm) | 0.322 | 0.799 | 0.441 |++**Arm 3 call:** dual-k null. Recall@10 **0.322** < 0.397; Recall@50 **0.799** < 0.854. Better than MiniLM ANCE 0.302 / 0.750, still below hybrid passages. p50 encode+retrieve 408 ms.++**Three-arm call:** no arm beat the passages control on its predeclared gate. Keep default omitted `channels` = passages. Do not raise `RERANK_MAX_K`. Do not fuse ColBERT or BGE into passages. Do not wipe `searchbenchesci74`.++---++## Exhaustive catalog rank then two-stage (2026-08-19)++**Focal question:** (A) does scoring **every** JSONL doc with the unweighted 4-class L-12 e2 checkpoint beat hybrid k=50; (B) do complementary sidecar lists raise dual-k recall without diluting Recall@10; (C) ship opt-in `mode=catalog` on `/retrieve/search` without changing the default path. Graph off. Frozen `searchbenchesci74` / `--skip-ingest`. Default `RERANK_MAX_K` stayed **10**. Do **not** mix these rows with CE-on-pool **0.695** or Reddy **0.857**. n=74, no p-values. B and C ran even though A was null.++**Predeclared gates:** A nDCG@10 **≥ 0.515** **and** Recall@10 **≥ 0.397**. B Recall@10 **≥ 0.397** **and** Recall@50 **≥ 0.854**, and Recall@10 not below **0.379**. C (same k=50 hits, deeper rerank): A’s nDCG@10 gate with Recall@50 held at **0.834**.++### A — exhaustive 4-class on 2043 docs (located evidence)++Harness `rank-corpus` scored all 2043 JSONL passages per query with `benchmarks/data/models/esci-minilm-l12-4class-nowt-e2` (74 × 2043 ≈ 151k pairs). Protocol `exhaustive-catalog`. Did **not** POST onto `searchbenchesci74`. Run `search-esci-74-exhaustive-ce`. Channels `exhaustive-4class`. Cut @ 5,10,20,50.++| Arm | nDCG@10 | Recall@10 | Recall@50 | p50 |+| --- | --- | --- | --- | --- |+| k=50 passages | **0.500** | **0.379** | **0.834** | 60 ms retrieve |+| Exhaustive 4-class L-12 e2 | 0.416 | 0.266 | 0.603 | 94 s labeled |++**A call:** null. nDCG@10 **0.416** < **0.515** and below hybrid 0.500. Recall@10 **0.266** < **0.397**. p50 is **labeled**, not a 200 ms product claim. MiniLM 4-class over the full file does not beat hybrid RRF on this slice.++### B1 — U01 unique-gold overlap (located evidence)++`list-overlap` on stored `hit_ids` (`search-esci-74-passages-k50` vs `search-esci-74-bge-base-k50` / `search-esci-74-colbert-k50`). Gold ASINs in sidecar top-50 absent from passages k=50:++| Sidecar | unique gold hits | queries with unique golds |+| --- | --- | --- |+| BGE-base k=50 | **40** | 19 / 74 |+| ColBERT k=50 | **21** | 13 / 74 |++Unique golds were nonzero, so B2 included ColBERT.++### B2 — harness RRF union (located evidence)++RRF of passages k=50 + BGE k=50 + ColBERT k=50. Channels `harness-union`. Not fused into live graph. Run `search-esci-74-union-bge-k50`.++| Arm | nDCG@10 | Recall@10 | Recall@50 |+| --- | --- | --- | --- |+| k=50 passages | **0.500** | **0.379** | **0.834** |+| Union passages+BGE+ColBERT | 0.479 | 0.363 | 0.827 |++**B call:** null. Recall@10 **0.363** is below control **0.379** (graph-style dilution). Dual-k **≥0.397 and ≥0.854** failed. Extra lists steal top-10 slots; unique golds in the tail do not raise Recall@50 here (0.827 vs 0.834). Do not fuse these lists into production ranking.++### C — live `mode=catalog` (located evidence)++Shipped: `SearchRequestBody.mode` default `"default"` (retrieve `k`, rerank `min(10, len)`). `mode=catalog` retrieves `k_ret = min(200, max(k, 50))`, reranks `min(len, CATALOG_RERANK_MAX_K=50)`, cuts to request `k`. Live `--skip-ingest` on frozen `searchbenchesci74`, `SEARCH_RERANK_MODEL` = nowt-e2 (same checkpoint as A, not retrieved-neg). Run `search-esci-74-catalog-ce-k50`. Request `k=50` so first-stage depth matches the k=50 control; this tests **plugin rerank of 50** (not 10).++| Arm | nDCG@10 | Recall@10 | Recall@50 | p50 retrieve | p50 client wall |+| --- | --- | --- | --- | --- | --- |+| k=50 passages (`rerank=none`) | **0.500** | **0.379** | **0.834** | 60 ms | — |+| `mode=catalog` + plugin CE | 0.467 | 0.341 | **0.834** | 59 ms | 877 ms |++**C call:** architecture **shipped**; quality gate **null**. Recall@50 held at **0.834** (same 50-hit retrieve). nDCG@10 **0.467** < **0.515** and below first-stage 0.500 — same direction as harness CE-on-retrieved 4-class L-12 e2 (0.467). Catalog+CE is **not** the ADR-007 200 ms default. Omitted `mode` and `RERANK_MAX_K=10` stay the product path. Do not wipe `searchbenchesci74`.++**A/B/C call:** exhaustive MiniLM cannot beat hybrid on this 2043-doc slice; list union dilutes Recall@10; catalog mode is the two-stage hook. Keep default omitted `channels` = passages. Do not cite Reddy **0.857** against these rows.++---++## Frozen-head cascade (2026-08-19)++**Focal question:** can sidecar unique golds raise Recall@50 if hybrid top-10 is frozen? Graph off. Harness only. Not live fusion. `RERANK_MAX_K` unchanged. Do **not** mix with CE-on-pool **0.695** or Reddy **0.857**. n=74, no p-values.++**Predeclared gate:** Recall@10 **≥ 0.379** (no dilution) **and** Recall@50 **≥ 0.854**.++**Located evidence.** `cascade-lists`: passages ranks 1–10 copied; unique golds from BGE+ColBERT top-50 that were absent from passages k=50 inserted at 11–50; remaining passages tail golds kept before non-golds. Run `search-esci-74-cascade-tail-k50`. Channels `harness-cascade`. Protocol `frozen-head-cascade`. Injected **51** unique gold ASINs (BGE 40 ∪ ColBERT 21).++| Arm | nDCG@10 | Recall@10 | Recall@20 | Recall@50 | nDCG@20 |+| --- | --- | --- | --- | --- | --- |+| k=50 passages | **0.500** | **0.379** | 0.590 | 0.834 | 0.552 |+| RRF union (prior, not this arm) | 0.479 | 0.363 | 0.577 | 0.827 | 0.538 |+| Frozen-head cascade | **0.500** | **0.379** | **0.750** | **0.889** | **0.619** |++**Call:** win. Recall@10 **0.379** matches the passages control to floating point (head frozen). Recall@50 **0.889 ≥ 0.854**. nDCG@10 held; nDCG@20 rose because extras land in 11–20. RRF union failed by stealing the head; cascade uses the same unique golds without that. Not fused into live `/retrieve/search`. Do not raise default `RERANK_MAX_K`. Do not wipe `searchbenchesci74`.++---++## DeBERTa-v3-base 4-class on catalog k=50 (2026-08-19)++**Focal question:** does replacing MiniLM with DeBERTa-v3-base (same 4-class weighted-gain recipe) promote golds already in hybrid ranks 11–50 into the top-10? Graph off. Frozen `searchbenchesci74` / no wipe. Default `RERANK_MAX_K` stayed **10**. Live `mode=catalog` only if harness gates fired. Do **not** mix first-stage rows with CE-on-pool **0.695** / **0.710** or Reddy **0.857**. n=74, no p-values.++**Predeclared first-stage win:** Recall@10 **≥ 0.397** **and** nDCG@10 **≥ 0.515**, Recall@50 held at **0.834**.++**Train (located).** `finetune-4class` `--base microsoft/deberta-v3-base`, US Task-1 train, test qids held out (`n_holdout_qids=148`), 80k pairs, 2 epochs, batch 8, unweighted, max_length 192, label smoothing 0.1. Checkpoint `benchmarks/data/models/esci-deberta-v3-base-4class`. Parent venv needed `sentencepiece` (installed 0.2.2). HuggingFace DeBERTa-v3 tokenizer shipped `model_max_length` as an unbounded sentinel; it was set to **192** to match train. Not v3-large. Not the www ensemble.++**Pool discriminator (ranking-in-pool only; not the R@10 gate).** `rank_pool_4class` run `search-esci-74-ce-pool-deberta-base`. nDCG@20 **0.710** vs MiniLM 4-class e2 **0.695**; full-list nDCG **0.777** vs **0.765**. n=74 ≠ ~4477. Do not cite Reddy **0.857** against this row.++**First-stage harness (primary).** `rerank-retrieved` on stored `search-esci-74-passages-k50` `hit_ids`. Run `search-esci-74-passages-k50-ce-deberta`. Plugin 4-logit softmax gains. No live API.++| Arm | nDCG@10 | Recall@10 | Recall@50 |+| --- | --- | --- | --- |+| k=50 passages | **0.500** | **0.379** | **0.834** |+| MiniLM 4-class e2 on those hits | 0.467 | 0.341 | **0.834** |+| DeBERTa-v3-base 4-class on those hits | 0.510 | 0.363 | **0.834** |++**Call:** first-stage **null**. Recall@10 **0.363 < 0.397** (also below passages 0.379). nDCG@10 **0.510 < 0.515** (above MiniLM 0.467 and slightly above passages 0.500). Recall@50 **0.834** held (same hit set). Live `mode=catalog` was **not** run: MiniLM catalog already matched harness CE-on-retrieved, and both quality gates missed. Cascade remains the Recall@50 win. Do not raise `RERANK_MAX_K`. Do not wipe `searchbenchesci74`.++**Interpretation (not a finding beyond this slice).** Pool nDCG@20 moved up a little while first-stage Recall@10 moved down vs hybrid: extra backbone capacity helped ranking-in-pool more than promoting golds through the hybrid head. Compatible with the alternative that labeled pools (coverage 1.0) are easier than the k=50 retrieved list (coverage 0.778). Zhang **0.90** is still ranking-in-pool + large + ensemble, not this y-axis.++---++## Multilingual first-stage (2026-08-19)++Not a US n=74 quality rerun. Not Reddy 0.857. Details in [22-multilingual-ecommerce-search.md](22-multilingual-ecommerce-search.md).++**Located evidence.** ESCI locales are US/ES/JP (`product_locale` in Reddy HTML). No Italian in ESCI. Amazon-M2 has locale IT but rec/title-generation tasks (`2307.09688`). mMARCO has Italian as MT passages (`2108.13897`), not catalog qrels. arXiv `product search` + Italian + `cs.IR` returned 0.++**Decision.** ES download must not clobber US JSONL. Italian smoke fixture is not ESCI. Product default stays passages + `rerank=none`. `searchbenchesci74` not wiped.++**ES first-stage (located evidence).** Run `search-esci-es-passages-k50`, brain `searchbenchescies`, n=62, 2000 docs, k=50 passages, `rerank=none`. nDCG@10 **0.577**, Recall@10 **0.353**, Recall@50 **0.914**, p50 84 ms. Not US n=74 0.500/0.379. Not Reddy ES 0.849 ranking-in-pool.++---++## Production first-stage arms on US n=74 (2026-08-19)++**Focal question:** can cheap first-stage changes raise Recall@50 and/or nDCG@10 on skip-ingest `searchbenchesci74` without CE-on-retrieved or graph-as-default? Graph off. Frozen brain / no wipe. `RERANK_MAX_K` stayed **10**. Do **not** mix with CE-on-pool **0.710** or Reddy **0.857**. n=74, no p-values.++**Predeclared gates.** Hold: nDCG@10 **≥ 0.500**, Recall@10 **≥ 0.379**, p50 retrieve **< 200 ms** (ex-embed). Recall@50 win: **≥ 0.854** and hold head. nDCG@10 win: **≥ 0.520** and Recall not down. Null → leave hybrid RRF, `rerank=none`.++### Frozen-head cascade (opt-in lists)++**Located evidence.** Helper `frozen_head_merge` (hybrid top-10 frozen; unique extra ids fill 11–50). Replay unit test on stored `search-esci-74-passages-k50` + BGE + ColBERT reproduces Recall@50 **0.889**, nDCG@10 **0.500**, Recall@10 **0.379**. Live `/retrieve/search` uses this merge **only** when plugin retriever lists (or `SEARCH_LITERAL_FILL`) are requested — not default RRF of plugins into the head. Default omitted `channels=["passages"]` unchanged.++**Live n=74 sidecar:** not run. `search-colbert` / `search-splade` load, but the ColBERT in-memory index for `searchbenchesci74` is empty (retrieve returns `[]`). Indexing 2043 chunks at query time was not started. **Decision:** cascade stays harness-only. Honest production Recall@50 on the default path stays **0.834**.++### C1 — `fusion=cc` alpha sweep (null)++Skip-ingest, passages, k=50, `rerank=none`. Request `fusion=cc` + `fusion_alpha`. p50 61–63 ms (hold).++| Arm | nDCG@10 | Recall@10 | Recall@50 | p50 retrieve |+| --- | --- | --- | --- | --- |+| k=50 passages RRF | **0.500** | **0.379** | **0.834** | ~60 ms |+| CC α=0.3 (`search-esci-74-cc-a03`) | 0.483 | 0.376 | 0.832 | 63 ms |+| CC α=0.5 (`search-esci-74-cc-a05`) | 0.486 | 0.377 | 0.832 | 61 ms |+| CC α=0.7 (`search-esci-74-cc-a07`) | 0.493 | 0.378 | 0.834 | 61 ms |++**Call:** null. Best nDCG@10 **0.493 < 0.520**. Recall@10 slightly below hold. Product default stays RRF.++### C2 — title-token literal residual (null)++**Predeclared fusion:** frozen-head (not RRF) so the top-10 cannot be stolen. Env `SEARCH_LITERAL_FILL` default **false**. Live skip-ingest `search-esci-74-literal-fill-k50`.++| Arm | nDCG@10 | Recall@10 | Recall@50 | p50 retrieve |+| --- | --- | --- | --- | --- |+| k=50 passages | **0.500** | **0.379** | **0.834** | ~60 ms |+| Literal fill + frozen head | **0.500** | **0.379** | **0.655** | 49 ms (`search.retrieve` only on this run) |++**Call:** null. Recall@10 held (frozen head). Recall@50 **0.655** dropped because non-gold ILIKE token hits filled 11–50 and evicted hybrid tail golds. Flag stays false. Do not treat this as a Recall@50 win.++### C3 — pairwise LTR on the stored k=50 head (null)++**Focal question:** can a cheap linear RankNet on features already in the hybrid list promote the 728 golds sitting in ranks 11–50 into the top-10, without changing the retrieved set? Graph off. Frozen `searchbenchesci74` / no wipe / skip-ingest. Not live `/retrieve/search`. Do **not** mix with CE-on-pool **0.710** or Reddy **0.857**. n=74, no p-values.++**Protocol (located evidence).** `./search.sh ltr-head --from-run search-esci-74-passages-k50 --dataset data/search_esci_74.jsonl --run search-esci-74-ltr-head-k50`. Query-grouped 5-fold CV (OOF predictions; seed=0, 40 epochs, lr=0.05, l2=1e-3, max 400 pairs/query). Features: `rrf_inv`, `bm25_inv`, `dense_inv`, `title_overlap`, `brand_hit`, `query_in_title`. Unlabeled docs in the 50 scored as 0. Overlap-only sort stored as `overlap_only_metrics`, not the gated arm.++| Arm | nDCG@10 | Recall@10 | Recall@20 | Recall@50 | MRR |+| --- | --- | --- | --- | --- | --- |+| k=50 passages RRF | **0.500** | **0.379** | 0.590 | **0.834** | 0.765 |+| Overlap-only (diagnostic) | 0.476 | 0.339 | 0.580 | 0.834 | 0.732 |+| LTR OOF (`search-esci-74-ltr-head-k50`) | 0.515 | 0.384 | 0.583 | **0.834** | 0.753 |+| Predeclared win | **≥ 0.520** | **≥ 0.397** | — | hold | — |++Mean CV weights (not a quality claim): `title_overlap` 3.35, `dense_inv` 2.06, `rrf_inv` 1.57, `brand_hit` 0.51, `bm25_inv` 0.02, `query_in_title` ~0.++**Call (decision):** null. nDCG@10 **0.515 < 0.520**. Recall@10 **0.384 < 0.397**. Recall@50 held. Directional nDCG lift is small and is **not** promotion of the 728 (Recall@10 +0.005; Recall@20 and MRR slightly down vs RRF). Overlap-only is worse than RRF — naive title ranking is not the product. Default stays hybrid RRF, `rerank=none`. Do not wire this ranker into `search.py`.++### C4 — LTR + 4-class CE as a feature (harness win; not live)++**Focal question:** can blending the already-trained 4-class MiniLM score with hybrid RRF via query-grouped LTR raise nDCG@10 on the frozen k=50 list, with Recall@10 held? Graph off. Frozen `searchbenchesci74` / no wipe. Not live `/retrieve/search`. Do **not** mix with CE-on-pool **0.710**, Reddy **0.857**, or CE-as-sole-ranker (located evidence: Recall@10 0.331–0.351). n=74, no p-values.++**Protocol (located evidence).** `./search.sh ltr-head --from-run search-esci-74-passages-k50 --ce-model data/models/esci-minilm-l12-4class-nowt-e2 --pair-policy other_query_neg --run search-esci-74-ltr-cefeat-k50`. Same hypers as C3 (seed=0, 40 epochs, lr=0.05, l2=1e-3, max 400 pairs/query, 5-fold OOF). Feature `ce_gain` = 1·P(E)+0.1·P(S)+0.01·P(C); lists are not sorted by CE. Pair policy: this-query graded golds vs other-query golds in the 50; unlabeled skipped. Predeclared this round: nDCG@10 ≥ **0.520**, Recall@10 ≥ **0.379**, Recall@50 held. Recall@10 ≥ 0.397 is not required to call the harness win.++| Arm | nDCG@10 | Recall@10 | Recall@20 | Recall@50 | MRR |+| --- | --- | --- | --- | --- | --- |+| k=50 passages RRF | 0.500 | 0.379 | 0.590 | **0.834** | 0.765 |+| CE-as-sole-ranker (prior, MiniLM on retrieved) | 0.448 | 0.331 | — | 0.834 | 0.720 |+| LTR OOF no CE (`search-esci-74-ltr-head-k50`) | 0.515 | 0.384 | 0.583 | **0.834** | 0.753 |+| LTR + `ce_gain` (`search-esci-74-ltr-cefeat-k50`) | **0.524** | 0.383 | 0.569 | **0.834** | **0.779** |+| Predeclared this round | **≥ 0.520** | **≥ 0.379** | — | hold | — |++Mean CV weights (not a quality claim): `ce_gain` 3.68, `title_overlap` 2.56, `dense_inv` 1.23, `rrf_inv` 0.66, `brand_hit` 0.45, `bm25_inv` −0.13, `query_in_title` ~0.++**Call (located evidence + decision):** harness win on the predeclared nDCG/hold. nDCG@10 **0.524 ≥ 0.520**. Recall@10 **0.383 ≥ 0.379**. Recall@50 held. MRR 0.765 → 0.779. LightGBM not run (gated only if nDCG close-but-short). **Do not wire into `search.py` without human review:** live CE on 50 hits is extra latency vs the ~60 ms retrieve path; n=74 has no CI; Recall@20 fell 0.590 → 0.569. CE-alone remains a null. Default stays hybrid RRF, `rerank=none` until review.++**A/B/C call:** Italian OR-FTS is a pipeline feature on an allowlisted searchbench. US n=74 live default is unchanged: hybrid BM25+dense, passages, `rerank=none`. Harness nDCG@10 0.524 is not Reddy **0.857**. Later DeBERTa RankNet blend (C5) is the current harness best at **0.542**; still not live.++### C5 — DeBERTa `ce_gain` LTR blend (harness win vs MiniLM; not live)++**Focal question:** on frozen [`search-esci-74-passages-k50`](../../benchmarks/runs/search-esci-74-passages-k50), does replacing MiniLM `ce_gain` with the existing DeBERTa-v3-base 4-class checkpoint raise first-stage nDCG@10 **above 0.524**, with Recall@10 ≥ **0.379** and Recall@50 **0.834** held? Graph off. Frozen `searchbenchesci74` / no wipe. Not live `/retrieve/search`. Do **not** mix with Reddy **0.857**, pool nDCG@20 **0.710**, or DeBERTa-as-sole-sorter (located: nDCG@10 **0.510**, Recall@10 **0.363**). n=74, no p-values. Horizon nDCG@10 ≥ **0.70** is not a fail of this arm.++**Protocol (located evidence).** `./search.sh ltr-head --from-run search-esci-74-passages-k50 --ce-model data/models/esci-deberta-v3-base-4class --pair-policy other_query_neg --run search-esci-74-ltr-deberta-k50`. Same RankNet hypers as C3/C4 (seed=0, 40 epochs, lr=0.05, l2=1e-3, max 400 pairs/query, 5-fold OOF). Lists are not sorted by DeBERTa. Scores cached at `runs/search-esci-74-passages-k50/ce_gain_esci-deberta-v3-base-4class.json` (MiniLM cache left in place). Tokenizer already had `model_max_length=192`; `load_4class_predict` was not edited. Predeclared this round: nDCG@10 **> 0.524**, Recall@10 ≥ **0.379**, Recall@50 held.++Gated I03 (beat 0.524, far from 0.70, `|ce_gain|` not ~0): LightGBM lambdarank, frozen `n_estimators=100`, `max_depth=3`, `learning_rate=0.05`, same CV/features/pair policy, run `search-esci-74-ltr-deberta-lgbm-k50`. No hypersearch.++| Arm | nDCG@10 | Recall@10 | Recall@20 | Recall@50 | MRR |+| --- | --- | --- | --- | --- | --- |+| k=50 passages RRF | 0.500 | 0.379 | 0.590 | **0.834** | 0.765 |+| DeBERTa as sole sorter (prior) | 0.510 | 0.363 | — | 0.834 | — |+| LTR + MiniLM `ce_gain` (`search-esci-74-ltr-cefeat-k50`) | 0.524 | 0.383 | 0.569 | **0.834** | 0.779 |+| LTR + DeBERTa `ce_gain` RankNet (`search-esci-74-ltr-deberta-k50`) | **0.542** | **0.387** | 0.586 | **0.834** | 0.774 |+| LightGBM + DeBERTa `ce_gain` (`search-esci-74-ltr-deberta-lgbm-k50`) | 0.516 | 0.379 | 0.592 | **0.834** | 0.710 |+| Must-beat this round | **> 0.524** | **≥ 0.379** | — | hold | — |+| Horizon (not a fail) | **≥ 0.70** | — | — | — | — |+| Oracle on these 50 | 0.876 | 0.548 | — | 0.834 | — |++Mean RankNet CV weights (not a quality claim): `ce_gain` 3.65, `title_overlap` 2.43, `dense_inv` 1.26, `brand_hit` 0.51, `rrf_inv` 0.45, `query_in_title` 0.02, `bm25_inv` −0.08.++Mean LightGBM gain importances (not a quality claim): `ce_gain` 646, `dense_inv` 561, `rrf_inv` 337, `title_overlap` 162, `brand_hit` 82, `bm25_inv` 19, `query_in_title` 0. LightGBM Recall@10 unrounded **0.3788** (quoted 0.379 in three decimals; hold is strict ≥ 0.379).++**Call (located evidence + decision):** I01 **win vs MiniLM**. nDCG@10 **0.542 > 0.524**. Recall@10 **0.387 ≥ 0.379**. Recall@50 held. Horizon 0.70 missed (~0.16 short; k=50 oracle 0.876). I03 LightGBM **null**: nDCG@10 **0.516** loses to RankNet DeBERTa and to MiniLM 0.524; Recall@10 misses the hold unrounded; `ce_gain` is live, so the miss is the GBDT head, not a dead feature. **Do not wire into `search.py`:** live DeBERTa on 50 hits is extra latency vs ~60 ms retrieve; n=74 has no CI; 0.542 vs 0.524 is a point-estimate gate, not a significance test. Default stays hybrid RRF, `rerank=none` until human review. Do not replace the MiniLM cache.++**I04 (this session):** both heads missed 0.70 with large `ce_gain`. Train queries on matched hybrid k=50 were run on a new `searchbench*` (C6).++**A/B/C call:** US n=74 live default unchanged. Harness best after C5 was DeBERTa RankNet CV **0.542**; C6 apply-from-train is **0.544**. Still not Reddy **0.857** and still not product ranking.++### C6 — LTR trained on 170 matched hybrid lists, applied to frozen n=74 (harness win vs 0.542; not live)++**Focal question:** does fitting RankNet+DeBERTa `ce_gain` on **matched hybrid k=50** from extra US train queries, then applying those weights to frozen [`search-esci-74-passages-k50`](../../benchmarks/runs/search-esci-74-passages-k50), raise nDCG@10 **above 0.542**, with Recall@10 ≥ **0.379** and Recall@50 **0.834** held? Graph off. Frozen `searchbenchesci74` / no wipe. New brain only. Not live `/retrieve/search`. Do **not** mix with Reddy **0.857**, pool nDCG@20 **0.710**, or official-pool LTR. n=74, no p-values. Horizon nDCG@10 ≥ **0.70** is not a fail of 170q.++**Protocol (located evidence).** Download `--split train --max-queries 200 --max-docs 4000 --out data/search_esci_ltr200.jsonl --holdout-dataset data/search_esci_74.jsonl` (did not clobber `search_esci.jsonl` / `_74`). Doc budget bound the slice to **170** queries / **4000** docs; qid overlap with the 74 is **0**. Ingest+evaluate passages k=50, `fusion=rrf`, `rerank=none`, skip_enrichment, brain **`searchbenchesciltr2`** (first `searchbenchesciltr` attempt timed out queued before the Celery worker was up; did not retry on `searchbenchesci74`). Train first-stage diagnostic (different catalog, not the gate): nDCG@10 **0.387**, Recall@10 **0.367**, Recall@50 **0.744**, p50 retrieve 76 ms (`search-esci-ltr200-passages-k50`).++Then `./search.sh ltr-head --from-run search-esci-74-passages-k50 --train-from-run search-esci-ltr200-passages-k50 --train-dataset data/search_esci_ltr200.jsonl --ce-model data/models/esci-deberta-v3-base-4class --pair-policy other_query_neg --run search-esci-74-ltr-deberta-train200`. Fit **once** on 170 train lists (no CV on the 74). Reused frozen-74 DeBERTa cache; scored train lists to `runs/search-esci-ltr200-passages-k50/ce_gain_esci-deberta-v3-base-4class.json`. MiniLM cache left in place. RankNet hypers unchanged. LightGBM not re-run.++**Assumption (catalog shift):** train lists come from a 4000-doc train-qrel index; test lists stay on the 2043-doc `searchbenchesci74` index. Feature space matches; IDF/dense geometry may not.++| Arm | nDCG@10 | Recall@10 | Recall@20 | Recall@50 | MRR |+| --- | --- | --- | --- | --- | --- |+| k=50 passages RRF | 0.500 | 0.379 | 0.590 | **0.834** | 0.765 |+| LTR + DeBERTa RankNet CV (`search-esci-74-ltr-deberta-k50`) | 0.542 | 0.387 | 0.586 | **0.834** | 0.774 |+| LTR + DeBERTa applied from 170 train lists (`search-esci-74-ltr-deberta-train200`) | **0.544** | **0.391** | 0.585 | **0.834** | 0.762 |+| Must-beat this round | **> 0.542** | **≥ 0.379** | — | hold | — |+| Horizon (not a fail) | **≥ 0.70** | — | — | — | — |++Mean applied RankNet weights (not a quality claim): `ce_gain` 4.27, `title_overlap` 3.12, `rrf_inv` 1.40, `dense_inv` 1.31, `bm25_inv` 0.49, `brand_hit` −0.23, `query_in_title` −0.22.++**Call (located evidence + decision):** I04 **win vs I01** on the predeclared point-estimate gate. nDCG@10 **0.544 > 0.542**. Recall@10 **0.391 ≥ 0.379**. Recall@50 held. Horizon 0.70 missed (~0.16 short). Lift vs CV DeBERTa is **+0.002** nDCG / **+0.004** Recall@10; n=74 has no CI, so this can be noise. Train Recall@50 **0.744** shows the 170q lists are not empty of golds, but the catalog is not the 74-index. **Do not wire into `search.py`.** Default stays hybrid RRF, `rerank=none`. Next gated step if chasing 0.70 is 400q matched lists (new `searchbench*` only), not LightGBM on these 170.++**A/B/C call:** US n=74 live default unchanged. Harness best is **0.544**, still not Reddy **0.857**.++### C7 — LightGBM applied from 170 matched hybrid lists (null vs RankNet 0.544; not live)++**Focal question:** on frozen [`search-esci-74-passages-k50`](../../benchmarks/runs/search-esci-74-passages-k50), does LightGBM lambdarank fit on the 170 train hybrid lists beat RankNet apply nDCG@10 **0.544**, with Recall@10 ≥ **0.379** and Recall@50 **0.834** held? Graph off. Frozen `searchbenchesci74` / no wipe. Not live `/retrieve/search`. Do **not** mix with Reddy **0.857** or pool nDCG@20 **0.710**. n=74, no p-values. Horizon nDCG@10 ≥ **0.70** is not a fail of this arm. Predeclared stop: nDCG@10 ≥ **0.58** would halt CE retrain.++**Protocol (located evidence).** `./search.sh ltr-head --from-run search-esci-74-passages-k50 --dataset data/search_esci_74.jsonl --train-from-run search-esci-ltr200-passages-k50 --train-dataset data/search_esci_ltr200.jsonl --ce-model data/models/esci-deberta-v3-base-4class --pair-policy other_query_neg --ltr-model lightgbm --run search-esci-74-ltr-deberta-lgbm-train200`. Same DeBERTa caches as C6. Frozen LightGBM hypers. No hypersearch.++| Arm | nDCG@10 | Recall@10 | Recall@20 | Recall@50 | MRR |+| --- | --- | --- | --- | --- | --- |+| k=50 passages RRF | 0.500 | 0.379 | 0.590 | **0.834** | 0.765 |+| LightGBM 5-fold on the 74 (`search-esci-74-ltr-deberta-lgbm-k50`) | 0.516 | 0.379 | 0.592 | **0.834** | 0.710 |+| RankNet applied from 170 (`search-esci-74-ltr-deberta-train200`) | **0.544** | **0.391** | 0.585 | **0.834** | 0.762 |+| LightGBM applied from 170 (`search-esci-74-ltr-deberta-lgbm-train200`) | 0.533 | 0.389 | 0.591 | **0.834** | 0.752 |+| Must-beat this round | **> 0.544** | **≥ 0.379** | — | hold | — |+| Horizon (not a fail) | **≥ 0.70** | — | — | — | — |++Mean LightGBM gain importances (not a quality claim): `ce_gain` 1684, `dense_inv` 1151, `rrf_inv` 998, `title_overlap` 273, `bm25_inv` 41, `brand_hit` 7, `query_in_title` 0.++**Call (located evidence + decision):** I-LGBM-APPLY **null vs C6 RankNet**. nDCG@10 **0.533 < 0.544**. Recall@10 **0.389 ≥ 0.379**. Recall@50 held. Stop bar 0.58 not reached. Horizon 0.70 missed. Linear vs tree blender on these 170 lists is not the unused 0.33. **Do not wire into `search.py`.** Continue to hybrid-list CE as `ce_gain` (C8).++**A/B/C call:** US n=74 live default unchanged. Harness best remains RankNet apply **0.544**, still not Reddy **0.857**.++### C8 — hybrid-list DeBERTa as `ce_gain` (null vs C6 0.544; not live)++**Focal question:** does continuing the 4-class DeBERTa on **matched hybrid k=50** lists (170 queries, unlabeled=`I`), then using the new scores as `ce_gain` in RankNet apply, beat nDCG@10 **0.544**, with Recall@10 ≥ **0.379** and Recall@50 **0.834** held? Graph off. Frozen `searchbenchesci74` / no wipe. Not live `/retrieve/search`. Do **not** mix with Reddy **0.857**, pool nDCG@20 **0.710**, or CE-as-sorter. n=74, no p-values. Horizon nDCG@10 ≥ **0.70** is not a fail of 8500 pairs.++**Protocol (located evidence).** Export `./search.sh export-hybrid-lists --from-run search-esci-ltr200-passages-k50 --dataset data/search_esci_ltr200.jsonl --holdout-dataset data/search_esci_74.jsonl --out data/esci_hybrid_lists_ltr200.jsonl` (did not clobber `esci_retrieved_lists.jsonl`). 170 queries / 8500 pairs; 0 holdout-qid rows; labels E 984 / S 953 / C 117 / I 6446. Then `./search.sh finetune-4class --dataset data/search_esci_74.jsonl --from-lists data/esci_hybrid_lists_ltr200.jsonl --lists-source hybrid-k50 --base data/models/esci-deberta-v3-base-4class --out data/models/esci-deberta-v3-base-4class-hybrid170 --epochs 1 --max-length 192` (266 steps, MPS; pool checkpoint left in place; `source=hybrid-k50`). Then `./search.sh ltr-head --from-run search-esci-74-passages-k50 --train-from-run search-esci-ltr200-passages-k50 --train-dataset data/search_esci_ltr200.jsonl --ce-model data/models/esci-deberta-v3-base-4class-hybrid170 --pair-policy other_query_neg --run search-esci-74-ltr-deberta-hybrid170`. New caches only (`ce_gain_esci-deberta-v3-base-4class-hybrid170.json`). Lists not sorted by CE.++**Assumption:** 8500 pairs / 170 lists may be too few to retune DeBERTa; catalog shift 4k vs 2043 remains.++| Arm | nDCG@10 | Recall@10 | Recall@20 | Recall@50 | MRR |+| --- | --- | --- | --- | --- | --- |+| k=50 passages RRF | 0.500 | 0.379 | 0.590 | **0.834** | 0.765 |+| RankNet + pool DeBERTa from 170 (`search-esci-74-ltr-deberta-train200`) | **0.544** | **0.391** | 0.585 | **0.834** | 0.762 |+| LightGBM from 170 (`search-esci-74-ltr-deberta-lgbm-train200`) | 0.533 | 0.389 | 0.591 | **0.834** | 0.752 |+| RankNet + hybrid DeBERTa (`search-esci-74-ltr-deberta-hybrid170`) | 0.530 | 0.385 | 0.588 | **0.834** | 0.765 |+| Must-beat this round | **> 0.544** | **≥ 0.379** | — | hold | — |+| Horizon (not a fail) | **≥ 0.70** | — | — | — | — |+| Oracle on these 50 | 0.876 | 0.548 | — | 0.834 | — |++Mean applied RankNet weights (not a quality claim): `ce_gain` 6.92, `title_overlap` 2.29, `rrf_inv` 1.14, `dense_inv` 0.84, `bm25_inv` 0.36, `brand_hit` −0.17, `query_in_title` −0.27.++**Call (located evidence + decision):** I-CE-HYB **null vs C6**. nDCG@10 **0.530 < 0.544**. Recall@10 **0.385 ≥ 0.379**. Recall@50 held. Horizon 0.70 missed. One epoch of hybrid hard-negatives on 170 lists did not unlock the unused 0.33; a null here can mean too little data, not that first-stage negatives never matter. Do **not** relabel a 6k BM25-list run as this construct. **Do not wire into `search.py`.** Default stays hybrid RRF, `rerank=none`.++**A/B/C call:** US n=74 live default unchanged. Harness best remains **0.544**, still not Reddy **0.857**.+diff --git a/docs/research/20-esci-sota-targets.md b/docs/research/20-esci-sota-targets.mdnew file mode 100644index 0000000..fb19c79--- /dev/null+++ b/docs/research/20-esci-sota-targets.md@@ -0,0 +1,350 @@+# 20 — ESCI search-quality SOTA numbers to aim for++Workstream: evaluation protocol for `/retrieve/search` on `searchbench*` only. Ledger: `benchmarks.search` only. This document is **located evidence plus protocol targets**, not a ranking-code change and not a published BrainAPI finding.++**Focal question:** Which published numbers are valid aim points for BrainAPI search quality, given the current 11-query / 200-doc ESCI slice?++Every numeric claim below is labeled **located evidence**, **slice-internal**, or **not-checked**. Do **not** claim the slice is above or below full-ESCI SOTA.++Search date: **2026-08-18**. Tools: harness read; arXiv MCP `search_papers` / `get_abstract` / `download_paper`; AIcrowd/Amazon Science web pages for leaderboard confirmation.++---++## What this workstream does++The eval scores a ranked list from `POST /retrieve/search` against catalog qrels, not LoCoMo judge accuracy and not `/retrieve/recommend`.++### Scoring path (located evidence)++1. Catalog JSONL carries `gold_grades` from ESCI letters via `ESCI_GAINS = {"E": 1.0, "S": 0.1, "C": 0.01, "I": 0.0}` (`benchmarks/search/catalog.py:31`, applied at `catalog.py:298-302`). Complement **is** in the gain table. Irrelevant is dropped (`gain > 0` filter).+2. After ingest, hits are collapsed to `doc_id` (chunk map or node uuid) and deduped (`evaluate.py:172-184`, `267-285`).+3. nDCG@10 is **graded** when `gold_grades` is present (`metrics.py:36-56`; wired at `evaluate.py:285`). Discount is `rel / log2(rank + 1)` (`metrics.py:29-33`) — the classic Järvelin DCG, not the `2^rel − 1` variant.+4. Recall@{5,10,20} is **binary** over the positive-gain set (Exact + Substitute + Complement), not Exact-only (`metrics.py:7-16`; gold union at `evaluate.py:224-245`).+5. MRR is the reciprocal rank of the **first positive-gain** hit (`metrics.py:19-26`). It is not MRR@10 as a cutoff in code; the list is whatever `k` the search returned (default evaluate `k=20`, `evaluate.py:304`).+6. Headline ledger fields are `ndcg@10`, `recall@10`, `mrr`, retrieve p50 (`report.py:65-68`).++### Slice under test (located evidence)++`benchmarks/data/search_esci_slice.jsonl`: 200 docs, 11 queries, US locale, all product columns, brain `searchbenchesci20`. Gold sizes: min 1 / median 20 / max 20. Gain histogram: E=127, S=37, C=2. Passages-only: nDCG@10 **0.758**, Recall@10 **0.511**, MRR **0.848**, p50 retrieve 28–29 ms (`docs/research/19-search-esci-quality.md`).++Later condition, not a substitute: `benchmarks/data/search_esci.jsonl` is 2000 docs / 74 queries (title-heavy; 43/2000 docs lack Brand/Color in text; 0 `class` fields). Gold min/median/max 1/15/38, mean 17.2.++Catalog builder defaults (`catalog.py:34-36, 64-71, 396-397`): locale `us`, split `test`, `small_version == 1` (Task 1 small), caps 80 queries / 2000 docs / 40 candidates per query. The JSONL is then searched as a **shared corpus**, not as a per-query candidate pool.++WANDS download exists in the same harness (`catalog.py:13-32`, gains Exact=1 / Partial=0.5 / Irrelevant=0). No BEIR or TREC Product Search harness in `benchmarks/search/`.++---++## Guarantees and where they break++**Stated guarantee:** graded nDCG@10 / Recall@10 / MRR on this slice measure product-search quality in a way that can be compared to published ESCI numbers.++Where that guarantee cannot be delivered:++1. **Pool size and task (critical).** Published Task 1 is *ranking-in-pool*: reorder ≤40 already-retrieved, fully labeled products per query (`2206.06588` §3.1). We retrieve from a 200-doc (or 2000-doc) shared index. Different candidate generation, different label density, different IDCG. **Gap**, not a bug in the scorer.+2. **Cutoff k (important).** Official Task 1 is nDCG on the provided list. Reddy does not state `@10`. Jeronymo et al. report the KDD Cup metric as **nDCG@20** (`2208.06264`). We always cut at 10 (`metrics.py:121`). Same gain table, different statistic.+3. **Recall@10 / MRR are not the Task 1 metrics (important).** Task 1 publishes nDCG. Tasks 2/3 publish micro-F1. Nobody in the primary ESCI papers reports Recall@10 or MRR as we compute them.+4. **n=11 (critical for inference).** Means have no interval. One total miss (`esci-72`) and one single-gold query (`esci-67`) dominate averages.+5. **Recall@10 ceiling (important).** Mean of `min(10, |gold|) / |gold|` on this slice is **0.667**. Six of eleven queries have `|gold|=20`, so Recall@10 cannot exceed 0.5 on those queries even if every top-10 hit is gold. Current 0.511 is already near that structural cap.+6. **MRR saturation (important).** Mean 0.848 leaves little room except the total miss. Rank-2–10 work will not show up.++Deliberate trade-off: slice size and `skip_enrichment` for ingest cost. Not a claim about full ESCI.++---++## Open questions for the maintainer++1. For a confirmatory ESCI run, is the target **ranking-in-pool nDCG** on the labeled candidate list (comparable to `2206.06588` / KDD Cup), or **first-stage nDCG@10** from a shared ingested corpus?+2. Should the headline cutoff stay nDCG@10, or switch to nDCG@20 / full-list nDCG when the goal is published-number comparability?+3. Should Recall@10 gold stay E+S+C, or Exact-only, given Task 1’s ranking order is E then S then C then I?+4. Is `search_esci.jsonl` (2000 docs, 74 queries) the first larger condition, or should the next run restrict scoring to each query’s original ≤40 labeled candidates?+5. Is p50 retrieve <200 ms still binding if a confirmatory arm uses a cross-encoder the way the published 0.85–0.90 numbers do?++---++## Frontier techniques++### ESCI / Shopping Queries Task 1 ranking-in-pool (canonical)++- **Mechanism:** For each query, reorder a provided list of up to 40 products. Graded nDCG with gains E=1, S=0.1, C=0.01, I=0. Complement is in the gain table. Locales US/ES/JP. Small version (Task 1) filters easier queries; US public test is 4,477 queries, avg depth 20.3 (`2206.06588` Tables 1–4, §3.1).+- **arXiv:** `2206.06588` (Reddy et al., 2022). Amazon Science: [dataset page](https://www.amazon.science/code-and-datasets/shopping-queries-dataset-a-large-scale-esci-benchmark-for-improving-product-search), [KDD Cup recap](https://www.amazon.science/blog/amazon-product-query-competition-draws-more-than-9-200-submissions).+- **Reported numbers (located evidence):**++| Method | Year | Metric | Split | Number | Source |+| --- | --- | --- | --- | --- | --- |+| Terrier BM25 (title, all locales together) | 2022 | nDCG (Task 1, no `@10` in paper) | Public test overall / EN / ES / JP | 0.563 / **0.675** / 0.697 / 0.136 | `2206.06588` Table 4 |+| Fine-tuned MS MARCO MiniLM cross-encoder (EN) + MPNet (ES, JP); Exact→1 else 0 at train time | 2022 | nDCG | Public test overall / EN / ES / JP | **0.852** / **0.857** / 0.849 / 0.840 | `2206.06588` Table 4 |+| Organizer “baseline” cited in the KDD recap | 2022 | nDCG | KDD private/public (unspecified cutoff in blog) | **0.8503** | [Amazon Science blog](https://www.amazon.science/blog/amazon-product-query-competition-draws-more-than-9-200-submissions) |+| Team www (DeBERTa/XLM/RemBERT ensemble; 4-class probs → weighted gain) | 2022 | NDCG (AIcrowd Task 1) | Public / private, multilingual Task 1 | **0.9057 / 0.9043** | `2208.02958`; [AIcrowd winners](https://discourse.aicrowd.com/t/final-winners-announcement/7974) |+| Team qinpersevere / day-day-up | 2022 | NDCG | Private Task 1 | 0.9036 / 0.9035 | AIcrowd winners table |+| mMonoT5-3.7B (NeuralMind) | 2022 | nDCG@20 (their label for the same leaderboard) | Public / private | 0.9012 / 0.9007 | `2208.06264` Table 1 |+| 20th place | 2022 | nDCG@20 | Private | 0.8929 | `2208.06264` Table 1 |+| SQID reproduction of ESCI_baseline on US Task 1 test (8,956 queries, ~20 judgments/query) | 2024 | NDCG (Terrier, corrected S/C mapping) | US small test | **0.8562** | `2405.15190` Table 2 |+| SQID random rank of the same pools | 2024 | NDCG | US small test | 0.7483 | `2405.15190` Table 2 |+| SQID SBERT title cosine (no fine-tune) | 2024 | NDCG | US small test | 0.8292 | `2405.15190` Table 2 |++- **Cost:** Cross-encoder or LLM ensembles over ~20–40 pairs per query. Not a first-stage ANN over a catalog.+- **Fit:** Gain table matches ours. Task, n, pool, and k do not.+- **Verdict:** **adopt** as the label protocol and as the **horizon-B ranking-in-pool target**. **reject** as a numeric bar for the 11-query / 200-doc slice.++### KDD Cup 2022 as “SOTA on ESCI”++- **Mechanism:** Same Task 1, private multilingual test. Top 10 private scores sit in **0.8998–0.9043** (AIcrowd). Jeronymo argues top-20 teams cluster near 0.90 nDCG@20 and that the dataset is too easy to discriminate retrievers (`2208.06264` abstract).+- **Cost:** Heavy fine-tune + ensemble. Wu et al. (`2208.00108`) rank 6th on Task 1 but do not publish a Task 1 nDCG in the paper body (leaderboard 6th is ETS-Lab 0.9014 private; Wu’s team name is not in the top-10 table — **not-checked** which public name maps to their 6th-place claim vs AIcrowd’s ETS-Lab).+- **Verdict:** **adopt** 0.9043 private NDCG as the published ceiling for *ranking-in-pool, multilingual Task 1*. Fashionable only if BrainAPI starts scoring labeled pools with a second-stage ranker.++### Full-catalog ESCI retrieval (BLaIR)++- **Mechanism:** Treat ESCI as product search over **1,367,729 items / 27,643 test queries**; report **NDCG@100** (`2403.03952` Table 3, Table 10). This is first-stage retrieval, not Task 1 reranking.+- **Reported (located evidence):** SFR-Embedding-Mistral **0.2560**; GritLM-7B 0.2537; e5-mistral-7b-instruct 0.2437; Qwen3-Embedding-8B 0.2328; text-embedding-3-large 0.2366; gemini-embedding-001 0.2233 (`2403.03952` Table 10).+- **Cost:** Embed the catalog; ANN. No cross-encoder over 40 candidates.+- **Fit:** Closer to “search a brain” than Task 1, but k=100, corpus ~1.4M, and nDCG@100 ≠ our nDCG@10. Recall@10 / MRR **not reported**.+- **Verdict:** **adapt** as the published bar if a later run is true first-stage over a large catalog. **reject** as a target for the 200-doc slice (our 0.758 lives on a different y-axis).++### Papers that use ESCI but are not SOTA bars here++- **ARR / SToICaL (`2601.05588`, 2026):** Custom in-context ranking, Gecko-mined lists, log-scale gains, n=310, nDCG ~95–97 and a nonstandard R@k (`|π[:k] ∩ π̂[:k]| / k`). **reject** — not ESCI Task 1 gains, not our recall.+- **GraphRAG / EDRM / LoCoMo:** Wrong task. Out of scope.+- **WANDS (ECIR 2022 Chen et al.):** Dataset paper identity **support-located** (2026-08-20): DOI [10.1007/978-3-030-99736-6_9](https://doi.org/10.1007/978-3-030-99736-6_9), OpenAlex `W4225565720`, Springer LNCS, `is_oa=false`, abstract index empty. arXiv title search did not return the chapter. Dataset size **located** on GitHub `wayfair/WANDS`: 480 queries / 42,994 products / 233,448 labels. Published BM25/hybrid **nDCG still not located**. Do not invent a paper nDCG. Medium notebooks are not primary. Local first-stage control is a different y-axis (≤80q / ≤2000 docs / linear Exact=1, Partial=0.5). See [23-search-wands-quality.md](23-search-wands-quality.md).+- **TREC 2023 Product Search (`2311.07861`):** Converts SQD toward end-to-end retrieval. **No harness in this repo.** Leftover.+- **BEIR (`2104.08663`):** No search harness here. Leftover.++---++## Implementation plan++No ranking-code change. Protocol only. Reuse `searchbenchesci20` with `--skip-ingest`. Never wipe brains.++### Phase 1 — Slice-internal aims (this week)++Treat passages-only **0.758 / 0.511 / 0.848** as the control. “Beating the slice” means:++| Metric | Now | Structural ceiling (slice-internal) | What a real win looks like | What is not a win |+| --- | --- | --- | --- | --- |+| nDCG@10 | 0.758 | Not 1.0: several queries have mixed E/S gold; `esci-72` is 0/11 of the mean | +0.02–0.05 with Recall@20 not down; or `esci-72` leaving 0 (alone +~0.09 if that query went to 1.0) | Graph fusion that drops nDCG while MRR stays flat |+| Recall@10 | 0.511 | **0.667** mean of `min(10,\|gold\|)/\|gold\|`; **0.5** on the six `|gold|=20` queries | Move `esci-72` (0 vs ceiling 1.0), `esci-19` (0.333 vs 0.667), `esci-37` (0.667 vs 1.0) | Raising retrieve `k` then cutting @10 |+| MRR | 0.848 | ~0.94 if only `esci-72` is 0 and is fixed to rank-1 | Almost only the total-miss query | Any rank-2–10 cleanup |++**Task 1 — Publish slice ceilings next to every report**++- **Acceptance:** Report or canvas shows Recall@10 ceiling 0.667 and per-query `|gold|`.+- **Verification:** `python` over `search_esci_slice.jsonl` gold sizes (already computed 2026-08-18).+- **Scope:** S. Files: report helper or canvas only.++**Checkpoint:** Humans agree the 0.758/0.511/0.848 triple is a **control**, not a SOTA claim.++### Phase 2 — Comparable confirmatory protocol++Pick one of two designs. Do not mix their numbers.++**Design B-rank (comparable to published 0.85–0.90):** For each query, score **only the original labeled candidates** (≤40), same E/S/C/I gains, report nDCG@20 **and** full-list nDCG, US `small_version` test, n in the thousands if ingest allows, else the 74-query 2000-doc file **restricted to each query’s gold pool**. Cite Reddy EN CE **0.857** as the organizer-style bar and www **0.9043** as the 2022 ceiling.++**Design B-retrieve (comparable to BLaIR, not to 0.90):** Search a shared corpus. Report nDCG@10 plus nDCG@100 if k allows. Cite SFR-Embedding-Mistral **0.256 nDCG@100** on 1.37M items only if the corpus is actually large. The 2000-doc file is still not that condition.++**Task 2 — Ranking-in-pool eval mode (if B-rank is chosen)**++- **Acceptance:** Harness can restrict hits to `gold_grades` keys plus labeled irrelevants for that `qid`; metrics include nDCG@20.+- **Verification:** One `--skip-ingest` run on a toy query with a 40-id pool; nDCG matches a manual Terrier-style recompute.+- **Scope:** M. Files: `benchmarks/search/evaluate.py`, `metrics.py`, `report.py`.+- **Depends:** Maintainer answer to open question 1.++**Task 3 — 2000-doc condition as a separate row**++- **Acceptance:** `search_esci.jsonl` run is labeled `dataset=search_esci.jsonl`, `n_queries=74`, never averaged with the 11-query slice.+- **Verification:** `./search.sh evaluate --dataset data/search_esci.jsonl --brain searchbenchesci --skip-ingest` (or a new `searchbench*` brain — no wipe of `searchbenchesci20`).+- **Scope:** S. Eval only.++**Checkpoint:** A table with three rows that are never compared as one number: slice n=11, 2000-doc retrieve, ranking-in-pool (if implemented).++### Measured n=11 ranking-in-pool smoke (2026-08-18)++Not a published-number claim. Rows must not be averaged.++| Row | nDCG@10 | nDCG@20 | full nDCG | n | Cite 0.857? |+| --- | --- | --- | --- | --- | --- |+| Slice first-stage passages | 0.758 | — | — | 11 | no |+| BrainAPI post-filter in-pool | 0.788 | 0.811 | 0.814 | 11 | no |+| MiniLM CE-on-pool (zero-shot MS MARCO) | 0.598 | 0.669 | 0.786 | 11 | **only this arm’s protocol**, still not the Reddy split |++Harness: `--rank-pool` on evaluate; `rank-pool-ce` subcommand. Catalog persists `candidate_grades` including I=0. `searchbenchesci20` was not re-ingested.++### 74-query brain `searchbenchesci74` (2026-08-18)++New brain; dataset `search_esci_74.jsonl` (74 queries, 2043 pool products, mean 28.8 labeled candidates). Passages ingest onto this brain is in progress and is **not** required for CE-on-pool.++Cite Reddy **0.857** only against CE-on-pool. n=74 is still not the 4,477-query public test.++| Arm | nDCG@20 | full nDCG | n | vs 0.857 |+| --- | --- | --- | --- | --- |+| Zero-shot MS MARCO MiniLM-L-6 | 0.631 | 0.717 | 74 | below |+| Fine-tuned 1 epoch, 30k US train pairs, Exact→1 else 0, test qids held out | 0.669 | 0.745 | 74 | below |++**Located evidence:** one-epoch Exact-vs-rest fine-tune lifted nDCG@20 +0.038 vs zero-shot and did not beat Reddy 0.857 / KDD 0.9043 (those use full Task 1 and, for the winner, ensembles).++### 74-query CE-on-pool and first-stage (2026-08-18, this workstream)++Do not average with n=11 or with Reddy n≈4477. Cite **0.857 only against CE-on-pool**.++| Arm | n | nDCG@10 | nDCG@20 | full nDCG | notes |+| --- | --- | --- | --- | --- | --- |+| Live passages skip-ingest (control) | 74 | **0.495** | 0.549 | 0.549 | R@10 0.377, R@20 0.592, MRR 0.758, p50 62 ms; `search-esci-74-passages-control` |+| Local BM25 all-text | 74 | 0.424 | 0.472 | — | harness JSONL; not BrainAPI |+| Local BM25 title-boost | 74 | 0.458 | 0.508 | — | best BM25 variant; **below** 0.495 |+| Local SPLADE-cocondenser | 74 | 0.465 | 0.520 | — | `naver/splade-cocondenser-ensembledistil`; **below** 0.495 |+| CE-on-pool FT Exact→1 MiniLM-L-6 30k | 74 | — | 0.669 | 0.745 | prior row |+| CE-on-pool 4-class MiniLM-L-12, 40k US train, catalog fields, score 1·P(E)+0.1·P(S)+0.01·P(C) | 74 | 0.597 | **0.678** | **0.747** | `esci-minilm-l12-4class`; **> 0.669 progress**; **not** 0.857 |++**Calls:** Reddy EN CE **0.857 was not beaten**. First-stage nDCG@10 did **not** beat 0.495 by ≥0.02 (live control matched 0.495; local BM25/SPLADE were lower). n=74 is still not the public test.++### 74-query 4-class weighted CE (2026-08-18, isolated workstream)++Ranking-in-pool only (I=0 stays in the list). Dataset `search_esci_74.jsonl`. Brain `searchbenchesci74` was **not** wiped or re-ingested. Did **not** repeat Exact→1 MiniLM-L-6. Cite **0.857 only against this protocol**; n=74 is not Reddy’s ~4,477-query EN public test.++**Chosen path:** team www (`2208.02958`) 4-class softmax → score `1.0 P(E)+0.1 P(S)+0.01 P(C)+0.0 P(I)`, backbone `cross-encoder/ms-marco-MiniLM-L-12-v2` (Reddy footnote 6, different **loss** from sibling Exact→1). New files: `benchmarks/search/finetune_esci_4class.py`, `benchmarks/search/rank_pool_4class.py`. Ledger runs under `benchmarks.search` only.++**Rejected (same lookup):** Reddy Exact→1 MiniLM (sibling already running that replica). Jeronymo mMonoT5-3.7B (`2208.06264`) 72 h TPU. www ensemble / DeBERTa-v3-large / RemBERT (0.90 is ensemble). SQID unfine-tuned title SBERT 0.8292 (`2405.15190` Table 2) already below 0.857. Wu (`2208.00108`) DeBERTaV3 + LightGBM group features. Pool-only ColBERT: extra infra, not a published 0.85–0.90 recipe we could finish in hours. `sentencepiece` is now in the parent venv (0.2.2); a **v3-base** 4-class slice was run 2026-08-19 (pool nDCG@20 **0.710**, still below 0.857; first-stage catalog R@10 **null**). v3-large / ensemble remain out of local MPS this pass.++**Critique (why 0.857/0.90 may be unreachable with this small CE on n=74):** construct-valid ranking-in-pool + ESCI gains. External validity fails: n=74 vs ~4477 (no interval). Training used 80k/419,653 US small-train pairs, not full KDD train+aug+ensemble. Inverse-freq class weights (C weight 5.53) **hurt** nDCG: on Exact golds mean P(I)>P(E). Unweighted 4-class (www-like) restored E>S>C>I in expected gain. Second epoch +0.006 only. 0.90 remains ensemble/3.7B.++| Arm | nDCG@20 | full nDCG | n | vs 0.669 / 0.857 |+| --- | --- | --- | --- | --- |+| Inverse-freq 4-class, 80k, 1 ep (`search-esci-74-ce-pool-4class`, `esci-minilm-l12-4class`) | 0.670 | 0.742 | 74 | ~tied / below |+| Unweighted 4-class, 80k, 1 ep (`search-esci-74-ce-pool-4class-nowt`) | 0.688 | 0.759 | 74 | **+0.019** / below |+| Unweighted 4-class, 80k, 2 ep (`search-esci-74-ce-pool-4class-e2`) | 0.695 | 0.765 | 74 | **+0.026** / below |+| Unweighted 4-class DeBERTa-v3-base, 80k, 2 ep (`search-esci-74-ce-pool-deberta-base`) | **0.710** | **0.777** | 74 | **+0.041** / below |++**Call:** Reddy EN CE **0.857 was not beaten**. Progress vs our Exact→1 FT 0.669: yes (best MiniLM 0.695; DeBERTa-v3-base **0.710** on the same 74 pools). On-disk `esci-minilm-l12-4class` is this workstream’s 80k inverse-freq checkpoint, not the 40k row above. Do not average pool **0.710** with first-stage nDCG@10.++### Deeper k then harness CE (2026-08-18, not 0.857)++Separate y-axis from the CE-on-pool rows above. First-stage retrieve from the 2043-doc shared corpus, then (Phase 2) reorder **retrieved** `hit_ids` only. Graph off. Default API `k` / `RERANK_MAX_K=10` unchanged. Cite **0.857 only against CE-on-pool**; do not average with 0.695.++**Phase 1 (located evidence).** Predeclared win: pool_coverage ≥ 0.57 **or** Recall@50 ≥ 0.65.++| Arm | k | Recall@50 | pool_coverage | nDCG@10 | p50 ms |+| --- | --- | --- | --- | --- | --- |+| Passages control | 20 | — | 0.517 (`--rank-pool`) | **0.495** | 62 |+| `search-esci-74-passages-k50` | 50 | **0.834** | — | 0.500 | 60 |+| `search-esci-74-passages-k50-pool` | 50 | 0.832 | **0.778** | 0.567 | 60 |+| `search-esci-74-passages-k100` | 100 | 0.831 | — | 0.500 | 58 |+| `search-esci-74-passages-k100-pool` | 100 | 0.906 | **0.846** | 0.574 | 59 |++**Phase 1 call:** win (Recall@50 0.834; coverage 0.778). nDCG@10 stayed flat, as predicted: extra golds are below rank 10. `missing_from_brain` stayed 0. k=100 does not raise Recall@50 vs k=50.++**Phase 2 (located evidence).** Harness `rerank-retrieved` on `search-esci-74-passages-k50`. Win needed nDCG@10 ≥ 0.515 with Recall@50 not down.++| Arm | nDCG@10 | Recall@50 |+| --- | --- | --- |+| k=50 first-stage | 0.500 | 0.834 |+| MiniLM-L-6 on retrieved hits | 0.448 | 0.834 |+| 4-class L-12 e2 on retrieved hits | 0.467 | 0.834 |++**Phase 2 call:** null. Both CEs moved nDCG@10 **down**. Same hit set, so Recall@50 is unchanged. Do not treat this as a Reddy comparison.++### Dual-k matching (2026-08-18, not 0.857)++Goal: more gold in **both** Recall@10 and Recall@50. Predeclared win: ≥0.397 and ≥0.854. Sidecar / skip-ingest only. `searchbenchesci74` not wiped.++**Task 1 located evidence.** k=50 golds: 424 in top-10, 728 in ranks 11–50, 118 missed (5 total-miss queries). Counts from `miss_strata.json` on `search-esci-74-passages-k50`.++**Task 2 located evidence.** Rewrote `esci-113` and `esci-267` only; skipped `esci-72`. Live k=50 qrewrite Recall@10 **0.379** / Recall@50 **0.834** (unchanged). Those two qids stayed at 0.++**Task 3 located evidence.** Local ANCE MiniLM (`search-esci-74-dense-ance-k50`): Recall@10 **0.302**, Recall@50 **0.750**, nDCG@10 **0.396**. Dual win **failed**. Not BrainAPI embeddings; not 0.857.++**Call:** no first-stage matching arm beat passages on both cutoffs. Stop encoder fishing on this isolation set. Do not fuse the sidecar. Do not mix with CE-on-pool 0.695.++### Retrieved-neg CE, ColBERT, BGE (2026-08-18, not 0.857)++Separate y-axis from CE-on-pool **0.695** / Reddy **0.857**. Frozen `searchbenchesci74`. Graph off. `RERANK_MAX_K=10` unchanged. n=74.++| Arm | protocol | nDCG@10 | Recall@10 | Recall@50 | vs gate |+| --- | --- | --- | --- | --- | --- |+| k=50 passages | hybrid RRF | **0.500** | 0.379 | **0.834** | control |+| retrieved-neg 4-class L-12 | CE on stored k=50 hits | 0.465 | 0.351 | 0.834 | nDCG@10 ≥0.515 **failed**; R@50 held |+| ColBERT MaxSim sidecar | local JSONL, not fused | 0.434 | 0.311 | 0.714 | dual-k **failed** |+| BGE-base (not MiniLM) | local JSONL, no `query:` | 0.441 | 0.322 | 0.799 | dual-k **failed** |++ColBERT p50 encode+retrieve **10063 ms** (200 ms labeled, not a stop). BGE p50 **408 ms**. MiniLM ANCE was **0.302 / 0.750**, not this BGE row.++**Call:** all three null vs their predeclared gates. Passages hybrid remains the first-stage control. Do not average with 0.695.++### Exhaustive catalog then two-stage (2026-08-19, not 0.857)++Separate y-axis from CE-on-pool **0.695** / Reddy **0.857**. Frozen `searchbenchesci74`. Graph off. Default `RERANK_MAX_K=10` unchanged. n=74. Cite **0.857 only against CE-on-pool**.++| Arm | protocol | nDCG@10 | Recall@10 | Recall@50 | vs gate |+| --- | --- | --- | --- | --- | --- |+| k=50 passages | hybrid RRF | **0.500** | 0.379 | **0.834** | control |+| Exhaustive 4-class L-12 e2 | exhaustive-catalog, 2043 docs | 0.416 | 0.266 | 0.603 | nDCG@10 ≥0.515 **and** R@10 ≥0.397 **failed** |+| Union passages+BGE+ColBERT | harness-union RRF | 0.479 | 0.363 | 0.827 | dual-k **failed**; R@10 diluted below 0.379 |+| `mode=catalog` + plugin CE | live two-stage, rerank 50 | 0.467 | 0.341 | **0.834** | nDCG@10 ≥0.515 **failed**; R@50 held |+| Frozen-head cascade | harness; hybrid top-10 frozen | **0.500** | **0.379** | **0.889** | R@10 held **and** R@50 ≥0.854 **win** |++U01: BGE top-50 held **40** gold ASINs absent from passages k=50 (19 qids); ColBERT **21** (13 qids). Unique golds were nonzero, so the union included ColBERT. Cascade injected **51** unique golds (set union) into ranks 11–50. Exhaustive p50 **94 s** is labeled, not a 200 ms claim. Catalog p50 retrieve **59 ms**; p50 client wall **877 ms** (not the ADR-007 default).++**Call:** A/B quality null; C ships the opt-in hook; frozen-head cascade is the first dual-k **win** on this isolation set. Do not average with 0.695. Do not treat C’s 0.467 as a Reddy comparison. Cascade is harness-only until a live fill-tail path is an explicit product decision.++### DeBERTa-v3-base 4-class catalog slice (2026-08-19, not 0.857)++C06 was previously “out of compute this pass.” This pass ran a **single** `microsoft/deberta-v3-base` 4-class CE (not v3-large, not www ensemble). Same 80k/2ep/unweighted recipe as MiniLM nowt-e2. Frozen `searchbenchesci74` not wiped. Default `RERANK_MAX_K=10` unchanged. n=74, no p-values.++**Ranking-in-pool (separate y-axis).** `search-esci-74-ce-pool-deberta-base` nDCG@20 **0.710** vs MiniLM e2 **0.695**. Still below Reddy **0.857**. n=74 ≠ ~4477.++**First-stage (R@10 gate).** Harness `search-esci-74-passages-k50-ce-deberta` on stored k=50 hits:++| Arm | protocol | nDCG@10 | Recall@10 | Recall@50 | vs gate |+| --- | --- | --- | --- | --- | --- |+| k=50 passages | hybrid RRF | **0.500** | **0.379** | **0.834** | control |+| MiniLM 4-class e2 on k=50 hits | CE on stored hits | 0.467 | 0.341 | **0.834** | nDCG@10 ≥0.515 **failed** |+| DeBERTa-v3-base 4-class on k=50 hits | CE on stored hits | 0.510 | 0.363 | **0.834** | R@10 ≥0.397 **and** nDCG@10 ≥0.515 **failed** |++**Call:** first-stage **null**. Recall@10 **0.363** missed **0.397** (below passages). nDCG@10 **0.510** missed **0.515**. Recall@50 held. Live `mode=catalog` skipped (gates miss; MiniLM catalog already matched harness). Cascade remains the Recall@50 result. Do not average with pool **0.710** / MiniLM **0.695** / Reddy **0.857**. v3-large / ensemble still out.++### Phase 3 — Optional adjacent harness++WANDS first-stage control is a **separate y-axis** from ESCI n=74. Chen et al. identity located (DOI `10.1007/978-3-030-99736-6_9`); published nDCG **not located**. Do not set a Chen-paper nDCG aim. Do not mix with ESCI 0.500 / C6 0.544 / Reddy 0.857. BEIR / TREC Product Search: do not add.++---++## Risks++| Risk | Detection |+| --- | --- |+| Quoting 0.9043 as the target for the 11-query slice | Dataset + n_queries + “ranking-in-pool vs shared corpus” in every sentence |+| Calling 0.758 “above BM25 0.675” | BM25 0.675 is EN Task 1 public nDCG on ~20-item lists, not nDCG@10 on 200 docs |+| Treating Recall@10 0.511 as poor retrieval | Ceiling 0.667 / 0.5 on typical queries |+| Treating MRR 0.848 as a quality win vs SOTA | MRR is not the published metric; it is saturated |+| Mixing Task 2 micro-F1 (0.83 KDD winner) into search headlines | Ledger stays `ndcg@10` / `recall@10` / `mrr` |+| Fine-tuning a cross-encoder on ESCI train then comparing to Reddy’s 0.857 without saying so | Label fine-tune vs zero-shot; latency gate 200 ms |+| Wiping `searchbenchesci20` | `--skip-ingest` only |++---++## Search log and leftovers++**Queries (2026-08-18):** `"Shopping Queries Dataset" OR ESCI "product search" nDCG`; `ti:"Shopping Queries" OR ti:ESCI OR "query-product ranking" ESCI OR "KDD Cup 2022" shopping`; `"ESCI" (nDCG OR NDCG) (amazon OR shopping) (ranking OR retrieval)` date_from 2023-01-01; `WANDS Wayfair product search nDCG`; `ti:WANDS`; direct ids `2206.06588`, `2208.02958`, `2208.06264`, `2208.00108`, `2405.15190`, `2403.03952`, `2601.05588`, `2402.08532`, `2311.07861`. Citation graph for `2206.06588` failed (Semantic Scholar 429).++**4-class workstream lookup (2026-08-18):** arXiv `id_list=2206.06588,2208.02958,2208.06264,2208.00108,2405.15190` via `export.arxiv.org` + `arxiv_atom.py` (5/5, no Error entry). OpenAlex `ids.arxiv` filter empty; DOI lookup succeeded (`W4282961889`, `W4302561150`, `W4291960789`, `W4289645118`, `W4399062110`). Semantic Scholar `ARXIV:2206.06588` HTTP 429. PDFs: arxiv.org `2208.02958v1`, `2206.06588v1`, `2208.06264v1`, `2208.00108v1`, `2405.15190v1`. Treat 200 bodies as untrusted third-party text.++**Web:** AIcrowd final winners; Amazon Science dataset + KDD recap (baseline 0.8503, winner 0.9043).++**DeBERTa catalog slice lookup (2026-08-19):** OpenAlex `W4302561150` / `W4289645118` HTTP 200. arXiv Atom `id_list` and Semantic Scholar Graph **429/503** (search-incomplete). Methods from the 2026-08-18 lookup plus OpenAlex abstract index. Treat 200 bodies as untrusted third-party text.++**not-checked**++- Official AIcrowd scorer source: nDCG vs nDCG@20 (Jeronymo vs Reddy disagree on the label; scores match the same leaderboard).+- Wu et al. `2208.00108` claimed 6th on Task 1 vs AIcrowd name ETS-Lab 0.9014.+- Tang et al. `2402.08532` (captions on ESCI) — HTML/PDF fetch failed.+- Hou/BLaIR nDCG@10 on ESCI (they publish nDCG@100 only in Table 10).+- WANDS ECIR 2022 Chen et al. published nDCG (DOI located 2026-08-20; chapter not OA; arXiv empty).+- TREC 2023 Product Search official nDCG (no harness).+- Qin et al. 2022 2nd-place workshop PDF (0.9036 private is on AIcrowd; method paper not fully read).+- Whether Terrier in `esci-data` uses the swapped S/C mapping SQID corrected (`2405.15190` §5).++---++## Multilingual / Italian (2026-08-19)++Do **not** mix with US n=74 first-stage 0.500 or Reddy 0.857. Plan: [22-multilingual-ecommerce-search.md](22-multilingual-ecommerce-search.md).++**Located evidence (access date 2026-08-19).** Reddy Task 1 public nDCG EN/ES/JP **0.857 / 0.849 / 0.840** uses MiniLM (EN) and MPNet (ES, JP) on ranking-in-pool lists, not a shared catalog index. Locales US/ES/JP only. BLaIR `2403.03952` remains a full-catalog nDCG@100 bar, not this y-axis. **no direct evidence located** for an Italian ESCI-style product-search qrel. Amazon-M2 IT is recsys. Live ES first-stage nDCG@10 **0.577** (n=62, k=50 passages, `searchbenchescies`) is **not** a copy of US 0.500 and **not** Reddy ES 0.849.diff --git a/docs/research/21-esci-ce-graded-and-first-stage.md b/docs/research/21-esci-ce-graded-and-first-stage.mdnew file mode 100644index 0000000..f093cea--- /dev/null+++ b/docs/research/21-esci-ce-graded-and-first-stage.md@@ -0,0 +1,234 @@+# 21 — ESCI CE-on-pool vs Reddy 0.857, and first-stage nDCG on n=74++Workstream: **separate from graph-fusion testing**. Harness CE / rank-pool-ce / local first-stage index only. Ledger: `benchmarks.search` only. Brains: `searchbench*` only; no wipe of `searchbenchesci20` / `searchbenchesci74`.++**Focal questions**++1. Can a MiniLM-scale CE, trained and scored on labeled pools, beat **our FT 0.669 nDCG@20** and, on a comparable protocol only, Reddy EN CE **0.857**?+2. Can a first-stage intervention raise nDCG@10 on the shared 2043-doc `search_esci_74.jsonl` corpus by **≥0.02** vs **0.495**, without mixing y-axes with 0.857?++n=74 is **not** Reddy’s ~4477-query US public test. Do not claim full-ESCI SOTA.++Search date: **2026-08-18**. Origin of ideas: AI-assisted. Stage labels follow the scientific-brainstorming skill.++---++## Scope++- Purpose: pick 1–2 harness interventions; measure them.+- In: `finetune_esci_ce.py`, `rank_pool.py`, `plugins/search-rerank/`, new harness scripts, this note plus append-only claims in `19` / `20`.+- Out: `search.py`, `hybrid.py`, `fact_filter.py`, `graph_channels.py`, `replay_fusion.py`, `evaluate.py` instrumentation, fusion unit tests, production `RERANK_MAX_K`, default `channels=["passages"]`, default rerank.+- Constraints: `--brain` on parent `./search.sh`; `BRAINPAT_TOKEN` never printed; this checkout not `~/.brainapi/source`.++---++## Independent ideas (before literature)++Stage `independent`. Contributor: this workstream. Origin: AI-assisted.++### CE-on-pool (ranking-in-pool)++| ID | Statement | Assumption | Prediction | Disconfirm |+| --- | --- | --- | --- | --- |+| C01 | Train with nDCG gains (E=1, S=0.1, C=0.01, I=0) instead of Exact→1 else 0 | Binary labels collapse S and I, which nDCG still separates | nDCG@20 > 0.669 on the same 74 pools | Graded MSE ≤ 0.669 |+| C02 | 4-class CE, score = 1·P(E)+0.1·P(S)+0.01·P(C) | Softmax over ESCI matches Task 1 gains better than a single logit | nDCG@20 > 0.669; still < 0.857 on MiniLM / n=74 | 4-class ≤ graded or ≤ binary FT |+| C03 | More US Task-1 train pairs (hold out test qids) | 30k is under-capacity vs ~420k US train judgements | Modest lift vs 30k, diminishing after tens of thousands | Extra pairs do not move nDCG@20 |+| C04 | MiniLM-L-12-v2 (Reddy’s EN backbone) | L-6 is the main capacity gap vs 0.857 | Lift vs L-6; still short of 0.857 | L-12 ≈ L-6 on n=74 |+| C05 | Concatenate title+brand+color+bullets+description | Reddy used titles; KDD-style fields add attributes | Helps attribute queries; can add noise | Catalog text ≤ title-only |+| C06 | Ensemble DeBERTa / XLM / RemBERT | Winner margin is ensemble + size | Could approach 0.90 on full Task 1 | v3-base catalog slice run 2026-08-19; large/ensemble still out |+| C07 | monoT5 / mT5 generative ranker | Jeronymo: 580M mMARCO z.s. 0.864, 3.7B FT ~0.90 | Beats MiniLM | Too large for this harness |+| C08 | Distill a large CE into MiniLM | Teacher signal > binary ESCI labels | Lift at MiniLM cost | Distillation data/time not available |+| C09 | Listwise / LambdaRank on the pool | Metric-aware listwise fits nDCG | Lift vs pairwise CE | Needs list construction; easy to overfit n=74 |+| C10 | Query expansion before CE | CE already sees the full pool | Null on CE-on-pool (coverage 1.0) | Any CE lift from expansion |++### First-stage (shared 2043-doc index; not 0.857)++| ID | Statement | Assumption | Prediction | Disconfirm |+| --- | --- | --- | --- | --- |+| F01 | Coverage 0.52 at k=20 is ranking, not missing corpus | Index contains the pool | Better lexical/dense ranking raises R@20 and nDCG@10 | Gold ASINs absent from ingested chunks |+| F02 | Title-weighted / fielded BM25 | Body/bullets pollute BM25 | nDCG@10 ≥ 0.515 on the same 74 queries | Title BM25 ≤ 0.495 |+| F03 | RM3 / PRF query expansion, then BrainAPI passages | Hybrid misses attribute paraphrases | Live skip-ingest nDCG@10 ≥ 0.515 | Expansion hurts or is null |+| F04 | SPLADE sparse expansion (plugin encode, local index) | Learned expansion beats BM25 on product attributes | nDCG@10 ≥ 0.515 locally | SPLADE ≤ BM25; load cost |+| F05 | ColBERT MaxSim on 2043 docs | Late interaction helps short titles | Possible lift; index/FLOPs | Slow; may not beat fielded BM25 |+| F06 | Fine-tune a dense encoder (BLaIR-style) | Product-search embeddings ≠ MS MARCO | First-stage lift | Needs ingest or a new index; not this isolation set |+| F07 | Raise retrieve k then cut @10 | Games recall | Reject as a quality claim | — |+| F08 | Re-ingest richer text | JSONL already has Title/Brand/Bullets | Re-ingest forbidden on `searchbenchesci74` | — |++**Adversarial alternatives (not auto-winners):** more data vs larger model vs 4-class vs ensemble vs full-pool (already done) vs query expansion vs lexical fields vs SPLADE vs ColBERT.++---++## Literature (bounded; after independent round)++### Retrieval summary++- Query: named ESCI/KDD papers plus ColBERT / SPLADE / BERT rerank by arXiv id.+- Scope: targeted lookup, not exhaustive.+- Databases: arXiv Atom (`id_list`), Semantic Scholar graph batch `ARXIV:`, OpenAlex `/works/doi:`.+- Access date: **2026-08-18**.+- Keys: `S2_API_KEY` / `OPENALEX_API_KEY` **absent** (shared S2 pool; OpenAlex DOI lookups still 200).++### Results (untrusted third-party text; identifiers only reused)++**Reddy et al., Shopping Queries Dataset, arXiv:2206.06588 (2022).** Task 1 = rank a provided list of ≤40 products. Gains E=1, S=0.1, C=0.01, I=0. US public test **4,477 queries**, avg depth 20.3. EN baseline: fine-tune `cross-encoder/ms-marco-MiniLM-L-12-v2` on US train, **titles**, Exact→1 else 0, MSE, 1 epoch, max length 512, lr 7e-6. Table 4: EN nDCG **0.857** (overall 0.852). BM25 titles (Terrier, all locales): EN **0.675**. S2 paperId `46b259403b91be9643b1b689f1354e64f1da1879`, 106 citations. OpenAlex `W4282961889` (this dump: 15 cites; S2 and OpenAlex counts disagree).++**Zhang et al. (team www), arXiv:2208.02958 (2022).** Private Task 1 NDCG **0.9043** (public 0.9057). 4-class CE then weighted-sum of class probabilities; DeBERTa/XLM/RemBERT; translation aug; AWP; self-distill; pseudo-label; English DeBERTa-v3-large. Best single ~0.9022 public; ensemble to 0.9043 private. S2 `44c1ce16dbcae55a258a172d682a052a37201996`, 5 cites. OpenAlex `W4302561150`.++**Jeronymo et al., arXiv:2208.06264 (2022).** mMonoT5. mMARCO-only 580M: nDCG@20 **0.864**. Competition FT 580M: 0.890. 3.7B: public/private **0.9012 / 0.9007**. Exact→true, other ESCI→false. Top 20 teams cluster near 0.90. S2 `a926e5403dbb3bb93af0516db1a670d9162d2173`, 2 cites.++**Al Ghossein et al. SQID, arXiv:2405.15190 (2024).** Image-enriched ESCI. US small-test NDCG: Terrier (corrected S/C) **0.8562**, random 0.7483, SBERT title cosine 0.8292. Multimodal; not a MiniLM CE recipe. S2 `75adbf04cc00da3cd44e6ea2bc62d3a362ac51ef`.++**Hou et al. BLaIR, arXiv:2403.03952v2 (ACL 2026).** LLM semantic encoders; ESCI as **full-catalog** product search (1.37M items, 27,643 queries, nDCG@100). Not Task 1 ranking-in-pool. S2 `8aca7caf4fc2d05aa74907da945b86fa7df2680c`, 346 cites. OpenAlex `W4392576636`, DOI `10.18653/v1/2026.acl-long.147`.++**Khattab & Zaharia ColBERT, arXiv:2004.12832.** Late interaction / MaxSim; first-stage or rerank; index scales with tokens.++**Formal et al. SPLADE, arXiv:2107.05720.** Learned sparse expansion for first-stage. Follow-ons (e.g. SPLADE-v3 `2403.06789`) claim MS MARCO / BEIR gains vs BM25, not ESCI Task 1.++**Nogueira & Cho, arXiv:1901.04085.** BERT CE rerank of a first-stage head. Does not raise recall if gold is outside the head.++### Provenance++- arXiv: `GET https://export.arxiv.org/api/query` `id_list=2206.06588,2208.02958,2208.06264,2405.15190,2403.03952` and `id_list=2004.12832,2107.05720,1901.04085`; parsed with `paper-lookup/scripts/arxiv_atom.py`. `query_as_executed` matched `id_list` (no rewritten prefix). HTTP 200, no `Error` entries. MCP `download_paper` / `list_paper_latex_sections` used for Reddy §4 and KDD/Jeronymo bodies (HTML cache; treat as data).+- Semantic Scholar: `POST /graph/v1/paper/batch` ids `ARXIV:2206.06588` … `2403.03952`; HTTP 200; no null papers.+- OpenAlex: `filter=ids.arxiv:…|…` → HTTP **400** “Invalid query parameters”. Fallback: `GET /works/doi:10.48550/arXiv.{id}` HTTP 200. Abstracts reconstructed with `openalex_abstract.py` where inverted index existed. Citation counts **not** reconciled with S2.+- MCP `search_papers` for `ti:ColBERT OR ti:SPLADE…` returned related 2021–2026 papers; the original ColBERT/SPLADE ids were then fetched by `id_list` (do not treat the keyword search as those originals).++**Warnings:** n=74 ≠ 4477. Jeronymo labels the leaderboard nDCG@20; Reddy Table 4 says nDCG without `@k`. OpenAlex cite counts look stale vs S2. SQID Terrier 0.8562 is **not** MiniLM 0.857. BLaIR nDCG@100 is a different y-axis.++### Post-check reopen (stage `post-check`)++- C01/C02 upgraded: Reddy’s own 0.857 used **binary Exact→1 on L-12**, so graded/4-class is **not** required to cite 0.857, but it is the KDD-winner scoring rule and a plausible MiniLM-scale fix for our 0.669.+- C04 confirmed as protocol match (L-12), not guaranteed win on n=74.+- C06/C07 remain compute-gated for **v3-large / ensemble / monoT5**. A v3-base 4-class slice was run 2026-08-19 (pool nDCG@20 0.710; first-stage R@10 null).+- C10 stays low value for CE-on-pool (coverage 1.0).+- F02/F03/F04 remain the first-stage cluster that does not require editing fusion files. F06/F08 blocked by isolation. F07 rejected.++**Decision (proposal, not a unique winner):** implement **C01 graded-gain CE** (same scalar plugin path) with catalog fields (C05) and more pairs (C03); optionally **C02** if plugin scoring is cheap. For first-stage, **F02 fielded BM25** locally on the 2043 texts, then **F03 RM3 → live skip-ingest** if the API is healthy. Do not auto-declare 0.857 reachable.++---++## Critique of 0.857 comparability and this plan++### Summary++The published 0.857 is organizer EN Task 1 ranking-in-pool nDCG on ~4477 public-test queries, MiniLM-L-12, titles, Exact→1 MSE, 1 epoch. Our comparable arm is CE-on-pool on **74** US Task-1-style pools (2043 products, mean ~29 candidates). First-stage 0.495/0.549 is a **different estimand**.++### Strengths++- Gain table already matches Reddy §3.1 (`ESCI_GAINS`).+- CE-on-pool coverage 1.0; test qids held out of FT.+- Slice n=11 and n=74 are labeled separately in `20-esci-sota-targets.md`.++### Concerns++**Critical**++- **n=74 vs 4477.** No interval. A 0.857 miss (or hit) on 74 queries is not a replication. Small-n mean is sensitive to a few qrels (statistical-pitfalls: underpowered comparison; do not treat a 0.02 delta as confirmatory).+- **Wrong y-axis if 0.857 is cited vs first-stage or `--rank-pool` BrainAPI (coverage 0.517).** That would be a construct-validity failure, not a ranking win.+- **Selection of “hard” Task 1 small queries.** Reddy’s small split filters easier NDCG queries. Our 74 is a cap from the catalog builder, not the official public test. Even a CE above 0.857 here would not license “we beat Reddy.”++**Important**++- **Architecture mismatch:** our FT is MiniLM-**L-6**, 30k pairs, titles. Reddy is L-12, full US train (~419k judgements). Gap 0.669 vs 0.857 is over-determined (model, data, n, split).+- **Label function mismatch vs KDD 0.9043:** winner used 4-class probabilities; Reddy used binary. Our Exact→1 FT copied Reddy’s label map, not the winner’s.+- **Multiple arms without a predeclared primary** inflate false wins (multiplicity). Predeclare: CE progress = nDCG@20 **> 0.669**; Reddy bar = **0.857** on CE-on-pool only; first-stage win = nDCG@10 **≥ 0.515** on skip-ingest passages or an explicitly labeled local first-stage on the same 74 queries.+- **HARKing risk** if we train until n=74 looks good. One graded run + optional 4-class; stop if neither beats 0.669.++**Minor**++- Full-list nDCG vs nDCG@20: report both; cite 0.857 only next to nDCG@20 / full-list pool nDCG, never @10 first-stage.+- SQID 0.8562 Terrier is a different system than Reddy CE 0.857.++### Overall++It is **reasonable** to try to beat 0.857 on this 74-query pool task as an **optimistic smoke**. It is **not** reasonable to treat success or failure as a statement about the public test. MiniLM-L-6 + n=74 makes 0.857 **unlikely**; say so unless evidence appears.++---++## Tasks (win / stop)++### Task 1: Graded-gain CE trainer (C01, C03, C05)++**Acceptance:** `finetune_esci_ce.py` can emit Exact→1 **or** ESCI-gain labels; catalog passage text; hold out test qids; save under `benchmarks/data/models/`.++**Win:** code path runs; pair counts logged. **Stop:** parquet missing.++### Task 2: Rank-pool-ce scores scalar FT (and 4-class if present)++**Acceptance:** `SEARCH_RERANK_MODEL=…` + `rank-pool-ce` on `search_esci_74.jsonl` writes nDCG@20 and full nDCG. Plugin maps 4-logit models to gain-weighted scores without changing production default model.++**Win for CE:** nDCG@20 **> 0.669**. **Reddy bar:** **> 0.857** (may fail). **Stop:** ≤ 0.669 after one graded run (optional one 4-class retry, then stop).++### Task 3: Fielded BM25 first-stage on the 2043-doc JSONL (F02)++**Acceptance:** Local BM25 (all-text / title / title-boost / optional RM3) reports nDCG@10/@20. **Not** cited as Reddy. **Not** a brain wipe.++**Win:** nDCG@10 **≥ 0.515**. **Stop:** all variants < 0.495+0.02.++### Task 4: RM3 expanded queries → live skip-ingest passages (F03)++**Acceptance:** Copy JSONL with expanded `query` text; `./search.sh --brain searchbenchesci74 evaluate --skip-ingest --channels passages`. No re-ingest.++**Win:** same as Task 3 on the **BrainAPI** list. **Stop:** API down or nDCG@10 < 0.515.++### Checkpoint++- CE-on-pool row vs 0.669 / 0.857, n=74 stated.+- First-stage row vs 0.495, protocol labeled (local BM25 vs live passages).+- No mix with slice n=11 0.758.++---++## Located numbers (do not average)++| Arm | n | nDCG@20 | notes |+| --- | --- | --- | --- |+| CE-on-pool z.s. MiniLM-L-6 MS MARCO | 74 | 0.631 | coverage 1.0 |+| CE-on-pool FT 1 ep, 30k, Exact→1 | 74 | 0.669 | did not beat 0.857 |+| Reddy EN CE | ~4477 | **0.857** | ranking-in-pool, FT L-12 |+| KDD www | private | 0.9043 | ensembles |+| BrainAPI `--rank-pool` k=20 | 74 | 0.579 | coverage 0.517 |+| First-stage passages 2043-doc | 74 | 0.549 (@20), **0.495 (@10)** | not 0.857 |+| Slice passages | 11 | — | nDCG@10 0.758; frozen `searchbenchesci20` |++---++## Outcomes (located evidence, 2026-08-18)++**CE-on-pool.** 4-class MiniLM-L-12 (40k US Task-1 pairs, test qids held out, catalog fields, score = 1·P(E)+0.1·P(S)+0.01·P(C)): nDCG@20 **0.678**, full-list nDCG **0.747**, nDCG@10 0.597, missing_text=0 (`runs/search-esci-74-ce-pool-l12-4class`). **Progress vs 0.669: yes (+0.009).** **Reddy 0.857: not beaten.** n=74 ≠ ~4477. Confounded vs prior FT (L-12 vs L-6, 4-class vs Exact→1, 40k vs 30k, catalog vs title).++**First-stage.** Live skip-ingest passages: nDCG@10 **0.495**, nDCG@20 0.549, R@10 0.377, R@20 0.592, MRR 0.758, p50 62 ms (`search-esci-74-passages-control`). Local BM25 title-boost 0.458; SPLADE 0.465; RM3 hurt. **Win ≥0.515: failed.** BrainAPI hybrid already beats these harness first-stage indexes on this 2043-doc set. Task 4 RM3 live expand was **not** run after local RM3 lost.++**Decision log:** implemented C02+C04 (4-class L-12) and F02/F04 (fielded BM25 + local SPLADE). C06 v3-base catalog slice run 2026-08-19 (first-stage null; pool 0.710 ≠ 0.857). C06 large/ensemble and C07 not run. F03 stopped. Product default rerank unchanged. `searchbenchesci74` not wiped.++### Deeper retrieve k, then CE on retrieved hits (2026-08-18)++Not CE-on-pool. Not Reddy 0.857. Details in `19-search-esci-quality.md` and `20-esci-sota-targets.md`.++**Phase 1 located evidence:** raising search `k` to 50 on frozen `searchbenchesci74` passages raised Recall@50 to **0.834** and `--rank-pool` coverage to **0.778** (from 0.517 at k=20). nDCG@10 stayed **~0.500** vs control **0.495**. k=100 Recall@100 0.906; Recall@50 did not beat k=50.++**Phase 2 located evidence:** harness CE over stored k=50 `hit_ids` (MiniLM-L-6 and 4-class L-12 e2) left Recall@50 at 0.834 and dropped nDCG@10 to **0.448** / **0.467**. Win ≥0.515 failed. `RERANK_MAX_K` stayed 10.++### Dual-k first-stage matching (2026-08-18)++Not CE-on-pool. Not Reddy 0.857. Dual win needed Recall@10 ≥ 0.397 **and** Recall@50 ≥ 0.854.++**Located evidence:** miss taxonomy 67 head-ok / 2 rank-too-low / 5 total-miss (1270 golds: 424 / 728 / 118). Query rewrite of `esci-113` and `esci-267` did not move those qids or the mean. Local ANCE MiniLM sidecar Recall@10 **0.302** / Recall@50 **0.750**. **Dual win failed.** Passages hybrid remains the control.++### Retrieved-neg CE, ColBERT, BGE (2026-08-18)++Not CE-on-pool. Not Reddy 0.857. Details in `19-search-esci-quality.md` and `20-esci-sota-targets.md`.++**Located evidence:** retrieved-neg 4-class L-12 on stored k=50 hits nDCG@10 **0.465** (gate ≥0.515 failed; Recall@50 held 0.834). ColBERT MaxSim sidecar Recall@10 **0.311** / Recall@50 **0.714**. BGE-base (not MiniLM; no `query:` prefix) Recall@10 **0.322** / Recall@50 **0.799**. Dual-k gates ≥0.397 **and** ≥0.854 failed. `RERANK_MAX_K` stayed 10. `searchbenchesci74` not wiped.++### DeBERTa-v3-base 4-class on retrieved k=50 (2026-08-19)++Not Reddy 0.857 as a first-stage target. Details in `19-search-esci-quality.md` and `20-esci-sota-targets.md`.++**Located evidence:** unweighted 4-class `microsoft/deberta-v3-base`, 80k pairs, 2 epochs, test qids held out. Ranking-in-pool nDCG@20 **0.710** vs MiniLM e2 **0.695** (`search-esci-74-ce-pool-deberta-base`; n=74 ≠ ~4477). Harness CE on stored k=50 hits (`search-esci-74-passages-k50-ce-deberta`): nDCG@10 **0.510**, Recall@10 **0.363**, Recall@50 **0.834**. Predeclared first-stage win Recall@10 ≥0.397 **and** nDCG@10 ≥0.515 **failed**. Live `mode=catalog` skipped. `RERANK_MAX_K` stayed 10. `searchbenchesci74` not wiped.++### Multilingual first-stage foundation (2026-08-19)++Not CE-on-pool. Not Reddy 0.857. Not a US n=74 rerun. Details in `22-multilingual-ecommerce-search.md`.++**Located evidence:** Reddy locales US/ES/JP only; ES/JP organizer neural baseline is MPNet, EN is MiniLM. No Italian product-search qrel located. Live first-stage ES (`search-esci-es-passages-k50`, `searchbenchescies`, n=62, k=50 passages): nDCG@10 **0.577**, Recall@10 **0.353**, Recall@50 **0.914**. Not Reddy ES 0.849. `searchbenchesci74` not wiped.diff --git a/docs/research/22-multilingual-ecommerce-search.md b/docs/research/22-multilingual-ecommerce-search.mdnew file mode 100644index 0000000..faf9b95--- /dev/null+++ b/docs/research/22-multilingual-ecommerce-search.md@@ -0,0 +1,227 @@+# 22 — Multilingual first-stage and production e-commerce search++Workstream: `/retrieve/search` only on `searchbench*`. Ledger: `benchmarks.search` only. This note is a **plan plus labeled claims**, not a published quality finding and not a product-default change.++**Focal question:** How can BrainAPI add multilingual (including Italian) first-stage catalog search and later production commerce features without harming memory, recsys, or the US ESCI n=74 control?++Session: 2026-08-19. Origin of ideas: AI-assisted, independent round before literature. Decision owner: the user.++---++## Isolation (decision)++- Brains: `searchbench*` only. Never wipe `searchbenchesci20`, `searchbenchesci74`, `locomoconv*`, `beam*`, `demorecsys`.+- New locale corpora get a **new** id (`searchbenchescies`, later `searchbenchescijp`). Skip-ingest on frozen brains.+- Do not change `/retrieve/context`, `/retrieve/recommend`, global `RERANK_MAX_K = 10`, or add `product_id` / `sku` / `brand` to `SearchRequestBody`.+- Default omitted `channels` stays `["passages"]`. Graph off ranking unless a predeclared isolated arm.+- n=74 rows stay first-stage; do not mix with CE-on-pool 0.695/0.710 or Reddy 0.857.+- Never print `BRAINPAT_TOKEN`. Benchmarks do not import `src/`.++---++## Independent ideation (before literature)++Stage `independent`. Contributor: this workstream.++### Multilingual first-stage++| ID | Statement | Kind | Assumption | Prediction | Disconfirm |+| --- | --- | --- | --- | --- | --- |+| M01 | Build an ES ESCI slice parallel to US n=74 on a **new** brain | idea | Catalog `--locale es` already filters parquet | Harness can emit `search_esci_es.jsonl` without touching `search_esci.jsonl` | ES download overwrites US JSONL |+| M02 | JP is a separate, costlier arm | idea | English `to_tsvector` will tokenize JP poorly | JP BM25 will look like Reddy’s JP BM25 collapse; dense may carry more | JP BM25 ≈ US BM25 |+| M03 | Do not MT the US 74 and call it Italian quality | idea | ESCI has no IT locale | An MT “IT” nDCG is not an Italian product-search estimand | A real Italian product qrel appears |+| M04 | Default OpenAI/Azure `text-embedding-3-large` is already multilingual enough for a first ES ingest | assumption | Provider docs; not re-checked on this corpus | ES ingest on a new brain does not require a new encoder; do **not** re-ingest `searchbenchesci74` | ES lexical overlap with EN embeddings is unusable |+| M05 | English Snowball on `search_tsv` hurts Italian morphology | idea | PostgreSQL `english` config stems EN, not IT | Italian smoke may still match exact tokens; inflected queries will miss | Italian analyzer on an isolated arm lifts smoke recall without US regression |+| M06 | Per-brain or per-request FTS language must not flip the global generated column | idea | `_SEARCH_DDL` is shared | Changing `'english'` globally would retokenize memory brains | Isolated search-only language column |+| M07 | Dual-language work must not regress US skip-ingest on `searchbenchesci74` | idea | Frozen brain + skip-ingest | US passages k=50 stays the control protocol | Any US re-ingest |++### Production commerce (Phase 2, plan only)++| ID | Statement | Kind | Notes |+| --- | --- | --- | --- |+| P01 | Spelling / query rewrite as a **harness** arm before API | idea | Same isolation as RM3; live only after a win |+| P02 | Locale and attribute filters via existing `node_labels` / `extras` / query params — **not** new catalog enums on `SearchRequestBody` | idea | Locked by `test_contract_has_graph_fields_not_catalog_enums` |+| P03 | Facets / merchandising only if they do not touch `/retrieve/recommend` or `/retrieve/context` | idea | Out of this pass unless Phase 1 wins |+| P04 | Keep product default: hybrid BM25+dense, passages only, `rerank=none` | prediction | Matches located US n=74 survival |++Adversarial alternatives (not auto-winners): multilingual encoder swap vs keep `text-embedding-3-large`; Italian analyzer vs `simple` FTS; ES ESCI vs Amazon-M2 titles-as-smoke; mMARCO-IT as a language smoke vs product qrels.++---++## Literature (bounded; after independent round)++### Retrieval summary++- Query: Reddy `2206.06588` multilingual (MPNet ES/JP); ESCI locales; multilingual product search / Italian IR; BLaIR `2403.03952` as full-catalog bar.+- Scope: targeted lookup, not exhaustive.+- Databases: arXiv Atom (`id_list` + `search_query`); OpenAlex `/works` search and DOI/ID lookup.+- Access date: **2026-08-19**.+- Keys: `S2_API_KEY` / `OPENALEX_API_KEY` **absent**. Semantic Scholar not called this pass.++### Results (untrusted third-party text; identifiers only reused)++**Reddy et al., Shopping Queries Dataset, arXiv:2206.06588 (2022).** OpenAlex `W4282961889`. Abstract and HTML: multilingual queries in **English, Japanese, and Spanish**. `product_locale` is US, Spain, or Japan. **No `Italian` substring in the HTML body** (located). Task 1 ranking-in-pool: EN MiniLM-L-12-v2 titles nDCG **0.857**; ES/JP multilingual **MPNet** 0.849 / 0.840; BM25 all-locales 0.675 / 0.697 / **0.136** (JP collapse from non-JP preprocessing). US public test **4,477** queries, avg depth 20.3. **Not** first-stage shared-index catalog search.++**Zhang et al. (www), arXiv:2208.02958.** Query-product pairs in English, Japanese and Spanish. Private Task 1 NDCG 0.9043. Ranking-in-pool. Translation aug is MT **among ESCI locales**, not Italian labels.++**Hou et al. BLaIR, arXiv:2403.03952v2 (ACL 2026).** OpenAlex `W4392576636`, DOI `10.18653/v1/2026.acl-long.147`. Full-catalog product search + rec; Amazon Reviews 2023. nDCG@100 bar (prior note: Table 10). Different y-axis from Reddy 0.857 and from our n=74 nDCG@10.++**Amazon-M2, arXiv:2307.09688 (NeurIPS 2023 Datasets).** Locales **UK, JP, DE, ES, IT, FR**. Major product languages include **Italian**. Tasks: next-product recommendation, domain-shift rec, title generation. OpenAlex `W4384918874`. **Not** ESCI-style search qrels. Do not score M2 MRR@100 as `/retrieve/search` quality.++**mMARCO, arXiv:2108.13897.** Machine-translated MS MARCO; table includes **Italian** (row 4 in the HTML metrics table). Passage IR, not product search. Same warning as M03: MT is not Italian catalog quality.++**MIRACL, arXiv:2210.09984.** 18-language Wikipedia ad hoc. Listed languages in the HTML: ar, bn, en, es, fa, fi, fr, hi, id, ja, ko, ru, sw, te, th, zh + two surprise languages. **No `Italian` substring in the HTML.** Not a product-search set.++**arXiv `all:"product search" AND (Italian OR italiano) AND cat:cs.IR`:** totalResults **0** (genuine no-match). **`all:ESCI AND (Italian OR italiano)`:** totalResults **0**.++**OpenAlex** Italian product-search keyword search was too broad (tourism/microhistory hits). Tighter ESCI query recovered Reddy / Zhang / KDD papers (EN/ES/JP). XMarket (Bonab et al., DOI `10.1145/3459637.3482493`) is **cross-market recommendation**, not search qrels.++### Provenance++- arXiv: `GET https://export.arxiv.org/api/query` `id_list=2206.06588,2403.03952` then (after ≥3s) `id_list=2307.09688,2108.13897,2210.09984`; parsed with `paper-lookup/scripts/arxiv_atom.py`. `query_as_executed` matched `id_list`. HTTP 200, no `Error` entries.+- arXiv keyword: `all:"product search" AND (Italian|italiano) AND cat:cs.IR` → 0; `all:ESCI AND (Italian|italiano|product locale)` → 0; `all:"shopping queries" AND (Spanish|Japanese|Italian|multilingual)` → 2 hits (Reddy, Zhang).+- OpenAlex: `/works?search=…` (ESCI multilingual; XMarket; mMARCO; MIRACL; Amazon-M2); `/works/doi:10.48550/arxiv.2206.06588`; `/works/W4392576636`; `/works/W4384918874`. Abstracts via `openalex_abstract.py` where inverted index existed. BLaIR `doi:10.48550/arxiv.2403.03952` was **not valid JSON** this pass; ID lookup 200.+- Reddy / MIRACL / mMARCO / Amazon-M2 HTML: `https://arxiv.org/html/{id}` on 2026-08-19.++**Warnings:** 429s not observed on arXiv/OpenAlex this pass. Do not invent Italian-in-ESCI. mMARCO-IT and Amazon-M2-IT are **not** ESCI Task 1. BLaIR nDCG@100 ≠ first-stage nDCG@10.++### Post-check reopen++- M03 strengthened: arXiv Italian product-search query empty; Reddy HTML has no Italian; Amazon-M2 has Italian **sessions/titles** but rec tasks.+- M02 aligned with Reddy JP BM25 0.136 — our `'english'` tsvector is the same class of mismatch.+- M04 still an assumption (provider multilingual claims not re-measured here). Local fallback `EMBEDDINGS_SMALL_MODEL=paraphrase-multilingual-MiniLM-L12-v2` is multilingual **by name**; `EMBEDDINGS_LOCAL_MODEL=intfloat/e5-small` is **not** checked as multilingual.+- P02 unchanged: no catalog enums on the search body.++**Italian product-search qrels:** **no direct evidence located.** Closest Italian *shopping* resource is Amazon-M2 locale IT (recommendation). Closest Italian *IR* resource in this lookup is mMARCO-IT (MT passages). Neither is an ESCI Task 1 stand-in.++### Phase 2 Italian qrel re-query (2026-08-19)++**Kind:** located evidence (empty). **Do not** treat Amazon-M2 or mMARCO-IT as `/retrieve/search` gold. Do not machine-translate US/ES ESCI.++| Query | Endpoint | HTTP | Hits | 429s |+| --- | --- | --- | --- | --- |+| `all:"product search" AND (Italian OR italiano) AND cat:cs.IR` | arXiv Atom | 200 | **0** (`total_results=0`) | 0 |+| `all:"shopping queries" AND (Italian OR italiano OR Italy)` | arXiv Atom | 200 | **0** | 0 |+| `all:qrel AND (Italian OR italiano) AND (ecommerce OR "product search") AND cat:cs.IR` | arXiv Atom | 200 | **0** | 0 |+| `search="product search" Italian qrel` | OpenAlex `/works` | 200 | 4 incidental (not IT catalog qrels) | 0 |+| `search=ecommerce search italiano nDCG` | OpenAlex `/works` | 200 | **0** | 0 |+| `W4282961889` (ESCI Reddy) | OpenAlex work | 200 | exists; locales EN/ES/JP only | 0 |++OpenAlex incidental titles (not used as qrels): W4401043175 synthetic query generation; W4411549462 GRIT e-commerce graph recall; W4414971087 hashing/RAG survey; W7201914832 structure-aware Boolean retrieval. None are Italian product-search qrels.++**Decision:** `--locale it` remains an ESCI error. No new Italian ESCI locale. Italian work stays the n=3 pipeline smoke fixture.++---++## Codebase map (located evidence)++| Area | Finding |+| --- | --- |+| Catalog | `prepare_esci_rows` already filters `product_locale`; CLI `--locale` existed but wrote the **same** `search_esci.jsonl` as US |+| Embeddings | Default `text-embedding-3-large` (OpenAI/Azure `.env.example`); TUI may run from `~/.brainapi/source` |+| BM25 | `src/lib/postgresql/data.py` `_SEARCH_DDL`: `to_tsvector('english', …)` generated column. Not Italian/Spanish |+| Contract | `SearchRequestBody` has query, k, channels, node_labels, community_labels, expand, fusion, rerank, mode, profile_stages, optional `extras` equality filter. Hits have `labels` / `extras`. Response may include `facets`. No sku/brand/product_id **fields** |+| Eval | `evaluate.py` already records `slice`; `metrics.py` `by_slice`. Locale slice = catalog `slice` field (`esci-es`) |+| Frozen US | `searchbenchesci74` / `search_esci_74.jsonl` — skip-ingest only |++---++## Phased plan++### Phase 1 — multilingual first-stage (this pass: foundation only)++Predeclared gates (**before** any live ES eval):++1. US default JSONL path remains `search_esci.jsonl`. `--locale es` writes `search_esci_es.jsonl`. `--locale it` errors.+2. `to_tsvector('english'` stays the product FTS config. Dual-language must not regress US skip-ingest on `searchbenchesci74`.+3. ES passages nDCG@10 is **not** a copy of US 0.500. Report ES separately. No p-values. n may differ from 74.+4. Italian smoke (`search_italian_smoke.jsonl`) is a **pipeline** check: ingest+search returns hits. It is **not** ESCI Task 1 and not an Italian quality claim.+5. Product default remains hybrid BM25+dense, passages, `rerank=none`. Do not re-ingest `searchbenchesci74`.++**Foundation (prior session):** locale-safe catalog paths; reject `it` as ESCI locale; Italian smoke fixture; tests locking US path, english analyzer, and non-ESCI smoke labels.++**Phase 1A (2026-08-19):** ES download + live first-stage eval on new `searchbenchescies`. JP still out. Product FTS stays english in `_SEARCH_DDL`.++---++## Located ES first-stage (2026-08-19)++**Kind:** located evidence. **Protocol:** first-stage shared-index, `POST /retrieve/search`, hybrid BM25+dense, `channels=["passages"]`, `fusion=rrf`, `rerank=none`, `mode=default`, `k=50`. Brain `searchbenchescies` (new; did not wipe `searchbenchesci74`). Dataset `search_esci_es.jsonl` (`slice=esci-es`). n=**62** queries, **2000** docs. English `to_tsvector` (same as US). No p-values.++| Metric | ES n=62 k=50 |+| --- | --- |+| nDCG@10 | **0.577** |+| Recall@10 | **0.353** |+| Recall@50 | **0.914** |+| MRR | 0.870 |+| p50 retrieve | 84 ms |++**Not** a copy of US n=74 nDCG@10 0.500 / Recall@10 0.379. Different n, different catalog, not a paired test. Do not average with US. **Not** Reddy ES Task 1 MPNet **0.849** (ranking-in-pool, ~20-item lists). **Not** CE-on-pool 0.695/0.710.++**Finding:** Recall@50 **0.914** is not a Reddy-JP-BM25-collapse class outcome (Reddy JP BM25 0.136 on Task 1 pools). Product default unchanged.++**Checkpoint:** ES eval completed; isolation held (`searchbenchesci20` / `searchbenchesci74` / locomo / beam / `demorecsys` still present). Human review of these numbers is the gate before any FTS schema work. JP arm stays out.++**Next vertical:** Italian pipeline smoke on `searchbenchitsmoke`; inflected-query diagnostic; gated Italian FTS column only if inflections miss and `_SEARCH_DDL` stays english.++**Command:** `./search.sh --brain searchbenchescies evaluate --dataset data/search_esci_es.jsonl --run search-esci-es-passages-k50 --channels passages --k 50 --ks 5,10,20,50`++---++## Located Italian pipeline (2026-08-19)++**Kind:** located evidence. **Not** an Italian product-search quality estimand (n=3; not ESCI).++Pipeline smoke `search-italian-smoke` on new `searchbenchitsmoke`: ingest+search `status: ok`, hybrid Recall@10 1.0. Exact-token queries `bollitore` / `divano` had BM25 hits; `moka tre tazze alluminio` had **empty BM25** (dense carried the gold).++Inflected diagnostic `data/search_italian_smoke_inflect.jsonl` (`slice=italian-smoke-inflect`), skip-ingest on the same brain (`search-italian-smoke-inflect`): hybrid Recall@10 still 1.0 because **dense** retrieved the three docs. **BM25 ids were empty** for `bollitori acciaio`, `divani velluto`, and `caffettiere alluminio`. Idea M05: english Snowball missed Italian morphology. Not a quality claim.++**Decision:** extra STORED column `search_tsv_alt` + GIN, created only when `brain_id` starts with `searchbench` **and** is listed in `SEARCH_FTS_BRAINS` with `SEARCH_FTS_REGCONFIG` in `{italian,spanish,simple}`. `_SEARCH_DDL` stays `to_tsvector('english'`. Memory/recsys brains never get the column.++Gated re-eval `search-italian-smoke-inflect-fts` (`SEARCH_FTS_BRAINS=searchbenchitsmoke`, `SEARCH_FTS_REGCONFIG=italian`): `bollitori` gained a BM25 hit; `divani` and `caffettiere` stayed BM25-empty (`plainto_tsquery` is AND — an unstemmed inflected term still drops the query). `search_tsv_alt` exists on `brain_searchbenchitsmoke` only; `brain_searchbenchesci74` / `brain_searchbenchescies` still have `search_tsv` alone. `RERANK_MAX_K` stayed 10. Product default unchanged.++**FTS OR on alt BM25 (2026-08-19, located evidence).** Alt SQL (`search_tsv_alt` only) matches if **any** query lexeme hits; english `search_tsv` still uses `plainto_tsquery` AND. Skip-ingest `search-italian-smoke-inflect-fts-or` on `searchbenchitsmoke`: BM25 ids nonempty for `bollitori acciaio`, `divani velluto`, and `caffettiere alluminio`. Hybrid Recall@10 1.0 is still dense+BM25 on n=3 — **not** an Italian nDCG claim. Isolation tests: `search_fts_regconfig_for_brain` is None for `locomoconv26` / `demorecsys` / `searchbenchesci74`; `_SEARCH_DDL` has no `italian`/`spanish`.++**Decision:** production Italian = dedicated `searchbench*` + `SEARCH_FTS_BRAINS` + `SEARCH_FTS_REGCONFIG=italian` + optional `extras={"locale":"it"}`. Do not enable alt FTS on memory/recsys brains. Do not MT US/ES ESCI.++### Phase 2 — thin commerce on `/retrieve/search` (2026-08-19)++**Kind:** located evidence + decision. Isolation held: no edits to `retrieve.py` / `entities.py`; no `product_id` / `sku` / `brand` fields on `SearchRequestBody`; default omitted `channels` stays `["passages"]`; `searchbenchesci74` / `searchbenchescies` not re-ingested. Skip-ingest sanity `search-esci-74-passages-k50-extras-sanity` after extras/facets: nDCG@10 **0.500**, Recall@10 **0.379**, Recall@50 **0.834** (bit-identical to the frozen control). p50 63 ms vs ~60 ms. Product default unchanged.++#### Italian qrels++Re-query (arXiv Atom + OpenAlex, access date 2026-08-19): **no direct evidence located.** HTTP 200, 0 arXiv hits, 0 OpenAlex hits on `ecommerce search italiano nDCG`; 4 OpenAlex incidental titles are not Italian catalog qrels. Amazon-M2 / mMARCO-IT still not `/retrieve/search` gold. `--locale it` still errors.++#### 2.1 Spelling harness (null → not live)++Miss-strata on `search-esci-es-passages-k50` / `search_esci_es.jsonl`: n=62, **0 total-miss** (`stratum_counts={'head-ok': 62}`). Query-level head-ok is not Recall@10 0.353 (item-level). Harness `spell-normalize` wrote `data/search_esci_es_spell.jsonl` (23 queries changed: NFKC, junk punctuation, conservative accent fold). No live LLM.++Skip-ingest eval `search-esci-es-spell-k50` on `searchbenchescies`, passages k=50:++| Metric | ES baseline | Spell arm | Gate |+| --- | --- | --- | --- |+| nDCG@10 | 0.577 | **0.595** | win if ≥ 0.597 — **miss** (+0.018) |+| Recall@10 | 0.353 | **0.362** | hold ≥ 0.353 — pass |+| Recall@50 | 0.914 | **0.914** | hold ≥ 0.914 — pass |++No previously total-miss qid to lift. **Decision:** spelling stays harness-only. Production `search.py` query text is unchanged. Do not mix with Reddy ES 0.849 or US n=74 0.500.++#### 2.2–2.3 Extras filter + hit-list facets++`SearchRequestBody.extras: dict[str, str] | None` is an equality filter after fusion, before cutting `k`. Catalog keys live inside that object (and on `SearchHit.extras`), not as body fields named `brand`. Search ingest now copies JSONL `brand` / `color` / `locale` via `meta_keys`. `SearchResponse.facets` counts extras values on the **current** hit list. No merchandising, no `/retrieve/recommend`.++Live on `searchbenchitsmoke` (3-doc re-ingest allowed): `extras={"locale":"it"}` keeps the three IT docs (Recall@10 1.0); facets `{locale: {it: 3}, brand: {CasaLuce, AtelierNord, FornoBasso}, color: {argento, blu navy, alluminio}}`. `extras={"color":"nope"}` returns empty hits / empty facets. p50 retrieve ~17 ms on this 3-doc brain — **not** a 200 ms SLO claim. Contract test still bans catalog enum field names.++#### Encoder smoke (M04 Italian-only)++Harness `local-dense` `paraphrase-multilingual-MiniLM-L12-v2` on `data/search_italian_smoke.jsonl`, labeled brain `searchbenchitmmini`, run `search-italian-minilm-pipeline`. **Did not** reuse `searchbenchitsmoke` 3-large vectors; **did not** re-ingest 74 or ES. n=3, `status: ok`, all three golds retrieved at k=10 (same pipeline recall as `search-italian-smoke` 3-large). **Not** a claim that MiniLM beats 3-large. **Not** Reddy. **Not** ES n=62. Product embedder unchanged.++### Out of scope++Wiping brains; raising default `RERANK_MAX_K`; CE as default; graph fusion as default; recsys; LoCoMo/BEAM; claiming Reddy 0.857; MT-US-as-Italian; Amazon-M2 MRR as search quality; merchandising that needs recommend/context.++---++## Located US control (do not redo)++Passages k=50 n=74: nDCG@10 **0.500**, Recall@10 **0.379**, Recall@50 **0.834**, p50 ~60ms. MiniLM/DeBERTa CE on those hits hurt Recall@10. Live `mode=catalog` MiniLM 0.467/0.341 null. Cascade R@50 0.889 harness-only (live sidecar not indexed). `fusion=cc` alpha sweep best nDCG@10 **0.493** (null vs 0.520). Literal residual Recall@50 **0.655** (null). Head LTR 5-fold CV on stored k=50: nDCG@10 **0.515**, Recall@10 **0.384** (null vs 0.520 / 0.397); overlap-only 0.476 / 0.339. LTR + MiniLM `ce_gain`: nDCG@10 **0.524**, Recall@10 **0.383**, Recall@50 **0.834** (harness win vs 0.520 / 0.379 hold; not live). LTR + DeBERTa-v3-base `ce_gain` RankNet CV: nDCG@10 **0.542**, Recall@10 **0.387**, Recall@50 **0.834** (win vs MiniLM 0.524; horizon 0.70 missed; not live). Gated LightGBM on the same DeBERTa matrix: nDCG@10 **0.516** (null vs RankNet). LTR + DeBERTa applied from 170 train hybrid lists (`searchbenchesciltr2`): nDCG@10 **0.544**, Recall@10 **0.391**, Recall@50 **0.834** (win vs 0.542; horizon missed; not live). LightGBM apply on the same 170 lists: nDCG@10 **0.533** (null vs RankNet 0.544; R@10 0.389 / R@50 0.834 held; not live). RankNet + hybrid-list DeBERTa `ce_gain` (`esci-deberta-v3-base-4class-hybrid170`): nDCG@10 **0.530** (null vs 0.544; R@10 0.385 / R@50 0.834 held; not live). Default path that survived: hybrid BM25+dense, passages, rerank=none, Italian via gated FTS+OR.diff --git a/docs/research/23-search-wands-quality.md b/docs/research/23-search-wands-quality.mdnew file mode 100644index 0000000..0f8fe0a--- /dev/null+++ b/docs/research/23-search-wands-quality.md@@ -0,0 +1,100 @@+# 23 — WANDS first-stage search quality++Workstream: `/retrieve/search` first-stage quality on a frozen Wayfair WANDS slice. Ledger: `benchmarks.search` only. This document records the **control protocol** and **located evidence**. It is not Chen et al. SOTA and not an ESCI number.++Every claim is labeled **idea**, **assumption**, **prediction**, **located evidence**, or **decision**.++---++## Focal question++What is BrainAPI’s live hybrid first-stage quality on a frozen WANDS catalog slice (passages, RRF, `rerank=none`, k=50), and what can that number actually support?++**Claim this run can support:** on this slice, with this product path, nDCG@10 / Recall@k / p50 were X.++**Claim this run cannot support:** we match Chen / WANDS SOTA / ESCI 0.500 / Reddy 0.857 / C6 0.544.++---++## Protocol (decision)++Mirror the ESCI n=74 first-stage control on a **different** brain, catalog, and gain table.++| Knob | Value |+| --- | --- |+| Dataset | Frozen [`benchmarks/data/search_wands.jsonl`](../../benchmarks/data/search_wands.jsonl) |+| Caps used to build it | `--max-queries 80 --max-docs 2000 --candidates-per-query 40` |+| Actual n | **66** queries / **2000** docs (`_select_catalog` dropped queries with no remaining gold once `max_docs` filled) |+| Slice label | `wands-80-2000` (actual n=66) |+| Brain | `searchbenchwands` (new; never wipe `searchbenchesci*`, `searchbenchesciltr2`, `locomoconv*`, `beam*`, `demorecsys`) |+| Ingest | `skip_enrichment` (default). No `--ingest-graph`. No `--enrich`. |+| Retrieve | `--channels passages --k 50 --ks 5,10,20,50`, fusion RRF, `rerank=none`. No `--rank-pool`. |+| Run | `search-wands-passages-k50` |+| After first eval | `--skip-ingest` only |+| Download freeze | `catalog_overwrite_blocked`: `--force` cannot clobber `search_wands.jsonl` once it exists |++```bash+./search.sh --brain searchbenchwands evaluate \+ --dataset data/search_wands.jsonl \+ --run search-wands-passages-k50 \+ --channels passages --k 50 --ks 5,10,20,50+# after first eval:+./search.sh --brain searchbenchwands evaluate \+ --dataset data/search_wands.jsonl \+ --run search-wands-passages-k50 \+ --channels passages --k 50 --ks 5,10,20,50 --skip-ingest+```++**Metrics (decision):**++- **nDCG@10:** linear DCG `rel / log2(rank+1)` with Exact=1, Partial=0.5, Irrelevant=0. Unlabeled retrieved docs gain 0. Affine-equivalent to linear 2/1/0; **not** TREC `2^rel-1`.+- **Recall@10 / @50:** binary over **Exact + Partial** (`gold_doc_ids` = gain > 0). Not ESCI E+S+C.+- **p50 retrieve ms** excluding embed.query.+- n=66, no CIs. Do not mix with ESCI 0.500 / C6 0.544.++**Subset caveat:** full WANDS is 480 queries / 42,994 products / 233,448 labels. This control searches the ingested 2000-doc brain, not the 40-candidate judged pool and not the 43k catalog.++Do **not** edit `src/services/api/retrieve.py`, `entities.py`, or `search.py` for this measurement.++---++## Literature (located evidence, 2026-08-20)++| Item | Status |+| --- | --- |+| Chen, Liu, Liu, Sun, Baltrunas, Schroeder. *WANDS: Dataset for Product Search Relevance Assessment*. ECIR 2022. DOI [10.1007/978-3-030-99736-6_9](https://doi.org/10.1007/978-3-030-99736-6_9). OpenAlex `W4225565720`. | **support-located** (identity). `is_oa=false`, `abstract_inverted_index=null`. Springer LNCS, not OA. |+| arXiv title search `ti:WANDS AND (Wayfair OR "product search relevance")` | **no Chen paper**. Hits were unrelated IR papers. |+| Dataset size ([wayfair/WANDS](https://github.com/wayfair/WANDS/)) | **located evidence:** 480 queries, 42,994 products, 233,448 labels. Exact / Partial / Irrelevant. |+| Published BM25 / hybrid nDCG in Chen et al. | **not located**. Do not invent. Medium notebooks are not primary. |++---++## First-stage control numbers++**Located evidence (2026-08-20).** Live `search-wands-passages-k50` on `searchbenchwands`. `eval.json`: `skip_ingest=false`, `ingest.status=completed`, `n_docs_mapped=2000`, `ingest_graph=false`, channels `passages`, fusion `rrf`, `rerank=none`, k=50. Ingest wall ~14 min. Ledger: `benchmarks.search` run `search-wands-passages-k50`.++| Metric | Value |+| --- | --- |+| n | 66 queries / 2000 docs |+| nDCG@10 | **0.823** |+| nDCG@20 | 0.812 |+| Recall@10 | **0.269** |+| Recall@20 | 0.465 |+| Recall@50 | **0.837** |+| MRR | **0.925** |+| p50 retrieve | **87 ms** |+| p95 retrieve | 97 ms |++Gold (Exact+Partial) min/median/max = 1 / 40 / 40; mean 34.1. Structural Recall@10 ceiling = mean `min(10,|gold|)/|gold|` = **0.365**. Observed 0.269 is below that ceiling, not below ESCI 0.379 (different gold density and gain table). 60/66 queries have a relevant at rank 1. Two queries have nDCG@10 = 0: `Bath Rugs & Mats` (Recall@50 also 0) and `Vanities` (first relevant at rank 27; Recall@50 = 1.0).++**Do not mix** with US ESCI n=74 nDCG@10 0.500 / C6 0.544, Reddy 0.857, or unpublished Medium WANDS notebooks. Chen et al. nDCG remains **not located**.++After this run, freeze the index: `--skip-ingest` only. Do not re-ingest `searchbenchwands`. Do not wipe ESCI brains.++---++## Isolation++- Brains: `searchbenchwands` only for this catalog. Skip-ingest on `searchbenchesci74`.+- Product default stays hybrid RRF, `rerank=none`, omitted `channels=["passages"]`.+- Ranking / LTR / CE / graph on WANDS are out of scope until a new human decision.diff --git a/docs/research/24-search-catalog-graph.md b/docs/research/24-search-catalog-graph.mdnew file mode 100644index 0000000..c2215ce--- /dev/null+++ b/docs/research/24-search-catalog-graph.md@@ -0,0 +1,175 @@+# 24 — Opt-in catalog graph + hybrid search++Workstream: structured catalog ingest + `channels=["communities"]` product retrieve. Ledger: `benchmarks.search` only, **architecture demo**. This is not a quality-default flip and not a claim against frozen WANDS passages nDCG@10 **0.823** or ESCI n=74 **0.500**.++Every claim is labeled **idea**, **assumption**, **prediction**, **located evidence**, or **decision**.++---++## Focal question++Can BrainAPI ship an **opt-in** catalog path: labeled product/hub nodes with `search_text` embeddings + node BM25, hub intersection (CLASS ∩ ATTR), then hybrid-rank **product nodes** inside the communities channel — without changing the live omitted-`channels` default (`["passages"]`)?++**Claim this run can support (located evidence, 2026-08-21):** on brain `searchbenchwandsgraph`, isolated communities nDCG@10 **0.409** / Recall@10 **0.148** / Recall@50 **0.287** / p50 retrieve **5609 ms** (ex-embed); paired passages nDCG@10 **0.823** / Recall@10 **0.269** / Recall@50 **0.837** / p50 retrieve **99 ms**. Architecture demo only.++**Claim the 2026-08-21 join round can support (decision + prediction):** architecture-demo that (a) paired passages nDCG@10 on `searchbenchwandsgraph` stays ~0.823, (b) catalog passage hits expose `node_id` = `doc_id` = ENTITY uuid, (c) that uuid round-trips to `GET /retrieve/entities/neighbors`.++**Claim this run cannot support:** we beat frozen `searchbenchwands` 0.823, ESCI 0.500, Reddy 0.857, or Chen et al. Communities beat passages. We should change the product default. Mixing chunk ids and node ids in default RRF.++---++## Architecture (decision)++One KB, three contracts (ADR-007). Search stays `POST /retrieve/search`. Recs stay `/retrieve/recommend`. Interactions may be ingested onto the same graph; they are not scored as HitRate and `user_id` is not a `SearchRequestBody` field. Optional search `target` (query-gated rerank of retrieved hits) is [ADR-008](../decisions/008-query-gated-search-personalization.md) / [25-personalized-search.md](25-personalized-search.md), not this workstream.++```text+catalog JSONL+ → skip_enrichment chunk ingest (passages)+ → POST /ingest/structured (HAS triples; entity uuid = doc_id)+ → CLASS / ATTR / TYPE hubs + ENTITY products (search_text on nodes)+ → channels=["communities"]: match hubs → intersect if ≥2 hub kinds else union+ → RRF(node BM25 ∪ node dense) on the product set (inside graph_channels)+```++- **Passages stay the live default (decision, 2026-08-21).** Omitted `channels` remains `["passages"]`. First-stage is query → passage. The catalog graph is not a competing candidate generator. `rerank=none`. `search.py` fusion does not mix chunk ids and node ids on the default path.+- **`node_id` is a sidecar, not fusion (decision).** After ranking, catalog passage hits attach `SearchHit.node_id` (and unique `SearchResponse.node_ids` in hit order) by parsing `DOCID <id>` from fetched chunk text. `entity_uuid(doc_id)` is `str(doc_id)`, so the join is deterministic and does not extra-round-trip the graph. Graph-channel item hits set `node_id = id`. Missing marker → `node_id` is null. The field is **`node_id`**, not `product_id` / `sku` / `brand` / `user_id`.+- **Clients use existing graph reads.** `node_id` is the product ENTITY uuid for `/retrieve/entities/neighbors`, `/retrieve/entity/synergies`, `/retrieve/recommend`, entity info. Recsys and memory already consume the graph. This round does not change `/retrieve/context` (ADR-006).+- **Product ENTITY `search_text` is the catalog blob (decision).** Prefer `doc["text"]` when present, else title + class + features + description (plus hierarchy/brand/color when composed). Hub `search_text` stays short name/key text.+- **Static catalog attributes are a direct edge (decision, 2026-08-21).** Mapper `has_triple` writes `ENTITY -HAS-> CLASS|TYPE|ATTR` with no EVENT mid-node. Structured ingest collapses the same shape if an old 5-tuple `HAS` wrapper arrives with no `happened_at`. Keep EVENT for real occurrences (`USER -MADE-> View/Purchase -TARGETED-> PRODUCT`, with `happened_at`). Polarity is not a reason to keep HAS wrappers. **Located evidence:** `searchbenchwandsgraph` was persisted on the 2-edge wrapper (≈32k HAS events); that brain is not rewritten this change.+- **Catalog product path is explicit:** `channels=["communities"]` (optionally `entities`) remains an opt-in architecture demo. Do not re-eval communities nDCG as a quality gate this round.+- **Hubs, not Louvain.** Communities already means TYPE/CLASS/TOPIC/ATTR hubs.+- **Continuous attributes are not hubs.** Price / measures / rating stay node properties when present. Style / color / material / class become ATTR/CLASS hubs.+- **Eval isolation:** brain `searchbenchwandsgraph` only. Never wipe `searchbenchwands`, `searchbenchesci74`, `searchbenchescies`, `searchbenchesciltr2`, `locomoconv*`, `beam*`, `demorecsys`. Frozen JSONL `benchmarks/data/search_wands.jsonl` is reuse-only; do not wipe frozen structured brains. ENTITY text/embed refresh, if run, refuses `FROZEN_STRUCTURED_BRAINS` and does not resubmit Celery architect.++Mapper lives in product code [`src/core/search/catalog_graph.py`](../../src/core/search/catalog_graph.py). The harness re-exports it from [`benchmarks/search/mapping.py`](../../benchmarks/search/mapping.py) (repo root on `sys.path`). `--ingest-graph` / `--interactions` targeting frozen structured brains raises `SystemExit`.++---++## Literature (located evidence, 2026-08-20)++**Support-located for query–document first-stage (not hub-walk):**++- Nigam et al., *Semantic Product Search*, arXiv `1907.00937` (KDD 2019) — dense semantic matching complements lexical; query → document, not query → hub → walk.+- Choi et al., *Semantic Product Search for Matching Structured Product Catalogs*, arXiv `2008.08180` — fielded catalog text plus lexical features.+- Balog / Garigliotti entity-oriented search, arXiv `1802.08010` — query → typed entities → fulfill (hubs then products) is a graph-read pattern, not the live search default.++**Challenge-located / do not overclaim (PKG as the only candidate generator):**++- Xu et al., *Product Knowledge Graph Embedding for E-commerce*, arXiv `1911.12481`, DOI `10.1145/3336191.3371778` — product KG used in ranking **and** recs; does not validate symbolic hub intersection as first-stage IR.+- Ai et al. DREM, arXiv `1909.07212` — user–item KG embeddings for search; learned relations, not HAS expansion.+- Zhang et al. PKGM, arXiv `2105.00388` — PKG for recs/classification, not first-stage nDCG.+- No located paper that community-detection retrieval beats BM25+dense on WANDS/ESCI. Absence from this bound is not “never studied.”++Prior BrainAPI **located evidence:** peer RRF of graph lists with passages **hurt** an ESCI slice (nDCG@10 0.682 vs 0.758 passages). Do not treat extra graph RRF peers as a quality win.++---++## Protocol (decision)++| Knob | Value |+| --- | --- |+| Dataset | Frozen [`benchmarks/data/search_wands.jsonl`](../../benchmarks/data/search_wands.jsonl) (reuse only) |+| Actual n | **66** queries / **2000** docs |+| Brain | `searchbenchwandsgraph` (**new**; never `searchbenchwands`) |+| Ingest | `skip_enrichment` chunks, then `--ingest-graph`. After first eval, `--skip-ingest` |+| Communities | `--channels communities --k 50 --ks 5,10,20,50`, fusion RRF, `rerank=none` |+| Paired passages | same brain, `--channels passages`, `--skip-ingest` |+| Runs | `search-wandsgraph-communities-k50`, `search-wandsgraph-passages-k50` |+| Ledger claim | `architecture-demo` |++```bash+./search.sh --brain searchbenchwandsgraph evaluate \+ --dataset data/search_wands.jsonl \+ --run search-wandsgraph-communities-k50 \+ --channels communities --k 50 --ks 5,10,20,50 \+ --ingest-graph --skip-ingest --timeout 10800+./search.sh --brain searchbenchwandsgraph evaluate \+ --dataset data/search_wands.jsonl \+ --run search-wandsgraph-passages-k50 \+ --channels passages --k 50 --ks 5,10,20,50 \+ --skip-ingest+./search.sh --brain searchbenchwandsgraph smoke \+ --dataset data/search_wands.jsonl \+ --interactions data/search_wands_interactions_smoke.jsonl \+ --channels events --skip-ingest --timeout 1800+```++**Go/no-go for a later default (not executed this round):** communities nDCG@10 ≥ paired passages **on this brain**. **Located evidence:** 0.409 < 0.823 — gate fails. Still must not regress frozen ESCI 0.500 / WANDS 0.823 protocols. Do not quote either row as beating those frozen numbers.++---++## Mapper / graph write (located evidence, 2026-08-20)++Offline count on the frozen JSONL via `docs_to_triples` (not a quality claim):++| Count | Value |+| --- | --- |+| HAS triples | 32208 |+| Product ENTITY nodes | 2000 |+| Hub nodes | 3641 |+| CLASS hubs (unique names) | 157 |+| ATTR hubs (unique names) | 3484 |+| TYPE hubs | 0 (frozen JSONL has no `hierarchy` / `category_hierarchy` field) |+| Docs with `class` | 1863 |+| Docs with `description` field | **0** |+| ATTR `modern` present | yes (`dsprimaryproductstyle : modern` and similar) |++Those 32208 HAS triples were written as **two edges + EVENT mid-node** on `searchbenchwandsgraph`. **Decision (2026-08-21):** new mapper/ingest writes them as one `HAS` edge. Do not treat the live 37383/64416 counts as the post-change shape. Do not wipe that brain to apply this.++**Located limitation (ingest already on disk, 2026-08-21):** the first `searchbenchwandsgraph` persist wrote title-only ENTITY `search_text` (mapper used title + missing `description` field). Frozen WANDS JSONL rows expose `title` / `class` / `features` and a full `text` blob, but not a separate `description` / `hierarchy` field. **Decision:** new mapper writes prefer `doc["text"]` (the same catalog blob passages embed). Already-persisted nodes do not rewrite until an ENTITY-only `search_text` + `nodes` vector refresh (2000 products, not 32k HAS events, not Celery architect). Overwrite of `search_wands.jsonl` remains blocked. Node BM25 after that refresh is **architecture-demo**, not a quality claim vs passages 0.823.++**Assumption:** many furniture queries are class-like; CLASS∩ATTR intersection may rarely fire on this slice. Fixture tests guarantee the operator; WANDS eval is still valid on union+hybrid. Feature keys are capped (`FEATURE_KEY_CAP=16`) and measure/price tokens are skipped; unique ATTR values across 2000 products still numbered in the thousands.++Structured ingest embeds `properties.search_text` into store `"nodes"` (cache key `uuid:{uuid}`). Nodes without `search_text` still embed **name only** (memory path). Node BM25 (`kg_nodes.search_tsv`, english) is gated on `SEARCH_ENABLED` like chunk DDL. Italian node FTS is not added this round.++---++## Isolated architecture eval++**Located evidence (2026-08-21).** Brain `searchbenchwandsgraph`. `rerank=none`, k=50, n=66. Ledger `claim: architecture-demo`. Do **not** quote vs frozen WANDS passages 0.823 or ESCI 0.500.++Live graph after Celery persist (exact counts): **37383** `kg_nodes`, **64416** `kg_relationships` (32208 HAS triples × 2 EVENT-mid edges). `vectors_relationships` leftover HAS from a killed first run is not a success signal.++Ingest harness timed out twice (`1800s` on batch 1; `10800s` on the remaining-triple parent while Celery was still writing). Persist finished in the worker; communities metrics below are **skip-ingest** search on that completed graph. Do not treat harness timeout as a failed graph write.++| Run | Channels | nDCG@10 | Recall@10 | Recall@50 | p50 retrieve ms (ex-embed) |+| --- | --- | --- | --- | --- | --- |+| `search-wandsgraph-communities-k50` | communities | 0.409 | 0.148 | 0.287 | 5609 |+| `search-wandsgraph-passages-k50` | passages | 0.823 | 0.269 | 0.837 | 99 |++ADR-007 architecture SLO: p50 < 200 ms excluding embed. Passages **99 ms** meets it. Communities **5609 ms** misses it (p50 embed 205 ms recorded separately). SLO check, not a quality win.++Interactions smoke (`channels=["events"]`, `--skip-ingest`, synthetic view/cart/purchase on WANDS `doc_id`s `0` / `10` / `13`) **passed** as pipeline-only (`retrieve/search(wands-0) -> 0 hits`). Not mixed into the communities nDCG ledger row. Not recsys HitRate. No `user_id` on the search body.++---++## Passages first-stage + node_id (located evidence, 2026-08-21)++Skip-ingest only on `searchbenchwandsgraph`. No new architect persist. Ledger `claim: architecture-demo`. Do **not** mix with frozen ESCI 0.500.++- Smoke (`channels=["passages"]`, 2 queries): catalog hits expose `node_id` = `doc_id` = ENTITY uuid (example `7468`); `GET /retrieve/entities/neighbors?uuid=7468` returned **200**. Neighbor `count` may be 0 depending on the neighbors filter; 200 is the round-trip gate.+- Skip-ingest k=50 eval `search-wandsgraph-passages-nodeid-k50`: nDCG@10 **0.823**, Recall@50 **0.837**, p50 retrieve **84 ms** (ex-embed). Same nDCG as paired passages `search-wandsgraph-passages-k50`. All 50 hits on `wands-0` had `node_id` matching DOCID. Join is string parse only.++Graph shape at this check: **37393** `kg_nodes` / **64429** `kg_relationships` (~1996 ENTITY / ~31756 EVENT). Close to the persist snapshot; not a rewrite.++ENTITY-only `search_text` + `nodes` vector refresh (`backfill-entity-text` on `searchbenchwandsgraph`, 2026-08-21): **1996** ENTITY nodes updated, **4** missing, **0** skipped. EVENT count stayed **31756**; total nodes stayed **37393**. Postgres node BM25 for `rubberwood` / `modern` hits product ENTITY `0` (`solid wood platform bed`). `channels=["entities"]` query `rubberwood` returned 200 with `node_id == id`. Refuses `FROZEN_STRUCTURED_BRAINS` (`searchbenchwands` exits 1). **Architecture-demo only** — not a quality claim vs passages 0.823.++---++## Passages first-stage + node_id (decision, 2026-08-21)++Recall@50 0.287 vs 0.837 was **candidate generation**, not ranking. Hub-walk cannot recover gold that sits outside the matched CLASS. Fanout/intersection/HAS-ANN remain valid **graph-channel** hygiene, not the search default. Keep `channels=["communities"]` as an opt-in demo; do not treat 0.409 as a quality gate.++**Out of scope this round:** raising communities nDCG@10 from 0.409; flipping the live default; mixed-ID RRF; `product_id` on `SearchRequestBody` / `SearchHit`; wiping frozen brains; editing `retrieve.py` / `entities.py`; a new 20k-triple architect persist.++---++## Explicitly later++- Fusion of passages + communities in `search.py` (needs this `node_id` join, then rank **products**, not mixed chunk/node RRF).+- Communities fanout/union/HAS-ANN hygiene as opt-in channel quality.+- `/retrieve/context` using catalog node text.+- Quality-default flip.+- Range filters for price/measures; SKU/configurable-option child nodes; mood taxonomies beyond feature strings.+- Learned PKG embeddings (Xu/DREM/PKGM) or GNN recs on `/retrieve/search`.+- Query-gated personalized search is a **separate** workstream, already shipped: [25-personalized-search.md](25-personalized-search.md), [ADR-008](../decisions/008-query-gated-search-personalization.md). Not a first-stage change and not a quality-default flip.diff --git a/docs/research/25-personalized-search.md b/docs/research/25-personalized-search.mdnew file mode 100644index 0000000..b41be86--- /dev/null+++ b/docs/research/25-personalized-search.md@@ -0,0 +1,98 @@+# 25 — Query-gated personalized search++Workstream: optional `target` on `/retrieve/search` reranks **already retrieved** hits. Ledger, if used: `benchmarks.search` only, **architecture demo**. This is not a quality-default flip, not recsys HitRate, and not a claim against frozen WANDS passages nDCG@10 **0.823** or ESCI n=74 **0.500**.++Every claim is labeled **idea**, **assumption**, **prediction**, **located evidence**, or **decision**. Binding contract: [ADR-008](../decisions/008-query-gated-search-personalization.md). Catalog graph / `node_id` join: [24-search-catalog-graph.md](24-search-catalog-graph.md). Recs isolation: [16-recsys-eval-protocol.md](16-recsys-eval-protocol.md).++---++## Focal question++Can BrainAPI personalize product search **without** (a) changing first-stage candidate generation, (b) sharing `/retrieve/recommend` request semantics, or (c) turning inferred taste into a hard `extras` filter?++**Claim this ship can support (decision, 2026-08-21):** omit `target` → anonymous ranking unchanged. Set `target` → query-gated blend of retrieve score vs user ATTR prefs over hits that already have `node_id`. Unknown user and all-zero prefs preserve order. `extras` AND stays user-asserted.++**Claim this ship cannot support:** we beat frozen WANDS 0.823 or ESCI 0.500. Recsys HitRate on `demorecsys` is search nDCG. HEM/TEM-class query–user–item training. Flipping live default `channels=["passages"]`. Using `user_id` on `SearchRequestBody`.++---++## Architecture (decision)++```text+query+ → first-stage (default: passages BM25∪dense)+ → extras AND (user-asserted only)+ → node_id attach (DOCID parse / graph hit id)+ → if target set: score given node_ids vs user prefs, blend with λ(query)+ → cut to k+```++- **Identity is `target`, not `user_id` (decision).** Same field name as `/retrieve/recommend`. GET `/retrieve/search?target=` matches the body. Schema tests ban `user_id` / `product_id` / `sku` / `brand` on the search body.+- **Rerank, not candidate generation (decision).** Do not call `EntityRecommendRetriever.recommend()`. Communities as first-stage already failed on `searchbenchwandsgraph` (Recall@50 **0.287** vs passages **0.837**; [24](24-search-catalog-graph.md)). Personalize scores only the retrieved `node_id`s.+- **Query λ (decision).** Digits or SKU-like tokens → 0 (navigational / specific). One content token ≈ 0.85; two ≈ 0.5; three ≈ 0.25; four+ ≈ 0.1. Blend is min-max retrieve vs pref; λ=0 or all-zero prefs preserve retrieve order and never drop ids.+- **Pref vector (decision).** `user_pref_weights` sums write-time `USER -PREFERS-> ATTR` (long-term) **and** recency-decayed `USER → EVENT → ITEM → ATTR` (short-term, 14-day half-life). Search does **not** skip events when PREFERS exist — unlike recommend’s `_attribute_pref_targets`, which still does. Treat ENTITY and PRODUCT as items.+- **Write path (decision).** Dated view / cart / purchase / favorite stay EVENT wrappers. Selected `options` / `attributes` emit `USER -PREFERS-> ATTR` with the **same hub uuid** as catalog HAS (`hub:attr:70s`). Options are not merged into the product `catalog_doc` (would stamp the SKU). Catalog `brand` / `color` / `class` on the row still PRODUCT-HAS. Architect persists any no-event direct triple (`HAS`, `PREFERS`, later `AVOIDS`), not only static HAS.+- **I4 weights (decision).** Favorite / wishlist **0.7**; unknown behaviors **0.2** (was 1.0). Shared by `src/core/search/recommend.py` and `plugins/features-rec/models/mapping.py`.+- **I7 rejected (decision).** Inferred style is blend-only. Do not write `extras={style: 70s}` from history. User-asserted `extras` remains a hard AND filter, applied **before** personalize.+- **SLO (decision).** Omitted-`target` path is the ADR-007 200 ms default. Personalized path is opt-in; do not cite its latency as that SLO.+- **Hits without `node_id` stay at pref 0.** Catalog mode personalizes `k_ret` then cuts to request `k`.++Code: [`src/core/search/personalize.py`](../../src/core/search/personalize.py), wired in [`src/services/api/controllers/search.py`](../../src/services/api/controllers/search.py) after extras / `node_id`. Mapper: [`src/core/search/catalog_graph.py`](../../src/core/search/catalog_graph.py) `prefers_triple` / `interaction_to_triples`.++---++## Literature (located evidence, 2026-08-21)++**Support-located for query-dependent personalization (not always-on):**++- Ai, Hill, Vishwanathan, Croft, *A Zero Attention Model for Personalized Product Search*, arXiv `1908.11322` (CIKM 2019) — commercial logs: personalization **depends on query characteristics**; a zero vector lets the model pay **no** attention to history. Maps to λ=0 on specific / digit queries, not to training ZAM.+- Bi, Ai, Croft, *A Transformer-based Embedding Model for Personalized Product Search*, arXiv `2005.08936` (SIGIR 2020) — TEM: “personalization does not always improve product search quality”; undifferentiated personalization is the failure mode. Maps to query-gated blend. **Does not** authorize training TEM here.++**Challenge-located / deferred (learned query–user–item first-stage):**++- Ai, Zhang, Bi, Chen, Croft, *Learning a Hierarchical Embedding Model for Personalized Product Search*, SIGIR 2017, DOI `10.1145/3077136.3080813` — HEM jointly embeds query, user, item from purchase language. Needs query×user logs. **I6 deferred.**+- Ai et al. DREM, arXiv `1909.07212` — learned user–item KG for search; not ATTR-hub overlap on a retrieved list. Already bounded in [24](24-search-catalog-graph.md).+- Nigam et al., *Semantic Product Search*, arXiv `1907.00937` — first-stage remains query → document. Personalize does not replace it.++**Wrong metric if cited as search quality:**++- Recsys next-item HitRate on `demorecsys` via `/retrieve/recommend` ([16](16-recsys-eval-protocol.md)) is a different API, task, brain, and label. WANDS / ESCI have no users.++---++## Protocol (decision)++| Knob | Value |+| --- | --- |+| Identity | Optional `target`; omit = control |+| First-stage | Unchanged; default omitted `channels=["passages"]` |+| Filter | User-asserted `extras` AND, then personalize |+| Unit tests | `tests/test_search_personalize.py` (λ, blend, PREFERS+recency, extras AND, schema ban) |+| Optional live smoke | skip-ingest on `searchbenchwandsgraph` only; fixture `benchmarks/data/search_personalize_smoke.jsonl` (`wands-u70`, favorite + `style=70s` options). Mapping unit-tested; live ingest is maintainer-gated |+| Ledger claim | If ledgered: `architecture-demo` only |++```bash+# Mapping-only fixture (gitignored under benchmarks/data/*). Compare omitted+# target vs target=wands-u70 on a generic query (lamp) vs a specific query+# (oak dining table 180cm). Skip-ingest on searchbenchwandsgraph after+# maintainer OK; do not wipe or re-architect.+# Fixture: benchmarks/data/search_personalize_smoke.jsonl+```++**Go/no-go for a later default (not executed):** personalized nDCG on a query×user labeled set beats omitted-`target` without regressing specific queries. WANDS/ESCI cannot run that gate. Do not quote vs frozen 0.823 / 0.500.++---++## Isolation++Never wipe `searchbenchwands`, `searchbenchesci74`, `searchbenchescies`, `searchbenchesciltr2`, `locomoconv*`, `beam*`, `demorecsys`. Optional smoke only on `searchbenchwandsgraph`. Do not score recsys HitRate as search nDCG. Do not mix architecture-demo numbers with frozen WANDS / ESCI quality rows.++---++## Explicitly later++- **I6 deferred:** HEM / TEM / ZAM training. Needs real query+user logs, not WANDS. Not in this ship.+- Learned query-conditioned attention (ZAM/TEM) instead of the token-count λ heuristic.+- Recommend `_attribute_pref_targets` combining PREFERS + EVENT (search already does; recs still skip EVENT when PREFERS exist).+- `AVOIDS` polarity as a direct edge.+- Using personalized search latency as the ADR-007 200 ms product claim.