Skip to content

#1 Persistent memory for AI coding agents based on real-world benchmarks

bench: add LongMemEval-S benchmark (95.2% R@5 hybrid, 86.2% BM25) (#91)

Rohit Ghumare565c2388a2ac
diff --git a/benchmark/LONGMEMEVAL.md b/benchmark/LONGMEMEVAL.md
new file mode 100644
index 00000000..00de938e
--- /dev/null
+++ b/benchmark/LONGMEMEVAL.md
@@ -0,0 +1,79 @@
+# LongMemEval-S Benchmark Results
+
+[LongMemEval](https://arxiv.org/abs/2410.10813) (ICLR 2025) is an academic benchmark for evaluating long-term memory in chat assistants. It tests 5 core abilities: information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention.
+
+## Setup
+
+- **Dataset**: LongMemEval-S (500 questions, ~48 sessions per question, ~115K tokens)
+- **Source**: [xiaowu0162/longmemeval-cleaned](https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned)
+- **Metric**: `recall_any@K` — does ANY gold session appear in top-K retrieved results?
+- **Embedding model**: `all-MiniLM-L6-v2` (384 dimensions, local, no API key)
+- **No LLM in the loop**: Pure retrieval evaluation, no answer generation or judge
+
+## Results
+
+| System | R@5 | R@10 | R@20 | NDCG@10 | MRR |
+|---|---|---|---|---|---|
+| **agentmemory BM25+Vector** | **95.2%** | **98.6%** | **99.4%** | **87.9%** | **88.2%** |
+| agentmemory BM25-only | 86.2% | 94.6% | 98.6% | 73.0% | 71.5% |
+| MemPalace raw (vector-only) | 96.6% | ~97.6% | — | — | — |
+
+### By Question Type (BM25+Vector)
+
+| Type | R@5 | R@10 | Count |
+|---|---|---|---|
+| knowledge-update | 98.7% | 100.0% | 78 |
+| multi-session | 97.7% | 100.0% | 133 |
+| single-session-assistant | 96.4% | 98.2% | 56 |
+| temporal-reasoning | 95.5% | 97.7% | 133 |
+| single-session-user | 90.0% | 97.1% | 70 |
+| single-session-preference | 83.3% | 96.7% | 30 |
+
+### By Question Type (BM25-only)
+
+| Type | R@5 | R@10 | Count |
+|---|---|---|---|
+| knowledge-update | 92.3% | 98.7% | 78 |
+| single-session-user | 91.4% | 95.7% | 70 |
+| temporal-reasoning | 88.0% | 94.7% | 133 |
+| multi-session | 86.5% | 96.2% | 133 |
+| single-session-assistant | 80.4% | 91.1% | 56 |
+| single-session-preference | 60.0% | 80.0% | 30 |
+
+## Analysis
+
+1. **BM25+Vector (95.2%) nearly matches pure vector search (96.6%)** with only a 1.4pp gap. Both use the same embedding model (all-MiniLM-L6-v2).
+
+2. **BM25 alone gets 86.2%** — keyword search with Porter stemming and synonym expansion is surprisingly effective on conversational data.
+
+3. **Adding vectors to BM25 gives +9pp** (86.2% → 95.2%), the largest improvement from any single component.
+
+4. **Preferences are the hardest category** for both BM25 (60%) and hybrid (83.3%). These require understanding implicit/indirect statements.
+
+5. **Multi-session and knowledge-update are strongest** (97.7%+ hybrid). The hybrid approach excels when facts are distributed across sessions.
+
+6. **R@10 reaches 98.6%** — nearly all gold sessions are found within the top 10 results.
+
+## Important Notes on Methodology
+
+- These are **retrieval recall** scores, not end-to-end QA accuracy. The official LongMemEval metric is QA accuracy (retrieve + generate answer + GPT-4o judge).
+- Systems on the actual LongMemEval QA leaderboard score 60-95% depending on the LLM reader (Oracle GPT-4o gets ~82.4%).
+- We do NOT claim these as "LongMemEval scores" — they are retrieval-only evaluations on the LongMemEval-S haystack.
+- Each question builds a fresh index from its ~48 sessions, searches with the question text, and checks if gold session IDs appear in results.
+
+## Reproducibility
+
+```bash
+# Download dataset (264 MB)
+pip install huggingface_hub
+python3 -c "
+from huggingface_hub import hf_hub_download
+hf_hub_download(repo_id='xiaowu0162/longmemeval-cleaned', filename='longmemeval_s_cleaned.json', repo_type='dataset', local_dir='benchmark/data')
+"
+
+# Run BM25-only
+npx tsx benchmark/longmemeval-bench.ts bm25
+
+# Run BM25+Vector hybrid (requires @xenova/transformers)
+npx tsx benchmark/longmemeval-bench.ts hybrid
+```
diff --git a/benchmark/data/.gitignore b/benchmark/data/.gitignore
new file mode 100644
index 00000000..baae31f7
--- /dev/null
+++ b/benchmark/data/.gitignore
@@ -0,0 +1 @@
+longmemeval_s_cleaned.json
diff --git a/benchmark/data/longmemeval_results_bm25.json b/benchmark/data/longmemeval_results_bm25.json
new file mode 100644
index 00000000..eaaaba21
--- /dev/null
+++ b/benchmark/data/longmemeval_results_bm25.json
@@ -0,0 +1,12491 @@
+{
+ "mode": "bm25",
+ "questions": 500,
+ "recall_any_at_5": 0.862,
+ "recall_any_at_10": 0.946,
+ "recall_any_at_20": 0.986,
+ "ndcg_at_10": 0.7302373205176367,
+ "mrr": 0.7153796961208726,
+ "per_type": {
+ "single-session-user": {
+ "count": 70,
+ "recall_any_at_5": 0.9142857142857143,
+ "recall_any_at_10": 0.9571428571428572
+ },
+ "multi-session": {
+ "count": 133,
+ "recall_any_at_5": 0.8646616541353384,
+ "recall_any_at_10": 0.9624060150375939
+ },
+ "single-session-preference": {
+ "count": 30,
+ "recall_any_at_5": 0.6,
+ "recall_any_at_10": 0.8
+ },
+ "temporal-reasoning": {
+ "count": 133,
+ "recall_any_at_5": 0.8796992481203008,
+ "recall_any_at_10": 0.9473684210526315
+ },
+ "knowledge-update": {
+ "count": 78,
+ "recall_any_at_5": 0.9230769230769231,
+ "recall_any_at_10": 0.9871794871794872
+ },
+ "single-session-assistant": {
+ "count": 56,
+ "recall_any_at_5": 0.8035714285714286,
+ "recall_any_at_10": 0.9107142857142857
+ }
+ },
+ "per_question": [
+ {
+ "question_id": "e47becba",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_280352e9",
+ "02bd2b90_3",
+ "ultrachat_214101",
+ "5dac7cc2_1",
+ "ultrachat_336116",
+ "f6859b48_2",
+ "sharegpt_Cr2tc1f_0",
+ "41abc171_4",
+ "3b3a77d9",
+ "aae4411b_2"
+ ],
+ "gold_session_ids": [
+ "answer_280352e9"
+ ]
+ },
+ {
+ "question_id": "118b2229",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_40a90d51",
+ "728deb4d_5",
+ "5209e813_2",
+ "db50c0f6",
+ "db73b7e4_4",
+ "628c958c_7",
+ "5dce60dd",
+ "ultrachat_311002",
+ "sharegpt_dabXXRO_10",
+ "ultrachat_392094"
+ ],
+ "gold_session_ids": [
+ "answer_40a90d51"
+ ]
+ },
+ {
+ "question_id": "51a45a95",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_d61669c7",
+ "98c198fb",
+ "5c44d9fe_1",
+ "07ba9acd_2",
+ "55e0c6db_2",
+ "ultrachat_56839",
+ "217debf7",
+ "sharegpt_hChsWOp_128",
+ "83addb61_4",
+ "6e0b1800_2"
+ ],
+ "gold_session_ids": [
+ "answer_d61669c7"
+ ]
+ },
+ {
+ "question_id": "58bf7951",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_355c48bb",
+ "sharegpt_gmLF3PE_17",
+ "ultrachat_304942",
+ "ultrachat_268434",
+ "793382f9_2",
+ "ultrachat_221066",
+ "ultrachat_278141",
+ "sharegpt_CvVLg2J_18",
+ "07b7a667_1",
+ "1587bd78_1"
+ ],
+ "gold_session_ids": [
+ "answer_355c48bb"
+ ]
+ },
+ {
+ "question_id": "1e043500",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_3e012175",
+ "01bd9def",
+ "be324b2e_1",
+ "3548cae3",
+ "d8a6c437",
+ "4d0d7984",
+ "sharegpt_6QUDIXG_24",
+ "sharegpt_uQMdL9t_0",
+ "sharegpt_85Y0l0J_365",
+ "c285b705_2"
+ ],
+ "gold_session_ids": [
+ "answer_3e012175"
+ ]
+ },
+ {
+ "question_id": "c5e8278d",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_f6168136",
+ "42e69993_2",
+ "aaf71ce2_2",
+ "sharegpt_DB8U4oH_9",
+ "f58317c6",
+ "d777e7da_1",
+ "bdf735de_2",
+ "dc832f73",
+ "193c23bd_2",
+ "e4881cbd_4"
+ ],
+ "gold_session_ids": [
+ "answer_f6168136"
+ ]
+ },
+ {
+ "question_id": "6ade9755",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_9398da02",
+ "sharegpt_BghMUHh_0",
+ "910ed479_3",
+ "0b64c6cb_1",
+ "ec8fa65d_2",
+ "7c5ab02a_2",
+ "ultrachat_329610",
+ "1ddc89e9_3",
+ "ultrachat_84599",
+ "sharegpt_YsgacQi_0"
+ ],
+ "gold_session_ids": [
+ "answer_9398da02"
+ ]
+ },
+ {
+ "question_id": "6f9b354f",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_feb5200f",
+ "sharegpt_BYHT3Xz_0",
+ "ultrachat_549317",
+ "sharegpt_VAELSB3_0",
+ "60207b41_1",
+ "sharegpt_IuLT3VI_9",
+ "a79f1a04_1",
+ "54d06eeb",
+ "36828c66_2",
+ "sharegpt_kRjBUsA_79"
+ ],
+ "gold_session_ids": [
+ "answer_feb5200f"
+ ]
+ },
+ {
+ "question_id": "58ef2f1c",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_59547700",
+ "13f3da23_3",
+ "e6ab6a7b_2",
+ "ultrachat_5212",
+ "e82ce7cd_1",
+ "ultrachat_246324",
+ "b17c7f18",
+ "ultrachat_483737",
+ "ultrachat_152126",
+ "sharegpt_eHZgwR0_9"
+ ],
+ "gold_session_ids": [
+ "answer_59547700"
+ ]
+ },
+ {
+ "question_id": "f8c5f88b",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_c3567066",
+ "sharegpt_TKj3yO8_0",
+ "15232887_1",
+ "68ace657_1",
+ "91880423",
+ "58de8f9b",
+ "54e9197a",
+ "7ff89315_3",
+ "sharegpt_n2aGNgW_9",
+ "121efb3b_1"
+ ],
+ "gold_session_ids": [
+ "answer_c3567066"
+ ]
+ },
+ {
+ "question_id": "5d3d2817",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 0,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0,
+ "mrr": 0.0625,
+ "retrieved_session_ids": [
+ "85d16d01",
+ "4ae45145",
+ "sharegpt_mtS0qOD_65",
+ "sharegpt_ipLglky_48",
+ "e7b0637e_3",
+ "7dc6f276_1",
+ "72f053da_2",
+ "4785e1f2_1",
+ "25d1830a_1",
+ "670ca40e"
+ ],
+ "gold_session_ids": [
+ "answer_235eb6fb"
+ ]
+ },
+ {
+ "question_id": "7527f7e2",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_7cb94507",
+ "sharegpt_cRrfdsc_0",
+ "18a06652_3",
+ "d81d9846",
+ "sharegpt_ToxFax1_535",
+ "864a563d_3",
+ "4e2acecc",
+ "ultrachat_62919",
+ "ultrachat_389020",
+ "60147f90_1"
+ ],
+ "gold_session_ids": [
+ "answer_7cb94507"
+ ]
+ },
+ {
+ "question_id": "c960da58",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_e05e4612",
+ "f7ffddbc",
+ "124f5dc3_2",
+ "sharegpt_9iUZNsL_0",
+ "b65f51fb_5",
+ "ultrachat_374640",
+ "89d3053f",
+ "sharegpt_ErOTMZ3_59",
+ "52381716_1",
+ "sharegpt_e328KPR_0"
+ ],
+ "gold_session_ids": [
+ "answer_e05e4612"
+ ]
+ },
+ {
+ "question_id": "3b6f954b",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6309297535714575,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "c271c071",
+ "answer_94030872",
+ "1f0c62de",
+ "c566edbb_3",
+ "6d3f5c68_4",
+ "34590cab_1",
+ "f999b05c_2",
+ "sharegpt_opQbgPL_7",
+ "e27891d3_2",
+ "sharegpt_kpnwrGe_12"
+ ],
+ "gold_session_ids": [
+ "answer_94030872"
+ ]
+ },
+ {
+ "question_id": "726462e0",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 0,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0,
+ "mrr": 0.09090909090909091,
+ "retrieved_session_ids": [
+ "9c05acfe_2",
+ "af2d26f1_1",
+ "741c6781_1",
+ "29e76903_1",
+ "ce144f08",
+ "b84b7a2b_1",
+ "082889ba_1",
+ "5b83c26e_1",
+ "6ef99651_2",
+ "322a96e8"
+ ],
+ "gold_session_ids": [
+ "answer_f38f679b"
+ ]
+ },
+ {
+ "question_id": "94f70d80",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_c63c0458",
+ "ultrachat_397420",
+ "sharegpt_PCqCVmR_18",
+ "2479cbae",
+ "67045503",
+ "09b651aa_3",
+ "58de8f9b",
+ "eee3f7d3",
+ "sharegpt_WQr7EZT_0",
+ "sharegpt_mJ7MnQS_31"
+ ],
+ "gold_session_ids": [
+ "answer_c63c0458"
+ ]
+ },
+ {
+ "question_id": "66f24dbb",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_fea2e4d3",
+ "89452526_2",
+ "8d015d9d_4",
+ "7732b1cc",
+ "ca3a4e4f_1",
+ "7a10d327_2",
+ "798e4ba3_2",
+ "8d46b8fa_1",
+ "b6858c2a_2",
+ "ultrachat_428656"
+ ],
+ "gold_session_ids": [
+ "answer_fea2e4d3"
+ ]
+ },
+ {
+ "question_id": "ad7109d1",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_679840f8",
+ "a3d2af51_2",
+ "sharegpt_RzNjBHR_0",
+ "3dc02d26_2",
+ "495cf518_1",
+ "3fa35683_5",
+ "ultrachat_74974",
+ "ultrachat_202091",
+ "7ad0bd25_1",
+ "1d6744b5_1"
+ ],
+ "gold_session_ids": [
+ "answer_679840f8"
+ ]
+ },
+ {
+ "question_id": "af8d2e46",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_82a04f59",
+ "d5a6f455_2",
+ "32a67b35",
+ "ced213c6",
+ "4ebc182b",
+ "82a89303",
+ "73d5f10c",
+ "43668d77_1",
+ "db6ddaba",
+ "6f341f96_3"
+ ],
+ "gold_session_ids": [
+ "answer_82a04f59"
+ ]
+ },
+ {
+ "question_id": "dccbc061",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_8f276838",
+ "0b8a4a9d_1",
+ "25d1830a_2",
+ "5fc843cd",
+ "8d74c5f6",
+ "ultrachat_243849",
+ "0f417163",
+ "4af27eab_2",
+ "657ab20c_1",
+ "ultrachat_520430"
+ ],
+ "gold_session_ids": [
+ "answer_8f276838"
+ ]
+ },
+ {
+ "question_id": "c8c3f81d",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_761acef8",
+ "sharegpt_pRqHb1o_0",
+ "19ec83c5_1",
+ "66e94928",
+ "2b16f3d6_1",
+ "3c0ed3bf_4",
+ "b818e2a1_1",
+ "9bec5593_1",
+ "25432e10",
+ "e89b6be2_2"
+ ],
+ "gold_session_ids": [
+ "answer_761acef8"
+ ]
+ },
+ {
+ "question_id": "8ebdbe50",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_8ad8a34f",
+ "b53f447a_1",
+ "sharegpt_mJ7MnQS_31",
+ "27d378b2_3",
+ "90f16f14_2",
+ "01e806f0_1",
+ "d4f7a065_2",
+ "03b4d5d8_2",
+ "326f41d8",
+ "ultrachat_194514"
+ ],
+ "gold_session_ids": [
+ "answer_8ad8a34f"
+ ]
+ },
+ {
+ "question_id": "6b168ec8",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.5,
+ "mrr": 0.3333333333333333,
+ "retrieved_session_ids": [
+ "sharegpt_D9kTAIU_0",
+ "dded1725_3",
+ "answer_e623ae87",
+ "ultrachat_414934",
+ "fb2368b8_3",
+ "ultrachat_473",
+ "258645ba_1",
+ "d58e8baa",
+ "afffd8ec_3",
+ "ultrachat_377741"
+ ],
+ "gold_session_ids": [
+ "answer_e623ae87"
+ ]
+ },
+ {
+ "question_id": "75499fd8",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_723bf11f",
+ "5328c3c2_2",
+ "ultrachat_428593",
+ "ultrachat_360018",
+ "3b028282",
+ "168feef7_1",
+ "ultrachat_189069",
+ "sharegpt_jBafAKm_0",
+ "sharegpt_mMNr1qm_0",
+ "ultrachat_466236"
+ ],
+ "gold_session_ids": [
+ "answer_723bf11f"
+ ]
+ },
+ {
+ "question_id": "21436231",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_1e6d4567",
+ "3fa35683_6",
+ "7a7a4bf0_4",
+ "0d0a89dc_1",
+ "b307196d",
+ "6bd0b406_1",
+ "d0785ee9_1",
+ "a0f8468c_3",
+ "4bebc783_2",
+ "2fd8bf73_1"
+ ],
+ "gold_session_ids": [
+ "answer_1e6d4567"
+ ]
+ },
+ {
+ "question_id": "95bcc1c8",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_cb742a61",
+ "99e5e380",
+ "f64c4793_2",
+ "sharegpt_hRba933_47",
+ "abeda028",
+ "e201572a",
+ "1c1f5ccc_2",
+ "c9763bff",
+ "sharegpt_J3KJuOr_0",
+ "56266c38_1"
+ ],
+ "gold_session_ids": [
+ "answer_cb742a61"
+ ]
+ },
+ {
+ "question_id": "0862e8bf",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6309297535714575,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "sharegpt_MTUnqho_0",
+ "answer_c6fd8ebd",
+ "ultrachat_240637",
+ "10541a2c_1",
+ "ultrachat_518141",
+ "sharegpt_WhKs0Le_0",
+ "95c36d37",
+ "sharegpt_2I1TxKW_0",
+ "ultrachat_357044",
+ "sharegpt_3BkB8g5_0"
+ ],
+ "gold_session_ids": [
+ "answer_c6fd8ebd"
+ ]
+ },
+ {
+ "question_id": "853b0a1d",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_69811d4a",
+ "c70f9f9c",
+ "1fdbdfff_5",
+ "ebb5bc7c_2",
+ "c3feb3bb",
+ "sharegpt_i8xL0ih_109",
+ "ultrachat_378413",
+ "f71bf532_1",
+ "sharegpt_n3MgsgL_0",
+ "sharegpt_hRba933_47"
+ ],
+ "gold_session_ids": [
+ "answer_69811d4a"
+ ]
+ },
+ {
+ "question_id": "a06e4cfe",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_6fe9fb49",
+ "84fb50bb_3",
+ "af257b0b_2",
+ "sharegpt_e1JmYDm_0",
+ "0c891c0b_1",
+ "9de2c795",
+ "ultrachat_113982",
+ "8dd2d3cc",
+ "de64539a_2",
+ "sharegpt_3swiY8I_36"
+ ],
+ "gold_session_ids": [
+ "answer_6fe9fb49"
+ ]
+ },
+ {
+ "question_id": "37d43f65",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_55161935",
+ "1cafd864_3",
+ "fec04f53_2",
+ "3ab79434_2",
+ "sharegpt_brc2wJS_264",
+ "4050ebff_5",
+ "ultrachat_370801",
+ "ultrachat_10500",
+ "sharegpt_MOpCbrr_0",
+ "ffe473f3_1"
+ ],
+ "gold_session_ids": [
+ "answer_55161935"
+ ]
+ },
+ {
+ "question_id": "b86304ba",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.3333333333333333,
+ "mrr": 0.14285714285714285,
+ "retrieved_session_ids": [
+ "sharegpt_E0YL5SX_142",
+ "e831a29f_1",
+ "fe1e4351_1",
+ "c734e31e_1",
+ "ultrachat_328696",
+ "91074ce7",
+ "answer_645b0329",
+ "8e9425ef_1",
+ "sharegpt_xGoJZ6Z_0",
+ "c1634164_2"
+ ],
+ "gold_session_ids": [
+ "answer_645b0329"
+ ]
+ },
+ {
+ "question_id": "d52b4f67",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_0df6aa4b",
+ "62a6d083",
+ "sharegpt_n6Ej34Y_75",
+ "d31c3ec8_3",
+ "ultrachat_258008",
+ "18a06652_6",
+ "ultrachat_127631",
+ "c578da1f_1",
+ "29545df8_1",
+ "da1797c4_1"
+ ],
+ "gold_session_ids": [
+ "answer_0df6aa4b"
+ ]
+ },
+ {
+ "question_id": "25e5aa4f",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_986de8c3",
+ "18807892_4",
+ "8953c8c8_1",
+ "91e581ad_1",
+ "7a389bd1_2",
+ "1de862d6_2",
+ "a459d477",
+ "sharegpt_8cFNcVG_0",
+ "478e02c2",
+ "b049dbb8_2"
+ ],
+ "gold_session_ids": [
+ "answer_986de8c3"
+ ]
+ },
+ {
+ "question_id": "caf9ead2",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.31546487678572877,
+ "mrr": 0.125,
+ "retrieved_session_ids": [
+ "86974d49",
+ "f8ba3f69",
+ "7ae303b7_1",
+ "58dd3f35",
+ "ultrachat_532107",
+ "fc167d00",
+ "sharegpt_3D3oQC0_213",
+ "answer_0714183a",
+ "48a72204_1",
+ "ultrachat_495546"
+ ],
+ "gold_session_ids": [
+ "answer_0714183a"
+ ]
+ },
+ {
+ "question_id": "8550ddae",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_c8354ae9",
+ "a7189b75",
+ "71cf5aeb_3",
+ "9deebbc2_2",
+ "8ec23b2c",
+ "3b13915f",
+ "sharegpt_H8FrXvr_25",
+ "7b78417e_1",
+ "4f234e2c",
+ "e10dfffb_1"
+ ],
+ "gold_session_ids": [
+ "answer_c8354ae9"
+ ]
+ },
+ {
+ "question_id": "60d45044",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_9cddca88",
+ "795a13a7",
+ "a76e7e3c_2",
+ "35a61ccd_1",
+ "e5713180_1",
+ "4ebf9095_1",
+ "025c9e24_1",
+ "af4d4ecd_2",
+ "fec059c5",
+ "f90c44e6_2"
+ ],
+ "gold_session_ids": [
+ "answer_9cddca88"
+ ]
+ },
+ {
+ "question_id": "3f1e9474",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_57fc1954",
+ "3aac691d_2",
+ "e7497c07_2",
+ "954d4451_2",
+ "4d408e9d",
+ "4f23a396_1",
+ "cdf068b1_1",
+ "ultrachat_393184",
+ "ultrachat_177477",
+ "ultrachat_462827"
+ ],
+ "gold_session_ids": [
+ "answer_57fc1954"
+ ]
+ },
+ {
+ "question_id": "86b68151",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_dc11c1eb",
+ "0abaca24_1",
+ "sharegpt_7rLq4MQ_9",
+ "d1bd26fb_2",
+ "3d72c0c0",
+ "ultrachat_522663",
+ "ultrachat_121159",
+ "4e671700",
+ "d8454317_4",
+ "e200d96c_2"
+ ],
+ "gold_session_ids": [
+ "answer_dc11c1eb"
+ ]
+ },
+ {
+ "question_id": "577d4d32",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.5,
+ "mrr": 0.3333333333333333,
+ "retrieved_session_ids": [
+ "ultrachat_125084",
+ "a126eeab_3",
+ "answer_0dd4d99a",
+ "2ef53f61_2",
+ "ultrachat_57579",
+ "b283f9eb_1",
+ "6e2cca63_2",
+ "dc700e27",
+ "92a6a6f7_2",
+ "sharegpt_101i5OR_0"
+ ],
+ "gold_session_ids": [
+ "answer_0dd4d99a"
+ ]
+ },
+ {
+ "question_id": "ec81a493",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ed1982fc",
+ "ab4643a2_4",
+ "d2fc150a_1",
+ "ultrachat_27488",
+ "6f55019d",
+ "ultrachat_447961",
+ "90fea364_3",
+ "616fdd27",
+ "ultrachat_413665",
+ "6c29f175"
+ ],
+ "gold_session_ids": [
+ "answer_ed1982fc"
+ ]
+ },
+ {
+ "question_id": "15745da0",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_586de428",
+ "07d33eaa",
+ "0d73afc8",
+ "ultrachat_155656",
+ "768544e1",
+ "sharegpt_f28uI6i_0",
+ "64a97d9b",
+ "sharegpt_KUXG6gY_2",
+ "a4f22269",
+ "sharegpt_iF4hlNs_5"
+ ],
+ "gold_session_ids": [
+ "answer_586de428"
+ ]
+ },
+ {
+ "question_id": "e01b8e2f",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.43067655807339306,
+ "mrr": 0.25,
+ "retrieved_session_ids": [
+ "d4cdb464",
+ "97338ddd_1",
+ "8acfa731",
+ "answer_5ca6cd28",
+ "1aaf8057_3",
+ "ultrachat_185433",
+ "ultrachat_556304",
+ "sharegpt_9s78QGH_0",
+ "b2543a58_3",
+ "ultrachat_433104"
+ ],
+ "gold_session_ids": [
+ "answer_5ca6cd28"
+ ]
+ },
+ {
+ "question_id": "bc8a6e93",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.5,
+ "mrr": 0.3333333333333333,
+ "retrieved_session_ids": [
+ "f62bfe2b_1",
+ "316dc9f8",
+ "answer_e6143162",
+ "sharegpt_qyWsgp8_13",
+ "552c7fd0",
+ "c6bed037_1",
+ "sharegpt_Us25gXO_18",
+ "ultrachat_133339",
+ "413b57cb_4",
+ "ultrachat_453950"
+ ],
+ "gold_session_ids": [
+ "answer_e6143162"
+ ]
+ },
+ {
+ "question_id": "ccb36322",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.38685280723454163,
+ "mrr": 0.2,
+ "retrieved_session_ids": [
+ "04a0b385",
+ "bdcee74e_3",
+ "ultrachat_395168",
+ "940629b2_3",
+ "answer_f1fbb330",
+ "f79a48dd_1",
+ "06d1a1a0",
+ "facfdfe2",
+ "162ff451",
+ "c38b33ae"
+ ],
+ "gold_session_ids": [
+ "answer_f1fbb330"
+ ]
+ },
+ {
+ "question_id": "001be529",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_530960c1",
+ "0f71db75_1",
+ "a244d459",
+ "f067b0ee",
+ "ultrachat_241736",
+ "ed9dad6c_1",
+ "sharegpt_6cz1Sq6_328",
+ "de877349",
+ "sharegpt_PKXxlMF_0",
+ "fb793c87_2"
+ ],
+ "gold_session_ids": [
+ "answer_530960c1"
+ ]
+ },
+ {
+ "question_id": "b320f3f8",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_5cc9b056",
+ "aae4411b_2",
+ "b89f68c4_4",
+ "1b71c896_2",
+ "40628271_1",
+ "sharegpt_4FCMpJR_0",
+ "ultrachat_151745",
+ "f2797fac",
+ "8bb8bc0d",
+ "514a7856_2"
+ ],
+ "gold_session_ids": [
+ "answer_5cc9b056"
+ ]
+ },
+ {
+ "question_id": "19b5f2b3",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6309297535714575,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "283e5c7e_1",
+ "answer_5ff494b9",
+ "275804a8_3",
+ "9970ac13_2",
+ "sharegpt_I0Wgney_11",
+ "ultrachat_447485",
+ "51adc57e_2",
+ "ultrachat_302261",
+ "ultrachat_268389",
+ "b951f8c6_2"
+ ],
+ "gold_session_ids": [
+ "answer_5ff494b9"
+ ]
+ },
+ {
+ "question_id": "4fd1909e",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_2952aee4",
+ "94c582bb_2",
+ "sharegpt_hO4FoVt_0",
+ "ad2135af",
+ "2479cbae",
+ "ultrachat_472809",
+ "sharegpt_I2cfkPp_0",
+ "sharegpt_xX1L0kg_29",
+ "ultrachat_59418",
+ "fef185b1"
+ ],
+ "gold_session_ids": [
+ "answer_2952aee4"
+ ]
+ },
+ {
+ "question_id": "545bd2b5",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_47ffab4c",
+ "b7f69a7d_1",
+ "5fb3b5ac",
+ "d1a1b9ea_1",
+ "cc021f81_2",
+ "cf8e352f_1",
+ "a79f1a04_1",
+ "1351505a_1",
+ "9f9384ed_1",
+ "700c16e2_1"
+ ],
+ "gold_session_ids": [
+ "answer_47ffab4c"
+ ]
+ },
+ {
+ "question_id": "8a137a7f",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_15d63a22",
+ "77908e43_1",
+ "64460e9d_1",
+ "sharegpt_o1SCEXR_23",
+ "e05e4612",
+ "ultrachat_90882",
+ "7738b24b",
+ "ultrachat_44939",
+ "e3e66b50",
+ "sharegpt_W35IQm3_0"
+ ],
+ "gold_session_ids": [
+ "answer_15d63a22"
+ ]
+ },
+ {
+ "question_id": "76d63226",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_bbdc7b0a",
+ "sharegpt_79vV0aT_377",
+ "cb289226_1",
+ "sharegpt_sRhcMiu_13",
+ "sharegpt_Wi7Op1u_91",
+ "6a2f9452_2",
+ "b0729ec8_1",
+ "5f73f2f3",
+ "50136b31",
+ "adc47df1_1"
+ ],
+ "gold_session_ids": [
+ "answer_bbdc7b0a"
+ ]
+ },
+ {
+ "question_id": "86f00804",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_96b8c9ee",
+ "sharegpt_ErOTMZ3_277",
+ "6e672b84_3",
+ "35c5419d_abs_3",
+ "137d8c6a_2",
+ "sharegpt_sCFErnY_22",
+ "6339aa1d_2",
+ "0424a40a_2",
+ "sharegpt_PQ71V9I_7",
+ "d98f48f8_3"
+ ],
+ "gold_session_ids": [
+ "answer_96b8c9ee"
+ ]
+ },
+ {
+ "question_id": "8e9d538c",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_7bdcbd23",
+ "sharegpt_tb5H6IH_3",
+ "ultrachat_570726",
+ "ultrachat_419107",
+ "2deed26f",
+ "ultrachat_169644",
+ "6aaf298d",
+ "654da2ba_1",
+ "ultrachat_26072",
+ "27d18d38"
+ ],
+ "gold_session_ids": [
+ "answer_7bdcbd23"
+ ]
+ },
+ {
+ "question_id": "311778f1",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_e40b054e",
+ "sharegpt_vXNQZ2I_0",
+ "d09360d3_1",
+ "8e6e058b_3",
+ "95cec590_1",
+ "ultrachat_473538",
+ "5db56d24_2",
+ "4fe91719_1",
+ "ultrachat_316997",
+ "ultrachat_256989"
+ ],
+ "gold_session_ids": [
+ "answer_e40b054e"
+ ]
+ },
+ {
+ "question_id": "c19f7a0b",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_f442ccbe",
+ "sharegpt_6ZeW9Vb_0",
+ "58ab5fc5",
+ "9530353d_1",
+ "0e829b8f",
+ "0dde26d4_8",
+ "c9292210_2",
+ "2fd445a2_3",
+ "1e739aef",
+ "7bc2c7a3_1"
+ ],
+ "gold_session_ids": [
+ "answer_f442ccbe"
+ ]
+ },
+ {
+ "question_id": "4100d0a0",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.5,
+ "mrr": 0.3333333333333333,
+ "retrieved_session_ids": [
+ "8d068598",
+ "ultrachat_130655",
+ "answer_83c13ff9",
+ "595902ff",
+ "sharegpt_eV0krsR_0",
+ "665c09b1_1",
+ "4fd958dd_2",
+ "fbfa105a",
+ "ultrachat_16500",
+ "ultrachat_187351"
+ ],
+ "gold_session_ids": [
+ "answer_83c13ff9"
+ ]
+ },
+ {
+ "question_id": "29f2956b",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_7cc5362f",
+ "00842aad_2",
+ "639cec58",
+ "a79db993",
+ "6d6ffac5",
+ "d50a8a33_1",
+ "88361fdd",
+ "ultrachat_238255",
+ "90f7041a_3",
+ "c4ed2287_3"
+ ],
+ "gold_session_ids": [
+ "answer_7cc5362f"
+ ]
+ },
+ {
+ "question_id": "1faac195",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_d01949bf",
+ "f2ccf83b",
+ "b2543a58_3",
+ "48d385f0_1",
+ "ultrachat_12436",
+ "d2905eb6_2",
+ "9d3d54f7",
+ "bcf6953d_3",
+ "8064b6ca",
+ "70b9616e"
+ ],
+ "gold_session_ids": [
+ "answer_d01949bf"
+ ]
+ },
+ {
+ "question_id": "faba32e5",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_39b12014",
+ "d3da4592_2",
+ "sharegpt_jxsIY6L_0",
+ "05944d87_2",
+ "sharegpt_OYMr8YY_44",
+ "db301f3c_1",
+ "sharegpt_3v69R59_0",
+ "6b9e8346",
+ "83cfab9f_2",
+ "770d11f5"
+ ],
+ "gold_session_ids": [
+ "answer_39b12014"
+ ]
+ },
+ {
+ "question_id": "f4f1d8a4",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6309297535714575,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "e45db473_3",
+ "answer_f5b33470",
+ "3aac691d_2",
+ "sharegpt_pX51ZUz_0",
+ "sharegpt_vXWOUqx_11",
+ "64934c76",
+ "76ea88ba_2",
+ "sharegpt_7ATc6lt_63",
+ "ultrachat_576354",
+ "44837adb"
+ ],
+ "gold_session_ids": [
+ "answer_f5b33470"
+ ]
+ },
+ {
+ "question_id": "c14c00dd",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_304511ce",
+ "sharegpt_KznkJ2p_0",
+ "sharegpt_gU0qvZu_0",
+ "c77250d2",
+ "cd1f5147",
+ "8de3201f_1",
+ "710f4e21_2",
+ "e2cb1647",
+ "541ecc45_1",
+ "7cd7c296_3"
+ ],
+ "gold_session_ids": [
+ "answer_304511ce"
+ ]
+ },
+ {
+ "question_id": "36580ce8",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_93e1bd22",
+ "ultrachat_149170",
+ "ultrachat_270984",
+ "9b38bda6_2",
+ "ca0a84a3_1",
+ "526354c8_2",
+ "ultrachat_32917",
+ "14076ea0_1",
+ "sharegpt_5L3F069_69",
+ "2aee3e70"
+ ],
+ "gold_session_ids": [
+ "answer_93e1bd22"
+ ]
+ },
+ {
+ "question_id": "3d86fd0a",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_19c24c11",
+ "7e4dab66_2",
+ "cf8e352f_1",
+ "1f643033_2",
+ "06d1a1a0",
+ "69674f25",
+ "ultrachat_231397",
+ "1c1a2b7f_2",
+ "ultrachat_96218",
+ "sharegpt_d7bP5fb_0"
+ ],
+ "gold_session_ids": [
+ "answer_19c24c11"
+ ]
+ },
+ {
+ "question_id": "a82c026e",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_787e6a6d",
+ "ccaabf0b_1",
+ "sharegpt_MBVSMQO_45",
+ "3009ba66_2",
+ "d9727262_4",
+ "752392bb_3",
+ "daa32134",
+ "sharegpt_vIkUTvg_0",
+ "sharegpt_u71pVhw_0",
+ "sharegpt_CMeL2P1_0"
+ ],
+ "gold_session_ids": [
+ "answer_787e6a6d"
+ ]
+ },
+ {
+ "question_id": "0862e8bf_abs",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 0,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0,
+ "mrr": 0.06666666666666667,
+ "retrieved_session_ids": [
+ "12cd736c",
+ "f238edb0_1",
+ "ultrachat_532726",
+ "94bc18df_3",
+ "32833e0b_5",
+ "6a8f8453_2",
+ "94f0d575_1",
+ "sharegpt_PSjpANO_35",
+ "ultrachat_210818",
+ "3301f749_1"
+ ],
+ "gold_session_ids": [
+ "answer_c6fd8ebd_abs"
+ ]
+ },
+ {
+ "question_id": "15745da0_abs",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_586de428_abs",
+ "565929a2_2",
+ "e839253d",
+ "375e27b9_2",
+ "9b638f21_4",
+ "66d120f4_2",
+ "83dae08b",
+ "ultrachat_375124",
+ "sharegpt_JSxa86k_19",
+ "ultrachat_360529"
+ ],
+ "gold_session_ids": [
+ "answer_586de428_abs"
+ ]
+ },
+ {
+ "question_id": "bc8a6e93_abs",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.43067655807339306,
+ "mrr": 0.25,
+ "retrieved_session_ids": [
+ "55cf4ce0_2",
+ "f51c7700",
+ "375354d1_2",
+ "answer_e6143162_abs",
+ "5abba521",
+ "sharegpt_MimvIAy_181",
+ "13e9de13_4",
+ "sharegpt_SF6KWw1_9",
+ "ba944cdc",
+ "sharegpt_hn3IS1q_0"
+ ],
+ "gold_session_ids": [
+ "answer_e6143162_abs"
+ ]
+ },
+ {
+ "question_id": "19b5f2b3_abs",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6309297535714575,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "ultrachat_159269",
+ "answer_5ff494b9_abs",
+ "66138e8d_1",
+ "3e59ee68_1",
+ "9f8fa5e4_2",
+ "sharegpt_v11Tpg9_483",
+ "f5cf8815_3",
+ "e6640284_1",
+ "45c29267",
+ "eb134165_2"
+ ],
+ "gold_session_ids": [
+ "answer_5ff494b9_abs"
+ ]
+ },
+ {
+ "question_id": "29f2956b_abs",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.43067655807339306,
+ "mrr": 0.25,
+ "retrieved_session_ids": [
+ "ultrachat_70054",
+ "503b2761",
+ "05b84f71_5",
+ "answer_7cc5362f_abs",
+ "cc50c6a9",
+ "64b7d9cc_2",
+ "f2fa97f5_3",
+ "7363604d_2",
+ "3b8f0d9d",
+ "ultrachat_441324"
+ ],
+ "gold_session_ids": [
+ "answer_7cc5362f_abs"
+ ]
+ },
+ {
+ "question_id": "f4f1d8a4_abs",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.3010299956639812,
+ "mrr": 0.1111111111111111,
+ "retrieved_session_ids": [
+ "afe04238_1",
+ "1dd13331_1",
+ "ea3db78e_2",
+ "080a4000_2",
+ "f252001e",
+ "ultrachat_317545",
+ "sharegpt_jerQm7S_17",
+ "402e0082_6",
+ "answer_f5b33470_abs",
+ "1a76dc7f_1"
+ ],
+ "gold_session_ids": [
+ "answer_f5b33470_abs"
+ ]
+ },
+ {
+ "question_id": "0a995998",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9674679834891693,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_afa9873b_1",
+ "answer_afa9873b_2",
+ "sharegpt_igR6Bmm_0",
+ "answer_afa9873b_3",
+ "e6790684_1",
+ "85846900_2",
+ "25091584_1",
+ "64b847a7_1",
+ "ad3d0815_1",
+ "439f9ed8_3"
+ ],
+ "gold_session_ids": [
+ "answer_afa9873b_2",
+ "answer_afa9873b_3",
+ "answer_afa9873b_1"
+ ]
+ },
+ {
+ "question_id": "6d550036",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.34620964322385606,
+ "mrr": 0.3333333333333333,
+ "retrieved_session_ids": [
+ "2e4430d8_2",
+ "0bab76de",
+ "answer_ec904b3c_1",
+ "dae3906e_2",
+ "answer_ec904b3c_2",
+ "sharegpt_nVi6tIt_0",
+ "b37a44e4_5",
+ "a9981dc6_3",
+ "187b4852_1",
+ "5558a42e_2"
+ ],
+ "gold_session_ids": [
+ "answer_ec904b3c_1",
+ "answer_ec904b3c_4",
+ "answer_ec904b3c_3",
+ "answer_ec904b3c_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_59c863d7",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.502373740777944,
+ "mrr": 0.3333333333333333,
+ "retrieved_session_ids": [
+ "cf7aad73_5",
+ "eb47739f_2",
+ "answer_593bdffd_4",
+ "answer_593bdffd_1",
+ "2f310ba0",
+ "answer_593bdffd_2",
+ "9ef698bc_2",
+ "61d3fec4_2",
+ "2f09d4c8",
+ "a7d014e4_1"
+ ],
+ "gold_session_ids": [
+ "answer_593bdffd_4",
+ "answer_593bdffd_1",
+ "answer_593bdffd_3",
+ "answer_593bdffd_2"
+ ]
+ },
+ {
+ "question_id": "b5ef892d",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.34870224750355494,
+ "mrr": 0.2,
+ "retrieved_session_ids": [
+ "sharegpt_hfWd2FK_0",
+ "e552e1f9_2",
+ "7294438c",
+ "sharegpt_esZWTJv_8",
+ "answer_a8b4290f_3",
+ "answer_a8b4290f_1",
+ "ultrachat_565056",
+ "35dcacdc_2",
+ "eb774a95",
+ "25ec099e_1"
+ ],
+ "gold_session_ids": [
+ "answer_a8b4290f_3",
+ "answer_a8b4290f_1",
+ "answer_a8b4290f_2"
+ ]
+ },
+ {
+ "question_id": "e831120c",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6240505200038379,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "773aebbd_4",
+ "answer_86c505e7_2",
+ "a63ad8e3_2",
+ "sharegpt_sDfu38Q_0",
+ "answer_86c505e7_1",
+ "sharegpt_CROow2Q_0",
+ "ultrachat_485311",
+ "8fb3fe3a_1",
+ "7c82a6c3",
+ "f07b6875_1"
+ ],
+ "gold_session_ids": [
+ "answer_86c505e7_1",
+ "answer_86c505e7_2"
+ ]
+ },
+ {
+ "question_id": "3a704032",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.20210734650054757,
+ "mrr": 0.25,
+ "retrieved_session_ids": [
+ "sharegpt_HrxBbBK_0",
+ "ultrachat_85818",
+ "sharegpt_tzJ6yxo_0",
+ "answer_c2204106_1",
+ "ultrachat_418858",
+ "aa41ac47",
+ "ultrachat_357807",
+ "f62bfe2b_2",
+ "7712aded",
+ "5cfdb3bc"
+ ],
+ "gold_session_ids": [
+ "answer_c2204106_2",
+ "answer_c2204106_3",
+ "answer_c2204106_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_d84a3211",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.2638647595767405,
+ "mrr": 0.2,
+ "retrieved_session_ids": [
+ "4a32d58c_2",
+ "ultrachat_234895",
+ "sharegpt_M5zaN8c_0",
+ "sharegpt_qHNCIfF_0",
+ "answer_2880eb6c_4",
+ "ultrachat_83526",
+ "51ec01a4",
+ "6acb2499",
+ "8dd540f8_1",
+ "answer_2880eb6c_1"
+ ],
+ "gold_session_ids": [
+ "answer_2880eb6c_2",
+ "answer_2880eb6c_4",
+ "answer_2880eb6c_1",
+ "answer_2880eb6c_3"
+ ]
+ },
+ {
+ "question_id": "aae3761f",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.619668607939653,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "3d85d91e_2",
+ "answer_526354c8_1",
+ "38e81260_7",
+ "sharegpt_bKkRqZc_0",
+ "2a86d0da_2",
+ "answer_526354c8_2",
+ "answer_526354c8_3",
+ "sharegpt_ynLoh9N_0",
+ "715176f5_1",
+ "d576152e_2"
+ ],
+ "gold_session_ids": [
+ "answer_526354c8_1",
+ "answer_526354c8_3",
+ "answer_526354c8_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_f2262a51",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6105456988825855,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_55a6940c_3",
+ "sharegpt_hChsWOp_128",
+ "sharegpt_eoEbthf_0",
+ "0984a772",
+ "ultrachat_268434",
+ "dd81b163_1",
+ "83e14506_1",
+ "c0d6fa6f_2",
+ "answer_55a6940c_2",
+ "sharegpt_BWMyoNr_0"
+ ],
+ "gold_session_ids": [
+ "answer_55a6940c_3",
+ "answer_55a6940c_1",
+ "answer_55a6940c_2"
+ ]
+ },
+ {
+ "question_id": "dd2973ad",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8175295903539445,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_f9de4602_2",
+ "sharegpt_O6NO7Vo_9",
+ "e5f08e40",
+ "6551be60_3",
+ "98c198fb",
+ "5b8ef613_4",
+ "answer_f9de4602_1",
+ "56266c38_1",
+ "0f73eee7_2",
+ "32c0dae1_3"
+ ],
+ "gold_session_ids": [
+ "answer_f9de4602_2",
+ "answer_f9de4602_1"
+ ]
+ },
+ {
+ "question_id": "c4a1ceb8",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9058653015743079,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_56d02cab_3",
+ "answer_56d02cab_1",
+ "40c77045",
+ "a09ae506_1",
+ "d0091194",
+ "answer_56d02cab_4",
+ "answer_56d02cab_2",
+ "ultrachat_441000",
+ "ultrachat_433406",
+ "44ba5b71_1"
+ ],
+ "gold_session_ids": [
+ "answer_56d02cab_3",
+ "answer_56d02cab_4",
+ "answer_56d02cab_2",
+ "answer_56d02cab_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_a56e767c",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8529278650606567,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_cf9e3940_3",
+ "88c8df0e_3",
+ "d75245ea",
+ "answer_cf9e3940_1",
+ "answer_cf9e3940_2",
+ "ultrachat_104129",
+ "71621be0",
+ "ultrachat_257906",
+ "7a819e6f_1",
+ "f56e6152_1"
+ ],
+ "gold_session_ids": [
+ "answer_cf9e3940_2",
+ "answer_cf9e3940_1",
+ "answer_cf9e3940_3"
+ ]
+ },
+ {
+ "question_id": "6cb6f249",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.5012658353418871,
+ "mrr": 0.25,
+ "retrieved_session_ids": [
+ "ultrachat_404221",
+ "sharegpt_IkYYvSd_79",
+ "9b6db1a9_2",
+ "answer_a4204937_2",
+ "answer_a4204937_1",
+ "ultrachat_374192",
+ "89a1800c_3",
+ "5edfe253_1",
+ "sharegpt_JTaoEJT_23",
+ "ultrachat_219372"
+ ],
+ "gold_session_ids": [
+ "answer_a4204937_2",
+ "answer_a4204937_1"
+ ]
+ },
+ {
+ "question_id": "46a3abf7",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.3295827480202853,
+ "mrr": 0.2,
+ "retrieved_session_ids": [
+ "e224317f_2",
+ "09f2c6c5",
+ "be133e38_1",
+ "c1e598cd_1",
+ "answer_c65042d7_2",
+ "4fe91719_2",
+ "ultrachat_42811",
+ "answer_c65042d7_3",
+ "a95bedca",
+ "28eb86ab"
+ ],
+ "gold_session_ids": [
+ "answer_c65042d7_3",
+ "answer_c65042d7_2",
+ "answer_c65042d7_1"
+ ]
+ },
+ {
+ "question_id": "36b9f61e",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.7039180890341347,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ef74281f_1",
+ "b283f9eb_5",
+ "answer_ef74281f_2",
+ "sharegpt_ba4V2an_2",
+ "0de92c3f_2",
+ "56911dc5_4",
+ "d6f5639a_1",
+ "93c5e61f",
+ "e419b7c3_3",
+ "889efc00_2"
+ ],
+ "gold_session_ids": [
+ "answer_ef74281f_2",
+ "answer_ef74281f_1",
+ "answer_ef74281f_3"
+ ]
+ },
+ {
+ "question_id": "28dc39ac",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9669850519151657,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_8d015d9d_1",
+ "answer_8d015d9d_4",
+ "answer_8d015d9d_3",
+ "5854eebc_1",
+ "answer_8d015d9d_5",
+ "3f5df138_1",
+ "answer_8d015d9d_2",
+ "49b7a8b7_2",
+ "1c1f5ccc_2",
+ "6ccadc2b"
+ ],
+ "gold_session_ids": [
+ "answer_8d015d9d_3",
+ "answer_8d015d9d_5",
+ "answer_8d015d9d_2",
+ "answer_8d015d9d_4",
+ "answer_8d015d9d_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_2f8be40d",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.5307212739772434,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "sharegpt_L64rHOA_0",
+ "answer_e7b0637e_2",
+ "answer_e7b0637e_1",
+ "1d6e01e8",
+ "acda6a4e_4",
+ "ultrachat_317479",
+ "81b971b8_2",
+ "62684a60",
+ "f0e71553_4",
+ "ultrachat_434633"
+ ],
+ "gold_session_ids": [
+ "answer_e7b0637e_2",
+ "answer_e7b0637e_1",
+ "answer_e7b0637e_3"
+ ]
+ },
+ {
+ "question_id": "2e6d26dc",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.4042974959288707,
+ "mrr": 0.2,
+ "retrieved_session_ids": [
+ "sharegpt_vXNQZ2I_0",
+ "ultrachat_121666",
+ "e8bfacec_1",
+ "14a520c8_2",
+ "answer_fa526fc0_3",
+ "f863a648_1",
+ "answer_fa526fc0_4",
+ "answer_fa526fc0_1",
+ "sharegpt_h4ZC1fl_152",
+ "73f14139"
+ ],
+ "gold_session_ids": [
+ "answer_fa526fc0_4",
+ "answer_fa526fc0_3",
+ "answer_fa526fc0_1",
+ "answer_fa526fc0_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_15e38248",
+ "question_type": "multi-session",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.2691828628414762,
+ "mrr": 0.16666666666666666,
+ "retrieved_session_ids": [
+ "295fcaeb_2",
+ "sharegpt_mm9tc5g_0",
+ "ultrachat_515012",
+ "263efe64_1",
+ "0785de5a",
+ "answer_8858d9dc_2",
+ "answer_8858d9dc_4",
+ "ultrachat_442952",
+ "8c9bc932_2",
+ "87e55e85_2"
+ ],
+ "gold_session_ids": [
+ "answer_8858d9dc_3",
+ "answer_8858d9dc_1",
+ "answer_8858d9dc_4",
+ "answer_8858d9dc_2"
+ ]
+ },
+ {
+ "question_id": "88432d0a",
+ "question_type": "multi-session",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 0,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0,
+ "mrr": 0.07142857142857142,
+ "retrieved_session_ids": [
+ "ultrachat_122782",
+ "ca0553dd_1",
+ "0e250059_2",
+ "f777f641",
+ "ultrachat_128882",
+ "27e262d5_2",
+ "ultrachat_414980",
+ "ultrachat_528370",
+ "c9d35c00_2",
+ "1c8832b4_2"
+ ],
+ "gold_session_ids": [
+ "answer_733e443a_3",
+ "answer_733e443a_4",
+ "answer_733e443a_2",
+ "answer_733e443a_1"
+ ]
+ },
+ {
+ "question_id": "80ec1f4f",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8179809735263114,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_990c8992_2",
+ "f987da7e_3",
+ "ultrachat_318441",
+ "ultrachat_497987",
+ "answer_990c8992_1",
+ "answer_990c8992_3",
+ "38aec47b_1",
+ "2a95f614_1",
+ "b6018747_3",
+ "sharegpt_nhknQtd_29"
+ ],
+ "gold_session_ids": [
+ "answer_990c8992_2",
+ "answer_990c8992_1",
+ "answer_990c8992_3"
+ ]
+ },
+ {
+ "question_id": "d23cf73b",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.298254219601818,
+ "mrr": 0.25,
+ "retrieved_session_ids": [
+ "c328cd4f",
+ "95b7c3cd_1",
+ "ef71a35e_2",
+ "answer_5a0d28f8_4",
+ "b124e4a1_1",
+ "9e21d6ab_2",
+ "answer_5a0d28f8_2",
+ "ad44d5c2_2",
+ "ultrachat_275858",
+ "ultrachat_553398"
+ ],
+ "gold_session_ids": [
+ "answer_5a0d28f8_4",
+ "answer_5a0d28f8_2",
+ "answer_5a0d28f8_3",
+ "answer_5a0d28f8_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_7fce9456",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.5592067393987519,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_a679a86a_4",
+ "1da409cd_1",
+ "288e7d30",
+ "191e1832",
+ "04328d1c",
+ "541ecc45_3",
+ "answer_a679a86a_5",
+ "answer_a679a86a_3",
+ "76707a65_1",
+ "sharegpt_tYc53EZ_13"
+ ],
+ "gold_session_ids": [
+ "answer_a679a86a_5",
+ "answer_a679a86a_4",
+ "answer_a679a86a_2",
+ "answer_a679a86a_3",
+ "answer_a679a86a_1"
+ ]
+ },
+ {
+ "question_id": "d682f1a2",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.5323325778787926,
+ "mrr": 0.3333333333333333,
+ "retrieved_session_ids": [
+ "ultrachat_214872",
+ "9de29219_1",
+ "answer_c008e5df_3",
+ "6e2cca63_8",
+ "sharegpt_E7VUcfj_0",
+ "sharegpt_LFPdgVp_22",
+ "answer_c008e5df_2",
+ "e30307d8_2",
+ "answer_c008e5df_1",
+ "58335f60_2"
+ ],
+ "gold_session_ids": [
+ "answer_c008e5df_1",
+ "answer_c008e5df_2",
+ "answer_c008e5df_3"
+ ]
+ },
+ {
+ "question_id": "7024f17c",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_a21f3697_3",
+ "answer_a21f3697_2",
+ "answer_a21f3697_1",
+ "d7281662_2",
+ "8c652eb0_1",
+ "3c12ff0c",
+ "336b281b_1",
+ "199a5225_2",
+ "sharegpt_rshCiS5_1",
+ "385683f0_1"
+ ],
+ "gold_session_ids": [
+ "answer_a21f3697_1",
+ "answer_a21f3697_2",
+ "answer_a21f3697_3"
+ ]
+ },
+ {
+ "question_id": "gpt4_5501fe77",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8710785440003369,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_203bf3fa_3",
+ "bc5827b5",
+ "answer_203bf3fa_1",
+ "542fa76e_1",
+ "021c0d09_2",
+ "answer_203bf3fa_2",
+ "426e7403_3",
+ "ultrachat_303597",
+ "631e4016",
+ "7a2c618c_1"
+ ],
+ "gold_session_ids": [
+ "answer_203bf3fa_1",
+ "answer_203bf3fa_3",
+ "answer_203bf3fa_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_2ba83207",
+ "question_type": "multi-session",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.13905617951077562,
+ "mrr": 0.16666666666666666,
+ "retrieved_session_ids": [
+ "33028509_2",
+ "325c2005",
+ "sharegpt_JXREDs0_4",
+ "131ff17e",
+ "41dc5d45_4",
+ "answer_6a3b5c13_2",
+ "ultrachat_224880",
+ "ultrachat_94578",
+ "f379d356_2",
+ "58d66fec"
+ ],
+ "gold_session_ids": [
+ "answer_6a3b5c13_3",
+ "answer_6a3b5c13_1",
+ "answer_6a3b5c13_2",
+ "answer_6a3b5c13_4"
+ ]
+ },
+ {
+ "question_id": "2318644b",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6131471927654584,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_eaa8e3ef_2",
+ "6ec23dc5_1",
+ "sharegpt_iwcVyKX_0",
+ "sharegpt_52wqzr7_0",
+ "ultrachat_254829",
+ "c5d99bff",
+ "bff09560_1",
+ "07a26dfa_2",
+ "sharegpt_7NcBHFs_0",
+ "e44e42dd"
+ ],
+ "gold_session_ids": [
+ "answer_eaa8e3ef_1",
+ "answer_eaa8e3ef_2"
+ ]
+ },
+ {
+ "question_id": "2ce6a0f2",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.24630238874073,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "f257aed3_1",
+ "answer_901a6763_4",
+ "sharegpt_8W46coJ_3",
+ "ultrachat_36844",
+ "ultrachat_463514",
+ "e4bacbcd_1",
+ "a1714d2c_2",
+ "89b2a0dc_1",
+ "600077b0_3",
+ "a267b11e"
+ ],
+ "gold_session_ids": [
+ "answer_901a6763_2",
+ "answer_901a6763_4",
+ "answer_901a6763_1",
+ "answer_901a6763_3"
+ ]
+ },
+ {
+ "question_id": "gpt4_d12ceb0e",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.3228087653971817,
+ "mrr": 0.2,
+ "retrieved_session_ids": [
+ "73f4798f",
+ "0672cd7b_1",
+ "33d16079_2",
+ "ced213c6",
+ "answer_2504635e_1",
+ "31dc9d66_3",
+ "9d97ff74",
+ "f0c6ddb9_2",
+ "answer_2504635e_2",
+ "ee28793c"
+ ],
+ "gold_session_ids": [
+ "answer_2504635e_3",
+ "answer_2504635e_2",
+ "answer_2504635e_1"
+ ]
+ },
+ {
+ "question_id": "00ca467f",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.35853358850813305,
+ "mrr": 0.25,
+ "retrieved_session_ids": [
+ "8a70ec05_2",
+ "e0fa8486_2",
+ "4951c0b9_1",
+ "answer_39900a0a_2",
+ "7d1de770_2",
+ "36828c66_1",
+ "answer_39900a0a_3",
+ "4a5ee697_1",
+ "ultrachat_145200",
+ "315a862c"
+ ],
+ "gold_session_ids": [
+ "answer_39900a0a_3",
+ "answer_39900a0a_2",
+ "answer_39900a0a_1"
+ ]
+ },
+ {
+ "question_id": "b3c15d39",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.42177340954886444,
+ "mrr": 0.2,
+ "retrieved_session_ids": [
+ "sharegpt_gPmunOt_37",
+ "sharegpt_inuIr9J_119",
+ "7bb01253",
+ "d298714d_1",
+ "answer_05d808e6_1",
+ "bd00a24d_2",
+ "6be23caa",
+ "5d4ff8db_1",
+ "answer_05d808e6_2",
+ "ultrachat_356582"
+ ],
+ "gold_session_ids": [
+ "answer_05d808e6_1",
+ "answer_05d808e6_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_31ff4165",
+ "question_type": "multi-session",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.4279411160210267,
+ "mrr": 0.16666666666666666,
+ "retrieved_session_ids": [
+ "e2393b63_2",
+ "sharegpt_WTYjzSr_0",
+ "sharegpt_tKmxch5_0",
+ "dc9bf721_1",
+ "2c185a2b_1",
+ "answer_02b63d04_4",
+ "2d0b80f1",
+ "answer_02b63d04_1",
+ "answer_02b63d04_3",
+ "answer_02b63d04_5"
+ ],
+ "gold_session_ids": [
+ "answer_02b63d04_1",
+ "answer_02b63d04_5",
+ "answer_02b63d04_2",
+ "answer_02b63d04_3",
+ "answer_02b63d04_4"
+ ]
+ },
+ {
+ "question_id": "eeda8a6d",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6934264036172708,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "a0f5d30b",
+ "answer_3e5fea0e_1",
+ "answer_3e5fea0e_2",
+ "cf7aad73_3",
+ "ultrachat_309574",
+ "ultrachat_307733",
+ "sharegpt_Caxpnd3_12",
+ "b2a021c9_1",
+ "sharegpt_khNFc2W_0",
+ "034ce94a_1"
+ ],
+ "gold_session_ids": [
+ "answer_3e5fea0e_1",
+ "answer_3e5fea0e_2"
+ ]
+ },
+ {
+ "question_id": "2788b940",
+ "question_type": "multi-session",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.25657228426720274,
+ "mrr": 0.16666666666666666,
+ "retrieved_session_ids": [
+ "sharegpt_dabXXRO_10",
+ "ff6e824b_1",
+ "544fe66c_2",
+ "8de5b7cf_2",
+ "sharegpt_vgBnRfE_0",
+ "answer_8f6b938d_4",
+ "ab483540",
+ "sharegpt_hV2NEYj_2",
+ "answer_8f6b938d_2",
+ "3f863722"
+ ],
+ "gold_session_ids": [
+ "answer_8f6b938d_1",
+ "answer_8f6b938d_3",
+ "answer_8f6b938d_4",
+ "answer_8f6b938d_2"
+ ]
+ },
+ {
+ "question_id": "60bf93ed",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.5802792108518124,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "d467e3fd",
+ "answer_e0956e0a_1",
+ "388c5534_3",
+ "6414f676_4",
+ "0e297c49",
+ "4fe27c66_4",
+ "ultrachat_7648",
+ "answer_e0956e0a_2",
+ "619e96bc",
+ "84f80d64"
+ ],
+ "gold_session_ids": [
+ "answer_e0956e0a_1",
+ "answer_e0956e0a_2"
+ ]
+ },
+ {
+ "question_id": "9d25d4e0",
+ "question_type": "multi-session",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.1480409554829326,
+ "mrr": 0.125,
+ "retrieved_session_ids": [
+ "sharegpt_hRba933_47",
+ "2366adbc",
+ "464f3821",
+ "82d6f027",
+ "5f558dee_1",
+ "46d9d476",
+ "380a9d4d_1",
+ "answer_fcff2dc4_2",
+ "39911f94",
+ "cd568521_3"
+ ],
+ "gold_session_ids": [
+ "answer_fcff2dc4_2",
+ "answer_fcff2dc4_1",
+ "answer_fcff2dc4_3"
+ ]
+ },
+ {
+ "question_id": "129d1232",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.46927872602275644,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_1de862d6_2",
+ "sharegpt_De0lanR_0",
+ "a0b8b0ad_7",
+ "ultrachat_227373",
+ "ultrachat_274599",
+ "ultrachat_88574",
+ "1ab255ba_1",
+ "eaa8e3ef_2",
+ "478c480d",
+ "49b78a55_2"
+ ],
+ "gold_session_ids": [
+ "answer_1de862d6_1",
+ "answer_1de862d6_3",
+ "answer_1de862d6_2"
+ ]
+ },
+ {
+ "question_id": "60472f9c",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.7653606369886217,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_e7fe8c8b_1",
+ "answer_e7fe8c8b_2",
+ "ultrachat_373250",
+ "b1815b81",
+ "sharegpt_kF8hHMt_45",
+ "e7e416f5_2",
+ "7b74d32d_3",
+ "fdd0f680_2",
+ "33da50d0_1",
+ "27f5e6e8_1"
+ ],
+ "gold_session_ids": [
+ "answer_e7fe8c8b_1",
+ "answer_e7fe8c8b_2",
+ "answer_e7fe8c8b_3"
+ ]
+ },
+ {
+ "question_id": "gpt4_194be4b3",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.5050778624218062,
+ "mrr": 0.25,
+ "retrieved_session_ids": [
+ "sharegpt_UbMVpdp_137",
+ "ultrachat_474217",
+ "ultrachat_58180",
+ "answer_3826dc55_4",
+ "answer_3826dc55_3",
+ "answer_3826dc55_1",
+ "sharegpt_VO7dTG3_0",
+ "answer_3826dc55_5",
+ "31403c51",
+ "ultrachat_163834"
+ ],
+ "gold_session_ids": [
+ "answer_3826dc55_1",
+ "answer_3826dc55_3",
+ "answer_3826dc55_5",
+ "answer_3826dc55_2",
+ "answer_3826dc55_4"
+ ]
+ },
+ {
+ "question_id": "a9f6b44c",
+ "question_type": "multi-session",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.16716045496620227,
+ "mrr": 0.16666666666666666,
+ "retrieved_session_ids": [
+ "ultrachat_539849",
+ "837f258b",
+ "949d51ef",
+ "e10dfffb_1",
+ "bb107057_4",
+ "answer_cc021f81_3",
+ "ultrachat_468681",
+ "3489fa38_2",
+ "ultrachat_279504",
+ "1126be1e_1"
+ ],
+ "gold_session_ids": [
+ "answer_cc021f81_2",
+ "answer_cc021f81_3",
+ "answer_cc021f81_1"
+ ]
+ },
+ {
+ "question_id": "d851d5ba",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6198898455343711,
+ "mrr": 0.3333333333333333,
+ "retrieved_session_ids": [
+ "561f3a6a_2",
+ "2a28999b",
+ "answer_5cdf9bd2_3",
+ "answer_5cdf9bd2_2",
+ "d6d8b854_4",
+ "answer_5cdf9bd2_1",
+ "9f8973ac",
+ "sharegpt_x7apNZ0_0",
+ "answer_5cdf9bd2_4",
+ "66a6a097_3"
+ ],
+ "gold_session_ids": [
+ "answer_5cdf9bd2_2",
+ "answer_5cdf9bd2_1",
+ "answer_5cdf9bd2_3",
+ "answer_5cdf9bd2_4"
+ ]
+ },
+ {
+ "question_id": "5a7937c8",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8385465274895063,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_4cef8a3c_2",
+ "b70ac29f_2",
+ "3b38be11_2",
+ "answer_4cef8a3c_3",
+ "559ce9c2",
+ "answer_4cef8a3c_1",
+ "ultrachat_564068",
+ "ultrachat_377263",
+ "ultrachat_355239",
+ "63c9e87c"
+ ],
+ "gold_session_ids": [
+ "answer_4cef8a3c_3",
+ "answer_4cef8a3c_1",
+ "answer_4cef8a3c_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_ab202e7f",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9103075074400794,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_728deb4d_3",
+ "answer_728deb4d_5",
+ "29bc69b3",
+ "answer_728deb4d_4",
+ "a08fbe88_1",
+ "dcafb5b3_5",
+ "answer_728deb4d_2",
+ "a394f6b5_2",
+ "ultrachat_433876",
+ "answer_728deb4d_1"
+ ],
+ "gold_session_ids": [
+ "answer_728deb4d_5",
+ "answer_728deb4d_2",
+ "answer_728deb4d_3",
+ "answer_728deb4d_1",
+ "answer_728deb4d_4"
+ ]
+ },
+ {
+ "question_id": "gpt4_e05b82a6",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9279611693743142,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_6350aa4f_1",
+ "answer_6350aa4f_2",
+ "386f6df9",
+ "answer_6350aa4f_4",
+ "e2cd250e_2",
+ "6172cc72_1",
+ "503791f7_1",
+ "answer_6350aa4f_3",
+ "ultrachat_43272",
+ "88e01130_1"
+ ],
+ "gold_session_ids": [
+ "answer_6350aa4f_1",
+ "answer_6350aa4f_2",
+ "answer_6350aa4f_3",
+ "answer_6350aa4f_4"
+ ]
+ },
+ {
+ "question_id": "gpt4_731e37d7",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_826d51da_3",
+ "answer_826d51da_1",
+ "answer_826d51da_2",
+ "answer_826d51da_4",
+ "ultrachat_109542",
+ "267a912c",
+ "5eec1ec6",
+ "65747d84_1",
+ "baa5b468_3",
+ "7e4aa7c2_1"
+ ],
+ "gold_session_ids": [
+ "answer_826d51da_3",
+ "answer_826d51da_4",
+ "answer_826d51da_2",
+ "answer_826d51da_1"
+ ]
+ },
+ {
+ "question_id": "edced276",
+ "question_type": "multi-session",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.3706656904013186,
+ "mrr": 0.125,
+ "retrieved_session_ids": [
+ "1da409cd_1",
+ "e4bbcd5f_4",
+ "ultrachat_195214",
+ "5726dc37_2",
+ "b86848e2_3",
+ "d89f6204",
+ "2dc55862_1",
+ "answer_60e8941a_2",
+ "93f23301_1",
+ "answer_60e8941a_1"
+ ],
+ "gold_session_ids": [
+ "answer_60e8941a_2",
+ "answer_60e8941a_1"
+ ]
+ },
+ {
+ "question_id": "10d9b85a",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6131471927654584,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_e0585cb5_2",
+ "c51583cd_3",
+ "3c8e7c0e_4",
+ "ultrachat_143866",
+ "5cbfaf3e_4",
+ "d576152e_1",
+ "ebe3b6bf_1",
+ "ultrachat_501022",
+ "9c5fa973",
+ "sharegpt_Sionhxj_115"
+ ],
+ "gold_session_ids": [
+ "answer_e0585cb5_2",
+ "answer_e0585cb5_1"
+ ]
+ },
+ {
+ "question_id": "e3038f8c",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.4303349101522581,
+ "mrr": 0.25,
+ "retrieved_session_ids": [
+ "e3d4f89e_3",
+ "d71c8b77_5",
+ "85846900_2",
+ "answer_b6018747_4",
+ "a3d8e134_2",
+ "answer_b6018747_3",
+ "ultrachat_48888",
+ "answer_b6018747_2",
+ "941d586a",
+ "6e2cca63_3"
+ ],
+ "gold_session_ids": [
+ "answer_b6018747_2",
+ "answer_b6018747_4",
+ "answer_b6018747_1",
+ "answer_b6018747_3"
+ ]
+ },
+ {
+ "question_id": "2b8f3739",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9674679834891693,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_23759615_3",
+ "answer_23759615_1",
+ "dd25eeb5_6",
+ "answer_23759615_2",
+ "b8ed5202_2",
+ "ce6cae1b_3",
+ "e9fb10e7_2",
+ "ec8691f5",
+ "52ccfdfa",
+ "465c3970"
+ ],
+ "gold_session_ids": [
+ "answer_23759615_2",
+ "answer_23759615_3",
+ "answer_23759615_1"
+ ]
+ },
+ {
+ "question_id": "1a8a66a6",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.3764290720714305,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "54aa4875",
+ "answer_2bd23659_1",
+ "sharegpt_mANdO1O_0",
+ "22eebf8a_2",
+ "d6792df8",
+ "sharegpt_P01BTRw_0",
+ "answer_2bd23659_2",
+ "ultrachat_280864",
+ "18fb3a22_3",
+ "sharegpt_tyukJll_27"
+ ],
+ "gold_session_ids": [
+ "answer_2bd23659_3",
+ "answer_2bd23659_2",
+ "answer_2bd23659_4",
+ "answer_2bd23659_1"
+ ]
+ },
+ {
+ "question_id": "c2ac3c61",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.5109559939712153,
+ "mrr": 0.3333333333333333,
+ "retrieved_session_ids": [
+ "8e4e215b",
+ "6009ba4e_5",
+ "answer_923c0221_2",
+ "0c483a3c",
+ "ultrachat_412857",
+ "sharegpt_z6pSZ19_0",
+ "answer_923c0221_1",
+ "sharegpt_gLb26GY_0",
+ "b403d6f0_4",
+ "97338ddd_1"
+ ],
+ "gold_session_ids": [
+ "answer_923c0221_1",
+ "answer_923c0221_2"
+ ]
+ },
+ {
+ "question_id": "bf659f65",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9674679834891693,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_7726e7e9_1",
+ "answer_7726e7e9_2",
+ "f2835879_2",
+ "answer_7726e7e9_3",
+ "ultrachat_365864",
+ "2917b130_2",
+ "sharegpt_inuIr9J_119",
+ "ultrachat_377765",
+ "258645ba_3",
+ "385a60c8_3"
+ ],
+ "gold_session_ids": [
+ "answer_7726e7e9_1",
+ "answer_7726e7e9_3",
+ "answer_7726e7e9_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_372c3eed",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6049306994552042,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_35c5419d_1",
+ "10647770",
+ "9a22ae03",
+ "ultrachat_400797",
+ "ultrachat_209842",
+ "sharegpt_6dPUCcK_69",
+ "8d969f3e_3",
+ "b661e1d3_1",
+ "c95f88fd_1",
+ "answer_35c5419d_3"
+ ],
+ "gold_session_ids": [
+ "answer_35c5419d_3",
+ "answer_35c5419d_2",
+ "answer_35c5419d_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_2f91af09",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8710785440003369,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_669318cf_3",
+ "e7e416f5_2",
+ "answer_669318cf_2",
+ "22b3af37_1",
+ "d75245ea",
+ "answer_669318cf_1",
+ "9b85a31d_1",
+ "51c82c8e_2",
+ "fd6f60f0_2",
+ "ultrachat_296153"
+ ],
+ "gold_session_ids": [
+ "answer_669318cf_2",
+ "answer_669318cf_1",
+ "answer_669318cf_3"
+ ]
+ },
+ {
+ "question_id": "81507db6",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.7809509050749017,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_da3c1266_5",
+ "answer_da3c1266_1",
+ "sharegpt_Caxpnd3_9",
+ "7da6f798_1",
+ "ultrachat_254248",
+ "answer_da3c1266_4",
+ "ultrachat_229430",
+ "answer_da3c1266_2",
+ "ultrachat_162129",
+ "sharegpt_AZELQqX_0"
+ ],
+ "gold_session_ids": [
+ "answer_da3c1266_3",
+ "answer_da3c1266_4",
+ "answer_da3c1266_1",
+ "answer_da3c1266_5",
+ "answer_da3c1266_2"
+ ]
+ },
+ {
+ "question_id": "88432d0a_abs",
+ "question_type": "multi-session",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.2476427880871277,
+ "mrr": 0.14285714285714285,
+ "retrieved_session_ids": [
+ "1e01dbcc_1",
+ "09d8b29e_1",
+ "ultrachat_453194",
+ "11f043cd_2",
+ "8e5b0424",
+ "4eff1baa",
+ "answer_733e443a_abs_1",
+ "9863a1c8_2",
+ "answer_733e443a_abs_3",
+ "1e215f06"
+ ],
+ "gold_session_ids": [
+ "answer_733e443a_abs_1",
+ "answer_733e443a_abs_3",
+ "answer_733e443a_abs_2",
+ "answer_733e443a_abs_4"
+ ]
+ },
+ {
+ "question_id": "80ec1f4f_abs",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.7122630665145961,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "b304b283_1",
+ "answer_990c8992_abs_2",
+ "answer_990c8992_abs_1",
+ "b1c6d332",
+ "answer_990c8992_abs_3",
+ "31829c1b_1",
+ "ultrachat_187094",
+ "ultrachat_544442",
+ "b341cf37_2",
+ "e5f9453d"
+ ],
+ "gold_session_ids": [
+ "answer_990c8992_abs_2",
+ "answer_990c8992_abs_3",
+ "answer_990c8992_abs_1"
+ ]
+ },
+ {
+ "question_id": "eeda8a6d_abs",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6509209298071326,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "a0b8b0ad_5",
+ "answer_3e5fea0e_abs_2",
+ "ultrachat_544853",
+ "answer_3e5fea0e_abs_1",
+ "sharegpt_l7gyE1o_17",
+ "f4eaef47",
+ "ultrachat_199093",
+ "d15d2899_4",
+ "ultrachat_124888",
+ "sharegpt_YpleI8A_285"
+ ],
+ "gold_session_ids": [
+ "answer_3e5fea0e_abs_2",
+ "answer_3e5fea0e_abs_1"
+ ]
+ },
+ {
+ "question_id": "60bf93ed_abs",
+ "question_type": "multi-session",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 0,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0,
+ "mrr": 0.08333333333333333,
+ "retrieved_session_ids": [
+ "ultrachat_490706",
+ "755c2d32_1",
+ "1e91cdf0",
+ "3e82fae8_1",
+ "0944933f_1",
+ "2946bf12",
+ "sharegpt_7mz4EV7_0",
+ "sharegpt_6cz1Sq6_130",
+ "9ef698bc_2",
+ "cdf068b1_3"
+ ],
+ "gold_session_ids": [
+ "answer_e0956e0a_abs_2",
+ "answer_e0956e0a_abs_1"
+ ]
+ },
+ {
+ "question_id": "edced276_abs",
+ "question_type": "multi-session",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.19342640361727081,
+ "mrr": 0.125,
+ "retrieved_session_ids": [
+ "sharegpt_u1AM5RT_88",
+ "66138e8d_1",
+ "ultrachat_410709",
+ "7a2bdd15_2",
+ "ultrachat_305072",
+ "7f4cd11e_1",
+ "sharegpt_1ZrnYv8_2",
+ "answer_60e8941a_abs_1",
+ "8b8e7cf8",
+ "sharegpt_YP2Ynd9_0"
+ ],
+ "gold_session_ids": [
+ "answer_60e8941a_abs_1",
+ "answer_60e8941a_abs_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_372c3eed_abs",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.46927872602275644,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_35c5419d_abs_1",
+ "b0a9de7f_1",
+ "6d3f5c68_4",
+ "54aa4875",
+ "9282ea7c_3",
+ "85dec0b4_1",
+ "3214139b_1",
+ "39948bcf_1",
+ "e6cc6157_4",
+ "48021f90_1"
+ ],
+ "gold_session_ids": [
+ "answer_35c5419d_abs_3",
+ "answer_35c5419d_abs_1",
+ "answer_35c5419d_abs_2"
+ ]
+ },
+ {
+ "question_id": "8a2466db",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6309297535714575,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "6a5b5a78",
+ "answer_edb03329",
+ "6be54739_3",
+ "3d0c9f89_1",
+ "sharegpt_MKMWjX0_25",
+ "e3cc9931_1",
+ "ultrachat_10500",
+ "a2e2cb72_3",
+ "b67748d1_1",
+ "6dcf5fa0_1"
+ ],
+ "gold_session_ids": [
+ "answer_edb03329"
+ ]
+ },
+ {
+ "question_id": "06878be2",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.3333333333333333,
+ "mrr": 0.14285714285714285,
+ "retrieved_session_ids": [
+ "465e7ef9",
+ "8fcfbe43_2",
+ "a6074da9_1",
+ "sharegpt_GSC090N_6",
+ "ultrachat_556731",
+ "ultrachat_319509",
+ "answer_555dfb94",
+ "a67c1862_4",
+ "sharegpt_5cxEGva_147",
+ "ca929779_1"
+ ],
+ "gold_session_ids": [
+ "answer_555dfb94"
+ ]
+ },
+ {
+ "question_id": "75832dbd",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 0,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0,
+ "mrr": 0.05,
+ "retrieved_session_ids": [
+ "1ef63c66",
+ "sharegpt_xni9yYO_6",
+ "6172cc72_1",
+ "47ec1674_2",
+ "c51583cd_1",
+ "d49b1a24_1",
+ "ultrachat_165893",
+ "e765a945_2",
+ "c884b67a_1",
+ "c7270a1b_1"
+ ],
+ "gold_session_ids": [
+ "answer_d87a6ef8"
+ ]
+ },
+ {
+ "question_id": "0edc2aef",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.3562071871080222,
+ "mrr": 0.16666666666666666,
+ "retrieved_session_ids": [
+ "8464304d_2",
+ "6a747f2e",
+ "f20e72e4_1",
+ "53dc1394",
+ "08ca1f31_2",
+ "answer_d586e9cd",
+ "25b1afbc",
+ "sharegpt_aVExml0_92",
+ "3f4611f4",
+ "33a813dd_2"
+ ],
+ "gold_session_ids": [
+ "answer_d586e9cd"
+ ]
+ },
+ {
+ "question_id": "35a27287",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.5,
+ "mrr": 0.3333333333333333,
+ "retrieved_session_ids": [
+ "880ee13e",
+ "58ae7c4f_3",
+ "answer_9b182436",
+ "203bf3fa_2",
+ "a98ff421_1",
+ "ultrachat_152094",
+ "e4a1b565_2",
+ "64071e19_2",
+ "c9b1c309_4",
+ "02f2ab8e_2"
+ ],
+ "gold_session_ids": [
+ "answer_9b182436"
+ ]
+ },
+ {
+ "question_id": "32260d93",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.3333333333333333,
+ "mrr": 0.14285714285714285,
+ "retrieved_session_ids": [
+ "sharegpt_q2uAW7H_0",
+ "3f787a78_4",
+ "573dc24b_1",
+ "6d3f5c68_4",
+ "sharegpt_zciCXP1_0",
+ "1b26bdd5_5",
+ "answer_0250ae1c",
+ "ultrachat_136348",
+ "ultrachat_171173",
+ "60ff3a8c"
+ ],
+ "gold_session_ids": [
+ "answer_0250ae1c"
+ ]
+ },
+ {
+ "question_id": "195a1a1b",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 0,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0,
+ "mrr": 0.0625,
+ "retrieved_session_ids": [
+ "32833e0b_3",
+ "sharegpt_WVr7Jiq_0",
+ "ce79a004",
+ "89a1800c_2",
+ "sharegpt_O3mtxv4_0",
+ "cc50c6a9",
+ "9f12daa1_1",
+ "af0ab6ab",
+ "ultrachat_510178",
+ "d03b6a05_2"
+ ],
+ "gold_session_ids": [
+ "answer_6dc4305e"
+ ]
+ },
+ {
+ "question_id": "afdc33df",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_8549e5e0",
+ "d03b6a05_3",
+ "ultrachat_214872",
+ "43b53ac4_1",
+ "d5128b51",
+ "ad8d9f0f",
+ "22f9f163_3",
+ "ultrachat_399348",
+ "96b6e5e8_2",
+ "b6558bfb"
+ ],
+ "gold_session_ids": [
+ "answer_8549e5e0"
+ ]
+ },
+ {
+ "question_id": "caf03d32",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_2fc6aabb",
+ "a52e072b",
+ "ec806228",
+ "6b137bce_4",
+ "eb0d8dc1_6",
+ "bd6d0687",
+ "ultrachat_200859",
+ "4b1af06f_1",
+ "78369dbb",
+ "6f0e184f"
+ ],
+ "gold_session_ids": [
+ "answer_2fc6aabb"
+ ]
+ },
+ {
+ "question_id": "54026fce",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.5,
+ "mrr": 0.3333333333333333,
+ "retrieved_session_ids": [
+ "670c49e6",
+ "1c95e152_1",
+ "answer_f7b22c66",
+ "518d26d3_6",
+ "6e92e417",
+ "5600db68_1",
+ "e68d6ffc_1",
+ "1fcb4134_3",
+ "52c34859_2",
+ "d60381f2_1"
+ ],
+ "gold_session_ids": [
+ "answer_f7b22c66"
+ ]
+ },
+ {
+ "question_id": "06f04340",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 0,
+ "recall_any_at_20": 0,
+ "ndcg_at_10": 0,
+ "mrr": 0,
+ "retrieved_session_ids": [
+ "sharegpt_Daapfhj_0",
+ "be2cb665",
+ "42924d15",
+ "14f9ee3c",
+ "b459f888_3",
+ "91223fd5_1",
+ "66bfa1db",
+ "0844dea6",
+ "728deb4d_4",
+ "4fba4832_1"
+ ],
+ "gold_session_ids": [
+ "answer_92d5f7cd"
+ ]
+ },
+ {
+ "question_id": "6b7dfb22",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_f6502d0f",
+ "56859d37",
+ "574df152_1",
+ "77a26018",
+ "8489e4ce_1",
+ "377f8fd9_1",
+ "e7b0637e_3",
+ "fc69fd44_3",
+ "e2e29a09",
+ "ultrachat_148325"
+ ],
+ "gold_session_ids": [
+ "answer_f6502d0f"
+ ]
+ },
+ {
+ "question_id": "1a1907b4",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 0,
+ "recall_any_at_20": 0,
+ "ndcg_at_10": 0,
+ "mrr": 0,
+ "retrieved_session_ids": [
+ "e42e7876_5",
+ "743f03a1_abs_1",
+ "4f3b36d4_5",
+ "d6f5639a_1",
+ "78dbd237",
+ "e622a1bd",
+ "61d3fec4_1",
+ "8e78fa70_1",
+ "4e3da326_1",
+ "aa930b56_1"
+ ],
+ "gold_session_ids": [
+ "answer_719502eb"
+ ]
+ },
+ {
+ "question_id": "09d032c9",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 0,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0,
+ "mrr": 0.08333333333333333,
+ "retrieved_session_ids": [
+ "d3971322_1",
+ "2a17ea02_1",
+ "3fc2244f",
+ "sharegpt_Wd2Dps4_0",
+ "26d9aaaf",
+ "21ef2d05_1",
+ "46bab85b",
+ "16ebc8f8_4",
+ "af631aa3_2",
+ "1fe34a92"
+ ],
+ "gold_session_ids": [
+ "answer_b10dce5e"
+ ]
+ },
+ {
+ "question_id": "38146c39",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.3562071871080222,
+ "mrr": 0.16666666666666666,
+ "retrieved_session_ids": [
+ "d22c2f04_5",
+ "a86b30e4",
+ "42234f98",
+ "15ed03f0_1",
+ "e5b1bc37",
+ "answer_772472c8",
+ "5806662d_1",
+ "87472dfd_3",
+ "286aa281_5",
+ "0908a2e6"
+ ],
+ "gold_session_ids": [
+ "answer_772472c8"
+ ]
+ },
+ {
+ "question_id": "d24813b1",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_7c0ade93",
+ "c2a34674_1",
+ "b7e64b5c_1",
+ "3124bb28_1",
+ "084802f9_3",
+ "22ce6625_2",
+ "9530353d_1",
+ "e08bf81f_4",
+ "61a46ff7_3",
+ "128f4e4d_3"
+ ],
+ "gold_session_ids": [
+ "answer_7c0ade93"
+ ]
+ },
+ {
+ "question_id": "57f827a0",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6309297535714575,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "f64c2982_1",
+ "answer_1bde8d3b",
+ "ultrachat_18651",
+ "d05d6391_1",
+ "1bff1675",
+ "6e110a53_1",
+ "762d288d_2",
+ "e85e0eaf_1",
+ "e9ad5914_1",
+ "2ef55f49_1"
+ ],
+ "gold_session_ids": [
+ "answer_1bde8d3b"
+ ]
+ },
+ {
+ "question_id": "95228167",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_5e613445",
+ "9f80607d",
+ "99e0fd9f_3",
+ "f75238a5_2",
+ "sharegpt_fwXicqL_262",
+ "aa71db11_1",
+ "0bac2789",
+ "3d2e66f2_2",
+ "7e17ae56_3",
+ "c34b6a1c_1"
+ ],
+ "gold_session_ids": [
+ "answer_5e613445"
+ ]
+ },
+ {
+ "question_id": "505af2f5",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6309297535714575,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "66bf4b03_1",
+ "answer_f3164f2c",
+ "4f2d6be4_2",
+ "9216c4d3_4",
+ "ultrachat_342931",
+ "56cdb8b8_2",
+ "69a42467_1",
+ "sharegpt_i59HUGf_0",
+ "ebeb4c51",
+ "629f17b4_2"
+ ],
+ "gold_session_ids": [
+ "answer_f3164f2c"
+ ]
+ },
+ {
+ "question_id": "75f70248",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_8ee04a2e",
+ "dcc8fc7f_2",
+ "ff67236f_3",
+ "adb1f904",
+ "95c9666f_5",
+ "2d74df23_4",
+ "15e366a1_1",
+ "6c5ef9fe",
+ "c1d13b9e",
+ "6ac43c5b_4"
+ ],
+ "gold_session_ids": [
+ "answer_8ee04a2e"
+ ]
+ },
+ {
+ "question_id": "d6233ab6",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 0,
+ "recall_any_at_20": 0,
+ "ndcg_at_10": 0,
+ "mrr": 0,
+ "retrieved_session_ids": [
+ "2c81cdf8_3",
+ "sharegpt_k4i6Hmp_8",
+ "87cfeb28_2",
+ "0e726047",
+ "cb05ed10",
+ "7af38385_2",
+ "e419b7c3_4",
+ "e064120a",
+ "426e7403_2",
+ "c927ffbb"
+ ],
+ "gold_session_ids": [
+ "answer_b0fac439"
+ ]
+ },
+ {
+ "question_id": "1da05512",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.43067655807339306,
+ "mrr": 0.25,
+ "retrieved_session_ids": [
+ "f5a2e179_1",
+ "sharegpt_mANdO1O_66",
+ "ae1cd5f3",
+ "answer_4d3be2ab",
+ "ce3ad11f_2",
+ "aea71da3_2",
+ "da1ea197_1",
+ "sharegpt_eSTYwMj_14",
+ "402e0082_4",
+ "6b93018e_2"
+ ],
+ "gold_session_ids": [
+ "answer_4d3be2ab"
+ ]
+ },
+ {
+ "question_id": "fca70973",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.3333333333333333,
+ "mrr": 0.14285714285714285,
+ "retrieved_session_ids": [
+ "f504b269",
+ "312bf939",
+ "fc005760",
+ "5d5e80c5_1",
+ "sharegpt_lSeiu11_0",
+ "2f2884ad_1",
+ "answer_a1e169b1",
+ "83ea6306_1",
+ "4e3f0831_1",
+ "09cff8ac"
+ ],
+ "gold_session_ids": [
+ "answer_a1e169b1"
+ ]
+ },
+ {
+ "question_id": "b6025781",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_8414cc57",
+ "f20c73f5_3",
+ "6446f6e6",
+ "2ef53f61_1",
+ "ddb3ab46",
+ "bbca6598_3",
+ "1a693999_4",
+ "22db3cc3_5",
+ "1d6744b5_1",
+ "89d49715_2"
+ ],
+ "gold_session_ids": [
+ "answer_8414cc57"
+ ]
+ },
+ {
+ "question_id": "a89d7624",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.38685280723454163,
+ "mrr": 0.2,
+ "retrieved_session_ids": [
+ "a479a7d6_4",
+ "082fc07f_1",
+ "99f2f5b1_1",
+ "sharegpt_G3RxeFJ_94",
+ "answer_8f15ac24",
+ "39a056ae",
+ "685d0b0d",
+ "ultrachat_30787",
+ "84f13d49_2",
+ "02f70006_1"
+ ],
+ "gold_session_ids": [
+ "answer_8f15ac24"
+ ]
+ },
+ {
+ "question_id": "b0479f84",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_30f63ddb",
+ "4f37e5ec",
+ "abc2ac45",
+ "6af947bf_1",
+ "1cde7ee4_1",
+ "79019170_1",
+ "ultrachat_263918",
+ "457b1adb_1",
+ "ultrachat_232424",
+ "94cffbd2_2"
+ ],
+ "gold_session_ids": [
+ "answer_30f63ddb"
+ ]
+ },
+ {
+ "question_id": "1d4e3b97",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.3333333333333333,
+ "mrr": 0.14285714285714285,
+ "retrieved_session_ids": [
+ "809cabca_1",
+ "fb2368b8_3",
+ "5cd6ab1b",
+ "243ca32d",
+ "c7772f2d_2",
+ "ecb24dd8_2",
+ "answer_e6b6353d",
+ "fd26f9f4_1",
+ "c51583cd_3",
+ "8c7ae899_2"
+ ],
+ "gold_session_ids": [
+ "answer_e6b6353d"
+ ]
+ },
+ {
+ "question_id": "07b6f563",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6309297535714575,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "5e7b1c98_1",
+ "answer_d03098f9",
+ "55c267aa",
+ "fd18e0dd_2",
+ "ddf41a24",
+ "sharegpt_72kPQPm_11",
+ "0ac1a6f9_2",
+ "66ab6260",
+ "dcc7f430_1",
+ "ultrachat_480403"
+ ],
+ "gold_session_ids": [
+ "answer_d03098f9"
+ ]
+ },
+ {
+ "question_id": "1c0ddc50",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_8da8c7e0",
+ "b33e89b5_1",
+ "2f0c1f4e",
+ "bdf33c73_2",
+ "2aa70c9c_1",
+ "c7ca6dff",
+ "a0aa5035",
+ "3d006c58_2",
+ "2566382f_2",
+ "0a7d5bf6_2"
+ ],
+ "gold_session_ids": [
+ "answer_8da8c7e0"
+ ]
+ },
+ {
+ "question_id": "0a34ad58",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6309297535714575,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "9ca2a195_1",
+ "answer_cebb7159",
+ "1e65e79a",
+ "90342357_3",
+ "f849dd04",
+ "4b495a04_1",
+ "58ad78d6_2",
+ "51fe163e_2",
+ "b869d7ed_2",
+ "dd3d8e73_1"
+ ],
+ "gold_session_ids": [
+ "answer_cebb7159"
+ ]
+ },
+ {
+ "question_id": "d3ab962e",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.5802792108518124,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "166d4800",
+ "answer_5237bb0b_2",
+ "1980bbfa_2",
+ "ultrachat_164834",
+ "ultrachat_155769",
+ "b66e4f25_2",
+ "013b320f",
+ "answer_5237bb0b_1",
+ "ultrachat_503039",
+ "ae15e8b6_3"
+ ],
+ "gold_session_ids": [
+ "answer_5237bb0b_2",
+ "answer_5237bb0b_1"
+ ]
+ },
+ {
+ "question_id": "2311e44b",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9197207891481876,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_bf633415_1",
+ "9b3611d7_1",
+ "answer_bf633415_2",
+ "16bc615e_1",
+ "ultrachat_458191",
+ "5854eebc_2",
+ "1e53c034",
+ "77753dd9_1",
+ "81ab8aec_1",
+ "sharegpt_hi2xiJT_14"
+ ],
+ "gold_session_ids": [
+ "answer_bf633415_2",
+ "answer_bf633415_1"
+ ]
+ },
+ {
+ "question_id": "cc06de0d",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8772153153380493,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_4fb01417_1",
+ "670c49e6",
+ "01551a39_2",
+ "answer_4fb01417_2",
+ "91c71874_2",
+ "d5eab084_2",
+ "ccb9811f",
+ "9863a1c8_3",
+ "92d74e73",
+ "55033f6f"
+ ],
+ "gold_session_ids": [
+ "answer_4fb01417_2",
+ "answer_4fb01417_1"
+ ]
+ },
+ {
+ "question_id": "a11281a2",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_c69ee1f9_2",
+ "answer_c69ee1f9_1",
+ "7a88956e",
+ "4b1af06f_2",
+ "9cda33e5",
+ "bb2180e9_3",
+ "c4c4d85a",
+ "ultrachat_55839",
+ "9208d3be",
+ "14feb919"
+ ],
+ "gold_session_ids": [
+ "answer_c69ee1f9_2",
+ "answer_c69ee1f9_1"
+ ]
+ },
+ {
+ "question_id": "4f54b7c9",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_50940cb7_1",
+ "answer_50940cb7_2",
+ "b89f68c4_1",
+ "22eebf8a_2",
+ "8a834aee_1",
+ "bcfd4ab7",
+ "663337ce_1",
+ "dde142b2_1",
+ "sharegpt_NPdQIWB_0",
+ "ultrachat_576691"
+ ],
+ "gold_session_ids": [
+ "answer_50940cb7_2",
+ "answer_50940cb7_1"
+ ]
+ },
+ {
+ "question_id": "85fa3a3f",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_dcb18b9b_2",
+ "answer_dcb18b9b_1",
+ "b7fd3b48",
+ "4f52229e",
+ "83f89c02",
+ "0d66562c_1",
+ "71110dbd_1",
+ "93ff2f73",
+ "8f707521_1",
+ "c6c06e6e"
+ ],
+ "gold_session_ids": [
+ "answer_dcb18b9b_2",
+ "answer_dcb18b9b_1"
+ ]
+ },
+ {
+ "question_id": "9aaed6a3",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8175295903539445,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_353d3c6d_2",
+ "c846422a_1",
+ "11e942da_2",
+ "3b715074",
+ "sharegpt_HdWyKUy_0",
+ "c95f2a09_2",
+ "answer_353d3c6d_1",
+ "ae332229_2",
+ "3f9f66f1_1",
+ "2e89bd03_1"
+ ],
+ "gold_session_ids": [
+ "answer_353d3c6d_2",
+ "answer_353d3c6d_1"
+ ]
+ },
+ {
+ "question_id": "1f2b8d4f",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9197207891481876,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_d8454588_1",
+ "9ee4da12_1",
+ "answer_d8454588_2",
+ "d3ee5958_2",
+ "be1c6872",
+ "76a39abc",
+ "ultrachat_401506",
+ "ultrachat_391569",
+ "c8c64252",
+ "ultrachat_413177"
+ ],
+ "gold_session_ids": [
+ "answer_d8454588_1",
+ "answer_d8454588_2"
+ ]
+ },
+ {
+ "question_id": "e6041065",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_4eb6d671_1",
+ "answer_4eb6d671_2",
+ "d79173aa_1",
+ "bdcee74e_3",
+ "4a006b77",
+ "ultrachat_407355",
+ "sharegpt_YV9bnz5_0",
+ "832e4ecf",
+ "4c053e7e_1",
+ "b73a4d04_1"
+ ],
+ "gold_session_ids": [
+ "answer_4eb6d671_2",
+ "answer_4eb6d671_1"
+ ]
+ },
+ {
+ "question_id": "51c32626",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_58820c75_1",
+ "answer_58820c75_2",
+ "sharegpt_hzwc0HI_13",
+ "c051e740_2",
+ "b38766c1_3",
+ "809cbce9_1",
+ "sharegpt_jBafAKm_0",
+ "932a97c6",
+ "932ea3bf_1",
+ "fb62ade8"
+ ],
+ "gold_session_ids": [
+ "answer_58820c75_1",
+ "answer_58820c75_2"
+ ]
+ },
+ {
+ "question_id": "d905b33f",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8503449055347546,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_85a77c48_1",
+ "8cfa91c3_2",
+ "8b69d1ae",
+ "6423d6a4_2",
+ "answer_85a77c48_2",
+ "ultrachat_381881",
+ "2d364fb0",
+ "3301f749_1",
+ "368276ab_3",
+ "dd4a0e15"
+ ],
+ "gold_session_ids": [
+ "answer_85a77c48_2",
+ "answer_85a77c48_1"
+ ]
+ },
+ {
+ "question_id": "7405e8b1",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_80323f3f_1",
+ "answer_80323f3f_2",
+ "f2f2a606_3",
+ "a6b00510_4",
+ "c578da1f_1",
+ "6e983235_1",
+ "8eeb4704",
+ "6c1c5bc3_2",
+ "643d285a_3",
+ "e40d733b_2"
+ ],
+ "gold_session_ids": [
+ "answer_80323f3f_1",
+ "answer_80323f3f_2"
+ ]
+ },
+ {
+ "question_id": "f35224e0",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6240505200038379,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "fd7becaf_1",
+ "answer_e9bb9500_2",
+ "8e608386",
+ "2ea6fda4_1",
+ "answer_e9bb9500_1",
+ "96da07f9_4",
+ "4b285f83",
+ "e184e4c3_1",
+ "sharegpt_0DiDwpY_23",
+ "sharegpt_Pqhvz5n_0"
+ ],
+ "gold_session_ids": [
+ "answer_e9bb9500_2",
+ "answer_e9bb9500_1"
+ ]
+ },
+ {
+ "question_id": "6456829e",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9197207891481876,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_743f03a1_2",
+ "c44d1533_2",
+ "answer_743f03a1_1",
+ "ultrachat_247641",
+ "a8c73552",
+ "97d3d674",
+ "fe7b6b40_1",
+ "385683f0_3",
+ "ultrachat_494325",
+ "sharegpt_LxQSGQ9_0"
+ ],
+ "gold_session_ids": [
+ "answer_743f03a1_2",
+ "answer_743f03a1_1"
+ ]
+ },
+ {
+ "question_id": "a4996e51",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_feb5f98a_2",
+ "answer_feb5f98a_1",
+ "sharegpt_LYRqOF7_9",
+ "28a64f28_2",
+ "4cc9270f",
+ "b1d9eb66_3",
+ "16bd5ea5_1",
+ "252715f8_3",
+ "600077b0_1",
+ "2ca99347_2"
+ ],
+ "gold_session_ids": [
+ "answer_feb5f98a_1",
+ "answer_feb5f98a_2"
+ ]
+ },
+ {
+ "question_id": "3c1045c8",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.38685280723454163,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "ultrachat_20893",
+ "answer_c8cc60d6_2",
+ "ecd24d5e_1",
+ "sharegpt_jUUvPID_0",
+ "b52f03a1_1",
+ "sharegpt_NTHvrGV_0",
+ "sharegpt_oIB2JG9_0",
+ "50b705ab_2",
+ "sharegpt_jSnesqJ_43",
+ "be32ad25_2"
+ ],
+ "gold_session_ids": [
+ "answer_c8cc60d6_2",
+ "answer_c8cc60d6_1"
+ ]
+ },
+ {
+ "question_id": "60036106",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_e552e1f9_1",
+ "answer_e552e1f9_2",
+ "sharegpt_moARQa7_0",
+ "sharegpt_mtS0qOD_65",
+ "763f028a",
+ "c575291e_2",
+ "fcc6d66d_2",
+ "10857212_2",
+ "e2cdd030",
+ "43283702_1"
+ ],
+ "gold_session_ids": [
+ "answer_e552e1f9_1",
+ "answer_e552e1f9_2"
+ ]
+ },
+ {
+ "question_id": "681a1674",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_3be95d43_2",
+ "answer_3be95d43_1",
+ "ultrachat_436833",
+ "ultrachat_64650",
+ "9e411500_1",
+ "7d33fcb4_2",
+ "sharegpt_8Kzmn95_0",
+ "ultrachat_11300",
+ "50e8f2b5_1",
+ "a21c4257"
+ ],
+ "gold_session_ids": [
+ "answer_3be95d43_1",
+ "answer_3be95d43_2"
+ ]
+ },
+ {
+ "question_id": "e25c3b8d",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6240505200038379,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "sharegpt_gQaSoiF_13",
+ "answer_6702277b_2",
+ "ff201786_1",
+ "90fea364_1",
+ "answer_6702277b_1",
+ "35b030f3_2",
+ "746d0c00_1",
+ "5237d8c6",
+ "sharegpt_Nk10TjE_29",
+ "20fa75e8_3"
+ ],
+ "gold_session_ids": [
+ "answer_6702277b_1",
+ "answer_6702277b_2"
+ ]
+ },
+ {
+ "question_id": "4adc0475",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.38685280723454163,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "ultrachat_463810",
+ "answer_6efce493_2",
+ "f82637ed_1",
+ "ultrachat_186558",
+ "debc34d7_2",
+ "fa0fa74d_2",
+ "bb27df5e_2",
+ "sharegpt_7nYPy2P_29",
+ "ultrachat_170415",
+ "183d2822_1"
+ ],
+ "gold_session_ids": [
+ "answer_6efce493_1",
+ "answer_6efce493_2"
+ ]
+ },
+ {
+ "question_id": "4bc144e2",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9197207891481876,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_9ef115d4_1",
+ "42bcee92_2",
+ "answer_9ef115d4_2",
+ "60207b41_3",
+ "2e2ae792_1",
+ "d75245ea",
+ "a177e66a_2",
+ "bf9c39e9_1",
+ "1fd71f84",
+ "9416c7de"
+ ],
+ "gold_session_ids": [
+ "answer_9ef115d4_1",
+ "answer_9ef115d4_2"
+ ]
+ },
+ {
+ "question_id": "ef66a6e5",
+ "question_type": "multi-session",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.21840743681816419,
+ "mrr": 0.16666666666666666,
+ "retrieved_session_ids": [
+ "sharegpt_bitlG4U_0",
+ "3f05a03e_1",
+ "147ab7e9_6",
+ "ultrachat_428073",
+ "ultrachat_61361",
+ "answer_f7fd1029_1",
+ "4c08398b_1",
+ "502b6b52_4",
+ "41e778e4_2",
+ "295645f5_1"
+ ],
+ "gold_session_ids": [
+ "answer_f7fd1029_2",
+ "answer_f7fd1029_1"
+ ]
+ },
+ {
+ "question_id": "5025383b",
+ "question_type": "multi-session",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.20438239758848611,
+ "mrr": 0.14285714285714285,
+ "retrieved_session_ids": [
+ "2d21f921",
+ "0e349efa",
+ "sharegpt_SwxX91i_5",
+ "2b8fe437_1",
+ "a268827b_2",
+ "f40fa3ed",
+ "answer_688f9a3f_2",
+ "22f9af5f",
+ "042c55a5",
+ "baf0243b_2"
+ ],
+ "gold_session_ids": [
+ "answer_688f9a3f_1",
+ "answer_688f9a3f_2"
+ ]
+ },
+ {
+ "question_id": "a1cc6108",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6131471927654584,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_17dc2f5b_2",
+ "001cefa7_2",
+ "a841ef89_1",
+ "5aa88956_2",
+ "sharegpt_9eEV9T3_8",
+ "sharegpt_jpiaPoJ_29",
+ "298a07de_5",
+ "59b87367",
+ "fb328ace_1",
+ "31e254b5"
+ ],
+ "gold_session_ids": [
+ "answer_17dc2f5b_2",
+ "answer_17dc2f5b_1"
+ ]
+ },
+ {
+ "question_id": "9ee3ecd6",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_66c23110_2",
+ "answer_66c23110_1",
+ "fd56c5bd_2",
+ "fa0fa74d_2",
+ "9b1b9069_1",
+ "6577a6d9_1",
+ "f550a835_1",
+ "42d9467c_1",
+ "a9eda378",
+ "ultrachat_185685"
+ ],
+ "gold_session_ids": [
+ "answer_66c23110_1",
+ "answer_66c23110_2"
+ ]
+ },
+ {
+ "question_id": "3fdac837",
+ "question_type": "multi-session",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 0,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0,
+ "mrr": 0.08333333333333333,
+ "retrieved_session_ids": [
+ "09d021e7_1",
+ "a98ff421_7",
+ "d1165be3_2",
+ "30c92c0c_1",
+ "ultrachat_293546",
+ "c85ef24e_1",
+ "92b06116_1",
+ "5298db7c_1",
+ "a1334327_1",
+ "9b446113_2"
+ ],
+ "gold_session_ids": [
+ "answer_419d21d5_2",
+ "answer_419d21d5_1"
+ ]
+ },
+ {
+ "question_id": "91b15a6e",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_5404a208_1",
+ "answer_5404a208_2",
+ "ultrachat_42513",
+ "ultrachat_326780",
+ "sharegpt_zS41R74_0",
+ "6169ef55_1",
+ "sharegpt_0ktU0jP_16",
+ "sharegpt_7r7RS6F_0",
+ "7cfeb89c_2",
+ "b28cdea2"
+ ],
+ "gold_session_ids": [
+ "answer_5404a208_1",
+ "answer_5404a208_2"
+ ]
+ },
+ {
+ "question_id": "27016adc",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9197207891481876,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_a37bdf22_1",
+ "ff5e0db0_1",
+ "answer_a37bdf22_2",
+ "sharegpt_ZbXRfgJ_0",
+ "28ea20bf",
+ "199a5225_2",
+ "46548fab_3",
+ "ultrachat_566359",
+ "dd25eeb5_8",
+ "fa858208_2"
+ ],
+ "gold_session_ids": [
+ "answer_a37bdf22_2",
+ "answer_a37bdf22_1"
+ ]
+ },
+ {
+ "question_id": "720133ac",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_c9dfeaea_2",
+ "answer_c9dfeaea_1",
+ "4c128d97_2",
+ "sharegpt_Km6yx9W_13",
+ "de43030f_2",
+ "ultrachat_479833",
+ "0cf86cbe_2",
+ "sharegpt_sG53ibg_116",
+ "ultrachat_508711",
+ "e200d96c_3"
+ ],
+ "gold_session_ids": [
+ "answer_c9dfeaea_1",
+ "answer_c9dfeaea_2"
+ ]
+ },
+ {
+ "question_id": "77eafa52",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8772153153380493,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_33c251f0_1",
+ "7b13b4d6_2",
+ "sharegpt_UUFCRmF_0",
+ "answer_33c251f0_2",
+ "sharegpt_qEukY6n_14",
+ "ultrachat_309057",
+ "41a87f13_1",
+ "fc9a62c0_1",
+ "174cccbc",
+ "sharegpt_C9GjDsF_0"
+ ],
+ "gold_session_ids": [
+ "answer_33c251f0_2",
+ "answer_33c251f0_1"
+ ]
+ },
+ {
+ "question_id": "8979f9ec",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_35e36341_2",
+ "answer_35e36341_1",
+ "bcc9382a_2",
+ "8540055e_1",
+ "1064ed80_2",
+ "1980bbfa_2",
+ "052d7def",
+ "a6b4c4c2_1",
+ "ultrachat_466891",
+ "ultrachat_562471"
+ ],
+ "gold_session_ids": [
+ "answer_35e36341_1",
+ "answer_35e36341_2"
+ ]
+ },
+ {
+ "question_id": "0100672e",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_35c9798c_2",
+ "answer_35c9798c_1",
+ "07313722_2",
+ "sharegpt_sDfu38Q_0",
+ "326e86e0_3",
+ "b167a4a8",
+ "f60c9de5",
+ "ultrachat_372868",
+ "c694fe7b_1",
+ "0f53a05e_4"
+ ],
+ "gold_session_ids": [
+ "answer_35c9798c_2",
+ "answer_35c9798c_1"
+ ]
+ },
+ {
+ "question_id": "a96c20ee",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.7903864795495061,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ef84b994_1",
+ "32e292f2",
+ "dff89d9d_1",
+ "9ca3534f_1",
+ "ultrachat_259222",
+ "sharegpt_00qHEQ6_0",
+ "sharegpt_lWLBUhQ_539",
+ "c69eb8ee_4",
+ "3d2e66f2_2",
+ "answer_ef84b994_2"
+ ],
+ "gold_session_ids": [
+ "answer_ef84b994_1",
+ "answer_ef84b994_2"
+ ]
+ },
+ {
+ "question_id": "92a0aa75",
+ "question_type": "multi-session",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 0,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0,
+ "mrr": 0.09090909090909091,
+ "retrieved_session_ids": [
+ "ultrachat_402091",
+ "sharegpt_hPax8pw_11",
+ "a5854a8d_2",
+ "bd00a24d_2",
+ "sharegpt_xkWMnwQ_0",
+ "edd89480_1",
+ "6da0bba2_1",
+ "2bdd78cb_1",
+ "1c0921ee_3",
+ "173e2e47"
+ ],
+ "gold_session_ids": [
+ "answer_6cb8f792_1",
+ "answer_6cb8f792_2"
+ ]
+ },
+ {
+ "question_id": "3fe836c9",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.48381288316677695,
+ "mrr": 0.3333333333333333,
+ "retrieved_session_ids": [
+ "c81897cd_2",
+ "0a42783a",
+ "answer_1bb63ea5_1",
+ "f0d7d364",
+ "8129a7a8",
+ "c4293e27_2",
+ "a3cfa973",
+ "ba184a34",
+ "e94b6708",
+ "answer_1bb63ea5_2"
+ ],
+ "gold_session_ids": [
+ "answer_1bb63ea5_2",
+ "answer_1bb63ea5_1"
+ ]
+ },
+ {
+ "question_id": "1c549ce4",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_4cb841a8_1",
+ "answer_4cb841a8_2",
+ "d743153b_2",
+ "f999b05c_1",
+ "ultrachat_441412",
+ "f1e4daa8_2",
+ "sharegpt_DwfIyV9_0",
+ "3d006c58_2",
+ "35f052f9_2",
+ "bd4110bc"
+ ],
+ "gold_session_ids": [
+ "answer_4cb841a8_1",
+ "answer_4cb841a8_2"
+ ]
+ },
+ {
+ "question_id": "6c49646a",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.23719771276929622,
+ "mrr": 0.2,
+ "retrieved_session_ids": [
+ "1a24ef7c_5",
+ "ultrachat_565061",
+ "fffa6e59",
+ "313404c6_1",
+ "answer_cc1ced21_1",
+ "8acb29f9_2",
+ "403ee298_3",
+ "9f220c66_1",
+ "6a2f306a",
+ "6155addd_2"
+ ],
+ "gold_session_ids": [
+ "answer_cc1ced21_2",
+ "answer_cc1ced21_1"
+ ]
+ },
+ {
+ "question_id": "1192316e",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_e184e4c3_1",
+ "answer_e184e4c3_2",
+ "c5db849c_1",
+ "69b90254",
+ "sharegpt_pZ3I356_11",
+ "66e58c90_3",
+ "55f7cc4f_3",
+ "199a5225_1",
+ "6f8cd952",
+ "52c6d59a_4"
+ ],
+ "gold_session_ids": [
+ "answer_e184e4c3_2",
+ "answer_e184e4c3_1"
+ ]
+ },
+ {
+ "question_id": "0ea62687",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6934264036172708,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "5dce60dd",
+ "answer_dc5e537d_2",
+ "answer_dc5e537d_1",
+ "92ee65e1",
+ "eb336de0_2",
+ "d1ea55c0_1",
+ "c5e9cbd8_1",
+ "2fd28490_1",
+ "fba28a35_3",
+ "4a5ee697_1"
+ ],
+ "gold_session_ids": [
+ "answer_dc5e537d_1",
+ "answer_dc5e537d_2"
+ ]
+ },
+ {
+ "question_id": "67e0d0f2",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8772153153380493,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_3a5010af_2",
+ "75a48603_1",
+ "e89b6be2_2",
+ "answer_3a5010af_1",
+ "sharegpt_uD7CBng_30",
+ "78e40ca2_2",
+ "ultrachat_170247",
+ "ultrachat_203993",
+ "2252d3dd",
+ "93bed3d8_2"
+ ],
+ "gold_session_ids": [
+ "answer_3a5010af_2",
+ "answer_3a5010af_1"
+ ]
+ },
+ {
+ "question_id": "bb7c3b45",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_de64539a_1",
+ "answer_de64539a_2",
+ "7a4e1601",
+ "sharegpt_I0uNTjm_36",
+ "b9220c40",
+ "752b10e6_1",
+ "07548ed6",
+ "f3d43a12_3",
+ "3891baea_4",
+ "ultrachat_139470"
+ ],
+ "gold_session_ids": [
+ "answer_de64539a_1",
+ "answer_de64539a_2"
+ ]
+ },
+ {
+ "question_id": "ba358f49",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.3065735963827292,
+ "mrr": 0.3333333333333333,
+ "retrieved_session_ids": [
+ "9cc709cb",
+ "ultrachat_317048",
+ "answer_cbd08e3c_1",
+ "91c71874_2",
+ "d964ed3b",
+ "95cb552e_1",
+ "sharegpt_bZQ9094_0",
+ "f6859b48_2",
+ "385683f0_2",
+ "4f838497_2"
+ ],
+ "gold_session_ids": [
+ "answer_cbd08e3c_1",
+ "answer_cbd08e3c_2"
+ ]
+ },
+ {
+ "question_id": "61f8c8f8",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9197207891481876,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_872e8da2_1",
+ "a3107e2a_1",
+ "answer_872e8da2_2",
+ "1a24ef7c_5",
+ "7d5d9dd3",
+ "7af38385_2",
+ "2154c810_1",
+ "sharegpt_7jGo66i_99",
+ "7b88c38b_2",
+ "ultrachat_211983"
+ ],
+ "gold_session_ids": [
+ "answer_872e8da2_2",
+ "answer_872e8da2_1"
+ ]
+ },
+ {
+ "question_id": "60159905",
+ "question_type": "multi-session",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.36181498356361597,
+ "mrr": 0.1111111111111111,
+ "retrieved_session_ids": [
+ "sharegpt_RtCgJK2_508",
+ "b3ef2cd4_1",
+ "15998e39_4",
+ "5faf5962_3",
+ "ultrachat_190672",
+ "afe04238_1",
+ "eef7c3fa_2",
+ "ultrachat_168392",
+ "answer_75eca223_2",
+ "answer_75eca223_1"
+ ],
+ "gold_session_ids": [
+ "answer_75eca223_2",
+ "answer_75eca223_1"
+ ]
+ },
+ {
+ "question_id": "ef9cf60a",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_87e3a1cb_2",
+ "answer_87e3a1cb_1",
+ "7b88c38b_1",
+ "sharegpt_mvtmr4e_19",
+ "2357de36_2",
+ "sharegpt_6fU0WOe_56",
+ "sharegpt_NelPQLT_0",
+ "0275bceb_2",
+ "d677eb06",
+ "3cd4a679"
+ ],
+ "gold_session_ids": [
+ "answer_87e3a1cb_1",
+ "answer_87e3a1cb_2"
+ ]
+ },
+ {
+ "question_id": "73d42213",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6934264036172708,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "ultrachat_275858",
+ "answer_1881e7db_1",
+ "answer_1881e7db_2",
+ "sharegpt_ab6IEma_0",
+ "ultrachat_3128",
+ "sharegpt_lNG1bAS_0",
+ "a1251fef_2",
+ "7c2ec9e9_2",
+ "sharegpt_GKjb2be_0",
+ "3cb41ab8_1"
+ ],
+ "gold_session_ids": [
+ "answer_1881e7db_1",
+ "answer_1881e7db_2"
+ ]
+ },
+ {
+ "question_id": "bc149d6b",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.45560514958746035,
+ "mrr": 0.2,
+ "retrieved_session_ids": [
+ "4f435d20_2",
+ "451ecb23_3",
+ "7211c69e",
+ "ultrachat_461193",
+ "answer_92147866_1",
+ "answer_92147866_2",
+ "27d378b2_3",
+ "sharegpt_HullwHG_71",
+ "86935301_2",
+ "1b474144"
+ ],
+ "gold_session_ids": [
+ "answer_92147866_1",
+ "answer_92147866_2"
+ ]
+ },
+ {
+ "question_id": "099778bb",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_80d6d664_1",
+ "answer_80d6d664_2",
+ "6bb8af95_1",
+ "b1793bba",
+ "sharegpt_s1I3UkG_0",
+ "5b014f9b_1",
+ "d00ba6d0_2",
+ "sharegpt_O8djord_90",
+ "sharegpt_qyWsgp8_13",
+ "0a042aef_2"
+ ],
+ "gold_session_ids": [
+ "answer_80d6d664_2",
+ "answer_80d6d664_1"
+ ]
+ },
+ {
+ "question_id": "09ba9854",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_96c743d0_1",
+ "answer_96c743d0_2",
+ "ultrachat_447648",
+ "8682e696_2",
+ "df963ea2_1",
+ "32e6c308",
+ "8c63238e",
+ "5faf5962_3",
+ "ae32e467_2",
+ "f7644391_2"
+ ],
+ "gold_session_ids": [
+ "answer_96c743d0_2",
+ "answer_96c743d0_1"
+ ]
+ },
+ {
+ "question_id": "d6062bb9",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_23f3a657_1",
+ "answer_23f3a657_2",
+ "sharegpt_xuhzAgG_19",
+ "sharegpt_hcaZN94_647",
+ "ultrachat_485311",
+ "cd4ae1ad",
+ "f7644391_2",
+ "65d28641_2",
+ "59c704ad_2",
+ "b35610f4_4"
+ ],
+ "gold_session_ids": [
+ "answer_23f3a657_2",
+ "answer_23f3a657_1"
+ ]
+ },
+ {
+ "question_id": "157a136e",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.48247555939075504,
+ "mrr": 0.25,
+ "retrieved_session_ids": [
+ "377f8fd9_4",
+ "9b3a7f2c_4",
+ "ultrachat_375609",
+ "answer_8de18468_2",
+ "ultrachat_255132",
+ "answer_8de18468_1",
+ "sharegpt_xQQ30VB_0",
+ "298bf7a3_1",
+ "a2e2cb72_5",
+ "d161536e"
+ ],
+ "gold_session_ids": [
+ "answer_8de18468_2",
+ "answer_8de18468_1"
+ ]
+ },
+ {
+ "question_id": "c18a7dc8",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6131471927654584,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_2e2085fa_1",
+ "9d5a389d",
+ "f716720c_3",
+ "5022812c",
+ "fc6c3985",
+ "805528d6_1",
+ "3421bdbe_1",
+ "4f52229e",
+ "1c177942_3",
+ "ultrachat_301566"
+ ],
+ "gold_session_ids": [
+ "answer_2e2085fa_2",
+ "answer_2e2085fa_1"
+ ]
+ },
+ {
+ "question_id": "a3332713",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_16ece55f_2",
+ "answer_16ece55f_1",
+ "1c8a264a_4",
+ "67388dfc_2",
+ "117098d9_3",
+ "29fbbc99",
+ "f6859b48_3",
+ "a864e7aa_1",
+ "ef2400e0_1",
+ "sharegpt_bYymtlW_0"
+ ],
+ "gold_session_ids": [
+ "answer_16ece55f_2",
+ "answer_16ece55f_1"
+ ]
+ },
+ {
+ "question_id": "55241a1f",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_fa08bf49_1",
+ "answer_fa08bf49_2",
+ "9c9a22d8",
+ "76491ee4_1",
+ "sharegpt_dUoAhvI_71",
+ "2aeb1268",
+ "d117e2da",
+ "sharegpt_TzwvePR_0",
+ "46e4a8eb_2",
+ "60153a02_4"
+ ],
+ "gold_session_ids": [
+ "answer_fa08bf49_1",
+ "answer_fa08bf49_2"
+ ]
+ },
+ {
+ "question_id": "a08a253f",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_47152166_1",
+ "answer_47152166_2",
+ "6e5130ed_2",
+ "28209b6a_4",
+ "798fa5f2",
+ "4524c250_2",
+ "d79173aa_3",
+ "771570c5_1",
+ "83ca9337",
+ "a79f1a04_1"
+ ],
+ "gold_session_ids": [
+ "answer_47152166_2",
+ "answer_47152166_1"
+ ]
+ },
+ {
+ "question_id": "f0e564bc",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8772153153380493,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_cfcf5340_2",
+ "ultrachat_384067",
+ "55da3cbf_3",
+ "answer_cfcf5340_1",
+ "ultrachat_330960",
+ "932a97c6",
+ "1788a980",
+ "65ca99fb_2",
+ "sharegpt_W3YFpVK_7",
+ "6dfc99bf_2"
+ ],
+ "gold_session_ids": [
+ "answer_cfcf5340_1",
+ "answer_cfcf5340_2"
+ ]
+ },
+ {
+ "question_id": "078150f1",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.5706417189553201,
+ "mrr": 0.3333333333333333,
+ "retrieved_session_ids": [
+ "e75d8a9c",
+ "46e4a8eb_1",
+ "answer_254d8b09_1",
+ "answer_254d8b09_2",
+ "0e218647_2",
+ "57bf2079",
+ "f54d3a48",
+ "61b27d67_3",
+ "b18ba8d0_1",
+ "ultrachat_202549"
+ ],
+ "gold_session_ids": [
+ "answer_254d8b09_1",
+ "answer_254d8b09_2"
+ ]
+ },
+ {
+ "question_id": "8cf4d046",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6131471927654584,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_e2278b24_2",
+ "ultrachat_548964",
+ "a30f52a4",
+ "sharegpt_P1Okivz_9",
+ "c36f4196",
+ "sharegpt_WjpyexI_15",
+ "961b965c_2",
+ "6c6c8df7",
+ "410ec447",
+ "c7270a1b_1"
+ ],
+ "gold_session_ids": [
+ "answer_e2278b24_1",
+ "answer_e2278b24_2"
+ ]
+ },
+ {
+ "question_id": "a346bb18",
+ "question_type": "multi-session",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.20438239758848611,
+ "mrr": 0.14285714285714285,
+ "retrieved_session_ids": [
+ "c51572bb_2",
+ "5726dc37_2",
+ "84fb50bb_1",
+ "b4003083",
+ "53949711_2",
+ "59e96888_1",
+ "answer_4934b2d7_2",
+ "ultrachat_49611",
+ "ultrachat_300801",
+ "d3a4eb00_1"
+ ],
+ "gold_session_ids": [
+ "answer_4934b2d7_2",
+ "answer_4934b2d7_1"
+ ]
+ },
+ {
+ "question_id": "37f165cf",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_6b9b2b1e_2",
+ "answer_6b9b2b1e_1",
+ "20b1b65f_2",
+ "c85d1682_1",
+ "cdcf791b_1",
+ "7e8e790a",
+ "ultrachat_319568",
+ "50dd6bc1_1",
+ "c41e97c9_2",
+ "ultrachat_453124"
+ ],
+ "gold_session_ids": [
+ "answer_6b9b2b1e_2",
+ "answer_6b9b2b1e_1"
+ ]
+ },
+ {
+ "question_id": "8e91e7d9",
+ "question_type": "multi-session",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 0,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0,
+ "mrr": 0.08333333333333333,
+ "retrieved_session_ids": [
+ "2f980ae0_2",
+ "ultrachat_147574",
+ "c3023a45_4",
+ "4e3da326_1",
+ "4f8caea3_2",
+ "7a389bd1_7",
+ "c2381e1a_2",
+ "42e69993_2",
+ "sharegpt_PZrBCF2_0",
+ "44fc2b1e"
+ ],
+ "gold_session_ids": [
+ "answer_477ae455_1",
+ "answer_477ae455_2"
+ ]
+ },
+ {
+ "question_id": "87f22b4a",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6240505200038379,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "ultrachat_182225",
+ "answer_f56e6152_1",
+ "02b99ec3_1",
+ "ea28bb75",
+ "answer_f56e6152_2",
+ "c6015a0c_3",
+ "889076d6",
+ "ultrachat_350147",
+ "0320f558_2",
+ "28741edc_1"
+ ],
+ "gold_session_ids": [
+ "answer_f56e6152_2",
+ "answer_f56e6152_1"
+ ]
+ },
+ {
+ "question_id": "e56a43b9",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_430d0a87_2",
+ "answer_430d0a87_1",
+ "7596ba6a",
+ "83e14506_2",
+ "38e81260_7",
+ "57a6a404_3",
+ "b28cdea2",
+ "sharegpt_iylDztx_14",
+ "ultrachat_267137",
+ "f684ac4c_3"
+ ],
+ "gold_session_ids": [
+ "answer_430d0a87_1",
+ "answer_430d0a87_2"
+ ]
+ },
+ {
+ "question_id": "efc3f7c2",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.7977228895450266,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_fc81d1a2_1",
+ "ultrachat_38938",
+ "fc005760",
+ "bbca6598_1",
+ "bbdc7b0a",
+ "88a7faa7_1",
+ "1ea13fbd_2",
+ "8a7df17e_1",
+ "answer_fc81d1a2_2",
+ "0f013a55"
+ ],
+ "gold_session_ids": [
+ "answer_fc81d1a2_1",
+ "answer_fc81d1a2_2"
+ ]
+ },
+ {
+ "question_id": "21d02d0d",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6240505200038379,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "42d9467c_2",
+ "answer_2c637141_1",
+ "ultrachat_71299",
+ "sharegpt_adeEi7y_7",
+ "answer_2c637141_2",
+ "2e28f3db",
+ "cf021b36_2",
+ "ultrachat_270892",
+ "sharegpt_B9jlITg_0",
+ "a15fa6eb"
+ ],
+ "gold_session_ids": [
+ "answer_2c637141_2",
+ "answer_2c637141_1"
+ ]
+ },
+ {
+ "question_id": "2311e44b_abs",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9197207891481876,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_bf633415_abs_1",
+ "313404c6_2",
+ "answer_bf633415_abs_2",
+ "4c8afa9c_1",
+ "90342357_2",
+ "5e906cb0_2",
+ "09d021e7_2",
+ "e9e9027c_2",
+ "35915bf5",
+ "393045ae_3"
+ ],
+ "gold_session_ids": [
+ "answer_bf633415_abs_1",
+ "answer_bf633415_abs_2"
+ ]
+ },
+ {
+ "question_id": "6456829e_abs",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9197207891481876,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_743f03a1_abs_2",
+ "b61cc91b_2",
+ "answer_743f03a1_abs_1",
+ "3d8d4828_2",
+ "sharegpt_Hc0Zm6i_0",
+ "ultrachat_417298",
+ "d20bb5aa_1",
+ "1b26bdd5_5",
+ "sharegpt_ZxGSwYN_7",
+ "4015360f_2"
+ ],
+ "gold_session_ids": [
+ "answer_743f03a1_abs_1",
+ "answer_743f03a1_abs_2"
+ ]
+ },
+ {
+ "question_id": "e5ba910e_abs",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9197207891481876,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_e49ed9d3_abs_2",
+ "ultrachat_282969",
+ "answer_e49ed9d3_abs_1",
+ "6339aa1d_2",
+ "7e0fa6a6",
+ "a239515b",
+ "ultrachat_402969",
+ "6d013bad_3",
+ "8e8c87fe_1",
+ "ultrachat_165962"
+ ],
+ "gold_session_ids": [
+ "answer_e49ed9d3_abs_1",
+ "answer_e49ed9d3_abs_2"
+ ]
+ },
+ {
+ "question_id": "a96c20ee_abs",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9197207891481876,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ef84b994_abs_1",
+ "4163e709_3",
+ "answer_ef84b994_abs_2",
+ "fee68529_2",
+ "47103d08",
+ "sharegpt_jM9e92F_0",
+ "ultrachat_231397",
+ "ultrachat_293509",
+ "1917d461",
+ "ultrachat_24513"
+ ],
+ "gold_session_ids": [
+ "answer_ef84b994_abs_1",
+ "answer_ef84b994_abs_2"
+ ]
+ },
+ {
+ "question_id": "ba358f49_abs",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.38685280723454163,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "730a1d1d_2",
+ "answer_cbd08e3c_abs_1",
+ "1162611c",
+ "5829ab38",
+ "9844f740",
+ "sharegpt_7kFao5h_0",
+ "ultrachat_387251",
+ "9b3a7f2c_2",
+ "11f043cd_2",
+ "sharegpt_XWLA0IO_0"
+ ],
+ "gold_session_ids": [
+ "answer_cbd08e3c_abs_1",
+ "answer_cbd08e3c_abs_2"
+ ]
+ },
+ {
+ "question_id": "09ba9854_abs",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9197207891481876,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_96c743d0_abs_1",
+ "85ca9420",
+ "answer_96c743d0_abs_2",
+ "66e58c90_2",
+ "a50b0d73_2",
+ "ultrachat_342361",
+ "338ac116_2",
+ "9beab022_1",
+ "25021ded",
+ "d8e9a53a_3"
+ ],
+ "gold_session_ids": [
+ "answer_96c743d0_abs_1",
+ "answer_96c743d0_abs_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_59149c77",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9197207891481876,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_d00ba6d0_2",
+ "9e2c2a6c_2",
+ "answer_d00ba6d0_1",
+ "9242fbe3_1",
+ "sharegpt_XlNs8Lz_199",
+ "148f87a0_2",
+ "54a05e72_2",
+ "31829c1b_1",
+ "sharegpt_ic0CCIq_0",
+ "c07ae000_2"
+ ],
+ "gold_session_ids": [
+ "answer_d00ba6d0_1",
+ "answer_d00ba6d0_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_f49edff3",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.431733884398313,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "4376945f",
+ "answer_3e9fce53_3",
+ "9f8fa5e4_2",
+ "acfdbdcb",
+ "9f091256_2",
+ "c73cebb3_2",
+ "40a289b7_2",
+ "dc7c53fa_4",
+ "7d33fcb4_1",
+ "answer_3e9fce53_2"
+ ],
+ "gold_session_ids": [
+ "answer_3e9fce53_1",
+ "answer_3e9fce53_2",
+ "answer_3e9fce53_3"
+ ]
+ },
+ {
+ "question_id": "71017276",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_0b4a8adc_1",
+ "4a545031_2",
+ "sharegpt_CjoQMV6_0",
+ "sharegpt_qevDNvo_0",
+ "sharegpt_tCLMKZY_0",
+ "58ec258f_2",
+ "398f23e7",
+ "ultrachat_491808",
+ "10889cf0_1",
+ "a80c8ed4"
+ ],
+ "gold_session_ids": [
+ "answer_0b4a8adc_1"
+ ]
+ },
+ {
+ "question_id": "b46e15ed",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.2856436410275446,
+ "mrr": 0.25,
+ "retrieved_session_ids": [
+ "a410bab2_3",
+ "47311a4c_1",
+ "9d754f54",
+ "answer_4bfcc250_4",
+ "53949711_2",
+ "ultrachat_265690",
+ "940629b2_2",
+ "ultrachat_228593",
+ "answer_4bfcc250_2",
+ "fa34119a_1"
+ ],
+ "gold_session_ids": [
+ "answer_4bfcc250_4",
+ "answer_4bfcc250_3",
+ "answer_4bfcc250_2",
+ "answer_4bfcc250_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_fa19884c",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ff201786_1",
+ "answer_ff201786_2",
+ "cbf48918_1",
+ "f4f4b504_2",
+ "6414f676_2",
+ "e8de3056_2",
+ "8414cc57",
+ "a7b1b27c",
+ "7ea7741b_2",
+ "128082f8_1"
+ ],
+ "gold_session_ids": [
+ "answer_ff201786_1",
+ "answer_ff201786_2"
+ ]
+ },
+ {
+ "question_id": "0bc8ad92",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9325210919548239,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_f4ea84fb_3",
+ "answer_f4ea84fb_1",
+ "ultrachat_349938",
+ "eca613fc",
+ "sharegpt_vClQxDr_21",
+ "answer_f4ea84fb_2",
+ "sharegpt_ntu1Cds_0",
+ "d3cc5bdc_1",
+ "0868982f_1",
+ "bcc37468_2"
+ ],
+ "gold_session_ids": [
+ "answer_f4ea84fb_3",
+ "answer_f4ea84fb_2",
+ "answer_f4ea84fb_1"
+ ]
+ },
+ {
+ "question_id": "af082822",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.3562071871080222,
+ "mrr": 0.16666666666666666,
+ "retrieved_session_ids": [
+ "6cf1848e_3",
+ "bc5827b5",
+ "b28990c8_7",
+ "c285b705_1",
+ "6bb8af95_2",
+ "answer_b51b6115_1",
+ "1dfef591_2",
+ "fbe6fa2c_2",
+ "facef6e0_1",
+ "09c3d921"
+ ],
+ "gold_session_ids": [
+ "answer_b51b6115_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_4929293a",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_add9b012_2",
+ "answer_add9b012_1",
+ "ultrachat_139245",
+ "73353aee_1",
+ "ultrachat_512282",
+ "ultrachat_329640",
+ "c1e598cd_1",
+ "8cfe5096_1",
+ "ultrachat_345984",
+ "sharegpt_oQacgfY_0"
+ ],
+ "gold_session_ids": [
+ "answer_add9b012_2",
+ "answer_add9b012_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_b5700ca9",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_a17423e7_1",
+ "44c62f26",
+ "33bdd89a_1",
+ "65698aa9_7",
+ "ultrachat_383774",
+ "ultrachat_366301",
+ "5053474c_2",
+ "31829c1b_1",
+ "ultrachat_346810",
+ "47382adf_1"
+ ],
+ "gold_session_ids": [
+ "answer_a17423e7_1"
+ ]
+ },
+ {
+ "question_id": "9a707b81",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6509209298071326,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "4abbeb2a_2",
+ "answer_dba89487_2",
+ "5a13d047_1",
+ "answer_dba89487_1",
+ "a8e24e22",
+ "e1d11615_1",
+ "3fcbb58c_4",
+ "ultrachat_233519",
+ "990f3ef9_6",
+ "ultrachat_550996"
+ ],
+ "gold_session_ids": [
+ "answer_dba89487_2",
+ "answer_dba89487_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_1d4ab0c9",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_febde667_2",
+ "answer_febde667_1",
+ "7257bd15_1",
+ "b3e0a3ab_3",
+ "sharegpt_Caxpnd3_12",
+ "4ff83df8",
+ "3f1d1224_3",
+ "5b83c26e_5",
+ "sharegpt_MIYBRJz_27",
+ "ab855b34_2"
+ ],
+ "gold_session_ids": [
+ "answer_febde667_1",
+ "answer_febde667_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_e072b769",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_c19bd2bf_1",
+ "5558a42e_1",
+ "b9e32ff8_1",
+ "0dd54b7c_1",
+ "4d6d013c",
+ "f39f7655_1",
+ "66d3fdb7_2",
+ "4f7b5dc9_1",
+ "78413cce_4",
+ "2fc9ce79_2"
+ ],
+ "gold_session_ids": [
+ "answer_c19bd2bf_1"
+ ]
+ },
+ {
+ "question_id": "0db4c65d",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_b9e32ff8_2",
+ "answer_b9e32ff8_1",
+ "953d5a4b_1",
+ "71dc2037_2",
+ "f78a22b6_3",
+ "e6c3a50a",
+ "sharegpt_5NoXULS_0",
+ "8d1f9505",
+ "ultrachat_324557",
+ "6461402b"
+ ],
+ "gold_session_ids": [
+ "answer_b9e32ff8_1",
+ "answer_b9e32ff8_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_1d80365e",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_661b711f_1",
+ "answer_661b711f_2",
+ "805e571c_1",
+ "sharegpt_yerbusu_1",
+ "0bad887e_1",
+ "1ecc86e7",
+ "554b054e_2",
+ "d6b17438_1",
+ "ef0341d4_2",
+ "sharegpt_5uZ7IdY_0"
+ ],
+ "gold_session_ids": [
+ "answer_661b711f_1",
+ "answer_661b711f_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_7f6b06db",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 0,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0,
+ "mrr": 0.08333333333333333,
+ "retrieved_session_ids": [
+ "64266f80_1",
+ "6f051087_2",
+ "4bf66c38_2",
+ "39c1bf6a",
+ "252715f8_2",
+ "da42526b_1",
+ "70dc7d08_1",
+ "42165950_8",
+ "sharegpt_v11Tpg9_255",
+ "ultrachat_6893"
+ ],
+ "gold_session_ids": [
+ "answer_5d8c99d3_1",
+ "answer_5d8c99d3_2",
+ "answer_5d8c99d3_3"
+ ]
+ },
+ {
+ "question_id": "gpt4_6dc9b45b",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_c4df007f_1",
+ "ultrachat_384387",
+ "9adc81d7_2",
+ "afa0c2e7_5",
+ "sharegpt_jyynvvk_0",
+ "cfcf5340_2",
+ "ultrachat_35147",
+ "9752e18d",
+ "d8f48b0a",
+ "56911dc5_2"
+ ],
+ "gold_session_ids": [
+ "answer_c4df007f_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_8279ba02",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6309297535714575,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "85f19a30_1",
+ "answer_56521e65_1",
+ "86c5d31d",
+ "ultrachat_302826",
+ "2fd445a2_5",
+ "ultrachat_520053",
+ "1a037873_2",
+ "ultrachat_78027",
+ "b909542d_1",
+ "e1c807f9"
+ ],
+ "gold_session_ids": [
+ "answer_56521e65_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_18c2b244",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_c862f65a_2",
+ "answer_c862f65a_3",
+ "answer_c862f65a_1",
+ "dfbc46bd_1",
+ "84896d13",
+ "d9e1d5fd",
+ "ultrachat_503659",
+ "ffd66315_2",
+ "746ae854_2",
+ "7e2c6065"
+ ],
+ "gold_session_ids": [
+ "answer_c862f65a_2",
+ "answer_c862f65a_3",
+ "answer_c862f65a_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_a1b77f9c",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.971476156593264,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_e9ad5914_5",
+ "answer_e9ad5914_3",
+ "answer_e9ad5914_2",
+ "answer_e9ad5914_4",
+ "47ec1674_2",
+ "199a5225_1",
+ "answer_e9ad5914_6",
+ "answer_e9ad5914_1",
+ "d52d478b",
+ "e09b6396"
+ ],
+ "gold_session_ids": [
+ "answer_e9ad5914_1",
+ "answer_e9ad5914_2",
+ "answer_e9ad5914_3",
+ "answer_e9ad5914_4",
+ "answer_e9ad5914_5",
+ "answer_e9ad5914_6"
+ ]
+ },
+ {
+ "question_id": "gpt4_1916e0ea",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_447052a5_2",
+ "answer_447052a5_1",
+ "ultrachat_306167",
+ "7a36e820_2",
+ "280352e9",
+ "ddf41a24",
+ "d3fe25ff",
+ "b9890c15_2",
+ "987b5d00",
+ "fa1ce406_2"
+ ],
+ "gold_session_ids": [
+ "answer_447052a5_2",
+ "answer_447052a5_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_7a0daae1",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_4d5490f1_1",
+ "answer_4d5490f1_2",
+ "014a5d36",
+ "7c08d74b_2",
+ "ultrachat_49899",
+ "bf3aebdb_2",
+ "445e6a7a_2",
+ "2ff297a8_1",
+ "8c68df0d_1",
+ "819dfad7_2"
+ ],
+ "gold_session_ids": [
+ "answer_4d5490f1_1",
+ "answer_4d5490f1_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_468eb063",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.43067655807339306,
+ "mrr": 0.25,
+ "retrieved_session_ids": [
+ "e60a93ff_2",
+ "sharegpt_Sf51OjE_8",
+ "ultrachat_91161",
+ "answer_9b09d95b_1",
+ "ultrachat_108278",
+ "sharegpt_1rIk8tS_0",
+ "sharegpt_gfhUw40_17",
+ "d97db962_2",
+ "23ecf657",
+ "982514b2_1"
+ ],
+ "gold_session_ids": [
+ "answer_9b09d95b_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_7abb270c",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.4296937870297496,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "087d2b0a_2",
+ "answer_7093d898_6",
+ "answer_7093d898_5",
+ "6e42386d_1",
+ "5808284e",
+ "85056d5f_1",
+ "3a7cdf9c",
+ "a211539c",
+ "cb787bfa_1",
+ "answer_7093d898_2"
+ ],
+ "gold_session_ids": [
+ "answer_7093d898_1",
+ "answer_7093d898_2",
+ "answer_7093d898_3",
+ "answer_7093d898_4",
+ "answer_7093d898_5",
+ "answer_7093d898_6"
+ ]
+ },
+ {
+ "question_id": "gpt4_1e4a8aeb",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_16bd5ea5_1",
+ "answer_16bd5ea5_2",
+ "63b51f8a_5",
+ "f7838a91_1",
+ "fd18e0dd_1",
+ "28621d6a_2",
+ "bc5827b5",
+ "6a7d7055_1",
+ "faad7d7a_1",
+ "ultrachat_239720"
+ ],
+ "gold_session_ids": [
+ "answer_16bd5ea5_1",
+ "answer_16bd5ea5_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_4fc4f797",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_be07688f_1",
+ "answer_be07688f_2",
+ "sharegpt_pRqHb1o_0",
+ "10d7d2b4_2",
+ "39566615_1",
+ "sharegpt_V2PULWC_13",
+ "47c5482a",
+ "0c783e4a_1",
+ "463ad596_2",
+ "86298245_2"
+ ],
+ "gold_session_ids": [
+ "answer_be07688f_1",
+ "answer_be07688f_2"
+ ]
+ },
+ {
+ "question_id": "4dfccbf7",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_4bebc782_1",
+ "answer_4bebc782_2",
+ "8fa624b2_6",
+ "e1b7b676",
+ "c2f86a02",
+ "b197b7fb",
+ "sharegpt_Dcshjlv_0",
+ "36f8b380_1",
+ "f2b8e6a8_1",
+ "491ebc5d_2"
+ ],
+ "gold_session_ids": [
+ "answer_4bebc782_1",
+ "answer_4bebc782_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_61e13b3c",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_e831a29f_1",
+ "answer_e831a29f_2",
+ "sharegpt_uD7CBng_4",
+ "cc380119_3",
+ "f3fcf4fd_2",
+ "ultrachat_441289",
+ "a5c89dac",
+ "b4923852",
+ "d4439279_1",
+ "e9f4f027_4"
+ ],
+ "gold_session_ids": [
+ "answer_e831a29f_1",
+ "answer_e831a29f_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_45189cb4",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.2960819109658652,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "dc3ee1d1_1",
+ "answer_e6c20e52_3",
+ "93395e5f_3",
+ "6593cb8b_1",
+ "22452c22",
+ "ultrachat_560291",
+ "sharegpt_AnH3ftB_0",
+ "ultrachat_307270",
+ "e789afdb_1",
+ "0eb4cb14"
+ ],
+ "gold_session_ids": [
+ "answer_e6c20e52_3",
+ "answer_e6c20e52_2",
+ "answer_e6c20e52_1"
+ ]
+ },
+ {
+ "question_id": "2ebe6c90",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6934264036172708,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "1850bfb1",
+ "answer_c9d35c00_1",
+ "answer_c9d35c00_2",
+ "9ccea2f0_2",
+ "sharegpt_jyMHY1J_31",
+ "ultrachat_27073",
+ "90e55108_3",
+ "c3c1f84a_1",
+ "c6bd5277",
+ "a4e89c45"
+ ],
+ "gold_session_ids": [
+ "answer_c9d35c00_1",
+ "answer_c9d35c00_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_e061b84f",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.13565197343244778,
+ "mrr": 0.1,
+ "retrieved_session_ids": [
+ "77974d02",
+ "539c2346_3",
+ "ultrachat_308277",
+ "ultrachat_18218",
+ "4cc9270f",
+ "0a6bf5e4_1",
+ "773287dc_1",
+ "ultrachat_436674",
+ "bab3f409",
+ "answer_8c64ce25_3"
+ ],
+ "gold_session_ids": [
+ "answer_8c64ce25_2",
+ "answer_8c64ce25_1",
+ "answer_8c64ce25_3"
+ ]
+ },
+ {
+ "question_id": "370a8ff4",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_61d1be50_1",
+ "answer_61d1be50_2",
+ "sharegpt_UbMVpdp_93",
+ "cfde748a",
+ "ultrachat_26182",
+ "2b4eaee0_1",
+ "sharegpt_JRQtEcd_115",
+ "9f8fa5e4_5",
+ "df53e1b7_2",
+ "sharegpt_jGPXxpI_17"
+ ],
+ "gold_session_ids": [
+ "answer_61d1be50_1",
+ "answer_61d1be50_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_d6585ce8",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8687949224876582,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_f999b05b_2",
+ "answer_f999b05b_5",
+ "answer_f999b05b_4",
+ "answer_f999b05b_3",
+ "a2b2d331_2",
+ "62f24315_1",
+ "23901329_2",
+ "sharegpt_Asf5ayG_41",
+ "6ec9bed9",
+ "sharegpt_JlSnb71_23"
+ ],
+ "gold_session_ids": [
+ "answer_f999b05b_1",
+ "answer_f999b05b_4",
+ "answer_f999b05b_2",
+ "answer_f999b05b_5",
+ "answer_f999b05b_3"
+ ]
+ },
+ {
+ "question_id": "gpt4_4ef30696",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.5714285040141098,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "8fa64936_2",
+ "answer_f964cea3_2",
+ "d9868305_2",
+ "sharegpt_ELYlA30_25",
+ "80b3f093",
+ "4067bede",
+ "f829b530_2",
+ "3aa868c7_1",
+ "answer_f964cea3_1",
+ "4b29957c_1"
+ ],
+ "gold_session_ids": [
+ "answer_f964cea3_1",
+ "answer_f964cea3_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_ec93e27f",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_e3aa84c4_1",
+ "answer_e3aa84c4_2",
+ "b76006cb_3",
+ "2a1d13f4",
+ "85846900_2",
+ "sharegpt_Sg0ctDe_14",
+ "6b63bdc5_2",
+ "5de5c1ed_1",
+ "f11b88d8_3",
+ "522dd987_1"
+ ],
+ "gold_session_ids": [
+ "answer_e3aa84c4_2",
+ "answer_e3aa84c4_1"
+ ]
+ },
+ {
+ "question_id": "6e984301",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6240505200038379,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "ultrachat_402356",
+ "answer_88841f26_2",
+ "b38766c1_2",
+ "b500d9be",
+ "answer_88841f26_1",
+ "2bc4ae7d",
+ "sharegpt_gVjorGx_0",
+ "c285b705_2",
+ "55c23a23_2",
+ "0daaefa7_1"
+ ],
+ "gold_session_ids": [
+ "answer_88841f26_1",
+ "answer_88841f26_2"
+ ]
+ },
+ {
+ "question_id": "8077ef71",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.5,
+ "mrr": 0.3333333333333333,
+ "retrieved_session_ids": [
+ "f7ffddbc",
+ "ultrachat_468198",
+ "answer_0dd54b7c_1",
+ "5365840b_1",
+ "sharegpt_9EM9xb8_46",
+ "sharegpt_11WxPMZ_0",
+ "1da90ded",
+ "sharegpt_Wi7Op1u_39",
+ "d87a6ef8",
+ "sharegpt_hoQsVGz_51"
+ ],
+ "gold_session_ids": [
+ "answer_0dd54b7c_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_f420262c",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.3347975674340217,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "sharegpt_B0riiep_0",
+ "answer_d8a1af6b_2",
+ "a823dc7c",
+ "ultrachat_58198",
+ "ultrachat_133350",
+ "answer_d8a1af6b_4",
+ "116bae09",
+ "94464f25",
+ "3418b277_5",
+ "3ea07188_2"
+ ],
+ "gold_session_ids": [
+ "answer_d8a1af6b_1",
+ "answer_d8a1af6b_2",
+ "answer_d8a1af6b_3",
+ "answer_d8a1af6b_4",
+ "answer_d8a1af6b_5"
+ ]
+ },
+ {
+ "question_id": "gpt4_8e165409",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_d97db962_2",
+ "answer_d97db962_1",
+ "sharegpt_B7mSlfQ_8",
+ "sharegpt_JRQtEcd_115",
+ "ultrachat_77564",
+ "4c2bd0fe_2",
+ "sharegpt_DrLV0RL_25",
+ "09599d45_1",
+ "aee13015_3",
+ "ultrachat_175639"
+ ],
+ "gold_session_ids": [
+ "answer_d97db962_1",
+ "answer_d97db962_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_74aed68e",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_aed8cf17_2",
+ "answer_aed8cf17_1",
+ "sharegpt_O3FeoHQ_153",
+ "470f29c0_1",
+ "ultrachat_176592",
+ "ultrachat_337472",
+ "50d66391_6",
+ "3f05a03e_1",
+ "34d446d0",
+ "db2f2ed2"
+ ],
+ "gold_session_ids": [
+ "answer_aed8cf17_1",
+ "answer_aed8cf17_2"
+ ]
+ },
+ {
+ "question_id": "bcbe585f",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_43ba3965_1",
+ "a98ff421_4",
+ "7fa8099b_1",
+ "d8a1af6c_5",
+ "5dbe11b9",
+ "8953c8c8_2",
+ "43c487f2_1",
+ "sharegpt_WXCvFir_31",
+ "a158b7e7",
+ "8166673a"
+ ],
+ "gold_session_ids": [
+ "answer_43ba3965_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_21adecb5",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.23719771276929622,
+ "mrr": 0.2,
+ "retrieved_session_ids": [
+ "128082f8_2",
+ "e1f37e24",
+ "sharegpt_ipLglky_42",
+ "0f0f9d80_1",
+ "answer_1e2369c9_1",
+ "b99bd2df",
+ "928890e4_1",
+ "ultrachat_444790",
+ "2880eb6c_1",
+ "1206a46c"
+ ],
+ "gold_session_ids": [
+ "answer_1e2369c9_1",
+ "answer_1e2369c9_2"
+ ]
+ },
+ {
+ "question_id": "5e1b23de",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.5,
+ "mrr": 0.3333333333333333,
+ "retrieved_session_ids": [
+ "1380576d_1",
+ "1a5344e9_3",
+ "answer_c18d480b_1",
+ "c3f945f3_1",
+ "0ba46bdc_2",
+ "a13abca5_2",
+ "ultrachat_153798",
+ "c28aed33",
+ "8278312e_1",
+ "sharegpt_z2MpvCJ_0"
+ ],
+ "gold_session_ids": [
+ "answer_c18d480b_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_98f46fc6",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_5850de18_1",
+ "answer_5850de18_2",
+ "ultrachat_45511",
+ "8e3b479f_1",
+ "28897044",
+ "c285b705_2",
+ "sharegpt_rnL5VQO_0",
+ "sharegpt_PLN69pg_19",
+ "29e1426c_2",
+ "6172cc72_3"
+ ],
+ "gold_session_ids": [
+ "answer_5850de18_2",
+ "answer_5850de18_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_af6db32f",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.5,
+ "mrr": 0.3333333333333333,
+ "retrieved_session_ids": [
+ "aee13015_6",
+ "sharegpt_hDjMGr7_70",
+ "answer_184c8f56_1",
+ "e0b8c937_1",
+ "sharegpt_PJmlduQ_0",
+ "sharegpt_Np41szj_13",
+ "b283f9eb_2",
+ "ec1b557a_1",
+ "43b449ce",
+ "f26846e1_3"
+ ],
+ "gold_session_ids": [
+ "answer_184c8f56_1"
+ ]
+ },
+ {
+ "question_id": "eac54adc",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_0d4d0347_2",
+ "answer_0d4d0347_1",
+ "bf234051_2",
+ "b40f0a7a",
+ "ae051325_3",
+ "sharegpt_0ktU0jP_16",
+ "9ee69ca6_1",
+ "32fc393b_2",
+ "490bb46d_3",
+ "sharegpt_KzDwaf1_253"
+ ],
+ "gold_session_ids": [
+ "answer_0d4d0347_1",
+ "answer_0d4d0347_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_7ddcf75f",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ad8a76cb_1",
+ "sharegpt_CWiYdQ8_152",
+ "a0b8b0ad_7",
+ "1781acb1",
+ "ultrachat_172010",
+ "9bfccfdc_2",
+ "9f8c432e_3",
+ "ultrachat_74113",
+ "8c652eb0_2",
+ "7ba84ed3_2"
+ ],
+ "gold_session_ids": [
+ "answer_ad8a76cb_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_a2d1d1f6",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_f6d6e33f_1",
+ "sharegpt_gI1U1eK_4",
+ "2c31d8be_3",
+ "894dd0c2_3",
+ "f5cf8815_4",
+ "a1bfb382_1",
+ "3b8076bf_1",
+ "bbe42a92",
+ "a21f3697_3",
+ "ultrachat_224894"
+ ],
+ "gold_session_ids": [
+ "answer_f6d6e33f_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_85da3956",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_581ab834_1",
+ "ultrachat_373754",
+ "14b000d9_1",
+ "97f8a545_2",
+ "fcc7fe77_2",
+ "8da8c7e0",
+ "sharegpt_h1UXagU_1",
+ "sharegpt_ewad5Sr_0",
+ "2fd445a2_4",
+ "8d952ece_2"
+ ],
+ "gold_session_ids": [
+ "answer_581ab834_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_b0863698",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.43067655807339306,
+ "mrr": 0.25,
+ "retrieved_session_ids": [
+ "b2fa108b_1",
+ "91e581ad_3",
+ "ce8b2a9e_2",
+ "answer_550bb2d1_1",
+ "5b014f9b_1",
+ "4a79d078_2",
+ "3b7c6429",
+ "sharegpt_FPbCOuq_30",
+ "sharegpt_T377N30_0",
+ "005ec7cf_1"
+ ],
+ "gold_session_ids": [
+ "answer_550bb2d1_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_68e94287",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.19342640361727081,
+ "mrr": 0.125,
+ "retrieved_session_ids": [
+ "43502c3c_2",
+ "52c6d59a_2",
+ "50f136f2_3",
+ "edfa358d_1",
+ "64460e9d_1",
+ "ultrachat_297774",
+ "50ca8a86",
+ "answer_9793daa3_2",
+ "sharegpt_35udPn0_1",
+ "sharegpt_Wm2ElWB_0"
+ ],
+ "gold_session_ids": [
+ "answer_9793daa3_2",
+ "answer_9793daa3_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_e414231e",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_e28c1f0d_1",
+ "answer_e28c1f0d_2",
+ "d3065d85",
+ "1bab001f",
+ "ultrachat_485311",
+ "7e3fa056_2",
+ "c87e6c56",
+ "c12d5181_3",
+ "642d1256",
+ "a459d477"
+ ],
+ "gold_session_ids": [
+ "answer_e28c1f0d_1",
+ "answer_e28c1f0d_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_7ca326fa",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_cf021b36_1",
+ "answer_cf021b36_3",
+ "answer_cf021b36_2",
+ "8d0ca228",
+ "sharegpt_7SCdbX2_0",
+ "16d79242_1",
+ "604de0a3_2",
+ "1301fbd1",
+ "sharegpt_CYVtC0w_0",
+ "b4025421"
+ ],
+ "gold_session_ids": [
+ "answer_cf021b36_1",
+ "answer_cf021b36_2",
+ "answer_cf021b36_3"
+ ]
+ },
+ {
+ "question_id": "gpt4_7bc6cf22",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_e22d6aef_1",
+ "09d021e7_1",
+ "3e43bf49_1",
+ "e6cc6157_1",
+ "225f3ed6_2",
+ "sharegpt_yzmMl34_0",
+ "eca3b71c_2",
+ "39566615_1",
+ "ultrachat_215356",
+ "9f793d6f_3"
+ ],
+ "gold_session_ids": [
+ "answer_e22d6aef_1"
+ ]
+ },
+ {
+ "question_id": "2ebe6c92",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.21840743681816419,
+ "mrr": 0.16666666666666666,
+ "retrieved_session_ids": [
+ "1ba79ea8_1",
+ "fb303dd2_2",
+ "16756728_1",
+ "9d4312f6_3",
+ "ca3a4e4f_1",
+ "answer_c9d35c00_2",
+ "d9e023f2_2",
+ "719a18dd_2",
+ "84048eba_5",
+ "b30a47be_2"
+ ],
+ "gold_session_ids": [
+ "answer_c9d35c00_1",
+ "answer_c9d35c00_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_e061b84g",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.18154179253735267,
+ "mrr": 0.2,
+ "retrieved_session_ids": [
+ "9dac9e37_1",
+ "de43030f_1",
+ "68705467_1",
+ "ultrachat_547960",
+ "answer_8c64ce26_3",
+ "ultrachat_213300",
+ "ae15e8b6_1",
+ "sharegpt_rB6IP4G_0",
+ "3929e6cc_1",
+ "ultrachat_391029"
+ ],
+ "gold_session_ids": [
+ "answer_8c64ce26_2",
+ "answer_8c64ce26_1",
+ "answer_8c64ce26_3"
+ ]
+ },
+ {
+ "question_id": "71017277",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_0b4a8adc_1",
+ "976ca0d9",
+ "85a1be56_2",
+ "f8de4e92_4",
+ "5854eebc_2",
+ "e64f9553_1",
+ "e5b86922_1",
+ "0aacd15f_5",
+ "dd99d002",
+ "40a1e681_1"
+ ],
+ "gold_session_ids": [
+ "answer_0b4a8adc_1"
+ ]
+ },
+ {
+ "question_id": "b46e15ee",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.4517623092632536,
+ "mrr": 0.3333333333333333,
+ "retrieved_session_ids": [
+ "sharegpt_1ZakfXH_14",
+ "495cf518_2",
+ "answer_4bfcc251_4",
+ "918c4788_1",
+ "f3fcf4fd_1",
+ "answer_4bfcc251_1",
+ "ultrachat_322794",
+ "c92239bc",
+ "answer_4bfcc251_2",
+ "d4446e25_2"
+ ],
+ "gold_session_ids": [
+ "answer_4bfcc251_4",
+ "answer_4bfcc251_3",
+ "answer_4bfcc251_2",
+ "answer_4bfcc251_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_d6585ce9",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.4973199614801841,
+ "mrr": 0.25,
+ "retrieved_session_ids": [
+ "58dd3f35",
+ "bb107057_4",
+ "a554ed79_4",
+ "answer_f999b05c_5",
+ "answer_f999b05c_2",
+ "17a561cb_2",
+ "answer_f999b05c_1",
+ "answer_f999b05c_4",
+ "a15fa6eb",
+ "b3763b6b_1"
+ ],
+ "gold_session_ids": [
+ "answer_f999b05c_1",
+ "answer_f999b05c_4",
+ "answer_f999b05c_2",
+ "answer_f999b05c_5",
+ "answer_f999b05c_3"
+ ]
+ },
+ {
+ "question_id": "gpt4_1e4a8aec",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 0,
+ "recall_any_at_20": 0,
+ "ndcg_at_10": 0,
+ "mrr": 0,
+ "retrieved_session_ids": [
+ "b1265fd7_1",
+ "9f1f3bf4",
+ "b34a52aa_3",
+ "e675c69a_1",
+ "1c662b7b_1",
+ "a8086137",
+ "sharegpt_9JZCfee_0",
+ "0b635369_2",
+ "49576fd6",
+ "e41b78c7_2"
+ ],
+ "gold_session_ids": [
+ "answer_16bd5ea6_1",
+ "answer_16bd5ea6_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_f420262d",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8550961823035497,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_d8a1af6c_2",
+ "27639dd8_3",
+ "answer_d8a1af6c_4",
+ "0dcfb511_1",
+ "answer_d8a1af6c_5",
+ "1032cbcf_1",
+ "answer_d8a1af6c_1",
+ "3c0ed3bf_4",
+ "answer_d8a1af6c_3",
+ "e1023f28"
+ ],
+ "gold_session_ids": [
+ "answer_d8a1af6c_1",
+ "answer_d8a1af6c_2",
+ "answer_d8a1af6c_5",
+ "answer_d8a1af6c_4",
+ "answer_d8a1af6c_3"
+ ]
+ },
+ {
+ "question_id": "gpt4_59149c78",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.5249810332008933,
+ "mrr": 0.3333333333333333,
+ "retrieved_session_ids": [
+ "415657e0",
+ "sharegpt_GI6737T_2",
+ "answer_d00ba6d1_2",
+ "afec1061_2",
+ "9848a1e3_2",
+ "answer_d00ba6d1_1",
+ "ultrachat_463998",
+ "23754665",
+ "765ce8a7_2",
+ "2c282c7c"
+ ],
+ "gold_session_ids": [
+ "answer_d00ba6d1_1",
+ "answer_d00ba6d1_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_e414231f",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8315546295836225,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_e28c1f0e_1",
+ "ultrachat_136119",
+ "ultrachat_150948",
+ "0bd59f15_2",
+ "deb5bb60",
+ "answer_e28c1f0e_2",
+ "752c438c_3",
+ "sharegpt_K8mXaIT_0",
+ "07a26dfa_5",
+ "135f761b_1"
+ ],
+ "gold_session_ids": [
+ "answer_e28c1f0e_1",
+ "answer_e28c1f0e_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_4929293b",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 0,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0,
+ "mrr": 0.0625,
+ "retrieved_session_ids": [
+ "b4ed7223_3",
+ "ultrachat_511609",
+ "e76430ea_3",
+ "7fa98614_1",
+ "bda611f6_3",
+ "c0dbabb8",
+ "ultrachat_156176",
+ "11d959b3_1",
+ "628c958c_1",
+ "ea8c38f8"
+ ],
+ "gold_session_ids": [
+ "answer_add9b013_2",
+ "answer_add9b013_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_468eb064",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 0,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0,
+ "mrr": 0.05555555555555555,
+ "retrieved_session_ids": [
+ "ultrachat_337251",
+ "1d74a734",
+ "db1aefb6_3",
+ "5e2b7e01_1",
+ "sharegpt_wrwD6bG_0",
+ "a2f0054f",
+ "ultrachat_59629",
+ "21ab8a2c_1",
+ "c3023a45_1",
+ "25e5b797_1"
+ ],
+ "gold_session_ids": [
+ "answer_9b09d95b_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_fa19884d",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6934264036172708,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "588a2123",
+ "answer_ff201787_2",
+ "answer_ff201787_1",
+ "f910dc31_4",
+ "b35fadaa",
+ "ccc87da9_1",
+ "040615bc_1",
+ "f7a61595_3",
+ "55da3cbf_2",
+ "ultrachat_424557"
+ ],
+ "gold_session_ids": [
+ "answer_ff201787_1",
+ "answer_ff201787_2"
+ ]
+ },
+ {
+ "question_id": "9a707b82",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 0,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0,
+ "mrr": 0.07142857142857142,
+ "retrieved_session_ids": [
+ "d33dbfd5_2",
+ "fab41c07",
+ "e8df5350",
+ "990f3ef9_2",
+ "8e78fa70_1",
+ "7966888b_2",
+ "f8e9d4fa",
+ "ed5c4c33_1",
+ "09e97c6f_2",
+ "778bedd4"
+ ],
+ "gold_session_ids": [
+ "answer_dba89488_2",
+ "answer_dba89488_1"
+ ]
+ },
+ {
+ "question_id": "eac54add",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 0,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0,
+ "mrr": 0.06666666666666667,
+ "retrieved_session_ids": [
+ "2ea6fda4_1",
+ "7966888b_2",
+ "eb336de0_3",
+ "7bb01253",
+ "773aebbd_3",
+ "2946bf12",
+ "072fac63_1",
+ "4d578deb",
+ "bdb49a42_2",
+ "1fb3bbed_1"
+ ],
+ "gold_session_ids": [
+ "answer_0d4d0348_1",
+ "answer_0d4d0348_2"
+ ]
+ },
+ {
+ "question_id": "4dfccbf8",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.21840743681816419,
+ "mrr": 0.16666666666666666,
+ "retrieved_session_ids": [
+ "sharegpt_uD7CBng_2",
+ "sharegpt_3D3oQC0_213",
+ "953aaad5_3",
+ "0ae4c9b0_2",
+ "7db28e68_1",
+ "answer_4bebc783_1",
+ "b4f63a70_3",
+ "edd89480_1",
+ "a63ad8e3_3",
+ "6c16c3ec_2"
+ ],
+ "gold_session_ids": [
+ "answer_4bebc783_1",
+ "answer_4bebc783_2"
+ ]
+ },
+ {
+ "question_id": "0bc8ad93",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_f4ea84fc_1",
+ "answer_f4ea84fc_3",
+ "answer_f4ea84fc_2",
+ "sharegpt_HOO59Ue_99",
+ "2c185a2b_1",
+ "ultrachat_260376",
+ "sharegpt_afCogMg_34",
+ "ef23ecbc_2",
+ "85736a07_1",
+ "97338ddd_1"
+ ],
+ "gold_session_ids": [
+ "answer_f4ea84fc_3",
+ "answer_f4ea84fc_2",
+ "answer_f4ea84fc_1"
+ ]
+ },
+ {
+ "question_id": "6e984302",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.430624116386567,
+ "mrr": 0.2,
+ "retrieved_session_ids": [
+ "c03fc56c_1",
+ "41dc5d45_1",
+ "7285299a_8",
+ "2ed9532f",
+ "answer_88841f27_1",
+ "cf3dc165_1",
+ "sharegpt_4Wjx3pH_0",
+ "answer_88841f27_2",
+ "b7702ee4_1",
+ "7cd7c296_1"
+ ],
+ "gold_session_ids": [
+ "answer_88841f27_1",
+ "answer_88841f27_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_8279ba03",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 0,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0,
+ "mrr": 0.07142857142857142,
+ "retrieved_session_ids": [
+ "864a563d_5",
+ "b357fb8b_2",
+ "606d0c4a_2",
+ "084bb880_2",
+ "518d26d3_4",
+ "d1a1b9ea_1",
+ "652c0717_3",
+ "dea400f8_2",
+ "50d66391_4",
+ "2ef55f49_3"
+ ],
+ "gold_session_ids": [
+ "answer_56521e66_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_b5700ca0",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_a17423e8_1",
+ "109e07e8_2",
+ "ultrachat_540666",
+ "b403d6f0_4",
+ "e789afdb_2",
+ "5829ab38",
+ "fd6f60f0_1",
+ "064f42ee_3",
+ "ultrachat_418858",
+ "sharegpt_FpTfRvR_0"
+ ],
+ "gold_session_ids": [
+ "answer_a17423e8_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_68e94288",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.23719771276929622,
+ "mrr": 0.2,
+ "retrieved_session_ids": [
+ "4c49e37f",
+ "6551be60_1",
+ "767b1b89",
+ "d2606fe9",
+ "answer_9793daa4_1",
+ "d8454317_7",
+ "0a7c4f1c_1",
+ "c99dcd81",
+ "128f4e4d_2",
+ "ultrachat_74747"
+ ],
+ "gold_session_ids": [
+ "answer_9793daa4_2",
+ "answer_9793daa4_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_2655b836",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_4be1b6b4_3",
+ "answer_4be1b6b4_2",
+ "answer_4be1b6b4_1",
+ "sharegpt_KzDwaf1_237",
+ "d8a1af6b_4",
+ "b876911c",
+ "be32ad25_2",
+ "d82cefd3",
+ "3bc55308",
+ "e6120ff9_2"
+ ],
+ "gold_session_ids": [
+ "answer_4be1b6b4_1",
+ "answer_4be1b6b4_2",
+ "answer_4be1b6b4_3"
+ ]
+ },
+ {
+ "question_id": "gpt4_2487a7cb",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9197207891481876,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_1c6b85ea_2",
+ "4f5880c6_6",
+ "answer_1c6b85ea_1",
+ "sharegpt_QN26oUg_0",
+ "f87fea58_4",
+ "sharegpt_CHaU4ax_10",
+ "67a8ef5f",
+ "sharegpt_gLb26GY_0",
+ "ultrachat_295777",
+ "b10c041b_2"
+ ],
+ "gold_session_ids": [
+ "answer_1c6b85ea_2",
+ "answer_1c6b85ea_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_76048e76",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9197207891481876,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_b535969f_1",
+ "409dfa26_1",
+ "answer_b535969f_2",
+ "33630c9e_2",
+ "8fcaf3a9_2",
+ "sharegpt_vx14orQ_11",
+ "sharegpt_inuIr9J_119",
+ "e2d9bcfe",
+ "6a78e959_2",
+ "ultrachat_185750"
+ ],
+ "gold_session_ids": [
+ "answer_b535969f_1",
+ "answer_b535969f_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_2312f94c",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_5328c3c2_2",
+ "answer_5328c3c2_1",
+ "e831a29f_2",
+ "sharegpt_XYtuANl_9",
+ "ultrachat_327449",
+ "ultrachat_500682",
+ "f3745025_2",
+ "a764b677_2",
+ "ultrachat_319398",
+ "b9f59a03_2"
+ ],
+ "gold_session_ids": [
+ "answer_5328c3c2_2",
+ "answer_5328c3c2_1"
+ ]
+ },
+ {
+ "question_id": "0bb5a684",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8065735963827293,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_e936197f_1",
+ "3c8e7c0e_3",
+ "1a14d6cb_1",
+ "ultrachat_62919",
+ "826d51da_3",
+ "565929a2_2",
+ "b2d46bf2",
+ "answer_e936197f_2",
+ "fce8bb8a_2",
+ "79a20d73"
+ ],
+ "gold_session_ids": [
+ "answer_e936197f_1",
+ "answer_e936197f_2"
+ ]
+ },
+ {
+ "question_id": "08f4fc43",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_6ea1541e_1",
+ "answer_6ea1541e_2",
+ "ultrachat_359609",
+ "9216c4d3_4",
+ "762a4ba1_3",
+ "sharegpt_JJu53iW_0",
+ "e295eb5f",
+ "ultrachat_157004",
+ "f3bd00ec_2",
+ "cdbd6862_1"
+ ],
+ "gold_session_ids": [
+ "answer_6ea1541e_2",
+ "answer_6ea1541e_1"
+ ]
+ },
+ {
+ "question_id": "2c63a862",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.20438239758848611,
+ "mrr": 0.14285714285714285,
+ "retrieved_session_ids": [
+ "93a5cfe6",
+ "33baaaab_1",
+ "ad175dc4_2",
+ "1854abc7_2",
+ "64eea322_1",
+ "0fccc817_1",
+ "answer_d39b7977_2",
+ "83c13ff9",
+ "acbc1954",
+ "9c549466_2"
+ ],
+ "gold_session_ids": [
+ "answer_d39b7977_1",
+ "answer_d39b7977_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_385a5000",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_7a4a93f1_2",
+ "answer_7a4a93f1_1",
+ "8062f4ab_2",
+ "7a4e1601",
+ "28f74b09_1",
+ "6380e5d6",
+ "cae65795_1",
+ "29d27eaa_2",
+ "dc832f73",
+ "sharegpt_Dcoc4IC_17"
+ ],
+ "gold_session_ids": [
+ "answer_7a4a93f1_2",
+ "answer_7a4a93f1_1"
+ ]
+ },
+ {
+ "question_id": "2a1811e2",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_1cc3cd0c_2",
+ "answer_1cc3cd0c_1",
+ "f5a4d792_1",
+ "901a6763_2",
+ "1aaf8057_3",
+ "ultrachat_187883",
+ "e9fb10e7_1",
+ "4019b1a8_4",
+ "2ac6f050_1",
+ "89e05490"
+ ],
+ "gold_session_ids": [
+ "answer_1cc3cd0c_1",
+ "answer_1cc3cd0c_2"
+ ]
+ },
+ {
+ "question_id": "bbf86515",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.5249810332008933,
+ "mrr": 0.3333333333333333,
+ "retrieved_session_ids": [
+ "592cb8d2",
+ "75b98d2a_1",
+ "answer_b3763b6b_1",
+ "05260a4b_2",
+ "sharegpt_GrLzv0V_432",
+ "answer_b3763b6b_2",
+ "ba85548f",
+ "df003c93_2",
+ "ultrachat_320650",
+ "29194fbe_1"
+ ],
+ "gold_session_ids": [
+ "answer_b3763b6b_1",
+ "answer_b3763b6b_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_5dcc0aab",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9277589908435726,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_099c1b6c_3",
+ "answer_099c1b6c_1",
+ "answer_099c1b6c_5",
+ "353d3c6d_1",
+ "3891baea_3",
+ "ultrachat_156812",
+ "3f787a78_6",
+ "answer_099c1b6c_2",
+ "sharegpt_5Rpp72b_87",
+ "answer_099c1b6c_4"
+ ],
+ "gold_session_ids": [
+ "answer_099c1b6c_5",
+ "answer_099c1b6c_2",
+ "answer_099c1b6c_4",
+ "answer_099c1b6c_3",
+ "answer_099c1b6c_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_0b2f1d21",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_c4e5d969_2",
+ "answer_c4e5d969_1",
+ "1cafd864_3",
+ "f31a0422_1",
+ "sharegpt_hDjMGr7_30",
+ "sharegpt_ME35OD8_167",
+ "sharegpt_uD7CBng_36",
+ "0418a4b1_1",
+ "0df69ec5",
+ "sharegpt_KBe10PZ_0"
+ ],
+ "gold_session_ids": [
+ "answer_c4e5d969_2",
+ "answer_c4e5d969_1"
+ ]
+ },
+ {
+ "question_id": "f0853d11",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_932ea3bf_1",
+ "answer_932ea3bf_2",
+ "b16a8656_2",
+ "10c0752c",
+ "sharegpt_Dcshjlv_33",
+ "sharegpt_9upOSlE_27",
+ "bd8708e2_2",
+ "6e436d54",
+ "sharegpt_hBX7Nqo_25",
+ "8c68df0d_2"
+ ],
+ "gold_session_ids": [
+ "answer_932ea3bf_2",
+ "answer_932ea3bf_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_6ed717ea",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_d50a8a33_2",
+ "answer_d50a8a33_1",
+ "15ed03f0_1",
+ "637bc32a_2",
+ "ultrachat_33224",
+ "d4b6ec0b",
+ "ultrachat_400856",
+ "e0585cb5_1",
+ "sharegpt_HQMYIkt_0",
+ "50940cb7_1"
+ ],
+ "gold_session_ids": [
+ "answer_d50a8a33_1",
+ "answer_d50a8a33_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_70e84552",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9197207891481876,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_b3070ec4_2",
+ "01f61248_1",
+ "answer_b3070ec4_1",
+ "2f8a3459_2",
+ "ultrachat_573894",
+ "30491f6f_1",
+ "67a8ef5f",
+ "298a07de_5",
+ "sharegpt_d7bP5fb_0",
+ "2b9b6224_2"
+ ],
+ "gold_session_ids": [
+ "answer_b3070ec4_2",
+ "answer_b3070ec4_1"
+ ]
+ },
+ {
+ "question_id": "a3838d2b",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.3482538302146053,
+ "mrr": 0.25,
+ "retrieved_session_ids": [
+ "sharegpt_GDFyxeT_0",
+ "sharegpt_JXREDs0_4",
+ "ultrachat_462884",
+ "answer_4ffa04a2_1",
+ "answer_4ffa04a2_6",
+ "8cf0d2a9",
+ "answer_4ffa04a2_4",
+ "ultrachat_95340",
+ "bd4110bc",
+ "ae73b0f7_2"
+ ],
+ "gold_session_ids": [
+ "answer_4ffa04a2_1",
+ "answer_4ffa04a2_6",
+ "answer_4ffa04a2_4",
+ "answer_4ffa04a2_3",
+ "answer_4ffa04a2_5",
+ "answer_4ffa04a2_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_93159ced",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8503449055347546,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_e5131a1b_2",
+ "263efe64_1",
+ "17d91635_1",
+ "ultrachat_387032",
+ "answer_e5131a1b_1",
+ "460be54a",
+ "c81d1a7b_2",
+ "e9ad5914_6",
+ "09b815e7_2",
+ "ultrachat_324774"
+ ],
+ "gold_session_ids": [
+ "answer_e5131a1b_2",
+ "answer_e5131a1b_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_2d58bcd6",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9197207891481876,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_3e11e0ae_2",
+ "0bd0a01c",
+ "answer_3e11e0ae_1",
+ "1bf8b212",
+ "sharegpt_XUxvLWu_0",
+ "be32ad25_1",
+ "sharegpt_DvZqBYf_93",
+ "343cbb3b",
+ "5d4bfa9d",
+ "c48a1e21_2"
+ ],
+ "gold_session_ids": [
+ "answer_3e11e0ae_1",
+ "answer_3e11e0ae_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_65aabe59",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6934264036172708,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "6a7e3aec",
+ "answer_30dfe889_1",
+ "answer_30dfe889_2",
+ "77b8fcd9_1",
+ "78bcce6a",
+ "ultrachat_190669",
+ "52fdad30_4",
+ "sharegpt_v2b1TzW_0",
+ "05a37787",
+ "9b1a5fb0"
+ ],
+ "gold_session_ids": [
+ "answer_30dfe889_1",
+ "answer_30dfe889_2"
+ ]
+ },
+ {
+ "question_id": "982b5123",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9197207891481876,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ab603dd5_1",
+ "04e05522_2",
+ "answer_ab603dd5_2",
+ "9a955705",
+ "6ff77d45_2",
+ "359a2a0d",
+ "bf8ab267",
+ "07ba9acd_2",
+ "1bfd5a8b_3",
+ "ultrachat_35755"
+ ],
+ "gold_session_ids": [
+ "answer_ab603dd5_1",
+ "answer_ab603dd5_2"
+ ]
+ },
+ {
+ "question_id": "b9cfe692",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_5e3bb940_1",
+ "answer_5e3bb940_2",
+ "answer_5e3bb940_3",
+ "91d847c1_6",
+ "da650f3b",
+ "sharegpt_oXgiN7q_53",
+ "0c27c633_2",
+ "ultrachat_265626",
+ "sharegpt_JRQtEcd_115",
+ "2c7bff3a_1"
+ ],
+ "gold_session_ids": [
+ "answer_5e3bb940_3",
+ "answer_5e3bb940_2",
+ "answer_5e3bb940_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_4edbafa2",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.48247555939075504,
+ "mrr": 0.25,
+ "retrieved_session_ids": [
+ "7533d546_5",
+ "3f5249ee_3",
+ "22be78fc_2",
+ "answer_0a00c163_2",
+ "6c68d9b4_1",
+ "answer_0a00c163_1",
+ "a65d076a",
+ "4a006b77",
+ "725f95e1",
+ "18c0af9c"
+ ],
+ "gold_session_ids": [
+ "answer_0a00c163_1",
+ "answer_0a00c163_2"
+ ]
+ },
+ {
+ "question_id": "c8090214",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.44130740935663865,
+ "mrr": 0.25,
+ "retrieved_session_ids": [
+ "ultrachat_518687",
+ "005ec7cf_1",
+ "162ad3bf_1",
+ "answer_70dc7d08_2",
+ "c21c9d97",
+ "sharegpt_xOzzaof_0",
+ "e2d9bcfe",
+ "ultrachat_381881",
+ "ultrachat_415426",
+ "answer_70dc7d08_1"
+ ],
+ "gold_session_ids": [
+ "answer_70dc7d08_2",
+ "answer_70dc7d08_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_483dd43c",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_fb793c87_1",
+ "answer_fb793c87_2",
+ "sharegpt_I6IzvEF_0",
+ "29695e1c_3",
+ "sharegpt_ItFPuha_0",
+ "4243ce01_1",
+ "52161c9e_1",
+ "19c24c11",
+ "sharegpt_E0YL5SX_145",
+ "73353aee_3"
+ ],
+ "gold_session_ids": [
+ "answer_fb793c87_1",
+ "answer_fb793c87_2"
+ ]
+ },
+ {
+ "question_id": "e4e14d04",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_cf425855_2",
+ "answer_cf425855_1",
+ "sharegpt_HO1gGmm_0",
+ "ultrachat_542420",
+ "ffc627f2",
+ "sharegpt_zX5iazJ_19",
+ "1c1f5ccc_3",
+ "5944b36a_1",
+ "f8de4e92_2",
+ "30f7f504"
+ ],
+ "gold_session_ids": [
+ "answer_cf425855_1",
+ "answer_cf425855_2"
+ ]
+ },
+ {
+ "question_id": "c9f37c46",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_cdba3d9f_1",
+ "answer_cdba3d9f_2",
+ "a8e24e22",
+ "0cf86cbe_1",
+ "ultrachat_418541",
+ "02df1225_1",
+ "b38766c1_3",
+ "211bb1c2_1",
+ "4ffa04a2_1",
+ "c3d6a282_1"
+ ],
+ "gold_session_ids": [
+ "answer_cdba3d9f_1",
+ "answer_cdba3d9f_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_2c50253f",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6509209298071326,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "sharegpt_4Su6bUM_11",
+ "answer_9af4e346_2",
+ "a8fae4b8_2",
+ "answer_9af4e346_1",
+ "57c8bf62_1",
+ "sharegpt_o0jtbUO_8",
+ "4b9ed528_2",
+ "4cef8a3c_3",
+ "462ce3bc_2",
+ "ultrachat_340289"
+ ],
+ "gold_session_ids": [
+ "answer_9af4e346_1",
+ "answer_9af4e346_2"
+ ]
+ },
+ {
+ "question_id": "dcfa8644",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_5e3eeb12_1",
+ "answer_5e3eeb12_2",
+ "b2243528_1",
+ "58dd3f35",
+ "0d129c99",
+ "7de51ffe_1",
+ "41f94af6",
+ "ultrachat_431004",
+ "sharegpt_oG3LXXo_6",
+ "595902ff"
+ ],
+ "gold_session_ids": [
+ "answer_5e3eeb12_2",
+ "answer_5e3eeb12_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_b4a80587",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_b9d9150e_2",
+ "answer_b9d9150e_1",
+ "sharegpt_inuIr9J_119",
+ "f79a48dd_1",
+ "3cb41ab8_1",
+ "ultrachat_91937",
+ "866de400",
+ "017fa8a9",
+ "b9c4fa1b",
+ "4ecbe1f0_1"
+ ],
+ "gold_session_ids": [
+ "answer_b9d9150e_2",
+ "answer_b9d9150e_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_9a159967",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6871475159848289,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "b2cae160_1",
+ "answer_8a42fedf_1",
+ "answer_8a42fedf_2",
+ "4ea94f85",
+ "253742b4_3",
+ "ultrachat_484212",
+ "answer_8a42fedf_3",
+ "0df54428_4",
+ "0ad608b3",
+ "bff09560_2"
+ ],
+ "gold_session_ids": [
+ "answer_8a42fedf_1",
+ "answer_8a42fedf_3",
+ "answer_8a42fedf_2"
+ ]
+ },
+ {
+ "question_id": "cc6d1ec1",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_be73098b_2",
+ "answer_be73098b_1",
+ "33baaaab_3",
+ "ultrachat_282969",
+ "ultrachat_95254",
+ "ultrachat_508911",
+ "ultrachat_436945",
+ "sharegpt_uD7CBng_20",
+ "sharegpt_j4070wP_0",
+ "3cb41ab8_1"
+ ],
+ "gold_session_ids": [
+ "answer_be73098b_2",
+ "answer_be73098b_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_8c8961ae",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_72d9aa58_1",
+ "answer_72d9aa58_2",
+ "3b22d17b",
+ "1cde7ee4_2",
+ "b77d0b58",
+ "05fad1cb_2",
+ "43a2c8af_1",
+ "37f0ce6b_1",
+ "02f9bedb_2",
+ "sharegpt_Ms1yTte_64"
+ ],
+ "gold_session_ids": [
+ "answer_72d9aa58_1",
+ "answer_72d9aa58_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_d9af6064",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_da704e79_1",
+ "answer_da704e79_2",
+ "03b4d5d8_2",
+ "ff49b5a5_2",
+ "bcebee27_3",
+ "sharegpt_VbIxRWJ_33",
+ "sharegpt_WjW4qki_0",
+ "sharegpt_w9YUs4c_0",
+ "ultrachat_182531",
+ "98c83fe4"
+ ],
+ "gold_session_ids": [
+ "answer_da704e79_2",
+ "answer_da704e79_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_7de946e7",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6934264036172708,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "caa00337_2",
+ "answer_6a78e959_1",
+ "answer_6a78e959_2",
+ "c4d370d3_1",
+ "58bec5fb_3",
+ "6155addd_2",
+ "ultrachat_220429",
+ "sharegpt_6Yfj9A3_5",
+ "3023b36f",
+ "ultrachat_105249"
+ ],
+ "gold_session_ids": [
+ "answer_6a78e959_2",
+ "answer_6a78e959_1"
+ ]
+ },
+ {
+ "question_id": "d01c6aa8",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.17723928678404774,
+ "mrr": 0.1,
+ "retrieved_session_ids": [
+ "f2fa97f5_1",
+ "fc9a62c0_1",
+ "51fe163e_2",
+ "6ebd9e18_1",
+ "5024758e",
+ "fbd67491_2",
+ "64460e9d_1",
+ "sharegpt_CWiYdQ8_13",
+ "debc34d7_2",
+ "answer_991d55e5_1"
+ ],
+ "gold_session_ids": [
+ "answer_991d55e5_1",
+ "answer_991d55e5_2"
+ ]
+ },
+ {
+ "question_id": "993da5e2",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8772153153380493,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_54f0d6f9_2",
+ "b16a8656_2",
+ "12cd736c",
+ "answer_54f0d6f9_1",
+ "c756d93e",
+ "a6c0f032_3",
+ "8953c8c8_3",
+ "13f3da23_2",
+ "c69a28f0",
+ "ultrachat_217815"
+ ],
+ "gold_session_ids": [
+ "answer_54f0d6f9_1",
+ "answer_54f0d6f9_2"
+ ]
+ },
+ {
+ "question_id": "a3045048",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6934264036172708,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "aaf71ce2_2",
+ "answer_016f6bd4_1",
+ "answer_016f6bd4_2",
+ "5cb58cc6",
+ "d12d5a98_1",
+ "9cdb028a",
+ "2def525b_1",
+ "536a334d",
+ "sharegpt_asoSnhq_0",
+ "752b10e6_2"
+ ],
+ "gold_session_ids": [
+ "answer_016f6bd4_2",
+ "answer_016f6bd4_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_d31cdae3",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_8464304d_2",
+ "answer_8464304d_1",
+ "20a5c39b_3",
+ "55033f6f",
+ "de695a4e",
+ "41743aae_2",
+ "9f8c432e_1",
+ "sharegpt_HDF4645_0",
+ "ultrachat_413288",
+ "ultrachat_169591"
+ ],
+ "gold_session_ids": [
+ "answer_8464304d_1",
+ "answer_8464304d_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_cd90e484",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_aa930b56_2",
+ "answer_aa930b56_1",
+ "6043dcf6_1",
+ "018eac29",
+ "83fb74bf_1",
+ "866de400",
+ "4bfcc251_1",
+ "sharegpt_obCC1BP_14",
+ "cc5a6057_2",
+ "af42ae9c_1"
+ ],
+ "gold_session_ids": [
+ "answer_aa930b56_1",
+ "answer_aa930b56_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_88806d6e",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.5912352048230277,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "ultrachat_562852",
+ "answer_e60a93ff_1",
+ "9ac3fa82_1",
+ "sharegpt_j4PmNxu_11",
+ "ultrachat_258611",
+ "c11fb634_1",
+ "answer_e60a93ff_2",
+ "24bfc009_5",
+ "ultrachat_220071",
+ "ef0341d4_2"
+ ],
+ "gold_session_ids": [
+ "answer_e60a93ff_1",
+ "answer_e60a93ff_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_4cd9eba1",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_5fcca8bc_1",
+ "answer_5fcca8bc_2",
+ "1b576f6e_2",
+ "dafc2440_2",
+ "f54d04cc",
+ "456860e7_2",
+ "sharegpt_3hCiKrk_0",
+ "d36a862c_1",
+ "3be2cefb_1",
+ "ultrachat_221041"
+ ],
+ "gold_session_ids": [
+ "answer_5fcca8bc_2",
+ "answer_5fcca8bc_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_93f6379c",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9010126104210695,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_544fe66c_2",
+ "answer_544fe66c_3",
+ "c07b8034",
+ "d3a34811",
+ "1b9b2960_3",
+ "b8a688f6",
+ "e06c6d99_5",
+ "sharegpt_cfESFSo_0",
+ "5024758e",
+ "answer_544fe66c_1"
+ ],
+ "gold_session_ids": [
+ "answer_544fe66c_2",
+ "answer_544fe66c_1",
+ "answer_544fe66c_3"
+ ]
+ },
+ {
+ "question_id": "b29f3365",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_436d4309_2",
+ "answer_436d4309_1",
+ "sharegpt_VhAV1pG_0",
+ "c279c578",
+ "ultrachat_58020",
+ "66b56e10",
+ "6b7605d1_1",
+ "ultrachat_533198",
+ "ultrachat_243663",
+ "95c9666f_5"
+ ],
+ "gold_session_ids": [
+ "answer_436d4309_2",
+ "answer_436d4309_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_2f56ae70",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.34870224750355494,
+ "mrr": 0.2,
+ "retrieved_session_ids": [
+ "3aac691d_2",
+ "dc3ee1d1_1",
+ "83cd701b_3",
+ "e61d966a_1",
+ "answer_7a36e820_2",
+ "answer_7a36e820_3",
+ "ultrachat_452651",
+ "sharegpt_KzDwaf1_165",
+ "647f5851",
+ "sharegpt_BZ0hFLo_0"
+ ],
+ "gold_session_ids": [
+ "answer_7a36e820_2",
+ "answer_7a36e820_3",
+ "answer_7a36e820_1"
+ ]
+ },
+ {
+ "question_id": "6613b389",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.7653606369886217,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_aaf71ce2_2",
+ "answer_aaf71ce2_3",
+ "e3d8da67",
+ "f1cc1ddb_1",
+ "2780323d_3",
+ "86c8f348_1",
+ "e87aa998_1",
+ "ultrachat_444602",
+ "ultrachat_31390",
+ "55bc2e1e"
+ ],
+ "gold_session_ids": [
+ "answer_aaf71ce2_3",
+ "answer_aaf71ce2_2",
+ "answer_aaf71ce2_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_78cf46a3",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8772153153380493,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_5a78688d_2",
+ "sharegpt_rx39ipj_0",
+ "7fd52e1c_2",
+ "answer_5a78688d_1",
+ "sharegpt_89M8Aiz_0",
+ "b89a3ec3",
+ "sharegpt_Fakxkr4_10",
+ "42b45f6e_2",
+ "e8e65a1f",
+ "52b19cba_1"
+ ],
+ "gold_session_ids": [
+ "answer_5a78688d_1",
+ "answer_5a78688d_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_0a05b494",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9197207891481876,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_a68db5db_2",
+ "56911dc5_6",
+ "answer_a68db5db_1",
+ "ultrachat_273213",
+ "0abe3e51",
+ "8fb3fe3a_2",
+ "sharegpt_eWQFoct_17",
+ "sharegpt_PfiDxfU_9",
+ "30677a32",
+ "05b84f71_1"
+ ],
+ "gold_session_ids": [
+ "answer_a68db5db_2",
+ "answer_a68db5db_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_1a1dc16d",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6934264036172708,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "87f1f950_2",
+ "answer_faad7d7a_2",
+ "answer_faad7d7a_1",
+ "ultrachat_480665",
+ "c12d5181_1",
+ "sharegpt_6cz1Sq6_328",
+ "ultrachat_63236",
+ "ultrachat_41485",
+ "ultrachat_428346",
+ "ultrachat_523153"
+ ],
+ "gold_session_ids": [
+ "answer_faad7d7a_2",
+ "answer_faad7d7a_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_2f584639",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_11a8f823_2",
+ "answer_11a8f823_1",
+ "ultrachat_430445",
+ "ultrachat_229781",
+ "c33707b0",
+ "sharegpt_bq9G6bT_2",
+ "13947b64_1",
+ "821242cd_2",
+ "68131948",
+ "d441f4c5"
+ ],
+ "gold_session_ids": [
+ "answer_11a8f823_2",
+ "answer_11a8f823_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_213fd887",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_53582e7e_2",
+ "answer_53582e7e_1",
+ "ultrachat_165091",
+ "dc1f9713_1",
+ "9bdbfc62_2",
+ "3b38bf46",
+ "dee39195_2",
+ "8aa17385",
+ "368276ab_3",
+ "6d734945"
+ ],
+ "gold_session_ids": [
+ "answer_53582e7e_2",
+ "answer_53582e7e_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_5438fa52",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.39780880120575696,
+ "mrr": 0.14285714285714285,
+ "retrieved_session_ids": [
+ "a8e24e22",
+ "ab8dce45",
+ "d68bab3c_1",
+ "0a699029_4",
+ "528f481d_6",
+ "ultrachat_69599",
+ "answer_b10f3828_2",
+ "answer_b10f3828_1",
+ "sharegpt_LVUZee8_35",
+ "290fe09a"
+ ],
+ "gold_session_ids": [
+ "answer_b10f3828_1",
+ "answer_b10f3828_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_c27434e8",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_d8e33f5c_2",
+ "answer_d8e33f5c_1",
+ "ultrachat_70335",
+ "4d578deb",
+ "sharegpt_TRck0h4_0",
+ "sharegpt_9XlbWvL_0",
+ "e6c3a50a",
+ "5d1c7bf7",
+ "11cd43df",
+ "16933961"
+ ],
+ "gold_session_ids": [
+ "answer_d8e33f5c_1",
+ "answer_d8e33f5c_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_fe651585",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.5012658353418871,
+ "mrr": 0.25,
+ "retrieved_session_ids": [
+ "ff6b8624",
+ "e30c4a7b_2",
+ "9f220c66_1",
+ "answer_65600ff6_2",
+ "answer_65600ff6_1",
+ "231c60aa",
+ "ultrachat_576354",
+ "987b36db_3",
+ "da828711",
+ "cee0b8d5"
+ ],
+ "gold_session_ids": [
+ "answer_65600ff6_2",
+ "answer_65600ff6_1"
+ ]
+ },
+ {
+ "question_id": "8c18457d",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_124f5dc3_2",
+ "answer_124f5dc3_1",
+ "e8e65a1f",
+ "3c8e7c0e_2",
+ "98cd882c_2",
+ "c3023a45_2",
+ "ultrachat_417305",
+ "c8dcbe7f_1",
+ "9f803df5_1",
+ "ultrachat_166506"
+ ],
+ "gold_session_ids": [
+ "answer_124f5dc3_2",
+ "answer_124f5dc3_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_70e84552_abs",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_b3070ec4_abs_1",
+ "answer_b3070ec4_abs_2",
+ "e2cd250e_2",
+ "f6ba44fe_1",
+ "0a699029_4",
+ "4e0da76d",
+ "ultrachat_96694",
+ "sharegpt_3hJWfne_11",
+ "51d338f6",
+ "1572cf8a"
+ ],
+ "gold_session_ids": [
+ "answer_b3070ec4_abs_2",
+ "answer_b3070ec4_abs_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_93159ced_abs",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.5912352048230277,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "156988fe",
+ "answer_e5131a1b_abs_1",
+ "76385c25",
+ "f5eb52d6",
+ "168feef7_2",
+ "eb80877b_1",
+ "answer_e5131a1b_abs_2",
+ "7b9ee249_1",
+ "2f097fba_1",
+ "56bba837"
+ ],
+ "gold_session_ids": [
+ "answer_e5131a1b_abs_1",
+ "answer_e5131a1b_abs_2"
+ ]
+ },
+ {
+ "question_id": "982b5123_abs",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.18457569677956817,
+ "mrr": 0.1111111111111111,
+ "retrieved_session_ids": [
+ "ultrachat_447209",
+ "a8179676",
+ "sharegpt_kaEoitr_15",
+ "sharegpt_jxsIY6L_0",
+ "b1793bba",
+ "ultrachat_461193",
+ "bee1aec6_2",
+ "fe82a033_1",
+ "answer_ab603dd5_abs_1",
+ "5080119d_1"
+ ],
+ "gold_session_ids": [
+ "answer_ab603dd5_abs_1",
+ "answer_ab603dd5_abs_2"
+ ]
+ },
+ {
+ "question_id": "c8090214_abs",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.3065735963827292,
+ "mrr": 0.3333333333333333,
+ "retrieved_session_ids": [
+ "ca3aa07f_1",
+ "d6f5639a_3",
+ "answer_70dc7d08_abs_1",
+ "ultrachat_393699",
+ "64c71782",
+ "ultrachat_129606",
+ "687f4717",
+ "f10be626",
+ "2cfd748e_5",
+ "b192ae00_1"
+ ],
+ "gold_session_ids": [
+ "answer_70dc7d08_abs_1",
+ "answer_70dc7d08_abs_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_c27434e8_abs",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_d8e33f5c_abs_1",
+ "answer_d8e33f5c_abs_2",
+ "ultrachat_552313",
+ "25d1830a_1",
+ "5e0e1fd8_3",
+ "d2335d72",
+ "5c184f9f_5",
+ "59a6150d_3",
+ "9b8dea8c_1",
+ "sharegpt_hGtkNGL_12"
+ ],
+ "gold_session_ids": [
+ "answer_d8e33f5c_abs_1",
+ "answer_d8e33f5c_abs_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_fe651585_abs",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6131471927654584,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_65600ff6_abs_2",
+ "d9bce149_2",
+ "sharegpt_7PLgNll_0",
+ "adc9bd10",
+ "ultrachat_238311",
+ "sharegpt_TBRPvuE_0",
+ "e4bacbcd_1",
+ "sharegpt_kTBVvbN_0",
+ "caa00337_1",
+ "6b7abcd9"
+ ],
+ "gold_session_ids": [
+ "answer_65600ff6_abs_1",
+ "answer_65600ff6_abs_2"
+ ]
+ },
+ {
+ "question_id": "6a1eabeb",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.5640920940185894,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "afe04238_7",
+ "answer_a25d4a91_2",
+ "51a529d2_2",
+ "d3ee5958_2",
+ "4aaf678f_3",
+ "9f80607d",
+ "131ff17e",
+ "d03995c6",
+ "9b3a7f2c_1",
+ "answer_a25d4a91_1"
+ ],
+ "gold_session_ids": [
+ "answer_a25d4a91_1",
+ "answer_a25d4a91_2"
+ ]
+ },
+ {
+ "question_id": "6aeb4375",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_3f9693b7_1",
+ "answer_3f9693b7_2",
+ "sharegpt_a9ZjGW8_9",
+ "dc9bf721_1",
+ "29621a34",
+ "08256f84_2",
+ "ultrachat_115445",
+ "402e0082_5",
+ "8a5e1497_3",
+ "ultrachat_123028"
+ ],
+ "gold_session_ids": [
+ "answer_3f9693b7_1",
+ "answer_3f9693b7_2"
+ ]
+ },
+ {
+ "question_id": "830ce83f",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.5,
+ "mrr": 0.3333333333333333,
+ "retrieved_session_ids": [
+ "ad5a7d50_2",
+ "7a389bd1_7",
+ "answer_0b1a0942_2",
+ "a7f88bb3_2",
+ "dfbda5e9",
+ "639cec58",
+ "020149a5_1",
+ "answer_0b1a0942_1",
+ "8bc305e4_2",
+ "ultrachat_346816"
+ ],
+ "gold_session_ids": [
+ "answer_0b1a0942_1",
+ "answer_0b1a0942_2"
+ ]
+ },
+ {
+ "question_id": "852ce960",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6934264036172708,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "sharegpt_FYhsZ0Q_0",
+ "answer_3a6f1e82_1",
+ "answer_3a6f1e82_2",
+ "010a28ab_3",
+ "d36d11b9_2",
+ "e8bfacec_1",
+ "a110c4dc_1",
+ "a76e7e3c_1",
+ "aaf22693",
+ "f55fee5b_2"
+ ],
+ "gold_session_ids": [
+ "answer_3a6f1e82_1",
+ "answer_3a6f1e82_2"
+ ]
+ },
+ {
+ "question_id": "945e3d21",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8772153153380493,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_6a4f8626_1",
+ "dc880372_3",
+ "ef0341d4_2",
+ "answer_6a4f8626_2",
+ "9bbd38a6",
+ "5fcca8bc_1",
+ "6da0bba2_1",
+ "b916da64_1",
+ "4cbe4e20_4",
+ "c7847b66_1"
+ ],
+ "gold_session_ids": [
+ "answer_6a4f8626_1",
+ "answer_6a4f8626_2"
+ ]
+ },
+ {
+ "question_id": "d7c942c3",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_eecb10d9_2",
+ "answer_eecb10d9_1",
+ "sharegpt_LOF3smB_25",
+ "577da76d_2",
+ "f86a2788",
+ "8843878f_2",
+ "sharegpt_zxm712D_0",
+ "d15c9567_2",
+ "ddf0f116_3",
+ "b2243528_1"
+ ],
+ "gold_session_ids": [
+ "answer_eecb10d9_1",
+ "answer_eecb10d9_2"
+ ]
+ },
+ {
+ "question_id": "71315a70",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_c44b9df4_1",
+ "answer_c44b9df4_2",
+ "ultrachat_399278",
+ "3d82ba51_3",
+ "d1d3fd12_1",
+ "1a43e6cf",
+ "bcd5c30f_1",
+ "d8a74286",
+ "37d634c0_1",
+ "sharegpt_rhjZmCm_0"
+ ],
+ "gold_session_ids": [
+ "answer_c44b9df4_1",
+ "answer_c44b9df4_2"
+ ]
+ },
+ {
+ "question_id": "89941a93",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_e1403127_2",
+ "answer_e1403127_1",
+ "sharegpt_ZzCRfPu_30",
+ "44202c74_1",
+ "sharegpt_bn0XI2B_0",
+ "169213ae",
+ "dde142b2_2",
+ "ultrachat_62543",
+ "3581c390_3",
+ "sharegpt_ni8QFie_0"
+ ],
+ "gold_session_ids": [
+ "answer_e1403127_1",
+ "answer_e1403127_2"
+ ]
+ },
+ {
+ "question_id": "ce6d2d27",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.23719771276929622,
+ "mrr": 0.2,
+ "retrieved_session_ids": [
+ "ultrachat_553022",
+ "48ad2a93",
+ "4f5880c6_5",
+ "ultrachat_573386",
+ "answer_73540165_2",
+ "78dc3b29",
+ "a46d10a2_1",
+ "9f6bec4f",
+ "0e250059_1",
+ "7a4d00b3_2"
+ ],
+ "gold_session_ids": [
+ "answer_73540165_1",
+ "answer_73540165_2"
+ ]
+ },
+ {
+ "question_id": "9ea5eabc",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6934264036172708,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "418c27fa",
+ "answer_02e66dec_1",
+ "answer_02e66dec_2",
+ "6e2cca63_1",
+ "fcb08a46_2",
+ "sharegpt_KwbJJ66_18",
+ "sharegpt_3v69R59_0",
+ "390acb55_3",
+ "62e6593e_1",
+ "8c8b180e"
+ ],
+ "gold_session_ids": [
+ "answer_02e66dec_1",
+ "answer_02e66dec_2"
+ ]
+ },
+ {
+ "question_id": "07741c44",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_7e9ad7b4_2",
+ "answer_7e9ad7b4_1",
+ "8d969f3e_2",
+ "98ae7eef",
+ "ultrachat_541598",
+ "af4d4ecd_4",
+ "sharegpt_mUiNDhv_0",
+ "a4b136d1_2",
+ "sharegpt_vWO8y7i_35",
+ "f6b80ebe"
+ ],
+ "gold_session_ids": [
+ "answer_7e9ad7b4_1",
+ "answer_7e9ad7b4_2"
+ ]
+ },
+ {
+ "question_id": "a1eacc2a",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_0eb23770_1",
+ "answer_0eb23770_2",
+ "521255f7_2",
+ "1f035408",
+ "7daa121d_2",
+ "1ccb1920",
+ "93f23301_1",
+ "c910dd2c_3",
+ "ultrachat_187728",
+ "fde0ca18"
+ ],
+ "gold_session_ids": [
+ "answer_0eb23770_1",
+ "answer_0eb23770_2"
+ ]
+ },
+ {
+ "question_id": "184da446",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8315546295836225,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_e2f4f947_2",
+ "85a49fe0_2",
+ "fcddf7d1_2",
+ "bf633415_2",
+ "ultrachat_195877",
+ "answer_e2f4f947_1",
+ "82a89303",
+ "7ee6a6c0_1",
+ "sharegpt_nsHU229_0",
+ "ultrachat_116760"
+ ],
+ "gold_session_ids": [
+ "answer_e2f4f947_1",
+ "answer_e2f4f947_2"
+ ]
+ },
+ {
+ "question_id": "031748ae",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9197207891481876,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_8748f791_1",
+ "eff063bb_2",
+ "answer_8748f791_2",
+ "5ac7f9c5_3",
+ "d0488050",
+ "e0f71050_2",
+ "31702c20_3",
+ "f9e54d9c",
+ "bc660b5f_1",
+ "70764af1_4"
+ ],
+ "gold_session_ids": [
+ "answer_8748f791_1",
+ "answer_8748f791_2"
+ ]
+ },
+ {
+ "question_id": "4d6b87c8",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8175295903539445,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_766ab8da_2",
+ "4dd737a9_1",
+ "d87e86f6_1",
+ "sharegpt_yPCVYhX_0",
+ "5c85023a_1",
+ "9af4e346_1",
+ "answer_766ab8da_1",
+ "cffde796",
+ "85da7827",
+ "ultrachat_411499"
+ ],
+ "gold_session_ids": [
+ "answer_766ab8da_1",
+ "answer_766ab8da_2"
+ ]
+ },
+ {
+ "question_id": "0f05491a",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_d6d2eba8_2",
+ "answer_d6d2eba8_1",
+ "9b6db1a9_4",
+ "057cbbc4_1",
+ "9760f4ca",
+ "bb914c5c_1",
+ "d81d9846",
+ "9f4bec78",
+ "ee0921d4",
+ "0c260a71_3"
+ ],
+ "gold_session_ids": [
+ "answer_d6d2eba8_1",
+ "answer_d6d2eba8_2"
+ ]
+ },
+ {
+ "question_id": "08e075c7",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6934264036172708,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "25b3e36c_1",
+ "answer_cdbe2250_2",
+ "answer_cdbe2250_1",
+ "a2fa5ba3",
+ "b1d9d555_1",
+ "5ff94163_1",
+ "ultrachat_265107",
+ "ae77c245",
+ "77827bdf",
+ "sharegpt_Ah0tRO5_0"
+ ],
+ "gold_session_ids": [
+ "answer_cdbe2250_1",
+ "answer_cdbe2250_2"
+ ]
+ },
+ {
+ "question_id": "f9e8c073",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9197207891481876,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_b191df5b_2",
+ "sharegpt_GDFyxeT_0",
+ "answer_b191df5b_1",
+ "ultrachat_88473",
+ "22db3cc3_1",
+ "669318cf_3",
+ "0a042aef_2",
+ "50b32eed_2",
+ "ultrachat_219619",
+ "4604bc73_2"
+ ],
+ "gold_session_ids": [
+ "answer_b191df5b_1",
+ "answer_b191df5b_2"
+ ]
+ },
+ {
+ "question_id": "41698283",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_c7ddc051_1",
+ "answer_c7ddc051_2",
+ "431ae25c",
+ "28e948f5",
+ "0e8ba03b_1",
+ "631e4016",
+ "ff3f5781",
+ "47a2bd5a_1",
+ "f35e3cb7_1",
+ "f9d0bc67"
+ ],
+ "gold_session_ids": [
+ "answer_c7ddc051_1",
+ "answer_c7ddc051_2"
+ ]
+ },
+ {
+ "question_id": "2698e78f",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_9282283d_1",
+ "answer_9282283d_2",
+ "69674f25",
+ "sharegpt_DrLV0RL_0",
+ "d76030af_1",
+ "0fffb38f_2",
+ "5ac7f9c5_3",
+ "a0fcf806",
+ "ebd61d29_2",
+ "sharegpt_4qCDHY4_17"
+ ],
+ "gold_session_ids": [
+ "answer_9282283d_1",
+ "answer_9282283d_2"
+ ]
+ },
+ {
+ "question_id": "b6019101",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.39564672360221187,
+ "mrr": 0.16666666666666666,
+ "retrieved_session_ids": [
+ "sharegpt_CvVLg2J_18",
+ "de43030f_2",
+ "sharegpt_v9WEVZG_0",
+ "f5f1ff92_3",
+ "ultrachat_338597",
+ "answer_67074b4b_1",
+ "7b0dea50_2",
+ "ultrachat_331484",
+ "sharegpt_3u9F9yx_0",
+ "answer_67074b4b_2"
+ ],
+ "gold_session_ids": [
+ "answer_67074b4b_1",
+ "answer_67074b4b_2"
+ ]
+ },
+ {
+ "question_id": "45dc21b6",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9197207891481876,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_07664d43_1",
+ "5934b91f",
+ "answer_07664d43_2",
+ "28ad789a_1",
+ "0da2772d",
+ "5754046b",
+ "sharegpt_mMdyu3t_0",
+ "1c0921ee_3",
+ "de459c4b",
+ "b7f69a7d_2"
+ ],
+ "gold_session_ids": [
+ "answer_07664d43_1",
+ "answer_07664d43_2"
+ ]
+ },
+ {
+ "question_id": "5a4f22c0",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_b0f3dfff_2",
+ "answer_b0f3dfff_1",
+ "af778ece_1",
+ "5a1164c9_1",
+ "acb77ba4",
+ "007e7d81_2",
+ "sharegpt_I0uNTjm_22",
+ "b10f3828_1",
+ "c3f945f3_1",
+ "sharegpt_cXkL3cR_77"
+ ],
+ "gold_session_ids": [
+ "answer_b0f3dfff_1",
+ "answer_b0f3dfff_2"
+ ]
+ },
+ {
+ "question_id": "6071bd76",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.5437713091520254,
+ "mrr": 0.3333333333333333,
+ "retrieved_session_ids": [
+ "5c18cb32_1",
+ "9de53d68_1",
+ "answer_4dac77cb_2",
+ "sharegpt_3D3oQC0_12",
+ "answer_4dac77cb_1",
+ "ultrachat_174856",
+ "e6b6353d",
+ "ultrachat_69243",
+ "6f2c01fc_3",
+ "710833f3"
+ ],
+ "gold_session_ids": [
+ "answer_4dac77cb_1",
+ "answer_4dac77cb_2"
+ ]
+ },
+ {
+ "question_id": "e493bb7c",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_1a374afa_2",
+ "answer_1a374afa_1",
+ "ultrachat_185447",
+ "0aacd15f_1",
+ "9161500f_2",
+ "sharegpt_9CnlN2a_0",
+ "ultrachat_415114",
+ "ultrachat_287713",
+ "ultrachat_392200",
+ "ultrachat_513900"
+ ],
+ "gold_session_ids": [
+ "answer_1a374afa_1",
+ "answer_1a374afa_2"
+ ]
+ },
+ {
+ "question_id": "618f13b2",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_caf5b52e_1",
+ "answer_caf5b52e_2",
+ "sharegpt_PvScEsS_0",
+ "b3619c2c",
+ "01a120b5",
+ "d2366c37",
+ "dda70510_2",
+ "6172cc72_3",
+ "c51583cd_3",
+ "4632c2c6_1"
+ ],
+ "gold_session_ids": [
+ "answer_caf5b52e_1",
+ "answer_caf5b52e_2"
+ ]
+ },
+ {
+ "question_id": "72e3ee87",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_d7de9a6a_2",
+ "answer_d7de9a6a_1",
+ "ultrachat_195333",
+ "1c1f5ccc_4",
+ "73a2f346_3",
+ "sharegpt_KdbXhTW_9",
+ "ultrachat_523550",
+ "be8c3970_2",
+ "080a4000_2",
+ "ultrachat_421360"
+ ],
+ "gold_session_ids": [
+ "answer_d7de9a6a_1",
+ "answer_d7de9a6a_2"
+ ]
+ },
+ {
+ "question_id": "c4ea545c",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.7977228895450266,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_d3bf812b_2",
+ "5be74579",
+ "a2a8e2c9",
+ "sharegpt_71QV6Zs_7",
+ "sharegpt_I0uNTjm_16",
+ "b2d284b1",
+ "sharegpt_JJu53iW_0",
+ "9f9b402a_5",
+ "answer_d3bf812b_1",
+ "85a49fe0_1"
+ ],
+ "gold_session_ids": [
+ "answer_d3bf812b_1",
+ "answer_d3bf812b_2"
+ ]
+ },
+ {
+ "question_id": "01493427",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_a7b44747_2",
+ "answer_a7b44747_1",
+ "88e73f02",
+ "077f7bcb_3",
+ "291d8c82",
+ "5a840928_1",
+ "55e6ff53",
+ "3e8f07c9_1",
+ "087ac53a_2",
+ "a4d6c239_2"
+ ],
+ "gold_session_ids": [
+ "answer_a7b44747_1",
+ "answer_a7b44747_2"
+ ]
+ },
+ {
+ "question_id": "6a27ffc2",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9197207891481876,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_77f32504_2",
+ "4604bc73_3",
+ "answer_77f32504_1",
+ "6ae451d1_2",
+ "ultrachat_161210",
+ "sharegpt_KSCqghj_0",
+ "9ded9e01_2",
+ "sharegpt_kkqlRlo_13",
+ "eb87aced_1",
+ "ultrachat_505209"
+ ],
+ "gold_session_ids": [
+ "answer_77f32504_1",
+ "answer_77f32504_2"
+ ]
+ },
+ {
+ "question_id": "2133c1b5",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.2640681225725909,
+ "mrr": 0.25,
+ "retrieved_session_ids": [
+ "a63ad8e3_3",
+ "sharegpt_bSlIUYf_32",
+ "sharegpt_Olp439q_19",
+ "answer_52382508_2",
+ "3b655772_1",
+ "9a4c74d3_1",
+ "sharegpt_MvUpG5V_5",
+ "add9b012_1",
+ "ultrachat_460074",
+ "ultrachat_502724"
+ ],
+ "gold_session_ids": [
+ "answer_52382508_1",
+ "answer_52382508_2"
+ ]
+ },
+ {
+ "question_id": "18bc8abd",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_fff743f5_2",
+ "answer_fff743f5_1",
+ "a35679d5_2",
+ "2714885f_3",
+ "951df45e",
+ "ultrachat_529103",
+ "sharegpt_afCogMg_34",
+ "4ad82121_2",
+ "c6d55c42_2",
+ "ba6b67af_3"
+ ],
+ "gold_session_ids": [
+ "answer_fff743f5_1",
+ "answer_fff743f5_2"
+ ]
+ },
+ {
+ "question_id": "db467c8c",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.5714285040141098,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "sharegpt_ocf8Lud_39",
+ "answer_611b6e83_1",
+ "77cf4551_1",
+ "76967d22",
+ "sharegpt_7nYPy2P_29",
+ "sharegpt_HBmf6q0_0",
+ "ultrachat_553961",
+ "daac1a76",
+ "answer_611b6e83_2",
+ "sharegpt_TAL32at_0"
+ ],
+ "gold_session_ids": [
+ "answer_611b6e83_1",
+ "answer_611b6e83_2"
+ ]
+ },
+ {
+ "question_id": "7a87bd0c",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_d08a934d_1",
+ "answer_d08a934d_2",
+ "e250791e_3",
+ "3d98b7e6_2",
+ "15519944_1",
+ "9aaeecbf",
+ "93e9e82c_2",
+ "62369c56_1",
+ "81f318a2_1",
+ "sharegpt_AOimCZD_0"
+ ],
+ "gold_session_ids": [
+ "answer_d08a934d_1",
+ "answer_d08a934d_2"
+ ]
+ },
+ {
+ "question_id": "e61a7584",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_f25c32f5_1",
+ "answer_f25c32f5_2",
+ "ultrachat_367590",
+ "82159da4_1",
+ "285525b2_1",
+ "574df152_2",
+ "afa96a40",
+ "18a06652_4",
+ "6a4b4b7e_2",
+ "e1416816_2"
+ ],
+ "gold_session_ids": [
+ "answer_f25c32f5_1",
+ "answer_f25c32f5_2"
+ ]
+ },
+ {
+ "question_id": "1cea1afa",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_79c395a9_1",
+ "answer_79c395a9_2",
+ "e9ad5914_5",
+ "9c479e89_1",
+ "7063fad2",
+ "sharegpt_inuIr9J_119",
+ "e131bcd4",
+ "7af38385_1",
+ "ultrachat_51727",
+ "ultrachat_455127"
+ ],
+ "gold_session_ids": [
+ "answer_79c395a9_1",
+ "answer_79c395a9_2"
+ ]
+ },
+ {
+ "question_id": "ed4ddc30",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8503449055347546,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_babbaccb_2",
+ "4ffa04a2_4",
+ "01a120b5",
+ "c279c578",
+ "answer_babbaccb_1",
+ "57c91b14_2",
+ "e4969569_2",
+ "4ed9cd1b_2",
+ "8b726b64_1",
+ "aa31c397_2"
+ ],
+ "gold_session_ids": [
+ "answer_babbaccb_1",
+ "answer_babbaccb_2"
+ ]
+ },
+ {
+ "question_id": "8fb83627",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_966cecbb_1",
+ "answer_966cecbb_2",
+ "4019b1a8_5",
+ "9315f268_1",
+ "ultrachat_172298",
+ "be050ab3_2",
+ "02b81ad9",
+ "ultrachat_44939",
+ "b184aab6",
+ "88e01130_1"
+ ],
+ "gold_session_ids": [
+ "answer_966cecbb_1",
+ "answer_966cecbb_2"
+ ]
+ },
+ {
+ "question_id": "b01defab",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_8c0712af_2",
+ "answer_8c0712af_1",
+ "sharegpt_VbIxRWJ_33",
+ "sharegpt_zMUpvkc_57",
+ "59dc866c_1",
+ "ultrachat_76442",
+ "639cec58",
+ "fc69fd44_7",
+ "9aa07e25",
+ "6408b911"
+ ],
+ "gold_session_ids": [
+ "answer_8c0712af_1",
+ "answer_8c0712af_2"
+ ]
+ },
+ {
+ "question_id": "22d2cb42",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_bcce0b73_1",
+ "answer_bcce0b73_2",
+ "ultrachat_227551",
+ "74932466_1",
+ "060aeced",
+ "57c91b14_2",
+ "ff898925",
+ "sharegpt_ZXM9kAl_0",
+ "eec72a82_4",
+ "ultrachat_18212"
+ ],
+ "gold_session_ids": [
+ "answer_bcce0b73_1",
+ "answer_bcce0b73_2"
+ ]
+ },
+ {
+ "question_id": "0e4e4c46",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8315546295836225,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_f2f998c7_1",
+ "b42cbaf0_1",
+ "71d1e644",
+ "sharegpt_XOT5vdc_6",
+ "17822471",
+ "answer_f2f998c7_2",
+ "cbd08e3c_abs_2",
+ "0773943b",
+ "55e0c6db_1",
+ "e82ce7cd_3"
+ ],
+ "gold_session_ids": [
+ "answer_f2f998c7_1",
+ "answer_f2f998c7_2"
+ ]
+ },
+ {
+ "question_id": "4b24c848",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.36181498356361597,
+ "mrr": 0.1111111111111111,
+ "retrieved_session_ids": [
+ "ultrachat_244540",
+ "16e094b3_1",
+ "b0855671_2",
+ "sharegpt_Lj1CaQL_0",
+ "306cb175",
+ "f5cf8815_1",
+ "ultrachat_421733",
+ "1815d1b7",
+ "answer_2cec623b_2",
+ "answer_2cec623b_1"
+ ],
+ "gold_session_ids": [
+ "answer_2cec623b_1",
+ "answer_2cec623b_2"
+ ]
+ },
+ {
+ "question_id": "7e974930",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_c9f5693c_1",
+ "answer_c9f5693c_2",
+ "705b5399_3",
+ "ultrachat_251049",
+ "898ce7a5_1",
+ "ultrachat_113430",
+ "sharegpt_EuvEEnD_0",
+ "a1e169b1",
+ "sharegpt_cXkL3cR_77",
+ "ultrachat_380252"
+ ],
+ "gold_session_ids": [
+ "answer_c9f5693c_1",
+ "answer_c9f5693c_2"
+ ]
+ },
+ {
+ "question_id": "603deb26",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.21840743681816419,
+ "mrr": 0.16666666666666666,
+ "retrieved_session_ids": [
+ "c4661b3c_1",
+ "88611fff",
+ "a55a5afc_2",
+ "93191d30_2",
+ "81f6b12f_2",
+ "answer_8afdebac_2",
+ "ede65c84_2",
+ "f0e71553_4",
+ "5e867c2f_2",
+ "37f66729"
+ ],
+ "gold_session_ids": [
+ "answer_8afdebac_1",
+ "answer_8afdebac_2"
+ ]
+ },
+ {
+ "question_id": "59524333",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8315546295836225,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_b28f2c7a_1",
+ "ultrachat_150151",
+ "ultrachat_458180",
+ "638e7094_1",
+ "sharegpt_Clr6oyu_0",
+ "answer_b28f2c7a_2",
+ "sharegpt_NYFuCmY_0",
+ "c9e7e793",
+ "sharegpt_KhxXaow_0",
+ "sharegpt_S6bD7yp_0"
+ ],
+ "gold_session_ids": [
+ "answer_b28f2c7a_1",
+ "answer_b28f2c7a_2"
+ ]
+ },
+ {
+ "question_id": "5831f84d",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6052602440527057,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "7a88956e",
+ "answer_8d63a897_1",
+ "ef84b994_abs_2",
+ "41c90f4c_2",
+ "0a1e90d1_1",
+ "answer_8d63a897_2",
+ "sharegpt_6I57AvU_67",
+ "6a471944_1",
+ "b70358bc_6",
+ "0d66562c_4"
+ ],
+ "gold_session_ids": [
+ "answer_8d63a897_1",
+ "answer_8d63a897_2"
+ ]
+ },
+ {
+ "question_id": "eace081b",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_8a791264_1",
+ "answer_8a791264_2",
+ "sharegpt_HOO59Ue_137",
+ "9316aae3_1",
+ "4d04b866",
+ "4af27eab_2",
+ "413b57cb_3",
+ "1dcfb9a0_2",
+ "a9ab2024",
+ "f340a3db"
+ ],
+ "gold_session_ids": [
+ "answer_8a791264_1",
+ "answer_8a791264_2"
+ ]
+ },
+ {
+ "question_id": "affe2881",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6509209298071326,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "fb5dd87f_7",
+ "answer_90de9b4d_1",
+ "4604bc73_3",
+ "answer_90de9b4d_2",
+ "377f8fd9_2",
+ "0018b628",
+ "33994682_3",
+ "sharegpt_P0Qe2Gx_0",
+ "7badf4f3_1",
+ "3d895af0_3"
+ ],
+ "gold_session_ids": [
+ "answer_90de9b4d_1",
+ "answer_90de9b4d_2"
+ ]
+ },
+ {
+ "question_id": "50635ada",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_dcd74827_1",
+ "answer_dcd74827_2",
+ "b6f35f3a_1",
+ "42bcee92_2",
+ "03f76b6b",
+ "add7cc1e",
+ "sharegpt_eSTYwMj_10",
+ "2e53911b_1",
+ "419099c7_1",
+ "sharegpt_BVhcEnF_0"
+ ],
+ "gold_session_ids": [
+ "answer_dcd74827_1",
+ "answer_dcd74827_2"
+ ]
+ },
+ {
+ "question_id": "e66b632c",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8772153153380493,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ac0140ce_1",
+ "b876911c",
+ "84098c1b",
+ "answer_ac0140ce_2",
+ "259d58da_2",
+ "c2808c05_2",
+ "ultrachat_306973",
+ "47d18b61_1",
+ "ultrachat_223509",
+ "ultrachat_533755"
+ ],
+ "gold_session_ids": [
+ "answer_ac0140ce_1",
+ "answer_ac0140ce_2"
+ ]
+ },
+ {
+ "question_id": "0ddfec37",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8503449055347546,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_a22b654d_2",
+ "e184e4c3_2",
+ "0a699029_2",
+ "3422786d_2",
+ "answer_a22b654d_1",
+ "147ab7e9_7",
+ "f2199726_2",
+ "ultrachat_180221",
+ "e4881cbd_1",
+ "sharegpt_ckrlzN2_35"
+ ],
+ "gold_session_ids": [
+ "answer_a22b654d_1",
+ "answer_a22b654d_2"
+ ]
+ },
+ {
+ "question_id": "f685340e",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8772153153380493,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_25df025b_1",
+ "a4815e8e_2",
+ "142e3203_1",
+ "answer_25df025b_2",
+ "e7dcd5df",
+ "e862f726_1",
+ "992eb1f3_2",
+ "eef7c3fa_1",
+ "a8ef396f_1",
+ "c07ae000_2"
+ ],
+ "gold_session_ids": [
+ "answer_25df025b_1",
+ "answer_25df025b_2"
+ ]
+ },
+ {
+ "question_id": "cc5ded98",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_a5b68517_1",
+ "answer_a5b68517_2",
+ "c63b2c8f",
+ "ef785bb4_2",
+ "b13d5e4e_1",
+ "d973fa59_1",
+ "0a140d49_1",
+ "c937aab6",
+ "sharegpt_bTDMt3m_80",
+ "f14a4532_1"
+ ],
+ "gold_session_ids": [
+ "answer_a5b68517_1",
+ "answer_a5b68517_2"
+ ]
+ },
+ {
+ "question_id": "dfde3500",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8772153153380493,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_35d6c0be_1",
+ "ad1c7ac6",
+ "59c704ad_1",
+ "answer_35d6c0be_2",
+ "86e218e3_2",
+ "5c18cb32_2",
+ "779c0465",
+ "5fcca8bc_2",
+ "134127c2_2",
+ "5d5e80c5_1"
+ ],
+ "gold_session_ids": [
+ "answer_35d6c0be_1",
+ "answer_35d6c0be_2"
+ ]
+ },
+ {
+ "question_id": "69fee5aa",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_d6028d6e_1",
+ "answer_d6028d6e_2",
+ "a0b8b0ad_7",
+ "5d284d7d_2",
+ "991d55e5_1",
+ "42adb80d_2",
+ "ultrachat_327452",
+ "sharegpt_QxC97cL_0",
+ "sharegpt_hpWyNVd_0",
+ "6ff77d45_2"
+ ],
+ "gold_session_ids": [
+ "answer_d6028d6e_1",
+ "answer_d6028d6e_2"
+ ]
+ },
+ {
+ "question_id": "7401057b",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_94650bfa_1",
+ "answer_94650bfa_2",
+ "c3757c5b",
+ "ultrachat_389374",
+ "sharegpt_Ah0tRO5_0",
+ "42e69993_1",
+ "7be23135",
+ "1450ffad",
+ "30f20967_2",
+ "8de5b7cf_2"
+ ],
+ "gold_session_ids": [
+ "answer_94650bfa_1",
+ "answer_94650bfa_2"
+ ]
+ },
+ {
+ "question_id": "cf22b7bf",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ae3a122b_1",
+ "answer_ae3a122b_2",
+ "c8fcdf5a_1",
+ "e201572a",
+ "ultrachat_346016",
+ "36d5bbde_1",
+ "22fe1c93",
+ "457e26bf",
+ "c3c334ff_1",
+ "8ab0292c_1"
+ ],
+ "gold_session_ids": [
+ "answer_ae3a122b_1",
+ "answer_ae3a122b_2"
+ ]
+ },
+ {
+ "question_id": "a2f3aa27",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.7977228895450266,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_5126c02d_1",
+ "c98b3349_3",
+ "b65f51fb_5",
+ "c11b64f3_1",
+ "f188b48d_2",
+ "4cf2ea34",
+ "b4f28f96_1",
+ "01a467ee_2",
+ "answer_5126c02d_2",
+ "57bf4122_1"
+ ],
+ "gold_session_ids": [
+ "answer_5126c02d_1",
+ "answer_5126c02d_2"
+ ]
+ },
+ {
+ "question_id": "c7dc5443",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9197207891481876,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_0cdbca92_2",
+ "c2fb8e86_2",
+ "answer_0cdbca92_1",
+ "debc34d7_2",
+ "e200d96c_3",
+ "42d9467c_2",
+ "ultrachat_494538",
+ "sharegpt_hXYFkwv_35",
+ "9f0f59d1_3",
+ "728fc45a"
+ ],
+ "gold_session_ids": [
+ "answer_0cdbca92_1",
+ "answer_0cdbca92_2"
+ ]
+ },
+ {
+ "question_id": "06db6396",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_da72b1b4_1",
+ "answer_da72b1b4_2",
+ "ultrachat_333096",
+ "5bd9f1e6_1",
+ "sharegpt_2pwdGxJ_16",
+ "84d7650f_2",
+ "6ebdc1fe_2",
+ "da94a327_2",
+ "35038174_4",
+ "8235b87d"
+ ],
+ "gold_session_ids": [
+ "answer_da72b1b4_1",
+ "answer_da72b1b4_2"
+ ]
+ },
+ {
+ "question_id": "3ba21379",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6240505200038379,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "49576fd6",
+ "answer_cd345582_2",
+ "93aff0d7_2",
+ "sharegpt_QQQAWUd_19",
+ "answer_cd345582_1",
+ "f64c4793_1",
+ "ultrachat_224436",
+ "sharegpt_DPJevuk_0",
+ "5e50c81b",
+ "0f78155a_1"
+ ],
+ "gold_session_ids": [
+ "answer_cd345582_1",
+ "answer_cd345582_2"
+ ]
+ },
+ {
+ "question_id": "9bbe84a2",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_c6a0c6c2_1",
+ "answer_c6a0c6c2_2",
+ "ultrachat_148297",
+ "sharegpt_hcaZN94_546",
+ "e6e4fd83_2",
+ "cd81a69b",
+ "6b7605d1_2",
+ "1fc31387",
+ "ff0bb7fd_2",
+ "114c1fb7_6"
+ ],
+ "gold_session_ids": [
+ "answer_c6a0c6c2_1",
+ "answer_c6a0c6c2_2"
+ ]
+ },
+ {
+ "question_id": "10e09553",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_67be2c38_2",
+ "answer_67be2c38_1",
+ "7d1f5395",
+ "ff67236f_2",
+ "e5f785de",
+ "ultrachat_196381",
+ "3021992b_2",
+ "959e6cb0_7",
+ "58ab5fc5",
+ "918c4788_1"
+ ],
+ "gold_session_ids": [
+ "answer_67be2c38_1",
+ "answer_67be2c38_2"
+ ]
+ },
+ {
+ "question_id": "dad224aa",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9197207891481876,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_4a97ae40_1",
+ "6de8645d_1",
+ "answer_4a97ae40_2",
+ "756a4a02_2",
+ "56b5b760",
+ "b971ed75_3",
+ "91e581ad_1",
+ "5c690047_2",
+ "sharegpt_2Qvu6BM_34",
+ "d0bef272_1"
+ ],
+ "gold_session_ids": [
+ "answer_4a97ae40_1",
+ "answer_4a97ae40_2"
+ ]
+ },
+ {
+ "question_id": "ba61f0b9",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_f377cda7_1",
+ "answer_f377cda7_2",
+ "sharegpt_s8Opwwu_0",
+ "495cf518_1",
+ "c9213a03_1",
+ "221f2754_1",
+ "daca659d",
+ "ultrachat_421876",
+ "9316aae3_2",
+ "8c1da8f9_1"
+ ],
+ "gold_session_ids": [
+ "answer_f377cda7_1",
+ "answer_f377cda7_2"
+ ]
+ },
+ {
+ "question_id": "42ec0761",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_e3892371_2",
+ "answer_e3892371_1",
+ "53acd215",
+ "493bd421_2",
+ "812ccbd8_4",
+ "30aa70c6",
+ "sharegpt_JHGwlee_33",
+ "5dc74ba7",
+ "27d0d3e7",
+ "sharegpt_OLc5cYf_653"
+ ],
+ "gold_session_ids": [
+ "answer_e3892371_1",
+ "answer_e3892371_2"
+ ]
+ },
+ {
+ "question_id": "5c40ec5b",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_1cb52d0a_2",
+ "answer_1cb52d0a_1",
+ "be050ab3_2",
+ "d9868305_1",
+ "9eb7bb35",
+ "sharegpt_Ah0tRO5_0",
+ "0e8c2a62",
+ "5cc9e1ed_3",
+ "692075ef",
+ "25a2002c"
+ ],
+ "gold_session_ids": [
+ "answer_1cb52d0a_1",
+ "answer_1cb52d0a_2"
+ ]
+ },
+ {
+ "question_id": "c6853660",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.5012658353418871,
+ "mrr": 0.25,
+ "retrieved_session_ids": [
+ "ultrachat_435535",
+ "a4863bcb_2",
+ "33994682_3",
+ "answer_626e93c4_2",
+ "answer_626e93c4_1",
+ "sharegpt_PQ71V9I_7",
+ "0e931802_2",
+ "6ed1c85e",
+ "6474cfcc",
+ "71dc2037_2"
+ ],
+ "gold_session_ids": [
+ "answer_626e93c4_1",
+ "answer_626e93c4_2"
+ ]
+ },
+ {
+ "question_id": "26bdc477",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_f762ad8d_1",
+ "answer_f762ad8d_2",
+ "27ce9f13_2",
+ "37b3c75d",
+ "a223e307",
+ "8aa17385",
+ "2d79c1c3_2",
+ "sharegpt_bq9G6bT_2",
+ "551eae69_1",
+ "f684ac4c_1"
+ ],
+ "gold_session_ids": [
+ "answer_f762ad8d_1",
+ "answer_f762ad8d_2"
+ ]
+ },
+ {
+ "question_id": "0977f2af",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.21840743681816419,
+ "mrr": 0.16666666666666666,
+ "retrieved_session_ids": [
+ "e255d6fc_2",
+ "bdf45964_3",
+ "b24eddd9_1",
+ "ultrachat_554962",
+ "104be29f_1",
+ "answer_3bf5b73b_2",
+ "ultrachat_143766",
+ "sharegpt_l6aWsPc_0",
+ "9b083158_1",
+ "sharegpt_qDHHlQO_27"
+ ],
+ "gold_session_ids": [
+ "answer_3bf5b73b_1",
+ "answer_3bf5b73b_2"
+ ]
+ },
+ {
+ "question_id": "6aeb4375_abs",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6240505200038379,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "f21e11f1_2",
+ "answer_3f9693b7_abs_1",
+ "1cf6c966_2",
+ "ultrachat_209842",
+ "answer_3f9693b7_abs_2",
+ "710f4e21_1",
+ "fcff2f22",
+ "ea83c1fe_1",
+ "53c4fdd7_2",
+ "a5286353_1"
+ ],
+ "gold_session_ids": [
+ "answer_3f9693b7_abs_1",
+ "answer_3f9693b7_abs_2"
+ ]
+ },
+ {
+ "question_id": "031748ae_abs",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8503449055347546,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_8748f791_abs_1",
+ "1521a6c3_1",
+ "ultrachat_341706",
+ "sharegpt_EX0p0tj_19",
+ "answer_8748f791_abs_2",
+ "169cd54f",
+ "4d2c2c88_1",
+ "feba9363_1",
+ "210d7fd8_2",
+ "ultrachat_473482"
+ ],
+ "gold_session_ids": [
+ "answer_8748f791_abs_1",
+ "answer_8748f791_abs_2"
+ ]
+ },
+ {
+ "question_id": "2698e78f_abs",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6240505200038379,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "7ebf92f6_2",
+ "answer_9282283d_abs_1",
+ "3d94eae8",
+ "9bd6beaa_1",
+ "answer_9282283d_abs_2",
+ "9316aae3_1",
+ "f607e932_1",
+ "9e21d6ab_1",
+ "2e18c90b_1",
+ "ce291e98"
+ ],
+ "gold_session_ids": [
+ "answer_9282283d_abs_1",
+ "answer_9282283d_abs_2"
+ ]
+ },
+ {
+ "question_id": "2133c1b5_abs",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 0,
+ "recall_any_at_20": 0,
+ "ndcg_at_10": 0,
+ "mrr": 0,
+ "retrieved_session_ids": [
+ "6c7079a1",
+ "sharegpt_g3YFWLT_0",
+ "ef0341d4_1",
+ "fe6d8377_1",
+ "ef84b994_1",
+ "d0cfe938_1",
+ "18c1f00c_1",
+ "sharegpt_LmxRyal_9",
+ "sharegpt_BgSFVRW_0",
+ "ultrachat_543717"
+ ],
+ "gold_session_ids": [
+ "answer_52382508_abs_1",
+ "answer_52382508_abs_2"
+ ]
+ },
+ {
+ "question_id": "0ddfec37_abs",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.3706656904013186,
+ "mrr": 0.125,
+ "retrieved_session_ids": [
+ "6de70f46_1",
+ "3a11533e",
+ "4bd8d215_1",
+ "36a06ee8",
+ "25572732",
+ "a6b6067b",
+ "e3e061c4",
+ "answer_a22b654d_abs_1",
+ "a35959ba_2",
+ "answer_a22b654d_abs_2"
+ ],
+ "gold_session_ids": [
+ "answer_a22b654d_abs_1",
+ "answer_a22b654d_abs_2"
+ ]
+ },
+ {
+ "question_id": "f685340e_abs",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9197207891481876,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_25df025b_abs_1",
+ "6b895848_1",
+ "answer_25df025b_abs_2",
+ "bc6c5757_2",
+ "11fa2b8f",
+ "be4dc764_1",
+ "79bb1298_5",
+ "1970789e",
+ "sharegpt_oF0gxsp_0",
+ "7b49beea"
+ ],
+ "gold_session_ids": [
+ "answer_25df025b_abs_1",
+ "answer_25df025b_abs_2"
+ ]
+ },
+ {
+ "question_id": "89941a94",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.5437713091520254,
+ "mrr": 0.3333333333333333,
+ "retrieved_session_ids": [
+ "f7a61595_5",
+ "85b9848f",
+ "answer_e1403127_2",
+ "sharegpt_bvzjXyv_0",
+ "answer_e1403127_1",
+ "fe1e4351_2",
+ "0d758215_2",
+ "1f035408",
+ "de275215_3",
+ "ultrachat_556968"
+ ],
+ "gold_session_ids": [
+ "answer_e1403127_1",
+ "answer_e1403127_2"
+ ]
+ },
+ {
+ "question_id": "07741c45",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8772153153380493,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_7e9ad7b4_2",
+ "5300bfba",
+ "604de0a3_2",
+ "answer_7e9ad7b4_1",
+ "af4a2fd1",
+ "df846730",
+ "4c2bd0fe_1",
+ "37d65d40_3",
+ "sharegpt_hHGVWPU_4",
+ "465bc6a2"
+ ],
+ "gold_session_ids": [
+ "answer_7e9ad7b4_1",
+ "answer_7e9ad7b4_2"
+ ]
+ },
+ {
+ "question_id": "7161e7e2",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_sharegpt_5Lzox6N_0",
+ "3c3fee41",
+ "5aed819f",
+ "b4f63a70_3",
+ "5850de18_2",
+ "de93e859",
+ "e89b6be2_2",
+ "ultrachat_541021",
+ "6b17c99b_2",
+ "3070419a_1"
+ ],
+ "gold_session_ids": [
+ "answer_sharegpt_5Lzox6N_0"
+ ]
+ },
+ {
+ "question_id": "c4f10528",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ultrachat_234453",
+ "51c89181_1",
+ "8b1b7007",
+ "11248f50",
+ "c5c309b9_1",
+ "dc1f9713_2",
+ "e12cfbcd",
+ "sharegpt_yMwEvl7_557",
+ "7157bff1_2",
+ "5ecb3466_2"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_234453"
+ ]
+ },
+ {
+ "question_id": "89527b6b",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 0,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0,
+ "mrr": 0.0625,
+ "retrieved_session_ids": [
+ "9fa1d135_1",
+ "ultrachat_117600",
+ "1c6b85ea_1",
+ "8c1da8f9_1",
+ "4e147c90",
+ "48d385f0_1",
+ "689b46e2_4",
+ "51c262b6_2",
+ "9c722816",
+ "07f3f0e0_2"
+ ],
+ "gold_session_ids": [
+ "answer_sharegpt_YkWn1Ne_0"
+ ]
+ },
+ {
+ "question_id": "e9327a54",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ultrachat_480665",
+ "c58fdf1e_2",
+ "c38fb85b",
+ "c0d099e6_2",
+ "2d79c1c3_1",
+ "8221810f_3",
+ "c81897cd_2",
+ "b65f51fb_5",
+ "ba944cdc",
+ "28e0ea23_2"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_480665"
+ ]
+ },
+ {
+ "question_id": "4c36ccef",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.5,
+ "mrr": 0.3333333333333333,
+ "retrieved_session_ids": [
+ "f18b6c03_1",
+ "sharegpt_1PRjwYK_11",
+ "answer_ultrachat_448704",
+ "89749c78_1",
+ "940629b2_1",
+ "988baea9",
+ "5412d059",
+ "7b1d1f43_2",
+ "89d49715_1",
+ "ultrachat_149723"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_448704"
+ ]
+ },
+ {
+ "question_id": "6ae235be",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_sharegpt_IUWQYGQ_0",
+ "0a1e90d1_2",
+ "ec8fa65d_1",
+ "99188307",
+ "41b6e78c",
+ "ca3a4e4f_3",
+ "cc5a6057_1",
+ "809cabca_1",
+ "43811b62",
+ "c25b95c5_2"
+ ],
+ "gold_session_ids": [
+ "answer_sharegpt_IUWQYGQ_0"
+ ]
+ },
+ {
+ "question_id": "7e00a6cb",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.5,
+ "mrr": 0.3333333333333333,
+ "retrieved_session_ids": [
+ "4dd1ba8d_2",
+ "24f78a31_2",
+ "answer_ultrachat_370515",
+ "9998814e_2",
+ "b7dca8f1_1",
+ "5534efcb_2",
+ "a9981dc6_2",
+ "39358a85_3",
+ "ultrachat_564942",
+ "e2cd250e_1"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_370515"
+ ]
+ },
+ {
+ "question_id": "1903aded",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 0,
+ "recall_any_at_20": 0,
+ "ndcg_at_10": 0,
+ "mrr": 0,
+ "retrieved_session_ids": [
+ "9f8c432e_2",
+ "sharegpt_64onasJ_0",
+ "ultrachat_138198",
+ "9b3611d7_1",
+ "8adb232f_1",
+ "f8246a66_1",
+ "184efb91_3",
+ "ultrachat_134914",
+ "b163d176",
+ "sharegpt_MCI1EVK_39"
+ ],
+ "gold_session_ids": [
+ "answer_sharegpt_hA7AkP3_0"
+ ]
+ },
+ {
+ "question_id": "ceb54acb",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.3010299956639812,
+ "mrr": 0.1111111111111111,
+ "retrieved_session_ids": [
+ "de43030f_1",
+ "dddce60a_1",
+ "0840765f_1",
+ "c60d5c8a_1",
+ "2f23dd1a",
+ "7aa2a9af_2",
+ "0bbd7094_1",
+ "sharegpt_hUhgRkO_0",
+ "answer_sharegpt_cGdjmYo_0",
+ "197d99cc"
+ ],
+ "gold_session_ids": [
+ "answer_sharegpt_cGdjmYo_0"
+ ]
+ },
+ {
+ "question_id": "f523d9fe",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_sharegpt_m2xJfjo_0",
+ "ultrachat_391251",
+ "fbfda981",
+ "636e0e32_1",
+ "193c23bd_1",
+ "44165708",
+ "bed5fbab",
+ "73f75ab4",
+ "sharegpt_TF6GLMw_26",
+ "fd18e0dd_1"
+ ],
+ "gold_session_ids": [
+ "answer_sharegpt_m2xJfjo_0"
+ ]
+ },
+ {
+ "question_id": "0e5e2d1a",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ultrachat_113156",
+ "f84b4266",
+ "sharegpt_btv4SkQ_0",
+ "c756c34e",
+ "c4e5d969_2",
+ "5d79fd65_2",
+ "147ab7e9_7",
+ "22f9f163_1",
+ "ultrachat_222974",
+ "a803f50b_2"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_113156"
+ ]
+ },
+ {
+ "question_id": "fea54f57",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ultrachat_187684",
+ "5ac7f9c5_1",
+ "a98f8672_1",
+ "a1e90def_1",
+ "34590cab_1",
+ "c8854b28",
+ "d5f14d5f_1",
+ "0aa99e2e_1",
+ "3210adc8",
+ "d81c92cb"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_187684"
+ ]
+ },
+ {
+ "question_id": "cc539528",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6309297535714575,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "15a8ffb7_2",
+ "answer_ultrachat_374124",
+ "9e2e32c1_4",
+ "faa194ea",
+ "sharegpt_WBeb7tv_29",
+ "fb760626",
+ "18807892_4",
+ "ultrachat_490706",
+ "38e81260_5",
+ "3a1ea336_3"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_374124"
+ ]
+ },
+ {
+ "question_id": "dc439ea3",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ultrachat_459954",
+ "49d8a382",
+ "1bfd5a8b_2",
+ "5a81277c",
+ "ultrachat_186512",
+ "76ecc9b9_1",
+ "6ef99651_1",
+ "sharegpt_q2uAW7H_0",
+ "cb5637b3",
+ "ultrachat_4166"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_459954"
+ ]
+ },
+ {
+ "question_id": "18dcd5a5",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 0,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0,
+ "mrr": 0.06666666666666667,
+ "retrieved_session_ids": [
+ "sharegpt_d1tE544_5",
+ "33d16079_2",
+ "6ea1541e_1",
+ "47727b14",
+ "9844f740",
+ "sharegpt_MW0whoh_79",
+ "5e906cb0_2",
+ "aceffaab",
+ "sharegpt_2yaKUmC_0",
+ "959e6cb0_2"
+ ],
+ "gold_session_ids": [
+ "answer_sharegpt_hn3IS1q_0"
+ ]
+ },
+ {
+ "question_id": "488d3006",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ultrachat_275993",
+ "a68090b0_2",
+ "ce19df54_3",
+ "9a57d65b",
+ "82b82901",
+ "62c00fbc",
+ "1af2c0fb_1",
+ "b0f8fc7b_2",
+ "a394f6b5_1",
+ "3eb40515"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_275993"
+ ]
+ },
+ {
+ "question_id": "58470ed2",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6309297535714575,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "9f8973ac",
+ "answer_sharegpt_U4oCSfU_7",
+ "ultrachat_478785",
+ "4050ebff_6",
+ "4a324955",
+ "8ca0085a_3",
+ "5c0574b1",
+ "66bfa1db",
+ "69bd3d4a_1",
+ "ultrachat_173621"
+ ],
+ "gold_session_ids": [
+ "answer_sharegpt_U4oCSfU_7"
+ ]
+ },
+ {
+ "question_id": "8cf51dda",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_sharegpt_HFMn2ZX_0",
+ "daa09295",
+ "9beab022_1",
+ "e862f726_3",
+ "sharegpt_jpiaPoJ_738",
+ "7073a9cd_2",
+ "b500d9be",
+ "5df2f487",
+ "3358b5ca_2",
+ "ed383233_2"
+ ],
+ "gold_session_ids": [
+ "answer_sharegpt_HFMn2ZX_0"
+ ]
+ },
+ {
+ "question_id": "1d4da289",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.5,
+ "mrr": 0.3333333333333333,
+ "retrieved_session_ids": [
+ "517ad0f0",
+ "sharegpt_AfruXDt_0",
+ "answer_ultrachat_348449",
+ "bf85791d",
+ "a05c7d7d",
+ "f01df2d2_1",
+ "59c5f97e",
+ "ultrachat_213759",
+ "252715f8_4",
+ "76e56e2c_1"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_348449"
+ ]
+ },
+ {
+ "question_id": "8464fc84",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ultrachat_467053",
+ "8c65c687",
+ "1d5998ca_1",
+ "69a105f9_1",
+ "21d86ebb_2",
+ "98d39a6d",
+ "9ac77ac9_2",
+ "2d003fe4_2",
+ "602b4074_3",
+ "6ac43c5b_4"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_467053"
+ ]
+ },
+ {
+ "question_id": "8aef76bc",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ultrachat_563222",
+ "146dfabe",
+ "44b1595a",
+ "27d378b2_2",
+ "08e084f1_1",
+ "c7f6643e_2",
+ "05e258f4_2",
+ "014a5d36",
+ "5209e813_2",
+ "c80b8b74_1"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_563222"
+ ]
+ },
+ {
+ "question_id": "71a3fd6b",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.3333333333333333,
+ "mrr": 0.14285714285714285,
+ "retrieved_session_ids": [
+ "b7146673_1",
+ "d8f4b0f4",
+ "7f7e26d2_2",
+ "a29e9311",
+ "8419b385_3",
+ "f21fc078_1",
+ "answer_ultrachat_417348",
+ "eee3f7d3",
+ "36743359_3",
+ "d7a98679_2"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_417348"
+ ]
+ },
+ {
+ "question_id": "2bf43736",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6309297535714575,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "8897a2e5",
+ "answer_sharegpt_2kpncbX_13",
+ "16c3be68_1",
+ "1a5fb2cd",
+ "2e89bd03_1",
+ "d79173aa_3",
+ "5ca6cd28",
+ "fa858208_2",
+ "a93750ef_2",
+ "ultrachat_49111"
+ ],
+ "gold_session_ids": [
+ "answer_sharegpt_2kpncbX_13"
+ ]
+ },
+ {
+ "question_id": "70b3e69b",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ultrachat_334948",
+ "40afb91c_1",
+ "ultrachat_47234",
+ "sharegpt_O8qJclh_0",
+ "09e1a929_2",
+ "28cd2f40",
+ "1388b204_2",
+ "dd92723f_1",
+ "ultrachat_146029",
+ "35c5419d_3"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_334948"
+ ]
+ },
+ {
+ "question_id": "8752c811",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_sharegpt_6pWK9yx_0",
+ "9db30f1c_3",
+ "93f23301_2",
+ "56745369_2",
+ "9272a6a2_2",
+ "d0a41222_1",
+ "02b63d04_1",
+ "31f7c649_2",
+ "f0e71553_5",
+ "87afa197_1"
+ ],
+ "gold_session_ids": [
+ "answer_sharegpt_6pWK9yx_0"
+ ]
+ },
+ {
+ "question_id": "3249768e",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.2890648263178879,
+ "mrr": 0.1,
+ "retrieved_session_ids": [
+ "689b46e2_2",
+ "268e8532_2",
+ "71f2b8ba",
+ "0868982f_1",
+ "ec00a866_2",
+ "172b47cb_2",
+ "c6c5bb8b_2",
+ "11e9d591_2",
+ "779c0465",
+ "answer_sharegpt_CaxTGYP_0"
+ ],
+ "gold_session_ids": [
+ "answer_sharegpt_CaxTGYP_0"
+ ]
+ },
+ {
+ "question_id": "1b9b7252",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6309297535714575,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "0e726047",
+ "answer_ultrachat_115151",
+ "4f6e1b77_1",
+ "af4d4ecd_6",
+ "sharegpt_1AhFMpw_5",
+ "66e8a6dd",
+ "330c05e7_1",
+ "e8212ed9",
+ "064f42ee_1",
+ "3abaa836"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_115151"
+ ]
+ },
+ {
+ "question_id": "1568498a",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.43067655807339306,
+ "mrr": 0.25,
+ "retrieved_session_ids": [
+ "sharegpt_h87NLyS_49",
+ "f964cea3_1",
+ "70fed904",
+ "answer_sharegpt_d6JJiqH_76",
+ "28eb86ab",
+ "1fef8ffe_1",
+ "6ad46850_2",
+ "936b0969_1",
+ "a05cd79f_1",
+ "71e0db57"
+ ],
+ "gold_session_ids": [
+ "answer_sharegpt_d6JJiqH_76"
+ ]
+ },
+ {
+ "question_id": "6222b6eb",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_sharegpt_H9PiM5G_0",
+ "25ffc571",
+ "caf7480d_1",
+ "5209e813_2",
+ "7b2523d3",
+ "ab603dd5_abs_2",
+ "723bf11f",
+ "b35fadaa",
+ "40a90d51",
+ "sharegpt_cyc2xbb_0"
+ ],
+ "gold_session_ids": [
+ "answer_sharegpt_H9PiM5G_0"
+ ]
+ },
+ {
+ "question_id": "e8a79c70",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 0,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0,
+ "mrr": 0.058823529411764705,
+ "retrieved_session_ids": [
+ "c0d099e6_2",
+ "539778d4_2",
+ "c068abb2",
+ "486705ce",
+ "sharegpt_ha8kUEr_0",
+ "fc706dc5_2",
+ "c4c504d2_6",
+ "f731656b_2",
+ "45c29267",
+ "89aed484_2"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_13075"
+ ]
+ },
+ {
+ "question_id": "d596882b",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.5,
+ "mrr": 0.3333333333333333,
+ "retrieved_session_ids": [
+ "71888aff_2",
+ "0dc2efcc_2",
+ "answer_ultrachat_252214",
+ "5558a42e_3",
+ "456807b7_1",
+ "dc9bf721_1",
+ "53582e7e_1",
+ "bef3abcd",
+ "a864e7aa_4",
+ "692075ef"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_252214"
+ ]
+ },
+ {
+ "question_id": "e3fc4d6e",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_sharegpt_5m7gg5F_0",
+ "d31c3ec8_1",
+ "ultrachat_442052",
+ "42adb80d_3",
+ "4d261642",
+ "9b72b151",
+ "ultrachat_161238",
+ "bec86aec",
+ "05be3ff0_4",
+ "sharegpt_ltqfPQc_0"
+ ],
+ "gold_session_ids": [
+ "answer_sharegpt_5m7gg5F_0"
+ ]
+ },
+ {
+ "question_id": "51b23612",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ultrachat_427265",
+ "409015ef_1",
+ "fd78c83d_2",
+ "f35cb519_2",
+ "8672f398_2",
+ "255c753b_1",
+ "cbd1fe79_2",
+ "74b0227e_2",
+ "17f444f3",
+ "ultrachat_343"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_427265"
+ ]
+ },
+ {
+ "question_id": "3e321797",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ultrachat_94624",
+ "07805a44",
+ "57144028_2",
+ "e77fc218_2",
+ "5558a42e_3",
+ "193c23bd_1",
+ "9252145e_2",
+ "04b7b396_1",
+ "11302335_4",
+ "sharegpt_EZmhz8p_0"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_94624"
+ ]
+ },
+ {
+ "question_id": "e982271f",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ultrachat_195444",
+ "sharegpt_BdpBz7Q_0",
+ "eb0d8dc1_4",
+ "ultrachat_564244",
+ "8a7c65c1_1",
+ "sharegpt_f83u8ds_59",
+ "ultrachat_383774",
+ "ultrachat_380780",
+ "7b4bce3b_2",
+ "db65997e"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_195444"
+ ]
+ },
+ {
+ "question_id": "352ab8bd",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_sharegpt_NoDZzot_7",
+ "eee803c7",
+ "3826dc55_5",
+ "337cbfc8_2",
+ "9bafe9a2",
+ "d6d87935_2",
+ "sharegpt_UCVg3jO_0",
+ "3b0f28f6_2",
+ "ultrachat_1052",
+ "df08e7a0"
+ ],
+ "gold_session_ids": [
+ "answer_sharegpt_NoDZzot_7"
+ ]
+ },
+ {
+ "question_id": "fca762bc",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.3333333333333333,
+ "mrr": 0.14285714285714285,
+ "retrieved_session_ids": [
+ "a8bb9614_2",
+ "d0cdfddb_2",
+ "3ac6fe4c",
+ "d5b3b034_1",
+ "42d9467c_1",
+ "2bc4ae7d",
+ "answer_ultrachat_39395",
+ "9beab022_1",
+ "3b7a6ecd_1",
+ "99eec551"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_39395"
+ ]
+ },
+ {
+ "question_id": "7a8d0b71",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_sharegpt_i0tMT9q_9",
+ "d5eab084_2",
+ "147ab7e9_5",
+ "ccc6ab06_2",
+ "f07752be_1",
+ "ultrachat_136480",
+ "f20e72e4_2",
+ "4aaf678f_4",
+ "c14e3f09_2",
+ "14725f4a_1"
+ ],
+ "gold_session_ids": [
+ "answer_sharegpt_i0tMT9q_9"
+ ]
+ },
+ {
+ "question_id": "a40e080f",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6309297535714575,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "e63fbbeb_1",
+ "answer_ultrachat_269020",
+ "sharegpt_fyAkcGP_81",
+ "ef917f63",
+ "sharegpt_oXgiN7q_327",
+ "cd38b6ee",
+ "eb18a1b7",
+ "sharegpt_G3RxeFJ_94",
+ "528f481d_6",
+ "85736a07_2"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_269020"
+ ]
+ },
+ {
+ "question_id": "8b9d4367",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ultrachat_289157",
+ "7babf05b",
+ "c83bbe71_1",
+ "0bad887e_1",
+ "93d4bb28",
+ "20cf06b8",
+ "45d1c5c4",
+ "5880cdab",
+ "031061e6_4",
+ "ultrachat_321805"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_289157"
+ ]
+ },
+ {
+ "question_id": "5809eb10",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_sharegpt_4aJsGCH_0",
+ "sharegpt_GDFyxeT_0",
+ "fc3d6c80",
+ "922c819f_2",
+ "5237bb0b_1",
+ "sharegpt_fQhexkP_38",
+ "616b8721",
+ "35ee7421",
+ "sharegpt_n3MgsgL_0",
+ "sharegpt_QaiMtpK_0"
+ ],
+ "gold_session_ids": [
+ "answer_sharegpt_4aJsGCH_0"
+ ]
+ },
+ {
+ "question_id": "41275add",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.2890648263178879,
+ "mrr": 0.1,
+ "retrieved_session_ids": [
+ "a68db5db_1",
+ "eee803c7",
+ "acc3913b",
+ "sharegpt_dUoAhvI_71",
+ "9345f7dc_1",
+ "c07ae000_1",
+ "b3619c2c",
+ "53fe2138",
+ "2e0ff38d_2",
+ "answer_sharegpt_81riySf_0"
+ ],
+ "gold_session_ids": [
+ "answer_sharegpt_81riySf_0"
+ ]
+ },
+ {
+ "question_id": "4388e9dd",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_sharegpt_qTi81nS_0",
+ "0826def6_1",
+ "sharegpt_Tun3KJ4_0",
+ "313404c6_2",
+ "a6c0f032_3",
+ "cf9eda76",
+ "b6506028_2",
+ "3197d998_7",
+ "47311a4c_2",
+ "1ebd753f_2"
+ ],
+ "gold_session_ids": [
+ "answer_sharegpt_qTi81nS_0"
+ ]
+ },
+ {
+ "question_id": "4baee567",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_sharegpt_i9adwQn_0",
+ "sharegpt_wduYbsX_0",
+ "sharegpt_28Mwwk9_0",
+ "f6ec94b4_1",
+ "74711358_2",
+ "e00298a6_2",
+ "sharegpt_lWLBUhQ_547",
+ "ed3504e1",
+ "d5415748",
+ "4509e9fc"
+ ],
+ "gold_session_ids": [
+ "answer_sharegpt_i9adwQn_0"
+ ]
+ },
+ {
+ "question_id": "561fabcd",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_sharegpt_hChsWOp_97",
+ "aee13015_3",
+ "63c75b13_2",
+ "1c1f5ccc_6",
+ "cb442986",
+ "33da50d0_5",
+ "83fb74bf_1",
+ "sharegpt_QaiMtpK_0",
+ "864a563d_4",
+ "d5d7eff7"
+ ],
+ "gold_session_ids": [
+ "answer_sharegpt_hChsWOp_97"
+ ]
+ },
+ {
+ "question_id": "b759caee",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_sharegpt_2BSXlAr_0",
+ "eae516cd",
+ "fcae5c39_1",
+ "d3575920",
+ "f55fee5b_4",
+ "191f4c9b_2",
+ "3a667084_2",
+ "bab41bb6_5",
+ "22aa7cca_1",
+ "6be2f065_1"
+ ],
+ "gold_session_ids": [
+ "answer_sharegpt_2BSXlAr_0"
+ ]
+ },
+ {
+ "question_id": "ac031881",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_sharegpt_GYqnAhC_190",
+ "449b918c_3",
+ "sharegpt_cIj2dFl_4",
+ "e30307d8_1",
+ "eb95f93c",
+ "6608c1da",
+ "f43d9ae5",
+ "010a28ab_3",
+ "c8c3084d_3",
+ "57c91b14_1"
+ ],
+ "gold_session_ids": [
+ "answer_sharegpt_GYqnAhC_190"
+ ]
+ },
+ {
+ "question_id": "28bcfaac",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6309297535714575,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "8d410160",
+ "answer_ultrachat_446979",
+ "aafe556c",
+ "135a62d7_2",
+ "485f4604_2",
+ "774da34f_1",
+ "510b98f3",
+ "e7cc239c_2",
+ "7cd7c296_1",
+ "17cb8d2f_5"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_446979"
+ ]
+ },
+ {
+ "question_id": "16c90bf4",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ultrachat_294807",
+ "89d49715_1",
+ "afa0c2e7_5",
+ "cc830ab4_1",
+ "d576152e_4",
+ "101636c0_1",
+ "fa2714ed_1",
+ "b0da5097_2",
+ "8464304d_2",
+ "d5f4d9fa_1"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_294807"
+ ]
+ },
+ {
+ "question_id": "c8f1aeed",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ultrachat_519486",
+ "f7cd21e5",
+ "sharegpt_ebYKddM_87",
+ "ed8ba4b0_1",
+ "48a72204_3",
+ "f8a9cd01_2",
+ "b8620ffe_2",
+ "sharegpt_9ra36HI_32",
+ "12cd736c",
+ "199a5225_1"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_519486"
+ ]
+ },
+ {
+ "question_id": "eaca4986",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 0,
+ "recall_any_at_20": 0,
+ "ndcg_at_10": 0,
+ "mrr": 0,
+ "retrieved_session_ids": [
+ "931c521e_1",
+ "f987da7e_3",
+ "3477c37c_1",
+ "sharegpt_yC6nsDU_0",
+ "1c1a2b7f_2",
+ "564761aa",
+ "sharegpt_zY4q7i0_5",
+ "0e543e9e_2",
+ "sharegpt_kNi3QzP_24",
+ "07fb2f15_1"
+ ],
+ "gold_session_ids": [
+ "answer_sharegpt_SS141vi_0"
+ ]
+ },
+ {
+ "question_id": "c7cf7dfd",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ultrachat_456407",
+ "96b8c9ee",
+ "285a29e7_1",
+ "9db6d988_2",
+ "sharegpt_gQaSoiF_13",
+ "e24943fc",
+ "1382d6fb_1",
+ "1341f0b9",
+ "780b6c7c_1",
+ "e27fe923_5"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_456407"
+ ]
+ },
+ {
+ "question_id": "e48988bc",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.5,
+ "mrr": 0.3333333333333333,
+ "retrieved_session_ids": [
+ "f84b4266",
+ "6e110a53_1",
+ "answer_ultrachat_174360",
+ "476b1cbb",
+ "8fb3fe3a_3",
+ "ff6bccaa_3",
+ "ultrachat_494373",
+ "e388463b_1",
+ "ultrachat_283303",
+ "ultrachat_58258"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_174360"
+ ]
+ },
+ {
+ "question_id": "1de5cff2",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6309297535714575,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "848d721f_1",
+ "answer_ultrachat_440262",
+ "001cefa7_2",
+ "4c6775ed_1",
+ "64048e35",
+ "42d9467c_1",
+ "e7418361_2",
+ "28031c43_2",
+ "1c1f5ccc_1",
+ "c14dbe17"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_440262"
+ ]
+ },
+ {
+ "question_id": "65240037",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ultrachat_403752",
+ "f8dc1273",
+ "37141183",
+ "92466e8e_1",
+ "7df4c735_1",
+ "0504a710_3",
+ "9139676d",
+ "sharegpt_ME35OD8_167",
+ "d3d299a4",
+ "46f07bd0_2"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_403752"
+ ]
+ },
+ {
+ "question_id": "778164c6",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.3333333333333333,
+ "mrr": 0.14285714285714285,
+ "retrieved_session_ids": [
+ "b605741a_1",
+ "490bb46d_1",
+ "05b551b6",
+ "35201d43",
+ "483ddd90",
+ "81e406dd_2",
+ "answer_ultrachat_399000",
+ "ddaa5a15_2",
+ "1340826c",
+ "sharegpt_ki9IVDq_6"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_399000"
+ ]
+ }
+ ]
+}
\ No newline at end of file
diff --git a/benchmark/data/longmemeval_results_hybrid.json b/benchmark/data/longmemeval_results_hybrid.json
new file mode 100644
index 00000000..21f1ad41
--- /dev/null
+++ b/benchmark/data/longmemeval_results_hybrid.json
@@ -0,0 +1,12491 @@
+{
+ "mode": "hybrid",
+ "questions": 500,
+ "recall_any_at_5": 0.952,
+ "recall_any_at_10": 0.986,
+ "recall_any_at_20": 0.994,
+ "ndcg_at_10": 0.8785881269713454,
+ "mrr": 0.8821431318681319,
+ "per_type": {
+ "single-session-user": {
+ "count": 70,
+ "recall_any_at_5": 0.9,
+ "recall_any_at_10": 0.9714285714285714
+ },
+ "multi-session": {
+ "count": 133,
+ "recall_any_at_5": 0.9774436090225563,
+ "recall_any_at_10": 1
+ },
+ "single-session-preference": {
+ "count": 30,
+ "recall_any_at_5": 0.8333333333333334,
+ "recall_any_at_10": 0.9666666666666667
+ },
+ "temporal-reasoning": {
+ "count": 133,
+ "recall_any_at_5": 0.9548872180451128,
+ "recall_any_at_10": 0.9774436090225563
+ },
+ "knowledge-update": {
+ "count": 78,
+ "recall_any_at_5": 0.9871794871794872,
+ "recall_any_at_10": 1
+ },
+ "single-session-assistant": {
+ "count": 56,
+ "recall_any_at_5": 0.9642857142857143,
+ "recall_any_at_10": 0.9821428571428571
+ }
+ },
+ "per_question": [
+ {
+ "question_id": "e47becba",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_280352e9",
+ "f6859b48_2",
+ "sharegpt_T1EiHWI_13",
+ "41abc171_4",
+ "ultrachat_336116",
+ "02bd2b90_3",
+ "sharegpt_UnjngE7_65",
+ "ultrachat_214101",
+ "ultrachat_486410",
+ "ultrachat_514031"
+ ],
+ "gold_session_ids": [
+ "answer_280352e9"
+ ]
+ },
+ {
+ "question_id": "118b2229",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_40a90d51",
+ "628c958c_7",
+ "db73b7e4_4",
+ "db50c0f6",
+ "7d8b5834_1",
+ "0afa904a",
+ "728deb4d_5",
+ "79f74d4e_1",
+ "sharegpt_dabXXRO_10",
+ "004edb32"
+ ],
+ "gold_session_ids": [
+ "answer_40a90d51"
+ ]
+ },
+ {
+ "question_id": "51a45a95",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_d61669c7",
+ "07ba9acd_2",
+ "55e0c6db_2",
+ "98c198fb",
+ "sharegpt_MKMWjX0_25",
+ "6e0b1800_2",
+ "5c44d9fe_1",
+ "sharegpt_3vxz2Zr_0",
+ "07d1915d_1",
+ "dfe646a7_1"
+ ],
+ "gold_session_ids": [
+ "answer_d61669c7"
+ ]
+ },
+ {
+ "question_id": "58bf7951",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_355c48bb",
+ "ultrachat_304942",
+ "793382f9_2",
+ "ultrachat_268434",
+ "5b83c26e_1",
+ "ultrachat_278141",
+ "ultrachat_71618",
+ "1587bd78_1",
+ "ultrachat_143341",
+ "ultrachat_155546"
+ ],
+ "gold_session_ids": [
+ "answer_355c48bb"
+ ]
+ },
+ {
+ "question_id": "1e043500",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.43067655807339306,
+ "mrr": 0.25,
+ "retrieved_session_ids": [
+ "01bd9def",
+ "8020d945_2",
+ "sharegpt_bph7DTk_7",
+ "answer_3e012175",
+ "sharegpt_uQMdL9t_0",
+ "c285b705_2",
+ "sharegpt_Clr6oyu_0",
+ "4d0d7984",
+ "2d0d0004",
+ "b28990c8_1"
+ ],
+ "gold_session_ids": [
+ "answer_3e012175"
+ ]
+ },
+ {
+ "question_id": "c5e8278d",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_f6168136",
+ "f58317c6",
+ "sharegpt_DB8U4oH_9",
+ "e4881cbd_4",
+ "89b1028d_1",
+ "aaf71ce2_2",
+ "0fe848c5_3",
+ "dc832f73",
+ "e01eaa19_2",
+ "03362ec8_2"
+ ],
+ "gold_session_ids": [
+ "answer_f6168136"
+ ]
+ },
+ {
+ "question_id": "6ade9755",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_9398da02",
+ "a200b713_4",
+ "ultrachat_270925",
+ "sharegpt_BghMUHh_0",
+ "5df2f487",
+ "7c5ab02a_2",
+ "de5634fc_2",
+ "ultrachat_462",
+ "ec8fa65d_2",
+ "5d049e3c"
+ ],
+ "gold_session_ids": [
+ "answer_9398da02"
+ ]
+ },
+ {
+ "question_id": "6f9b354f",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_feb5200f",
+ "ultrachat_549317",
+ "de877349",
+ "0c0f2449_2",
+ "a79f1a04_1",
+ "c22eca8c_1",
+ "sharegpt_BYHT3Xz_0",
+ "a729934a_2",
+ "40643340_2",
+ "2c275178_1"
+ ],
+ "gold_session_ids": [
+ "answer_feb5200f"
+ ]
+ },
+ {
+ "question_id": "58ef2f1c",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_59547700",
+ "e6ab6a7b_2",
+ "ultrachat_5212",
+ "e82ce7cd_1",
+ "sharegpt_eHZgwR0_9",
+ "13f3da23_3",
+ "b546cb39_2",
+ "sharegpt_RFw4j0W_0",
+ "ultrachat_246324",
+ "ultrachat_477344"
+ ],
+ "gold_session_ids": [
+ "answer_59547700"
+ ]
+ },
+ {
+ "question_id": "f8c5f88b",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_c3567066",
+ "91880423",
+ "15232887_1",
+ "sharegpt_TKj3yO8_0",
+ "54e9197a",
+ "68ace657_1",
+ "66c23110_1",
+ "sharegpt_n2aGNgW_9",
+ "5520ca18",
+ "0a6bf5e4_2"
+ ],
+ "gold_session_ids": [
+ "answer_c3567066"
+ ]
+ },
+ {
+ "question_id": "5d3d2817",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.3010299956639812,
+ "mrr": 0.1111111111111111,
+ "retrieved_session_ids": [
+ "670ca40e",
+ "sharegpt_ebWP6Hz_0",
+ "sharegpt_ipLglky_48",
+ "sharegpt_bTDMt3m_64",
+ "sharegpt_BKl952A_23",
+ "49b78a55_1",
+ "e00298a6_1",
+ "e93efd9a_1",
+ "answer_235eb6fb",
+ "85d16d01"
+ ],
+ "gold_session_ids": [
+ "answer_235eb6fb"
+ ]
+ },
+ {
+ "question_id": "7527f7e2",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_7cb94507",
+ "18a06652_3",
+ "44bbd8cd_2",
+ "4e2acecc",
+ "afbd7193_1",
+ "sharegpt_xIDRgaY_0",
+ "sharegpt_cRrfdsc_0",
+ "sharegpt_ToxFax1_535",
+ "bee872da_2",
+ "88fed3ed_2"
+ ],
+ "gold_session_ids": [
+ "answer_7cb94507"
+ ]
+ },
+ {
+ "question_id": "c960da58",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_e05e4612",
+ "sharegpt_ErOTMZ3_59",
+ "sharegpt_CdnyHep_0",
+ "ultrachat_198948",
+ "f7ffddbc",
+ "sharegpt_OAUjazp_0",
+ "ultrachat_374640",
+ "b76006cb_3",
+ "sharegpt_SoyhGNj_0",
+ "52381716_1"
+ ],
+ "gold_session_ids": [
+ "answer_e05e4612"
+ ]
+ },
+ {
+ "question_id": "3b6f954b",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 0,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0,
+ "mrr": 0.07692307692307693,
+ "retrieved_session_ids": [
+ "c271c071",
+ "ultrachat_230611",
+ "ultrachat_64326",
+ "ultrachat_572251",
+ "sharegpt_cD2Kgkn_33",
+ "4ed9cd1b_1",
+ "e27891d3_2",
+ "ultrachat_294369",
+ "f999b05c_2",
+ "ultrachat_308865"
+ ],
+ "gold_session_ids": [
+ "answer_94030872"
+ ]
+ },
+ {
+ "question_id": "726462e0",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 0,
+ "recall_any_at_20": 0,
+ "ndcg_at_10": 0,
+ "mrr": 0,
+ "retrieved_session_ids": [
+ "741c6781_1",
+ "6ef99651_2",
+ "ce144f08",
+ "cdd9704d_1",
+ "9c05acfe_2",
+ "b84b7a2b_1",
+ "ultrachat_450587",
+ "7e4dab66_1",
+ "5b83c26e_1",
+ "sharegpt_IxOWCIE_9"
+ ],
+ "gold_session_ids": [
+ "answer_f38f679b"
+ ]
+ },
+ {
+ "question_id": "94f70d80",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_c63c0458",
+ "sharegpt_PCqCVmR_18",
+ "96eac086",
+ "sharegpt_ZvjRGRN_0",
+ "2479cbae",
+ "ultrachat_526427",
+ "b015c3c3_1",
+ "4e77dbbe_5",
+ "2ae4f277_1",
+ "09b651aa_3"
+ ],
+ "gold_session_ids": [
+ "answer_c63c0458"
+ ]
+ },
+ {
+ "question_id": "66f24dbb",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6309297535714575,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "89452526_2",
+ "answer_fea2e4d3",
+ "798e4ba3_2",
+ "8d015d9d_4",
+ "ca3a4e4f_1",
+ "7a10d327_2",
+ "8d46b8fa_1",
+ "aeb3818c_2",
+ "sharegpt_BvuMJIh_0",
+ "fce8bb8a_2"
+ ],
+ "gold_session_ids": [
+ "answer_fea2e4d3"
+ ]
+ },
+ {
+ "question_id": "ad7109d1",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6309297535714575,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "3dc02d26_2",
+ "answer_679840f8",
+ "22aa7cca_3",
+ "06121f7f_1",
+ "3030efec_2",
+ "7ad0bd25_1",
+ "0982a1fa_5",
+ "1d6744b5_1",
+ "sharegpt_zdDMlRC_0",
+ "02f0738e_4"
+ ],
+ "gold_session_ids": [
+ "answer_679840f8"
+ ]
+ },
+ {
+ "question_id": "af8d2e46",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_82a04f59",
+ "ced213c6",
+ "d5a6f455_2",
+ "43668d77_1",
+ "33c3b457",
+ "7f61409f",
+ "f75238a5_1",
+ "ultrachat_164840",
+ "38e4b795_2",
+ "73d5f10c"
+ ],
+ "gold_session_ids": [
+ "answer_82a04f59"
+ ]
+ },
+ {
+ "question_id": "dccbc061",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_8f276838",
+ "25d1830a_2",
+ "ultrachat_343",
+ "ultrachat_519037",
+ "0b8a4a9d_1",
+ "79bb1298_3",
+ "4af27eab_2",
+ "sharegpt_buGcwO5_33",
+ "sharegpt_MKMWjX0_25",
+ "657ab20c_1"
+ ],
+ "gold_session_ids": [
+ "answer_8f276838"
+ ]
+ },
+ {
+ "question_id": "c8c3f81d",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_761acef8",
+ "19ec83c5_1",
+ "b818e2a1_1",
+ "2b16f3d6_1",
+ "eb403c80_2",
+ "25432e10",
+ "sharegpt_pRqHb1o_0",
+ "sharegpt_6CsjCEy_0",
+ "eb5d5a93_1",
+ "e89b6be2_2"
+ ],
+ "gold_session_ids": [
+ "answer_761acef8"
+ ]
+ },
+ {
+ "question_id": "8ebdbe50",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_8ad8a34f",
+ "90f16f14_2",
+ "b53f447a_1",
+ "39911f94",
+ "27d378b2_3",
+ "03b4d5d8_2",
+ "sharegpt_c0PdB3l_0",
+ "66369fde_4",
+ "sharegpt_M84blrA_53",
+ "407dd888"
+ ],
+ "gold_session_ids": [
+ "answer_8ad8a34f"
+ ]
+ },
+ {
+ "question_id": "6b168ec8",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_e623ae87",
+ "ultrachat_414934",
+ "sharegpt_1kE6haW_52",
+ "afffd8ec_3",
+ "388c5534_1",
+ "32cf110f_1",
+ "443688cc_2",
+ "33dff20c_3",
+ "sharegpt_ba4V2an_2",
+ "518d26d3_5"
+ ],
+ "gold_session_ids": [
+ "answer_e623ae87"
+ ]
+ },
+ {
+ "question_id": "75499fd8",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_723bf11f",
+ "ultrachat_428593",
+ "5328c3c2_2",
+ "533372c8_3",
+ "sharegpt_v11Tpg9_195",
+ "sharegpt_WetAHhj_0",
+ "ddf0f116_3",
+ "sharegpt_u1AM5RT_273",
+ "ultrachat_371964",
+ "ultrachat_466236"
+ ],
+ "gold_session_ids": [
+ "answer_723bf11f"
+ ]
+ },
+ {
+ "question_id": "21436231",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_1e6d4567",
+ "7a7a4bf0_4",
+ "1b2b30d3",
+ "a0f8468c_3",
+ "a3118ef0_1",
+ "4bebc783_2",
+ "d0785ee9_1",
+ "a3058382_4",
+ "sharegpt_DGiqvVv_169",
+ "0d0a89dc_1"
+ ],
+ "gold_session_ids": [
+ "answer_1e6d4567"
+ ]
+ },
+ {
+ "question_id": "95bcc1c8",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_cb742a61",
+ "abeda028",
+ "56266c38_1",
+ "c9763bff",
+ "sharegpt_uATA5m5_41",
+ "ultrachat_320154",
+ "d59a335d",
+ "99e5e380",
+ "396238f9",
+ "ultrachat_203404"
+ ],
+ "gold_session_ids": [
+ "answer_cb742a61"
+ ]
+ },
+ {
+ "question_id": "0862e8bf",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_c6fd8ebd",
+ "sharegpt_MTUnqho_0",
+ "ultrachat_357044",
+ "sharegpt_A7KJApk_21",
+ "sharegpt_2I1TxKW_0",
+ "sharegpt_WhKs0Le_0",
+ "10541a2c_1",
+ "sharegpt_oLlUxYj_40",
+ "ultrachat_240637",
+ "ultrachat_518141"
+ ],
+ "gold_session_ids": [
+ "answer_c6fd8ebd"
+ ]
+ },
+ {
+ "question_id": "853b0a1d",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6309297535714575,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "c70f9f9c",
+ "answer_69811d4a",
+ "sharegpt_n3MgsgL_0",
+ "ultrachat_376940",
+ "sharegpt_hRba933_47",
+ "f71bf532_1",
+ "ultrachat_378413",
+ "sharegpt_F1lIiep_0",
+ "sharegpt_UnjngE7_65",
+ "sharegpt_oh3eExx_8"
+ ],
+ "gold_session_ids": [
+ "answer_69811d4a"
+ ]
+ },
+ {
+ "question_id": "a06e4cfe",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6309297535714575,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "af257b0b_2",
+ "answer_6fe9fb49",
+ "84fb50bb_3",
+ "b66c6397_1",
+ "ultrachat_442201",
+ "6dfb33f1_1",
+ "0c891c0b_1",
+ "de64539a_2",
+ "9de2c795",
+ "15519944_2"
+ ],
+ "gold_session_ids": [
+ "answer_6fe9fb49"
+ ]
+ },
+ {
+ "question_id": "37d43f65",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_55161935",
+ "1cafd864_3",
+ "782c0f84",
+ "3ab79434_2",
+ "ultrachat_415674",
+ "sharegpt_uNuzBLK_0",
+ "sharegpt_Bm91yJg_0",
+ "63587769",
+ "3dcf1e0b_2",
+ "ultrachat_107321"
+ ],
+ "gold_session_ids": [
+ "answer_55161935"
+ ]
+ },
+ {
+ "question_id": "b86304ba",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.3562071871080222,
+ "mrr": 0.16666666666666666,
+ "retrieved_session_ids": [
+ "ultrachat_328696",
+ "e831a29f_1",
+ "65f3c006_1",
+ "sharegpt_E0YL5SX_142",
+ "8e9425ef_1",
+ "answer_645b0329",
+ "ae00973d_4",
+ "sharegpt_xGoJZ6Z_0",
+ "5cc9b056",
+ "ea8bb4f8_2"
+ ],
+ "gold_session_ids": [
+ "answer_645b0329"
+ ]
+ },
+ {
+ "question_id": "d52b4f67",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_0df6aa4b",
+ "d31c3ec8_3",
+ "62a6d083",
+ "cd1f21d0_3",
+ "0840765f_1",
+ "sharegpt_n6Ej34Y_75",
+ "sharegpt_BVfC7JV_0",
+ "f999b05b_5",
+ "ultrachat_481735",
+ "9b17bb98"
+ ],
+ "gold_session_ids": [
+ "answer_0df6aa4b"
+ ]
+ },
+ {
+ "question_id": "25e5aa4f",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_986de8c3",
+ "18807892_4",
+ "ultrachat_225537",
+ "sharegpt_8cFNcVG_0",
+ "sharegpt_6MPnc5F_9",
+ "sharegpt_rqvR1Ep_0",
+ "8953c8c8_1",
+ "ultrachat_373294",
+ "sharegpt_SwdaeDm_15",
+ "1de862d6_2"
+ ],
+ "gold_session_ids": [
+ "answer_986de8c3"
+ ]
+ },
+ {
+ "question_id": "caf9ead2",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_0714183a",
+ "7ae303b7_1",
+ "86974d49",
+ "ultrachat_137778",
+ "cd568521_2",
+ "fcc6d66d_1",
+ "e18a6f0b_4",
+ "58dd3f35",
+ "f8ba3f69",
+ "5e014cd6_2"
+ ],
+ "gold_session_ids": [
+ "answer_0714183a"
+ ]
+ },
+ {
+ "question_id": "8550ddae",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_c8354ae9",
+ "71cf5aeb_3",
+ "a7189b75",
+ "9deebbc2_2",
+ "e10dfffb_1",
+ "c7a15bdc",
+ "3b13915f",
+ "d2cb1f8c",
+ "ultrachat_365941",
+ "38bbc2b7_2"
+ ],
+ "gold_session_ids": [
+ "answer_c8354ae9"
+ ]
+ },
+ {
+ "question_id": "60d45044",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_9cddca88",
+ "a76e7e3c_2",
+ "fec059c5",
+ "446173bb_1",
+ "26bc645b_5",
+ "f90c44e6_2",
+ "af4d4ecd_2",
+ "0c0f2449_1",
+ "de34d3f7",
+ "fcc6d66d_1"
+ ],
+ "gold_session_ids": [
+ "answer_9cddca88"
+ ]
+ },
+ {
+ "question_id": "3f1e9474",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_57fc1954",
+ "205f348b_2",
+ "e7497c07_2",
+ "ultrachat_393184",
+ "ultrachat_477598",
+ "sharegpt_hl7jQnw_0",
+ "ultrachat_489379",
+ "ultrachat_177477",
+ "3aac691d_2",
+ "sharegpt_5clkQjb_12"
+ ],
+ "gold_session_ids": [
+ "answer_57fc1954"
+ ]
+ },
+ {
+ "question_id": "86b68151",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6309297535714575,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "0abaca24_1",
+ "answer_dc11c1eb",
+ "4e671700",
+ "sharegpt_beV15ZO_0",
+ "d1bd26fb_2",
+ "sharegpt_rnL5VQO_0",
+ "d8454317_4",
+ "sharegpt_KdbXhTW_13",
+ "ultrachat_182718",
+ "159ab103"
+ ],
+ "gold_session_ids": [
+ "answer_dc11c1eb"
+ ]
+ },
+ {
+ "question_id": "577d4d32",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.3333333333333333,
+ "mrr": 0.14285714285714285,
+ "retrieved_session_ids": [
+ "6e2cca63_2",
+ "f60245ed",
+ "ultrachat_125084",
+ "dc700e27",
+ "26f9a255_1",
+ "2ef53f61_2",
+ "answer_0dd4d99a",
+ "sharegpt_101i5OR_0",
+ "b283f9eb_1",
+ "sharegpt_1ByFFwm_0"
+ ],
+ "gold_session_ids": [
+ "answer_0dd4d99a"
+ ]
+ },
+ {
+ "question_id": "ec81a493",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ed1982fc",
+ "6f55019d",
+ "ab4643a2_4",
+ "616fdd27",
+ "ultrachat_27488",
+ "90fea364_3",
+ "ffbd49a4",
+ "bb2180e9_1",
+ "d22a462a_2",
+ "ultrachat_74904"
+ ],
+ "gold_session_ids": [
+ "answer_ed1982fc"
+ ]
+ },
+ {
+ "question_id": "15745da0",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_586de428",
+ "07d33eaa",
+ "ultrachat_446016",
+ "1f5370f7_4",
+ "0d73afc8",
+ "ultrachat_155656",
+ "1c662b7b_1",
+ "768544e1",
+ "c1f03307_2",
+ "4cbb6dd2"
+ ],
+ "gold_session_ids": [
+ "answer_586de428"
+ ]
+ },
+ {
+ "question_id": "e01b8e2f",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_5ca6cd28",
+ "97338ddd_1",
+ "86c5d31d",
+ "8acfa731",
+ "b2543a58_3",
+ "1aaf8057_3",
+ "d31c3ec8_1",
+ "8be2c3f1",
+ "sharegpt_4ANhVKP_0",
+ "d4cdb464"
+ ],
+ "gold_session_ids": [
+ "answer_5ca6cd28"
+ ]
+ },
+ {
+ "question_id": "bc8a6e93",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_e6143162",
+ "ultrachat_453950",
+ "b869d7ed_2",
+ "ultrachat_184007",
+ "a1b3dd27_1",
+ "6f8c03ab",
+ "f62bfe2b_1",
+ "05594106_1",
+ "1cb7a6c4_4",
+ "sharegpt_W9DAhiE_0"
+ ],
+ "gold_session_ids": [
+ "answer_e6143162"
+ ]
+ },
+ {
+ "question_id": "ccb36322",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_f1fbb330",
+ "facfdfe2",
+ "bdcee74e_3",
+ "162ff451",
+ "04a0b385",
+ "06d1a1a0",
+ "940629b2_3",
+ "28a45124",
+ "ultrachat_203880",
+ "46d9d476"
+ ],
+ "gold_session_ids": [
+ "answer_f1fbb330"
+ ]
+ },
+ {
+ "question_id": "001be529",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_530960c1",
+ "sharegpt_6cz1Sq6_328",
+ "sharegpt_L5SCdrp_0",
+ "dffe8d2d_2",
+ "f067b0ee",
+ "752a01fe_3",
+ "a244d459",
+ "de877349",
+ "2c1537ff",
+ "ultrachat_241736"
+ ],
+ "gold_session_ids": [
+ "answer_530960c1"
+ ]
+ },
+ {
+ "question_id": "b320f3f8",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_5cc9b056",
+ "aae4411b_2",
+ "b89f68c4_4",
+ "sharegpt_4FCMpJR_0",
+ "1b71c896_2",
+ "5e4bb245_2",
+ "66ffbb8b_1",
+ "e28f75f4",
+ "sharegpt_7QD86cC_20",
+ "ultrachat_562554"
+ ],
+ "gold_session_ids": [
+ "answer_5cc9b056"
+ ]
+ },
+ {
+ "question_id": "19b5f2b3",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_5ff494b9",
+ "ultrachat_302261",
+ "sharegpt_I0Wgney_11",
+ "ultrachat_447485",
+ "ultrachat_138083",
+ "275804a8_3",
+ "ultrachat_8325",
+ "5214491c",
+ "51adc57e_2",
+ "fbe6fa2c_1"
+ ],
+ "gold_session_ids": [
+ "answer_5ff494b9"
+ ]
+ },
+ {
+ "question_id": "4fd1909e",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_2952aee4",
+ "ad2135af",
+ "ultrachat_186237",
+ "94c582bb_2",
+ "ultrachat_472809",
+ "502b6b52_2",
+ "f716720c_1",
+ "ultrachat_162975",
+ "65bc93cf_7",
+ "2479cbae"
+ ],
+ "gold_session_ids": [
+ "answer_2952aee4"
+ ]
+ },
+ {
+ "question_id": "545bd2b5",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6309297535714575,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "b7f69a7d_1",
+ "answer_47ffab4c",
+ "d1a1b9ea_1",
+ "1351505a_1",
+ "32615aa6_3",
+ "6e672b84_2",
+ "700c16e2_1",
+ "9f9384ed_1",
+ "10889cf0_1",
+ "cc021f81_2"
+ ],
+ "gold_session_ids": [
+ "answer_47ffab4c"
+ ]
+ },
+ {
+ "question_id": "8a137a7f",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_15d63a22",
+ "ac583170_2",
+ "e3e66b50",
+ "ultrachat_293136",
+ "sharegpt_tR1SJyb_0",
+ "ultrachat_44939",
+ "e2cb1647",
+ "sharegpt_CMbaRkD_37",
+ "77908e43_1",
+ "sharegpt_o1SCEXR_23"
+ ],
+ "gold_session_ids": [
+ "answer_15d63a22"
+ ]
+ },
+ {
+ "question_id": "76d63226",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.3562071871080222,
+ "mrr": 0.16666666666666666,
+ "retrieved_session_ids": [
+ "cb289226_1",
+ "50136b31",
+ "6a2f9452_2",
+ "ultrachat_180710",
+ "sharegpt_Wi7Op1u_91",
+ "answer_bbdc7b0a",
+ "11566271_2",
+ "sharegpt_sRhcMiu_13",
+ "sharegpt_jsjbgCR_0",
+ "372544ba"
+ ],
+ "gold_session_ids": [
+ "answer_bbdc7b0a"
+ ]
+ },
+ {
+ "question_id": "86f00804",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_96b8c9ee",
+ "6e672b84_3",
+ "35c5419d_abs_3",
+ "sharegpt_sCFErnY_22",
+ "sharegpt_BAHXiar_14",
+ "sharegpt_ErOTMZ3_277",
+ "6339aa1d_2",
+ "sharegpt_K3jwzdd_0",
+ "830c739e",
+ "cf9010c2_1"
+ ],
+ "gold_session_ids": [
+ "answer_96b8c9ee"
+ ]
+ },
+ {
+ "question_id": "8e9d538c",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_7bdcbd23",
+ "27d18d38",
+ "ultrachat_570726",
+ "2deed26f",
+ "4b5568a5",
+ "sharegpt_MoCDhww_0",
+ "ultrachat_185241",
+ "sharegpt_cADPA4R_0",
+ "sharegpt_tb5H6IH_3",
+ "ultrachat_419107"
+ ],
+ "gold_session_ids": [
+ "answer_7bdcbd23"
+ ]
+ },
+ {
+ "question_id": "311778f1",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_e40b054e",
+ "sharegpt_vXNQZ2I_0",
+ "4fe91719_1",
+ "8e6e058b_3",
+ "5db56d24_2",
+ "31903f54",
+ "414b9122",
+ "ultrachat_316997",
+ "ultrachat_514341",
+ "sharegpt_1kE6haW_276"
+ ],
+ "gold_session_ids": [
+ "answer_e40b054e"
+ ]
+ },
+ {
+ "question_id": "c19f7a0b",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_f442ccbe",
+ "9530353d_1",
+ "58ab5fc5",
+ "sharegpt_6ZeW9Vb_0",
+ "7bc2c7a3_1",
+ "ultrachat_352431",
+ "c9292210_2",
+ "5558a42e_1",
+ "ultrachat_80803",
+ "sharegpt_RjNU9sC_20"
+ ],
+ "gold_session_ids": [
+ "answer_f442ccbe"
+ ]
+ },
+ {
+ "question_id": "4100d0a0",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_83c13ff9",
+ "8d068598",
+ "eb888fcf_1",
+ "sharegpt_8w0MhIA_0",
+ "22a97fec",
+ "sharegpt_eV0krsR_0",
+ "sharegpt_Viy08ou_0",
+ "sharegpt_t3f69h7_23",
+ "4fd958dd_2",
+ "2b86336d_1"
+ ],
+ "gold_session_ids": [
+ "answer_83c13ff9"
+ ]
+ },
+ {
+ "question_id": "29f2956b",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_7cc5362f",
+ "00842aad_2",
+ "90f7041a_3",
+ "46548fab_3",
+ "a79db993",
+ "88361fdd",
+ "6d6ffac5",
+ "715b1b28",
+ "c4ed2287_3",
+ "d50a8a33_1"
+ ],
+ "gold_session_ids": [
+ "answer_7cc5362f"
+ ]
+ },
+ {
+ "question_id": "1faac195",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6309297535714575,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "b2543a58_3",
+ "answer_d01949bf",
+ "d2905eb6_2",
+ "2def525b_2",
+ "f2ccf83b",
+ "94f460be_1",
+ "sharegpt_wgTTTPq_0",
+ "bcf6953d_3",
+ "75f97838_1",
+ "bce91ca6"
+ ],
+ "gold_session_ids": [
+ "answer_d01949bf"
+ ]
+ },
+ {
+ "question_id": "faba32e5",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_39b12014",
+ "sharegpt_3v69R59_0",
+ "770d11f5",
+ "83cfab9f_2",
+ "sharegpt_benxw0S_9",
+ "f0ecaea5_2",
+ "69f3fc12_1",
+ "sharegpt_jxsIY6L_0",
+ "70323809_2",
+ "sharegpt_PdnvIns_0"
+ ],
+ "gold_session_ids": [
+ "answer_39b12014"
+ ]
+ },
+ {
+ "question_id": "f4f1d8a4",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.43067655807339306,
+ "mrr": 0.25,
+ "retrieved_session_ids": [
+ "3aac691d_2",
+ "76ea88ba_2",
+ "eb682e52_2",
+ "answer_f5b33470",
+ "6f051087_2",
+ "64934c76",
+ "5944b36a_2",
+ "04d84aa8_2",
+ "sharegpt_pX51ZUz_0",
+ "e45db473_3"
+ ],
+ "gold_session_ids": [
+ "answer_f5b33470"
+ ]
+ },
+ {
+ "question_id": "c14c00dd",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6309297535714575,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "c77250d2",
+ "answer_304511ce",
+ "5dac7cc2_1",
+ "sharegpt_XCNeusQ_9",
+ "sharegpt_zs9nI2N_0",
+ "f91cac51_1",
+ "cd1f5147",
+ "sharegpt_vXWOUqx_11",
+ "sharegpt_gU0qvZu_0",
+ "sharegpt_FfqJ2xI_0"
+ ],
+ "gold_session_ids": [
+ "answer_304511ce"
+ ]
+ },
+ {
+ "question_id": "36580ce8",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.5,
+ "mrr": 0.3333333333333333,
+ "retrieved_session_ids": [
+ "ultrachat_149170",
+ "ultrachat_32917",
+ "answer_93e1bd22",
+ "ultrachat_270984",
+ "ultrachat_318891",
+ "ultrachat_53784",
+ "sharegpt_5L3F069_69",
+ "2aee3e70",
+ "2865bf22_2",
+ "526354c8_2"
+ ],
+ "gold_session_ids": [
+ "answer_93e1bd22"
+ ]
+ },
+ {
+ "question_id": "3d86fd0a",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_19c24c11",
+ "cf8e352f_1",
+ "06d1a1a0",
+ "6227a491_2",
+ "5a13d047_1",
+ "7e4dab66_2",
+ "ultrachat_394611",
+ "1c1a2b7f_2",
+ "d1d3fd12_2",
+ "4ff568ba_2"
+ ],
+ "gold_session_ids": [
+ "answer_19c24c11"
+ ]
+ },
+ {
+ "question_id": "a82c026e",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.3562071871080222,
+ "mrr": 0.16666666666666666,
+ "retrieved_session_ids": [
+ "ccaabf0b_1",
+ "752392bb_3",
+ "sharegpt_u71pVhw_0",
+ "daa32134",
+ "1a693999_4",
+ "answer_787e6a6d",
+ "516b0aae_1",
+ "a6c41826_3",
+ "sharegpt_vIkUTvg_0",
+ "c71e1c7a_2"
+ ],
+ "gold_session_ids": [
+ "answer_787e6a6d"
+ ]
+ },
+ {
+ "question_id": "0862e8bf_abs",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6309297535714575,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "f238edb0_1",
+ "answer_c6fd8ebd_abs",
+ "sharegpt_xLtmCeT_0",
+ "sharegpt_StUzdLK_227",
+ "sharegpt_PSjpANO_35",
+ "12cd736c",
+ "3301f749_1",
+ "6a8f8453_2",
+ "ultrachat_460804",
+ "sharegpt_8T6d65A_83"
+ ],
+ "gold_session_ids": [
+ "answer_c6fd8ebd_abs"
+ ]
+ },
+ {
+ "question_id": "15745da0_abs",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_586de428_abs",
+ "375e27b9_2",
+ "565929a2_2",
+ "9b638f21_4",
+ "ultrachat_360529",
+ "7dfe0b9d_1",
+ "sharegpt_bmEgv3O_5",
+ "sharegpt_JSxa86k_19",
+ "be976c82_1",
+ "sharegpt_Ql85pW6_0"
+ ],
+ "gold_session_ids": [
+ "answer_586de428_abs"
+ ]
+ },
+ {
+ "question_id": "bc8a6e93_abs",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_e6143162_abs",
+ "ba944cdc",
+ "375354d1_2",
+ "31f3f40d_4",
+ "sharegpt_hn3IS1q_0",
+ "b3763b6b_2",
+ "9c68c22f",
+ "sharegpt_MimvIAy_181",
+ "de1f4aec_1",
+ "22356a93_4"
+ ],
+ "gold_session_ids": [
+ "answer_e6143162_abs"
+ ]
+ },
+ {
+ "question_id": "19b5f2b3_abs",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_5ff494b9_abs",
+ "ultrachat_159269",
+ "sharegpt_oPOTyid_28",
+ "56cdb8b8_1",
+ "e44a5733_1",
+ "sharegpt_v11Tpg9_483",
+ "d0b0dabe",
+ "f5cf8815_3",
+ "9f8fa5e4_2",
+ "8fa624b2_3"
+ ],
+ "gold_session_ids": [
+ "answer_5ff494b9_abs"
+ ]
+ },
+ {
+ "question_id": "29f2956b_abs",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_7cc5362f_abs",
+ "64b7d9cc_2",
+ "f2fa97f5_3",
+ "503b2761",
+ "ultrachat_13007",
+ "3b8f0d9d",
+ "1516d4c8",
+ "05b84f71_5",
+ "01ecd4fa_3",
+ "ultrachat_70054"
+ ],
+ "gold_session_ids": [
+ "answer_7cc5362f_abs"
+ ]
+ },
+ {
+ "question_id": "f4f1d8a4_abs",
+ "question_type": "single-session-user",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.5,
+ "mrr": 0.3333333333333333,
+ "retrieved_session_ids": [
+ "1dd13331_1",
+ "ea3db78e_2",
+ "answer_f5b33470_abs",
+ "080a4000_2",
+ "f252001e",
+ "56745369_1",
+ "31299b8e_2",
+ "793382f9_2",
+ "sharegpt_n7xJvjp_0",
+ "ultrachat_317545"
+ ],
+ "gold_session_ids": [
+ "answer_f5b33470_abs"
+ ]
+ },
+ {
+ "question_id": "0a995998",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_afa9873b_2",
+ "answer_afa9873b_1",
+ "answer_afa9873b_3",
+ "85846900_2",
+ "25091584_1",
+ "439f9ed8_3",
+ "ultrachat_331531",
+ "fa46a10e",
+ "ad3d0815_1",
+ "64b847a7_1"
+ ],
+ "gold_session_ids": [
+ "answer_afa9873b_2",
+ "answer_afa9873b_3",
+ "answer_afa9873b_1"
+ ]
+ },
+ {
+ "question_id": "6d550036",
+ "question_type": "multi-session",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.123151194370365,
+ "mrr": 0.125,
+ "retrieved_session_ids": [
+ "2e4430d8_2",
+ "sharegpt_zciCXP1_12",
+ "sharegpt_nVi6tIt_0",
+ "0bab76de",
+ "sharegpt_UbMVpdp_93",
+ "sharegpt_J7ZAFLd_0",
+ "dae3906e_2",
+ "answer_ec904b3c_1",
+ "sharegpt_oXgiN7q_53",
+ "25b2b29d"
+ ],
+ "gold_session_ids": [
+ "answer_ec904b3c_1",
+ "answer_ec904b3c_4",
+ "answer_ec904b3c_3",
+ "answer_ec904b3c_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_59c863d7",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9447176050778178,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_593bdffd_4",
+ "answer_593bdffd_1",
+ "answer_593bdffd_2",
+ "eb47739f_2",
+ "61d3fec4_2",
+ "a7d014e4_1",
+ "9ef698bc_2",
+ "sharegpt_Opm4kTV_0",
+ "ultrachat_233858",
+ "answer_593bdffd_3"
+ ],
+ "gold_session_ids": [
+ "answer_593bdffd_4",
+ "answer_593bdffd_1",
+ "answer_593bdffd_3",
+ "answer_593bdffd_2"
+ ]
+ },
+ {
+ "question_id": "b5ef892d",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9325210919548239,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_a8b4290f_1",
+ "answer_a8b4290f_3",
+ "35dcacdc_2",
+ "7294438c",
+ "eb774a95",
+ "answer_a8b4290f_2",
+ "1a44346c_1",
+ "ultrachat_565056",
+ "ultrachat_367439",
+ "dc968254_1"
+ ],
+ "gold_session_ids": [
+ "answer_a8b4290f_3",
+ "answer_a8b4290f_1",
+ "answer_a8b4290f_2"
+ ]
+ },
+ {
+ "question_id": "e831120c",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_86c505e7_2",
+ "answer_86c505e7_1",
+ "8fb3fe3a_1",
+ "63388005",
+ "sharegpt_CROow2Q_0",
+ "a63ad8e3_2",
+ "ba6b67af_2",
+ "sharegpt_sDfu38Q_0",
+ "db6ab13b_1",
+ "773aebbd_4"
+ ],
+ "gold_session_ids": [
+ "answer_86c505e7_1",
+ "answer_86c505e7_2"
+ ]
+ },
+ {
+ "question_id": "3a704032",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_c2204106_1",
+ "answer_c2204106_2",
+ "answer_c2204106_3",
+ "0babf5b6_2",
+ "7712aded",
+ "ultrachat_501733",
+ "014a5d36",
+ "sharegpt_tzJ6yxo_0",
+ "sharegpt_jlDfWk7_17",
+ "be24e3db_2"
+ ],
+ "gold_session_ids": [
+ "answer_c2204106_2",
+ "answer_c2204106_3",
+ "answer_c2204106_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_d84a3211",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9619991470595832,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_2880eb6c_4",
+ "answer_2880eb6c_1",
+ "answer_2880eb6c_2",
+ "18d68208_1",
+ "sharegpt_qHNCIfF_0",
+ "4889cd54_1",
+ "answer_2880eb6c_3",
+ "700cc131_2",
+ "83fa816b",
+ "ultrachat_83526"
+ ],
+ "gold_session_ids": [
+ "answer_2880eb6c_2",
+ "answer_2880eb6c_4",
+ "answer_2880eb6c_1",
+ "answer_2880eb6c_3"
+ ]
+ },
+ {
+ "question_id": "aae3761f",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9674679834891693,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_526354c8_1",
+ "answer_526354c8_2",
+ "38e81260_7",
+ "answer_526354c8_3",
+ "b192ae00_2",
+ "2a86d0da_2",
+ "sharegpt_ynLoh9N_0",
+ "ultrachat_120599",
+ "sharegpt_bKkRqZc_0",
+ "41e778e4_1"
+ ],
+ "gold_session_ids": [
+ "answer_526354c8_1",
+ "answer_526354c8_3",
+ "answer_526354c8_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_f2262a51",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8385465274895063,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_55a6940c_3",
+ "sharegpt_hChsWOp_128",
+ "dd81b163_1",
+ "answer_55a6940c_2",
+ "ultrachat_139167",
+ "answer_55a6940c_1",
+ "ultrachat_109996",
+ "ultrachat_268434",
+ "f2ffaf25",
+ "6f46ba40_1"
+ ],
+ "gold_session_ids": [
+ "answer_55a6940c_3",
+ "answer_55a6940c_1",
+ "answer_55a6940c_2"
+ ]
+ },
+ {
+ "question_id": "dd2973ad",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9197207891481876,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_f9de4602_2",
+ "56266c38_1",
+ "answer_f9de4602_1",
+ "sharegpt_O6NO7Vo_9",
+ "5b8ef613_4",
+ "41743aae_1",
+ "6551be60_3",
+ "bc542ef7_2",
+ "ultrachat_478199",
+ "98c198fb"
+ ],
+ "gold_session_ids": [
+ "answer_f9de4602_2",
+ "answer_f9de4602_1"
+ ]
+ },
+ {
+ "question_id": "c4a1ceb8",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_56d02cab_3",
+ "answer_56d02cab_1",
+ "answer_56d02cab_4",
+ "answer_56d02cab_2",
+ "ultrachat_441000",
+ "40c77045",
+ "1dc2b670_2",
+ "cef33b28_1",
+ "ba9f938b_2",
+ "d0091194"
+ ],
+ "gold_session_ids": [
+ "answer_56d02cab_3",
+ "answer_56d02cab_4",
+ "answer_56d02cab_2",
+ "answer_56d02cab_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_a56e767c",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_cf9e3940_3",
+ "answer_cf9e3940_1",
+ "answer_cf9e3940_2",
+ "d75245ea",
+ "ultrachat_104129",
+ "15998e39_2",
+ "dac54d1e_1",
+ "ultrachat_21691",
+ "88c8df0e_3",
+ "32da578d_2"
+ ],
+ "gold_session_ids": [
+ "answer_cf9e3940_2",
+ "answer_cf9e3940_1",
+ "answer_cf9e3940_3"
+ ]
+ },
+ {
+ "question_id": "6cb6f249",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6934264036172708,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "ultrachat_404221",
+ "answer_a4204937_1",
+ "answer_a4204937_2",
+ "9b6db1a9_2",
+ "67620dbc_2",
+ "32fc393b_2",
+ "sharegpt_JTaoEJT_23",
+ "f0e71553_5",
+ "579b27c1",
+ "sharegpt_WjpyexI_8"
+ ],
+ "gold_session_ids": [
+ "answer_a4204937_2",
+ "answer_a4204937_1"
+ ]
+ },
+ {
+ "question_id": "46a3abf7",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9325210919548239,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_c65042d7_2",
+ "answer_c65042d7_3",
+ "28eb86ab",
+ "2917b130_1",
+ "a95bedca",
+ "answer_c65042d7_1",
+ "sharegpt_v11Tpg9_58",
+ "8acfa731",
+ "e224317f_2",
+ "ultrachat_558017"
+ ],
+ "gold_session_ids": [
+ "answer_c65042d7_3",
+ "answer_c65042d7_2",
+ "answer_c65042d7_1"
+ ]
+ },
+ {
+ "question_id": "36b9f61e",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ef74281f_1",
+ "answer_ef74281f_2",
+ "answer_ef74281f_3",
+ "a4ad48d9",
+ "fc6f9c59",
+ "85cd56c7_1",
+ "sharegpt_Dvl0iLL_0",
+ "3159942d_2",
+ "56911dc5_4",
+ "ultrachat_18673"
+ ],
+ "gold_session_ids": [
+ "answer_ef74281f_2",
+ "answer_ef74281f_1",
+ "answer_ef74281f_3"
+ ]
+ },
+ {
+ "question_id": "28dc39ac",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_8d015d9d_4",
+ "answer_8d015d9d_1",
+ "answer_8d015d9d_5",
+ "answer_8d015d9d_2",
+ "answer_8d015d9d_3",
+ "42bcee92_2",
+ "df003c93_3",
+ "53985dc3",
+ "ultrachat_452868",
+ "5854eebc_1"
+ ],
+ "gold_session_ids": [
+ "answer_8d015d9d_3",
+ "answer_8d015d9d_5",
+ "answer_8d015d9d_2",
+ "answer_8d015d9d_4",
+ "answer_8d015d9d_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_2f8be40d",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_e7b0637e_1",
+ "answer_e7b0637e_2",
+ "answer_e7b0637e_3",
+ "acda6a4e_4",
+ "1d6e01e8",
+ "a3332b4c_2",
+ "81b971b8_2",
+ "ultrachat_217382",
+ "sharegpt_sScldBY_0",
+ "dfe646a7_2"
+ ],
+ "gold_session_ids": [
+ "answer_e7b0637e_2",
+ "answer_e7b0637e_1",
+ "answer_e7b0637e_3"
+ ]
+ },
+ {
+ "question_id": "2e6d26dc",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_fa526fc0_3",
+ "answer_fa526fc0_4",
+ "answer_fa526fc0_1",
+ "answer_fa526fc0_2",
+ "ultrachat_393167",
+ "ultrachat_541982",
+ "ultrachat_121666",
+ "73f14139",
+ "14a520c8_2",
+ "sharegpt_IigLRfw_0"
+ ],
+ "gold_session_ids": [
+ "answer_fa526fc0_4",
+ "answer_fa526fc0_3",
+ "answer_fa526fc0_1",
+ "answer_fa526fc0_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_15e38248",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.7668091220635322,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_8858d9dc_2",
+ "answer_8858d9dc_4",
+ "8c9bc932_2",
+ "263efe64_1",
+ "7784236e_2",
+ "295fcaeb_2",
+ "answer_8858d9dc_1",
+ "1e7f3172_5",
+ "sharegpt_XK3KWPT_0",
+ "e697b2dd_4"
+ ],
+ "gold_session_ids": [
+ "answer_8858d9dc_3",
+ "answer_8858d9dc_1",
+ "answer_8858d9dc_4",
+ "answer_8858d9dc_2"
+ ]
+ },
+ {
+ "question_id": "88432d0a",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.7827141072039429,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_733e443a_3",
+ "1c8832b4_2",
+ "fa460b49_2",
+ "ultrachat_359327",
+ "f777f641",
+ "answer_733e443a_2",
+ "answer_733e443a_4",
+ "answer_733e443a_1",
+ "0934d99a",
+ "0e250059_2"
+ ],
+ "gold_session_ids": [
+ "answer_733e443a_3",
+ "answer_733e443a_4",
+ "answer_733e443a_2",
+ "answer_733e443a_1"
+ ]
+ },
+ {
+ "question_id": "80ec1f4f",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_990c8992_2",
+ "answer_990c8992_3",
+ "answer_990c8992_1",
+ "ultrachat_318441",
+ "b909542d_1",
+ "sharegpt_nzDNOrq_29",
+ "32387419_4",
+ "f987da7e_3",
+ "9a584663_2",
+ "38aec47b_1"
+ ],
+ "gold_session_ids": [
+ "answer_990c8992_2",
+ "answer_990c8992_1",
+ "answer_990c8992_3"
+ ]
+ },
+ {
+ "question_id": "d23cf73b",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9267582364714126,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_5a0d28f8_4",
+ "answer_5a0d28f8_2",
+ "95b7c3cd_1",
+ "9705c90b",
+ "answer_5a0d28f8_1",
+ "answer_5a0d28f8_3",
+ "c328cd4f",
+ "be0962fb_1",
+ "20ea9554_2",
+ "9e21d6ab_2"
+ ],
+ "gold_session_ids": [
+ "answer_5a0d28f8_4",
+ "answer_5a0d28f8_2",
+ "answer_5a0d28f8_3",
+ "answer_5a0d28f8_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_7fce9456",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9896062251871525,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_a679a86a_4",
+ "answer_a679a86a_5",
+ "answer_a679a86a_3",
+ "answer_a679a86a_1",
+ "1da409cd_1",
+ "answer_a679a86a_2",
+ "541ecc45_3",
+ "04328d1c",
+ "288e7d30",
+ "3753e55f_2"
+ ],
+ "gold_session_ids": [
+ "answer_a679a86a_5",
+ "answer_a679a86a_4",
+ "answer_a679a86a_2",
+ "answer_a679a86a_3",
+ "answer_a679a86a_1"
+ ]
+ },
+ {
+ "question_id": "d682f1a2",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9674679834891693,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_c008e5df_3",
+ "answer_c008e5df_1",
+ "9de29219_1",
+ "answer_c008e5df_2",
+ "sharegpt_LFPdgVp_22",
+ "sharegpt_ysMparQ_41",
+ "6e2cca63_8",
+ "sharegpt_WjpyexI_15",
+ "7332a779_2",
+ "sharegpt_JTaoEJT_23"
+ ],
+ "gold_session_ids": [
+ "answer_c008e5df_1",
+ "answer_c008e5df_2",
+ "answer_c008e5df_3"
+ ]
+ },
+ {
+ "question_id": "7024f17c",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_a21f3697_2",
+ "answer_a21f3697_3",
+ "answer_a21f3697_1",
+ "d7281662_2",
+ "385683f0_1",
+ "f8476198",
+ "b5ef63be_2",
+ "8c652eb0_1",
+ "b7f69a7d_1",
+ "445e06ce_2"
+ ],
+ "gold_session_ids": [
+ "answer_a21f3697_1",
+ "answer_a21f3697_2",
+ "answer_a21f3697_3"
+ ]
+ },
+ {
+ "question_id": "gpt4_5501fe77",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_203bf3fa_3",
+ "answer_203bf3fa_1",
+ "answer_203bf3fa_2",
+ "021c0d09_2",
+ "9b083158_2",
+ "ultrachat_163385",
+ "542fa76e_1",
+ "ultrachat_46240",
+ "ultrachat_337168",
+ "ultrachat_43091"
+ ],
+ "gold_session_ids": [
+ "answer_203bf3fa_1",
+ "answer_203bf3fa_3",
+ "answer_203bf3fa_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_2ba83207",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9047172294870751,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_6a3b5c13_2",
+ "33028509_2",
+ "answer_6a3b5c13_1",
+ "answer_6a3b5c13_3",
+ "answer_6a3b5c13_4",
+ "ea336da0",
+ "ultrachat_224880",
+ "41dc5d45_4",
+ "325c2005",
+ "ultrachat_568233"
+ ],
+ "gold_session_ids": [
+ "answer_6a3b5c13_3",
+ "answer_6a3b5c13_1",
+ "answer_6a3b5c13_2",
+ "answer_6a3b5c13_4"
+ ]
+ },
+ {
+ "question_id": "2318644b",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_eaa8e3ef_2",
+ "answer_eaa8e3ef_1",
+ "c4ea8219_1",
+ "ultrachat_135444",
+ "sharegpt_iwcVyKX_0",
+ "7ff89315_3",
+ "6ec23dc5_1",
+ "fc733b20_2",
+ "sharegpt_36VQ7Bq_0",
+ "7d02ef5c_5"
+ ],
+ "gold_session_ids": [
+ "answer_eaa8e3ef_1",
+ "answer_eaa8e3ef_2"
+ ]
+ },
+ {
+ "question_id": "2ce6a0f2",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.7365896932159578,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_901a6763_4",
+ "600077b0_3",
+ "answer_901a6763_1",
+ "e4bacbcd_1",
+ "answer_901a6763_2",
+ "a267b11e",
+ "ultrachat_36844",
+ "f257aed3_1",
+ "ultrachat_192754",
+ "sharegpt_8W46coJ_3"
+ ],
+ "gold_session_ids": [
+ "answer_901a6763_2",
+ "answer_901a6763_4",
+ "answer_901a6763_1",
+ "answer_901a6763_3"
+ ]
+ },
+ {
+ "question_id": "gpt4_d12ceb0e",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6546154994739983,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "73f4798f",
+ "answer_2504635e_1",
+ "0672cd7b_1",
+ "answer_2504635e_2",
+ "f0c6ddb9_2",
+ "d8e33f5c_abs_2",
+ "answer_2504635e_3",
+ "a1166ecc_3",
+ "31dc9d66_3",
+ "33d16079_2"
+ ],
+ "gold_session_ids": [
+ "answer_2504635e_3",
+ "answer_2504635e_2",
+ "answer_2504635e_1"
+ ]
+ },
+ {
+ "question_id": "00ca467f",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9325210919548239,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_39900a0a_2",
+ "answer_39900a0a_3",
+ "7d1de770_2",
+ "10857212_2",
+ "4951c0b9_1",
+ "answer_39900a0a_1",
+ "e0fa8486_2",
+ "4a5ee697_1",
+ "ultrachat_360972",
+ "426e7403_3"
+ ],
+ "gold_session_ids": [
+ "answer_39900a0a_3",
+ "answer_39900a0a_2",
+ "answer_39900a0a_1"
+ ]
+ },
+ {
+ "question_id": "b3c15d39",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6934264036172708,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "sharegpt_gPmunOt_37",
+ "answer_05d808e6_1",
+ "answer_05d808e6_2",
+ "d298714d_1",
+ "sharegpt_inuIr9J_119",
+ "sharegpt_V2j1zkI_0",
+ "7bb01253",
+ "5d4ff8db_1",
+ "6be23caa",
+ "bd00a24d_2"
+ ],
+ "gold_session_ids": [
+ "answer_05d808e6_1",
+ "answer_05d808e6_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_31ff4165",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_02b63d04_1",
+ "answer_02b63d04_3",
+ "answer_02b63d04_4",
+ "answer_02b63d04_5",
+ "answer_02b63d04_2",
+ "63f8f5a8",
+ "2c185a2b_1",
+ "2d0b80f1",
+ "e2393b63_2",
+ "35201d43"
+ ],
+ "gold_session_ids": [
+ "answer_02b63d04_1",
+ "answer_02b63d04_5",
+ "answer_02b63d04_2",
+ "answer_02b63d04_3",
+ "answer_02b63d04_4"
+ ]
+ },
+ {
+ "question_id": "eeda8a6d",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_3e5fea0e_2",
+ "answer_3e5fea0e_1",
+ "sharegpt_Caxpnd3_12",
+ "sharegpt_an5DTCa_0",
+ "cf7aad73_3",
+ "be6eb800",
+ "ultrachat_309574",
+ "sharegpt_JsCcnH4_0",
+ "a2609501_2",
+ "ultrachat_313541"
+ ],
+ "gold_session_ids": [
+ "answer_3e5fea0e_1",
+ "answer_3e5fea0e_2"
+ ]
+ },
+ {
+ "question_id": "2788b940",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9550236580992476,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_8f6b938d_4",
+ "answer_8f6b938d_1",
+ "answer_8f6b938d_2",
+ "ff6e824b_1",
+ "8de5b7cf_2",
+ "sharegpt_dabXXRO_10",
+ "ultrachat_410014",
+ "answer_8f6b938d_3",
+ "69f3386e",
+ "544fe66c_2"
+ ],
+ "gold_session_ids": [
+ "answer_8f6b938d_1",
+ "answer_8f6b938d_3",
+ "answer_8f6b938d_4",
+ "answer_8f6b938d_2"
+ ]
+ },
+ {
+ "question_id": "60bf93ed",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6052602440527057,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "6414f676_4",
+ "answer_e0956e0a_2",
+ "ultrachat_7648",
+ "388c5534_3",
+ "4fe27c66_4",
+ "answer_e0956e0a_1",
+ "0e297c49",
+ "a200b713_2",
+ "3197603c",
+ "ultrachat_259496"
+ ],
+ "gold_session_ids": [
+ "answer_e0956e0a_1",
+ "answer_e0956e0a_2"
+ ]
+ },
+ {
+ "question_id": "9d25d4e0",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9674679834891693,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_fcff2dc4_2",
+ "answer_fcff2dc4_1",
+ "cd568521_3",
+ "answer_fcff2dc4_3",
+ "5f558dee_1",
+ "380a9d4d_1",
+ "464f3821",
+ "9b38bda6_1",
+ "sharegpt_eOKax2Z_0",
+ "sharegpt_hRba933_47"
+ ],
+ "gold_session_ids": [
+ "answer_fcff2dc4_2",
+ "answer_fcff2dc4_1",
+ "answer_fcff2dc4_3"
+ ]
+ },
+ {
+ "question_id": "129d1232",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.5937751258332797,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "sharegpt_YfIEYo1_0",
+ "answer_1de862d6_3",
+ "ultrachat_274599",
+ "sharegpt_V2j1zkI_0",
+ "5e4bb245_2",
+ "69a9211c_2",
+ "answer_1de862d6_2",
+ "ultrachat_88574",
+ "answer_1de862d6_1",
+ "bf3aebdb_1"
+ ],
+ "gold_session_ids": [
+ "answer_1de862d6_1",
+ "answer_1de862d6_3",
+ "answer_1de862d6_2"
+ ]
+ },
+ {
+ "question_id": "60472f9c",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_e7fe8c8b_1",
+ "answer_e7fe8c8b_2",
+ "answer_e7fe8c8b_3",
+ "33da50d0_1",
+ "ultrachat_373250",
+ "b1815b81",
+ "ultrachat_5271",
+ "sharegpt_LQbL4dQ_0",
+ "7b74d32d_3",
+ "sharegpt_paNxxAc_0"
+ ],
+ "gold_session_ids": [
+ "answer_e7fe8c8b_1",
+ "answer_e7fe8c8b_2",
+ "answer_e7fe8c8b_3"
+ ]
+ },
+ {
+ "question_id": "gpt4_194be4b3",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9330795243082165,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_3826dc55_3",
+ "answer_3826dc55_4",
+ "ultrachat_474217",
+ "answer_3826dc55_5",
+ "sharegpt_VO7dTG3_0",
+ "answer_3826dc55_1",
+ "answer_3826dc55_2",
+ "212ce1f4_4",
+ "407031a9_2",
+ "ultrachat_385467"
+ ],
+ "gold_session_ids": [
+ "answer_3826dc55_1",
+ "answer_3826dc55_3",
+ "answer_3826dc55_5",
+ "answer_3826dc55_2",
+ "answer_3826dc55_4"
+ ]
+ },
+ {
+ "question_id": "a9f6b44c",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8854598815714874,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_cc021f81_3",
+ "d4912cd9",
+ "answer_cc021f81_2",
+ "1126be1e_1",
+ "answer_cc021f81_1",
+ "ca896bdc_1",
+ "35bc3132_2",
+ "bb107057_4",
+ "ec8d10c7_2",
+ "fdd0f680_1"
+ ],
+ "gold_session_ids": [
+ "answer_cc021f81_2",
+ "answer_cc021f81_3",
+ "answer_cc021f81_1"
+ ]
+ },
+ {
+ "question_id": "d851d5ba",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_5cdf9bd2_1",
+ "answer_5cdf9bd2_3",
+ "answer_5cdf9bd2_2",
+ "answer_5cdf9bd2_4",
+ "70764af1_4",
+ "68ace657_2",
+ "2a28999b",
+ "sharegpt_dxirwR4_25",
+ "561f3a6a_2",
+ "47183c60_3"
+ ],
+ "gold_session_ids": [
+ "answer_5cdf9bd2_2",
+ "answer_5cdf9bd2_1",
+ "answer_5cdf9bd2_3",
+ "answer_5cdf9bd2_4"
+ ]
+ },
+ {
+ "question_id": "5a7937c8",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_4cef8a3c_2",
+ "answer_4cef8a3c_1",
+ "answer_4cef8a3c_3",
+ "63c9e87c",
+ "9bd49ce9_1",
+ "40d799e8_2",
+ "55cc1ba3_1",
+ "ultrachat_288747",
+ "ultrachat_564068",
+ "df54d2be"
+ ],
+ "gold_session_ids": [
+ "answer_4cef8a3c_3",
+ "answer_4cef8a3c_1",
+ "answer_4cef8a3c_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_ab202e7f",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8539316501572937,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_728deb4d_5",
+ "answer_728deb4d_3",
+ "answer_728deb4d_2",
+ "dcafb5b3_5",
+ "answer_728deb4d_4",
+ "a08fbe88_1",
+ "29bc69b3",
+ "sharegpt_nc62Spr_7",
+ "10541a2c_2",
+ "ultrachat_464603"
+ ],
+ "gold_session_ids": [
+ "answer_728deb4d_5",
+ "answer_728deb4d_2",
+ "answer_728deb4d_3",
+ "answer_728deb4d_1",
+ "answer_728deb4d_4"
+ ]
+ },
+ {
+ "question_id": "gpt4_e05b82a6",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9550236580992476,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_6350aa4f_2",
+ "answer_6350aa4f_4",
+ "answer_6350aa4f_3",
+ "386f6df9",
+ "ultrachat_43272",
+ "aed6b1b6_1",
+ "6a0d566a_2",
+ "answer_6350aa4f_1",
+ "503791f7_1",
+ "88e01130_1"
+ ],
+ "gold_session_ids": [
+ "answer_6350aa4f_1",
+ "answer_6350aa4f_2",
+ "answer_6350aa4f_3",
+ "answer_6350aa4f_4"
+ ]
+ },
+ {
+ "question_id": "gpt4_731e37d7",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_826d51da_1",
+ "answer_826d51da_4",
+ "answer_826d51da_3",
+ "answer_826d51da_2",
+ "7e4aa7c2_1",
+ "a93750ef_1",
+ "b52f03a1_2",
+ "ultrachat_109542",
+ "267a912c",
+ "sharegpt_vonEwUo_17"
+ ],
+ "gold_session_ids": [
+ "answer_826d51da_3",
+ "answer_826d51da_4",
+ "answer_826d51da_2",
+ "answer_826d51da_1"
+ ]
+ },
+ {
+ "question_id": "edced276",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_60e8941a_2",
+ "answer_60e8941a_1",
+ "1da409cd_1",
+ "ultrachat_475154",
+ "ultrachat_421943",
+ "sharegpt_IFLajLc_0",
+ "d89f6204",
+ "59a6150d_3",
+ "ce1abc42_1",
+ "f584ba36_2"
+ ],
+ "gold_session_ids": [
+ "answer_60e8941a_2",
+ "answer_60e8941a_1"
+ ]
+ },
+ {
+ "question_id": "10d9b85a",
+ "question_type": "multi-session",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.21840743681816419,
+ "mrr": 0.16666666666666666,
+ "retrieved_session_ids": [
+ "c51583cd_3",
+ "84889496_1",
+ "4e59fb02_2",
+ "d576152e_1",
+ "ultrachat_143866",
+ "answer_e0585cb5_2",
+ "cbd1fe79_2",
+ "45823393_2",
+ "3c8e7c0e_4",
+ "aa6afba8"
+ ],
+ "gold_session_ids": [
+ "answer_e0585cb5_2",
+ "answer_e0585cb5_1"
+ ]
+ },
+ {
+ "question_id": "e3038f8c",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6084170171049966,
+ "mrr": 0.3333333333333333,
+ "retrieved_session_ids": [
+ "a3d8e134_2",
+ "85846900_2",
+ "answer_b6018747_3",
+ "sharegpt_hgGAUvu_0",
+ "answer_b6018747_2",
+ "answer_b6018747_1",
+ "e3d4f89e_3",
+ "answer_b6018747_4",
+ "f2df24cd_4",
+ "sharegpt_WxCMnMO_5"
+ ],
+ "gold_session_ids": [
+ "answer_b6018747_2",
+ "answer_b6018747_4",
+ "answer_b6018747_1",
+ "answer_b6018747_3"
+ ]
+ },
+ {
+ "question_id": "2b8f3739",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_23759615_1",
+ "answer_23759615_2",
+ "answer_23759615_3",
+ "e9fb10e7_2",
+ "ce6cae1b_3",
+ "c075835c",
+ "ec8691f5",
+ "2154c810_1",
+ "67d6f18f_1",
+ "285525b2_1"
+ ],
+ "gold_session_ids": [
+ "answer_23759615_2",
+ "answer_23759615_3",
+ "answer_23759615_1"
+ ]
+ },
+ {
+ "question_id": "1a8a66a6",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.5855700749881525,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_2bd23659_1",
+ "d6792df8",
+ "answer_2bd23659_3",
+ "22eebf8a_2",
+ "5eed6833_2",
+ "f829b530_3",
+ "b01aafcb_2",
+ "ebb5d262",
+ "ultrachat_310819",
+ "sharegpt_lsiJM0Y_0"
+ ],
+ "gold_session_ids": [
+ "answer_2bd23659_3",
+ "answer_2bd23659_2",
+ "answer_2bd23659_4",
+ "answer_2bd23659_1"
+ ]
+ },
+ {
+ "question_id": "c2ac3c61",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_923c0221_2",
+ "answer_923c0221_1",
+ "ultrachat_412857",
+ "6009ba4e_5",
+ "9f803df5_1",
+ "sharegpt_z6pSZ19_0",
+ "8e4e215b",
+ "80dc42f7_1",
+ "ultrachat_46891",
+ "sharegpt_CQ3HZO9_53"
+ ],
+ "gold_session_ids": [
+ "answer_923c0221_1",
+ "answer_923c0221_2"
+ ]
+ },
+ {
+ "question_id": "bf659f65",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8278123145308894,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_7726e7e9_1",
+ "f2835879_2",
+ "ultrachat_365864",
+ "answer_7726e7e9_2",
+ "258645ba_3",
+ "sharegpt_inuIr9J_119",
+ "answer_7726e7e9_3",
+ "ultrachat_167088",
+ "71619051_1",
+ "32833e0b_3"
+ ],
+ "gold_session_ids": [
+ "answer_7726e7e9_1",
+ "answer_7726e7e9_3",
+ "answer_7726e7e9_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_372c3eed",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9134015924715543,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_35c5419d_1",
+ "answer_35c5419d_3",
+ "sharegpt_DRsWWMJ_0",
+ "8d969f3e_3",
+ "8dd672a3_3",
+ "sharegpt_6VqlSoL_25",
+ "sharegpt_6dPUCcK_69",
+ "answer_35c5419d_2",
+ "9a22ae03",
+ "sharegpt_2hMUduQ_1"
+ ],
+ "gold_session_ids": [
+ "answer_35c5419d_3",
+ "answer_35c5419d_2",
+ "answer_35c5419d_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_2f91af09",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_669318cf_3",
+ "answer_669318cf_2",
+ "answer_669318cf_1",
+ "51c82c8e_2",
+ "sharegpt_d7bP5fb_0",
+ "22b3af37_1",
+ "ultrachat_296153",
+ "49b78a55_1",
+ "sharegpt_YmzANeT_0",
+ "e7e416f5_2"
+ ],
+ "gold_session_ids": [
+ "answer_669318cf_2",
+ "answer_669318cf_1",
+ "answer_669318cf_3"
+ ]
+ },
+ {
+ "question_id": "81507db6",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8495981447350542,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_da3c1266_5",
+ "9071ba70_3",
+ "answer_da3c1266_2",
+ "ultrachat_254248",
+ "2ca99347_2",
+ "answer_da3c1266_3",
+ "answer_da3c1266_4",
+ "answer_da3c1266_1",
+ "0f6b9698_2",
+ "sharegpt_Caxpnd3_9"
+ ],
+ "gold_session_ids": [
+ "answer_da3c1266_3",
+ "answer_da3c1266_4",
+ "answer_da3c1266_1",
+ "answer_da3c1266_5",
+ "answer_da3c1266_2"
+ ]
+ },
+ {
+ "question_id": "88432d0a_abs",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9349366583346498,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_733e443a_abs_3",
+ "answer_733e443a_abs_1",
+ "752b10e6_2",
+ "answer_733e443a_abs_4",
+ "b2ecce8d_1",
+ "31ca5871_1",
+ "answer_733e443a_abs_2",
+ "8e5b0424",
+ "11f043cd_2",
+ "sharegpt_conu8MS_14"
+ ],
+ "gold_session_ids": [
+ "answer_733e443a_abs_1",
+ "answer_733e443a_abs_3",
+ "answer_733e443a_abs_2",
+ "answer_733e443a_abs_4"
+ ]
+ },
+ {
+ "question_id": "80ec1f4f_abs",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_990c8992_abs_1",
+ "answer_990c8992_abs_3",
+ "answer_990c8992_abs_2",
+ "b341cf37_2",
+ "b304b283_1",
+ "sharegpt_7uSzi6w_0",
+ "7eda4f2a_1",
+ "ultrachat_169512",
+ "ultrachat_330542",
+ "sharegpt_Xf63XgQ_0"
+ ],
+ "gold_session_ids": [
+ "answer_990c8992_abs_2",
+ "answer_990c8992_abs_3",
+ "answer_990c8992_abs_1"
+ ]
+ },
+ {
+ "question_id": "eeda8a6d_abs",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_3e5fea0e_abs_2",
+ "answer_3e5fea0e_abs_1",
+ "9530353d_2",
+ "d15d2899_4",
+ "44ba5b71_2",
+ "c8cc60d6_1",
+ "628c958c_5",
+ "da71721f",
+ "ultrachat_124888",
+ "sharegpt_l7gyE1o_17"
+ ],
+ "gold_session_ids": [
+ "answer_3e5fea0e_abs_2",
+ "answer_3e5fea0e_abs_1"
+ ]
+ },
+ {
+ "question_id": "60bf93ed_abs",
+ "question_type": "multi-session",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.18457569677956817,
+ "mrr": 0.1111111111111111,
+ "retrieved_session_ids": [
+ "cdf068b1_3",
+ "a95d014c_3",
+ "1e91cdf0",
+ "sharegpt_6cz1Sq6_130",
+ "6cd203f7_2",
+ "0944933f_1",
+ "755c2d32_1",
+ "9a6c8d3c",
+ "answer_e0956e0a_abs_2",
+ "36d5bbde_1"
+ ],
+ "gold_session_ids": [
+ "answer_e0956e0a_abs_2",
+ "answer_e0956e0a_abs_1"
+ ]
+ },
+ {
+ "question_id": "edced276_abs",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_60e8941a_abs_1",
+ "answer_60e8941a_abs_2",
+ "a7a64da9_3",
+ "66138e8d_1",
+ "76a6b5fb_1",
+ "8b8e7cf8",
+ "8e8288b3",
+ "sharegpt_u1AM5RT_88",
+ "b68a96c0_2",
+ "7a2bdd15_2"
+ ],
+ "gold_session_ids": [
+ "answer_60e8941a_abs_1",
+ "answer_60e8941a_abs_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_372c3eed_abs",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8278123145308894,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_35c5419d_abs_1",
+ "b0a9de7f_1",
+ "31e254b5",
+ "answer_35c5419d_abs_3",
+ "ultrachat_563740",
+ "85dec0b4_1",
+ "answer_35c5419d_abs_2",
+ "f002a741_2",
+ "sharegpt_xEpi2SM_55",
+ "sharegpt_JuqFCEz_71"
+ ],
+ "gold_session_ids": [
+ "answer_35c5419d_abs_3",
+ "answer_35c5419d_abs_1",
+ "answer_35c5419d_abs_2"
+ ]
+ },
+ {
+ "question_id": "8a2466db",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_edb03329",
+ "6dcf5fa0_1",
+ "b67748d1_1",
+ "6a5b5a78",
+ "e3cc9931_1",
+ "3392c0c7",
+ "sharegpt_osTHjYi_0",
+ "3d0c9f89_1",
+ "6be54739_3",
+ "898bef66"
+ ],
+ "gold_session_ids": [
+ "answer_edb03329"
+ ]
+ },
+ {
+ "question_id": "06878be2",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_555dfb94",
+ "3e9fce53_1",
+ "ca929779_1",
+ "a6074da9_1",
+ "e58109ed_1",
+ "465e7ef9",
+ "be1c6872",
+ "a67c1862_4",
+ "sharegpt_cn3iUbp_7",
+ "9863a1c8_3"
+ ],
+ "gold_session_ids": [
+ "answer_555dfb94"
+ ]
+ },
+ {
+ "question_id": "75832dbd",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.3562071871080222,
+ "mrr": 0.16666666666666666,
+ "retrieved_session_ids": [
+ "sharegpt_xni9yYO_6",
+ "c51583cd_1",
+ "47ec1674_2",
+ "6172cc72_1",
+ "d49b1a24_1",
+ "answer_d87a6ef8",
+ "be753d18",
+ "ultrachat_165893",
+ "c7270a1b_1",
+ "c884b67a_1"
+ ],
+ "gold_session_ids": [
+ "answer_d87a6ef8"
+ ]
+ },
+ {
+ "question_id": "0edc2aef",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.5,
+ "mrr": 0.3333333333333333,
+ "retrieved_session_ids": [
+ "8464304d_2",
+ "f20e72e4_1",
+ "answer_d586e9cd",
+ "08ca1f31_2",
+ "3c3a9042_3",
+ "c3023a45_1",
+ "6a747f2e",
+ "c6c3a982_1",
+ "53dc1394",
+ "ultrachat_166065"
+ ],
+ "gold_session_ids": [
+ "answer_d586e9cd"
+ ]
+ },
+ {
+ "question_id": "35a27287",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.43067655807339306,
+ "mrr": 0.25,
+ "retrieved_session_ids": [
+ "58ae7c4f_3",
+ "880ee13e",
+ "a98ff421_1",
+ "answer_9b182436",
+ "02f2ab8e_2",
+ "f863a648_3",
+ "d20be4f3_1",
+ "203bf3fa_2",
+ "ultrachat_267380",
+ "c9b1c309_4"
+ ],
+ "gold_session_ids": [
+ "answer_9b182436"
+ ]
+ },
+ {
+ "question_id": "32260d93",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6309297535714575,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "573dc24b_1",
+ "answer_0250ae1c",
+ "6d3f5c68_4",
+ "b84b7a2b_2",
+ "1b26bdd5_5",
+ "04bef53b",
+ "d6bbe94d_3",
+ "2d6a38fe_1",
+ "5ace29d7_1",
+ "a33cf36a_1"
+ ],
+ "gold_session_ids": [
+ "answer_0250ae1c"
+ ]
+ },
+ {
+ "question_id": "195a1a1b",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.3333333333333333,
+ "mrr": 0.14285714285714285,
+ "retrieved_session_ids": [
+ "89a1800c_2",
+ "87fff4b4_1",
+ "ed9dad6c_2",
+ "9f12daa1_1",
+ "1b71c896_2",
+ "ce79a004",
+ "answer_6dc4305e",
+ "sharegpt_WVr7Jiq_0",
+ "d03b6a05_2",
+ "eaebcc81_2"
+ ],
+ "gold_session_ids": [
+ "answer_6dc4305e"
+ ]
+ },
+ {
+ "question_id": "afdc33df",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_8549e5e0",
+ "ad8d9f0f",
+ "b6558bfb",
+ "22f9f163_3",
+ "d03b6a05_3",
+ "d5128b51",
+ "e58109ed_1",
+ "96b6e5e8_2",
+ "43b53ac4_1",
+ "afa0c2e7_2"
+ ],
+ "gold_session_ids": [
+ "answer_8549e5e0"
+ ]
+ },
+ {
+ "question_id": "caf03d32",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_2fc6aabb",
+ "bd6d0687",
+ "ec806228",
+ "a52e072b",
+ "af631aa3_1",
+ "6f0e184f",
+ "sharegpt_kLAPfLd_0",
+ "0bab76de",
+ "sharegpt_3D7Qpuq_14",
+ "4b1af06f_1"
+ ],
+ "gold_session_ids": [
+ "answer_2fc6aabb"
+ ]
+ },
+ {
+ "question_id": "54026fce",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_f7b22c66",
+ "670c49e6",
+ "6e92e417",
+ "e6195229",
+ "1c95e152_1",
+ "1fcb4134_3",
+ "ef5d4aa7_1",
+ "1a9f5723_6",
+ "eb91c46b",
+ "ultrachat_491226"
+ ],
+ "gold_session_ids": [
+ "answer_f7b22c66"
+ ]
+ },
+ {
+ "question_id": "06f04340",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.3010299956639812,
+ "mrr": 0.1111111111111111,
+ "retrieved_session_ids": [
+ "91223fd5_1",
+ "42924d15",
+ "728deb4d_4",
+ "be2cb665",
+ "b459f888_3",
+ "66bfa1db",
+ "8b156015_2",
+ "6e6fbb6b",
+ "answer_92d5f7cd",
+ "de1f4aec_2"
+ ],
+ "gold_session_ids": [
+ "answer_92d5f7cd"
+ ]
+ },
+ {
+ "question_id": "6b7dfb22",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_f6502d0f",
+ "574df152_1",
+ "56859d37",
+ "9ee69ca6_3",
+ "fc69fd44_3",
+ "377f8fd9_1",
+ "1c1f5ccc_4",
+ "8489e4ce_1",
+ "e7b0637e_3",
+ "d8454317_3"
+ ],
+ "gold_session_ids": [
+ "answer_f6502d0f"
+ ]
+ },
+ {
+ "question_id": "1a1907b4",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.3333333333333333,
+ "mrr": 0.14285714285714285,
+ "retrieved_session_ids": [
+ "e42e7876_5",
+ "743f03a1_abs_1",
+ "8e78fa70_1",
+ "ae77c245",
+ "4f3b36d4_5",
+ "4e3da326_1",
+ "answer_719502eb",
+ "9585e0c6_1",
+ "61d3fec4_1",
+ "65b01f7c"
+ ],
+ "gold_session_ids": [
+ "answer_719502eb"
+ ]
+ },
+ {
+ "question_id": "09d032c9",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6309297535714575,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "21ef2d05_1",
+ "answer_b10dce5e",
+ "e8bfacec_2",
+ "af631aa3_2",
+ "c7f7949e_3",
+ "1fe34a92",
+ "87e8ec02",
+ "ultrachat_274584",
+ "54c5a89c_2",
+ "16ebc8f8_4"
+ ],
+ "gold_session_ids": [
+ "answer_b10dce5e"
+ ]
+ },
+ {
+ "question_id": "38146c39",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6309297535714575,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "5806662d_1",
+ "answer_772472c8",
+ "42234f98",
+ "15ed03f0_1",
+ "d22c2f04_5",
+ "28116d8a_1",
+ "5263736d_1",
+ "87aaee77",
+ "467db12a",
+ "381b80a3"
+ ],
+ "gold_session_ids": [
+ "answer_772472c8"
+ ]
+ },
+ {
+ "question_id": "d24813b1",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6309297535714575,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "e08bf81f_4",
+ "answer_7c0ade93",
+ "3124bb28_1",
+ "61a46ff7_3",
+ "c2a34674_1",
+ "084802f9_3",
+ "b7e64b5c_1",
+ "446173bb_1",
+ "29e25716_1",
+ "22ce6625_2"
+ ],
+ "gold_session_ids": [
+ "answer_7c0ade93"
+ ]
+ },
+ {
+ "question_id": "57f827a0",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_1bde8d3b",
+ "e85e0eaf_1",
+ "6e110a53_1",
+ "d0f42e3f",
+ "670ca40e",
+ "2ef55f49_1",
+ "f1dd798a_1",
+ "762d288d_2",
+ "ultrachat_18651",
+ "f64c2982_1"
+ ],
+ "gold_session_ids": [
+ "answer_1bde8d3b"
+ ]
+ },
+ {
+ "question_id": "95228167",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_5e613445",
+ "3d2e66f2_2",
+ "5d8c99d3_3",
+ "b916da64_2",
+ "7e17ae56_3",
+ "8199fb3a_4",
+ "084802f9_2",
+ "9f80607d",
+ "57d6e39b_2",
+ "c34b6a1c_1"
+ ],
+ "gold_session_ids": [
+ "answer_5e613445"
+ ]
+ },
+ {
+ "question_id": "505af2f5",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_f3164f2c",
+ "4f2d6be4_2",
+ "9216c4d3_4",
+ "af631aa3_1",
+ "sharegpt_i59HUGf_0",
+ "66bf4b03_1",
+ "sharegpt_kjeGJvK_11",
+ "0e193841_8",
+ "7d52ca48_2",
+ "ebeb4c51"
+ ],
+ "gold_session_ids": [
+ "answer_f3164f2c"
+ ]
+ },
+ {
+ "question_id": "75f70248",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_8ee04a2e",
+ "2d74df23_4",
+ "6ac43c5b_4",
+ "95c9666f_5",
+ "adb1f904",
+ "c6ec5d11",
+ "a394f6b5_1",
+ "sharegpt_KdbXhTW_13",
+ "ultrachat_32122",
+ "c1d13b9e"
+ ],
+ "gold_session_ids": [
+ "answer_8ee04a2e"
+ ]
+ },
+ {
+ "question_id": "d6233ab6",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 0,
+ "recall_any_at_20": 0,
+ "ndcg_at_10": 0,
+ "mrr": 0,
+ "retrieved_session_ids": [
+ "e419b7c3_4",
+ "2c81cdf8_3",
+ "0e726047",
+ "sharegpt_xni9yYO_6",
+ "7af38385_2",
+ "f916c63a_2",
+ "87cfeb28_2",
+ "cf7aad73_3",
+ "32f28c7b_1",
+ "ultrachat_237324"
+ ],
+ "gold_session_ids": [
+ "answer_b0fac439"
+ ]
+ },
+ {
+ "question_id": "1da05512",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_4d3be2ab",
+ "aea71da3_2",
+ "sharegpt_eSTYwMj_14",
+ "f5a2e179_1",
+ "f2565b3b_2",
+ "6de70f46_1",
+ "da1ea197_1",
+ "e4adef2a",
+ "6b93018e_2",
+ "58de8f9b"
+ ],
+ "gold_session_ids": [
+ "answer_4d3be2ab"
+ ]
+ },
+ {
+ "question_id": "fca70973",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6309297535714575,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "fc005760",
+ "answer_a1e169b1",
+ "5d5e80c5_1",
+ "2f2884ad_1",
+ "99b061c5_1",
+ "4e3f0831_1",
+ "312bf939",
+ "f504b269",
+ "09cff8ac",
+ "762a8b45_2"
+ ],
+ "gold_session_ids": [
+ "answer_a1e169b1"
+ ]
+ },
+ {
+ "question_id": "b6025781",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6309297535714575,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "f20c73f5_3",
+ "answer_8414cc57",
+ "2ef53f61_1",
+ "1a693999_4",
+ "612c8368_2",
+ "d15d2899_5",
+ "6446f6e6",
+ "02592f97_1",
+ "89d49715_2",
+ "3a1ea336_1"
+ ],
+ "gold_session_ids": [
+ "answer_8414cc57"
+ ]
+ },
+ {
+ "question_id": "a89d7624",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.43067655807339306,
+ "mrr": 0.25,
+ "retrieved_session_ids": [
+ "a479a7d6_4",
+ "99f2f5b1_1",
+ "082fc07f_1",
+ "answer_8f15ac24",
+ "39a056ae",
+ "ultrachat_30787",
+ "3cb41ab8_4",
+ "ultrachat_383054",
+ "b14d9f4e_2",
+ "168776ed_2"
+ ],
+ "gold_session_ids": [
+ "answer_8f15ac24"
+ ]
+ },
+ {
+ "question_id": "b0479f84",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_30f63ddb",
+ "4f37e5ec",
+ "386f6df9",
+ "79019170_1",
+ "0cf86cbe_2",
+ "4b563ae6_1",
+ "abc2ac45",
+ "fbfda981",
+ "818545a6_2",
+ "ddf41a24"
+ ],
+ "gold_session_ids": [
+ "answer_30f63ddb"
+ ]
+ },
+ {
+ "question_id": "1d4e3b97",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6309297535714575,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "fb2368b8_3",
+ "answer_e6b6353d",
+ "ecb24dd8_2",
+ "809cabca_1",
+ "044e200f_2",
+ "5cd6ab1b",
+ "8c7ae899_2",
+ "03633a51",
+ "c51583cd_3",
+ "fd26f9f4_1"
+ ],
+ "gold_session_ids": [
+ "answer_e6b6353d"
+ ]
+ },
+ {
+ "question_id": "07b6f563",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_d03098f9",
+ "55c267aa",
+ "4bebc782_1",
+ "5e7b1c98_1",
+ "fd18e0dd_2",
+ "66ab6260",
+ "9b2f4dd7_2",
+ "0ac1a6f9_2",
+ "6af947bf_1",
+ "bbccfe79"
+ ],
+ "gold_session_ids": [
+ "answer_d03098f9"
+ ]
+ },
+ {
+ "question_id": "1c0ddc50",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_8da8c7e0",
+ "2f0c1f4e",
+ "2aa70c9c_1",
+ "b33e89b5_1",
+ "c7ca6dff",
+ "a0aa5035",
+ "bdf33c73_2",
+ "0a7d5bf6_2",
+ "2566382f_2",
+ "b0c6fffa"
+ ],
+ "gold_session_ids": [
+ "answer_8da8c7e0"
+ ]
+ },
+ {
+ "question_id": "0a34ad58",
+ "question_type": "single-session-preference",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_cebb7159",
+ "9ca2a195_1",
+ "58ad78d6_2",
+ "90342357_3",
+ "5c0574b1",
+ "b4c5414d_2",
+ "51fe163e_2",
+ "f7e682c3_5",
+ "dd3d8e73_1",
+ "f849dd04"
+ ],
+ "gold_session_ids": [
+ "answer_cebb7159"
+ ]
+ },
+ {
+ "question_id": "d3ab962e",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_5237bb0b_2",
+ "answer_5237bb0b_1",
+ "ultrachat_248741",
+ "3329e5e8_1",
+ "be679e09",
+ "ultrachat_564044",
+ "ultrachat_164834",
+ "aa235649_1",
+ "92f1ea4d_1",
+ "24e44a64"
+ ],
+ "gold_session_ids": [
+ "answer_5237bb0b_2",
+ "answer_5237bb0b_1"
+ ]
+ },
+ {
+ "question_id": "2311e44b",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_bf633415_1",
+ "answer_bf633415_2",
+ "ultrachat_458191",
+ "9b3611d7_1",
+ "1e53c034",
+ "sharegpt_hi2xiJT_14",
+ "eb3eb62b_2",
+ "sharegpt_ELyN256_13",
+ "77753dd9_1",
+ "2714885f_1"
+ ],
+ "gold_session_ids": [
+ "answer_bf633415_2",
+ "answer_bf633415_1"
+ ]
+ },
+ {
+ "question_id": "cc06de0d",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_4fb01417_1",
+ "answer_4fb01417_2",
+ "91c71874_2",
+ "01551a39_2",
+ "4ad63a03_3",
+ "92d74e73",
+ "55033f6f",
+ "53ae82be_3",
+ "9863a1c8_3",
+ "d5eab084_2"
+ ],
+ "gold_session_ids": [
+ "answer_4fb01417_2",
+ "answer_4fb01417_1"
+ ]
+ },
+ {
+ "question_id": "a11281a2",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_c69ee1f9_2",
+ "answer_c69ee1f9_1",
+ "bb2180e9_3",
+ "4b1af06f_2",
+ "ultrachat_55839",
+ "d95e64a7_4",
+ "ultrachat_103691",
+ "sharegpt_u3MVGYG_3",
+ "ultrachat_234016",
+ "7a88956e"
+ ],
+ "gold_session_ids": [
+ "answer_c69ee1f9_2",
+ "answer_c69ee1f9_1"
+ ]
+ },
+ {
+ "question_id": "4f54b7c9",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_50940cb7_2",
+ "answer_50940cb7_1",
+ "dde142b2_1",
+ "22eebf8a_2",
+ "663337ce_1",
+ "sharegpt_NPdQIWB_0",
+ "8a834aee_1",
+ "aafe556c",
+ "b89f68c4_1",
+ "a6c0f032_3"
+ ],
+ "gold_session_ids": [
+ "answer_50940cb7_2",
+ "answer_50940cb7_1"
+ ]
+ },
+ {
+ "question_id": "85fa3a3f",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_dcb18b9b_2",
+ "answer_dcb18b9b_1",
+ "sharegpt_O1gqpsW_0",
+ "a4b223a7_1",
+ "421605fc_1",
+ "989ad9e6_1",
+ "22b3af37_2",
+ "93ff2f73",
+ "0d66562c_1",
+ "e4881cbd_1"
+ ],
+ "gold_session_ids": [
+ "answer_dcb18b9b_2",
+ "answer_dcb18b9b_1"
+ ]
+ },
+ {
+ "question_id": "9aaed6a3",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9197207891481876,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_353d3c6d_2",
+ "c846422a_1",
+ "answer_353d3c6d_1",
+ "3b715074",
+ "c95f2a09_2",
+ "819dfad7_2",
+ "sharegpt_Le00ERV_0",
+ "ultrachat_216636",
+ "sharegpt_HpdNUlz_9",
+ "a861a671_1"
+ ],
+ "gold_session_ids": [
+ "answer_353d3c6d_2",
+ "answer_353d3c6d_1"
+ ]
+ },
+ {
+ "question_id": "1f2b8d4f",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_d8454588_1",
+ "answer_d8454588_2",
+ "b5c147c7_2",
+ "be1c6872",
+ "8d015d9d_5",
+ "9ee4da12_1",
+ "628c958c_1",
+ "ultrachat_401506",
+ "0bd59f15_3",
+ "ultrachat_571561"
+ ],
+ "gold_session_ids": [
+ "answer_d8454588_1",
+ "answer_d8454588_2"
+ ]
+ },
+ {
+ "question_id": "e6041065",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_4eb6d671_2",
+ "answer_4eb6d671_1",
+ "b73a4d04_1",
+ "2a86d0da_2",
+ "bdcee74e_3",
+ "ultrachat_407355",
+ "ec616e7e_4",
+ "49f2e396_2",
+ "4a006b77",
+ "3da98832_2"
+ ],
+ "gold_session_ids": [
+ "answer_4eb6d671_2",
+ "answer_4eb6d671_1"
+ ]
+ },
+ {
+ "question_id": "51c32626",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_58820c75_2",
+ "answer_58820c75_1",
+ "sharegpt_TRck0h4_0",
+ "c051e740_2",
+ "sharegpt_hzwc0HI_13",
+ "fb62ade8",
+ "ultrachat_260793",
+ "ultrachat_301579",
+ "ultrachat_488942",
+ "f29704ac_2"
+ ],
+ "gold_session_ids": [
+ "answer_58820c75_1",
+ "answer_58820c75_2"
+ ]
+ },
+ {
+ "question_id": "d905b33f",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_85a77c48_2",
+ "answer_85a77c48_1",
+ "3301f749_1",
+ "8cfa91c3_2",
+ "368276ab_3",
+ "7613d5ec_2",
+ "70d0ae5d",
+ "2d364fb0",
+ "6423d6a4_2",
+ "a9b7f49e_1"
+ ],
+ "gold_session_ids": [
+ "answer_85a77c48_2",
+ "answer_85a77c48_1"
+ ]
+ },
+ {
+ "question_id": "7405e8b1",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_80323f3f_2",
+ "answer_80323f3f_1",
+ "f2f2a606_3",
+ "a6b00510_4",
+ "6c1c5bc3_2",
+ "sharegpt_uiECyCD_93",
+ "sharegpt_2pwdGxJ_1",
+ "89568a93_1",
+ "8eeb4704",
+ "7d7f11d0_1"
+ ],
+ "gold_session_ids": [
+ "answer_80323f3f_1",
+ "answer_80323f3f_2"
+ ]
+ },
+ {
+ "question_id": "f35224e0",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_e9bb9500_2",
+ "answer_e9bb9500_1",
+ "8e608386",
+ "96da07f9_4",
+ "42234f98",
+ "176594fa_3",
+ "sharegpt_gGriMhQ_181",
+ "sharegpt_d8dUtMG_0",
+ "fd7becaf_1",
+ "4b285f83"
+ ],
+ "gold_session_ids": [
+ "answer_e9bb9500_2",
+ "answer_e9bb9500_1"
+ ]
+ },
+ {
+ "question_id": "6456829e",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_743f03a1_2",
+ "answer_743f03a1_1",
+ "c44d1533_2",
+ "ultrachat_555576",
+ "5bed6828",
+ "sharegpt_LxQSGQ9_0",
+ "sharegpt_3xXQihW_45",
+ "385683f0_3",
+ "793ff37a",
+ "47b924c1_1"
+ ],
+ "gold_session_ids": [
+ "answer_743f03a1_2",
+ "answer_743f03a1_1"
+ ]
+ },
+ {
+ "question_id": "a4996e51",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_feb5f98a_1",
+ "answer_feb5f98a_2",
+ "252715f8_3",
+ "b1d9eb66_3",
+ "726e9a1f_3",
+ "ef0341d4_2",
+ "4f2d6be4_2",
+ "sharegpt_tvcY0lQ_0",
+ "2ca99347_2",
+ "ultrachat_295714"
+ ],
+ "gold_session_ids": [
+ "answer_feb5f98a_1",
+ "answer_feb5f98a_2"
+ ]
+ },
+ {
+ "question_id": "3c1045c8",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9197207891481876,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_c8cc60d6_2",
+ "ultrachat_20893",
+ "answer_c8cc60d6_1",
+ "ultrachat_251919",
+ "sharegpt_GYqnAhC_190",
+ "ultrachat_484591",
+ "sharegpt_xY3p1o7_127",
+ "sharegpt_NTHvrGV_0",
+ "ef71a35e_1",
+ "4f3b36d4_3"
+ ],
+ "gold_session_ids": [
+ "answer_c8cc60d6_2",
+ "answer_c8cc60d6_1"
+ ]
+ },
+ {
+ "question_id": "60036106",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_e552e1f9_1",
+ "answer_e552e1f9_2",
+ "sharegpt_moARQa7_0",
+ "763f028a",
+ "sharegpt_ckvrgF1_99",
+ "sharegpt_mtS0qOD_65",
+ "e2cdd030",
+ "ultrachat_262356",
+ "10857212_2",
+ "c575291e_2"
+ ],
+ "gold_session_ids": [
+ "answer_e552e1f9_1",
+ "answer_e552e1f9_2"
+ ]
+ },
+ {
+ "question_id": "681a1674",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_3be95d43_2",
+ "answer_3be95d43_1",
+ "d81e853b",
+ "ultrachat_64650",
+ "cbdbcfbc_6",
+ "9146213d_3",
+ "d5b3cf54_2",
+ "sharegpt_2wTEFHv_47",
+ "f1cf6f00_2",
+ "sharegpt_8Kzmn95_0"
+ ],
+ "gold_session_ids": [
+ "answer_3be95d43_1",
+ "answer_3be95d43_2"
+ ]
+ },
+ {
+ "question_id": "e25c3b8d",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_6702277b_2",
+ "answer_6702277b_1",
+ "099c1b6c_3",
+ "824634d9_1",
+ "sharegpt_gQaSoiF_13",
+ "c04c626c_1",
+ "90fea364_1",
+ "ff201786_1",
+ "sharegpt_rtgucSC_0",
+ "20fa75e8_3"
+ ],
+ "gold_session_ids": [
+ "answer_6702277b_1",
+ "answer_6702277b_2"
+ ]
+ },
+ {
+ "question_id": "4adc0475",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8065735963827293,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_6efce493_2",
+ "sharegpt_7nYPy2P_29",
+ "ultrachat_186558",
+ "183d2822_1",
+ "sharegpt_wcTU09Q_31",
+ "f82637ed_1",
+ "bb27df5e_2",
+ "answer_6efce493_1",
+ "29e283f4_1",
+ "ultrachat_30883"
+ ],
+ "gold_session_ids": [
+ "answer_6efce493_1",
+ "answer_6efce493_2"
+ ]
+ },
+ {
+ "question_id": "4bc144e2",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_9ef115d4_1",
+ "answer_9ef115d4_2",
+ "2e2ae792_1",
+ "42bcee92_2",
+ "32a203b4_1",
+ "60207b41_3",
+ "ultrachat_380790",
+ "7b13b4d6_2",
+ "099c1b6c_3",
+ "sharegpt_lwNINvp_0"
+ ],
+ "gold_session_ids": [
+ "answer_9ef115d4_1",
+ "answer_9ef115d4_2"
+ ]
+ },
+ {
+ "question_id": "ef66a6e5",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8065735963827293,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_f7fd1029_1",
+ "afb58af0",
+ "ultrachat_61361",
+ "41e778e4_2",
+ "295645f5_1",
+ "sharegpt_P8MHt59_0",
+ "147ab7e9_6",
+ "answer_f7fd1029_2",
+ "7c08d74b_1",
+ "4c08398b_1"
+ ],
+ "gold_session_ids": [
+ "answer_f7fd1029_2",
+ "answer_f7fd1029_1"
+ ]
+ },
+ {
+ "question_id": "5025383b",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8503449055347546,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_688f9a3f_2",
+ "064f42ee_1",
+ "sharegpt_Yuw2LhQ_4",
+ "cf8e352f_4",
+ "answer_688f9a3f_1",
+ "2d21f921",
+ "bab41bb6_5",
+ "f40fa3ed",
+ "de293134_1",
+ "d10f0307_1"
+ ],
+ "gold_session_ids": [
+ "answer_688f9a3f_1",
+ "answer_688f9a3f_2"
+ ]
+ },
+ {
+ "question_id": "a1cc6108",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8065735963827293,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_17dc2f5b_2",
+ "31e254b5",
+ "sharegpt_jpiaPoJ_29",
+ "fb328ace_1",
+ "sharegpt_wrN9uUo_43",
+ "001cefa7_2",
+ "sharegpt_9eEV9T3_8",
+ "answer_17dc2f5b_1",
+ "ac72b02a_2",
+ "ultrachat_152948"
+ ],
+ "gold_session_ids": [
+ "answer_17dc2f5b_2",
+ "answer_17dc2f5b_1"
+ ]
+ },
+ {
+ "question_id": "9ee3ecd6",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_66c23110_1",
+ "answer_66c23110_2",
+ "fa0fa74d_2",
+ "fd56c5bd_2",
+ "ultrachat_282862",
+ "sharegpt_V2j1zkI_4",
+ "ultrachat_175931",
+ "a697b169_1",
+ "7a7a4bf0_4",
+ "f550a835_1"
+ ],
+ "gold_session_ids": [
+ "answer_66c23110_1",
+ "answer_66c23110_2"
+ ]
+ },
+ {
+ "question_id": "3fdac837",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6934264036172708,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "c85ef24e_1",
+ "answer_419d21d5_1",
+ "answer_419d21d5_2",
+ "d1165be3_2",
+ "a98ff421_7",
+ "88c8df0e_3",
+ "30c92c0c_1",
+ "ultrachat_526748",
+ "09d021e7_1",
+ "daa1fe97_1"
+ ],
+ "gold_session_ids": [
+ "answer_419d21d5_2",
+ "answer_419d21d5_1"
+ ]
+ },
+ {
+ "question_id": "91b15a6e",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_5404a208_1",
+ "answer_5404a208_2",
+ "6169ef55_1",
+ "7cfeb89c_2",
+ "c1e170f0_2",
+ "sharegpt_7r7RS6F_0",
+ "sharegpt_1xq5lFg_0",
+ "4c039bd7_1",
+ "409dfa26_1",
+ "d7cffd42_2"
+ ],
+ "gold_session_ids": [
+ "answer_5404a208_1",
+ "answer_5404a208_2"
+ ]
+ },
+ {
+ "question_id": "27016adc",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_a37bdf22_1",
+ "answer_a37bdf22_2",
+ "1a555998",
+ "fa858208_2",
+ "ultrachat_566359",
+ "27ce9f13_2",
+ "ultrachat_284316",
+ "f1597b1c",
+ "dd25eeb5_8",
+ "ff5e0db0_1"
+ ],
+ "gold_session_ids": [
+ "answer_a37bdf22_2",
+ "answer_a37bdf22_1"
+ ]
+ },
+ {
+ "question_id": "720133ac",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_c9dfeaea_1",
+ "answer_c9dfeaea_2",
+ "4c128d97_2",
+ "ultrachat_508711",
+ "e200d96c_3",
+ "ultrachat_479833",
+ "sharegpt_sG53ibg_116",
+ "10a598b1_1",
+ "87bce563",
+ "sharegpt_Km6yx9W_13"
+ ],
+ "gold_session_ids": [
+ "answer_c9dfeaea_1",
+ "answer_c9dfeaea_2"
+ ]
+ },
+ {
+ "question_id": "77eafa52",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_33c251f0_1",
+ "answer_33c251f0_2",
+ "sharegpt_qEukY6n_14",
+ "41a87f13_1",
+ "f87fea58_1",
+ "fc9a62c0_1",
+ "sharegpt_C9GjDsF_0",
+ "7b13b4d6_2",
+ "sharegpt_nCTZBKf_0",
+ "cd568521_3"
+ ],
+ "gold_session_ids": [
+ "answer_33c251f0_2",
+ "answer_33c251f0_1"
+ ]
+ },
+ {
+ "question_id": "8979f9ec",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_35e36341_1",
+ "answer_35e36341_2",
+ "bcc9382a_2",
+ "1064ed80_2",
+ "a6b4c4c2_1",
+ "ultrachat_562471",
+ "052d7def",
+ "91c71874_1",
+ "f18ebe36_7",
+ "4c967baa_1"
+ ],
+ "gold_session_ids": [
+ "answer_35e36341_1",
+ "answer_35e36341_2"
+ ]
+ },
+ {
+ "question_id": "0100672e",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_35c9798c_2",
+ "answer_35c9798c_1",
+ "b167a4a8",
+ "326e86e0_3",
+ "a4b223a7_1",
+ "562addbd_1",
+ "091aa510_2",
+ "8237de6a_2",
+ "07313722_2",
+ "c694fe7b_1"
+ ],
+ "gold_session_ids": [
+ "answer_35c9798c_2",
+ "answer_35c9798c_1"
+ ]
+ },
+ {
+ "question_id": "a96c20ee",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9197207891481876,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ef84b994_1",
+ "sharegpt_00qHEQ6_0",
+ "answer_ef84b994_2",
+ "c69eb8ee_4",
+ "ultrachat_259222",
+ "18cfdd41_1",
+ "ultrachat_286736",
+ "9ca3534f_1",
+ "dff89d9d_1",
+ "e4e64e80_6"
+ ],
+ "gold_session_ids": [
+ "answer_ef84b994_1",
+ "answer_ef84b994_2"
+ ]
+ },
+ {
+ "question_id": "92a0aa75",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.38685280723454163,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "a5854a8d_2",
+ "answer_6cb8f792_2",
+ "sharegpt_hPax8pw_11",
+ "sharegpt_9l7gjYP_17",
+ "1c0921ee_3",
+ "85eb2b94_2",
+ "ultrachat_61473",
+ "173e2e47",
+ "ultrachat_178564",
+ "6da0bba2_1"
+ ],
+ "gold_session_ids": [
+ "answer_6cb8f792_1",
+ "answer_6cb8f792_2"
+ ]
+ },
+ {
+ "question_id": "3fe836c9",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8772153153380493,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_1bb63ea5_1",
+ "c81897cd_2",
+ "f0d7d364",
+ "answer_1bb63ea5_2",
+ "0a42783a",
+ "dda70510_1",
+ "ff49b5a5_3",
+ "ultrachat_258020",
+ "ultrachat_199846",
+ "ultrachat_328173"
+ ],
+ "gold_session_ids": [
+ "answer_1bb63ea5_2",
+ "answer_1bb63ea5_1"
+ ]
+ },
+ {
+ "question_id": "1c549ce4",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_4cb841a8_1",
+ "answer_4cb841a8_2",
+ "sharegpt_miOKGt1_13",
+ "f1e4daa8_2",
+ "35f052f9_2",
+ "f999b05c_1",
+ "809cbce9_1",
+ "ddcd2c65_1",
+ "sharegpt_Ab5RUMY_0",
+ "d743153b_2"
+ ],
+ "gold_session_ids": [
+ "answer_4cb841a8_1",
+ "answer_4cb841a8_2"
+ ]
+ },
+ {
+ "question_id": "6c49646a",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8772153153380493,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_cc1ced21_1",
+ "9f220c66_1",
+ "ultrachat_565061",
+ "answer_cc1ced21_2",
+ "d3065d85",
+ "6155addd_2",
+ "1a24ef7c_5",
+ "2aeb1268",
+ "ultrachat_298957",
+ "8df3019b_1"
+ ],
+ "gold_session_ids": [
+ "answer_cc1ced21_2",
+ "answer_cc1ced21_1"
+ ]
+ },
+ {
+ "question_id": "1192316e",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_e184e4c3_2",
+ "answer_e184e4c3_1",
+ "c5db849c_1",
+ "f6ed7fa9_3",
+ "574df152_3",
+ "21ef2d05_1",
+ "55f7cc4f_3",
+ "6cd7234e_2",
+ "sharegpt_5HGh4Yy_0",
+ "6f8cd952"
+ ],
+ "gold_session_ids": [
+ "answer_e184e4c3_2",
+ "answer_e184e4c3_1"
+ ]
+ },
+ {
+ "question_id": "0ea62687",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_dc5e537d_1",
+ "answer_dc5e537d_2",
+ "d1ea55c0_1",
+ "d3ee5958_1",
+ "e22d6aef_1",
+ "5dce60dd",
+ "26eedae1_1",
+ "988c6b0e",
+ "50904946_2",
+ "b9e47a0f_2"
+ ],
+ "gold_session_ids": [
+ "answer_dc5e537d_1",
+ "answer_dc5e537d_2"
+ ]
+ },
+ {
+ "question_id": "67e0d0f2",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_3a5010af_2",
+ "answer_3a5010af_1",
+ "2252d3dd",
+ "ultrachat_217333",
+ "sharegpt_uD7CBng_30",
+ "6868c94f",
+ "b192ae00_2",
+ "sharegpt_RziX6kj_0",
+ "298382a6_2",
+ "ultrachat_25863"
+ ],
+ "gold_session_ids": [
+ "answer_3a5010af_2",
+ "answer_3a5010af_1"
+ ]
+ },
+ {
+ "question_id": "bb7c3b45",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_de64539a_1",
+ "answer_de64539a_2",
+ "3891baea_4",
+ "f3d43a12_3",
+ "4f2d6be4_1",
+ "0c061750_1",
+ "a51dc9ca",
+ "13b6b7ab_1",
+ "0287c177_1",
+ "sharegpt_l02agjs_0"
+ ],
+ "gold_session_ids": [
+ "answer_de64539a_1",
+ "answer_de64539a_2"
+ ]
+ },
+ {
+ "question_id": "ba358f49",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6131471927654584,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_cbd08e3c_1",
+ "fb117eb0_2",
+ "f6859b48_2",
+ "2a2ea005_1",
+ "385683f0_2",
+ "sharegpt_c2LF3xK_12",
+ "sharegpt_IqVsDp7_89",
+ "fb0f2ac2",
+ "31419155",
+ "4f838497_2"
+ ],
+ "gold_session_ids": [
+ "answer_cbd08e3c_1",
+ "answer_cbd08e3c_2"
+ ]
+ },
+ {
+ "question_id": "61f8c8f8",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_872e8da2_1",
+ "answer_872e8da2_2",
+ "2154c810_1",
+ "1a24ef7c_5",
+ "6e110a53_1",
+ "sharegpt_7jGo66i_99",
+ "76ea58ed_1",
+ "7b88c38b_2",
+ "8991014d_1",
+ "58335f60_2"
+ ],
+ "gold_session_ids": [
+ "answer_872e8da2_2",
+ "answer_872e8da2_1"
+ ]
+ },
+ {
+ "question_id": "60159905",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_75eca223_2",
+ "answer_75eca223_1",
+ "21f8f481_3",
+ "6ade84f0",
+ "c351fa3b_1",
+ "15998e39_4",
+ "b7917b32",
+ "02eae87c_5",
+ "b3ef2cd4_1",
+ "b4f94171_2"
+ ],
+ "gold_session_ids": [
+ "answer_75eca223_2",
+ "answer_75eca223_1"
+ ]
+ },
+ {
+ "question_id": "ef9cf60a",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_87e3a1cb_1",
+ "answer_87e3a1cb_2",
+ "2357de36_2",
+ "7b88c38b_1",
+ "a89815bf",
+ "d677eb06",
+ "71027306_1",
+ "sharegpt_Ha6kIaj_0",
+ "sharegpt_Lk4UN8n_159",
+ "ultrachat_247949"
+ ],
+ "gold_session_ids": [
+ "answer_87e3a1cb_1",
+ "answer_87e3a1cb_2"
+ ]
+ },
+ {
+ "question_id": "73d42213",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_1881e7db_2",
+ "answer_1881e7db_1",
+ "3cb41ab8_1",
+ "sharegpt_ab6IEma_0",
+ "sharegpt_YlgTw57_486",
+ "9614d065_1",
+ "02e9f7e3_3",
+ "sharegpt_ndynIbS_0",
+ "a1251fef_2",
+ "a13abca5_1"
+ ],
+ "gold_session_ids": [
+ "answer_1881e7db_1",
+ "answer_1881e7db_2"
+ ]
+ },
+ {
+ "question_id": "bc149d6b",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9197207891481876,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_92147866_1",
+ "7211c69e",
+ "answer_92147866_2",
+ "9806001e",
+ "4f435d20_2",
+ "9bfd836a",
+ "05be3ff0_3",
+ "dcb18b9b_1",
+ "93a7a1bb_1",
+ "ultrachat_13563"
+ ],
+ "gold_session_ids": [
+ "answer_92147866_1",
+ "answer_92147866_2"
+ ]
+ },
+ {
+ "question_id": "099778bb",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_80d6d664_1",
+ "answer_80d6d664_2",
+ "b1793bba",
+ "95ce0e21_1",
+ "sharegpt_TMWJOGD_21",
+ "5b014f9b_1",
+ "sharegpt_O8djord_90",
+ "4acdd888",
+ "sharegpt_1dQ2mgC_0",
+ "ultrachat_60492"
+ ],
+ "gold_session_ids": [
+ "answer_80d6d664_2",
+ "answer_80d6d664_1"
+ ]
+ },
+ {
+ "question_id": "09ba9854",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_96c743d0_2",
+ "answer_96c743d0_1",
+ "f7644391_2",
+ "8c63238e",
+ "32e6c308",
+ "5faf5962_3",
+ "69a42467_2",
+ "e760dc71_1",
+ "ultrachat_447648",
+ "2bd23659_1"
+ ],
+ "gold_session_ids": [
+ "answer_96c743d0_2",
+ "answer_96c743d0_1"
+ ]
+ },
+ {
+ "question_id": "d6062bb9",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_23f3a657_2",
+ "answer_23f3a657_1",
+ "86298245_2",
+ "5b6aedac",
+ "65d28641_2",
+ "sharegpt_xuhzAgG_19",
+ "f7644391_2",
+ "59c704ad_2",
+ "5c95d0d0",
+ "sharegpt_PjFSCFK_9"
+ ],
+ "gold_session_ids": [
+ "answer_23f3a657_2",
+ "answer_23f3a657_1"
+ ]
+ },
+ {
+ "question_id": "157a136e",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8315546295836225,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_8de18468_2",
+ "377f8fd9_4",
+ "298bf7a3_1",
+ "7746ba1b_1",
+ "ultrachat_506827",
+ "answer_8de18468_1",
+ "ultrachat_255132",
+ "9b3a7f2c_4",
+ "6715a0c6_1",
+ "7614c21f_2"
+ ],
+ "gold_session_ids": [
+ "answer_8de18468_2",
+ "answer_8de18468_1"
+ ]
+ },
+ {
+ "question_id": "c18a7dc8",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.38685280723454163,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "9d5a389d",
+ "answer_2e2085fa_1",
+ "ultrachat_353377",
+ "ultrachat_418857",
+ "2d74df23_1",
+ "sharegpt_YPC6Oll_18",
+ "ultrachat_463016",
+ "f716720c_3",
+ "5022812c",
+ "ultrachat_415226"
+ ],
+ "gold_session_ids": [
+ "answer_2e2085fa_2",
+ "answer_2e2085fa_1"
+ ]
+ },
+ {
+ "question_id": "a3332713",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_16ece55f_2",
+ "answer_16ece55f_1",
+ "67388dfc_2",
+ "1c8a264a_4",
+ "117098d9_3",
+ "62be624f",
+ "sharegpt_7uSzi6w_2",
+ "sharegpt_bYymtlW_0",
+ "202c6329_1",
+ "33994682_3"
+ ],
+ "gold_session_ids": [
+ "answer_16ece55f_2",
+ "answer_16ece55f_1"
+ ]
+ },
+ {
+ "question_id": "55241a1f",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_fa08bf49_1",
+ "answer_fa08bf49_2",
+ "9c9a22d8",
+ "d117e2da",
+ "46e4a8eb_2",
+ "76491ee4_1",
+ "2aeb1268",
+ "98c83fe4",
+ "ultrachat_348345",
+ "60153a02_4"
+ ],
+ "gold_session_ids": [
+ "answer_fa08bf49_1",
+ "answer_fa08bf49_2"
+ ]
+ },
+ {
+ "question_id": "a08a253f",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_47152166_1",
+ "answer_47152166_2",
+ "771570c5_1",
+ "d79173aa_3",
+ "6e5130ed_2",
+ "28209b6a_4",
+ "b1d9eb66_4",
+ "798fa5f2",
+ "4524c250_2",
+ "c7fc2bda_1"
+ ],
+ "gold_session_ids": [
+ "answer_47152166_2",
+ "answer_47152166_1"
+ ]
+ },
+ {
+ "question_id": "f0e564bc",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_cfcf5340_2",
+ "answer_cfcf5340_1",
+ "798e4ba3_2",
+ "6dfc99bf_2",
+ "932a97c6",
+ "9f35ac68",
+ "bc1f1051_1",
+ "sharegpt_W3YFpVK_7",
+ "683fdb17",
+ "658150c4"
+ ],
+ "gold_session_ids": [
+ "answer_cfcf5340_1",
+ "answer_cfcf5340_2"
+ ]
+ },
+ {
+ "question_id": "078150f1",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_254d8b09_1",
+ "answer_254d8b09_2",
+ "61b27d67_3",
+ "b18ba8d0_1",
+ "ultrachat_202549",
+ "203a48bb_1",
+ "ultrachat_276164",
+ "e4c97ea0_1",
+ "08256f84_1",
+ "2655efcc_1"
+ ],
+ "gold_session_ids": [
+ "answer_254d8b09_1",
+ "answer_254d8b09_2"
+ ]
+ },
+ {
+ "question_id": "8cf4d046",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9197207891481876,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_e2278b24_2",
+ "sharegpt_WjpyexI_15",
+ "answer_e2278b24_1",
+ "ultrachat_554855",
+ "sharegpt_P1Okivz_9",
+ "sharegpt_XsIUcaD_89",
+ "6c6c8df7",
+ "410ec447",
+ "sharegpt_kyCtrXT_0",
+ "7a5c32b0_3"
+ ],
+ "gold_session_ids": [
+ "answer_e2278b24_1",
+ "answer_e2278b24_2"
+ ]
+ },
+ {
+ "question_id": "a346bb18",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8315546295836225,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_4934b2d7_2",
+ "53949711_2",
+ "5726dc37_2",
+ "59e96888_1",
+ "771ba1d5",
+ "answer_4934b2d7_1",
+ "feb5f98a_1",
+ "c51572bb_2",
+ "fd5448f0",
+ "2655efcc_1"
+ ],
+ "gold_session_ids": [
+ "answer_4934b2d7_2",
+ "answer_4934b2d7_1"
+ ]
+ },
+ {
+ "question_id": "37f165cf",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_6b9b2b1e_2",
+ "answer_6b9b2b1e_1",
+ "20b1b65f_2",
+ "37b19c82",
+ "7e8e790a",
+ "0f134440",
+ "cdcf791b_1",
+ "395a2f92",
+ "sharegpt_2yaKUmC_0",
+ "38de4c2a_1"
+ ],
+ "gold_session_ids": [
+ "answer_6b9b2b1e_2",
+ "answer_6b9b2b1e_1"
+ ]
+ },
+ {
+ "question_id": "8e91e7d9",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.38685280723454163,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "ultrachat_147574",
+ "answer_477ae455_2",
+ "sharegpt_8lpX5R7_19",
+ "4e3da326_1",
+ "da71721f",
+ "sharegpt_PZrBCF2_0",
+ "4f8caea3_2",
+ "17cb8d2f_3",
+ "7a389bd1_7",
+ "c3023a45_4"
+ ],
+ "gold_session_ids": [
+ "answer_477ae455_1",
+ "answer_477ae455_2"
+ ]
+ },
+ {
+ "question_id": "87f22b4a",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_f56e6152_1",
+ "answer_f56e6152_2",
+ "ea28bb75",
+ "c6015a0c_3",
+ "a8ac3d1d_3",
+ "ultrachat_182225",
+ "7c2ec9e9_2",
+ "889076d6",
+ "02b99ec3_1",
+ "c4c504d2_6"
+ ],
+ "gold_session_ids": [
+ "answer_f56e6152_2",
+ "answer_f56e6152_1"
+ ]
+ },
+ {
+ "question_id": "e56a43b9",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9197207891481876,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_430d0a87_2",
+ "7596ba6a",
+ "answer_430d0a87_1",
+ "57a6a404_3",
+ "372544ba",
+ "38e81260_7",
+ "ultrachat_12201",
+ "3c0ed3bf_1",
+ "a59b73f6",
+ "ce21f0e4"
+ ],
+ "gold_session_ids": [
+ "answer_430d0a87_1",
+ "answer_430d0a87_2"
+ ]
+ },
+ {
+ "question_id": "efc3f7c2",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9197207891481876,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_fc81d1a2_1",
+ "88a7faa7_1",
+ "answer_fc81d1a2_2",
+ "bbdc7b0a",
+ "8a7df17e_1",
+ "0f013a55",
+ "bbca6598_1",
+ "fc005760",
+ "ultrachat_38938",
+ "c4251d8d_1"
+ ],
+ "gold_session_ids": [
+ "answer_fc81d1a2_1",
+ "answer_fc81d1a2_2"
+ ]
+ },
+ {
+ "question_id": "21d02d0d",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_2c637141_1",
+ "answer_2c637141_2",
+ "2e28f3db",
+ "a15fa6eb",
+ "42d9467c_2",
+ "ultrachat_36404",
+ "414b8ddb",
+ "6ff8954a_2",
+ "94bc18df_2",
+ "ultrachat_285752"
+ ],
+ "gold_session_ids": [
+ "answer_2c637141_2",
+ "answer_2c637141_1"
+ ]
+ },
+ {
+ "question_id": "2311e44b_abs",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_bf633415_abs_1",
+ "answer_bf633415_abs_2",
+ "35915bf5",
+ "313404c6_2",
+ "dc880372_2",
+ "4c8afa9c_1",
+ "sharegpt_l2SM5sp_0",
+ "sharegpt_3bZSvyG_0",
+ "369695b4_2",
+ "a7bd5b45_3"
+ ],
+ "gold_session_ids": [
+ "answer_bf633415_abs_1",
+ "answer_bf633415_abs_2"
+ ]
+ },
+ {
+ "question_id": "6456829e_abs",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_743f03a1_abs_2",
+ "answer_743f03a1_abs_1",
+ "b61cc91b_2",
+ "sharegpt_Hc0Zm6i_0",
+ "d20bb5aa_1",
+ "3d8d4828_2",
+ "ultrachat_567580",
+ "89cce497",
+ "ultrachat_345787",
+ "ultrachat_568772"
+ ],
+ "gold_session_ids": [
+ "answer_743f03a1_abs_1",
+ "answer_743f03a1_abs_2"
+ ]
+ },
+ {
+ "question_id": "e5ba910e_abs",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_e49ed9d3_abs_2",
+ "answer_e49ed9d3_abs_1",
+ "a239515b",
+ "6d013bad_3",
+ "6339aa1d_2",
+ "f2df24cd_2",
+ "ultrachat_282969",
+ "bd8a4fcb_2",
+ "cc3f0564_1",
+ "7e0fa6a6"
+ ],
+ "gold_session_ids": [
+ "answer_e49ed9d3_abs_1",
+ "answer_e49ed9d3_abs_2"
+ ]
+ },
+ {
+ "question_id": "a96c20ee_abs",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9197207891481876,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ef84b994_abs_2",
+ "4163e709_3",
+ "answer_ef84b994_abs_1",
+ "ultrachat_59306",
+ "ultrachat_194192",
+ "ultrachat_293509",
+ "sharegpt_jM9e92F_0",
+ "ultrachat_36312",
+ "ultrachat_184142",
+ "ultrachat_24979"
+ ],
+ "gold_session_ids": [
+ "answer_ef84b994_abs_1",
+ "answer_ef84b994_abs_2"
+ ]
+ },
+ {
+ "question_id": "ba358f49_abs",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.7977228895450266,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_cbd08e3c_abs_1",
+ "07ebc271",
+ "5829ab38",
+ "sharegpt_7kFao5h_0",
+ "1162611c",
+ "40648965",
+ "20ea9554_1",
+ "9844f740",
+ "answer_cbd08e3c_abs_2",
+ "sharegpt_HRpe8C8_0"
+ ],
+ "gold_session_ids": [
+ "answer_cbd08e3c_abs_1",
+ "answer_cbd08e3c_abs_2"
+ ]
+ },
+ {
+ "question_id": "09ba9854_abs",
+ "question_type": "multi-session",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_96c743d0_abs_1",
+ "answer_96c743d0_abs_2",
+ "ultrachat_342361",
+ "a50b0d73_2",
+ "66e58c90_2",
+ "sharegpt_JZm4ich_15",
+ "64460e9d_2",
+ "85ca9420",
+ "338ac116_2",
+ "e4250940"
+ ],
+ "gold_session_ids": [
+ "answer_96c743d0_abs_1",
+ "answer_96c743d0_abs_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_59149c77",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6934264036172708,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "9e2c2a6c_2",
+ "answer_d00ba6d0_2",
+ "answer_d00ba6d0_1",
+ "sharegpt_XlNs8Lz_199",
+ "ab520c81_1",
+ "a7403b3d_4",
+ "148f87a0_2",
+ "6e672b84_1",
+ "ultrachat_318178",
+ "64c2b3c8"
+ ],
+ "gold_session_ids": [
+ "answer_d00ba6d0_1",
+ "answer_d00ba6d0_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_f49edff3",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9469024295259745,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_3e9fce53_3",
+ "answer_3e9fce53_2",
+ "4376945f",
+ "40a289b7_2",
+ "answer_3e9fce53_1",
+ "7d33fcb4_1",
+ "cbc1588c_2",
+ "8e226a86",
+ "dc7c53fa_4",
+ "4798296c_2"
+ ],
+ "gold_session_ids": [
+ "answer_3e9fce53_1",
+ "answer_3e9fce53_2",
+ "answer_3e9fce53_3"
+ ]
+ },
+ {
+ "question_id": "71017276",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_0b4a8adc_1",
+ "sharegpt_CjoQMV6_0",
+ "sharegpt_tCLMKZY_0",
+ "sharegpt_qevDNvo_0",
+ "398f23e7",
+ "f78a22b6_4",
+ "sharegpt_duoAhsS_0",
+ "10889cf0_1",
+ "683c8358_1",
+ "sharegpt_ZUJIuGt_13"
+ ],
+ "gold_session_ids": [
+ "answer_0b4a8adc_1"
+ ]
+ },
+ {
+ "question_id": "b46e15ed",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6525874238732422,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_4bfcc250_4",
+ "aeab3296",
+ "ultrachat_265690",
+ "53949711_2",
+ "47311a4c_1",
+ "answer_4bfcc250_3",
+ "sharegpt_cXkL3cR_28",
+ "answer_4bfcc250_1",
+ "f598d30f_1",
+ "ultrachat_396986"
+ ],
+ "gold_session_ids": [
+ "answer_4bfcc250_4",
+ "answer_4bfcc250_3",
+ "answer_4bfcc250_2",
+ "answer_4bfcc250_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_fa19884c",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ff201786_2",
+ "answer_ff201786_1",
+ "e8de3056_2",
+ "cbf48918_1",
+ "f4f4b504_2",
+ "bb2180e9_3",
+ "6414f676_2",
+ "ultrachat_278956",
+ "ultrachat_423544",
+ "f3745025_1"
+ ],
+ "gold_session_ids": [
+ "answer_ff201786_1",
+ "answer_ff201786_2"
+ ]
+ },
+ {
+ "question_id": "0bc8ad92",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9674679834891693,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_f4ea84fb_3",
+ "answer_f4ea84fb_1",
+ "d3cc5bdc_1",
+ "answer_f4ea84fb_2",
+ "0868982f_1",
+ "717d1aea_1",
+ "ultrachat_349938",
+ "sharegpt_ntu1Cds_0",
+ "f58c36b8_1",
+ "sharegpt_vClQxDr_21"
+ ],
+ "gold_session_ids": [
+ "answer_f4ea84fb_3",
+ "answer_f4ea84fb_2",
+ "answer_f4ea84fb_1"
+ ]
+ },
+ {
+ "question_id": "af082822",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 0,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0,
+ "mrr": 0.06666666666666667,
+ "retrieved_session_ids": [
+ "1dfef591_2",
+ "6cf1848e_3",
+ "e132259f",
+ "bc5827b5",
+ "1a920713_2",
+ "facef6e0_1",
+ "bf38543d_2",
+ "1fcb4134_2",
+ "fbe6fa2c_2",
+ "1040e24b_1"
+ ],
+ "gold_session_ids": [
+ "answer_b51b6115_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_4929293a",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_add9b012_2",
+ "answer_add9b012_1",
+ "ultrachat_139245",
+ "ultrachat_512282",
+ "b8bc1464_4",
+ "ultrachat_329640",
+ "e2304e5e_1",
+ "73353aee_1",
+ "c1e598cd_1",
+ "ultrachat_127833"
+ ],
+ "gold_session_ids": [
+ "answer_add9b012_2",
+ "answer_add9b012_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_b5700ca9",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_a17423e7_1",
+ "5053474c_2",
+ "47382adf_1",
+ "33bdd89a_1",
+ "31829c1b_1",
+ "sharegpt_FqOC5e9_169",
+ "c81aa33c_2",
+ "d743153b_1",
+ "a53c7542_1",
+ "ultrachat_383774"
+ ],
+ "gold_session_ids": [
+ "answer_a17423e7_1"
+ ]
+ },
+ {
+ "question_id": "9a707b81",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9197207891481876,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_dba89487_2",
+ "4abbeb2a_2",
+ "answer_dba89487_1",
+ "e1d11615_1",
+ "990f3ef9_6",
+ "ultrachat_233519",
+ "e75c302b_2",
+ "3fcbb58c_4",
+ "sharegpt_sW9QRz1_0",
+ "5a13d047_1"
+ ],
+ "gold_session_ids": [
+ "answer_dba89487_2",
+ "answer_dba89487_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_1d4ab0c9",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_febde667_1",
+ "answer_febde667_2",
+ "sharegpt_Caxpnd3_12",
+ "7257bd15_1",
+ "sharegpt_MIYBRJz_27",
+ "sharegpt_htSCqmh_171",
+ "c14e3f09_1",
+ "ultrachat_125013",
+ "4ff83df8",
+ "28d52bff"
+ ],
+ "gold_session_ids": [
+ "answer_febde667_1",
+ "answer_febde667_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_e072b769",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_c19bd2bf_1",
+ "0dd54b7c_1",
+ "4f7b5dc9_1",
+ "5558a42e_1",
+ "sharegpt_9Ln5Ood_0",
+ "66d3fdb7_2",
+ "sharegpt_zdDMlRC_0",
+ "sharegpt_VbIxRWJ_33",
+ "ultrachat_423979",
+ "2fc9ce79_2"
+ ],
+ "gold_session_ids": [
+ "answer_c19bd2bf_1"
+ ]
+ },
+ {
+ "question_id": "0db4c65d",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_b9e32ff8_2",
+ "answer_b9e32ff8_1",
+ "f78a22b6_3",
+ "953d5a4b_1",
+ "71dc2037_2",
+ "sharegpt_UGg8d44_9",
+ "6461402b",
+ "ultrachat_324557",
+ "d36d11b9_2",
+ "e6c3a50a"
+ ],
+ "gold_session_ids": [
+ "answer_b9e32ff8_1",
+ "answer_b9e32ff8_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_1d80365e",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_661b711f_1",
+ "answer_661b711f_2",
+ "805e571c_1",
+ "d6b17438_1",
+ "554b054e_2",
+ "3be19aed_1",
+ "ef0341d4_2",
+ "18f6e3be_2",
+ "sharegpt_5uZ7IdY_0",
+ "f7a61595_2"
+ ],
+ "gold_session_ids": [
+ "answer_661b711f_1",
+ "answer_661b711f_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_7f6b06db",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.35853358850813305,
+ "mrr": 0.25,
+ "retrieved_session_ids": [
+ "4bf66c38_2",
+ "6f051087_2",
+ "da42526b_1",
+ "answer_5d8c99d3_2",
+ "252715f8_2",
+ "928890e4_3",
+ "answer_5d8c99d3_3",
+ "42165950_8",
+ "39c1bf6a",
+ "a6109135_1"
+ ],
+ "gold_session_ids": [
+ "answer_5d8c99d3_1",
+ "answer_5d8c99d3_2",
+ "answer_5d8c99d3_3"
+ ]
+ },
+ {
+ "question_id": "gpt4_6dc9b45b",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6309297535714575,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "ultrachat_384387",
+ "answer_c4df007f_1",
+ "afa0c2e7_5",
+ "9adc81d7_2",
+ "ultrachat_35147",
+ "27639dd8_1",
+ "cfcf5340_2",
+ "e7cc239c_4",
+ "d8f48b0a",
+ "ultrachat_453768"
+ ],
+ "gold_session_ids": [
+ "answer_c4df007f_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_8279ba02",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 0,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0,
+ "mrr": 0.08333333333333333,
+ "retrieved_session_ids": [
+ "b909542d_1",
+ "b5b8f8f9_1",
+ "01f1e64a_3",
+ "1a037873_2",
+ "b6018747_2",
+ "sharegpt_BSRNM84_16",
+ "86c5d31d",
+ "85f19a30_1",
+ "886b70b1_2",
+ "67d1de46_2"
+ ],
+ "gold_session_ids": [
+ "answer_56521e65_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_18c2b244",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_c862f65a_3",
+ "answer_c862f65a_2",
+ "answer_c862f65a_1",
+ "ultrachat_503659",
+ "7e2c6065",
+ "dfbc46bd_1",
+ "746ae854_2",
+ "ffd66315_2",
+ "7c4431e2_1",
+ "3176940c_2"
+ ],
+ "gold_session_ids": [
+ "answer_c862f65a_2",
+ "answer_c862f65a_3",
+ "answer_c862f65a_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_a1b77f9c",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_e9ad5914_4",
+ "answer_e9ad5914_2",
+ "answer_e9ad5914_3",
+ "answer_e9ad5914_5",
+ "answer_e9ad5914_1",
+ "answer_e9ad5914_6",
+ "d52d478b",
+ "47ec1674_2",
+ "0dcfb511_4",
+ "660c4dba_2"
+ ],
+ "gold_session_ids": [
+ "answer_e9ad5914_1",
+ "answer_e9ad5914_2",
+ "answer_e9ad5914_3",
+ "answer_e9ad5914_4",
+ "answer_e9ad5914_5",
+ "answer_e9ad5914_6"
+ ]
+ },
+ {
+ "question_id": "gpt4_1916e0ea",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.7977228895450266,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_447052a5_1",
+ "sharegpt_gw7Ym6y_6",
+ "fa1ce406_2",
+ "b9890c15_2",
+ "3f3013b4_1",
+ "d3fe25ff",
+ "280352e9",
+ "4b72fe4c_1",
+ "answer_447052a5_2",
+ "fe4dd702_2"
+ ],
+ "gold_session_ids": [
+ "answer_447052a5_2",
+ "answer_447052a5_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_7a0daae1",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_4d5490f1_2",
+ "answer_4d5490f1_1",
+ "bf3aebdb_2",
+ "2ff297a8_1",
+ "819dfad7_2",
+ "sharegpt_xcd9nky_8",
+ "facb94a8_3",
+ "ultrachat_253041",
+ "ultrachat_49899",
+ "ultrachat_9855"
+ ],
+ "gold_session_ids": [
+ "answer_4d5490f1_1",
+ "answer_4d5490f1_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_468eb063",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.38685280723454163,
+ "mrr": 0.2,
+ "retrieved_session_ids": [
+ "sharegpt_Sf51OjE_8",
+ "e60a93ff_2",
+ "sharegpt_gfhUw40_17",
+ "71888aff_2",
+ "answer_9b09d95b_1",
+ "sharegpt_ipLglky_42",
+ "7ca763c3_2",
+ "ce584ba0",
+ "6410c263_1",
+ "sharegpt_8M7HcM3_11"
+ ],
+ "gold_session_ids": [
+ "answer_9b09d95b_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_7abb270c",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.7976403384579975,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_7093d898_5",
+ "answer_7093d898_6",
+ "087d2b0a_2",
+ "1bfd5a8b_2",
+ "5808284e",
+ "answer_7093d898_2",
+ "answer_7093d898_1",
+ "answer_7093d898_4",
+ "85056d5f_1",
+ "3c9cdf01"
+ ],
+ "gold_session_ids": [
+ "answer_7093d898_1",
+ "answer_7093d898_2",
+ "answer_7093d898_3",
+ "answer_7093d898_4",
+ "answer_7093d898_5",
+ "answer_7093d898_6"
+ ]
+ },
+ {
+ "question_id": "gpt4_1e4a8aeb",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_16bd5ea5_2",
+ "answer_16bd5ea5_1",
+ "28621d6a_2",
+ "fd18e0dd_1",
+ "6be23caa",
+ "ae0cd456",
+ "f7838a91_1",
+ "sharegpt_Nk10TjE_29",
+ "6a7d7055_1",
+ "e83a2abc_1"
+ ],
+ "gold_session_ids": [
+ "answer_16bd5ea5_1",
+ "answer_16bd5ea5_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_4fc4f797",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_be07688f_1",
+ "answer_be07688f_2",
+ "39566615_1",
+ "bdcee74e_3",
+ "sharegpt_pRqHb1o_0",
+ "08834bf2_1",
+ "660cea1d",
+ "59fc4d6b",
+ "10d7d2b4_2",
+ "e77ef495_1"
+ ],
+ "gold_session_ids": [
+ "answer_be07688f_1",
+ "answer_be07688f_2"
+ ]
+ },
+ {
+ "question_id": "4dfccbf7",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6934264036172708,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "8fa624b2_6",
+ "answer_4bebc782_2",
+ "answer_4bebc782_1",
+ "e1b7b676",
+ "sharegpt_Dcshjlv_0",
+ "ultrachat_373047",
+ "36f8b380_1",
+ "aa71db11_2",
+ "f5493bf3",
+ "sharegpt_cXkL3cR_69"
+ ],
+ "gold_session_ids": [
+ "answer_4bebc782_1",
+ "answer_4bebc782_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_61e13b3c",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_e831a29f_1",
+ "answer_e831a29f_2",
+ "ultrachat_441289",
+ "2b5c911e_3",
+ "e75828ad_3",
+ "ultrachat_213012",
+ "8e10bf6b_1",
+ "cc380119_3",
+ "d4439279_1",
+ "a6f2320f_2"
+ ],
+ "gold_session_ids": [
+ "answer_e831a29f_1",
+ "answer_e831a29f_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_45189cb4",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8179809735263114,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_e6c20e52_3",
+ "dc3ee1d1_1",
+ "22452c22",
+ "5c16fe0b_1",
+ "answer_e6c20e52_1",
+ "answer_e6c20e52_2",
+ "36e72174_1",
+ "93395e5f_3",
+ "6593cb8b_1",
+ "ebf5b3bc_2"
+ ],
+ "gold_session_ids": [
+ "answer_e6c20e52_3",
+ "answer_e6c20e52_2",
+ "answer_e6c20e52_1"
+ ]
+ },
+ {
+ "question_id": "2ebe6c90",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_c9d35c00_2",
+ "answer_c9d35c00_1",
+ "sharegpt_jyMHY1J_31",
+ "ultrachat_4566",
+ "dddce60a_2",
+ "9ccea2f0_2",
+ "0fe848c5_3",
+ "c17f3491",
+ "453d354e_1",
+ "8fa64936_1"
+ ],
+ "gold_session_ids": [
+ "answer_c9d35c00_1",
+ "answer_c9d35c00_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_e061b84f",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6105456988825855,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_8c64ce25_3",
+ "78d28576_2",
+ "0a6bf5e4_1",
+ "773287dc_1",
+ "539c2346_3",
+ "bab3f409",
+ "ultrachat_129606",
+ "3275acf9",
+ "answer_8c64ce25_1",
+ "ultrachat_194928"
+ ],
+ "gold_session_ids": [
+ "answer_8c64ce25_2",
+ "answer_8c64ce25_1",
+ "answer_8c64ce25_3"
+ ]
+ },
+ {
+ "question_id": "370a8ff4",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_61d1be50_1",
+ "answer_61d1be50_2",
+ "sharegpt_jGPXxpI_17",
+ "2b4eaee0_1",
+ "8c64ce25_1",
+ "21a51dee_2",
+ "df53e1b7_2",
+ "ultrachat_26182",
+ "11c0b1c5",
+ "4e77dbbe_5"
+ ],
+ "gold_session_ids": [
+ "answer_61d1be50_1",
+ "answer_61d1be50_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_d6585ce8",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_f999b05b_4",
+ "answer_f999b05b_5",
+ "answer_f999b05b_2",
+ "answer_f999b05b_3",
+ "answer_f999b05b_1",
+ "ultrachat_531211",
+ "5d4010bd",
+ "23901329_2",
+ "sharegpt_Asf5ayG_41",
+ "dc0802a9_1"
+ ],
+ "gold_session_ids": [
+ "answer_f999b05b_1",
+ "answer_f999b05b_4",
+ "answer_f999b05b_2",
+ "answer_f999b05b_5",
+ "answer_f999b05b_3"
+ ]
+ },
+ {
+ "question_id": "gpt4_4ef30696",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8772153153380493,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_f964cea3_2",
+ "sharegpt_ELYlA30_25",
+ "4b29957c_1",
+ "answer_f964cea3_1",
+ "80b3f093",
+ "d9868305_2",
+ "ultrachat_356406",
+ "750b437b_1",
+ "1ccb08a0",
+ "ultrachat_511839"
+ ],
+ "gold_session_ids": [
+ "answer_f964cea3_1",
+ "answer_f964cea3_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_ec93e27f",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.7903864795495061,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_e3aa84c4_1",
+ "6b63bdc5_2",
+ "sharegpt_Sg0ctDe_14",
+ "85846900_2",
+ "ultrachat_133145",
+ "ultrachat_401609",
+ "ultrachat_29858",
+ "5a994ed3_2",
+ "b76a71b9_2",
+ "answer_e3aa84c4_2"
+ ],
+ "gold_session_ids": [
+ "answer_e3aa84c4_2",
+ "answer_e3aa84c4_1"
+ ]
+ },
+ {
+ "question_id": "6e984301",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_88841f26_1",
+ "answer_88841f26_2",
+ "0125f748_1",
+ "31903f54",
+ "b38766c1_2",
+ "ultrachat_402356",
+ "2bc4ae7d",
+ "8f6e71cf",
+ "ultrachat_111630",
+ "93bed3d8_2"
+ ],
+ "gold_session_ids": [
+ "answer_88841f26_1",
+ "answer_88841f26_2"
+ ]
+ },
+ {
+ "question_id": "8077ef71",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_0dd54b7c_1",
+ "f7ffddbc",
+ "sharegpt_Wi7Op1u_39",
+ "1da90ded",
+ "sharegpt_hoQsVGz_51",
+ "7fa3c60f_1",
+ "5365840b_1",
+ "30711c1e_1",
+ "56ee0a51",
+ "d87a6ef8"
+ ],
+ "gold_session_ids": [
+ "answer_0dd54b7c_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_f420262c",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9747429528567879,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_d8a1af6b_2",
+ "answer_d8a1af6b_4",
+ "answer_d8a1af6b_3",
+ "a823dc7c",
+ "answer_d8a1af6b_5",
+ "answer_d8a1af6b_1",
+ "3418b277_5",
+ "116bae09",
+ "3ea07188_2",
+ "sharegpt_jFDrHZ3_0"
+ ],
+ "gold_session_ids": [
+ "answer_d8a1af6b_1",
+ "answer_d8a1af6b_2",
+ "answer_d8a1af6b_3",
+ "answer_d8a1af6b_4",
+ "answer_d8a1af6b_5"
+ ]
+ },
+ {
+ "question_id": "gpt4_8e165409",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_d97db962_1",
+ "answer_d97db962_2",
+ "sharegpt_JRQtEcd_115",
+ "4c2bd0fe_2",
+ "58ab5fc5",
+ "ultrachat_77564",
+ "sharegpt_B7mSlfQ_8",
+ "4c967baa_1",
+ "4c9f6bf5_2",
+ "b49985d9_1"
+ ],
+ "gold_session_ids": [
+ "answer_d97db962_1",
+ "answer_d97db962_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_74aed68e",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_aed8cf17_2",
+ "answer_aed8cf17_1",
+ "2a86d0da_1",
+ "ultrachat_176592",
+ "69a9211c_1",
+ "318048af_2",
+ "470f29c0_1",
+ "ultrachat_35782",
+ "25b3e36c_2",
+ "da71721f"
+ ],
+ "gold_session_ids": [
+ "answer_aed8cf17_1",
+ "answer_aed8cf17_2"
+ ]
+ },
+ {
+ "question_id": "bcbe585f",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_43ba3965_1",
+ "5dbe11b9",
+ "7fa8099b_1",
+ "8953c8c8_2",
+ "6bed7862",
+ "896162a2",
+ "sharegpt_WsDgKko_13",
+ "5edfe253_1",
+ "89a16e7f_1",
+ "d8a1af6c_5"
+ ],
+ "gold_session_ids": [
+ "answer_43ba3965_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_21adecb5",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.44130740935663865,
+ "mrr": 0.25,
+ "retrieved_session_ids": [
+ "sharegpt_ipLglky_42",
+ "0f0f9d80_1",
+ "e1f37e24",
+ "answer_1e2369c9_2",
+ "19f2e2c5_2",
+ "ultrachat_444790",
+ "sharegpt_cUXo8S0_0",
+ "b99bd2df",
+ "sharegpt_MCmVQ20_0",
+ "answer_1e2369c9_1"
+ ],
+ "gold_session_ids": [
+ "answer_1e2369c9_1",
+ "answer_1e2369c9_2"
+ ]
+ },
+ {
+ "question_id": "5e1b23de",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_c18d480b_1",
+ "1a5344e9_3",
+ "c28aed33",
+ "7ff028d8_2",
+ "41782aae_1",
+ "cdf75ca3_2",
+ "1380576d_1",
+ "ultrachat_190837",
+ "ultrachat_524214",
+ "0ba46bdc_2"
+ ],
+ "gold_session_ids": [
+ "answer_c18d480b_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_98f46fc6",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_5850de18_1",
+ "answer_5850de18_2",
+ "sharegpt_rnL5VQO_0",
+ "32e292f2",
+ "sharegpt_FhQzMiW_0",
+ "sharegpt_Hj47IDr_0",
+ "ultrachat_238480",
+ "2640383e",
+ "28897044",
+ "sharegpt_PLN69pg_19"
+ ],
+ "gold_session_ids": [
+ "answer_5850de18_2",
+ "answer_5850de18_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_af6db32f",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_184c8f56_1",
+ "f26846e1_3",
+ "ec1b557a_1",
+ "aee13015_6",
+ "ultrachat_251919",
+ "sharegpt_1jjEIai_173",
+ "df6938a4_5",
+ "sharegpt_Np41szj_13",
+ "8ccbe6e0",
+ "8617737e"
+ ],
+ "gold_session_ids": [
+ "answer_184c8f56_1"
+ ]
+ },
+ {
+ "question_id": "eac54adc",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_0d4d0347_2",
+ "answer_0d4d0347_1",
+ "sharegpt_PgmAp0p_0",
+ "b40f0a7a",
+ "sharegpt_Jm8wWJN_0",
+ "sharegpt_z88NqeI_0",
+ "ae051325_3",
+ "32fc393b_2",
+ "80a6c3b7",
+ "dba97bb1_2"
+ ],
+ "gold_session_ids": [
+ "answer_0d4d0347_1",
+ "answer_0d4d0347_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_7ddcf75f",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ad8a76cb_1",
+ "9bfccfdc_2",
+ "8c652eb0_2",
+ "9283c152",
+ "sharegpt_ZUxBndS_26",
+ "sharegpt_mMdyu3t_0",
+ "836f5ae4",
+ "9f8c432e_3",
+ "ultrachat_281183",
+ "a0b8b0ad_7"
+ ],
+ "gold_session_ids": [
+ "answer_ad8a76cb_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_a2d1d1f6",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_f6d6e33f_1",
+ "a21f3697_3",
+ "e40d733b_2",
+ "2c31d8be_3",
+ "9a584663_1",
+ "sharegpt_gI1U1eK_4",
+ "894dd0c2_3",
+ "534be93d_1",
+ "ultrachat_5344",
+ "a1bfb382_1"
+ ],
+ "gold_session_ids": [
+ "answer_f6d6e33f_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_85da3956",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_581ab834_1",
+ "14b000d9_1",
+ "ultrachat_373754",
+ "2fd445a2_4",
+ "sharegpt_ewad5Sr_0",
+ "sharegpt_opQbgPL_13",
+ "3a4012a5_4",
+ "3e6b6802_3",
+ "d8b3e1c8_1",
+ "e6fddf6c"
+ ],
+ "gold_session_ids": [
+ "answer_581ab834_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_b0863698",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_550bb2d1_1",
+ "91e581ad_3",
+ "4a79d078_2",
+ "5b014f9b_1",
+ "ce8b2a9e_2",
+ "554b054e_2",
+ "710833f3",
+ "b2fa108b_1",
+ "sharegpt_FPbCOuq_30",
+ "sharegpt_vGZ5lH1_21"
+ ],
+ "gold_session_ids": [
+ "answer_550bb2d1_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_68e94287",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8772153153380493,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_9793daa3_2",
+ "50f136f2_3",
+ "64460e9d_1",
+ "answer_9793daa3_1",
+ "74bcc8f4_1",
+ "edfa358d_1",
+ "sharegpt_XzAEFL4_0",
+ "dadd6379_1",
+ "50ca8a86",
+ "43502c3c_2"
+ ],
+ "gold_session_ids": [
+ "answer_9793daa3_2",
+ "answer_9793daa3_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_e414231e",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_e28c1f0d_2",
+ "answer_e28c1f0d_1",
+ "7e3fa056_2",
+ "d3065d85",
+ "9f8fc173_3",
+ "aee32463_1",
+ "0254514e_1",
+ "d4439279_2",
+ "sharegpt_CdnyHep_0",
+ "c87e6c56"
+ ],
+ "gold_session_ids": [
+ "answer_e28c1f0d_1",
+ "answer_e28c1f0d_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_7ca326fa",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_cf021b36_1",
+ "answer_cf021b36_3",
+ "answer_cf021b36_2",
+ "sharegpt_LDpwZD3_0",
+ "c70f9f9c",
+ "1301fbd1",
+ "8d0ca228",
+ "ultrachat_147001",
+ "sharegpt_7SCdbX2_0",
+ "5429e768_1"
+ ],
+ "gold_session_ids": [
+ "answer_cf021b36_1",
+ "answer_cf021b36_2",
+ "answer_cf021b36_3"
+ ]
+ },
+ {
+ "question_id": "gpt4_7bc6cf22",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_e22d6aef_1",
+ "3e43bf49_1",
+ "70b9616e",
+ "ultrachat_330542",
+ "09d021e7_1",
+ "ultrachat_215356",
+ "68396ed3_2",
+ "eca3b71c_2",
+ "e6cc6157_1",
+ "7a4d00b3_1"
+ ],
+ "gold_session_ids": [
+ "answer_e22d6aef_1"
+ ]
+ },
+ {
+ "question_id": "2ebe6c92",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6052602440527057,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "1ba79ea8_1",
+ "answer_c9d35c00_2",
+ "ca3a4e4f_1",
+ "d9e023f2_2",
+ "9d4312f6_3",
+ "answer_c9d35c00_1",
+ "719a18dd_2",
+ "acbb04c7",
+ "16756728_1",
+ "sharegpt_iQp7qQ2_0"
+ ],
+ "gold_session_ids": [
+ "answer_c9d35c00_1",
+ "answer_c9d35c00_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_e061b84g",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.5307212739772434,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "9dac9e37_1",
+ "answer_8c64ce26_3",
+ "answer_8c64ce26_2",
+ "3929e6cc_1",
+ "ae15e8b6_1",
+ "sharegpt_i33zh1e_15",
+ "ce584ba0",
+ "68705467_1",
+ "ultrachat_547960",
+ "51221cac"
+ ],
+ "gold_session_ids": [
+ "answer_8c64ce26_2",
+ "answer_8c64ce26_1",
+ "answer_8c64ce26_3"
+ ]
+ },
+ {
+ "question_id": "71017277",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_0b4a8adc_1",
+ "f8de4e92_4",
+ "ultrachat_557308",
+ "ad175dc4_5",
+ "0aacd15f_5",
+ "6de70f46_2",
+ "85a1be56_2",
+ "fd749c9a_1",
+ "451ecb23_3",
+ "e64f9553_1"
+ ],
+ "gold_session_ids": [
+ "answer_0b4a8adc_1"
+ ]
+ },
+ {
+ "question_id": "b46e15ee",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9619991470595832,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_4bfcc251_4",
+ "answer_4bfcc251_1",
+ "answer_4bfcc251_2",
+ "495cf518_2",
+ "d4446e25_2",
+ "e03c71c5_1",
+ "answer_4bfcc251_3",
+ "ultrachat_436509",
+ "f3fcf4fd_1",
+ "c92239bc"
+ ],
+ "gold_session_ids": [
+ "answer_4bfcc251_4",
+ "answer_4bfcc251_3",
+ "answer_4bfcc251_2",
+ "answer_4bfcc251_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_d6585ce9",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8738100898801323,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_f999b05c_5",
+ "58dd3f35",
+ "answer_f999b05c_4",
+ "a554ed79_4",
+ "answer_f999b05c_2",
+ "answer_f999b05c_1",
+ "answer_f999b05c_3",
+ "b3763b6b_1",
+ "a15fa6eb",
+ "a5532feb_2"
+ ],
+ "gold_session_ids": [
+ "answer_f999b05c_1",
+ "answer_f999b05c_4",
+ "answer_f999b05c_2",
+ "answer_f999b05c_5",
+ "answer_f999b05c_3"
+ ]
+ },
+ {
+ "question_id": "gpt4_1e4a8aec",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.20438239758848611,
+ "mrr": 0.14285714285714285,
+ "retrieved_session_ids": [
+ "9f1f3bf4",
+ "b1265fd7_1",
+ "e675c69a_1",
+ "1c662b7b_1",
+ "b34a52aa_3",
+ "0b635369_2",
+ "answer_16bd5ea6_1",
+ "sharegpt_sRhcMiu_0",
+ "4c8455cb_2",
+ "743b1fc3_2"
+ ],
+ "gold_session_ids": [
+ "answer_16bd5ea6_1",
+ "answer_16bd5ea6_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_f420262d",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_d8a1af6c_2",
+ "answer_d8a1af6c_5",
+ "answer_d8a1af6c_4",
+ "answer_d8a1af6c_1",
+ "answer_d8a1af6c_3",
+ "27639dd8_3",
+ "dfd4e4b7_1",
+ "1032cbcf_1",
+ "534716de",
+ "a7c62fd2_2"
+ ],
+ "gold_session_ids": [
+ "answer_d8a1af6c_1",
+ "answer_d8a1af6c_2",
+ "answer_d8a1af6c_5",
+ "answer_d8a1af6c_4",
+ "answer_d8a1af6c_3"
+ ]
+ },
+ {
+ "question_id": "gpt4_59149c78",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8065735963827293,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_d00ba6d1_1",
+ "23754665",
+ "ultrachat_463998",
+ "765ce8a7_2",
+ "a8ac3d1d_1",
+ "ultrachat_131385",
+ "665f8750_1",
+ "answer_d00ba6d1_2",
+ "78e40ca2_2",
+ "9848a1e3_2"
+ ],
+ "gold_session_ids": [
+ "answer_d00ba6d1_1",
+ "answer_d00ba6d1_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_e414231f",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9197207891481876,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_e28c1f0e_1",
+ "ultrachat_150948",
+ "answer_e28c1f0e_2",
+ "752c438c_3",
+ "deb5bb60",
+ "78ef59d0_1",
+ "sharegpt_p57YcDV_15",
+ "3acb50ef_4",
+ "135f761b_1",
+ "sharegpt_mFa5dxA_11"
+ ],
+ "gold_session_ids": [
+ "answer_e28c1f0e_1",
+ "answer_e28c1f0e_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_4929293b",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.20438239758848611,
+ "mrr": 0.14285714285714285,
+ "retrieved_session_ids": [
+ "7fa98614_1",
+ "ultrachat_342539",
+ "ultrachat_156176",
+ "ultrachat_326769",
+ "cae65795_1",
+ "sharegpt_bq9G6bT_0",
+ "answer_add9b013_2",
+ "bda611f6_3",
+ "e76430ea_3",
+ "ea8c38f8"
+ ],
+ "gold_session_ids": [
+ "answer_add9b013_2",
+ "answer_add9b013_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_468eb064",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.2890648263178879,
+ "mrr": 0.1,
+ "retrieved_session_ids": [
+ "sharegpt_wrwD6bG_0",
+ "c3023a45_1",
+ "sharegpt_0PLBHdX_15",
+ "0944933f_1",
+ "25e5b797_1",
+ "a2f0054f",
+ "db1aefb6_3",
+ "ultrachat_59629",
+ "1e5bd28d_2",
+ "answer_9b09d95b_1"
+ ],
+ "gold_session_ids": [
+ "answer_9b09d95b_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_fa19884d",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8315546295836225,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ff201787_2",
+ "ccc87da9_1",
+ "588a2123",
+ "f910dc31_4",
+ "5f9dd782",
+ "answer_ff201787_1",
+ "b35fadaa",
+ "040615bc_1",
+ "ebd2a973",
+ "8f8f2ced_1"
+ ],
+ "gold_session_ids": [
+ "answer_ff201787_1",
+ "answer_ff201787_2"
+ ]
+ },
+ {
+ "question_id": "9a707b82",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.5,
+ "mrr": 0.3333333333333333,
+ "retrieved_session_ids": [
+ "990f3ef9_2",
+ "fab41c07",
+ "answer_dba89488_1",
+ "f8e9d4fa",
+ "68d35085_1",
+ "8e78fa70_1",
+ "ultrachat_466884",
+ "answer_dba89488_2",
+ "7a4d00b3_2",
+ "e627b1aa_1"
+ ],
+ "gold_session_ids": [
+ "answer_dba89488_2",
+ "answer_dba89488_1"
+ ]
+ },
+ {
+ "question_id": "eac54add",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.38685280723454163,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "7bb01253",
+ "answer_0d4d0348_1",
+ "ultrachat_91693",
+ "2946bf12",
+ "773aebbd_3",
+ "577da76d_2",
+ "ultrachat_134291",
+ "sharegpt_PJmlduQ_0",
+ "79987901_2",
+ "e552e1f9_2"
+ ],
+ "gold_session_ids": [
+ "answer_0d4d0348_1",
+ "answer_0d4d0348_2"
+ ]
+ },
+ {
+ "question_id": "4dfccbf8",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 0,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0,
+ "mrr": 0.0625,
+ "retrieved_session_ids": [
+ "953aaad5_3",
+ "7db28e68_1",
+ "87317e05_1",
+ "edd89480_1",
+ "0ae4c9b0_2",
+ "a63ad8e3_3",
+ "1c177942_4",
+ "sharegpt_3D3oQC0_213",
+ "sharegpt_oXgiN7q_131",
+ "b4f63a70_3"
+ ],
+ "gold_session_ids": [
+ "answer_4bebc783_1",
+ "answer_4bebc783_2"
+ ]
+ },
+ {
+ "question_id": "0bc8ad93",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_f4ea84fc_3",
+ "answer_f4ea84fc_1",
+ "answer_f4ea84fc_2",
+ "sharegpt_HOO59Ue_99",
+ "dd25eeb5_2",
+ "85736a07_1",
+ "ultrachat_260376",
+ "2c185a2b_1",
+ "sharegpt_afCogMg_34",
+ "7285299a_2"
+ ],
+ "gold_session_ids": [
+ "answer_f4ea84fc_3",
+ "answer_f4ea84fc_2",
+ "answer_f4ea84fc_1"
+ ]
+ },
+ {
+ "question_id": "6e984302",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8503449055347546,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_88841f27_1",
+ "41dc5d45_1",
+ "c03fc56c_1",
+ "7cd7c296_1",
+ "answer_88841f27_2",
+ "b7702ee4_1",
+ "7285299a_8",
+ "33a39aa7_3",
+ "cf3dc165_1",
+ "ultrachat_542864"
+ ],
+ "gold_session_ids": [
+ "answer_88841f27_1",
+ "answer_88841f27_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_8279ba03",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.5,
+ "mrr": 0.3333333333333333,
+ "retrieved_session_ids": [
+ "49b78a55_1",
+ "bb107057_2",
+ "answer_56521e66_1",
+ "864a563d_5",
+ "50d66391_4",
+ "518d26d3_4",
+ "c4661b3c_2",
+ "ultrachat_526748",
+ "084bb880_2",
+ "d9868305_1"
+ ],
+ "gold_session_ids": [
+ "answer_56521e66_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_b5700ca0",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6309297535714575,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "109e07e8_2",
+ "answer_a17423e8_1",
+ "ultrachat_540666",
+ "b403d6f0_4",
+ "534be93d_1",
+ "064f42ee_3",
+ "e789afdb_2",
+ "0f09447b",
+ "c4251d8d_2",
+ "14b000d9_4"
+ ],
+ "gold_session_ids": [
+ "answer_a17423e8_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_68e94288",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6131471927654584,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_9793daa4_1",
+ "128f4e4d_2",
+ "ultrachat_74747",
+ "ba80721c",
+ "9d2fdc71",
+ "d2606fe9",
+ "sharegpt_RD37kJL_31",
+ "d8454317_7",
+ "4c49e37f",
+ "ultrachat_301272"
+ ],
+ "gold_session_ids": [
+ "answer_9793daa4_2",
+ "answer_9793daa4_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_2655b836",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_4be1b6b4_2",
+ "answer_4be1b6b4_1",
+ "answer_4be1b6b4_3",
+ "d8a1af6b_4",
+ "sharegpt_KzDwaf1_237",
+ "sharegpt_Tv9eXvf_0",
+ "sharegpt_CUs8n0A_0",
+ "ea8cc87b",
+ "f220dd04_1",
+ "e6120ff9_2"
+ ],
+ "gold_session_ids": [
+ "answer_4be1b6b4_1",
+ "answer_4be1b6b4_2",
+ "answer_4be1b6b4_3"
+ ]
+ },
+ {
+ "question_id": "gpt4_2487a7cb",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_1c6b85ea_2",
+ "answer_1c6b85ea_1",
+ "4f5880c6_6",
+ "67a8ef5f",
+ "4c1982d8_2",
+ "f87fea58_4",
+ "b7901ed7",
+ "sharegpt_QN26oUg_0",
+ "3afa6478",
+ "sharegpt_gLb26GY_0"
+ ],
+ "gold_session_ids": [
+ "answer_1c6b85ea_2",
+ "answer_1c6b85ea_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_76048e76",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_b535969f_1",
+ "answer_b535969f_2",
+ "sharegpt_vx14orQ_11",
+ "33630c9e_2",
+ "6a78e959_2",
+ "e2d9bcfe",
+ "0125f748_1",
+ "sharegpt_JprREQx_0",
+ "33a39aa7_2",
+ "907b6ed7_1"
+ ],
+ "gold_session_ids": [
+ "answer_b535969f_1",
+ "answer_b535969f_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_2312f94c",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_5328c3c2_2",
+ "answer_5328c3c2_1",
+ "sharegpt_XYtuANl_9",
+ "e831a29f_2",
+ "ultrachat_437795",
+ "3db86aab_1",
+ "a764b677_2",
+ "00ebfcf5",
+ "5e39a233_1",
+ "f34bf5e1"
+ ],
+ "gold_session_ids": [
+ "answer_5328c3c2_2",
+ "answer_5328c3c2_1"
+ ]
+ },
+ {
+ "question_id": "0bb5a684",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9197207891481876,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_e936197f_1",
+ "826d51da_3",
+ "answer_e936197f_2",
+ "565929a2_2",
+ "3c8e7c0e_3",
+ "ultrachat_62919",
+ "3548cae3",
+ "d794dec9_2",
+ "sharegpt_AzkYB5s_0",
+ "sharegpt_6dPUCcK_13"
+ ],
+ "gold_session_ids": [
+ "answer_e936197f_1",
+ "answer_e936197f_2"
+ ]
+ },
+ {
+ "question_id": "08f4fc43",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_6ea1541e_2",
+ "answer_6ea1541e_1",
+ "58d11d0c_2",
+ "sharegpt_wXqtiUe_0",
+ "9216c4d3_4",
+ "cdbd6862_1",
+ "sharegpt_JJu53iW_0",
+ "ultrachat_359609",
+ "ultrachat_292416",
+ "f3bd00ec_2"
+ ],
+ "gold_session_ids": [
+ "answer_6ea1541e_2",
+ "answer_6ea1541e_1"
+ ]
+ },
+ {
+ "question_id": "2c63a862",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.5706417189553201,
+ "mrr": 0.3333333333333333,
+ "retrieved_session_ids": [
+ "ad175dc4_2",
+ "33baaaab_1",
+ "answer_d39b7977_2",
+ "answer_d39b7977_1",
+ "sharegpt_lXtWJ7d_0",
+ "918c4788_2",
+ "1854abc7_2",
+ "0fccc817_1",
+ "ultrachat_179062",
+ "b25f9c97_3"
+ ],
+ "gold_session_ids": [
+ "answer_d39b7977_1",
+ "answer_d39b7977_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_385a5000",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_7a4a93f1_2",
+ "answer_7a4a93f1_1",
+ "8062f4ab_2",
+ "dc832f73",
+ "cae65795_1",
+ "28f74b09_1",
+ "ultrachat_286149",
+ "ultrachat_480527",
+ "90019a03_2",
+ "sharegpt_A4N0rHY_13"
+ ],
+ "gold_session_ids": [
+ "answer_7a4a93f1_2",
+ "answer_7a4a93f1_1"
+ ]
+ },
+ {
+ "question_id": "2a1811e2",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_1cc3cd0c_1",
+ "answer_1cc3cd0c_2",
+ "ultrachat_187883",
+ "1aaf8057_3",
+ "89e05490",
+ "aac025d1_1",
+ "b0e040c1",
+ "b94ad1bc_4",
+ "88ad913f_2",
+ "sharegpt_v2b1TzW_35"
+ ],
+ "gold_session_ids": [
+ "answer_1cc3cd0c_1",
+ "answer_1cc3cd0c_2"
+ ]
+ },
+ {
+ "question_id": "bbf86515",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8772153153380493,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_b3763b6b_1",
+ "75b98d2a_1",
+ "592cb8d2",
+ "answer_b3763b6b_2",
+ "df003c93_2",
+ "e6fad20f",
+ "sharegpt_GrLzv0V_432",
+ "05260a4b_2",
+ "6cd203f7_2",
+ "e9e19917_1"
+ ],
+ "gold_session_ids": [
+ "answer_b3763b6b_1",
+ "answer_b3763b6b_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_5dcc0aab",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_099c1b6c_3",
+ "answer_099c1b6c_5",
+ "answer_099c1b6c_4",
+ "answer_099c1b6c_1",
+ "answer_099c1b6c_2",
+ "3891baea_3",
+ "353d3c6d_1",
+ "ultrachat_156812",
+ "b10c041b_1",
+ "3a192d16_1"
+ ],
+ "gold_session_ids": [
+ "answer_099c1b6c_5",
+ "answer_099c1b6c_2",
+ "answer_099c1b6c_4",
+ "answer_099c1b6c_3",
+ "answer_099c1b6c_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_0b2f1d21",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_c4e5d969_1",
+ "answer_c4e5d969_2",
+ "sharegpt_hDjMGr7_30",
+ "f31a0422_1",
+ "1cafd864_3",
+ "sharegpt_35udPn0_1",
+ "sharegpt_ItYxNpw_0",
+ "sharegpt_9JZCfee_42",
+ "ultrachat_425364",
+ "sharegpt_rhWNt54_19"
+ ],
+ "gold_session_ids": [
+ "answer_c4e5d969_2",
+ "answer_c4e5d969_1"
+ ]
+ },
+ {
+ "question_id": "f0853d11",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_932ea3bf_1",
+ "answer_932ea3bf_2",
+ "ultrachat_114660",
+ "ultrachat_32771",
+ "8c68df0d_2",
+ "sharegpt_Dcshjlv_33",
+ "b16a8656_2",
+ "661e2cf8_3",
+ "ultrachat_325202",
+ "ultrachat_78285"
+ ],
+ "gold_session_ids": [
+ "answer_932ea3bf_2",
+ "answer_932ea3bf_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_6ed717ea",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_d50a8a33_2",
+ "answer_d50a8a33_1",
+ "15ed03f0_1",
+ "e0585cb5_1",
+ "637bc32a_2",
+ "50940cb7_1",
+ "24bfc009_3",
+ "f849dd04",
+ "d4b6ec0b",
+ "sharegpt_nccc9Ke_21"
+ ],
+ "gold_session_ids": [
+ "answer_d50a8a33_1",
+ "answer_d50a8a33_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_70e84552",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_b3070ec4_2",
+ "answer_b3070ec4_1",
+ "67a8ef5f",
+ "2f8a3459_2",
+ "30491f6f_1",
+ "sharegpt_znI61kP_0",
+ "sharegpt_d7bP5fb_0",
+ "f479774c",
+ "ultrachat_573894",
+ "ultrachat_468422"
+ ],
+ "gold_session_ids": [
+ "answer_b3070ec4_2",
+ "answer_b3070ec4_1"
+ ]
+ },
+ {
+ "question_id": "a3838d2b",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9796825595768971,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_4ffa04a2_1",
+ "answer_4ffa04a2_6",
+ "answer_4ffa04a2_2",
+ "answer_4ffa04a2_3",
+ "answer_4ffa04a2_5",
+ "06317e04",
+ "e8de3056_1",
+ "ae73b0f7_2",
+ "ultrachat_154465",
+ "answer_4ffa04a2_4"
+ ],
+ "gold_session_ids": [
+ "answer_4ffa04a2_1",
+ "answer_4ffa04a2_6",
+ "answer_4ffa04a2_4",
+ "answer_4ffa04a2_3",
+ "answer_4ffa04a2_5",
+ "answer_4ffa04a2_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_93159ced",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_e5131a1b_2",
+ "answer_e5131a1b_1",
+ "17d91635_1",
+ "460be54a",
+ "sharegpt_A83arI9_383",
+ "ultrachat_387032",
+ "61a989b5_3",
+ "ultrachat_256765",
+ "263efe64_1",
+ "ultrachat_324774"
+ ],
+ "gold_session_ids": [
+ "answer_e5131a1b_2",
+ "answer_e5131a1b_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_2d58bcd6",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_3e11e0ae_2",
+ "answer_3e11e0ae_1",
+ "0bd0a01c",
+ "1b205d70_1",
+ "1bf8b212",
+ "sharegpt_XUxvLWu_0",
+ "sharegpt_4tLDGmp_0",
+ "be32ad25_1",
+ "sharegpt_Ntr6bOu_0",
+ "5e50c81b"
+ ],
+ "gold_session_ids": [
+ "answer_3e11e0ae_1",
+ "answer_3e11e0ae_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_65aabe59",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9197207891481876,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_30dfe889_1",
+ "6a7e3aec",
+ "answer_30dfe889_2",
+ "ultrachat_190669",
+ "ultrachat_509625",
+ "f0158960",
+ "ultrachat_386890",
+ "ultrachat_216833",
+ "sharegpt_F0NKEOn_2",
+ "ultrachat_16520"
+ ],
+ "gold_session_ids": [
+ "answer_30dfe889_1",
+ "answer_30dfe889_2"
+ ]
+ },
+ {
+ "question_id": "982b5123",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ab603dd5_1",
+ "answer_ab603dd5_2",
+ "bf8ab267",
+ "e36b0031",
+ "1bfd5a8b_3",
+ "07ba9acd_2",
+ "a200b713_4",
+ "04e05522_2",
+ "ce8b2a9e_1",
+ "359a2a0d"
+ ],
+ "gold_session_ids": [
+ "answer_ab603dd5_1",
+ "answer_ab603dd5_2"
+ ]
+ },
+ {
+ "question_id": "b9cfe692",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_5e3bb940_2",
+ "answer_5e3bb940_1",
+ "answer_5e3bb940_3",
+ "sharegpt_JRQtEcd_115",
+ "0c27c633_2",
+ "sharegpt_oXgiN7q_53",
+ "da650f3b",
+ "2c7bff3a_1",
+ "91d847c1_6",
+ "sharegpt_ErOTMZ3_149"
+ ],
+ "gold_session_ids": [
+ "answer_5e3bb940_3",
+ "answer_5e3bb940_2",
+ "answer_5e3bb940_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_4edbafa2",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_0a00c163_2",
+ "answer_0a00c163_1",
+ "a6f2320f_2",
+ "7a7a4bf0_3",
+ "7533d546_5",
+ "d36a36f2",
+ "2ba7fdd0_2",
+ "22be78fc_2",
+ "a65d076a",
+ "3f5249ee_3"
+ ],
+ "gold_session_ids": [
+ "answer_0a00c163_1",
+ "answer_0a00c163_2"
+ ]
+ },
+ {
+ "question_id": "c8090214",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8772153153380493,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_70dc7d08_2",
+ "162ad3bf_1",
+ "c21c9d97",
+ "answer_70dc7d08_1",
+ "005ec7cf_1",
+ "sharegpt_yXAlFu3_0",
+ "sharegpt_xOzzaof_0",
+ "5b415488_2",
+ "ultrachat_518687",
+ "f2939479_1"
+ ],
+ "gold_session_ids": [
+ "answer_70dc7d08_2",
+ "answer_70dc7d08_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_483dd43c",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_fb793c87_1",
+ "answer_fb793c87_2",
+ "73353aee_3",
+ "ultrachat_375572",
+ "3fd56ebf",
+ "9d0f4bfa_2",
+ "sharegpt_I6IzvEF_0",
+ "sharegpt_ItFPuha_0",
+ "de9f265f",
+ "4798296c_1"
+ ],
+ "gold_session_ids": [
+ "answer_fb793c87_1",
+ "answer_fb793c87_2"
+ ]
+ },
+ {
+ "question_id": "e4e14d04",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_cf425855_2",
+ "answer_cf425855_1",
+ "5944b36a_1",
+ "sharegpt_HO1gGmm_0",
+ "ffc627f2",
+ "d64a0896",
+ "f8de4e92_2",
+ "1c1f5ccc_3",
+ "ultrachat_200098",
+ "a92165a5"
+ ],
+ "gold_session_ids": [
+ "answer_cf425855_1",
+ "answer_cf425855_2"
+ ]
+ },
+ {
+ "question_id": "c9f37c46",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_cdba3d9f_2",
+ "answer_cdba3d9f_1",
+ "c3d6a282_1",
+ "0cf86cbe_1",
+ "b38766c1_3",
+ "4ffa04a2_1",
+ "9d4312f6_3",
+ "sharegpt_94CjWOc_0",
+ "211bb1c2_1",
+ "50136b31"
+ ],
+ "gold_session_ids": [
+ "answer_cdba3d9f_1",
+ "answer_cdba3d9f_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_2c50253f",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_9af4e346_2",
+ "answer_9af4e346_1",
+ "4b9ed528_2",
+ "4cef8a3c_3",
+ "sharegpt_Bmic1Id_51",
+ "0138fdec",
+ "57c8bf62_1",
+ "69a9211c_3",
+ "b6f35f3a_1",
+ "sharegpt_CWlW9UH_0"
+ ],
+ "gold_session_ids": [
+ "answer_9af4e346_1",
+ "answer_9af4e346_2"
+ ]
+ },
+ {
+ "question_id": "dcfa8644",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_5e3eeb12_2",
+ "answer_5e3eeb12_1",
+ "b2243528_1",
+ "sharegpt_oG3LXXo_6",
+ "f36d82f7",
+ "29a7a78d_1",
+ "09f15689",
+ "d15d2899_2",
+ "30bbae12_2",
+ "41f94af6"
+ ],
+ "gold_session_ids": [
+ "answer_5e3eeb12_2",
+ "answer_5e3eeb12_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_b4a80587",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_b9d9150e_1",
+ "answer_b9d9150e_2",
+ "sharegpt_inuIr9J_119",
+ "f79a48dd_1",
+ "sharegpt_HxOhjmq_29",
+ "3cb41ab8_1",
+ "55bcfb77",
+ "sharegpt_zLYVdSv_0",
+ "866de400",
+ "ultrachat_175745"
+ ],
+ "gold_session_ids": [
+ "answer_b9d9150e_2",
+ "answer_b9d9150e_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_9a159967",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_8a42fedf_1",
+ "answer_8a42fedf_3",
+ "answer_8a42fedf_2",
+ "4ea94f85",
+ "75671e3f_1",
+ "6c9bcb0f_1",
+ "b2cae160_1",
+ "ca529c5b_2",
+ "ultrachat_534927",
+ "0df54428_4"
+ ],
+ "gold_session_ids": [
+ "answer_8a42fedf_1",
+ "answer_8a42fedf_3",
+ "answer_8a42fedf_2"
+ ]
+ },
+ {
+ "question_id": "cc6d1ec1",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_be73098b_1",
+ "answer_be73098b_2",
+ "3cb41ab8_1",
+ "22db3cc3_4",
+ "8a42fedf_2",
+ "sharegpt_ckqShdI_21",
+ "ultrachat_508911",
+ "33baaaab_3",
+ "e549b6b2_2",
+ "336b281b_2"
+ ],
+ "gold_session_ids": [
+ "answer_be73098b_2",
+ "answer_be73098b_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_8c8961ae",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_72d9aa58_1",
+ "answer_72d9aa58_2",
+ "3b22d17b",
+ "37f0ce6b_1",
+ "05fad1cb_2",
+ "10466528_3",
+ "ultrachat_334521",
+ "1cde7ee4_2",
+ "b77d0b58",
+ "ultrachat_197141"
+ ],
+ "gold_session_ids": [
+ "answer_72d9aa58_1",
+ "answer_72d9aa58_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_d9af6064",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_da704e79_1",
+ "answer_da704e79_2",
+ "ultrachat_182531",
+ "sharegpt_VbIxRWJ_33",
+ "03b4d5d8_2",
+ "ultrachat_519452",
+ "sharegpt_w9YUs4c_0",
+ "a1b3dd27_1",
+ "ff49b5a5_2",
+ "0d9a6f01"
+ ],
+ "gold_session_ids": [
+ "answer_da704e79_2",
+ "answer_da704e79_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_7de946e7",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_6a78e959_2",
+ "answer_6a78e959_1",
+ "caa00337_2",
+ "58bec5fb_3",
+ "ultrachat_105249",
+ "3023b36f",
+ "ultrachat_293723",
+ "ultrachat_167604",
+ "sharegpt_6Yfj9A3_5",
+ "6155addd_2"
+ ],
+ "gold_session_ids": [
+ "answer_6a78e959_2",
+ "answer_6a78e959_1"
+ ]
+ },
+ {
+ "question_id": "d01c6aa8",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8503449055347546,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_991d55e5_1",
+ "fc9a62c0_1",
+ "7d1f5395",
+ "a68ad96e_1",
+ "answer_991d55e5_2",
+ "sharegpt_xd1G5uV_0",
+ "64460e9d_1",
+ "ultrachat_10123",
+ "fbd67491_2",
+ "5024758e"
+ ],
+ "gold_session_ids": [
+ "answer_991d55e5_1",
+ "answer_991d55e5_2"
+ ]
+ },
+ {
+ "question_id": "993da5e2",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_54f0d6f9_2",
+ "answer_54f0d6f9_1",
+ "12cd736c",
+ "ultrachat_217815",
+ "ultrachat_73982",
+ "13f3da23_2",
+ "8953c8c8_3",
+ "b16a8656_2",
+ "c756d93e",
+ "3ac33554_2"
+ ],
+ "gold_session_ids": [
+ "answer_54f0d6f9_1",
+ "answer_54f0d6f9_2"
+ ]
+ },
+ {
+ "question_id": "a3045048",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.5912352048230277,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "aaf71ce2_2",
+ "answer_016f6bd4_1",
+ "d12d5a98_1",
+ "2def525b_1",
+ "sharegpt_asoSnhq_0",
+ "sharegpt_YblmreD_53",
+ "answer_016f6bd4_2",
+ "sharegpt_LC9Pzkr_0",
+ "ultrachat_19072",
+ "536a334d"
+ ],
+ "gold_session_ids": [
+ "answer_016f6bd4_2",
+ "answer_016f6bd4_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_d31cdae3",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6131471927654584,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_8464304d_2",
+ "de695a4e",
+ "55033f6f",
+ "ultrachat_324099",
+ "sharegpt_HDF4645_0",
+ "sharegpt_g6KbKeo_0",
+ "41743aae_2",
+ "809cabca_2",
+ "f87fea58_1",
+ "sharegpt_7W1Sg8f_0"
+ ],
+ "gold_session_ids": [
+ "answer_8464304d_1",
+ "answer_8464304d_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_cd90e484",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_aa930b56_2",
+ "answer_aa930b56_1",
+ "6043dcf6_1",
+ "4bfcc251_1",
+ "866de400",
+ "08c306e2",
+ "ultrachat_347068",
+ "61b27d67_3",
+ "018eac29",
+ "cc5a6057_2"
+ ],
+ "gold_session_ids": [
+ "answer_aa930b56_1",
+ "answer_aa930b56_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_88806d6e",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_e60a93ff_1",
+ "answer_e60a93ff_2",
+ "9ac3fa82_1",
+ "24bfc009_5",
+ "sharegpt_WT6LnXo_0",
+ "743b1fc3_2",
+ "sharegpt_5RbA8hu_0",
+ "sharegpt_mMdyu3t_0",
+ "3c11dc79",
+ "c11fb634_1"
+ ],
+ "gold_session_ids": [
+ "answer_e60a93ff_1",
+ "answer_e60a93ff_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_4cd9eba1",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_5fcca8bc_1",
+ "answer_5fcca8bc_2",
+ "1b576f6e_2",
+ "d36a862c_1",
+ "sharegpt_3hCiKrk_0",
+ "dafc2440_2",
+ "456860e7_2",
+ "6b9b2b1e_1",
+ "8242fa3f_1",
+ "ultrachat_221041"
+ ],
+ "gold_session_ids": [
+ "answer_5fcca8bc_2",
+ "answer_5fcca8bc_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_93f6379c",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9469024295259745,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_544fe66c_3",
+ "answer_544fe66c_2",
+ "sharegpt_cfESFSo_0",
+ "aee99715",
+ "answer_544fe66c_1",
+ "b8a688f6",
+ "sharegpt_ZY5xXZa_7",
+ "sharegpt_y7eQRmk_0",
+ "de695a4e",
+ "c07b8034"
+ ],
+ "gold_session_ids": [
+ "answer_544fe66c_2",
+ "answer_544fe66c_1",
+ "answer_544fe66c_3"
+ ]
+ },
+ {
+ "question_id": "b29f3365",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_436d4309_2",
+ "answer_436d4309_1",
+ "6b7605d1_1",
+ "6a2f9452_1",
+ "06ee3665",
+ "sharegpt_WjkZjl4_28",
+ "53785a15",
+ "1dae02fe",
+ "c279c578",
+ "4ccc2061"
+ ],
+ "gold_session_ids": [
+ "answer_436d4309_2",
+ "answer_436d4309_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_2f56ae70",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_7a36e820_2",
+ "answer_7a36e820_3",
+ "answer_7a36e820_1",
+ "647f5851",
+ "dc3ee1d1_1",
+ "c578da1f_2",
+ "541ecc45_2",
+ "f2835879_2",
+ "2889e785_1",
+ "74c8f7bb"
+ ],
+ "gold_session_ids": [
+ "answer_7a36e820_2",
+ "answer_7a36e820_3",
+ "answer_7a36e820_1"
+ ]
+ },
+ {
+ "question_id": "6613b389",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9674679834891693,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_aaf71ce2_2",
+ "answer_aaf71ce2_3",
+ "e3d8da67",
+ "answer_aaf71ce2_1",
+ "86c8f348_1",
+ "ultrachat_31390",
+ "20d6c376_1",
+ "df8a7455",
+ "f1cc1ddb_1",
+ "1a65880d"
+ ],
+ "gold_session_ids": [
+ "answer_aaf71ce2_3",
+ "answer_aaf71ce2_2",
+ "answer_aaf71ce2_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_78cf46a3",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6934264036172708,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "sharegpt_rx39ipj_0",
+ "answer_5a78688d_1",
+ "answer_5a78688d_2",
+ "ultrachat_305042",
+ "52b19cba_1",
+ "ultrachat_370708",
+ "42b45f6e_2",
+ "sharegpt_ZWqMvoL_593",
+ "b89a3ec3",
+ "e8e65a1f"
+ ],
+ "gold_session_ids": [
+ "answer_5a78688d_1",
+ "answer_5a78688d_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_0a05b494",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_a68db5db_2",
+ "answer_a68db5db_1",
+ "0abe3e51",
+ "ultrachat_273213",
+ "sharegpt_eWQFoct_17",
+ "ab603dd5_2",
+ "56911dc5_6",
+ "8c4995d0",
+ "05b84f71_1",
+ "30677a32"
+ ],
+ "gold_session_ids": [
+ "answer_a68db5db_2",
+ "answer_a68db5db_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_1a1dc16d",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8772153153380493,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_faad7d7a_2",
+ "c12d5181_1",
+ "87f1f950_2",
+ "answer_faad7d7a_1",
+ "ultrachat_480665",
+ "8bd3814a_2",
+ "ultrachat_428346",
+ "sharegpt_6cz1Sq6_328",
+ "sharegpt_u1aJmpT_0",
+ "14940349_2"
+ ],
+ "gold_session_ids": [
+ "answer_faad7d7a_2",
+ "answer_faad7d7a_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_2f584639",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_11a8f823_1",
+ "answer_11a8f823_2",
+ "c6eb17f0_1",
+ "821242cd_2",
+ "13947b64_1",
+ "sharegpt_bq9G6bT_2",
+ "ultrachat_464469",
+ "ultrachat_430445",
+ "c3d89115",
+ "d441f4c5"
+ ],
+ "gold_session_ids": [
+ "answer_11a8f823_2",
+ "answer_11a8f823_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_213fd887",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_53582e7e_2",
+ "answer_53582e7e_1",
+ "9bdbfc62_2",
+ "3b0a7d5d_1",
+ "368276ab_3",
+ "3b38bf46",
+ "d033262b_2",
+ "ultrachat_165091",
+ "sharegpt_PvScEsS_0",
+ "fb3ec1ff"
+ ],
+ "gold_session_ids": [
+ "answer_53582e7e_2",
+ "answer_53582e7e_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_5438fa52",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.46845052016107697,
+ "mrr": 0.25,
+ "retrieved_session_ids": [
+ "a8e24e22",
+ "528f481d_6",
+ "d68bab3c_1",
+ "answer_b10f3828_1",
+ "ultrachat_69599",
+ "290fe09a",
+ "answer_b10f3828_2",
+ "ab8dce45",
+ "ultrachat_160450",
+ "828c3d75"
+ ],
+ "gold_session_ids": [
+ "answer_b10f3828_1",
+ "answer_b10f3828_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_c27434e8",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_d8e33f5c_2",
+ "answer_d8e33f5c_1",
+ "4d578deb",
+ "sharegpt_9XlbWvL_0",
+ "sharegpt_TRck0h4_0",
+ "11cd43df",
+ "ultrachat_556731",
+ "820c1a20",
+ "ultrachat_444552",
+ "ultrachat_70335"
+ ],
+ "gold_session_ids": [
+ "answer_d8e33f5c_1",
+ "answer_d8e33f5c_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_fe651585",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_65600ff6_2",
+ "answer_65600ff6_1",
+ "sharegpt_gupA3qh_0",
+ "sharegpt_mJ7MnQS_219",
+ "9f220c66_1",
+ "ultrachat_576354",
+ "sharegpt_izuEpAx_20",
+ "987b36db_3",
+ "e30c4a7b_2",
+ "bc1f1051_1"
+ ],
+ "gold_session_ids": [
+ "answer_65600ff6_2",
+ "answer_65600ff6_1"
+ ]
+ },
+ {
+ "question_id": "8c18457d",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_124f5dc3_1",
+ "answer_124f5dc3_2",
+ "98cd882c_2",
+ "c8dcbe7f_1",
+ "1a5369b9",
+ "ultrachat_417305",
+ "3c8e7c0e_2",
+ "4745bb2e",
+ "c3023a45_2",
+ "9f803df5_1"
+ ],
+ "gold_session_ids": [
+ "answer_124f5dc3_2",
+ "answer_124f5dc3_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_70e84552_abs",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_b3070ec4_abs_1",
+ "answer_b3070ec4_abs_2",
+ "f6ba44fe_1",
+ "e2b58a46_1",
+ "e2cd250e_2",
+ "430d0a87_2",
+ "4e0da76d",
+ "sharegpt_EypOKzQ_0",
+ "59264d33_2",
+ "ultrachat_475184"
+ ],
+ "gold_session_ids": [
+ "answer_b3070ec4_abs_2",
+ "answer_b3070ec4_abs_1"
+ ]
+ },
+ {
+ "question_id": "gpt4_93159ced_abs",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_e5131a1b_abs_1",
+ "answer_e5131a1b_abs_2",
+ "sharegpt_CyJ3dal_43",
+ "ultrachat_195762",
+ "56bba837",
+ "sharegpt_gYShON7_0",
+ "156988fe",
+ "2f097fba_1",
+ "76385c25",
+ "168feef7_2"
+ ],
+ "gold_session_ids": [
+ "answer_e5131a1b_abs_1",
+ "answer_e5131a1b_abs_2"
+ ]
+ },
+ {
+ "question_id": "982b5123_abs",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8175295903539445,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ab603dd5_abs_1",
+ "11e942da_1",
+ "fe82a033_1",
+ "4f3b0353_2",
+ "a8179676",
+ "3fc2244f",
+ "answer_ab603dd5_abs_2",
+ "b1793bba",
+ "5080119d_1",
+ "6e5130ed_1"
+ ],
+ "gold_session_ids": [
+ "answer_ab603dd5_abs_1",
+ "answer_ab603dd5_abs_2"
+ ]
+ },
+ {
+ "question_id": "c8090214_abs",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.38685280723454163,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "64c71782",
+ "answer_70dc7d08_abs_1",
+ "b192ae00_1",
+ "ultrachat_393699",
+ "d4f02577_2",
+ "ca3aa07f_1",
+ "ultrachat_129606",
+ "65c21f34_1",
+ "ultrachat_133575",
+ "fda42399_3"
+ ],
+ "gold_session_ids": [
+ "answer_70dc7d08_abs_1",
+ "answer_70dc7d08_abs_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_c27434e8_abs",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_d8e33f5c_abs_1",
+ "answer_d8e33f5c_abs_2",
+ "ultrachat_106944",
+ "fccc9400",
+ "ultrachat_552313",
+ "ae0cd456",
+ "sharegpt_hGtkNGL_12",
+ "2e28f3db",
+ "sharegpt_iYoVtKC_0",
+ "59a6150d_3"
+ ],
+ "gold_session_ids": [
+ "answer_d8e33f5c_abs_1",
+ "answer_d8e33f5c_abs_2"
+ ]
+ },
+ {
+ "question_id": "gpt4_fe651585_abs",
+ "question_type": "temporal-reasoning",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9197207891481876,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_65600ff6_abs_2",
+ "sharegpt_7PLgNll_0",
+ "answer_65600ff6_abs_1",
+ "sharegpt_v6ZRge5_0",
+ "sharegpt_TBRPvuE_0",
+ "sharegpt_wrN9uUo_53",
+ "caa00337_1",
+ "sharegpt_jVFHJsP_0",
+ "sharegpt_NMLgKqu_7",
+ "85a1c17b_1"
+ ],
+ "gold_session_ids": [
+ "answer_65600ff6_abs_1",
+ "answer_65600ff6_abs_2"
+ ]
+ },
+ {
+ "question_id": "6a1eabeb",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8772153153380493,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_a25d4a91_2",
+ "4aaf678f_3",
+ "afe04238_7",
+ "answer_a25d4a91_1",
+ "9b3a7f2c_1",
+ "6da0bba2_1",
+ "d03995c6",
+ "9f80607d",
+ "sharegpt_3Jpu3tr_13",
+ "4c49e37f"
+ ],
+ "gold_session_ids": [
+ "answer_a25d4a91_1",
+ "answer_a25d4a91_2"
+ ]
+ },
+ {
+ "question_id": "6aeb4375",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_3f9693b7_1",
+ "answer_3f9693b7_2",
+ "402e0082_5",
+ "29621a34",
+ "08256f84_2",
+ "1bc87711_1",
+ "sharegpt_MWlvrvZ_0",
+ "2fd43bba",
+ "03105808_2",
+ "dc9bf721_1"
+ ],
+ "gold_session_ids": [
+ "answer_3f9693b7_1",
+ "answer_3f9693b7_2"
+ ]
+ },
+ {
+ "question_id": "830ce83f",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.5706417189553201,
+ "mrr": 0.3333333333333333,
+ "retrieved_session_ids": [
+ "a7f88bb3_2",
+ "dfbda5e9",
+ "answer_0b1a0942_2",
+ "answer_0b1a0942_1",
+ "d1e7d11c_2",
+ "ultrachat_288747",
+ "ultrachat_441391",
+ "7ddf575b",
+ "7a389bd1_7",
+ "ultrachat_346816"
+ ],
+ "gold_session_ids": [
+ "answer_0b1a0942_1",
+ "answer_0b1a0942_2"
+ ]
+ },
+ {
+ "question_id": "852ce960",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_3a6f1e82_2",
+ "answer_3a6f1e82_1",
+ "010a28ab_3",
+ "sharegpt_FYhsZ0Q_0",
+ "a76e7e3c_1",
+ "e8bfacec_1",
+ "bdfcb4d3",
+ "sharegpt_eN4o9Zr_0",
+ "sharegpt_V2j1zkI_1",
+ "sharegpt_uiECyCD_66"
+ ],
+ "gold_session_ids": [
+ "answer_3a6f1e82_1",
+ "answer_3a6f1e82_2"
+ ]
+ },
+ {
+ "question_id": "945e3d21",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.40298313359773236,
+ "mrr": 0.16666666666666666,
+ "retrieved_session_ids": [
+ "ef0341d4_2",
+ "5fcca8bc_1",
+ "ultrachat_242046",
+ "f5025173_2",
+ "sharegpt_E7VUcfj_0",
+ "answer_6a4f8626_2",
+ "fc69fd44_7",
+ "940629b2_1",
+ "answer_6a4f8626_1",
+ "4019b1a8_4"
+ ],
+ "gold_session_ids": [
+ "answer_6a4f8626_1",
+ "answer_6a4f8626_2"
+ ]
+ },
+ {
+ "question_id": "d7c942c3",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.5706417189553201,
+ "mrr": 0.3333333333333333,
+ "retrieved_session_ids": [
+ "cf8e2761",
+ "ddf0f116_3",
+ "answer_eecb10d9_1",
+ "answer_eecb10d9_2",
+ "sharegpt_zxm712D_0",
+ "695bc054_1",
+ "sharegpt_vYQdLPe_2",
+ "8bd5b3c4_2",
+ "f0b803b0_2",
+ "4e70b8fd"
+ ],
+ "gold_session_ids": [
+ "answer_eecb10d9_1",
+ "answer_eecb10d9_2"
+ ]
+ },
+ {
+ "question_id": "71315a70",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_c44b9df4_2",
+ "answer_c44b9df4_1",
+ "a08fbe88_1",
+ "ultrachat_42515",
+ "8cfa91c3_1",
+ "b5e94516_2",
+ "37d634c0_1",
+ "ultrachat_399278",
+ "773aebbd_1",
+ "ultrachat_208238"
+ ],
+ "gold_session_ids": [
+ "answer_c44b9df4_1",
+ "answer_c44b9df4_2"
+ ]
+ },
+ {
+ "question_id": "89941a93",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_e1403127_2",
+ "answer_e1403127_1",
+ "1e88ebca",
+ "dde142b2_2",
+ "25956eab_1",
+ "169213ae",
+ "ultrachat_214088",
+ "ultrachat_453348",
+ "8e72316f",
+ "sharegpt_jxsIY6L_0"
+ ],
+ "gold_session_ids": [
+ "answer_e1403127_1",
+ "answer_e1403127_2"
+ ]
+ },
+ {
+ "question_id": "ce6d2d27",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8772153153380493,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_73540165_2",
+ "7a4d00b3_2",
+ "48ad2a93",
+ "answer_73540165_1",
+ "168776ed_1",
+ "cd92ac85_1",
+ "a1c4f65a",
+ "54f0d6f9_2",
+ "9f6bec4f",
+ "sharegpt_WQctw3k_0"
+ ],
+ "gold_session_ids": [
+ "answer_73540165_1",
+ "answer_73540165_2"
+ ]
+ },
+ {
+ "question_id": "9ea5eabc",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9197207891481876,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_02e66dec_2",
+ "6e2cca63_1",
+ "answer_02e66dec_1",
+ "8c8b180e",
+ "9bf5da2d",
+ "418c27fa",
+ "d1f30ac6_5",
+ "62e6593e_1",
+ "7c44bc2d_1",
+ "29695e1c_2"
+ ],
+ "gold_session_ids": [
+ "answer_02e66dec_1",
+ "answer_02e66dec_2"
+ ]
+ },
+ {
+ "question_id": "07741c44",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_7e9ad7b4_2",
+ "answer_7e9ad7b4_1",
+ "8d969f3e_2",
+ "a4b136d1_2",
+ "01d22d4d",
+ "98ae7eef",
+ "sharegpt_CEWG2XW_0",
+ "aa71db11_1",
+ "61fc1d85",
+ "f6b80ebe"
+ ],
+ "gold_session_ids": [
+ "answer_7e9ad7b4_1",
+ "answer_7e9ad7b4_2"
+ ]
+ },
+ {
+ "question_id": "a1eacc2a",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_0eb23770_2",
+ "answer_0eb23770_1",
+ "521255f7_2",
+ "1f035408",
+ "93f23301_1",
+ "1ccb1920",
+ "20d6c376_2",
+ "c910dd2c_3",
+ "sharegpt_ePBzYD0_57",
+ "ultrachat_187728"
+ ],
+ "gold_session_ids": [
+ "answer_0eb23770_1",
+ "answer_0eb23770_2"
+ ]
+ },
+ {
+ "question_id": "184da446",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9197207891481876,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_e2f4f947_2",
+ "bf633415_2",
+ "answer_e2f4f947_1",
+ "sharegpt_IMtsj65_0",
+ "61ad6074",
+ "341a737e",
+ "e5f08e40",
+ "30a3c930",
+ "sharegpt_nsHU229_0",
+ "82a89303"
+ ],
+ "gold_session_ids": [
+ "answer_e2f4f947_1",
+ "answer_e2f4f947_2"
+ ]
+ },
+ {
+ "question_id": "031748ae",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8503449055347546,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_8748f791_1",
+ "d0488050",
+ "5ac7f9c5_3",
+ "70764af1_4",
+ "answer_8748f791_2",
+ "ultrachat_435543",
+ "sharegpt_Ukl9zsG_0",
+ "920fa36f",
+ "sharegpt_pPNVaqR_0",
+ "sharegpt_3D7sTTH_0"
+ ],
+ "gold_session_ids": [
+ "answer_8748f791_1",
+ "answer_8748f791_2"
+ ]
+ },
+ {
+ "question_id": "4d6b87c8",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_766ab8da_2",
+ "answer_766ab8da_1",
+ "d87e86f6_1",
+ "sharegpt_xLjgprT_0",
+ "4dd737a9_1",
+ "5c85023a_1",
+ "60d4332d",
+ "sharegpt_yPCVYhX_0",
+ "a0201607_3",
+ "85da7827"
+ ],
+ "gold_session_ids": [
+ "answer_766ab8da_1",
+ "answer_766ab8da_2"
+ ]
+ },
+ {
+ "question_id": "0f05491a",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6052602440527057,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "9f4bec78",
+ "answer_d6d2eba8_1",
+ "057cbbc4_1",
+ "27aba903_2",
+ "sharegpt_HO1gGmm_4",
+ "answer_d6d2eba8_2",
+ "28000804",
+ "44678992_1",
+ "sharegpt_UpYQ1kh_79",
+ "d81d9846"
+ ],
+ "gold_session_ids": [
+ "answer_d6d2eba8_1",
+ "answer_d6d2eba8_2"
+ ]
+ },
+ {
+ "question_id": "08e075c7",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6509209298071326,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "25b3e36c_1",
+ "answer_cdbe2250_1",
+ "a2fa5ba3",
+ "answer_cdbe2250_2",
+ "0bd59f15_3",
+ "0fea08a2",
+ "5ff94163_1",
+ "385683f0_1",
+ "b76006cb_2",
+ "b1d9d555_1"
+ ],
+ "gold_session_ids": [
+ "answer_cdbe2250_1",
+ "answer_cdbe2250_2"
+ ]
+ },
+ {
+ "question_id": "f9e8c073",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.7977228895450266,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_b191df5b_1",
+ "ultrachat_88473",
+ "sharegpt_Fhigp7F_1",
+ "50b32eed_2",
+ "sharegpt_WQun2Qq_0",
+ "sharegpt_GDFyxeT_0",
+ "4604bc73_2",
+ "669318cf_3",
+ "answer_b191df5b_2",
+ "d5f4d9fa_3"
+ ],
+ "gold_session_ids": [
+ "answer_b191df5b_1",
+ "answer_b191df5b_2"
+ ]
+ },
+ {
+ "question_id": "41698283",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_c7ddc051_1",
+ "answer_c7ddc051_2",
+ "431ae25c",
+ "631e4016",
+ "f35e3cb7_1",
+ "0e8ba03b_1",
+ "8f3246db",
+ "bf386a67_3",
+ "4464626e_1",
+ "28e948f5"
+ ],
+ "gold_session_ids": [
+ "answer_c7ddc051_1",
+ "answer_c7ddc051_2"
+ ]
+ },
+ {
+ "question_id": "2698e78f",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8772153153380493,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_9282283d_2",
+ "a0fcf806",
+ "sharegpt_DrLV0RL_0",
+ "answer_9282283d_1",
+ "0fffb38f_2",
+ "ultrachat_455955",
+ "7ca763c3_2",
+ "d31c3ec8_4",
+ "ultrachat_110771",
+ "ultrachat_35438"
+ ],
+ "gold_session_ids": [
+ "answer_9282283d_1",
+ "answer_9282283d_2"
+ ]
+ },
+ {
+ "question_id": "b6019101",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_67074b4b_1",
+ "answer_67074b4b_2",
+ "de43030f_2",
+ "sharegpt_CvVLg2J_18",
+ "ultrachat_72569",
+ "0d66562c_4",
+ "ultrachat_338597",
+ "sharegpt_aVExml0_235",
+ "bf4fdf8f_1",
+ "ultrachat_242137"
+ ],
+ "gold_session_ids": [
+ "answer_67074b4b_1",
+ "answer_67074b4b_2"
+ ]
+ },
+ {
+ "question_id": "45dc21b6",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6131471927654584,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_07664d43_1",
+ "8acb29f9_1",
+ "0ad56ba6",
+ "0504a710_4",
+ "eb34144a_2",
+ "sharegpt_gGriMhQ_91",
+ "369695b4_2",
+ "sharegpt_mMdyu3t_0",
+ "de459c4b",
+ "ultrachat_494140"
+ ],
+ "gold_session_ids": [
+ "answer_07664d43_1",
+ "answer_07664d43_2"
+ ]
+ },
+ {
+ "question_id": "5a4f22c0",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_b0f3dfff_2",
+ "answer_b0f3dfff_1",
+ "195f2929",
+ "007e7d81_2",
+ "c3f945f3_1",
+ "f188b48d_1",
+ "sharegpt_I0uNTjm_22",
+ "ultrachat_208478",
+ "75302ce3",
+ "af778ece_1"
+ ],
+ "gold_session_ids": [
+ "answer_b0f3dfff_1",
+ "answer_b0f3dfff_2"
+ ]
+ },
+ {
+ "question_id": "6071bd76",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_4dac77cb_2",
+ "answer_4dac77cb_1",
+ "9de53d68_1",
+ "e6b6353d",
+ "sharegpt_3D3oQC0_12",
+ "8d0b4363",
+ "ultrachat_334948",
+ "4e8b74ca_1",
+ "710833f3",
+ "ultrachat_431164"
+ ],
+ "gold_session_ids": [
+ "answer_4dac77cb_1",
+ "answer_4dac77cb_2"
+ ]
+ },
+ {
+ "question_id": "e493bb7c",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9197207891481876,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_1a374afa_2",
+ "ultrachat_185447",
+ "answer_1a374afa_1",
+ "7aaa23d7_2",
+ "0aacd15f_1",
+ "ultrachat_363167",
+ "ultrachat_415114",
+ "ultrachat_392200",
+ "ultrachat_555062",
+ "ultrachat_203191"
+ ],
+ "gold_session_ids": [
+ "answer_1a374afa_1",
+ "answer_1a374afa_2"
+ ]
+ },
+ {
+ "question_id": "618f13b2",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_caf5b52e_2",
+ "answer_caf5b52e_1",
+ "a20bc58f",
+ "dda70510_2",
+ "3a980e89_3",
+ "86b56135_2",
+ "748eb152_2",
+ "4632c2c6_1",
+ "c51583cd_3",
+ "ultrachat_558040"
+ ],
+ "gold_session_ids": [
+ "answer_caf5b52e_1",
+ "answer_caf5b52e_2"
+ ]
+ },
+ {
+ "question_id": "72e3ee87",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_d7de9a6a_1",
+ "answer_d7de9a6a_2",
+ "ultrachat_195333",
+ "080a4000_2",
+ "73a2f346_3",
+ "be8c3970_2",
+ "ultrachat_421360",
+ "sharegpt_i6wl0uh_0",
+ "ultrachat_233144",
+ "6f2c01fc_1"
+ ],
+ "gold_session_ids": [
+ "answer_d7de9a6a_1",
+ "answer_d7de9a6a_2"
+ ]
+ },
+ {
+ "question_id": "c4ea545c",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_d3bf812b_2",
+ "answer_d3bf812b_1",
+ "sharegpt_71QV6Zs_7",
+ "5be74579",
+ "ea0e2434",
+ "a2a8e2c9",
+ "fcddf7d1_2",
+ "9f9b402a_5",
+ "493e23ac",
+ "508d552e_5"
+ ],
+ "gold_session_ids": [
+ "answer_d3bf812b_1",
+ "answer_d3bf812b_2"
+ ]
+ },
+ {
+ "question_id": "01493427",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_a7b44747_2",
+ "answer_a7b44747_1",
+ "88e73f02",
+ "077f7bcb_3",
+ "5a840928_1",
+ "780b6c7c_1",
+ "616fdd27",
+ "58e5adf2",
+ "291d8c82",
+ "087ac53a_2"
+ ],
+ "gold_session_ids": [
+ "answer_a7b44747_1",
+ "answer_a7b44747_2"
+ ]
+ },
+ {
+ "question_id": "6a27ffc2",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_77f32504_2",
+ "answer_77f32504_1",
+ "4604bc73_3",
+ "6ae451d1_2",
+ "sharegpt_tco8eKE_0",
+ "de26a9d4",
+ "ultrachat_183936",
+ "sharegpt_CW8gAFB_8",
+ "1388b204_2",
+ "sharegpt_kkqlRlo_13"
+ ],
+ "gold_session_ids": [
+ "answer_77f32504_1",
+ "answer_77f32504_2"
+ ]
+ },
+ {
+ "question_id": "2133c1b5",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6131471927654584,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_52382508_2",
+ "a52d98bf_2",
+ "8ca0085a_1",
+ "ultrachat_328330",
+ "sharegpt_I1DGUU4_0",
+ "ultrachat_354956",
+ "sharegpt_bSlIUYf_32",
+ "add9b012_1",
+ "sharegpt_MvUpG5V_5",
+ "a63ad8e3_3"
+ ],
+ "gold_session_ids": [
+ "answer_52382508_1",
+ "answer_52382508_2"
+ ]
+ },
+ {
+ "question_id": "18bc8abd",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_fff743f5_1",
+ "answer_fff743f5_2",
+ "951df45e",
+ "7169e342_1",
+ "ultrachat_529103",
+ "ba6b67af_3",
+ "2714885f_3",
+ "ac549598_1",
+ "4ad82121_2",
+ "ultrachat_225891"
+ ],
+ "gold_session_ids": [
+ "answer_fff743f5_1",
+ "answer_fff743f5_2"
+ ]
+ },
+ {
+ "question_id": "db467c8c",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_611b6e83_1",
+ "answer_611b6e83_2",
+ "ultrachat_553961",
+ "77cf4551_1",
+ "fb5dd87f_4",
+ "sharegpt_HBmf6q0_0",
+ "sharegpt_TAL32at_0",
+ "ultrachat_527913",
+ "6cc835bd",
+ "88d42833"
+ ],
+ "gold_session_ids": [
+ "answer_611b6e83_1",
+ "answer_611b6e83_2"
+ ]
+ },
+ {
+ "question_id": "7a87bd0c",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_d08a934d_1",
+ "answer_d08a934d_2",
+ "3d98b7e6_2",
+ "93e9e82c_2",
+ "81f318a2_1",
+ "62369c56_1",
+ "15519944_1",
+ "e250791e_3",
+ "sharegpt_e1Mbyy2_28",
+ "2c543c87"
+ ],
+ "gold_session_ids": [
+ "answer_d08a934d_1",
+ "answer_d08a934d_2"
+ ]
+ },
+ {
+ "question_id": "e61a7584",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_f25c32f5_1",
+ "answer_f25c32f5_2",
+ "6a4b4b7e_2",
+ "82159da4_1",
+ "sharegpt_o0MporM_0",
+ "e900ff31_2",
+ "e1416816_2",
+ "ultrachat_567152",
+ "f379d356_3",
+ "afa96a40"
+ ],
+ "gold_session_ids": [
+ "answer_f25c32f5_1",
+ "answer_f25c32f5_2"
+ ]
+ },
+ {
+ "question_id": "1cea1afa",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_79c395a9_1",
+ "answer_79c395a9_2",
+ "e131bcd4",
+ "9c479e89_1",
+ "ultrachat_82739",
+ "ultrachat_165628",
+ "sharegpt_inuIr9J_119",
+ "97f564ef",
+ "602ad002_1",
+ "c8fcdf5a_2"
+ ],
+ "gold_session_ids": [
+ "answer_79c395a9_1",
+ "answer_79c395a9_2"
+ ]
+ },
+ {
+ "question_id": "ed4ddc30",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_babbaccb_2",
+ "answer_babbaccb_1",
+ "8b726b64_1",
+ "c279c578",
+ "499652a0_3",
+ "5a6dc9cc",
+ "e4969569_2",
+ "2490f2af_3",
+ "5170dcc7_2",
+ "75621e39_1"
+ ],
+ "gold_session_ids": [
+ "answer_babbaccb_1",
+ "answer_babbaccb_2"
+ ]
+ },
+ {
+ "question_id": "8fb83627",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_966cecbb_1",
+ "answer_966cecbb_2",
+ "ultrachat_172298",
+ "be050ab3_2",
+ "ultrachat_44939",
+ "02b81ad9",
+ "4019b1a8_5",
+ "ultrachat_307166",
+ "0638f6f6_1",
+ "9315f268_1"
+ ],
+ "gold_session_ids": [
+ "answer_966cecbb_1",
+ "answer_966cecbb_2"
+ ]
+ },
+ {
+ "question_id": "b01defab",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9197207891481876,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_8c0712af_2",
+ "59dc866c_1",
+ "answer_8c0712af_1",
+ "sharegpt_VbIxRWJ_33",
+ "639cec58",
+ "9aa07e25",
+ "ultrachat_407511",
+ "b1d9eb66_2",
+ "3891baea_2",
+ "dc378711_2"
+ ],
+ "gold_session_ids": [
+ "answer_8c0712af_1",
+ "answer_8c0712af_2"
+ ]
+ },
+ {
+ "question_id": "22d2cb42",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_bcce0b73_1",
+ "answer_bcce0b73_2",
+ "060aeced",
+ "74932466_1",
+ "b63954ef",
+ "d1e4ee27_1",
+ "e1416816_1",
+ "a2defa66_2",
+ "5a76fadb_2",
+ "ultrachat_312304"
+ ],
+ "gold_session_ids": [
+ "answer_bcce0b73_1",
+ "answer_bcce0b73_2"
+ ]
+ },
+ {
+ "question_id": "0e4e4c46",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_f2f998c7_1",
+ "answer_f2f998c7_2",
+ "60876def_2",
+ "ultrachat_467820",
+ "5d1c7bf7",
+ "0773943b",
+ "b7702ee4_1",
+ "sharegpt_1YtwybC_15",
+ "70cae729_3",
+ "998616b4_2"
+ ],
+ "gold_session_ids": [
+ "answer_f2f998c7_1",
+ "answer_f2f998c7_2"
+ ]
+ },
+ {
+ "question_id": "4b24c848",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9197207891481876,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_2cec623b_2",
+ "1815d1b7",
+ "answer_2cec623b_1",
+ "16e094b3_1",
+ "d38e5b38",
+ "f5cf8815_1",
+ "ultrachat_421733",
+ "afa0c2e7_1",
+ "d0c1453b_1",
+ "95ae7b23_1"
+ ],
+ "gold_session_ids": [
+ "answer_2cec623b_1",
+ "answer_2cec623b_2"
+ ]
+ },
+ {
+ "question_id": "7e974930",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8772153153380493,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_c9f5693c_1",
+ "898ce7a5_1",
+ "ultrachat_251049",
+ "answer_c9f5693c_2",
+ "ultrachat_113430",
+ "705b5399_3",
+ "0a24e95e",
+ "bf139883",
+ "ultrachat_271314",
+ "a1e169b1"
+ ],
+ "gold_session_ids": [
+ "answer_c9f5693c_1",
+ "answer_c9f5693c_2"
+ ]
+ },
+ {
+ "question_id": "603deb26",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6131471927654584,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_8afdebac_2",
+ "88611fff",
+ "a55a5afc_2",
+ "ede65c84_2",
+ "37f66729",
+ "7db28e68_1",
+ "5e523d80_3",
+ "ultrachat_104321",
+ "5e867c2f_2",
+ "sharegpt_mMTudFY_0"
+ ],
+ "gold_session_ids": [
+ "answer_8afdebac_1",
+ "answer_8afdebac_2"
+ ]
+ },
+ {
+ "question_id": "59524333",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6934264036172708,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "638e7094_1",
+ "answer_b28f2c7a_2",
+ "answer_b28f2c7a_1",
+ "sharegpt_Clr6oyu_0",
+ "2d5d1c10",
+ "8953c8c8_2",
+ "1286422b_1",
+ "e3d4f89e_3",
+ "sharegpt_NYFuCmY_0",
+ "ultrachat_458180"
+ ],
+ "gold_session_ids": [
+ "answer_b28f2c7a_1",
+ "answer_b28f2c7a_2"
+ ]
+ },
+ {
+ "question_id": "5831f84d",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_8d63a897_1",
+ "answer_8d63a897_2",
+ "41c90f4c_2",
+ "sharegpt_6I57AvU_67",
+ "ef84b994_abs_2",
+ "0d66562c_4",
+ "b818e2a1_1",
+ "b70358bc_6",
+ "sharegpt_NHidbra_0",
+ "6a471944_1"
+ ],
+ "gold_session_ids": [
+ "answer_8d63a897_1",
+ "answer_8d63a897_2"
+ ]
+ },
+ {
+ "question_id": "eace081b",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_8a791264_2",
+ "answer_8a791264_1",
+ "4d04b866",
+ "413b57cb_3",
+ "9316aae3_1",
+ "1dcfb9a0_2",
+ "sharegpt_HOO59Ue_137",
+ "02eae87c_1",
+ "164bd5a4",
+ "4dae77d3"
+ ],
+ "gold_session_ids": [
+ "answer_8a791264_1",
+ "answer_8a791264_2"
+ ]
+ },
+ {
+ "question_id": "affe2881",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6934264036172708,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "fb5dd87f_7",
+ "answer_90de9b4d_1",
+ "answer_90de9b4d_2",
+ "ultrachat_443550",
+ "3d895af0_3",
+ "f334878c",
+ "1b71c896_2",
+ "84daebd7_2",
+ "5b4d2438_5",
+ "d53b6bfe_1"
+ ],
+ "gold_session_ids": [
+ "answer_90de9b4d_1",
+ "answer_90de9b4d_2"
+ ]
+ },
+ {
+ "question_id": "50635ada",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_dcd74827_2",
+ "answer_dcd74827_1",
+ "sharegpt_eSTYwMj_10",
+ "b6f35f3a_1",
+ "03f76b6b",
+ "42bcee92_2",
+ "35bc3132_2",
+ "9f9b402a_5",
+ "sharegpt_BVhcEnF_0",
+ "dcbd175f_2"
+ ],
+ "gold_session_ids": [
+ "answer_dcd74827_1",
+ "answer_dcd74827_2"
+ ]
+ },
+ {
+ "question_id": "e66b632c",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ac0140ce_1",
+ "answer_ac0140ce_2",
+ "259d58da_2",
+ "92f1ea4d_1",
+ "ultrachat_306973",
+ "47d18b61_1",
+ "9b820206",
+ "b3070ec4_1",
+ "57c8bf62_3",
+ "3304f57b_1"
+ ],
+ "gold_session_ids": [
+ "answer_ac0140ce_1",
+ "answer_ac0140ce_2"
+ ]
+ },
+ {
+ "question_id": "0ddfec37",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_a22b654d_2",
+ "answer_a22b654d_1",
+ "147ab7e9_7",
+ "e4881cbd_1",
+ "3422786d_2",
+ "a5959675_2",
+ "92509977_1",
+ "60f9246d",
+ "2878574e",
+ "sharegpt_h0fDljQ_12"
+ ],
+ "gold_session_ids": [
+ "answer_a22b654d_1",
+ "answer_a22b654d_2"
+ ]
+ },
+ {
+ "question_id": "f685340e",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_25df025b_1",
+ "answer_25df025b_2",
+ "a4815e8e_2",
+ "142e3203_1",
+ "e7dcd5df",
+ "4a9eb139_2",
+ "9e7446eb",
+ "bb2180e9_2",
+ "992eb1f3_2",
+ "8fcfbe43_2"
+ ],
+ "gold_session_ids": [
+ "answer_25df025b_1",
+ "answer_25df025b_2"
+ ]
+ },
+ {
+ "question_id": "cc5ded98",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_a5b68517_1",
+ "answer_a5b68517_2",
+ "8a5fe67a",
+ "f14a4532_1",
+ "c63b2c8f",
+ "c937aab6",
+ "d973fa59_1",
+ "ef785bb4_2",
+ "sharegpt_IEsU45Z_23",
+ "b13d5e4e_1"
+ ],
+ "gold_session_ids": [
+ "answer_a5b68517_1",
+ "answer_a5b68517_2"
+ ]
+ },
+ {
+ "question_id": "dfde3500",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_35d6c0be_1",
+ "answer_35d6c0be_2",
+ "ad1c7ac6",
+ "5fcca8bc_2",
+ "779c0465",
+ "5c18cb32_2",
+ "86e218e3_2",
+ "5d5e80c5_1",
+ "sharegpt_DiRAKDo_0",
+ "7de51ffe_1"
+ ],
+ "gold_session_ids": [
+ "answer_35d6c0be_1",
+ "answer_35d6c0be_2"
+ ]
+ },
+ {
+ "question_id": "69fee5aa",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_d6028d6e_2",
+ "answer_d6028d6e_1",
+ "5d284d7d_2",
+ "991d55e5_1",
+ "a0b8b0ad_7",
+ "8464304d_2",
+ "a554ed79_2",
+ "ultrachat_152449",
+ "9d11d844",
+ "2af27403_2"
+ ],
+ "gold_session_ids": [
+ "answer_d6028d6e_1",
+ "answer_d6028d6e_2"
+ ]
+ },
+ {
+ "question_id": "7401057b",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_94650bfa_1",
+ "answer_94650bfa_2",
+ "42e69993_1",
+ "0f09447b",
+ "c3757c5b",
+ "8de5b7cf_2",
+ "b7146673_1",
+ "fb2ae167",
+ "5e906cb0_1",
+ "ultrachat_389374"
+ ],
+ "gold_session_ids": [
+ "answer_94650bfa_1",
+ "answer_94650bfa_2"
+ ]
+ },
+ {
+ "question_id": "cf22b7bf",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ae3a122b_1",
+ "answer_ae3a122b_2",
+ "457e26bf",
+ "c8fcdf5a_1",
+ "36d5bbde_1",
+ "8ab0292c_1",
+ "8064b6ca",
+ "ultrachat_346016",
+ "22fe1c93",
+ "8b540c1b"
+ ],
+ "gold_session_ids": [
+ "answer_ae3a122b_1",
+ "answer_ae3a122b_2"
+ ]
+ },
+ {
+ "question_id": "a2f3aa27",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_5126c02d_1",
+ "answer_5126c02d_2",
+ "b65f51fb_5",
+ "2fa50779_2",
+ "01a467ee_2",
+ "ultrachat_75332",
+ "c98b3349_3",
+ "39709ef6_2",
+ "a8028ce7_2",
+ "57bf4122_1"
+ ],
+ "gold_session_ids": [
+ "answer_5126c02d_1",
+ "answer_5126c02d_2"
+ ]
+ },
+ {
+ "question_id": "c7dc5443",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_0cdbca92_2",
+ "answer_0cdbca92_1",
+ "9f0f59d1_3",
+ "debc34d7_2",
+ "3f9f66f1_2",
+ "8fd3c71b",
+ "sharegpt_6cz1Sq6_264",
+ "sharegpt_oSemQhs_0",
+ "sharegpt_35cK50k_0",
+ "e200d96c_3"
+ ],
+ "gold_session_ids": [
+ "answer_0cdbca92_1",
+ "answer_0cdbca92_2"
+ ]
+ },
+ {
+ "question_id": "06db6396",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_da72b1b4_2",
+ "answer_da72b1b4_1",
+ "6ebdc1fe_2",
+ "5bd9f1e6_1",
+ "ultrachat_456942",
+ "da94a327_2",
+ "sharegpt_SwdaeDm_15",
+ "57144028_1",
+ "84d7650f_2",
+ "ultrachat_279487"
+ ],
+ "gold_session_ids": [
+ "answer_da72b1b4_1",
+ "answer_da72b1b4_2"
+ ]
+ },
+ {
+ "question_id": "3ba21379",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_cd345582_2",
+ "answer_cd345582_1",
+ "0f78155a_1",
+ "d557e57a",
+ "sharegpt_xYURy32_0",
+ "a946dbf5_3",
+ "ultrachat_224436",
+ "5e50c81b",
+ "sharegpt_QQQAWUd_19",
+ "d68838d0_2"
+ ],
+ "gold_session_ids": [
+ "answer_cd345582_1",
+ "answer_cd345582_2"
+ ]
+ },
+ {
+ "question_id": "9bbe84a2",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9197207891481876,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_c6a0c6c2_1",
+ "ff0bb7fd_2",
+ "answer_c6a0c6c2_2",
+ "6b7605d1_2",
+ "ultrachat_148297",
+ "ad611725_2",
+ "114c1fb7_6",
+ "sharegpt_hcaZN94_546",
+ "cd81a69b",
+ "sharegpt_flmt4T1_5"
+ ],
+ "gold_session_ids": [
+ "answer_c6a0c6c2_1",
+ "answer_c6a0c6c2_2"
+ ]
+ },
+ {
+ "question_id": "10e09553",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_67be2c38_2",
+ "answer_67be2c38_1",
+ "58ab5fc5",
+ "3021992b_2",
+ "918c4788_1",
+ "7d1f5395",
+ "959e6cb0_7",
+ "88a3c944_3",
+ "ultrachat_313128",
+ "sharegpt_PAveUvy_0"
+ ],
+ "gold_session_ids": [
+ "answer_67be2c38_1",
+ "answer_67be2c38_2"
+ ]
+ },
+ {
+ "question_id": "dad224aa",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6934264036172708,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "6de8645d_1",
+ "answer_4a97ae40_1",
+ "answer_4a97ae40_2",
+ "ultrachat_106352",
+ "56b5b760",
+ "3dcbca49",
+ "6ff8954a_2",
+ "91e581ad_1",
+ "42234f98",
+ "ultrachat_468289"
+ ],
+ "gold_session_ids": [
+ "answer_4a97ae40_1",
+ "answer_4a97ae40_2"
+ ]
+ },
+ {
+ "question_id": "ba61f0b9",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9197207891481876,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_f377cda7_2",
+ "sharegpt_s8Opwwu_0",
+ "answer_f377cda7_1",
+ "daca659d",
+ "ultrachat_421876",
+ "sharegpt_5m7gg5F_0",
+ "ultrachat_245108",
+ "c9213a03_1",
+ "ultrachat_169691",
+ "ultrachat_328271"
+ ],
+ "gold_session_ids": [
+ "answer_f377cda7_1",
+ "answer_f377cda7_2"
+ ]
+ },
+ {
+ "question_id": "42ec0761",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.8503449055347546,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_e3892371_1",
+ "53acd215",
+ "27d0d3e7",
+ "6fb15dce_1",
+ "answer_e3892371_2",
+ "b5b8f8f9_1",
+ "sharegpt_OLc5cYf_653",
+ "e3150810",
+ "b37a44e4_2",
+ "493bd421_2"
+ ],
+ "gold_session_ids": [
+ "answer_e3892371_1",
+ "answer_e3892371_2"
+ ]
+ },
+ {
+ "question_id": "5c40ec5b",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_1cb52d0a_2",
+ "answer_1cb52d0a_1",
+ "d9868305_1",
+ "sharegpt_Ah0tRO5_0",
+ "be050ab3_2",
+ "sharegpt_XkVyVGq_0",
+ "25a2002c",
+ "sharegpt_RVk16qK_0",
+ "5cc9e1ed_3",
+ "a200b713_3"
+ ],
+ "gold_session_ids": [
+ "answer_1cb52d0a_1",
+ "answer_1cb52d0a_2"
+ ]
+ },
+ {
+ "question_id": "c6853660",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_626e93c4_2",
+ "answer_626e93c4_1",
+ "ultrachat_435535",
+ "6ed1c85e",
+ "dcbd175f_2",
+ "0e931802_2",
+ "ultrachat_223464",
+ "48a72204_2",
+ "6474cfcc",
+ "ultrachat_223382"
+ ],
+ "gold_session_ids": [
+ "answer_626e93c4_1",
+ "answer_626e93c4_2"
+ ]
+ },
+ {
+ "question_id": "26bdc477",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_f762ad8d_2",
+ "answer_f762ad8d_1",
+ "a223e307",
+ "201d188a_4",
+ "27ce9f13_2",
+ "sharegpt_bq9G6bT_2",
+ "f684ac4c_1",
+ "e4bbcd5f_2",
+ "551eae69_1",
+ "ultrachat_525275"
+ ],
+ "gold_session_ids": [
+ "answer_f762ad8d_1",
+ "answer_f762ad8d_2"
+ ]
+ },
+ {
+ "question_id": "0977f2af",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.38685280723454163,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "ultrachat_554962",
+ "answer_3bf5b73b_2",
+ "b24eddd9_1",
+ "b3642bd9_3",
+ "sharegpt_FqOC5e9_45",
+ "9b083158_1",
+ "ultrachat_143766",
+ "bdf45964_3",
+ "f115a3db_1",
+ "e0bfe67a_2"
+ ],
+ "gold_session_ids": [
+ "answer_3bf5b73b_1",
+ "answer_3bf5b73b_2"
+ ]
+ },
+ {
+ "question_id": "6aeb4375_abs",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9197207891481876,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_3f9693b7_abs_1",
+ "a5286353_1",
+ "answer_3f9693b7_abs_2",
+ "bf25a889",
+ "24cb335f",
+ "ultrachat_181672",
+ "9205e608_1",
+ "fcff2f22",
+ "c6c5bb8b_1",
+ "710f4e21_1"
+ ],
+ "gold_session_ids": [
+ "answer_3f9693b7_abs_1",
+ "answer_3f9693b7_abs_2"
+ ]
+ },
+ {
+ "question_id": "031748ae_abs",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.9197207891481876,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_8748f791_abs_1",
+ "169cd54f",
+ "answer_8748f791_abs_2",
+ "ultrachat_473482",
+ "sharegpt_EX0p0tj_19",
+ "sharegpt_wN1Z65m_0",
+ "sharegpt_7PLgNll_0",
+ "7e76059f_2",
+ "ultrachat_561369",
+ "sharegpt_Us1zWGN_19"
+ ],
+ "gold_session_ids": [
+ "answer_8748f791_abs_1",
+ "answer_8748f791_abs_2"
+ ]
+ },
+ {
+ "question_id": "2698e78f_abs",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6509209298071326,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "2e18c90b_1",
+ "answer_9282283d_abs_1",
+ "cdba3d9f_1",
+ "answer_9282283d_abs_2",
+ "e4e64e80_8",
+ "eae760f6",
+ "sharegpt_dZJOxsS_0",
+ "9e21d6ab_1",
+ "7ebf92f6_2",
+ "f607e932_1"
+ ],
+ "gold_session_ids": [
+ "answer_9282283d_abs_1",
+ "answer_9282283d_abs_2"
+ ]
+ },
+ {
+ "question_id": "2133c1b5_abs",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.45749452618986164,
+ "mrr": 0.25,
+ "retrieved_session_ids": [
+ "a864e7aa_5",
+ "fe6d8377_1",
+ "sharegpt_asoSnhq_0",
+ "answer_52382508_abs_2",
+ "688246d1",
+ "ultrachat_420387",
+ "9998814e_1",
+ "answer_52382508_abs_1",
+ "sharegpt_M5JnA12_12",
+ "ef0341d4_1"
+ ],
+ "gold_session_ids": [
+ "answer_52382508_abs_1",
+ "answer_52382508_abs_2"
+ ]
+ },
+ {
+ "question_id": "0ddfec37_abs",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6934264036172708,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "6de70f46_1",
+ "answer_a22b654d_abs_1",
+ "answer_a22b654d_abs_2",
+ "6eb55dc4_2",
+ "25572732",
+ "3a11533e",
+ "cb24e5d5_1",
+ "sharegpt_V2j1zkI_0",
+ "e3e061c4",
+ "ultrachat_200650"
+ ],
+ "gold_session_ids": [
+ "answer_a22b654d_abs_1",
+ "answer_a22b654d_abs_2"
+ ]
+ },
+ {
+ "question_id": "f685340e_abs",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_25df025b_abs_1",
+ "answer_25df025b_abs_2",
+ "6b895848_1",
+ "1970789e",
+ "79bb1298_5",
+ "bc6c5757_2",
+ "ultrachat_276410",
+ "be4dc764_1",
+ "5a30c8b2",
+ "11fa2b8f"
+ ],
+ "gold_session_ids": [
+ "answer_25df025b_abs_1",
+ "answer_25df025b_abs_2"
+ ]
+ },
+ {
+ "question_id": "89941a94",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_e1403127_2",
+ "answer_e1403127_1",
+ "de275215_3",
+ "f7a61595_5",
+ "522dd987_1",
+ "ultrachat_312851",
+ "6a7d7055_2",
+ "2ec60f51_3",
+ "85b9848f",
+ "0d758215_2"
+ ],
+ "gold_session_ids": [
+ "answer_e1403127_1",
+ "answer_e1403127_2"
+ ]
+ },
+ {
+ "question_id": "07741c45",
+ "question_type": "knowledge-update",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_7e9ad7b4_2",
+ "answer_7e9ad7b4_1",
+ "af4a2fd1",
+ "df846730",
+ "5300bfba",
+ "37d65d40_3",
+ "523e716b_2",
+ "ce21f0e4",
+ "604de0a3_2",
+ "443688cc_2"
+ ],
+ "gold_session_ids": [
+ "answer_7e9ad7b4_1",
+ "answer_7e9ad7b4_2"
+ ]
+ },
+ {
+ "question_id": "7161e7e2",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_sharegpt_5Lzox6N_0",
+ "3c3fee41",
+ "sharegpt_bYTKGgl_18",
+ "2bd9990a_3",
+ "ae73b0f7_2",
+ "a849dd16",
+ "ultrachat_447051",
+ "ultrachat_89092",
+ "3070419a_1",
+ "ultrachat_20447"
+ ],
+ "gold_session_ids": [
+ "answer_sharegpt_5Lzox6N_0"
+ ]
+ },
+ {
+ "question_id": "c4f10528",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ultrachat_234453",
+ "51c89181_1",
+ "dc1f9713_2",
+ "sharegpt_yMwEvl7_557",
+ "11248f50",
+ "7613d5ec_1",
+ "de55485d_1",
+ "107d40f0",
+ "8b1b7007",
+ "e12cfbcd"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_234453"
+ ]
+ },
+ {
+ "question_id": "89527b6b",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_sharegpt_YkWn1Ne_0",
+ "07f3f0e0_2",
+ "ultrachat_117600",
+ "689b46e2_4",
+ "ultrachat_66044",
+ "ultrachat_468629",
+ "ultrachat_11001",
+ "644e1d00",
+ "ultrachat_442154",
+ "990f3ef9_7"
+ ],
+ "gold_session_ids": [
+ "answer_sharegpt_YkWn1Ne_0"
+ ]
+ },
+ {
+ "question_id": "e9327a54",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ultrachat_480665",
+ "c0d099e6_2",
+ "c58fdf1e_2",
+ "ba944cdc",
+ "c38fb85b",
+ "61e1ba53_2",
+ "e91e8a2e_1",
+ "8221810f_3",
+ "7613d5ec_2",
+ "ultrachat_423650"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_480665"
+ ]
+ },
+ {
+ "question_id": "4c36ccef",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ultrachat_448704",
+ "3dc8ba5f_2",
+ "89749c78_1",
+ "7b1d1f43_2",
+ "5fc83435_2",
+ "52381716_2",
+ "940629b2_1",
+ "07e6a3bd_1",
+ "ultrachat_149723",
+ "sharegpt_m8F4NxL_0"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_448704"
+ ]
+ },
+ {
+ "question_id": "6ae235be",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_sharegpt_IUWQYGQ_0",
+ "ec8fa65d_1",
+ "41b6e78c",
+ "99188307",
+ "0a1e90d1_2",
+ "ultrachat_497582",
+ "cc5a6057_1",
+ "ultrachat_431987",
+ "43811b62",
+ "sharegpt_TVK4gjw_24"
+ ],
+ "gold_session_ids": [
+ "answer_sharegpt_IUWQYGQ_0"
+ ]
+ },
+ {
+ "question_id": "7e00a6cb",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ultrachat_370515",
+ "24f78a31_2",
+ "4dd1ba8d_2",
+ "9998814e_2",
+ "ultrachat_188131",
+ "956ce238",
+ "a9981dc6_2",
+ "sharegpt_n2aGNgW_9",
+ "64c2b3c8",
+ "sharegpt_0dfpdJm_0"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_370515"
+ ]
+ },
+ {
+ "question_id": "1903aded",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 0,
+ "recall_any_at_20": 0,
+ "ndcg_at_10": 0,
+ "mrr": 0,
+ "retrieved_session_ids": [
+ "9f8c432e_2",
+ "sharegpt_ELyN256_27",
+ "9b4c070c",
+ "66369fde_1",
+ "b163d176",
+ "f8246a66_1",
+ "4d287828_2",
+ "ultrachat_138198",
+ "9b3611d7_1",
+ "8adb232f_1"
+ ],
+ "gold_session_ids": [
+ "answer_sharegpt_hA7AkP3_0"
+ ]
+ },
+ {
+ "question_id": "ceb54acb",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_sharegpt_cGdjmYo_0",
+ "7aa2a9af_2",
+ "de43030f_1",
+ "de293134_1",
+ "2f23dd1a",
+ "a1cbe83a",
+ "0bbd7094_1",
+ "92762eb7",
+ "dddce60a_1",
+ "f6fd00cf"
+ ],
+ "gold_session_ids": [
+ "answer_sharegpt_cGdjmYo_0"
+ ]
+ },
+ {
+ "question_id": "f523d9fe",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_sharegpt_m2xJfjo_0",
+ "fbfda981",
+ "44165708",
+ "a07f9d5c_1",
+ "636e0e32_1",
+ "sharegpt_4RnbgQt_0",
+ "debc34d7_1",
+ "193c23bd_1",
+ "4376945f",
+ "ultrachat_418857"
+ ],
+ "gold_session_ids": [
+ "answer_sharegpt_m2xJfjo_0"
+ ]
+ },
+ {
+ "question_id": "0e5e2d1a",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ultrachat_113156",
+ "c756c34e",
+ "a803f50b_2",
+ "sharegpt_btv4SkQ_0",
+ "ultrachat_201585",
+ "ultrachat_180166",
+ "a6c41826_3",
+ "ultrachat_222974",
+ "5d79fd65_2",
+ "147ab7e9_7"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_113156"
+ ]
+ },
+ {
+ "question_id": "fea54f57",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ultrachat_187684",
+ "ultrachat_129340",
+ "d81c92cb",
+ "3210adc8",
+ "a1e90def_1",
+ "c8854b28",
+ "712e923c",
+ "5ac7f9c5_1",
+ "19ec83c5_1",
+ "sharegpt_UGg8d44_4"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_187684"
+ ]
+ },
+ {
+ "question_id": "cc539528",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ultrachat_374124",
+ "18807892_4",
+ "38e81260_5",
+ "faa194ea",
+ "1692563c_2",
+ "fb760626",
+ "4f859a31_1",
+ "8bd3814a_1",
+ "3a1ea336_3",
+ "15a8ffb7_2"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_374124"
+ ]
+ },
+ {
+ "question_id": "dc439ea3",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ultrachat_459954",
+ "1bfd5a8b_2",
+ "ultrachat_186512",
+ "cb5637b3",
+ "49d8a382",
+ "76ecc9b9_1",
+ "eb7366f2_1",
+ "ultrachat_480287",
+ "sharegpt_aVExml0_92",
+ "sharegpt_q2uAW7H_0"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_459954"
+ ]
+ },
+ {
+ "question_id": "18dcd5a5",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.5,
+ "mrr": 0.3333333333333333,
+ "retrieved_session_ids": [
+ "sharegpt_d1tE544_5",
+ "aceffaab",
+ "answer_sharegpt_hn3IS1q_0",
+ "5e906cb0_2",
+ "6ea1541e_1",
+ "c00077f5_2",
+ "sharegpt_2yaKUmC_0",
+ "d89f6204",
+ "ultrachat_3370",
+ "47727b14"
+ ],
+ "gold_session_ids": [
+ "answer_sharegpt_hn3IS1q_0"
+ ]
+ },
+ {
+ "question_id": "488d3006",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ultrachat_275993",
+ "ce19df54_3",
+ "a394f6b5_1",
+ "82b82901",
+ "a68090b0_2",
+ "b0f8fc7b_2",
+ "990c8992_1",
+ "d53b6bfe_1",
+ "7e4dab66_1",
+ "4646c83d_2"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_275993"
+ ]
+ },
+ {
+ "question_id": "58470ed2",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_sharegpt_U4oCSfU_7",
+ "ultrachat_478785",
+ "66bfa1db",
+ "sharegpt_6QUDIXG_104",
+ "ff49b5a5_3",
+ "sharegpt_ty6EeyH_0",
+ "718d0e40",
+ "b6018747_4",
+ "69bd3d4a_1",
+ "ultrachat_251269"
+ ],
+ "gold_session_ids": [
+ "answer_sharegpt_U4oCSfU_7"
+ ]
+ },
+ {
+ "question_id": "8cf51dda",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_sharegpt_HFMn2ZX_0",
+ "e862f726_3",
+ "sharegpt_fUrzIST_0",
+ "sharegpt_ftoCcOn_0",
+ "ultrachat_365938",
+ "9f0f59d1_1",
+ "sharegpt_jpiaPoJ_738",
+ "sharegpt_eTtpNeO_5",
+ "507514d9_1",
+ "ultrachat_479158"
+ ],
+ "gold_session_ids": [
+ "answer_sharegpt_HFMn2ZX_0"
+ ]
+ },
+ {
+ "question_id": "1d4da289",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ultrachat_348449",
+ "sharegpt_AfruXDt_0",
+ "ultrachat_213759",
+ "517ad0f0",
+ "sharegpt_eVmxjQZ_0",
+ "sharegpt_NaYyUjd_0",
+ "sharegpt_jltxhMe_53",
+ "bf85791d",
+ "4cd929f8_1",
+ "33baaaab_6"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_348449"
+ ]
+ },
+ {
+ "question_id": "8464fc84",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ultrachat_467053",
+ "98d39a6d",
+ "21d86ebb_2",
+ "d43b63a6",
+ "2d003fe4_2",
+ "e6ab6a7b_1",
+ "6ac43c5b_4",
+ "6095245e",
+ "9ac77ac9_2",
+ "69a105f9_1"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_467053"
+ ]
+ },
+ {
+ "question_id": "8aef76bc",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ultrachat_563222",
+ "146dfabe",
+ "5209e813_2",
+ "05e258f4_2",
+ "08e084f1_1",
+ "ultrachat_368669",
+ "c7f6643e_2",
+ "sharegpt_5VMruqC_0",
+ "a14e0137",
+ "87a183c1_3"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_563222"
+ ]
+ },
+ {
+ "question_id": "71a3fd6b",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6309297535714575,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "7f7e26d2_2",
+ "answer_ultrachat_417348",
+ "b7146673_1",
+ "36743359_3",
+ "8419b385_3",
+ "cc40f3ff",
+ "4bebc783_2",
+ "d8f4b0f4",
+ "sharegpt_69zYTa4_0",
+ "ultrachat_168420"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_417348"
+ ]
+ },
+ {
+ "question_id": "2bf43736",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_sharegpt_2kpncbX_13",
+ "8897a2e5",
+ "1a5fb2cd",
+ "16c3be68_1",
+ "a93750ef_2",
+ "5ca6cd28",
+ "2e89bd03_1",
+ "ultrachat_49111",
+ "445e6a7a_1",
+ "9b1d5656_1"
+ ],
+ "gold_session_ids": [
+ "answer_sharegpt_2kpncbX_13"
+ ]
+ },
+ {
+ "question_id": "70b3e69b",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ultrachat_334948",
+ "sharegpt_O8qJclh_0",
+ "ultrachat_47234",
+ "ultrachat_146029",
+ "ultrachat_302829",
+ "3169d075_2",
+ "09e1a929_2",
+ "28cd2f40",
+ "ab3af6c5_2",
+ "ultrachat_361790"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_334948"
+ ]
+ },
+ {
+ "question_id": "8752c811",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_sharegpt_6pWK9yx_0",
+ "sharegpt_cTla0Yd_0",
+ "02b63d04_1",
+ "5c6a5b7c_1",
+ "56745369_2",
+ "sharegpt_Gb68gx3_0",
+ "sharegpt_Y6MteNX_0",
+ "sharegpt_7E53b59_0",
+ "9db30f1c_3",
+ "sharegpt_lAfTTUo_0"
+ ],
+ "gold_session_ids": [
+ "answer_sharegpt_6pWK9yx_0"
+ ]
+ },
+ {
+ "question_id": "3249768e",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6309297535714575,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "0868982f_1",
+ "answer_sharegpt_CaxTGYP_0",
+ "689b46e2_2",
+ "11e9d591_2",
+ "c6c5bb8b_2",
+ "e0bd5f14_2",
+ "23450b93_3",
+ "ed1d68f1_1",
+ "773aebbd_3",
+ "268e8532_2"
+ ],
+ "gold_session_ids": [
+ "answer_sharegpt_CaxTGYP_0"
+ ]
+ },
+ {
+ "question_id": "1b9b7252",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ultrachat_115151",
+ "sharegpt_1AhFMpw_5",
+ "d3cc5bdc_4",
+ "53ab4734",
+ "0e726047",
+ "f7796d96_1",
+ "7798452b_6",
+ "3abaa836",
+ "b2e5a019",
+ "af4d4ecd_6"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_115151"
+ ]
+ },
+ {
+ "question_id": "1568498a",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_sharegpt_d6JJiqH_76",
+ "sharegpt_h87NLyS_49",
+ "71e0db57",
+ "sharegpt_YfakX5Z_0",
+ "6ad46850_2",
+ "ultrachat_344985",
+ "ultrachat_131295",
+ "1fef8ffe_1",
+ "ultrachat_35159",
+ "sharegpt_69agvDU_27"
+ ],
+ "gold_session_ids": [
+ "answer_sharegpt_d6JJiqH_76"
+ ]
+ },
+ {
+ "question_id": "6222b6eb",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6309297535714575,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "04a0b385",
+ "answer_sharegpt_H9PiM5G_0",
+ "d1b9993e_3",
+ "723bf11f",
+ "25ffc571",
+ "sharegpt_MvUpG5V_5",
+ "caf7480d_1",
+ "2bbbe083",
+ "ultrachat_418305",
+ "sharegpt_zS41R74_0"
+ ],
+ "gold_session_ids": [
+ "answer_sharegpt_H9PiM5G_0"
+ ]
+ },
+ {
+ "question_id": "e8a79c70",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.5,
+ "mrr": 0.3333333333333333,
+ "retrieved_session_ids": [
+ "c0d099e6_2",
+ "89aed484_2",
+ "answer_ultrachat_13075",
+ "4050ebff_9",
+ "sharegpt_ha8kUEr_0",
+ "2b5c911e_4",
+ "c2b2b2ea_1",
+ "08ca1f31_2",
+ "45c29267",
+ "85a49fe0_1"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_13075"
+ ]
+ },
+ {
+ "question_id": "d596882b",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ultrachat_252214",
+ "0dc2efcc_2",
+ "456807b7_1",
+ "5558a42e_3",
+ "73ba8380_3",
+ "ultrachat_299672",
+ "71888aff_2",
+ "5fb3b5ac",
+ "692075ef",
+ "c324a5bc_1"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_252214"
+ ]
+ },
+ {
+ "question_id": "e3fc4d6e",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_sharegpt_5m7gg5F_0",
+ "ultrachat_442052",
+ "ultrachat_161238",
+ "sharegpt_sXKNzPE_0",
+ "9b72b151",
+ "3b38bf46",
+ "d31c3ec8_1",
+ "bec86aec",
+ "6aaf298d",
+ "ultrachat_61952"
+ ],
+ "gold_session_ids": [
+ "answer_sharegpt_5m7gg5F_0"
+ ]
+ },
+ {
+ "question_id": "51b23612",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ultrachat_427265",
+ "17f444f3",
+ "ultrachat_555062",
+ "8672f398_2",
+ "409015ef_1",
+ "d3c3cd97",
+ "8d30ec5b",
+ "ultrachat_522170",
+ "ultrachat_221034",
+ "fe5894c5"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_427265"
+ ]
+ },
+ {
+ "question_id": "3e321797",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ultrachat_94624",
+ "sharegpt_EZmhz8p_0",
+ "0382dbfe_2",
+ "96b6e5e8_1",
+ "57144028_2",
+ "04b7b396_1",
+ "00305dd2_1",
+ "e77fc218_2",
+ "385a60c8_1",
+ "5558a42e_3"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_94624"
+ ]
+ },
+ {
+ "question_id": "e982271f",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ultrachat_195444",
+ "db65997e",
+ "7b4bce3b_2",
+ "eb0d8dc1_4",
+ "333a59e5_1",
+ "091aa510_5",
+ "ultrachat_383774",
+ "e760dc71_1",
+ "3e8f07c9_1",
+ "ultrachat_137024"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_195444"
+ ]
+ },
+ {
+ "question_id": "352ab8bd",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_sharegpt_NoDZzot_7",
+ "337cbfc8_2",
+ "df08e7a0",
+ "3b0f28f6_2",
+ "6593cb8b_6",
+ "sharegpt_UCVg3jO_0",
+ "sharegpt_cDlYqSf_21",
+ "c7ca6dff",
+ "ultrachat_1052",
+ "sharegpt_7tOPXDd_50"
+ ],
+ "gold_session_ids": [
+ "answer_sharegpt_NoDZzot_7"
+ ]
+ },
+ {
+ "question_id": "fca762bc",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ultrachat_39395",
+ "2bc4ae7d",
+ "d0cdfddb_2",
+ "99eec551",
+ "d5b3b034_1",
+ "9beab022_1",
+ "a8bb9614_2",
+ "06a34d82_1",
+ "f1bdf7f3_4",
+ "8aee9f4c_2"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_39395"
+ ]
+ },
+ {
+ "question_id": "7a8d0b71",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_sharegpt_i0tMT9q_9",
+ "ultrachat_136480",
+ "147ab7e9_5",
+ "f20e72e4_2",
+ "f8169a6d",
+ "1e26715f",
+ "8969b3ca_2",
+ "ultrachat_324501",
+ "369695b4_1",
+ "sharegpt_P0Qe2Gx_0"
+ ],
+ "gold_session_ids": [
+ "answer_sharegpt_i0tMT9q_9"
+ ]
+ },
+ {
+ "question_id": "a40e080f",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ultrachat_269020",
+ "4d32ba1b_1",
+ "528f481d_6",
+ "1c1541c3",
+ "eb18a1b7",
+ "4fd6129f_1",
+ "89c7a48a_2",
+ "cd38b6ee",
+ "ultrachat_483678",
+ "12c28d3f_1"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_269020"
+ ]
+ },
+ {
+ "question_id": "8b9d4367",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ultrachat_289157",
+ "ultrachat_321805",
+ "7babf05b",
+ "sharegpt_WZzri9J_0",
+ "07b94dbf_1",
+ "ultrachat_563661",
+ "sharegpt_ZSnZKMn_13",
+ "c77250d2",
+ "007e7d81_3",
+ "5534efcb_1"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_289157"
+ ]
+ },
+ {
+ "question_id": "5809eb10",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_sharegpt_4aJsGCH_0",
+ "sharegpt_GDFyxeT_0",
+ "5237bb0b_1",
+ "ultrachat_255793",
+ "sharegpt_9M4WbfX_0",
+ "922c819f_2",
+ "58ae7c4f_1",
+ "sharegpt_oKiXiX8_0",
+ "ultrachat_549762",
+ "701ea427_6"
+ ],
+ "gold_session_ids": [
+ "answer_sharegpt_4aJsGCH_0"
+ ]
+ },
+ {
+ "question_id": "41275add",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.6309297535714575,
+ "mrr": 0.5,
+ "retrieved_session_ids": [
+ "b3619c2c",
+ "answer_sharegpt_81riySf_0",
+ "acc3913b",
+ "9345f7dc_1",
+ "4d84cbfa",
+ "a68db5db_1",
+ "6d96a14d_2",
+ "5fe4d2ee_1",
+ "sharegpt_dUoAhvI_71",
+ "5b83c26e_3"
+ ],
+ "gold_session_ids": [
+ "answer_sharegpt_81riySf_0"
+ ]
+ },
+ {
+ "question_id": "4388e9dd",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_sharegpt_qTi81nS_0",
+ "sharegpt_Tun3KJ4_0",
+ "cfd21744_2",
+ "3197d998_7",
+ "ultrachat_412706",
+ "f508f11f_2",
+ "sharegpt_2LrB18X_51",
+ "234c8721",
+ "sharegpt_rdaQ1VE_13",
+ "cf021b36_3"
+ ],
+ "gold_session_ids": [
+ "answer_sharegpt_qTi81nS_0"
+ ]
+ },
+ {
+ "question_id": "4baee567",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_sharegpt_i9adwQn_0",
+ "74711358_2",
+ "sharegpt_wduYbsX_0",
+ "6169ef55_1",
+ "sharegpt_28Mwwk9_0",
+ "ultrachat_487211",
+ "b762a658",
+ "2c22e0e8_1",
+ "ultrachat_323343",
+ "d5415748"
+ ],
+ "gold_session_ids": [
+ "answer_sharegpt_i9adwQn_0"
+ ]
+ },
+ {
+ "question_id": "561fabcd",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_sharegpt_hChsWOp_97",
+ "aee13015_3",
+ "33da50d0_5",
+ "83fb74bf_1",
+ "63c75b13_2",
+ "d75f4fb5_1",
+ "9a992902",
+ "d5d7eff7",
+ "1c1f5ccc_6",
+ "34b38398_2"
+ ],
+ "gold_session_ids": [
+ "answer_sharegpt_hChsWOp_97"
+ ]
+ },
+ {
+ "question_id": "b759caee",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_sharegpt_2BSXlAr_0",
+ "191f4c9b_2",
+ "abecc1b6",
+ "ffa51434_3",
+ "f55fee5b_4",
+ "d3575920",
+ "0bad887e_2",
+ "bab41bb6_5",
+ "b41aae7f_1",
+ "fcae5c39_1"
+ ],
+ "gold_session_ids": [
+ "answer_sharegpt_2BSXlAr_0"
+ ]
+ },
+ {
+ "question_id": "ac031881",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_sharegpt_GYqnAhC_190",
+ "c986f83a",
+ "sharegpt_cIj2dFl_4",
+ "449b918c_3",
+ "e30307d8_1",
+ "sharegpt_1X61wv2_2",
+ "sharegpt_RWoA8C4_0",
+ "19c24c11",
+ "83f89c02",
+ "ebb47912"
+ ],
+ "gold_session_ids": [
+ "answer_sharegpt_GYqnAhC_190"
+ ]
+ },
+ {
+ "question_id": "28bcfaac",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ultrachat_446979",
+ "135a62d7_2",
+ "be05680b_2",
+ "96e1688e",
+ "4646c83d_1",
+ "ultrachat_181176",
+ "sharegpt_hi2xiJT_14",
+ "510b98f3",
+ "a27a2811",
+ "485f4604_2"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_446979"
+ ]
+ },
+ {
+ "question_id": "16c90bf4",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ultrachat_294807",
+ "b8770374_2",
+ "fa2714ed_1",
+ "89d49715_1",
+ "b0da5097_2",
+ "afa0c2e7_5",
+ "d4230511_1",
+ "be74993b_1",
+ "cc830ab4_1",
+ "8464304d_2"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_294807"
+ ]
+ },
+ {
+ "question_id": "c8f1aeed",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ultrachat_519486",
+ "sharegpt_ebYKddM_87",
+ "ultrachat_220071",
+ "ultrachat_39392",
+ "c05dde5b_2",
+ "f8a9cd01_2",
+ "199a5225_1",
+ "a53fdd02_2",
+ "af631aa3_3",
+ "ea27c6f5_1"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_519486"
+ ]
+ },
+ {
+ "question_id": "eaca4986",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 0,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 0.3333333333333333,
+ "mrr": 0.14285714285714285,
+ "retrieved_session_ids": [
+ "931c521e_1",
+ "1c1a2b7f_2",
+ "ultrachat_311235",
+ "3477c37c_1",
+ "ultrachat_474091",
+ "37db6f4e_1",
+ "answer_sharegpt_SS141vi_0",
+ "ultrachat_175444",
+ "0e543e9e_2",
+ "sharegpt_zhAMIY7_0"
+ ],
+ "gold_session_ids": [
+ "answer_sharegpt_SS141vi_0"
+ ]
+ },
+ {
+ "question_id": "c7cf7dfd",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ultrachat_456407",
+ "e27fe923_5",
+ "780b6c7c_1",
+ "9db6d988_2",
+ "ultrachat_328173",
+ "b90109d2_1",
+ "ultrachat_139629",
+ "ultrachat_419371",
+ "285a29e7_1",
+ "4acdd888"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_456407"
+ ]
+ },
+ {
+ "question_id": "e48988bc",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ultrachat_174360",
+ "476b1cbb",
+ "ultrachat_494373",
+ "ultrachat_307163",
+ "ultrachat_283303",
+ "8fb3fe3a_3",
+ "0781daa1_2",
+ "ultrachat_172828",
+ "ultrachat_99935",
+ "bae244c4"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_174360"
+ ]
+ },
+ {
+ "question_id": "1de5cff2",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ultrachat_440262",
+ "28031c43_2",
+ "8fcfbe43_3",
+ "c14dbe17",
+ "ultrachat_186422",
+ "ultrachat_58031",
+ "1a693999_4",
+ "f863a648_3",
+ "4cb33063",
+ "848d721f_1"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_440262"
+ ]
+ },
+ {
+ "question_id": "65240037",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ultrachat_403752",
+ "f8dc1273",
+ "37141183",
+ "faad7d7a_2",
+ "dba5a924",
+ "fd6f60f0_4",
+ "0984a772",
+ "92466e8e_1",
+ "7df4c735_1",
+ "sharegpt_ME35OD8_167"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_403752"
+ ]
+ },
+ {
+ "question_id": "778164c6",
+ "question_type": "single-session-assistant",
+ "recall_any_at_5": 1,
+ "recall_any_at_10": 1,
+ "recall_any_at_20": 1,
+ "ndcg_at_10": 1,
+ "mrr": 1,
+ "retrieved_session_ids": [
+ "answer_ultrachat_399000",
+ "b605741a_1",
+ "sharegpt_WXCvFir_11",
+ "c15dadce_4",
+ "ddaa5a15_2",
+ "sharegpt_ki9IVDq_6",
+ "1340826c",
+ "ultrachat_346016",
+ "c6bd5277",
+ "025836be_1"
+ ],
+ "gold_session_ids": [
+ "answer_ultrachat_399000"
+ ]
+ }
+ ]
+}
\ No newline at end of file
diff --git a/benchmark/longmemeval-bench.ts b/benchmark/longmemeval-bench.ts
new file mode 100644
index 00000000..d826e1ac
--- /dev/null
+++ b/benchmark/longmemeval-bench.ts
@@ -0,0 +1,317 @@
+import { SearchIndex } from "../src/state/search-index.js";
+import { VectorIndex } from "../src/state/vector-index.js";
+import { HybridSearch } from "../src/state/hybrid-search.js";
+import type {
+ CompressedObservation,
+ EmbeddingProvider,
+} from "../src/types.js";
+import { readFileSync, writeFileSync, existsSync } from "node:fs";
+
+interface LongMemEvalEntry {
+ question_id: string;
+ question_type: string;
+ question: string;
+ question_date: string;
+ answer: string;
+ answer_session_ids: string[];
+ haystack_dates: string[];
+ haystack_session_ids: string[];
+ haystack_sessions: Array<Array<{ role: string; content: string; has_answer?: boolean }>>;
+}
+
+interface SessionChunk {
+ sessionId: string;
+ text: string;
+ turnCount: number;
+}
+
+interface BenchResult {
+ question_id: string;
+ question_type: string;
+ recall_any_at_5: number;
+ recall_any_at_10: number;
+ recall_any_at_20: number;
+ ndcg_at_10: number;
+ mrr: number;
+ retrieved_session_ids: string[];
+ gold_session_ids: string[];
+}
+
+function chunkSessionToText(
+ turns: Array<{ role: string; content: string }>,
+): string {
+ return turns
+ .map((t) => `${t.role}: ${t.content}`)
+ .join("\n");
+}
+
+function recallAny(
+ retrievedSessionIds: string[],
+ goldSessionIds: string[],
+ k: number,
+): number {
+ const topK = new Set(retrievedSessionIds.slice(0, k));
+ return goldSessionIds.some((gid) => topK.has(gid)) ? 1.0 : 0.0;
+}
+
+function dcg(relevances: boolean[], k: number): number {
+ let sum = 0;
+ for (let i = 0; i < Math.min(k, relevances.length); i++) {
+ sum += (relevances[i] ? 1 : 0) / Math.log2(i + 2);
+ }
+ return sum;
+}
+
+function ndcg(
+ retrievedSessionIds: string[],
+ goldSessionIds: Set<string>,
+ k: number,
+): number {
+ const rels = retrievedSessionIds
+ .slice(0, k)
+ .map((id) => goldSessionIds.has(id));
+ const idealRels = Array.from(
+ { length: Math.min(k, goldSessionIds.size) },
+ () => true,
+ );
+ const idealDCG = dcg(idealRels, k);
+ if (idealDCG === 0) return 0;
+ return dcg(rels, k) / idealDCG;
+}
+
+function mrr(
+ retrievedSessionIds: string[],
+ goldSessionIds: Set<string>,
+): number {
+ for (let i = 0; i < retrievedSessionIds.length; i++) {
+ if (goldSessionIds.has(retrievedSessionIds[i])) return 1 / (i + 1);
+ }
+ return 0;
+}
+
+class MockKV {
+ private store = new Map<string, Map<string, unknown>>();
+ async get<T>(scope: string, key: string): Promise<T> {
+ const m = this.store.get(scope);
+ if (!m || !m.has(key)) throw new Error(`Not found: ${scope}/${key}`);
+ return m.get(key) as T;
+ }
+ async set(scope: string, key: string, value: unknown): Promise<void> {
+ if (!this.store.has(scope)) this.store.set(scope, new Map());
+ this.store.get(scope)!.set(key, value);
+ }
+ async list<T>(scope: string): Promise<T[]> {
+ const m = this.store.get(scope);
+ if (!m) return [];
+ return Array.from(m.values()) as T[];
+ }
+ async delete(scope: string, key: string): Promise<void> {
+ this.store.get(scope)?.delete(key);
+ }
+}
+
+async function runBenchmark(
+ mode: "bm25" | "vector" | "hybrid",
+ embeddingProvider?: EmbeddingProvider,
+) {
+ const dataPath = new URL("./data/longmemeval_s_cleaned.json", import.meta.url).pathname;
+ if (!existsSync(dataPath)) {
+ console.error(`Dataset not found at ${dataPath}`);
+ console.error("Download from: https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned");
+ process.exit(1);
+ }
+
+ console.log(`Loading LongMemEval-S dataset...`);
+ const raw = JSON.parse(readFileSync(dataPath, "utf-8")) as LongMemEvalEntry[];
+
+ const abstentionTypes = new Set([
+ "single-session-user_abs",
+ "multi-session_abs",
+ "knowledge-update_abs",
+ "temporal-reasoning_abs",
+ ]);
+ const entries = raw.filter((e) => !abstentionTypes.has(e.question_type));
+ console.log(
+ `Loaded ${entries.length} questions (${raw.length - entries.length} abstention excluded)`,
+ );
+
+ const results: BenchResult[] = [];
+ let processed = 0;
+
+ for (const entry of entries) {
+ const sessionChunks: SessionChunk[] = [];
+ for (let i = 0; i < entry.haystack_sessions.length; i++) {
+ const sessionId = entry.haystack_session_ids[i];
+ const turns = entry.haystack_sessions[i];
+ const text = chunkSessionToText(turns);
+ sessionChunks.push({ sessionId, text, turnCount: turns.length });
+ }
+
+ const bm25 = new SearchIndex();
+ const vector = mode !== "bm25" ? new VectorIndex() : null;
+ const kv = new MockKV();
+
+ const observations: CompressedObservation[] = [];
+ for (const chunk of sessionChunks) {
+ const obs: CompressedObservation = {
+ id: `obs_${chunk.sessionId}`,
+ sessionId: chunk.sessionId,
+ timestamp: new Date().toISOString(),
+ type: "conversation",
+ title: chunk.text.slice(0, 80),
+ facts: [],
+ narrative: chunk.text,
+ concepts: [],
+ files: [],
+ importance: 5,
+ };
+ observations.push(obs);
+ bm25.add(obs);
+
+ if (vector && embeddingProvider) {
+ try {
+ const embedding = await embeddingProvider.embed(
+ chunk.text.slice(0, 512),
+ );
+ vector.add(obs.id, chunk.sessionId, embedding);
+ } catch {}
+ }
+
+ await kv.set(`mem:obs:${chunk.sessionId}`, obs.id, obs);
+ }
+
+ let retrievedObsIds: string[];
+
+ if (mode === "bm25") {
+ const bm25Results = bm25.search(entry.question, 20);
+ retrievedObsIds = bm25Results.map((r) => r.obsId);
+ } else {
+ const hybridSearch = new HybridSearch(
+ bm25,
+ vector,
+ embeddingProvider || null,
+ kv as any,
+ 0.4,
+ 0.6,
+ 0.0,
+ false,
+ );
+ const hybridResults = await hybridSearch.search(entry.question, 20);
+ retrievedObsIds = hybridResults.map((r) => r.observation.id);
+ }
+
+ const retrievedSessionIds = retrievedObsIds.map((oid) =>
+ oid.replace(/^obs_/, ""),
+ );
+ const goldSet = new Set(entry.answer_session_ids);
+
+ const result: BenchResult = {
+ question_id: entry.question_id,
+ question_type: entry.question_type,
+ recall_any_at_5: recallAny(retrievedSessionIds, entry.answer_session_ids, 5),
+ recall_any_at_10: recallAny(retrievedSessionIds, entry.answer_session_ids, 10),
+ recall_any_at_20: recallAny(retrievedSessionIds, entry.answer_session_ids, 20),
+ ndcg_at_10: ndcg(retrievedSessionIds, goldSet, 10),
+ mrr: mrr(retrievedSessionIds, goldSet),
+ retrieved_session_ids: retrievedSessionIds.slice(0, 10),
+ gold_session_ids: entry.answer_session_ids,
+ };
+ results.push(result);
+ processed++;
+
+ if (processed % 50 === 0) {
+ const avgRecall5 =
+ results.reduce((s, r) => s + r.recall_any_at_5, 0) / results.length;
+ console.log(
+ ` [${processed}/${entries.length}] running recall_any@5: ${(avgRecall5 * 100).toFixed(1)}%`,
+ );
+ }
+ }
+
+ const avgRecallAny5 =
+ results.reduce((s, r) => s + r.recall_any_at_5, 0) / results.length;
+ const avgRecallAny10 =
+ results.reduce((s, r) => s + r.recall_any_at_10, 0) / results.length;
+ const avgRecallAny20 =
+ results.reduce((s, r) => s + r.recall_any_at_20, 0) / results.length;
+ const avgNdcg10 =
+ results.reduce((s, r) => s + r.ndcg_at_10, 0) / results.length;
+ const avgMrr =
+ results.reduce((s, r) => s + r.mrr, 0) / results.length;
+
+ const byType = new Map<string, BenchResult[]>();
+ for (const r of results) {
+ if (!byType.has(r.question_type)) byType.set(r.question_type, []);
+ byType.get(r.question_type)!.push(r);
+ }
+
+ console.log(`\n=== LongMemEval-S Results (${mode}) ===`);
+ console.log(`Questions: ${results.length} (excl. abstention)`);
+ console.log(`recall_any@5: ${(avgRecallAny5 * 100).toFixed(1)}%`);
+ console.log(`recall_any@10: ${(avgRecallAny10 * 100).toFixed(1)}%`);
+ console.log(`recall_any@20: ${(avgRecallAny20 * 100).toFixed(1)}%`);
+ console.log(`NDCG@10: ${(avgNdcg10 * 100).toFixed(1)}%`);
+ console.log(`MRR: ${(avgMrr * 100).toFixed(1)}%`);
+
+ console.log(`\nBy question type:`);
+ for (const [type, typeResults] of byType) {
+ const r5 =
+ typeResults.reduce((s, r) => s + r.recall_any_at_5, 0) /
+ typeResults.length;
+ const r10 =
+ typeResults.reduce((s, r) => s + r.recall_any_at_10, 0) /
+ typeResults.length;
+ console.log(
+ ` ${type.padEnd(30)} R@5: ${(r5 * 100).toFixed(1)}% R@10: ${(r10 * 100).toFixed(1)}% (n=${typeResults.length})`,
+ );
+ }
+
+ const outPath = new URL(
+ `./data/longmemeval_results_${mode}.json`,
+ import.meta.url,
+ ).pathname;
+ writeFileSync(
+ outPath,
+ JSON.stringify(
+ {
+ mode,
+ questions: results.length,
+ recall_any_at_5: avgRecallAny5,
+ recall_any_at_10: avgRecallAny10,
+ recall_any_at_20: avgRecallAny20,
+ ndcg_at_10: avgNdcg10,
+ mrr: avgMrr,
+ per_type: Object.fromEntries(
+ Array.from(byType).map(([type, tr]) => [
+ type,
+ {
+ count: tr.length,
+ recall_any_at_5:
+ tr.reduce((s, r) => s + r.recall_any_at_5, 0) / tr.length,
+ recall_any_at_10:
+ tr.reduce((s, r) => s + r.recall_any_at_10, 0) / tr.length,
+ },
+ ]),
+ ),
+ per_question: results,
+ },
+ null,
+ 2,
+ ),
+ );
+ console.log(`\nResults saved to ${outPath}`);
+}
+
+const mode = (process.argv[2] || "bm25") as "bm25" | "vector" | "hybrid";
+console.log(`Running LongMemEval-S benchmark in ${mode} mode...`);
+
+if (mode === "bm25") {
+ runBenchmark("bm25").catch(console.error);
+} else {
+ import("../src/providers/embedding/local.js")
+ .then(({ LocalEmbeddingProvider }) => {
+ const provider = new LocalEmbeddingProvider();
+ return runBenchmark(mode, provider);
+ })
+ .catch(console.error);
+}