Skip to content

World's first open-source enterprise world model.

A lease amendment chain answers as-of questions (#683)

WaylandYang2250383a0e3b
diff --git a/.gitignore b/.gitignore
index 335341ec..ab2f1c63 100644
--- a/.gitignore
+++ b/.gitignore
@@ -35,3 +35,6 @@ _*.txt
# 命令写进跑测记录,语料本身不进仓库。
/scripts/bench/corpora/wiki-*-subset.json
/scripts/bench/corpora/wiki-nov2023.json
+/scripts/bench/corpora/contracts/edgar/
+/scripts/bench/corpora/contracts/cuad/
+/scripts/bench/corpora/contracts/lease-kb-*.json
diff --git a/scripts/bench/corpora/contracts/README.md b/scripts/bench/corpora/contracts/README.md
new file mode 100644
index 00000000..a30db330
--- /dev/null
+++ b/scripts/bench/corpora/contracts/README.md
@@ -0,0 +1,68 @@
+# 合同链测量台
+
+问的是:**一份合同被一连串补充协议改写之后,按时点问它的条款,答不答得对。**
+
+召回测量台(`recall.mjs`)量的是事实进没进图;这里量的是进了图之后,时间线排得对不对。
+Blackbaud 总部租约的「二期行权截止日」在 2020 年上半年被补充协议连续改了五次,第十份
+补充协议又改成「触发日后 45 天」;同一年物业从 HPBB1 卖给 Blackbaud 的关联方 BBHQ1,
+第十一份补充协议起房东换了人;买卖价格因为验房下调了 49,435.72 美元。18 道题问的就是
+这些值在某一天是多少。
+
+## 跑一轮
+
+```
+SEC_USER_AGENT="你的名字 [email protected]" py -3 fetch.py
+py -3 lease_bench.py setup --label <名字>
+# 等这个库的 documents.graph_status 全部 done(13 份补充协议约 10 分钟)
+py -3 lease_bench.py score <kb id>
+```
+
+`lease_bench.py` 头部写着环境变量与打分规则。**每轮一个新库,单轮波动约 3 题**,一个
+版本至少跑两轮。服务要连一个配好对话与嵌入模型的工作区。
+
+## 目录
+
+| 路径 | 内容 |
+|---|---|
+| `edgar-manifest.json` | 7 条合同链、62 份 EDGAR 文件:CIK、accession、提交日期、角色、标题 |
+| `fetch.py` | 按清单抓原件到 `edgar/`,每份一个 HTML、一个纯文本 |
+| `lease_bench.py` | `setup` 建库、声明本体、导入总部租约链;`score` 答 18 道按时点的题 |
+| `edgar/`、`lease-kb-*.json` | 抓下来的文件与每轮的库 id,不进仓库(见 `.gitignore`) |
+
+文件是各公司提交给 SEC 的,版权不属于我们,所以只有脚本与清单进仓库,与英伟达语料的
+`fetch-sec-filings.mjs` 同一个处理。清单里除了总部租约链,还有 Blackbaud 的三净租约
+(修订、终止,终止费 14,759,500 美元)、Arrowhead 的四份租约,和金额大量涂黑的 CSG 与
+Comcast 服务协议(默认不抓)。
+
+标题括号里的日期是逐份读开头定的。提交日期不是签署日期:一份 10-Q 常一次附上几份早先
+签的补充协议。从正文自动读签署日期不可靠——会读到它引用的原合同日期——所以清单里没有。
+
+## 真值
+
+读原文定的,写在 `lease_bench.py` 的 `QUESTIONS` 里:
+
+| 补充协议(生效日) | 二期行权截止日 | 二期预算交付截止日 |
+|---|---|---|
+| 第五份(2020-02-18) | 2020-03-17 | 2020-05-07 |
+| 第六份(2020-03-17) | 2020-04-14 | 2020-06-04 |
+| 第七份(2020-04-14) | 2020-05-26 | 2020-07-16 |
+| 第八份(2020-05-26) | 2020-06-09 | 2020-07-30 |
+| 第九份(2020-06-08) | 2020-06-23 | 2020-08-13 |
+| 第十份(2020-06-26) | 触发日后 45 天 | 触发日后 96 天 |
+
+房东 2020-08-13(第十一份)起是 BBHQ1,之前是 HPBB1;租户 Blackbaud;场地 12.98 英亩;
+买价 76,272,484.66 美元,买方 BBHQ1,最晚交割日 2020-12-14。
+
+## 记录(2026-09-13,DeepSeek-V3.2 + bge-m3)
+
+| 版本 | 18 题答对 | 带房东或租户的租约实体 |
+|---|---|---|
+| dev | 5 | 8 |
+| dev + 每块带文件开头 | 5、8 | 9、8 |
+| dev + #670 + 文件开头 | 8、8 | 9、3 |
+| 再加日期格式与本体里的补充协议类 | 10、5 | 3、4 |
+| 再加时态引擎乱序修复(#679 第一版) | 8、11 | 2、2 |
+| 同上,人工合并一次两个租约实体之后 | 9、12 | 1、1 |
+
+剩下的错主要是:模型偶尔不给改写后的值起点(#681 §1)、正确的合并被 10% 人工抽样挡着
+(0026,照设计)、二期的截止日偶尔挂到二期租约上(#681 §3)。
diff --git a/scripts/bench/corpora/contracts/edgar-manifest.json b/scripts/bench/corpora/contracts/edgar-manifest.json
new file mode 100644
index 00000000..502cd9ed
--- /dev/null
+++ b/scripts/bench/corpora/contracts/edgar-manifest.json
@@ -0,0 +1,607 @@
+{
+ "source": "SEC EDGAR, public filings",
+ "note": "文件版权属于各提交公司,只存脚本和清单,不进仓库。",
+ "chains": [
+ {
+ "id": "blackbaud-hq-lease",
+ "summary": "Blackbaud 查尔斯顿总部租约。2016 年签,到 2020 年共 11 份补充协议;2020 年房东 HPBB1 把物业卖给 BBHQ1,买卖协议及其两份修订一并收入,第 11 份补充协议的房东已是 BBHQ1。",
+ "docs": [
+ {
+ "cik": "1280058",
+ "filed": "2016-08-04",
+ "accession": "0001280058-16-000072",
+ "file": "a2016q210-qexx1084.htm",
+ "role": "base",
+ "title": "Lease Agreement by and between HPBB1, LLC and Blackbaud, Inc.",
+ "seq": 0
+ },
+ {
+ "cik": "1280058",
+ "filed": "2016-11-04",
+ "accession": "0001280058-16-000081",
+ "file": "a2016q310-qexx1087.htm",
+ "role": "amendment",
+ "title": "First Amendment to Lease Agreement (effective 2016-08-22)",
+ "seq": 1
+ },
+ {
+ "cik": "1280058",
+ "filed": "2018-02-20",
+ "accession": "0001280058-18-000007",
+ "file": "a201710-kexx1093.htm",
+ "role": "amendment",
+ "title": "Second Amendment to Lease Agreement (effective 2017-05-18)",
+ "seq": 2
+ },
+ {
+ "cik": "1280058",
+ "filed": "2018-02-20",
+ "accession": "0001280058-18-000007",
+ "file": "a201710-kexx1094.htm",
+ "role": "amendment",
+ "title": "Third Amendment to Lease Agreement (effective 2017-12-11)",
+ "seq": 3
+ },
+ {
+ "cik": "1280058",
+ "filed": "2018-05-04",
+ "accession": "0001280058-18-000021",
+ "file": "a2018q110-qexx1095.htm",
+ "role": "amendment",
+ "title": "Fourth Amendment to Lease Agreement (effective 2018-02-28)",
+ "seq": 4
+ },
+ {
+ "cik": "1280058",
+ "filed": "2020-08-04",
+ "accession": "0001280058-20-000038",
+ "file": "a2020q210-qexx101.htm",
+ "role": "amendment",
+ "title": "Fifth Amendment to Lease Agreement (effective 2020-02-18)",
+ "seq": 5
+ },
+ {
+ "cik": "1280058",
+ "filed": "2020-08-04",
+ "accession": "0001280058-20-000038",
+ "file": "a2020q210-qexx102.htm",
+ "role": "amendment",
+ "title": "Sixth Amendment to Lease Agreement (effective 2020-03-17)",
+ "seq": 6
+ },
+ {
+ "cik": "1280058",
+ "filed": "2020-08-04",
+ "accession": "0001280058-20-000038",
+ "file": "a2020q210-qexx103.htm",
+ "role": "amendment",
+ "title": "Seventh Amendment to Lease Agreement (effective 2020-04-14)",
+ "seq": 7
+ },
+ {
+ "cik": "1280058",
+ "filed": "2020-08-04",
+ "accession": "0001280058-20-000038",
+ "file": "a2020q210-qexx104.htm",
+ "role": "amendment",
+ "title": "Eighth Amendment to Lease Agreement (effective 2020-05-26)",
+ "seq": 8
+ },
+ {
+ "cik": "1280058",
+ "filed": "2020-08-04",
+ "accession": "0001280058-20-000038",
+ "file": "a2020q210-qexx105.htm",
+ "role": "amendment",
+ "title": "Ninth Amendment to Lease Agreement (effective 2020-06-08)",
+ "seq": 9
+ },
+ {
+ "cik": "1280058",
+ "filed": "2020-08-04",
+ "accession": "0001280058-20-000038",
+ "file": "a2020q210-qexx106.htm",
+ "role": "sale",
+ "title": "Purchase and Sale Agreement, HPBB1, LLC to BBHQ1, LLC (as of 2020-06-17)",
+ "seq": 10
+ },
+ {
+ "cik": "1280058",
+ "filed": "2020-08-04",
+ "accession": "0001280058-20-000038",
+ "file": "a2020q210-qexx107.htm",
+ "role": "amendment",
+ "title": "Tenth Amendment to Lease Agreement (effective 2020-06-26)",
+ "seq": 11
+ },
+ {
+ "cik": "1280058",
+ "filed": "2020-11-03",
+ "accession": "0001280058-20-000052",
+ "file": "a2020q310-qexx101.htm",
+ "role": "sale-amendment",
+ "title": "First Amendment to Purchase and Sale Agreement (2020-07-08)",
+ "seq": 12
+ },
+ {
+ "cik": "1280058",
+ "filed": "2020-11-03",
+ "accession": "0001280058-20-000052",
+ "file": "a2020q310-qexx102.htm",
+ "role": "sale-amendment",
+ "title": "Second Amendment to Purchase and Sale Agreement (July 2020)",
+ "seq": 13
+ },
+ {
+ "cik": "1280058",
+ "filed": "2020-11-03",
+ "accession": "0001280058-20-000052",
+ "file": "a2020q310-qexx103.htm",
+ "role": "amendment",
+ "title": "Eleventh Amendment to Lease Agreement, landlord BBHQ1, LLC (effective 2020-08-13)",
+ "seq": 14
+ }
+ ]
+ },
+ {
+ "id": "blackbaud-triple-net-lease",
+ "summary": "Blackbaud 的三净租约。2008 年签,2013 年与 Duck Pond Creek-SPE 签修订及整治协议,2020 年 11 月与 HP 2000DI 签终止协议,终止费 14,759,500 美元。房东在链条中换过人。",
+ "docs": [
+ {
+ "cik": "1280058",
+ "filed": "2008-12-11",
+ "accession": "0001193125-08-251891",
+ "file": "dex1037.htm",
+ "role": "base",
+ "title": "Triple Net Lease Agreement (dated 2008-10-01)",
+ "seq": 0
+ },
+ {
+ "cik": "1280058",
+ "filed": "2013-03-28",
+ "accession": "0001280058-13-000009",
+ "file": "ex1066mar2013.htm",
+ "role": "amendment",
+ "title": "Lease Amendment and Remediation Agreement, Duck Pond Creek-SPE, LLC",
+ "seq": 1
+ },
+ {
+ "cik": "1280058",
+ "filed": "2021-02-23",
+ "accession": "0001280058-21-000006",
+ "file": "a202010-kexx1040.htm",
+ "role": "termination",
+ "title": "Termination of Triple Net Lease and Mutual Release Agreement (effective 2020-11-04)",
+ "seq": 2
+ }
+ ]
+ },
+ {
+ "id": "arrowhead-madison-lease",
+ "summary": "Arrowhead 在威斯康星麦迪逊的实验室租约,房东 University Research Park。2016 年签,2018 到 2021 年共 7 份补充协议。",
+ "docs": [
+ {
+ "cik": "879407",
+ "filed": "2016-02-09",
+ "accession": "0001564590-16-012515",
+ "file": "arwr-ex101_83.htm",
+ "role": "base",
+ "title": "Lease Agreement, University Research Park, Incorporated and Arrowhead Madison Inc.",
+ "seq": 0
+ },
+ {
+ "cik": "879407",
+ "filed": "2020-11-23",
+ "accession": "0001564590-20-054893",
+ "file": "arwr-ex1023_125.htm",
+ "role": "amendment",
+ "title": "First Amendment to Lease Agreement (effective 2018-10-22)",
+ "seq": 1
+ },
+ {
+ "cik": "879407",
+ "filed": "2020-11-23",
+ "accession": "0001564590-20-054893",
+ "file": "arwr-ex1024_124.htm",
+ "role": "amendment",
+ "title": "Second Amendment to Lease Agreement (effective 2019-01-10)",
+ "seq": 2
+ },
+ {
+ "cik": "879407",
+ "filed": "2020-11-23",
+ "accession": "0001564590-20-054893",
+ "file": "arwr-ex1025_123.htm",
+ "role": "amendment",
+ "title": "Third Amendment to Lease Agreement (effective 2019-01-11)",
+ "seq": 3
+ },
+ {
+ "cik": "879407",
+ "filed": "2020-11-23",
+ "accession": "0001564590-20-054893",
+ "file": "arwr-ex1026_121.htm",
+ "role": "amendment",
+ "title": "Fourth Amendment to Lease Agreement (dated 2019-09-19)",
+ "seq": 4
+ },
+ {
+ "cik": "879407",
+ "filed": "2020-11-23",
+ "accession": "0001564590-20-054893",
+ "file": "arwr-ex1027_122.htm",
+ "role": "amendment",
+ "title": "Fifth Amendment to Lease Agreement",
+ "seq": 5
+ },
+ {
+ "cik": "879407",
+ "filed": "2021-02-04",
+ "accession": "0001564590-21-004319",
+ "file": "arwr-ex103_51.htm",
+ "role": "amendment",
+ "title": "Sixth Amendment to Lease Agreement",
+ "seq": 6
+ },
+ {
+ "cik": "879407",
+ "filed": "2021-02-04",
+ "accession": "0001564590-21-004319",
+ "file": "arwr-ex104_52.htm",
+ "role": "amendment",
+ "title": "Seventh Amendment to Lease Agreement",
+ "seq": 7
+ }
+ ]
+ },
+ {
+ "id": "arrowhead-pasadena-lease",
+ "summary": "Arrowhead 帕萨迪纳总部租约,177 E. Colorado Boulevard。2019 年签,2020 年第一份补充协议,2026 年第三份补充协议。第二份补充协议没有在 EDGAR 公开,第三份会引用它。",
+ "docs": [
+ {
+ "cik": "879407",
+ "filed": "2019-08-05",
+ "accession": "0001564590-19-028760",
+ "file": "arwr-ex101_67.htm",
+ "role": "base",
+ "title": "Office Lease, 177 E. Colorado Boulevard",
+ "seq": 0
+ },
+ {
+ "cik": "879407",
+ "filed": "2021-02-04",
+ "accession": "0001564590-21-004319",
+ "file": "arwr-ex102_50.htm",
+ "role": "amendment",
+ "title": "First Amendment to Office Lease (effective 2020-10-23)",
+ "seq": 1
+ },
+ {
+ "cik": "879407",
+ "filed": "2026-08-04",
+ "accession": "0000879407-26-000057",
+ "file": "exhibit104pasadenalease.htm",
+ "role": "amendment",
+ "title": "Third Amendment to Office Lease (effective 2026-04-27)",
+ "seq": 3
+ }
+ ]
+ },
+ {
+ "id": "arrowhead-south-lake-lease",
+ "summary": "Arrowhead 搬去 177 E. Colorado 之前的帕萨迪纳办公室租约,房东 South Lake Avenue Investors,2012 年签。可用来问某个时点公司同时有哪些租约。",
+ "docs": [
+ {
+ "cik": "879407",
+ "filed": "2012-05-08",
+ "accession": "0001193125-12-218387",
+ "file": "d336351dex101.htm",
+ "role": "base",
+ "title": "Office Lease Agreement, South Lake Avenue Investors LLC",
+ "seq": 0
+ }
+ ]
+ },
+ {
+ "id": "arrowhead-san-diego-lease",
+ "summary": "Arrowhead 圣迭戈租约,房东 ARE-SD Region No. 72。2021 年 11 月签,2023 年第一份补充协议。",
+ "docs": [
+ {
+ "cik": "879407",
+ "filed": "2022-02-02",
+ "accession": "0001564590-22-003616",
+ "file": "arwr-ex102_515.htm",
+ "role": "base",
+ "title": "Lease Agreement, ARE-SD Region No. 72, LLC (effective 2021-11-19)",
+ "seq": 0
+ },
+ {
+ "cik": "879407",
+ "filed": "2023-11-29",
+ "accession": "0001628280-23-040219",
+ "file": "a1039-firstamendmenttole.htm",
+ "role": "amendment",
+ "title": "First Amendment to Lease (dated September 2023)",
+ "seq": 1
+ }
+ ]
+ },
+ {
+ "id": "csg-comcast-services",
+ "summary": "CSG 与 Comcast 的主服务协议。2020 年 1 月 1 日生效的重签版本,加上到 2025 年 11 月为止的 29 份补充协议。金额、期限年份等被涂黑,主协议约 74 万字符,抽取成本高,默认不进演示。",
+ "redacted": true,
+ "docs": [
+ {
+ "cik": "1005757",
+ "filed": "2020-02-21",
+ "accession": "0001564590-20-005691",
+ "file": "csgs-ex1027_251.htm",
+ "role": "base",
+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27",
+ "seq": 0
+ },
+ {
+ "cik": "1005757",
+ "filed": "2020-02-21",
+ "accession": "0001564590-20-005691",
+ "file": "csgs-ex1027a_199.htm",
+ "role": "amendment",
+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27A",
+ "seq": 1
+ },
+ {
+ "cik": "1005757",
+ "filed": "2020-05-07",
+ "accession": "0001564590-20-022554",
+ "file": "csgs-ex1027b_58.htm",
+ "role": "amendment",
+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27B",
+ "seq": 2
+ },
+ {
+ "cik": "1005757",
+ "filed": "2021-02-19",
+ "accession": "0001564590-21-006874",
+ "file": "csgs-ex1027c_34.htm",
+ "role": "amendment",
+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27C",
+ "seq": 3
+ },
+ {
+ "cik": "1005757",
+ "filed": "2021-05-06",
+ "accession": "0001564590-21-024767",
+ "file": "csgs-ex1027d_106.htm",
+ "role": "amendment",
+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27D",
+ "seq": 4
+ },
+ {
+ "cik": "1005757",
+ "filed": "2021-08-05",
+ "accession": "0001564590-21-041434",
+ "file": "csgs-ex1027e_105.htm",
+ "role": "amendment",
+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27E",
+ "seq": 5
+ },
+ {
+ "cik": "1005757",
+ "filed": "2021-08-05",
+ "accession": "0001564590-21-041434",
+ "file": "csgs-ex1027f_102.htm",
+ "role": "amendment",
+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27F",
+ "seq": 6
+ },
+ {
+ "cik": "1005757",
+ "filed": "2021-08-05",
+ "accession": "0001564590-21-041434",
+ "file": "csgs-ex1027g_203.htm",
+ "role": "amendment",
+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27G",
+ "seq": 7
+ },
+ {
+ "cik": "1005757",
+ "filed": "2021-08-05",
+ "accession": "0001564590-21-041434",
+ "file": "csgs-ex1027h_101.htm",
+ "role": "amendment",
+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27H",
+ "seq": 8
+ },
+ {
+ "cik": "1005757",
+ "filed": "2021-11-04",
+ "accession": "0001564590-21-054408",
+ "file": "csgs-ex10_xxviii.htm",
+ "role": "amendment",
+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27I",
+ "seq": 9
+ },
+ {
+ "cik": "1005757",
+ "filed": "2021-11-04",
+ "accession": "0001564590-21-054408",
+ "file": "csgs-ex10_xxviij.htm",
+ "role": "amendment",
+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27J",
+ "seq": 10
+ },
+ {
+ "cik": "1005757",
+ "filed": "2022-02-18",
+ "accession": "0000950170-22-001463",
+ "file": "csgs-ex10_271.htm",
+ "role": "amendment",
+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27K",
+ "seq": 11
+ },
+ {
+ "cik": "1005757",
+ "filed": "2022-02-18",
+ "accession": "0000950170-22-001463",
+ "file": "csgs-ex10_272.htm",
+ "role": "amendment",
+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27L",
+ "seq": 12
+ },
+ {
+ "cik": "1005757",
+ "filed": "2022-02-18",
+ "accession": "0000950170-22-001463",
+ "file": "csgs-ex10_273.htm",
+ "role": "amendment",
+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27M",
+ "seq": 13
+ },
+ {
+ "cik": "1005757",
+ "filed": "2022-02-18",
+ "accession": "0000950170-22-001463",
+ "file": "csgs-ex10_274.htm",
+ "role": "amendment",
+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27N",
+ "seq": 14
+ },
+ {
+ "cik": "1005757",
+ "filed": "2022-05-05",
+ "accession": "0001564590-22-018177",
+ "file": "csgs-ex10_271.htm",
+ "role": "amendment",
+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27O",
+ "seq": 15
+ },
+ {
+ "cik": "1005757",
+ "filed": "2022-05-05",
+ "accession": "0001564590-22-018177",
+ "file": "csgs-ex10_272.htm",
+ "role": "amendment",
+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27P",
+ "seq": 16
+ },
+ {
+ "cik": "1005757",
+ "filed": "2022-05-05",
+ "accession": "0001564590-22-018177",
+ "file": "csgs-ex10_273.htm",
+ "role": "amendment",
+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27Q",
+ "seq": 17
+ },
+ {
+ "cik": "1005757",
+ "filed": "2022-05-05",
+ "accession": "0001564590-22-018177",
+ "file": "csgs-ex10_274.htm",
+ "role": "amendment",
+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27R",
+ "seq": 18
+ },
+ {
+ "cik": "1005757",
+ "filed": "2022-05-05",
+ "accession": "0001564590-22-018177",
+ "file": "csgs-ex10_275.htm",
+ "role": "amendment",
+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27S",
+ "seq": 19
+ },
+ {
+ "cik": "1005757",
+ "filed": "2022-05-05",
+ "accession": "0001564590-22-018177",
+ "file": "csgs-ex10_276.htm",
+ "role": "amendment",
+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27T",
+ "seq": 20
+ },
+ {
+ "cik": "1005757",
+ "filed": "2023-08-03",
+ "accession": "0000950170-23-037789",
+ "file": "csgs-ex10_27u.htm",
+ "role": "amendment",
+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27U",
+ "seq": 21
+ },
+ {
+ "cik": "1005757",
+ "filed": "2023-08-03",
+ "accession": "0000950170-23-037789",
+ "file": "csgs-ex10_27v.htm",
+ "role": "amendment",
+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27V",
+ "seq": 22
+ },
+ {
+ "cik": "1005757",
+ "filed": "2023-08-03",
+ "accession": "0000950170-23-037789",
+ "file": "csgs-ex10_27w.htm",
+ "role": "amendment",
+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27W",
+ "seq": 23
+ },
+ {
+ "cik": "1005757",
+ "filed": "2024-02-16",
+ "accession": "0000950170-24-016286",
+ "file": "csgs-ex10_27x.htm",
+ "role": "amendment",
+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27X",
+ "seq": 24
+ },
+ {
+ "cik": "1005757",
+ "filed": "2024-08-08",
+ "accession": "0000950170-24-093482",
+ "file": "csgs-ex10_27y.htm",
+ "role": "amendment",
+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27Y",
+ "seq": 25
+ },
+ {
+ "cik": "1005757",
+ "filed": "2024-11-07",
+ "accession": "0000950170-24-122869",
+ "file": "csgs-ex10_27z.htm",
+ "role": "amendment",
+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27Z",
+ "seq": 26
+ },
+ {
+ "cik": "1005757",
+ "filed": "2025-02-20",
+ "accession": "0000950170-25-023962",
+ "file": "csgs-ex10_27aa.htm",
+ "role": "amendment",
+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27AA",
+ "seq": 27
+ },
+ {
+ "cik": "1005757",
+ "filed": "2025-08-07",
+ "accession": "0000950170-25-104694",
+ "file": "csgs-ex10_27ab.htm",
+ "role": "amendment",
+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27AB",
+ "seq": 28
+ },
+ {
+ "cik": "1005757",
+ "filed": "2025-11-06",
+ "accession": "0001193125-25-269064",
+ "file": "csgs-ex10_27ac.htm",
+ "role": "amendment",
+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27AC",
+ "seq": 29
+ }
+ ]
+ }
+ ]
+}
\ No newline at end of file
diff --git a/scripts/bench/corpora/contracts/fetch.py b/scripts/bench/corpora/contracts/fetch.py
new file mode 100644
index 00000000..cecad6f1
--- /dev/null
+++ b/scripts/bench/corpora/contracts/fetch.py
@@ -0,0 +1,86 @@
+"""抓合同链语料:按 edgar-manifest.json 从 SEC EDGAR 取原件。
+
+一条链是一份原合同,加上后来改写它的补充协议、物业转让与终止协议。每份存原始 HTML,
+旁边放一份纯文本便于阅读与比对。
+
+**文件不进仓库。** 它们是各公司提交给 SEC 的文件,公开可取但版权不属于我们,与
+`fetch-sec-filings.mjs` 抓的英伟达语料同一个处理:仓库里只有脚本和清单,跑一次就有。
+
+用法:
+ SEC_USER_AGENT="你的名字 [email protected]" py -3 fetch.py [--with-redacted]
+
+SEC 要求 User-Agent 带能联系上的地址,限速每秒 10 次;这个脚本每秒约 3 次。
+CSG 与 Comcast 那条链金额涂黑、主协议约 4 MB,默认不抓,要 `--with-redacted`。
+"""
+
+import html
+import json
+import os
+import re
+import subprocess
+import sys
+import time
+from pathlib import Path
+
+HERE = Path(__file__).resolve().parent
+OUT = HERE / "edgar"
+UA = os.environ.get("SEC_USER_AGENT", "Utopia-bench ([email protected])")
+
+
+def url_of(doc):
+ acc = doc["accession"].replace("-", "")
+ return f"https://www.sec.gov/Archives/edgar/data/{int(doc['cik'])}/{acc}/{doc['file']}"
+
+
+def to_text(raw: bytes) -> str:
+ t = raw.decode("utf-8", "replace")
+ t = re.sub(r"(?is)<(script|style).*?</\1>", " ", t)
+ t = re.sub(r"(?i)<br\s*/?>|</(p|div|tr|li|h[1-6])>", "\n", t)
+ t = re.sub(r"<[^>]+>", " ", t)
+ t = html.unescape(t).replace("\xa0", " ")
+ t = re.sub(r"[ \t]+", " ", t)
+ return re.sub(r"\n\s*\n+", "\n\n", t).strip() + "\n"
+
+
+def fetch(url: str) -> bytes:
+ # 走 curl 不走 urllib:与 fetch-sec-filings.mjs 同一个理由,本机的代理只有 curl 读
+ code = b""
+ for attempt in range(4):
+ r = subprocess.run(
+ ["curl", "-sSL", "--max-time", "120", "-H", f"User-Agent: {UA}", "-w", "%{http_code}", url],
+ capture_output=True,
+ )
+ body, code = r.stdout[:-3], r.stdout[-3:]
+ if code == b"200" and body:
+ return body
+ time.sleep(3 + attempt * 5)
+ raise RuntimeError(f"failed after retries: {url} ({code!r})")
+
+
+def main():
+ with_redacted = "--with-redacted" in sys.argv
+ manifest = json.loads((HERE / "edgar-manifest.json").read_text(encoding="utf-8"))
+ total = 0
+ for chain in manifest["chains"]:
+ if chain.get("redacted") and not with_redacted:
+ print(f"skip {chain['id']}(涂黑,要抓就加 --with-redacted)")
+ continue
+ folder = OUT / chain["id"]
+ folder.mkdir(parents=True, exist_ok=True)
+ for doc in chain["docs"]:
+ stem = f"{doc['seq']:02d}-{doc['role']}-{doc['filed']}"
+ dest = folder / f"{stem}.html"
+ if dest.exists() and dest.stat().st_size > 0:
+ total += 1
+ continue
+ body = fetch(url_of(doc))
+ dest.write_bytes(body)
+ (folder / f"{stem}.txt").write_text(to_text(body), encoding="utf-8")
+ total += 1
+ print(f"ok {chain['id']}/{stem} {len(body):>8} bytes")
+ time.sleep(0.35)
+ print(f"\n{total} 份文件在 {OUT}")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/scripts/bench/corpora/contracts/lease_bench.py b/scripts/bench/corpora/contracts/lease_bench.py
new file mode 100644
index 00000000..dfe2411d
--- /dev/null
+++ b/scripts/bench/corpora/contracts/lease_bench.py
@@ -0,0 +1,265 @@
+"""租约链测量台:Blackbaud 总部租约与它的补充协议,按时点问 18 道题。
+
+ py -3 lease_bench.py setup --label <名字> [--with-base]
+ py -3 lease_bench.py score <kb id>
+
+**setup** 建一个新库,声明一套通用的租约与物业买卖本体,把 `edgar/blackbaud-hq-lease/`
+里的文件导进去(先跑 `fetch.py`)。默认不导 2016 年的原租约——它有 145 块,一轮要两个
+小时,而 18 道题问的全在补充协议里;`--with-base` 才导。库 id 写进 `lease-kb-<名字>.json`。
+等 `documents.graph_status` 全部 done 再打分。
+
+**score** 用产品自己的读法答题:取每个租约、买卖协议实体的事实,留下服务端算出的成立区间
+(`holds_from` / `holds_to`)盖住那个时刻的,比那一刻的值。那一刻所有同类实体上的值
+**恰好**是真值算对;真值在、但还有别的值算部分对。真值是逐份读原文定的,写在下面 QUESTIONS 里。
+
+本体是通用的:它说任何租约、买卖协议都有的东西(房东、租户、起租日、选择权截止日、买价),
+单值的标 functional——后一份补充协议才关得上前一份写的值。里面没有一个字是这份合同特有的。
+
+**每轮一个新库,单轮波动约 3 题**,一个版本至少跑两轮再下结论(2026-09-13 的记录见 README)。
+
+环境变量:
+ BENCH_BASE 服务地址,缺省 http://127.0.0.1:1516
+ BENCH_TOKEN 登录令牌;没有就用下面两个登录
+ BENCH_EMAIL / BENCH_PASSWORD 同 recall.mjs,缺省 [email protected] / benchbench123
+"""
+
+import json
+import os
+import sys
+import time
+import urllib.error
+import urllib.request
+from collections import defaultdict
+from pathlib import Path
+
+HERE = Path(__file__).resolve().parent
+BASE = os.environ.get("BENCH_BASE", "http://127.0.0.1:1516")
+# 本机的 HTTP(S)_PROXY 指向本地代理,连 localhost 不该经过它
+OPENER = urllib.request.build_opener(urllib.request.ProxyHandler({}))
+TOKEN = None
+
+
+def api(method, path, body=None):
+ data = json.dumps(body).encode() if body is not None else None
+ req = urllib.request.Request(BASE + "/api/v1" + path, data=data, method=method)
+ if TOKEN:
+ req.add_header("Authorization", f"Bearer {TOKEN}")
+ if data is not None:
+ req.add_header("Content-Type", "application/json")
+ try:
+ with OPENER.open(req, timeout=120) as res:
+ raw = res.read()
+ return json.loads(raw) if raw else None
+ except urllib.error.HTTPError as e:
+ raise SystemExit(f"{method} {path} -> {e.code}: {e.read()[:400]!r}")
+
+
+def login():
+ global TOKEN
+ TOKEN = os.environ.get("BENCH_TOKEN")
+ if TOKEN:
+ return
+ email = os.environ.get("BENCH_EMAIL", "[email protected]")
+ password = os.environ.get("BENCH_PASSWORD", "benchbench123")
+ TOKEN = api("POST", "/auth/login", {"email": email, "password": password})["token"]
+
+
+# ---------------------------------------------------------------- setup
+
+CLASSES = [
+ ("organization", "Organization", "A company, LLC, fund or public body that can be a party to an agreement."),
+ ("lease", "Lease", "A lease agreement together with all its amendments; one lease is one entity however many amendments change it."),
+ ("amendment", "Amendment", "A document that changes an existing agreement. The terms it changes belong to the agreement it amends, not to the amendment."),
+ ("property", "Property", "A parcel of real property or a building."),
+ ("purchase_agreement", "Purchase agreement", "An agreement to buy and sell real property, together with its amendments."),
+]
+
+# (key, label, kind, domain, range 或 datatype, 单位, functional, 描述)
+AXIOMS = [
+ ("landlord", "landlord", "relation", "lease", "organization", None, True,
+ "The party leasing the premises out. A lease has one landlord at a time; a new owner of the property becomes the new landlord."),
+ ("tenant", "tenant", "relation", "lease", "organization", None, True,
+ "The party leasing the premises."),
+ ("leased_property", "leased property", "relation", "lease", "property", None, False,
+ "Real property covered by the lease."),
+ ("seller", "seller", "relation", "purchase_agreement", "organization", None, True,
+ "The party selling the property under a purchase agreement."),
+ ("purchaser", "purchaser", "relation", "purchase_agreement", "organization", None, True,
+ "The party buying the property under a purchase agreement."),
+ ("property_sold", "property sold", "relation", "purchase_agreement", "property", None, False,
+ "The property a purchase agreement sells."),
+ ("affiliate_of", "affiliate of", "relation", "organization", "organization", None, False,
+ "An organization controlled by, or under common control with, another."),
+ ("amends", "amends", "relation", "amendment", "lease|purchase_agreement", None, False,
+ "The agreement an amendment changes."),
+ ("effective_date", "effective date", "attribute", "amendment", "date", None, True,
+ "The date an amendment takes effect."),
+ ("premises_acres", "premises area", "attribute", "lease", "number", "acres", True,
+ "Land area of the leased premises, in acres."),
+ ("lease_term_years", "initial term", "attribute", "lease", "number", "years", True,
+ "Length of the initial term, in years."),
+ ("commencement_date", "commencement date", "attribute", "lease", "date", None, True,
+ "The date the lease term begins. When the date is defined by a condition, record the fixed date the text gives."),
+ ("expansion_option_deadline", "expansion option exercise deadline", "attribute", "lease", "date", None, True,
+ "The last day the tenant may exercise an option to expand the premises, such as a Phase 2 exercise deadline."),
+ ("expansion_lease_deadline", "expansion lease signing deadline", "attribute", "lease", "date", None, True,
+ "The last day for the parties to sign the lease for the expansion phase."),
+ ("expansion_budget_deadline", "expansion budget delivery deadline", "attribute", "lease", "date", None, True,
+ "The last day for the landlord to deliver the preliminary budget for the expansion phase."),
+ ("purchase_price", "purchase price", "attribute", "purchase_agreement", "number", "USD", True,
+ "The price payable for the property, in US dollars."),
+ ("inspection_period_end", "inspection period end", "attribute", "purchase_agreement", "date", None, True,
+ "The date the purchaser's inspection period ends."),
+ ("outside_closing_date", "outside closing date", "attribute", "purchase_agreement", "date", None, True,
+ "The latest date by which the sale must close."),
+]
+
+# (edgar/blackbaud-hq-lease 里的文件, 开头读出的文件日期, 上传名)。第二份买卖协议修订只写了
+# 「July 2020」没写哪天,不给日期
+DOCS = [
+ ("00-base-2016-08-04.html", "2016-05-16", "hq-lease-2016-05-16-lease-agreement.html"),
+ ("01-amendment-2016-11-04.html", "2016-08-22", "hq-lease-2016-08-22-first-amendment.html"),
+ ("02-amendment-2018-02-20.html", "2017-05-18", "hq-lease-2017-05-18-second-amendment.html"),
+ ("03-amendment-2018-02-20.html", "2017-12-11", "hq-lease-2017-12-11-third-amendment.html"),
+ ("04-amendment-2018-05-04.html", "2018-02-28", "hq-lease-2018-02-28-fourth-amendment.html"),
+ ("05-amendment-2020-08-04.html", "2020-02-18", "hq-lease-2020-02-18-fifth-amendment.html"),
+ ("06-amendment-2020-08-04.html", "2020-03-17", "hq-lease-2020-03-17-sixth-amendment.html"),
+ ("07-amendment-2020-08-04.html", "2020-04-14", "hq-lease-2020-04-14-seventh-amendment.html"),
+ ("08-amendment-2020-08-04.html", "2020-05-26", "hq-lease-2020-05-26-eighth-amendment.html"),
+ ("09-amendment-2020-08-04.html", "2020-06-08", "hq-lease-2020-06-08-ninth-amendment.html"),
+ ("11-amendment-2020-08-04.html", "2020-06-26", "hq-lease-2020-06-26-tenth-amendment.html"),
+ ("12-sale-amendment-2020-11-03.html", "2020-07-08", "hq-sale-2020-07-08-psa-first-amendment.html"),
+ ("13-sale-amendment-2020-11-03.html", None, "hq-sale-2020-07-psa-second-amendment.html"),
+ ("14-amendment-2020-11-03.html", "2020-08-13", "hq-lease-2020-08-13-eleventh-amendment.html"),
+]
+
+
+def setup(label, with_base):
+ folder = HERE / "edgar" / "blackbaud-hq-lease"
+ if not folder.exists():
+ raise SystemExit("先跑 fetch.py")
+ ws = api("GET", "/workspaces")[0]["id"]
+ kb = api("POST", f"/workspaces/{ws}/kbs", {"name": f"Blackbaud HQ lease ({label})", "ontology_packs": []})["id"]
+ print("kb", kb)
+ time.sleep(4)
+ # 与召回测量台同一个理由:抽取之后会改图的开关一律关掉,量的是抽取本身
+ api("PATCH", f"/kbs/{kb}", {"auto_extend_ontology": False, "materialize_inferences": False, "governance": False})
+ ids = {}
+ for key, name, desc in CLASSES:
+ ids[key] = api("POST", f"/kbs/{kb}/ontology/entity-types", {"key": key, "label": name, "description": desc})["id"]
+ for key, name, kind, domain, rng, unit, functional, desc in AXIOMS:
+ body = {"key": key, "label": name, "kind": kind, "temporal": "state", "functional": functional,
+ "description": desc, "domains": [ids[domain]]}
+ if kind == "attribute":
+ body["datatype"] = rng
+ if unit:
+ body["unit"] = unit
+ else:
+ body["ranges"] = [ids[r] for r in rng.split("|")]
+ api("POST", f"/kbs/{kb}/ontology/relation-types", body)
+ print(f"声明 {len(CLASSES)} 个类、{len(AXIOMS)} 条关系与属性")
+ for src, date, name in DOCS:
+ if src.startswith("00-") and not with_base:
+ continue
+ body = {"filename": name, "content": (folder / src).read_text(encoding="utf-8", errors="replace")}
+ if date:
+ body["doc_time"] = f"{date}T00:00:00Z"
+ api("POST", f"/kbs/{kb}/ingest", body)
+ print("导入", name)
+ (HERE / f"lease-kb-{label}.json").write_text(json.dumps({"kb": kb, "base": BASE}, indent=1), encoding="utf-8")
+
+
+# ---------------------------------------------------------------- score
+
+# (题目, 类, 谓词, 时刻, 真值)
+QUESTIONS = [
+ ("expansion option deadline, 2020-03-01", "lease", "expansion_option_deadline", "2020-03-01", "2020-03-17"),
+ ("expansion option deadline, 2020-04-01", "lease", "expansion_option_deadline", "2020-04-01", "2020-04-14"),
+ ("expansion option deadline, 2020-05-01", "lease", "expansion_option_deadline", "2020-05-01", "2020-05-26"),
+ ("expansion option deadline, 2020-06-01", "lease", "expansion_option_deadline", "2020-06-01", "2020-06-09"),
+ ("expansion option deadline, 2020-06-15", "lease", "expansion_option_deadline", "2020-06-15", "2020-06-23"),
+ ("expansion budget deadline, 2020-03-01", "lease", "expansion_budget_deadline", "2020-03-01", "2020-05-07"),
+ ("expansion budget deadline, 2020-04-01", "lease", "expansion_budget_deadline", "2020-04-01", "2020-06-04"),
+ ("expansion budget deadline, 2020-05-01", "lease", "expansion_budget_deadline", "2020-05-01", "2020-07-16"),
+ ("expansion budget deadline, 2020-06-01", "lease", "expansion_budget_deadline", "2020-06-01", "2020-07-30"),
+ ("expansion budget deadline, 2020-06-15", "lease", "expansion_budget_deadline", "2020-06-15", "2020-08-13"),
+ ("landlord, 2019-01-01", "lease", "landlord", "2019-01-01", "HPBB1"),
+ ("landlord, 2020-07-01", "lease", "landlord", "2020-07-01", "HPBB1"),
+ ("landlord, 2020-09-01", "lease", "landlord", "2020-09-01", "BBHQ1"),
+ ("tenant, 2020-09-01", "lease", "tenant", "2020-09-01", "BLACKBAUD"),
+ ("premises acres, 2019-01-01", "lease", "premises_acres", "2019-01-01", "12.98"),
+ ("purchase price, 2020-08-01", "purchase_agreement", "purchase_price", "2020-08-01", "76272484.66"),
+ ("purchaser, 2020-08-01", "purchase_agreement", "purchaser", "2020-08-01", "BBHQ1"),
+ ("outside closing date, 2020-08-01", "purchase_agreement", "outside_closing_date", "2020-08-01", "2020-12-14"),
+]
+
+
+def norm(v):
+ s = str(v).strip().upper()
+ s = s.split(",")[0].strip() if not s.replace(",", "").replace(".", "").isdigit() else s.replace(",", "")
+ return s[:10] if len(s) >= 10 and s[4:5] == "-" else s
+
+
+def holds(f, t):
+ start, end = f.get("holds_from"), f.get("holds_to")
+ return (start is None or start[:10] <= t) and (end is None or end[:10] > t)
+
+
+def score(kb):
+ by_class = defaultdict(list)
+ type_ids = {t["key"]: t["id"] for t in api("GET", f"/kbs/{kb}/ontology")["entity_types"]}
+ for cls in {q[1] for q in QUESTIONS}:
+ # 这个接口分页,默认一页 12 个:不翻页就只看到前 12 个实体
+ page_no = 0
+ while True:
+ page = api("GET", f"/kbs/{kb}/ontology/entity-types/{type_ids[cls]}/entities?page={page_no}&per=100")
+ by_class[cls].extend(page["entities"])
+ if not page["entities"] or len(by_class[cls]) >= page.get("total", 0):
+ break
+ page_no += 1
+ facts = {}
+ for ents in by_class.values():
+ for e in ents:
+ facts[e["id"]] = [f for f in api("GET", f"/kbs/{kb}/entities/{e['id']}")["facts"] if f.get("direction") == "out"]
+
+ carriers = [e["name"] for e in by_class["lease"] if any(f["predicate_key"] in ("landlord", "tenant") for f in facts[e["id"]])]
+ right = partial = 0
+ for label, cls, pred, t, truth in QUESTIONS:
+ values = set()
+ for e in by_class[cls]:
+ for f in facts[e["id"]]:
+ if f["predicate_key"] == pred and holds(f, t):
+ v = f["other_name"] if f.get("other_name") else (f.get("object_value") or {}).get("value")
+ values.add(norm(v))
+ want = norm(truth)
+ hit = any(v.startswith(want) for v in values)
+ if hit and len(values) == 1:
+ right += 1
+ mark = "right"
+ elif hit:
+ partial += 1
+ mark = "partial"
+ else:
+ mark = "wrong"
+ print(f"{mark:8} {label:40} want {truth:14} got {sorted(values)}")
+ print(f"\nright {right}/{len(QUESTIONS)}, partial {partial}")
+ print(f"lease entities carrying a landlord or tenant: {len(carriers)}")
+ for name in carriers:
+ print(" ", name[:90])
+
+
+def main():
+ if len(sys.argv) < 3 or sys.argv[1] not in ("setup", "score"):
+ raise SystemExit(__doc__)
+ login()
+ if sys.argv[1] == "setup":
+ label = next((a.split("=", 1)[1] for a in sys.argv if a.startswith("--label=")), None)
+ if not label and "--label" in sys.argv:
+ label = sys.argv[sys.argv.index("--label") + 1]
+ setup(label or "run", "--with-base" in sys.argv)
+ else:
+ score(sys.argv[2])
+
+
+if __name__ == "__main__":
+ main()