World's first open-source enterprise world model.
Rust agent-memorybitemporalgraphragknowledge-baseknowledge-graphllmontologypgvectorpostgresqlragrustself-hostedsemantic-searchtemporal-knowledge-graphworld-model
A lease amendment chain answers as-of questions (#683)
WaylandYang2250383a0e3b
diff --git a/.gitignore b/.gitignoreindex 335341ec..ab2f1c63 100644--- a/.gitignore+++ b/.gitignore@@ -35,3 +35,6 @@ _*.txt# 命令写进跑测记录,语料本身不进仓库。/scripts/bench/corpora/wiki-*-subset.json/scripts/bench/corpora/wiki-nov2023.json+/scripts/bench/corpora/contracts/edgar/+/scripts/bench/corpora/contracts/cuad/+/scripts/bench/corpora/contracts/lease-kb-*.jsondiff --git a/scripts/bench/corpora/contracts/README.md b/scripts/bench/corpora/contracts/README.mdnew file mode 100644index 00000000..a30db330--- /dev/null+++ b/scripts/bench/corpora/contracts/README.md@@ -0,0 +1,68 @@+# 合同链测量台++问的是:**一份合同被一连串补充协议改写之后,按时点问它的条款,答不答得对。**++召回测量台(`recall.mjs`)量的是事实进没进图;这里量的是进了图之后,时间线排得对不对。+Blackbaud 总部租约的「二期行权截止日」在 2020 年上半年被补充协议连续改了五次,第十份+补充协议又改成「触发日后 45 天」;同一年物业从 HPBB1 卖给 Blackbaud 的关联方 BBHQ1,+第十一份补充协议起房东换了人;买卖价格因为验房下调了 49,435.72 美元。18 道题问的就是+这些值在某一天是多少。++## 跑一轮++```+SEC_USER_AGENT="你的名字 [email protected]" py -3 fetch.py+py -3 lease_bench.py setup --label <名字>+# 等这个库的 documents.graph_status 全部 done(13 份补充协议约 10 分钟)+py -3 lease_bench.py score <kb id>+```++`lease_bench.py` 头部写着环境变量与打分规则。**每轮一个新库,单轮波动约 3 题**,一个+版本至少跑两轮。服务要连一个配好对话与嵌入模型的工作区。++## 目录++| 路径 | 内容 |+|---|---|+| `edgar-manifest.json` | 7 条合同链、62 份 EDGAR 文件:CIK、accession、提交日期、角色、标题 |+| `fetch.py` | 按清单抓原件到 `edgar/`,每份一个 HTML、一个纯文本 |+| `lease_bench.py` | `setup` 建库、声明本体、导入总部租约链;`score` 答 18 道按时点的题 |+| `edgar/`、`lease-kb-*.json` | 抓下来的文件与每轮的库 id,不进仓库(见 `.gitignore`) |++文件是各公司提交给 SEC 的,版权不属于我们,所以只有脚本与清单进仓库,与英伟达语料的+`fetch-sec-filings.mjs` 同一个处理。清单里除了总部租约链,还有 Blackbaud 的三净租约+(修订、终止,终止费 14,759,500 美元)、Arrowhead 的四份租约,和金额大量涂黑的 CSG 与+Comcast 服务协议(默认不抓)。++标题括号里的日期是逐份读开头定的。提交日期不是签署日期:一份 10-Q 常一次附上几份早先+签的补充协议。从正文自动读签署日期不可靠——会读到它引用的原合同日期——所以清单里没有。++## 真值++读原文定的,写在 `lease_bench.py` 的 `QUESTIONS` 里:++| 补充协议(生效日) | 二期行权截止日 | 二期预算交付截止日 |+|---|---|---|+| 第五份(2020-02-18) | 2020-03-17 | 2020-05-07 |+| 第六份(2020-03-17) | 2020-04-14 | 2020-06-04 |+| 第七份(2020-04-14) | 2020-05-26 | 2020-07-16 |+| 第八份(2020-05-26) | 2020-06-09 | 2020-07-30 |+| 第九份(2020-06-08) | 2020-06-23 | 2020-08-13 |+| 第十份(2020-06-26) | 触发日后 45 天 | 触发日后 96 天 |++房东 2020-08-13(第十一份)起是 BBHQ1,之前是 HPBB1;租户 Blackbaud;场地 12.98 英亩;+买价 76,272,484.66 美元,买方 BBHQ1,最晚交割日 2020-12-14。++## 记录(2026-09-13,DeepSeek-V3.2 + bge-m3)++| 版本 | 18 题答对 | 带房东或租户的租约实体 |+|---|---|---|+| dev | 5 | 8 |+| dev + 每块带文件开头 | 5、8 | 9、8 |+| dev + #670 + 文件开头 | 8、8 | 9、3 |+| 再加日期格式与本体里的补充协议类 | 10、5 | 3、4 |+| 再加时态引擎乱序修复(#679 第一版) | 8、11 | 2、2 |+| 同上,人工合并一次两个租约实体之后 | 9、12 | 1、1 |++剩下的错主要是:模型偶尔不给改写后的值起点(#681 §1)、正确的合并被 10% 人工抽样挡着+(0026,照设计)、二期的截止日偶尔挂到二期租约上(#681 §3)。diff --git a/scripts/bench/corpora/contracts/edgar-manifest.json b/scripts/bench/corpora/contracts/edgar-manifest.jsonnew file mode 100644index 00000000..502cd9ed--- /dev/null+++ b/scripts/bench/corpora/contracts/edgar-manifest.json@@ -0,0 +1,607 @@+{+ "source": "SEC EDGAR, public filings",+ "note": "文件版权属于各提交公司,只存脚本和清单,不进仓库。",+ "chains": [+ {+ "id": "blackbaud-hq-lease",+ "summary": "Blackbaud 查尔斯顿总部租约。2016 年签,到 2020 年共 11 份补充协议;2020 年房东 HPBB1 把物业卖给 BBHQ1,买卖协议及其两份修订一并收入,第 11 份补充协议的房东已是 BBHQ1。",+ "docs": [+ {+ "cik": "1280058",+ "filed": "2016-08-04",+ "accession": "0001280058-16-000072",+ "file": "a2016q210-qexx1084.htm",+ "role": "base",+ "title": "Lease Agreement by and between HPBB1, LLC and Blackbaud, Inc.",+ "seq": 0+ },+ {+ "cik": "1280058",+ "filed": "2016-11-04",+ "accession": "0001280058-16-000081",+ "file": "a2016q310-qexx1087.htm",+ "role": "amendment",+ "title": "First Amendment to Lease Agreement (effective 2016-08-22)",+ "seq": 1+ },+ {+ "cik": "1280058",+ "filed": "2018-02-20",+ "accession": "0001280058-18-000007",+ "file": "a201710-kexx1093.htm",+ "role": "amendment",+ "title": "Second Amendment to Lease Agreement (effective 2017-05-18)",+ "seq": 2+ },+ {+ "cik": "1280058",+ "filed": "2018-02-20",+ "accession": "0001280058-18-000007",+ "file": "a201710-kexx1094.htm",+ "role": "amendment",+ "title": "Third Amendment to Lease Agreement (effective 2017-12-11)",+ "seq": 3+ },+ {+ "cik": "1280058",+ "filed": "2018-05-04",+ "accession": "0001280058-18-000021",+ "file": "a2018q110-qexx1095.htm",+ "role": "amendment",+ "title": "Fourth Amendment to Lease Agreement (effective 2018-02-28)",+ "seq": 4+ },+ {+ "cik": "1280058",+ "filed": "2020-08-04",+ "accession": "0001280058-20-000038",+ "file": "a2020q210-qexx101.htm",+ "role": "amendment",+ "title": "Fifth Amendment to Lease Agreement (effective 2020-02-18)",+ "seq": 5+ },+ {+ "cik": "1280058",+ "filed": "2020-08-04",+ "accession": "0001280058-20-000038",+ "file": "a2020q210-qexx102.htm",+ "role": "amendment",+ "title": "Sixth Amendment to Lease Agreement (effective 2020-03-17)",+ "seq": 6+ },+ {+ "cik": "1280058",+ "filed": "2020-08-04",+ "accession": "0001280058-20-000038",+ "file": "a2020q210-qexx103.htm",+ "role": "amendment",+ "title": "Seventh Amendment to Lease Agreement (effective 2020-04-14)",+ "seq": 7+ },+ {+ "cik": "1280058",+ "filed": "2020-08-04",+ "accession": "0001280058-20-000038",+ "file": "a2020q210-qexx104.htm",+ "role": "amendment",+ "title": "Eighth Amendment to Lease Agreement (effective 2020-05-26)",+ "seq": 8+ },+ {+ "cik": "1280058",+ "filed": "2020-08-04",+ "accession": "0001280058-20-000038",+ "file": "a2020q210-qexx105.htm",+ "role": "amendment",+ "title": "Ninth Amendment to Lease Agreement (effective 2020-06-08)",+ "seq": 9+ },+ {+ "cik": "1280058",+ "filed": "2020-08-04",+ "accession": "0001280058-20-000038",+ "file": "a2020q210-qexx106.htm",+ "role": "sale",+ "title": "Purchase and Sale Agreement, HPBB1, LLC to BBHQ1, LLC (as of 2020-06-17)",+ "seq": 10+ },+ {+ "cik": "1280058",+ "filed": "2020-08-04",+ "accession": "0001280058-20-000038",+ "file": "a2020q210-qexx107.htm",+ "role": "amendment",+ "title": "Tenth Amendment to Lease Agreement (effective 2020-06-26)",+ "seq": 11+ },+ {+ "cik": "1280058",+ "filed": "2020-11-03",+ "accession": "0001280058-20-000052",+ "file": "a2020q310-qexx101.htm",+ "role": "sale-amendment",+ "title": "First Amendment to Purchase and Sale Agreement (2020-07-08)",+ "seq": 12+ },+ {+ "cik": "1280058",+ "filed": "2020-11-03",+ "accession": "0001280058-20-000052",+ "file": "a2020q310-qexx102.htm",+ "role": "sale-amendment",+ "title": "Second Amendment to Purchase and Sale Agreement (July 2020)",+ "seq": 13+ },+ {+ "cik": "1280058",+ "filed": "2020-11-03",+ "accession": "0001280058-20-000052",+ "file": "a2020q310-qexx103.htm",+ "role": "amendment",+ "title": "Eleventh Amendment to Lease Agreement, landlord BBHQ1, LLC (effective 2020-08-13)",+ "seq": 14+ }+ ]+ },+ {+ "id": "blackbaud-triple-net-lease",+ "summary": "Blackbaud 的三净租约。2008 年签,2013 年与 Duck Pond Creek-SPE 签修订及整治协议,2020 年 11 月与 HP 2000DI 签终止协议,终止费 14,759,500 美元。房东在链条中换过人。",+ "docs": [+ {+ "cik": "1280058",+ "filed": "2008-12-11",+ "accession": "0001193125-08-251891",+ "file": "dex1037.htm",+ "role": "base",+ "title": "Triple Net Lease Agreement (dated 2008-10-01)",+ "seq": 0+ },+ {+ "cik": "1280058",+ "filed": "2013-03-28",+ "accession": "0001280058-13-000009",+ "file": "ex1066mar2013.htm",+ "role": "amendment",+ "title": "Lease Amendment and Remediation Agreement, Duck Pond Creek-SPE, LLC",+ "seq": 1+ },+ {+ "cik": "1280058",+ "filed": "2021-02-23",+ "accession": "0001280058-21-000006",+ "file": "a202010-kexx1040.htm",+ "role": "termination",+ "title": "Termination of Triple Net Lease and Mutual Release Agreement (effective 2020-11-04)",+ "seq": 2+ }+ ]+ },+ {+ "id": "arrowhead-madison-lease",+ "summary": "Arrowhead 在威斯康星麦迪逊的实验室租约,房东 University Research Park。2016 年签,2018 到 2021 年共 7 份补充协议。",+ "docs": [+ {+ "cik": "879407",+ "filed": "2016-02-09",+ "accession": "0001564590-16-012515",+ "file": "arwr-ex101_83.htm",+ "role": "base",+ "title": "Lease Agreement, University Research Park, Incorporated and Arrowhead Madison Inc.",+ "seq": 0+ },+ {+ "cik": "879407",+ "filed": "2020-11-23",+ "accession": "0001564590-20-054893",+ "file": "arwr-ex1023_125.htm",+ "role": "amendment",+ "title": "First Amendment to Lease Agreement (effective 2018-10-22)",+ "seq": 1+ },+ {+ "cik": "879407",+ "filed": "2020-11-23",+ "accession": "0001564590-20-054893",+ "file": "arwr-ex1024_124.htm",+ "role": "amendment",+ "title": "Second Amendment to Lease Agreement (effective 2019-01-10)",+ "seq": 2+ },+ {+ "cik": "879407",+ "filed": "2020-11-23",+ "accession": "0001564590-20-054893",+ "file": "arwr-ex1025_123.htm",+ "role": "amendment",+ "title": "Third Amendment to Lease Agreement (effective 2019-01-11)",+ "seq": 3+ },+ {+ "cik": "879407",+ "filed": "2020-11-23",+ "accession": "0001564590-20-054893",+ "file": "arwr-ex1026_121.htm",+ "role": "amendment",+ "title": "Fourth Amendment to Lease Agreement (dated 2019-09-19)",+ "seq": 4+ },+ {+ "cik": "879407",+ "filed": "2020-11-23",+ "accession": "0001564590-20-054893",+ "file": "arwr-ex1027_122.htm",+ "role": "amendment",+ "title": "Fifth Amendment to Lease Agreement",+ "seq": 5+ },+ {+ "cik": "879407",+ "filed": "2021-02-04",+ "accession": "0001564590-21-004319",+ "file": "arwr-ex103_51.htm",+ "role": "amendment",+ "title": "Sixth Amendment to Lease Agreement",+ "seq": 6+ },+ {+ "cik": "879407",+ "filed": "2021-02-04",+ "accession": "0001564590-21-004319",+ "file": "arwr-ex104_52.htm",+ "role": "amendment",+ "title": "Seventh Amendment to Lease Agreement",+ "seq": 7+ }+ ]+ },+ {+ "id": "arrowhead-pasadena-lease",+ "summary": "Arrowhead 帕萨迪纳总部租约,177 E. Colorado Boulevard。2019 年签,2020 年第一份补充协议,2026 年第三份补充协议。第二份补充协议没有在 EDGAR 公开,第三份会引用它。",+ "docs": [+ {+ "cik": "879407",+ "filed": "2019-08-05",+ "accession": "0001564590-19-028760",+ "file": "arwr-ex101_67.htm",+ "role": "base",+ "title": "Office Lease, 177 E. Colorado Boulevard",+ "seq": 0+ },+ {+ "cik": "879407",+ "filed": "2021-02-04",+ "accession": "0001564590-21-004319",+ "file": "arwr-ex102_50.htm",+ "role": "amendment",+ "title": "First Amendment to Office Lease (effective 2020-10-23)",+ "seq": 1+ },+ {+ "cik": "879407",+ "filed": "2026-08-04",+ "accession": "0000879407-26-000057",+ "file": "exhibit104pasadenalease.htm",+ "role": "amendment",+ "title": "Third Amendment to Office Lease (effective 2026-04-27)",+ "seq": 3+ }+ ]+ },+ {+ "id": "arrowhead-south-lake-lease",+ "summary": "Arrowhead 搬去 177 E. Colorado 之前的帕萨迪纳办公室租约,房东 South Lake Avenue Investors,2012 年签。可用来问某个时点公司同时有哪些租约。",+ "docs": [+ {+ "cik": "879407",+ "filed": "2012-05-08",+ "accession": "0001193125-12-218387",+ "file": "d336351dex101.htm",+ "role": "base",+ "title": "Office Lease Agreement, South Lake Avenue Investors LLC",+ "seq": 0+ }+ ]+ },+ {+ "id": "arrowhead-san-diego-lease",+ "summary": "Arrowhead 圣迭戈租约,房东 ARE-SD Region No. 72。2021 年 11 月签,2023 年第一份补充协议。",+ "docs": [+ {+ "cik": "879407",+ "filed": "2022-02-02",+ "accession": "0001564590-22-003616",+ "file": "arwr-ex102_515.htm",+ "role": "base",+ "title": "Lease Agreement, ARE-SD Region No. 72, LLC (effective 2021-11-19)",+ "seq": 0+ },+ {+ "cik": "879407",+ "filed": "2023-11-29",+ "accession": "0001628280-23-040219",+ "file": "a1039-firstamendmenttole.htm",+ "role": "amendment",+ "title": "First Amendment to Lease (dated September 2023)",+ "seq": 1+ }+ ]+ },+ {+ "id": "csg-comcast-services",+ "summary": "CSG 与 Comcast 的主服务协议。2020 年 1 月 1 日生效的重签版本,加上到 2025 年 11 月为止的 29 份补充协议。金额、期限年份等被涂黑,主协议约 74 万字符,抽取成本高,默认不进演示。",+ "redacted": true,+ "docs": [+ {+ "cik": "1005757",+ "filed": "2020-02-21",+ "accession": "0001564590-20-005691",+ "file": "csgs-ex1027_251.htm",+ "role": "base",+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27",+ "seq": 0+ },+ {+ "cik": "1005757",+ "filed": "2020-02-21",+ "accession": "0001564590-20-005691",+ "file": "csgs-ex1027a_199.htm",+ "role": "amendment",+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27A",+ "seq": 1+ },+ {+ "cik": "1005757",+ "filed": "2020-05-07",+ "accession": "0001564590-20-022554",+ "file": "csgs-ex1027b_58.htm",+ "role": "amendment",+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27B",+ "seq": 2+ },+ {+ "cik": "1005757",+ "filed": "2021-02-19",+ "accession": "0001564590-21-006874",+ "file": "csgs-ex1027c_34.htm",+ "role": "amendment",+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27C",+ "seq": 3+ },+ {+ "cik": "1005757",+ "filed": "2021-05-06",+ "accession": "0001564590-21-024767",+ "file": "csgs-ex1027d_106.htm",+ "role": "amendment",+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27D",+ "seq": 4+ },+ {+ "cik": "1005757",+ "filed": "2021-08-05",+ "accession": "0001564590-21-041434",+ "file": "csgs-ex1027e_105.htm",+ "role": "amendment",+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27E",+ "seq": 5+ },+ {+ "cik": "1005757",+ "filed": "2021-08-05",+ "accession": "0001564590-21-041434",+ "file": "csgs-ex1027f_102.htm",+ "role": "amendment",+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27F",+ "seq": 6+ },+ {+ "cik": "1005757",+ "filed": "2021-08-05",+ "accession": "0001564590-21-041434",+ "file": "csgs-ex1027g_203.htm",+ "role": "amendment",+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27G",+ "seq": 7+ },+ {+ "cik": "1005757",+ "filed": "2021-08-05",+ "accession": "0001564590-21-041434",+ "file": "csgs-ex1027h_101.htm",+ "role": "amendment",+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27H",+ "seq": 8+ },+ {+ "cik": "1005757",+ "filed": "2021-11-04",+ "accession": "0001564590-21-054408",+ "file": "csgs-ex10_xxviii.htm",+ "role": "amendment",+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27I",+ "seq": 9+ },+ {+ "cik": "1005757",+ "filed": "2021-11-04",+ "accession": "0001564590-21-054408",+ "file": "csgs-ex10_xxviij.htm",+ "role": "amendment",+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27J",+ "seq": 10+ },+ {+ "cik": "1005757",+ "filed": "2022-02-18",+ "accession": "0000950170-22-001463",+ "file": "csgs-ex10_271.htm",+ "role": "amendment",+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27K",+ "seq": 11+ },+ {+ "cik": "1005757",+ "filed": "2022-02-18",+ "accession": "0000950170-22-001463",+ "file": "csgs-ex10_272.htm",+ "role": "amendment",+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27L",+ "seq": 12+ },+ {+ "cik": "1005757",+ "filed": "2022-02-18",+ "accession": "0000950170-22-001463",+ "file": "csgs-ex10_273.htm",+ "role": "amendment",+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27M",+ "seq": 13+ },+ {+ "cik": "1005757",+ "filed": "2022-02-18",+ "accession": "0000950170-22-001463",+ "file": "csgs-ex10_274.htm",+ "role": "amendment",+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27N",+ "seq": 14+ },+ {+ "cik": "1005757",+ "filed": "2022-05-05",+ "accession": "0001564590-22-018177",+ "file": "csgs-ex10_271.htm",+ "role": "amendment",+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27O",+ "seq": 15+ },+ {+ "cik": "1005757",+ "filed": "2022-05-05",+ "accession": "0001564590-22-018177",+ "file": "csgs-ex10_272.htm",+ "role": "amendment",+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27P",+ "seq": 16+ },+ {+ "cik": "1005757",+ "filed": "2022-05-05",+ "accession": "0001564590-22-018177",+ "file": "csgs-ex10_273.htm",+ "role": "amendment",+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27Q",+ "seq": 17+ },+ {+ "cik": "1005757",+ "filed": "2022-05-05",+ "accession": "0001564590-22-018177",+ "file": "csgs-ex10_274.htm",+ "role": "amendment",+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27R",+ "seq": 18+ },+ {+ "cik": "1005757",+ "filed": "2022-05-05",+ "accession": "0001564590-22-018177",+ "file": "csgs-ex10_275.htm",+ "role": "amendment",+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27S",+ "seq": 19+ },+ {+ "cik": "1005757",+ "filed": "2022-05-05",+ "accession": "0001564590-22-018177",+ "file": "csgs-ex10_276.htm",+ "role": "amendment",+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27T",+ "seq": 20+ },+ {+ "cik": "1005757",+ "filed": "2023-08-03",+ "accession": "0000950170-23-037789",+ "file": "csgs-ex10_27u.htm",+ "role": "amendment",+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27U",+ "seq": 21+ },+ {+ "cik": "1005757",+ "filed": "2023-08-03",+ "accession": "0000950170-23-037789",+ "file": "csgs-ex10_27v.htm",+ "role": "amendment",+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27V",+ "seq": 22+ },+ {+ "cik": "1005757",+ "filed": "2023-08-03",+ "accession": "0000950170-23-037789",+ "file": "csgs-ex10_27w.htm",+ "role": "amendment",+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27W",+ "seq": 23+ },+ {+ "cik": "1005757",+ "filed": "2024-02-16",+ "accession": "0000950170-24-016286",+ "file": "csgs-ex10_27x.htm",+ "role": "amendment",+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27X",+ "seq": 24+ },+ {+ "cik": "1005757",+ "filed": "2024-08-08",+ "accession": "0000950170-24-093482",+ "file": "csgs-ex10_27y.htm",+ "role": "amendment",+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27Y",+ "seq": 25+ },+ {+ "cik": "1005757",+ "filed": "2024-11-07",+ "accession": "0000950170-24-122869",+ "file": "csgs-ex10_27z.htm",+ "role": "amendment",+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27Z",+ "seq": 26+ },+ {+ "cik": "1005757",+ "filed": "2025-02-20",+ "accession": "0000950170-25-023962",+ "file": "csgs-ex10_27aa.htm",+ "role": "amendment",+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27AA",+ "seq": 27+ },+ {+ "cik": "1005757",+ "filed": "2025-08-07",+ "accession": "0000950170-25-104694",+ "file": "csgs-ex10_27ab.htm",+ "role": "amendment",+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27AB",+ "seq": 28+ },+ {+ "cik": "1005757",+ "filed": "2025-11-06",+ "accession": "0001193125-25-269064",+ "file": "csgs-ex10_27ac.htm",+ "role": "amendment",+ "title": "CSG Master Subscriber Management System Agreement, Exhibit 10.27AC",+ "seq": 29+ }+ ]+ }+ ]+}\ No newline at end of filediff --git a/scripts/bench/corpora/contracts/fetch.py b/scripts/bench/corpora/contracts/fetch.pynew file mode 100644index 00000000..cecad6f1--- /dev/null+++ b/scripts/bench/corpora/contracts/fetch.py@@ -0,0 +1,86 @@+"""抓合同链语料:按 edgar-manifest.json 从 SEC EDGAR 取原件。++一条链是一份原合同,加上后来改写它的补充协议、物业转让与终止协议。每份存原始 HTML,+旁边放一份纯文本便于阅读与比对。++**文件不进仓库。** 它们是各公司提交给 SEC 的文件,公开可取但版权不属于我们,与+`fetch-sec-filings.mjs` 抓的英伟达语料同一个处理:仓库里只有脚本和清单,跑一次就有。++用法:+ SEC_USER_AGENT="你的名字 [email protected]" py -3 fetch.py [--with-redacted]++SEC 要求 User-Agent 带能联系上的地址,限速每秒 10 次;这个脚本每秒约 3 次。+CSG 与 Comcast 那条链金额涂黑、主协议约 4 MB,默认不抓,要 `--with-redacted`。+"""++import html+import json+import os+import re+import subprocess+import sys+import time+from pathlib import Path++HERE = Path(__file__).resolve().parent+OUT = HERE / "edgar"+UA = os.environ.get("SEC_USER_AGENT", "Utopia-bench ([email protected])")+++def url_of(doc):+ acc = doc["accession"].replace("-", "")+ return f"https://www.sec.gov/Archives/edgar/data/{int(doc['cik'])}/{acc}/{doc['file']}"+++def to_text(raw: bytes) -> str:+ t = raw.decode("utf-8", "replace")+ t = re.sub(r"(?is)<(script|style).*?</\1>", " ", t)+ t = re.sub(r"(?i)<br\s*/?>|</(p|div|tr|li|h[1-6])>", "\n", t)+ t = re.sub(r"<[^>]+>", " ", t)+ t = html.unescape(t).replace("\xa0", " ")+ t = re.sub(r"[ \t]+", " ", t)+ return re.sub(r"\n\s*\n+", "\n\n", t).strip() + "\n"+++def fetch(url: str) -> bytes:+ # 走 curl 不走 urllib:与 fetch-sec-filings.mjs 同一个理由,本机的代理只有 curl 读+ code = b""+ for attempt in range(4):+ r = subprocess.run(+ ["curl", "-sSL", "--max-time", "120", "-H", f"User-Agent: {UA}", "-w", "%{http_code}", url],+ capture_output=True,+ )+ body, code = r.stdout[:-3], r.stdout[-3:]+ if code == b"200" and body:+ return body+ time.sleep(3 + attempt * 5)+ raise RuntimeError(f"failed after retries: {url} ({code!r})")+++def main():+ with_redacted = "--with-redacted" in sys.argv+ manifest = json.loads((HERE / "edgar-manifest.json").read_text(encoding="utf-8"))+ total = 0+ for chain in manifest["chains"]:+ if chain.get("redacted") and not with_redacted:+ print(f"skip {chain['id']}(涂黑,要抓就加 --with-redacted)")+ continue+ folder = OUT / chain["id"]+ folder.mkdir(parents=True, exist_ok=True)+ for doc in chain["docs"]:+ stem = f"{doc['seq']:02d}-{doc['role']}-{doc['filed']}"+ dest = folder / f"{stem}.html"+ if dest.exists() and dest.stat().st_size > 0:+ total += 1+ continue+ body = fetch(url_of(doc))+ dest.write_bytes(body)+ (folder / f"{stem}.txt").write_text(to_text(body), encoding="utf-8")+ total += 1+ print(f"ok {chain['id']}/{stem} {len(body):>8} bytes")+ time.sleep(0.35)+ print(f"\n{total} 份文件在 {OUT}")+++if __name__ == "__main__":+ main()diff --git a/scripts/bench/corpora/contracts/lease_bench.py b/scripts/bench/corpora/contracts/lease_bench.pynew file mode 100644index 00000000..dfe2411d--- /dev/null+++ b/scripts/bench/corpora/contracts/lease_bench.py@@ -0,0 +1,265 @@+"""租约链测量台:Blackbaud 总部租约与它的补充协议,按时点问 18 道题。++ py -3 lease_bench.py setup --label <名字> [--with-base]+ py -3 lease_bench.py score <kb id>++**setup** 建一个新库,声明一套通用的租约与物业买卖本体,把 `edgar/blackbaud-hq-lease/`+里的文件导进去(先跑 `fetch.py`)。默认不导 2016 年的原租约——它有 145 块,一轮要两个+小时,而 18 道题问的全在补充协议里;`--with-base` 才导。库 id 写进 `lease-kb-<名字>.json`。+等 `documents.graph_status` 全部 done 再打分。++**score** 用产品自己的读法答题:取每个租约、买卖协议实体的事实,留下服务端算出的成立区间+(`holds_from` / `holds_to`)盖住那个时刻的,比那一刻的值。那一刻所有同类实体上的值+**恰好**是真值算对;真值在、但还有别的值算部分对。真值是逐份读原文定的,写在下面 QUESTIONS 里。++本体是通用的:它说任何租约、买卖协议都有的东西(房东、租户、起租日、选择权截止日、买价),+单值的标 functional——后一份补充协议才关得上前一份写的值。里面没有一个字是这份合同特有的。++**每轮一个新库,单轮波动约 3 题**,一个版本至少跑两轮再下结论(2026-09-13 的记录见 README)。++环境变量:+ BENCH_BASE 服务地址,缺省 http://127.0.0.1:1516+ BENCH_TOKEN 登录令牌;没有就用下面两个登录+ BENCH_EMAIL / BENCH_PASSWORD 同 recall.mjs,缺省 [email protected] / benchbench123+"""++import json+import os+import sys+import time+import urllib.error+import urllib.request+from collections import defaultdict+from pathlib import Path++HERE = Path(__file__).resolve().parent+BASE = os.environ.get("BENCH_BASE", "http://127.0.0.1:1516")+# 本机的 HTTP(S)_PROXY 指向本地代理,连 localhost 不该经过它+OPENER = urllib.request.build_opener(urllib.request.ProxyHandler({}))+TOKEN = None+++def api(method, path, body=None):+ data = json.dumps(body).encode() if body is not None else None+ req = urllib.request.Request(BASE + "/api/v1" + path, data=data, method=method)+ if TOKEN:+ req.add_header("Authorization", f"Bearer {TOKEN}")+ if data is not None:+ req.add_header("Content-Type", "application/json")+ try:+ with OPENER.open(req, timeout=120) as res:+ raw = res.read()+ return json.loads(raw) if raw else None+ except urllib.error.HTTPError as e:+ raise SystemExit(f"{method} {path} -> {e.code}: {e.read()[:400]!r}")+++def login():+ global TOKEN+ TOKEN = os.environ.get("BENCH_TOKEN")+ if TOKEN:+ return+ email = os.environ.get("BENCH_EMAIL", "[email protected]")+ password = os.environ.get("BENCH_PASSWORD", "benchbench123")+ TOKEN = api("POST", "/auth/login", {"email": email, "password": password})["token"]+++# ---------------------------------------------------------------- setup++CLASSES = [+ ("organization", "Organization", "A company, LLC, fund or public body that can be a party to an agreement."),+ ("lease", "Lease", "A lease agreement together with all its amendments; one lease is one entity however many amendments change it."),+ ("amendment", "Amendment", "A document that changes an existing agreement. The terms it changes belong to the agreement it amends, not to the amendment."),+ ("property", "Property", "A parcel of real property or a building."),+ ("purchase_agreement", "Purchase agreement", "An agreement to buy and sell real property, together with its amendments."),+]++# (key, label, kind, domain, range 或 datatype, 单位, functional, 描述)+AXIOMS = [+ ("landlord", "landlord", "relation", "lease", "organization", None, True,+ "The party leasing the premises out. A lease has one landlord at a time; a new owner of the property becomes the new landlord."),+ ("tenant", "tenant", "relation", "lease", "organization", None, True,+ "The party leasing the premises."),+ ("leased_property", "leased property", "relation", "lease", "property", None, False,+ "Real property covered by the lease."),+ ("seller", "seller", "relation", "purchase_agreement", "organization", None, True,+ "The party selling the property under a purchase agreement."),+ ("purchaser", "purchaser", "relation", "purchase_agreement", "organization", None, True,+ "The party buying the property under a purchase agreement."),+ ("property_sold", "property sold", "relation", "purchase_agreement", "property", None, False,+ "The property a purchase agreement sells."),+ ("affiliate_of", "affiliate of", "relation", "organization", "organization", None, False,+ "An organization controlled by, or under common control with, another."),+ ("amends", "amends", "relation", "amendment", "lease|purchase_agreement", None, False,+ "The agreement an amendment changes."),+ ("effective_date", "effective date", "attribute", "amendment", "date", None, True,+ "The date an amendment takes effect."),+ ("premises_acres", "premises area", "attribute", "lease", "number", "acres", True,+ "Land area of the leased premises, in acres."),+ ("lease_term_years", "initial term", "attribute", "lease", "number", "years", True,+ "Length of the initial term, in years."),+ ("commencement_date", "commencement date", "attribute", "lease", "date", None, True,+ "The date the lease term begins. When the date is defined by a condition, record the fixed date the text gives."),+ ("expansion_option_deadline", "expansion option exercise deadline", "attribute", "lease", "date", None, True,+ "The last day the tenant may exercise an option to expand the premises, such as a Phase 2 exercise deadline."),+ ("expansion_lease_deadline", "expansion lease signing deadline", "attribute", "lease", "date", None, True,+ "The last day for the parties to sign the lease for the expansion phase."),+ ("expansion_budget_deadline", "expansion budget delivery deadline", "attribute", "lease", "date", None, True,+ "The last day for the landlord to deliver the preliminary budget for the expansion phase."),+ ("purchase_price", "purchase price", "attribute", "purchase_agreement", "number", "USD", True,+ "The price payable for the property, in US dollars."),+ ("inspection_period_end", "inspection period end", "attribute", "purchase_agreement", "date", None, True,+ "The date the purchaser's inspection period ends."),+ ("outside_closing_date", "outside closing date", "attribute", "purchase_agreement", "date", None, True,+ "The latest date by which the sale must close."),+]++# (edgar/blackbaud-hq-lease 里的文件, 开头读出的文件日期, 上传名)。第二份买卖协议修订只写了+# 「July 2020」没写哪天,不给日期+DOCS = [+ ("00-base-2016-08-04.html", "2016-05-16", "hq-lease-2016-05-16-lease-agreement.html"),+ ("01-amendment-2016-11-04.html", "2016-08-22", "hq-lease-2016-08-22-first-amendment.html"),+ ("02-amendment-2018-02-20.html", "2017-05-18", "hq-lease-2017-05-18-second-amendment.html"),+ ("03-amendment-2018-02-20.html", "2017-12-11", "hq-lease-2017-12-11-third-amendment.html"),+ ("04-amendment-2018-05-04.html", "2018-02-28", "hq-lease-2018-02-28-fourth-amendment.html"),+ ("05-amendment-2020-08-04.html", "2020-02-18", "hq-lease-2020-02-18-fifth-amendment.html"),+ ("06-amendment-2020-08-04.html", "2020-03-17", "hq-lease-2020-03-17-sixth-amendment.html"),+ ("07-amendment-2020-08-04.html", "2020-04-14", "hq-lease-2020-04-14-seventh-amendment.html"),+ ("08-amendment-2020-08-04.html", "2020-05-26", "hq-lease-2020-05-26-eighth-amendment.html"),+ ("09-amendment-2020-08-04.html", "2020-06-08", "hq-lease-2020-06-08-ninth-amendment.html"),+ ("11-amendment-2020-08-04.html", "2020-06-26", "hq-lease-2020-06-26-tenth-amendment.html"),+ ("12-sale-amendment-2020-11-03.html", "2020-07-08", "hq-sale-2020-07-08-psa-first-amendment.html"),+ ("13-sale-amendment-2020-11-03.html", None, "hq-sale-2020-07-psa-second-amendment.html"),+ ("14-amendment-2020-11-03.html", "2020-08-13", "hq-lease-2020-08-13-eleventh-amendment.html"),+]+++def setup(label, with_base):+ folder = HERE / "edgar" / "blackbaud-hq-lease"+ if not folder.exists():+ raise SystemExit("先跑 fetch.py")+ ws = api("GET", "/workspaces")[0]["id"]+ kb = api("POST", f"/workspaces/{ws}/kbs", {"name": f"Blackbaud HQ lease ({label})", "ontology_packs": []})["id"]+ print("kb", kb)+ time.sleep(4)+ # 与召回测量台同一个理由:抽取之后会改图的开关一律关掉,量的是抽取本身+ api("PATCH", f"/kbs/{kb}", {"auto_extend_ontology": False, "materialize_inferences": False, "governance": False})+ ids = {}+ for key, name, desc in CLASSES:+ ids[key] = api("POST", f"/kbs/{kb}/ontology/entity-types", {"key": key, "label": name, "description": desc})["id"]+ for key, name, kind, domain, rng, unit, functional, desc in AXIOMS:+ body = {"key": key, "label": name, "kind": kind, "temporal": "state", "functional": functional,+ "description": desc, "domains": [ids[domain]]}+ if kind == "attribute":+ body["datatype"] = rng+ if unit:+ body["unit"] = unit+ else:+ body["ranges"] = [ids[r] for r in rng.split("|")]+ api("POST", f"/kbs/{kb}/ontology/relation-types", body)+ print(f"声明 {len(CLASSES)} 个类、{len(AXIOMS)} 条关系与属性")+ for src, date, name in DOCS:+ if src.startswith("00-") and not with_base:+ continue+ body = {"filename": name, "content": (folder / src).read_text(encoding="utf-8", errors="replace")}+ if date:+ body["doc_time"] = f"{date}T00:00:00Z"+ api("POST", f"/kbs/{kb}/ingest", body)+ print("导入", name)+ (HERE / f"lease-kb-{label}.json").write_text(json.dumps({"kb": kb, "base": BASE}, indent=1), encoding="utf-8")+++# ---------------------------------------------------------------- score++# (题目, 类, 谓词, 时刻, 真值)+QUESTIONS = [+ ("expansion option deadline, 2020-03-01", "lease", "expansion_option_deadline", "2020-03-01", "2020-03-17"),+ ("expansion option deadline, 2020-04-01", "lease", "expansion_option_deadline", "2020-04-01", "2020-04-14"),+ ("expansion option deadline, 2020-05-01", "lease", "expansion_option_deadline", "2020-05-01", "2020-05-26"),+ ("expansion option deadline, 2020-06-01", "lease", "expansion_option_deadline", "2020-06-01", "2020-06-09"),+ ("expansion option deadline, 2020-06-15", "lease", "expansion_option_deadline", "2020-06-15", "2020-06-23"),+ ("expansion budget deadline, 2020-03-01", "lease", "expansion_budget_deadline", "2020-03-01", "2020-05-07"),+ ("expansion budget deadline, 2020-04-01", "lease", "expansion_budget_deadline", "2020-04-01", "2020-06-04"),+ ("expansion budget deadline, 2020-05-01", "lease", "expansion_budget_deadline", "2020-05-01", "2020-07-16"),+ ("expansion budget deadline, 2020-06-01", "lease", "expansion_budget_deadline", "2020-06-01", "2020-07-30"),+ ("expansion budget deadline, 2020-06-15", "lease", "expansion_budget_deadline", "2020-06-15", "2020-08-13"),+ ("landlord, 2019-01-01", "lease", "landlord", "2019-01-01", "HPBB1"),+ ("landlord, 2020-07-01", "lease", "landlord", "2020-07-01", "HPBB1"),+ ("landlord, 2020-09-01", "lease", "landlord", "2020-09-01", "BBHQ1"),+ ("tenant, 2020-09-01", "lease", "tenant", "2020-09-01", "BLACKBAUD"),+ ("premises acres, 2019-01-01", "lease", "premises_acres", "2019-01-01", "12.98"),+ ("purchase price, 2020-08-01", "purchase_agreement", "purchase_price", "2020-08-01", "76272484.66"),+ ("purchaser, 2020-08-01", "purchase_agreement", "purchaser", "2020-08-01", "BBHQ1"),+ ("outside closing date, 2020-08-01", "purchase_agreement", "outside_closing_date", "2020-08-01", "2020-12-14"),+]+++def norm(v):+ s = str(v).strip().upper()+ s = s.split(",")[0].strip() if not s.replace(",", "").replace(".", "").isdigit() else s.replace(",", "")+ return s[:10] if len(s) >= 10 and s[4:5] == "-" else s+++def holds(f, t):+ start, end = f.get("holds_from"), f.get("holds_to")+ return (start is None or start[:10] <= t) and (end is None or end[:10] > t)+++def score(kb):+ by_class = defaultdict(list)+ type_ids = {t["key"]: t["id"] for t in api("GET", f"/kbs/{kb}/ontology")["entity_types"]}+ for cls in {q[1] for q in QUESTIONS}:+ # 这个接口分页,默认一页 12 个:不翻页就只看到前 12 个实体+ page_no = 0+ while True:+ page = api("GET", f"/kbs/{kb}/ontology/entity-types/{type_ids[cls]}/entities?page={page_no}&per=100")+ by_class[cls].extend(page["entities"])+ if not page["entities"] or len(by_class[cls]) >= page.get("total", 0):+ break+ page_no += 1+ facts = {}+ for ents in by_class.values():+ for e in ents:+ facts[e["id"]] = [f for f in api("GET", f"/kbs/{kb}/entities/{e['id']}")["facts"] if f.get("direction") == "out"]++ carriers = [e["name"] for e in by_class["lease"] if any(f["predicate_key"] in ("landlord", "tenant") for f in facts[e["id"]])]+ right = partial = 0+ for label, cls, pred, t, truth in QUESTIONS:+ values = set()+ for e in by_class[cls]:+ for f in facts[e["id"]]:+ if f["predicate_key"] == pred and holds(f, t):+ v = f["other_name"] if f.get("other_name") else (f.get("object_value") or {}).get("value")+ values.add(norm(v))+ want = norm(truth)+ hit = any(v.startswith(want) for v in values)+ if hit and len(values) == 1:+ right += 1+ mark = "right"+ elif hit:+ partial += 1+ mark = "partial"+ else:+ mark = "wrong"+ print(f"{mark:8} {label:40} want {truth:14} got {sorted(values)}")+ print(f"\nright {right}/{len(QUESTIONS)}, partial {partial}")+ print(f"lease entities carrying a landlord or tenant: {len(carriers)}")+ for name in carriers:+ print(" ", name[:90])+++def main():+ if len(sys.argv) < 3 or sys.argv[1] not in ("setup", "score"):+ raise SystemExit(__doc__)+ login()+ if sys.argv[1] == "setup":+ label = next((a.split("=", 1)[1] for a in sys.argv if a.startswith("--label=")), None)+ if not label and "--label" in sys.argv:+ label = sys.argv[sys.argv.index("--label") + 1]+ setup(label or "run", "--with-base" in sys.argv)+ else:+ score(sys.argv[2])+++if __name__ == "__main__":+ main()