Skip to content

World's first open-source enterprise world model.

The agent judges identity by written rules, and a labeled set says whether it did (0025 decision 10) (#457)

WaylandYangc92a6f9affa8
diff --git a/crates/utopia-extract/src/governor.rs b/crates/utopia-extract/src/governor.rs
index beb1c95b..26c8c737 100644
--- a/crates/utopia-extract/src/governor.rs
+++ b/crates/utopia-extract/src/governor.rs
@@ -7,7 +7,7 @@
use serde_json::{json, Value};
-use crate::AdjudicationPair;
+use crate::{AdjudicationPair, IDENTITY_RULES};
/// 一对最多查几次再表态。够看两侧的事实与原文各一次、翻一次台账,还剩一次
pub const MAX_STEPS: usize = 6;
@@ -101,13 +101,7 @@ pub fn messages(pair: &AdjudicationPair, earlier: &EarlierLook) -> Vec<Value> {
records that an earlier, quicker look could not settle. Decide whether the two records \
refer to the SAME real-world entity or are namesakes.\n\
\n\
- Judge by the facts: employer or affiliation, role, time ranges, connected entities. \
- Identical names alone are NEVER sufficient evidence of sameness. Contradictory \
- affiliations in overlapping time periods mean different entities; non-overlapping \
- periods can be one person who changed jobs. One name containing the other is usually \
- the longer one abbreviated (documents drop the qualifier after first mention), unless \
- the longer one adds a noun or a phrase at the end: a project, a programme, a team or a \
- component is its own record.\n\
+ {IDENTITY_RULES}\n\
\n\
You may look things up before answering, at most {MAX_STEPS} lookups: the facts of a \
side, the source passages that mention a side, what people in this base decided about \
diff --git a/crates/utopia-extract/src/lib.rs b/crates/utopia-extract/src/lib.rs
index 86d18c0d..0e7358bc 100644
--- a/crates/utopia-extract/src/lib.rs
+++ b/crates/utopia-extract/src/lib.rs
@@ -531,46 +531,98 @@ struct AdjudicationReply {
}
/// 构造攒批裁决提示词。保守偏置:证据不足答 unsure(宁分勿合,合并要证据)。
+/// 身份规则(0025 第一轮迭代,2026-09-07):攒批与工具循环两个提示词共用。
+///
+/// 从 ai-timeline 真值上量出来的四种错:版本并进系列、「X 的 Y」并进 X、列表并进成员、
+/// 子公司并进母公司;以及同名被「类型标签不同」「合作伙伴不同」拆开。每一条都写成
+/// 一句规则加例子,例子取自那份语料。
+pub const IDENTITY_RULES: &str = "\
+A record is one specific thing in the world: a person, an organization, a legal entity, a \
+product, one version of a product, a document, an event, a place, or a concept. Two records \
+are the SAME only when they denote exactly the same thing.\n\
+\n\
+Names:\n\
+- The same proper name, compatible kinds, and no real contradiction: the same thing. Type \
+ labels were assigned by an extractor and are noisy: Organization, Corporation, \
+ ResearchOrganization, NGO and Store can all be one company; SoftwareApplication, \
+ CreativeWork, Intangible, Product, Service, Offer and ComputerLanguage can all be one \
+ product; Place and State can be one state. Only incompatible kinds count as a difference: \
+ a person against a company, a place against a product, an event against an organization. \
+ For identical proper names answer \"different\" only when you can name the contradiction \
+ in one sentence; a type label, a missing fact, fewer facts, or different partners, \
+ products, roles or events is not one. Never treat the absence of facts as a difference.\n\
+- A surname or a first name alone against a full name that contains it, in the same \
+ documents, is the same person unless another person with that name appears: \"Pachocki\" \
+ is \"Jakub Pachocki\", \"Kwon\" is \"Jason Kwon\", \"Nadella\" is \"Satya Nadella\". A person \
+ who moved between two organizations is still one person.\n\
+- A parenthetical acronym, an expanded acronym or a fuller product designation is the same \
+ thing: \"reinforcement learning (RL)\" is \"reinforcement learning\", \"US Federal Trade \
+ Commission (FTC)\" is \"Federal Trade Commission\", \"MI450\" is the \"AMD Instinct MI450\".\n\
+- A name that is the other name with a qualifier removed from the FRONT is usually the same \
+ thing abbreviated: \"Google DeepMind\" and \"DeepMind\", \"Adam D'Angelo\" and \"D'Angelo\", \
+ \"Meta Platforms\" and \"Meta\", \"Altimeter Capital\" and \"Altimeter\", \"The New York \
+ Times\" and \"New York Times\", \"Apple Inc.\" and \"Apple\". Documents drop the qualifier \
+ after first mention.\n\
+- A name that is the other name with something ADDED AT THE END is a different, more \
+ specific thing: a version or edition (\"Claude 4 Opus\" is not \"Claude\", \"AlphaFold2\" is \
+ not \"AlphaFold\", \"Genie 2\" is not \"Genie\", \"GPT-4.5\" is not \"GPT-4\", \"2024 \
+ International Mathematical Olympiad\" is not \"International Mathematical Olympiad\"), a \
+ variant or tier (\"Gemini Robotics-ER\" is not \"Gemini Robotics\", \"Claude 3 Haiku\" is \
+ not \"Haiku\"), a division, subsidiary or legal entity (\"DeepMind Health\" is not \
+ \"DeepMind\", \"OpenAI Ireland Ltd\" is not \"OpenAI\", \"Microsoft AI\" is not \
+ \"Microsoft\"), a project, programme, team, app or component. Never merge a version into \
+ its family or a part into its whole.\n\
+- A phrase that merely contains a name is not that name: \"Sam Altman's efforts\", \
+ \"psychological abuse from Sam Altman\", \"share sale led by Thrive Capital\", \"leaked \
+ letter from the National Data Guardian\", \"ChatGPT played a role in the campaign\", \"a \
+ consistent pattern of lying\". These describe something about the thing; they are not the \
+ thing.\n\
+- A list of names (\"MuZero, AlphaStar, AlphaGeometry\") is not any of its members.\n\
+- A common noun or generic phrase (\"employees\", \"users\", \"lawsuit\", \"investors\", \
+ \"event\", \"safety\") is not a proper name. Two such records are the same only when their \
+ facts show they are one specific instance; usually they are different.\n\
+- When the facts of either record describe ownership, control, a subsidiary, a holding or \
+ a parent relation between the two names, or show one of them as one legal entity among \
+ several in a group (\"OpenAI, Inc. controls the for-profit company\", \"OpenAI GP LLC \
+ controls OpenAI LP\"), they are two entities even if one name is the other plus a \
+ corporate suffix. A group and its legal entities are different records.\n\
+\n\
+Facts:\n\
+- Different facts are not contradictory facts. One company has many partnerships, investors, \
+ lawsuits and contracts; a person changes jobs; one product is praised in one document and \
+ criticised in another. A contradiction is two facts that cannot both hold of one thing at \
+ once: two different founders, two headquarters at the same time, two different birth dates, \
+ or affiliations that overlap in time and exclude each other. Only a contradiction is \
+ evidence of difference.\n\
+- A record with no facts contributes nothing; the name and the other record decide.";
+
pub fn build_adjudication_messages(pairs: &[AdjudicationPair]) -> Vec<ChatMessage> {
- let system = "You are an entity-resolution adjudicator for a knowledge graph. \
- For each numbered pair, decide whether the two records refer to the SAME real-world \
- entity or are namesakes (different entities that share a name).\n\
- \n\
- Judge by the facts attached to each record: employer/affiliation, role, time ranges, \
- and connected entities. Identical names alone are NEVER sufficient evidence of sameness. \
- Contradictory affiliations in overlapping time periods indicate different entities \
- (but people do change jobs — non-overlapping periods can belong to one person).\n\
- \n\
- One name containing the other is a different case, and the rule above does not apply \
- to it: \"星云科技上海研究院\" against \"上海研究院\", \"Nebula Technologies Inc.\" \
- against \"Nebula\". Documents drop the qualifier after first mention, so the shorter \
- form is usually the longer one abbreviated — treat the containment as evidence FOR \
- sameness and let the facts settle it. Shared people, parent or location confirm one \
- entity; a different parent or conflicting leadership means the shorter name belongs \
- to something else.\n\
- Abbreviation removes a qualifier from the FRONT. It never adds a noun or a \
- prepositional phrase at the end, so those are different entities however much text \
- they share: \"the operator library for the Canghai Platform\" is not the Canghai \
- Platform, \"Qiming X7 programme\" is not the Qiming X7, and \"沧海平台项目\" is not \
- \"沧海平台\" — a project, a programme, a team or a component is its own record.\n\
- \n\
- Some pairs carry precedents: decisions people made in this same knowledge base on \
- these names, or on pairs of the same two types. Treat them as how the owners of this \
- base want such cases judged. Follow a precedent on the same pair unless the facts of \
- this pair clearly differ from it; when precedents disagree with each other, answer \
- \"unsure\". A precedent never overrides a contradiction in the facts.\n\
- \n\
- Output exactly one JSON object and nothing else:\n\
- {\"verdicts\":[{\"i\":0,\"verdict\":\"same|different|unsure\",\"confidence\":0.9,\
- \"why\":\"one sentence\"}]}\n\
- \n\
- Rules:\n\
- 1. One verdict per pair, using the pair's number as \"i\".\n\
- 2. confidence in 0~1.\n\
- 3. \"why\" is one short sentence; when a precedent decided it, say which.\n\
- 4. Be conservative: if the evidence is insufficient to decide, answer \"unsure\" — \
- a wrong merge is far more damaging than leaving two records separate."
- .to_string();
+ let system = format!(
+ "You are an entity-resolution adjudicator for a knowledge graph. For each numbered \
+ pair, decide whether the two records refer to the SAME real-world thing or are two \
+ things that share a name.\n\
+ \n\
+ {IDENTITY_RULES}\n\
+ \n\
+ Some pairs carry precedents: decisions people made in this same knowledge base on \
+ these names, or on pairs of the same two types. Treat them as how the owners of this \
+ base want such cases judged. Follow a precedent on the same pair unless the facts of \
+ this pair clearly differ from it; when precedents disagree with each other, answer \
+ \"unsure\". A precedent never overrides a contradiction in the facts.\n\
+ \n\
+ Output exactly one JSON object and nothing else:\n\
+ {{\"verdicts\":[{{\"i\":0,\"verdict\":\"same|different|unsure\",\"confidence\":0.9,\
+ \"why\":\"one sentence\"}}]}}\n\
+ \n\
+ Rules:\n\
+ 1. One verdict per pair, using the pair's number as \"i\".\n\
+ 2. confidence in 0~1.\n\
+ 3. \"why\" is one short sentence; when a precedent decided it, say which.\n\
+ 4. A wrong merge is far more damaging than leaving two records separate: answer \
+ \"same\" only when the rules above make it so; when a rule says two things, say \
+ \"different\" with confidence, not \"unsure\"; \"unsure\" is for evidence that \
+ genuinely points both ways."
+ );
let mut user = String::new();
for (i, p) in pairs.iter().enumerate() {
diff --git a/crates/utopia-server/src/governance.rs b/crates/utopia-server/src/governance.rs
index 8c15b217..6db28070 100644
--- a/crates/utopia-server/src/governance.rs
+++ b/crates/utopia-server/src/governance.rs
@@ -206,9 +206,31 @@ async fn permit(ctx: &Ctx<'_>) -> Option<tokio::sync::OwnedSemaphorePermit> {
}
fn pair_of(item: &ReviewItem, p: &Precedents) -> utopia_extract::AdjudicationPair {
+ // 同名且大类不冲突的对:两侧写同一个类型标签。抽取器给同一家公司的两条记录
+ // Store 与 Organization,模型就拿这个当「不同」的理由——把拐杖拿掉,让它看事实
+ // 只在两侧都归得到同一个大类(人、组织、地点、事件)时才共用:Periodical 对 Service、
+ // VideoGame 对没类型,那些标签是有信息的,留着
+ let same_kind = gov::name_shape(&item.left.name, &item.right.name) == gov::NameShape::Identical
+ && matches!(
+ (
+ item.left.type_label.as_deref().and_then(gov::type_family),
+ item.right.type_label.as_deref().and_then(gov::type_family),
+ ),
+ (Some(a), Some(b)) if a == b
+ );
+ let shared = item
+ .left
+ .type_label
+ .clone()
+ .or_else(|| item.right.type_label.clone())
+ .unwrap_or_else(|| "untyped".into());
let side = |s: &utopia_core::models::ReviewSide| utopia_extract::AdjudicationSide {
name: s.name.clone(),
- type_label: s.type_label.clone().unwrap_or_else(|| "untyped".into()),
+ type_label: if same_kind {
+ shared.clone()
+ } else {
+ s.type_label.clone().unwrap_or_else(|| "untyped".into())
+ },
facts: s.top_facts.clone(),
};
utopia_extract::AdjudicationPair {
@@ -229,14 +251,22 @@ async fn settle(
) -> anyhow::Result<()> {
let pool = &ctx.state.pool;
let kb_id = ctx.kb_id;
- let types_conflict = matches!(
- (&item.left.type_label, &item.right.type_label),
- (Some(a), Some(b)) if a != b
+ let types_conflict = gov::types_conflict(
+ item.left.type_label.as_deref(),
+ item.right.type_label.as_deref(),
);
+ let shape = gov::name_shape(&item.left.name, &item.right.name);
// 第二层:只接判不定的,硬规则拦下的不进。预算用完了照第一刀写建议
- if gov::gate(look.same, look.conf, types_conflict, p) == Gate::Propose
+ // 同名、大类不冲突、模型却说不同:这是它最爱错的一种,先别采纳,让第二层带着
+ // 全部事实与原文再看一遍
+ let doubted_split = shape == gov::NameShape::Identical
+ && !types_conflict
+ && look.same == Some(false)
+ && look.calls == 0;
+ if (gov::gate(look.same, look.conf, types_conflict, shape, p) == Gate::Propose
&& look.uncertain()
+ || doubted_split)
&& p.reverts.is_empty()
{
let spent = gov::loop_calls_today(pool, kb_id).await?;
@@ -270,7 +300,7 @@ async fn settle(
calls: look.calls,
};
- match gov::gate(look.same, look.conf, types_conflict, p) {
+ match gov::gate(look.same, look.conf, types_conflict, shape, p) {
Gate::Apply if look.same == Some(true) => {
let reason = format!("governed|{conf:.2}");
// 同簇连锁:前一对合完,这一对的一侧可能已经并进了别人——合活着的那个
diff --git a/crates/utopia-store/src/governance.rs b/crates/utopia-store/src/governance.rs
index 5500852c..a1877093 100644
--- a/crates/utopia-store/src/governance.rs
+++ b/crates/utopia-store/src/governance.rs
@@ -243,6 +243,235 @@ pub fn precedents_json(p: &Precedents) -> serde_json::Value {
serde_json::Value::Array(all)
}
+/// 名字的形状:不靠模型也能看出来的那几种
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum NameShape {
+ /// 一字不差(忽略大小写与开头的 the)
+ Identical,
+ /// 一个是另一个去掉了前面的限定词:Google DeepMind / DeepMind
+ Abbreviation,
+ /// 一个是另一个加了公司后缀:Apple Inc. / Apple、Thrive Capital / Thrive
+ Suffix,
+ /// 一个是另一个加了版本、届次:Claude 4 / Claude、AlphaFold2 / AlphaFold、2024 IMO / IMO
+ Version,
+ /// 一个是另一个加了一两个词:DeepMind Health / DeepMind、Gemini Robotics-ER / Gemini Robotics
+ Extension,
+ /// 一个是含着另一个的一句话:Sam Altman's efforts / Sam Altman;或列表 / 成员
+ Phrase,
+ /// 互不包含
+ Unrelated,
+}
+
+fn norm_name(s: &str) -> String {
+ let s = s.trim().to_lowercase();
+ let s = s.strip_prefix("the ").unwrap_or(&s).to_string();
+ // 末尾括号里的缩写不是版本:reinforcement learning (RL)、Department of Defense (DoD)
+ let s = match (s.rfind(" ("), s.ends_with(')')) {
+ (Some(i), true)
+ if s[i + 2..s.len() - 1].chars().all(|c| c.is_alphanumeric()) && s.len() - i <= 12 =>
+ {
+ s[..i].to_string()
+ }
+ _ => s,
+ };
+ s.split_whitespace().collect::<Vec<_>>().join(" ")
+}
+
+const CORPORATE_SUFFIX: &[&str] = &[
+ "inc",
+ "ltd",
+ "llc",
+ "plc",
+ "corp",
+ "corporation",
+ "co",
+ "company",
+ "capital",
+ "group",
+ "platforms",
+ "industries",
+ "technologies",
+ "holdings",
+ "limited",
+ "gmbh",
+ "sa",
+ "ag",
+];
+
+pub fn name_shape(a: &str, b: &str) -> NameShape {
+ let (a, b) = (norm_name(a), norm_name(b));
+ if a == b {
+ return NameShape::Identical;
+ }
+ let (short, long) = if a.len() <= b.len() {
+ (&a, &b)
+ } else {
+ (&b, &a)
+ };
+ // 要整词出现:Time 在 Financial Times 里不算(后面跟着字母),AlphaFold 在 AlphaFold2
+ // 里算(后面跟着数字,那是版本)
+ let word_at = |long: &str, short: &str| -> Option<usize> {
+ let mut from = 0;
+ while let Some(i) = long[from..].find(short) {
+ let i = from + i;
+ let before_ok = i == 0
+ || !long[..i]
+ .chars()
+ .last()
+ .is_some_and(|c| c.is_alphanumeric());
+ let after_ok = long[i + short.len()..]
+ .chars()
+ .next()
+ .is_none_or(|c| !c.is_alphabetic());
+ if before_ok && after_ok {
+ return Some(i);
+ }
+ from = i + 1;
+ }
+ None
+ };
+ let Some(pos) = word_at(long, short) else {
+ return NameShape::Unrelated;
+ };
+ if short.is_empty() {
+ return NameShape::Unrelated;
+ }
+ let words = |s: &str| {
+ s.split(|c: char| !c.is_alphanumeric() && c != '\'')
+ .filter(|w| !w.is_empty())
+ .count()
+ };
+ let phrase_marker = |s: &str| {
+ s.contains("'s ")
+ || s.ends_with("'s")
+ || [
+ " of ",
+ " by ",
+ " from ",
+ " for ",
+ " against ",
+ " with ",
+ " to ",
+ " in ",
+ " on ",
+ " at ",
+ " and ",
+ " or ",
+ " led ",
+ " into ",
+ ]
+ .iter()
+ .any(|m| format!(" {s} ").contains(m))
+ };
+ if long.matches(", ").count() >= 2 {
+ return NameShape::Phrase;
+ }
+ if pos + short.len() == long.len() {
+ let front = long[..pos].trim().trim_end_matches([',', '-', ':']);
+ if front.chars().next().is_some_and(|c| c.is_ascii_digit()) {
+ return NameShape::Version;
+ }
+ if phrase_marker(front) || words(front) > 2 {
+ return NameShape::Phrase;
+ }
+ return NameShape::Abbreviation;
+ }
+ if pos == 0 {
+ let tail = long[short.len()..]
+ .trim()
+ .trim_start_matches([',', '-', ':', '.', 'v']);
+ let tail = tail.trim();
+ if tail.chars().next().is_some_and(|c| c.is_ascii_digit()) {
+ return NameShape::Version;
+ }
+ let tail_words: Vec<&str> = tail
+ .split(|c: char| !c.is_alphanumeric())
+ .filter(|w| !w.is_empty())
+ .collect();
+ if !tail_words.is_empty() && tail_words.iter().all(|w| CORPORATE_SUFFIX.contains(w)) {
+ return NameShape::Suffix;
+ }
+ // 尾巴两个词起就是一句话了:DeepMind Health data sharing、OpenAI Ireland Ltd、
+ // Meta Superintelligence Labs——都不是前面那个东西
+ if phrase_marker(tail) || tail_words.len() > 1 {
+ return NameShape::Phrase;
+ }
+ return NameShape::Extension;
+ }
+ NameShape::Phrase
+}
+
+/// 类型标签的大类:抽取器给的标签本身很吵(同一家公司在两篇里是 Organization 与
+/// Corporation),只有大类不同才算冲突。None = 说不上是哪一类,与谁都不冲突
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum TypeFamily {
+ Person,
+ Organization,
+ Place,
+ Event,
+}
+
+pub fn type_family(label: &str) -> Option<TypeFamily> {
+ let l = label.to_lowercase();
+ if l == "person" || l == "researcher" || l.ends_with("person") {
+ return Some(TypeFamily::Person);
+ }
+ if l.contains("event") {
+ return Some(TypeFamily::Event);
+ }
+ if [
+ "organization",
+ "corporation",
+ "business",
+ "store",
+ "hospital",
+ "university",
+ "college",
+ "school",
+ "consortium",
+ "ngo",
+ "party",
+ "agency",
+ "company",
+ "bank",
+ "fund",
+ "newspaper",
+ "governmentoffice",
+ "team",
+ ]
+ .iter()
+ .any(|k| l.contains(k))
+ {
+ return Some(TypeFamily::Organization);
+ }
+ if [
+ "place",
+ "city",
+ "country",
+ "state",
+ "continent",
+ "administrativearea",
+ "region",
+ "landmark",
+ "civicstructure",
+ "locality",
+ ]
+ .iter()
+ .any(|k| l.contains(k))
+ {
+ return Some(TypeFamily::Place);
+ }
+ None
+}
+
+/// 两边的类型标签是不是不同的大类
+pub fn types_conflict(a: Option<&str>, b: Option<&str>) -> bool {
+ matches!(
+ (a.and_then(type_family), b.and_then(type_family)),
+ (Some(x), Some(y)) if x != y
+ )
+}
+
#[derive(Debug, PartialEq, Eq)]
pub enum Gate {
/// 自己动手:合并可撤,分开可再合
@@ -272,14 +501,22 @@ impl Precedents {
/// agent 按自己的把握判,历史是参考(0025 决定 4,2026-09-06 修订)。历史反对就拦住:
/// 人分开过这一对却说合、合过却说分、涉及任一名字的撤回。历史同意就把线放低。其余
/// 按 agent 自己的把握。类型冲突永远不合——那是规则,不是把握
-pub fn gate(same: Option<bool>, conf: f32, types_conflict: bool, p: &Precedents) -> Gate {
+pub fn gate(
+ same: Option<bool>,
+ conf: f32,
+ types_conflict: bool,
+ shape: NameShape,
+ p: &Precedents,
+) -> Gate {
let Some(same) = same else {
return Gate::Propose;
};
if !p.reverts.is_empty() || p.same_pair.iter().any(|x| x.merged() != same) {
return Gate::Propose;
}
- if same && types_conflict {
+ // 两条不靠模型的规则:类型的大类不同不合;版本尾巴或含着名字的一句话不自动合——
+ // 模型在这两种上最爱说 same,而它们几乎从不是
+ if same && (types_conflict || matches!(shape, NameShape::Version | NameShape::Phrase)) {
return Gate::Propose;
}
let bar = if p.supports(same) {
@@ -754,68 +991,134 @@ mod tests {
#[test]
fn unsure_and_low_confidence_are_proposed() {
let none = Precedents::default();
- assert_eq!(gate(None, 0.99, false, &none), Gate::Propose);
- assert_eq!(gate(Some(false), 0.6, false, &none), Gate::Propose);
- assert_eq!(gate(Some(true), 0.84, false, &none), Gate::Propose);
+ assert_eq!(
+ gate(None, 0.99, false, NameShape::Unrelated, &none),
+ Gate::Propose
+ );
+ assert_eq!(
+ gate(Some(false), 0.6, false, NameShape::Unrelated, &none),
+ Gate::Propose
+ );
+ assert_eq!(
+ gate(Some(true), 0.84, false, NameShape::Unrelated, &none),
+ Gate::Propose
+ );
}
#[test]
fn keeping_apart_needs_confidence_only() {
let none = Precedents::default();
- assert_eq!(gate(Some(false), 0.9, false, &none), Gate::Apply);
+ assert_eq!(
+ gate(Some(false), 0.9, false, NameShape::Unrelated, &none),
+ Gate::Apply
+ );
// 类型冲突的对也能自己分开:分开不改图
- assert_eq!(gate(Some(false), 0.9, true, &none), Gate::Apply);
+ assert_eq!(
+ gate(Some(false), 0.9, true, NameShape::Unrelated, &none),
+ Gate::Apply
+ );
}
#[test]
fn the_agent_judges_and_history_moves_the_bar() {
// 没有任何历史:agent 自己的把握够就合,不够就问
let none = Precedents::default();
- assert_eq!(gate(Some(true), 0.9, false, &none), Gate::Apply);
- assert_eq!(gate(Some(true), 0.8, false, &none), Gate::Propose);
+ assert_eq!(
+ gate(Some(true), 0.9, false, NameShape::Unrelated, &none),
+ Gate::Apply
+ );
+ assert_eq!(
+ gate(Some(true), 0.8, false, NameShape::Unrelated, &none),
+ Gate::Propose
+ );
// 历史同意:线降到 SUPPORTED_CONF
let pair = Precedents {
same_pair: vec![p("review.merge")],
..Default::default()
};
- assert_eq!(gate(Some(true), 0.8, false, &pair), Gate::Apply);
- assert_eq!(gate(Some(true), 0.7, false, &pair), Gate::Propose);
+ assert_eq!(
+ gate(Some(true), 0.8, false, NameShape::Unrelated, &pair),
+ Gate::Apply
+ );
+ assert_eq!(
+ gate(Some(true), 0.7, false, NameShape::Unrelated, &pair),
+ Gate::Propose
+ );
let name = Precedents {
same_name: vec![p("merge.manual"), p("review.merge")],
..Default::default()
};
- assert_eq!(gate(Some(true), 0.8, false, &name), Gate::Apply);
+ assert_eq!(
+ gate(Some(true), 0.8, false, NameShape::Unrelated, &name),
+ Gate::Apply
+ );
// 同名的决定有合有分:不算同意,回到 agent 自己的线
let mixed = Precedents {
same_name: vec![p("review.merge"), p("review.keep")],
..Default::default()
};
- assert_eq!(gate(Some(true), 0.8, false, &mixed), Gate::Propose);
- assert_eq!(gate(Some(true), 0.9, false, &mixed), Gate::Apply);
+ assert_eq!(
+ gate(Some(true), 0.8, false, NameShape::Unrelated, &mixed),
+ Gate::Propose
+ );
+ assert_eq!(
+ gate(Some(true), 0.9, false, NameShape::Unrelated, &mixed),
+ Gate::Apply
+ );
// 类型对的习惯
assert_eq!(
- gate(Some(true), 0.8, false, &with_stats(6, 2, 0)),
+ gate(
+ Some(true),
+ 0.8,
+ false,
+ NameShape::Unrelated,
+ &with_stats(6, 2, 0)
+ ),
Gate::Apply
);
assert_eq!(
- gate(Some(true), 0.8, false, &with_stats(3, 1, 0)),
+ gate(
+ Some(true),
+ 0.8,
+ false,
+ NameShape::Unrelated,
+ &with_stats(3, 1, 0)
+ ),
Gate::Propose,
"习惯要够 {TYPE_PAIR_MIN} 笔才算"
);
assert_eq!(
- gate(Some(true), 0.8, false, &with_stats(6, 2, 1)),
+ gate(
+ Some(true),
+ 0.8,
+ false,
+ NameShape::Unrelated,
+ &with_stats(6, 2, 1)
+ ),
Gate::Propose,
"撤回过的类型对不算合并的习惯"
);
// 分开那一边同样:人一直分开的名字,分开的线也低
- assert_eq!(gate(Some(false), 0.8, false, &none), Gate::Propose);
+ assert_eq!(
+ gate(Some(false), 0.8, false, NameShape::Unrelated, &none),
+ Gate::Propose
+ );
let kept = Precedents {
same_pair: vec![p("review.keep")],
..Default::default()
};
- assert_eq!(gate(Some(false), 0.8, false, &kept), Gate::Apply);
assert_eq!(
- gate(Some(false), 0.8, false, &with_stats(1, 8, 0)),
+ gate(Some(false), 0.8, false, NameShape::Unrelated, &kept),
+ Gate::Apply
+ );
+ assert_eq!(
+ gate(
+ Some(false),
+ 0.8,
+ false,
+ NameShape::Unrelated,
+ &with_stats(1, 8, 0)
+ ),
Gate::Apply
);
}
@@ -861,6 +1164,118 @@ mod tests {
);
}
+ #[test]
+ fn a_name_has_a_shape() {
+ use NameShape::*;
+ assert_eq!(name_shape("Google", "google"), Identical);
+ assert_eq!(
+ name_shape("The New York Times", "New York Times"),
+ Identical
+ );
+ assert_eq!(name_shape("Google DeepMind", "DeepMind"), Abbreviation);
+ assert_eq!(name_shape("Adam D'Angelo", "D'Angelo"), Abbreviation);
+ assert_eq!(name_shape("Apple Inc.", "Apple"), Suffix);
+ assert_eq!(name_shape("Thrive Capital", "Thrive"), Suffix);
+ assert_eq!(name_shape("Meta Platforms", "Meta"), Suffix);
+ assert_eq!(name_shape("Claude 4 Opus", "Claude"), Version);
+ assert_eq!(name_shape("AlphaFold2", "AlphaFold"), Version);
+ assert_eq!(name_shape("GPT-4.5", "GPT-4"), Version);
+ assert_eq!(name_shape("Lyria 3", "Lyria"), Version);
+ assert_eq!(
+ name_shape(
+ "2024 International Mathematical Olympiad",
+ "International Mathematical Olympiad"
+ ),
+ Version
+ );
+ assert_eq!(name_shape("DeepMind Health", "DeepMind"), Extension);
+ assert_eq!(
+ name_shape("Gemini Robotics-ER", "Gemini Robotics"),
+ Extension
+ );
+ assert_eq!(name_shape("OpenAI Ireland Ltd", "OpenAI"), Phrase);
+ assert_eq!(
+ name_shape("DeepMind Health data sharing", "DeepMind Health"),
+ Phrase
+ );
+ assert_eq!(
+ name_shape("reinforcement learning (RL)", "reinforcement learning"),
+ Identical
+ );
+ assert_eq!(
+ name_shape(
+ "US Securities and Exchange Commission (SEC)",
+ "Securities and Exchange Commission"
+ ),
+ Abbreviation
+ );
+ assert_eq!(
+ name_shape(
+ "United States Department of Defense (DoD)",
+ "Department of Defense"
+ ),
+ Abbreviation
+ );
+ assert_eq!(
+ name_shape("Amazon Web Services (AWS)", "Amazon Web Services"),
+ Identical
+ );
+ assert_eq!(name_shape("Sam Altman's efforts", "Sam Altman"), Phrase);
+ assert_eq!(
+ name_shape("share sale led by Thrive Capital", "Thrive Capital"),
+ Phrase
+ );
+ assert_eq!(
+ name_shape("psychological abuse from Sam Altman", "Sam Altman"),
+ Phrase
+ );
+ assert_eq!(
+ name_shape("MuZero, AlphaStar, AlphaGeometry", "AlphaStar"),
+ Phrase
+ );
+ assert_eq!(
+ name_shape("MuZero, AlphaStar, AlphaGeometry", "MuZero"),
+ Phrase
+ );
+ assert_eq!(name_shape("Time", "Financial Times"), Unrelated);
+ assert_eq!(name_shape("Altimeter", "Altimeter Capital"), Suffix);
+ }
+
+ #[test]
+ fn types_conflict_only_across_families() {
+ assert!(!types_conflict(Some("Organization"), Some("Corporation")));
+ assert!(!types_conflict(
+ Some("SoftwareApplication"),
+ Some("CreativeWork")
+ ));
+ assert!(!types_conflict(Some("Organization"), None));
+ assert!(types_conflict(Some("Person"), Some("Organization")));
+ assert!(!types_conflict(Some("City"), Some("Product")));
+ assert!(types_conflict(Some("City"), Some("Corporation")));
+ assert!(types_conflict(Some("ConferenceEvent"), Some("Person")));
+ }
+
+ #[test]
+ fn a_version_or_a_phrase_never_merges_on_its_own() {
+ let none = Precedents::default();
+ assert_eq!(
+ gate(Some(true), 0.99, false, NameShape::Version, &none),
+ Gate::Propose
+ );
+ assert_eq!(
+ gate(Some(true), 0.99, false, NameShape::Phrase, &none),
+ Gate::Propose
+ );
+ assert_eq!(
+ gate(Some(true), 0.99, false, NameShape::Extension, &none),
+ Gate::Apply
+ );
+ assert_eq!(
+ gate(Some(false), 0.9, false, NameShape::Version, &none),
+ Gate::Apply
+ );
+ }
+
#[test]
fn hard_rules_come_first() {
let pair = Precedents {
@@ -868,7 +1283,7 @@ mod tests {
..Default::default()
};
assert_eq!(
- gate(Some(true), 0.99, true, &pair),
+ gate(Some(true), 0.99, true, NameShape::Unrelated, &pair),
Gate::Propose,
"类型冲突不合"
);
@@ -877,14 +1292,20 @@ mod tests {
reverts: vec![p("merge.revert")],
..Default::default()
};
- assert_eq!(gate(Some(true), 0.99, false, &reverted), Gate::Propose);
- assert_eq!(gate(Some(false), 0.99, false, &reverted), Gate::Propose);
+ assert_eq!(
+ gate(Some(true), 0.99, false, NameShape::Unrelated, &reverted),
+ Gate::Propose
+ );
+ assert_eq!(
+ gate(Some(false), 0.99, false, NameShape::Unrelated, &reverted),
+ Gate::Propose
+ );
let kept = Precedents {
same_pair: vec![p("review.keep")],
..Default::default()
};
assert_eq!(
- gate(Some(true), 0.99, false, &kept),
+ gate(Some(true), 0.99, false, NameShape::Unrelated, &kept),
Gate::Propose,
"人分开过这一对,模型说合,只建议"
);
@@ -893,7 +1314,7 @@ mod tests {
..Default::default()
};
assert_eq!(
- gate(Some(false), 0.99, false, &merged),
+ gate(Some(false), 0.99, false, NameShape::Unrelated, &merged),
Gate::Propose,
"人合过这一对,模型说分,只建议"
);
diff --git a/crates/utopia-store/tests/a_governor_reads_the_ledger.rs b/crates/utopia-store/tests/a_governor_reads_the_ledger.rs
index 9704817a..956f7485 100644
--- a/crates/utopia-store/tests/a_governor_reads_the_ledger.rs
+++ b/crates/utopia-store/tests/a_governor_reads_the_ledger.rs
@@ -233,9 +233,24 @@ async fn run(pool: &PgPool, s: &Seed) -> anyhow::Result<()> {
assert!(p.reverts.is_empty());
let t = p.type_pair.clone().expect("两边都有类型");
assert_eq!((t.merged, t.kept, t.reverted), (1, 3, 0));
- assert_eq!(governance::gate(Some(false), 0.9, false, &p), Gate::Apply);
assert_eq!(
- governance::gate(Some(true), 0.99, false, &p),
+ governance::gate(
+ Some(false),
+ 0.9,
+ false,
+ governance::NameShape::Unrelated,
+ &p
+ ),
+ Gate::Apply
+ );
+ assert_eq!(
+ governance::gate(
+ Some(true),
+ 0.99,
+ false,
+ governance::NameShape::Unrelated,
+ &p
+ ),
Gate::Propose,
"人分开过,模型说合,只建议"
);
@@ -248,8 +263,14 @@ async fn run(pool: &PgPool, s: &Seed) -> anyhow::Result<()> {
1,
"Apple 对 Apple Records 是这个名字对别的名字"
);
- assert_eq!(governance::gate(Some(true), 0.9, false, &p), Gate::Apply);
- assert_eq!(governance::gate(Some(true), 0.7, false, &p), Gate::Propose);
+ assert_eq!(
+ governance::gate(Some(true), 0.9, false, governance::NameShape::Unrelated, &p),
+ Gate::Apply
+ );
+ assert_eq!(
+ governance::gate(Some(true), 0.7, false, governance::NameShape::Unrelated, &p),
+ Gate::Propose
+ );
let lines = governance::render_lines(&p);
assert!(lines[0].starts_with("this same pair was merged by a person on "));
assert!(lines
@@ -258,17 +279,38 @@ async fn run(pool: &PgPool, s: &Seed) -> anyhow::Result<()> {
let p = governance::precedents_for(pool, s.kb, &by_id(s.orion)).await?;
assert_eq!(p.reverts.len(), 1);
- assert_eq!(governance::gate(Some(true), 0.99, false, &p), Gate::Propose);
assert_eq!(
- governance::gate(Some(false), 0.99, false, &p),
+ governance::gate(
+ Some(true),
+ 0.99,
+ false,
+ governance::NameShape::Unrelated,
+ &p
+ ),
+ Gate::Propose
+ );
+ assert_eq!(
+ governance::gate(
+ Some(false),
+ 0.99,
+ false,
+ governance::NameShape::Unrelated,
+ &p
+ ),
Gate::Propose
);
let m = by_id(s.mercury);
let p = governance::precedents_for(pool, s.kb, &m).await?;
assert!(p.type_pair.is_some());
- assert_eq!(governance::gate(Some(true), 0.99, true, &p), Gate::Propose);
- assert_eq!(governance::gate(Some(false), 0.9, true, &p), Gate::Apply);
+ assert_eq!(
+ governance::gate(Some(true), 0.99, true, governance::NameShape::Unrelated, &p),
+ Gate::Propose
+ );
+ assert_eq!(
+ governance::gate(Some(false), 0.9, true, governance::NameShape::Unrelated, &p),
+ Gate::Apply
+ );
// 写了建议的对不再进队列
let run_id = Uuid::now_v7();
diff --git a/docs/decisions/0025-governance-reads-the-ledger-before-it-decides.md b/docs/decisions/0025-governance-reads-the-ledger-before-it-decides.md
index d96f1298..8ac02523 100644
--- a/docs/decisions/0025-governance-reads-the-ledger-before-it-decides.md
+++ b/docs/decisions/0025-governance-reads-the-ledger-before-it-decides.md
@@ -1,6 +1,6 @@
# 0025 · Governance reads the ledger before it decides
-- **Status**: decision 4 revised 2026-09-06 (the agent's own confidence decides, history moves the bar or blocks) · cut 1 implemented (#434) · migration 0035 adds `knowledge_bases.governance` and `agent_decisions`; `governance` in the store holds the precedent families, the first-in-first-out queue with its clusters, the gate and the table; the `govern` job in the server reads the switch, calls the model with precedents and applies or proposes; `?queue=agent`, `ReviewCounts.agent` and `POST /kbs/{id}/review/agent/{decision_id}` on the API · cut 3 (UI, #437) implemented: the switch in base settings, the Agent queue with its rows and answers, the proposal chip on a duplicate card whose Merge / Keep answers the proposal, the Agent section on the Overview · cut 2 implemented: migration 0036 adds `question`, `trace` and `calls` to `agent_decisions`; `governor` in the extract crate holds the tools, the opening and the step reader; `investigate` in the server job runs the loop for pairs the batch could not settle (decision 8) · cut 4 implemented: migration 0037 adds `knowledge_bases.governance_since`; `fuse` in the server job turns the switch off after two reverts of the agent's merges since it was turned on, within seven days, raises `governance.tripped` and writes the ledger (decision 9)
+- **Status**: decision 10 added 2026-09-07 (identity rules, name shapes, type families, the labeled set and `govern.mjs`) · decision 4 revised 2026-09-06 (the agent's own confidence decides, history moves the bar or blocks) · cut 1 implemented (#434) · migration 0035 adds `knowledge_bases.governance` and `agent_decisions`; `governance` in the store holds the precedent families, the first-in-first-out queue with its clusters, the gate and the table; the `govern` job in the server reads the switch, calls the model with precedents and applies or proposes; `?queue=agent`, `ReviewCounts.agent` and `POST /kbs/{id}/review/agent/{decision_id}` on the API · cut 3 (UI, #437) implemented: the switch in base settings, the Agent queue with its rows and answers, the proposal chip on a duplicate card whose Merge / Keep answers the proposal, the Agent section on the Overview · cut 2 implemented: migration 0036 adds `question`, `trace` and `calls` to `agent_decisions`; `governor` in the extract crate holds the tools, the opening and the step reader; `investigate` in the server job runs the loop for pairs the batch could not settle (decision 8) · cut 4 implemented: migration 0037 adds `knowledge_bases.governance_since`; `fuse` in the server job turns the switch off after two reverts of the agent's merges since it was turned on, within seven days, raises `governance.tripped` and writes the ledger (decision 9)
- **Written**: 2026-09-06 (conventions in the [README](README.md))
- **Related**: [0016](0016-close-the-open-seams-before-cutting-new-ones.md) C2 gave a base its first automation switch, `auto_type_resolution`, and this record copies its shape; #428 asked for bulk and automatic handling of same-name pairs and got the batch path (#429, #430) this builds on; [0020](0020-an-auditor-reads-it-without-us.md) made the ledger complete enough to be read back; [0015](0015-recording-a-sentence-is-not-asserting-a-fact.md) keeps a person's own sentences out of any machine's reach.
@@ -60,6 +60,26 @@ A person reverting one of the agent's merges is the strongest signal the ledger
Both revert paths count: the Agent queue's Revert and the Merges page's Revert on a merge the agent made (the latter now marks the agent's row `reverted` too, so the queue and the history agree). A person merging over an applied keep, or overriding a proposal, does not count: those are corrections to a suggestion, not to a change in the graph.
+### 10. The agent judges by written identity rules, and a labeled set says whether it did
+
+The first big-file run (decision 4's revision note) showed two things at once: a fresh base needs the agent to decide on its own, and what the model decides on its own is wrong in patterned ways. So on 2026-09-07 every pair the resolver raised on one governed import of eight interlocking articles (615 pairs, 411 distinct name pairs) was labeled by hand from each side's facts and source passages: `same`, `different`, or `unknown` where no reader could tell either (a bare "judge"). The set lives in `scripts/bench/truth/ai-timeline.duplicates.json`, keyed by the two names, and `scripts/bench/govern.mjs` resets a base's agent decisions without touching the ledger, runs the governor, and scores: decided on its own, agreed with the labels, wrong merges, wrong keeps, left for people.
+
+The patterns became rules, written once (`IDENTITY_RULES`) and read by both prompts: a version, edition, tier, division, subsidiary or legal entity added at the end of a name is a different thing; a phrase that contains a name is not the name; a list is not its members; a common noun is not a proper name; a qualifier dropped from the front, a corporate suffix, a surname for a full name, a parenthetical acronym are the same thing; a different type label, a missing fact, a different partner or role is not a contradiction. Two of the rules are also mechanical, so the model cannot talk its way past them: `name_shape` classifies a pair as identical, abbreviation, suffix, version, extension, phrase or unrelated, and a `same` verdict on a version or a phrase is never applied on its own; `type_family` folds the extractor's labels into person, organization, place and event, and only a difference across families counts as a conflict. Identical names whose labels fall in one family are shown to the model with one label, since the label difference was the reason it gave most often for a wrong split; and a `different` verdict on identical, compatible names goes to the tool loop for a second look before it is applied.
+
+The same 589 pairs, reset and re-run after each change:
+
+| run | decided on its own | agreed with the labels | wrong merges | wrong keeps | left for people | wall clock · loop calls |
+|---|---|---|---|---|---|---|
+| before this decision (gate of 2026-09-06) | 67% | 91% | 1 | 35 | 186 | ran alongside the 63 min import; 301 loop calls |
+| rules, shapes and families | 86% | 95.3% | 6 | 18 | 14 | 26 min · 42 |
+| acronyms, surnames, two-word tails | 98.5% | 96.3% | 6 | 12 | 8 | 22 min · 12 |
+| one label for identical names, second look on a split | 98.2% | 97.3% | 4 | 9 | 9 | 22 min · 15 |
+| one label only within one family (the version shipped) | 97.9% | 96.9% | 4 | 11 | 11 | 25 min · 46 |
+
+Wall clock is for 589 pairs with the model capped at three concurrent calls (the endpoint rate-limits above that), so about twenty-five pairs a minute; each run makes one batch call per cluster of up to twelve pairs, roughly sixty, plus the loop calls shown. The loop budget of 2000 calls a day was never approached.
+
+The last two runs differ by run-to-run noise of the model; the shipped version keeps the extractor's labels wherever they carry information (a periodical against a service, a game against an untyped record), because the run before it merged two such pairs, and a wrong merge costs more than a wrong keep. What remains disagreeing is mostly judgment: whether "Claude Mythos 5" is "Claude Mythos", whether "the Claude series" is "Claude", whether a nonprofit legal entity is the organization it controls. Two of the four wrong merges follow a precedent a simulated person left by accepting proposals blindly, which is the ledger working as designed on a bad decision. And the share left for people does not go to zero on purpose: the version and phrase rules hold back a few right merges to stop dozens of wrong ones, and a record with no facts cannot be settled by anyone.
+
### 7. The governor does not touch the verdict cache
The adjudicator's cache is keyed on names, types and facts. The governor's answer also depends on the precedents, which change with every human decision; a cached verdict would be one taken without them. The governor neither reads nor writes the cache, and the two are kept apart by the switch: off, the adjudicator runs the adjudicating stage as before; on, extraction enqueues the governor instead and both stages flow through the one queue.
diff --git a/docs/decisions/README.md b/docs/decisions/README.md
index 315bebba..d3f87bbc 100644
--- a/docs/decisions/README.md
+++ b/docs/decisions/README.md
@@ -48,7 +48,7 @@ The test for writing one: if someone (including us) looks at a piece of code in
| 0022 | [An unknown date is not an open one](0022-an-unknown-date-is-not-an-open-one.md) | Implemented in two cuts (#394 and the derived cut) · `world_axis` predicate beside `record_axis`, `facts.attested_at` anchors a missing start or an undated end at the document that attests it, every read and both client filters on the read interval, an undated ending closes the dated row it ends, derived rows intersect premise intervals as read and carry no precision on an anchored bound · two anchors (`attested_from` / `attested_to`), so a bare open row closes too (#393) |
| 0023 | [RSS observations are not documents](0023-rss-observations-are-not-documents.md) | Implemented in #326 |
| 0024 | [The world axis reaches the second](0024-the-world-axis-reaches-the-second.md) | Implemented · the precision ladder runs year → second (not below: no source states less), a stored value is truncated to its precision by CHECK on all three tables, a clock time without a zone is a date, one list spells the ladder in the extractor, the renderers, the export and the evaluator; a derived bound takes the precision of the premise that set it |
-| 0025 | [Governance reads the ledger before it decides](0025-governance-reads-the-ledger-before-it-decides.md) | Cut 1 implemented · a per-base `governance` switch, a `govern` job that works the duplicates queue first in, first out with each head's cluster, precedents pulled from the ledger into the prompt, a gate where the agent's own confidence decides and history lowers the bar or blocks (revised after the first big-file run), every look a row in `agent_decisions`, answers through the person's own decide path · the switch, the Agent queue, the proposal chip and the Overview section in the UI · a pair the batch cannot settle is looked at again with tools and then decided or asked about, within a daily budget · two reverts in a week turn the switch off and raise an alert |
+| 0025 | [Governance reads the ledger before it decides](0025-governance-reads-the-ledger-before-it-decides.md) | Cut 1 implemented · a per-base `governance` switch, a `govern` job that works the duplicates queue first in, first out with each head's cluster, precedents pulled from the ledger into the prompt, a gate where the agent's own confidence decides and history lowers the bar or blocks (revised after the first big-file run), every look a row in `agent_decisions`, answers through the person's own decide path · the switch, the Agent queue, the proposal chip and the Overview section in the UI · a pair the batch cannot settle is looked at again with tools and then decided or asked about, within a daily budget · two reverts in a week turn the switch off and raise an alert · identity rules the model reads and two it cannot argue with (name shapes, type families), measured against a hand-labeled set of 411 name pairs by `scripts/bench/govern.mjs` |
## Not a decision record
diff --git a/scripts/bench/README.md b/scripts/bench/README.md
index ad7e60f0..4e172456 100644
--- a/scripts/bench/README.md
+++ b/scripts/bench/README.md
@@ -43,6 +43,35 @@ node scripts/bench/run.mjs --corpus pharma --ontology /tmp/schemaorg.ttl --label
因为 `supersedes` 只在同一条目的相邻快照之间发生,随机抽块会把时态那根轴废掉。
- `subset.mjs` —— 把 schema.org 的 TTL 切成前 N 个类,给退化曲线用。
+## 治理的测量台(0025)
+
+`govern.mjs` 量的是 agent 替人裁重复项裁得对不对、还剩多少等人。真值在
+`truth/ai-timeline.duplicates.json`:把八篇互相咬合的条目(OpenAI 那场风波、Anthropic、
+DeepMind、Mistral、Inflection、SSI、ChatGPT)灌进一个治理开着的新库,消解器提出的
+615 对、411 个名字对,2026-09-07 按两侧的事实与原文逐对手标:`same` 同一个东西;
+`different` 两个东西,含版本、部门、子公司、只是含着这个名字的一句话;`unknown` 一侧
+没有事实、名字本身定不了的(光秃秃的「judge」「Time」),人也定不了。按两个名字键
+(不分左右、不分大小写);同名的一对只有在那次导入里每一对都同一个答案时才带标签。
+
+```
+node scripts/bench/govern.mjs --fresh # 新库、灌语料、跑完打分(约一小时)
+node scripts/bench/govern.mjs --kb <id> # 已有的库:撤掉 agent 的决定、再跑、打分(约二十分钟)
+node scripts/bench/govern.mjs --kb <id> --score # 只打分
+node scripts/bench/govern.mjs --kb <id> --score --stuck # 连留给人的那些一起列出来
+```
+
+`--kb` 那条路不重新抽取:`--reset` 用 `revert_merge` 同一套还原把 agent(和模拟的人)做的
+合并撤掉、分开的重开、建议删掉,直接在库里做而不经 API——经 API 会在台账上留下
+`merge.revert`,下一轮 agent 会把它当成人的先例。所以改一次判断、二十分钟出一次数。
+
+读数:`decided_on_its_own` 是它自己裁的;`agreed_with_labels` 是其中与标注一致的;
+`wrong_merges` 是最要紧的那个数——合错了要人去撤;`left_for_people` 是留给人的建议,
+后面跟着按标注人会合几对、分几对;`unlabeled` 是这次导入抽出来、真值里没有的名字对,
+不算对也不算错。语料重抽一次名字会略有出入,那一栏不为零是正常的。
+
+同一份语料上的对照(2026-09-06,闸门还是「没先例不合」那版):治理关着老裁决器自动合
+146、留 50 给人;治理开着一对不合、留 200。这个数就是记录 0025 决定 4 修订的起因。
+
## 读数怎么算
- `prompt_tokens_est` 是**本体段**的估算,不是整个提示词。实测 4.0 字符 ≈ 1 token
diff --git a/scripts/bench/govern.mjs b/scripts/bench/govern.mjs
new file mode 100644
index 00000000..fade061e
--- /dev/null
+++ b/scripts/bench/govern.mjs
@@ -0,0 +1,231 @@
+#!/usr/bin/env node
+// 治理的测量台(0025):agent 自己裁的与人标的一致不一致,还剩多少等人。
+//
+// 真值是 truth/<corpus>.duplicates.json:一次导入里消解器提出的每一对,按两个名字
+// 键(不分左右、不分大小写)人工标成 same / different / unknown。同一份语料再导一次,
+// 抽出来的名字会略有出入,对不上真值的对单独算一栏(unlabeled),不算对也不算错。
+//
+// 用法:
+// node scripts/bench/govern.mjs --fresh # 新库、灌语料(治理开着)、跑完打分
+// node scripts/bench/govern.mjs --kb <id> # 已有的库:撤掉 agent 的决定 → 再跑 → 打分
+// node scripts/bench/govern.mjs --kb <id> --score # 只打分,不动库
+// node scripts/bench/govern.mjs --kb <id> --reset # 只撤(agent 与模拟的人做的合并、分开、建议全部还原成等人)
+//
+// --reset 走的是 revert_merge 同一套还原(事实搬回、作废的恢复、时态修正撤销、别名与
+// 画像回退),直接在库里做而不经 API:经 API 会在台账上留下 merge.revert,下一轮 agent
+// 会把它当成人的先例。台账里 agent 自己的行(actor 为空)留着无妨——先例只认人的。
+//
+// 环境变量:BENCH_BASE(默认 http://127.0.0.1:8322)、BENCH_EMAIL / BENCH_PASSWORD、
+// BENCH_PSQL(默认 docker exec … psql -d utopia)。
+
+import fs from "node:fs";
+import path from "node:path";
+import { execFileSync } from "node:child_process";
+import { fileURLToPath } from "node:url";
+
+const HERE = path.dirname(fileURLToPath(import.meta.url));
+const BASE = process.env.BENCH_BASE || "http://127.0.0.1:8322";
+const EMAIL = process.env.BENCH_EMAIL || "[email protected]";
+const PASSWORD = process.env.BENCH_PASSWORD || "benchbench123";
+
+const args = Object.fromEntries(
+ process.argv.slice(2).reduce((acc, cur, i, arr) => {
+ if (cur.startsWith("--")) acc.push([cur.slice(2), arr[i + 1]?.startsWith("--") ? true : (arr[i + 1] ?? true)]);
+ return acc;
+ }, []),
+);
+const corpusName = args.corpus || "ai-timeline";
+const truth = JSON.parse(fs.readFileSync(path.join(HERE, "truth", `${corpusName}.duplicates.json`), "utf8"));
+
+let cookie = "";
+async function api(method, url, body) {
+ const init = { method, headers: {} };
+ if (cookie) init.headers.cookie = cookie;
+ if (body !== undefined) { init.headers["content-type"] = "application/json"; init.body = JSON.stringify(body); }
+ const r = await fetch(BASE + url, init);
+ for (const c of r.headers.getSetCookie?.() ?? []) cookie = c.split(";")[0];
+ const text = await r.text();
+ if (!r.ok) throw new Error(`${method} ${url} -> ${r.status} ${text.slice(0, 200)}`);
+ return text ? JSON.parse(text) : null;
+}
+function psql(sql) {
+ const cmd = process.env.BENCH_PSQL || "docker exec -e PGPASSWORD=utopia landscapebi-db-1 psql -U utopia -d utopia -tAc";
+ const parts = cmd.split(" ");
+ return execFileSync(parts[0], [...parts.slice(1), sql], { encoding: "utf8", maxBuffer: 64 << 20 }).trim();
+}
+const num = (sql) => Number(psql(sql) || 0);
+const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
+const log = (...a) => console.error(new Date().toISOString().slice(11, 19), ...a);
+async function until(fn, everyMs, stallMs) {
+ let deadline = Date.now() + stallMs, last = null;
+ for (;;) {
+ const r = await fn();
+ if (r === true) return;
+ if (typeof r === "number" && r !== last) { last = r; deadline = Date.now() + stallMs; }
+ if (Date.now() > deadline) throw new Error(`等超时:${Math.round(stallMs / 60000)} 分钟没有任何进展`);
+ await sleep(everyMs);
+ }
+}
+
+// ---------- 新库 + 灌语料 ----------
+async function fresh() {
+ const corpus = JSON.parse(fs.readFileSync(path.join(HERE, "corpora", `${corpusName}.json`), "utf8"));
+ const keep = new Set(truth.docs || corpus.docs.map((d) => d[0]));
+ const docs = corpus.docs.filter((d) => keep.has(d[0]));
+ const ws = (await api("GET", "/api/v1/workspaces"))[0].id;
+ const stamp = new Date().toISOString().slice(0, 16).replace(/[:T]/g, "-");
+ const kb = (await api("POST", `/api/v1/workspaces/${ws}/kbs`, { name: `govern ${args.label || corpusName} ${stamp}`, ontology_packs: ["schema-org"] })).id;
+ await sleep(6000);
+ await api("PATCH", `/api/v1/kbs/${kb}`, { governance: true });
+ log(`kb ${kb}: ${docs.length} docs going in`);
+ for (const [filename, content, docTime] of docs) {
+ const body = { filename, content };
+ if (docTime) body.doc_time = docTime;
+ await api("POST", `/api/v1/kbs/${kb}/ingest`, body);
+ }
+ await until(async () => {
+ const done = num(`SELECT count(*) FROM documents WHERE kb_id='${kb}' AND graph_status IN ('done','failed')`);
+ if (done >= docs.length) return true;
+ const chunks = num(`SELECT count(*) FROM chunks WHERE kb_id='${kb}' AND extracted_at IS NOT NULL`);
+ log(` 抽取 ${chunks} 块 / ${done} 篇完成`);
+ return chunks;
+ }, 20000, 1200000);
+ return kb;
+}
+
+// ---------- 撤掉 agent(与模拟的人)的决定 ----------
+function reset(kb) {
+ psql(`DO $$
+DECLARE m RECORD; src RECORD;
+BEGIN
+ FOR m IN SELECT * FROM entity_merges WHERE kb_id = '${kb}' AND reverted_at IS NULL ORDER BY created_at DESC LOOP
+ UPDATE facts SET subject_id = m.source_id WHERE id = ANY(m.moved_subject_facts);
+ UPDATE facts SET object_id = m.source_id WHERE id = ANY(m.moved_object_facts);
+ UPDATE facts SET invalidated_at = NULL WHERE id = ANY(m.invalidated_facts);
+ UPDATE facts SET invalidated_at = NULL WHERE id IN (
+ SELECT supersedes FROM facts WHERE id = ANY(m.temporal_corrections) AND invalidated_at IS NULL AND supersedes IS NOT NULL);
+ UPDATE facts SET invalidated_at = now() WHERE id = ANY(m.temporal_corrections) AND invalidated_at IS NULL;
+ SELECT canonical_name, aliases INTO src FROM entities WHERE id = m.source_id;
+ UPDATE entities SET
+ aliases = (SELECT coalesce(array_agg(a), '{}') FROM unnest(aliases) a
+ WHERE lower(a) <> ALL (SELECT lower(x) FROM unnest(array_prepend(src.canonical_name, src.aliases)) x)),
+ profile_embedding = m.target_profile_before, profile_n = m.target_profile_n_before,
+ type_id = coalesce(m.target_type_before, type_id), updated_at = now()
+ WHERE id = m.target_id;
+ UPDATE entities SET merged_into = NULL, updated_at = now() WHERE id = m.source_id;
+ END LOOP;
+ DELETE FROM entity_merges WHERE kb_id = '${kb}';
+ -- 同一对实体可能有一行已裁、一行又等着(消解器裁完之后又提了一次):等着的那行留着,
+ -- 已裁的重复行删掉;一对只剩一行已裁的,重开成等人
+ DELETE FROM resolution_reviews rr WHERE rr.kb_id = '${kb}' AND rr.status <> 'pending'
+ AND EXISTS (SELECT 1 FROM resolution_reviews p WHERE p.kb_id = rr.kb_id AND p.status = 'pending'
+ AND least(p.left_id, p.right_id) = least(rr.left_id, rr.right_id)
+ AND greatest(p.left_id, p.right_id) = greatest(rr.left_id, rr.right_id));
+ DELETE FROM resolution_reviews rr WHERE rr.kb_id = '${kb}' AND rr.status <> 'pending'
+ AND rr.id <> (SELECT p.id FROM resolution_reviews p WHERE p.kb_id = rr.kb_id AND p.status <> 'pending'
+ AND least(p.left_id, p.right_id) = least(rr.left_id, rr.right_id)
+ AND greatest(p.left_id, p.right_id) = greatest(rr.left_id, rr.right_id)
+ ORDER BY p.decided_at DESC NULLS LAST, p.created_at DESC LIMIT 1);
+ UPDATE resolution_reviews SET status = 'pending', stage = 'human', decided_at = NULL, decided_by = NULL,
+ reason = CASE WHEN reason LIKE 'governed|%' OR reason = 'proposed' OR reason LIKE 'escalate_%' THEN NULL ELSE reason END
+ WHERE kb_id = '${kb}' AND status <> 'pending';
+ UPDATE resolution_reviews SET reason = NULL WHERE kb_id = '${kb}' AND (reason LIKE 'governed|%' OR reason = 'proposed' OR reason LIKE 'escalate_%');
+ DELETE FROM agent_decisions WHERE kb_id = '${kb}';
+END $$;`);
+ log(`reset: ${num(`SELECT count(*) FROM resolution_reviews WHERE kb_id='${kb}' AND status='pending'`)} pairs waiting, 0 agent rows`);
+}
+
+// ---------- 跑一轮 ----------
+async function run(kb) {
+ await api("PATCH", `/api/v1/kbs/${kb}`, { governance: true });
+ if (num(`SELECT count(*) FROM jobs WHERE kind='govern' AND status IN ('queued','running') AND payload->>'kb_id'='${kb}'`) === 0) {
+ psql(`INSERT INTO jobs (kind, payload) VALUES ('govern', jsonb_build_object('kb_id', '${kb}'))`);
+ }
+ const t0 = Date.now();
+ await until(async () => {
+ const busy = num(`SELECT count(*) FROM jobs WHERE kind='govern' AND status IN ('queued','running') AND payload->>'kb_id'='${kb}'`);
+ const left = num(`SELECT count(*) FROM resolution_reviews rr WHERE rr.kb_id='${kb}' AND rr.status='pending'
+ AND NOT EXISTS (SELECT 1 FROM agent_decisions d WHERE d.target_id = rr.id AND d.status = 'proposed')`);
+ if (busy === 0 && left === 0) return true;
+ if (busy === 0) {
+ // 任务停了但还有没看的:失败重试耗尽?再排一个
+ const failed = psql(`SELECT left(COALESCE(last_error,''),120) FROM jobs WHERE kind='govern' AND payload->>'kb_id'='${kb}' ORDER BY id DESC LIMIT 1`);
+ log(` job stopped with ${left} pairs unseen: ${failed}`);
+ psql(`INSERT INTO jobs (kind, payload) VALUES ('govern', jsonb_build_object('kb_id', '${kb}'))`);
+ }
+ const rows = num(`SELECT count(*) FROM agent_decisions WHERE kb_id='${kb}'`);
+ log(` ${left} pairs not yet looked at, ${rows} agent rows`);
+ return left * 10000 + rows;
+ }, 15000, 900000);
+ log(`run took ${Math.round((Date.now() - t0) / 60000)} min`);
+}
+
+// ---------- 打分 ----------
+function score(kb) {
+ const key = (a, b) => [a.toLowerCase(), b.toLowerCase()].sort().join(" | ");
+ const want = new Map(truth.pairs.map((p) => [key(p.left, p.right), p.verdict]));
+ const rows = JSON.parse(psql(`SELECT COALESCE(json_agg(json_build_object(
+ 'l', a.canonical_name, 'r', b.canonical_name, 'status', rr.status, 'reason', rr.reason, 'by_person', rr.decided_by IS NOT NULL,
+ 'agent', (SELECT json_build_object('action', d.action, 'status', d.status, 'confidence', d.confidence, 'calls', d.calls, 'reason', d.reason)
+ FROM agent_decisions d WHERE d.target_id = rr.id ORDER BY d.created_at DESC LIMIT 1))), '[]')
+ FROM resolution_reviews rr JOIN entities a ON a.id = rr.left_id JOIN entities b ON b.id = rr.right_id
+ WHERE rr.kb_id = '${kb}'`));
+ const c = { pairs: rows.length, labeled: 0, unlabeled: 0, unknown: 0,
+ applied: 0, right: 0, wrong_merge: 0, wrong_keep: 0, applied_unknown: 0,
+ proposed: 0, proposed_same: 0, proposed_different: 0, unsure: 0, by_person: 0, unseen: 0, moot: 0, loop_rows: 0, loop_calls: 0 };
+ const wrong = [], stuck = [];
+ for (const r of rows) {
+ const t = want.get(key(r.l, r.r));
+ if (t === undefined) { c.unlabeled++; }
+ else if (t === "unknown") c.unknown++;
+ else c.labeled++;
+ const d = r.agent;
+ if (d?.calls > 0) { c.loop_rows++; c.loop_calls += d.calls; }
+ if (r.by_person) { c.by_person++; continue; }
+ // 一侧并进了别人之后,这一对本身不存在了:库里关成 kept、reason 说明是连带的
+ if (!d && r.reason === "superseded by merge") { c.moot++; continue; }
+ if (!d) { c.unseen++; continue; }
+ if (d.status === "applied") {
+ c.applied++;
+ if (t === undefined || t === "unknown") { c.applied_unknown++; continue; }
+ const ok = (d.action === "merge" && t === "same") || (d.action === "keep" && t === "different");
+ if (ok) c.right++;
+ else { if (d.action === "merge") c.wrong_merge++; else c.wrong_keep++; wrong.push(`${d.action}@${d.confidence} on "${r.l}" ≟ "${r.r}" (truth ${t}) — ${d.reason ?? ""}`); }
+ } else if (d.status === "proposed") {
+ c.proposed++;
+ if (d.action === "unsure") c.unsure++;
+ if (t === "same") c.proposed_same++; else if (t === "different") c.proposed_different++;
+ stuck.push(`${d.action}@${d.confidence} "${r.l}" ≟ "${r.r}" (truth ${t ?? "?"})`);
+ } else c.unseen++;
+ }
+ const pct = (a, b) => (b ? `${Math.round((a / b) * 1000) / 10}%` : "-");
+ const out = {
+ kb, pairs: c.pairs, labeled: c.labeled, unlabeled: c.unlabeled, unknown: c.unknown,
+ decided_on_its_own: c.applied, agreed_with_labels: `${c.right}/${c.applied - c.applied_unknown} (${pct(c.right, c.applied - c.applied_unknown)})`,
+ wrong_merges: c.wrong_merge, wrong_keeps: c.wrong_keep,
+ left_for_people: c.proposed, of_which_unsure: c.unsure, people_would_merge: c.proposed_same, people_would_keep: c.proposed_different,
+ decided_by_person: c.by_person, moot_after_a_merge: c.moot, not_looked_at: c.unseen,
+ automatic_share: pct(c.applied, c.pairs - c.by_person - c.moot),
+ loop: `${c.loop_rows} pairs, ${c.loop_calls} calls`,
+ };
+ console.log(JSON.stringify(out, null, 2));
+ if (wrong.length) console.log("\nWRONG\n" + wrong.join("\n"));
+ if (args.stuck && stuck.length) console.log("\nLEFT FOR PEOPLE\n" + stuck.join("\n"));
+ return out;
+}
+
+async function main() {
+ await api("POST", "/api/v1/auth/login", { email: EMAIL, password: PASSWORD });
+ let kb = args.kb;
+ if (args.fresh) kb = await fresh();
+ if (!kb) throw new Error("--kb <id> or --fresh");
+ const only = args.score || args.reset;
+ if (args.reset) reset(kb);
+ if (!only || args.run) {
+ if (!args.fresh && !args.reset && !args.run) reset(kb);
+ if (!args.fresh) await run(kb);
+ else await run(kb);
+ }
+ if (!args.reset || args.score) score(kb);
+}
+main().catch((e) => { console.error(e); process.exit(1); });
diff --git a/scripts/bench/truth/ai-timeline.duplicates.json b/scripts/bench/truth/ai-timeline.duplicates.json
new file mode 100644
index 00000000..1828b431
--- /dev/null
+++ b/scripts/bench/truth/ai-timeline.duplicates.json
@@ -0,0 +1,2166 @@
+{
+ "name": "ai-timeline duplicates",
+ "corpus": "ai-timeline",
+ "docs": [
+ "removal-of-sam-altman-from-openai.txt",
+ "openai.txt",
+ "anthropic.txt",
+ "google-deepmind.txt",
+ "mistral-ai.txt",
+ "inflection-ai.txt",
+ "safe-superintelligence-inc.txt",
+ "chatgpt.txt"
+ ],
+ "match": "names",
+ "note": "Every duplicate pair the resolver raised on one governed import of the eight documents above (615 pairs, 411 distinct name pairs), labeled by hand on 2026-09-07 from each side's facts and source passages. same = one real-world thing; different = two things, including a version, a division, a subsidiary or a phrase that merely contains the name; unknown = a side with no facts and a name that settles nothing (a bare 'judge', a bare 'Time'), which no reader could decide either. Keyed by the two names, either order, case-insensitive: a pair of identical names carries the label only when every such pair in that import shared it.",
+ "counts": {
+ "different": 296,
+ "same": 110,
+ "unknown": 5
+ },
+ "pairs": [
+ {
+ "left": "$10 billion",
+ "right": "$10 billion investment",
+ "verdict": "different",
+ "why": "2025 share sale vs 2023 Microsoft investment"
+ },
+ {
+ "left": "$1 billion",
+ "right": "$1 billion investment",
+ "verdict": "different"
+ },
+ {
+ "left": "$1 billion",
+ "right": "US$1 billion",
+ "verdict": "different",
+ "why": "SSI raise vs OpenAI revenue"
+ },
+ {
+ "left": "$1 billion investment",
+ "right": "$1 billion investment in OpenAI LP",
+ "verdict": "same",
+ "why": "Microsoft's 2019 $1B investment in OpenAI LP"
+ },
+ {
+ "left": "$1 billion investment in OpenAI LP",
+ "right": "OpenAI",
+ "verdict": "different"
+ },
+ {
+ "left": "$4 billion",
+ "right": "another $4 billion",
+ "verdict": "unknown",
+ "why": "the bare \"$4 billion\" has no facts"
+ },
+ {
+ "left": "$4 billion",
+ "right": "US$1.3 billion at $4 billion valuation",
+ "verdict": "different"
+ },
+ {
+ "left": "$965 billion",
+ "right": "US$965 billion",
+ "verdict": "same",
+ "why": "Anthropic's $965B valuation, May 2026"
+ },
+ {
+ "left": "100 million monthly active users",
+ "right": "users",
+ "verdict": "different"
+ },
+ {
+ "left": "2024 International Mathematical Olympiad",
+ "right": "International Mathematical Olympiad",
+ "verdict": "different",
+ "why": "one edition vs the event series"
+ },
+ {
+ "left": "2026 United States intervention in Venezuela",
+ "right": "Venezuela",
+ "verdict": "different"
+ },
+ {
+ "left": "900 million weekly active users",
+ "right": "users",
+ "verdict": "different"
+ },
+ {
+ "left": "ability to create images, using Black Forest Labs' Flux Pro model",
+ "right": "Black Forest Labs' Flux Pro model",
+ "verdict": "different"
+ },
+ {
+ "left": "ability to create images, using Black Forest Labs' Flux Pro model",
+ "right": "Labs",
+ "verdict": "different"
+ },
+ {
+ "left": "ability to use Claude for all lawful purposes",
+ "right": "Claude",
+ "verdict": "different"
+ },
+ {
+ "left": "account belonging to the perpetrator",
+ "right": "the perpetrator",
+ "verdict": "different",
+ "why": "a person vs his account"
+ },
+ {
+ "left": "a consistent pattern of [...] Lying",
+ "right": "lying",
+ "verdict": "different",
+ "why": "two allegations from different documents"
+ },
+ {
+ "left": "actions",
+ "right": "autonomously performing tasks through web browser interactions",
+ "verdict": "different"
+ },
+ {
+ "left": "acute kidney injury",
+ "right": "acute kidney injury management",
+ "verdict": "different"
+ },
+ {
+ "left": "Adam D'Angelo",
+ "right": "D'Angelo",
+ "verdict": "same",
+ "why": "D'Angelo is Adam D'Angelo"
+ },
+ {
+ "left": "a decades-long project that eventually surpasses human intelligence",
+ "right": "event",
+ "verdict": "different"
+ },
+ {
+ "left": "a dozen employees",
+ "right": "employees",
+ "verdict": "different"
+ },
+ {
+ "left": "agents",
+ "right": "develop more helpful AI agents",
+ "verdict": "different"
+ },
+ {
+ "left": "AI Mode",
+ "right": "AI models",
+ "verdict": "different"
+ },
+ {
+ "left": "AI Mode",
+ "right": "An API that gives access to various OpenAI models",
+ "verdict": "different"
+ },
+ {
+ "left": "AI Mode",
+ "right": "unreleased AI model",
+ "verdict": "different"
+ },
+ {
+ "left": "AI models",
+ "right": "An API that gives access to various OpenAI models",
+ "verdict": "different"
+ },
+ {
+ "left": "AI safety researchers",
+ "right": "safety",
+ "verdict": "different"
+ },
+ {
+ "left": "AI system",
+ "right": "AI system",
+ "verdict": "different",
+ "why": "Anthropic's constitutional-AI system vs DeepMind's Atari agent"
+ },
+ {
+ "left": "AlphaDev",
+ "right": "MuZero, AlphaStar, AlphaGeometry, FunSearch, AlphaEvolve, AlphaDev, AlphaTensor",
+ "verdict": "different",
+ "why": "a list vs one of its members"
+ },
+ {
+ "left": "AlphaFold",
+ "right": "AlphaFold2",
+ "verdict": "different",
+ "why": "a version vs the product"
+ },
+ {
+ "left": "AlphaFold",
+ "right": "AlphaFold3",
+ "verdict": "different"
+ },
+ {
+ "left": "AlphaFold",
+ "right": "AlphaFold's achievement",
+ "verdict": "different"
+ },
+ {
+ "left": "AlphaGo",
+ "right": "AlphaGo Zero",
+ "verdict": "different",
+ "why": "AlphaGo Zero is its own program"
+ },
+ {
+ "left": "AlphaStar",
+ "right": "MuZero, AlphaStar, AlphaGeometry, FunSearch, AlphaEvolve, AlphaDev, AlphaTensor",
+ "verdict": "different"
+ },
+ {
+ "left": "Altimeter",
+ "right": "Altimeter Capital",
+ "verdict": "same",
+ "why": "Altimeter is Altimeter Capital"
+ },
+ {
+ "left": "Altimeter",
+ "right": "Time",
+ "verdict": "different"
+ },
+ {
+ "left": "Altimeter Capital",
+ "right": "Time",
+ "verdict": "different"
+ },
+ {
+ "left": "Altman and Brockman",
+ "right": "Brockman",
+ "verdict": "different"
+ },
+ {
+ "left": "Altman intended to return to OpenAI with support from Sutskever",
+ "right": "OpenAI",
+ "verdict": "different"
+ },
+ {
+ "left": "Amazon",
+ "right": "Amazon",
+ "verdict": "same",
+ "why": "Amazon the company in both"
+ },
+ {
+ "left": "Amazon",
+ "right": "Amazon Web Services",
+ "verdict": "different",
+ "why": "AWS is a subsidiary"
+ },
+ {
+ "left": "Amazon",
+ "right": "Amazon Web Services (AWS)",
+ "verdict": "different"
+ },
+ {
+ "left": "Amazon Web Services",
+ "right": "Amazon Web Services",
+ "verdict": "same"
+ },
+ {
+ "left": "Amazon Web Services",
+ "right": "Amazon Web Services (AWS)",
+ "verdict": "same",
+ "why": "AWS with and without the acronym"
+ },
+ {
+ "left": "AMD",
+ "right": "AMD",
+ "verdict": "same"
+ },
+ {
+ "left": "AMD Instinct MI450 GPUs",
+ "right": "MI450",
+ "verdict": "same",
+ "why": "MI450 is the AMD Instinct MI450"
+ },
+ {
+ "left": "American artificial intelligence (AI) public benefit corporation",
+ "right": "public benefit corporation",
+ "verdict": "different",
+ "why": "the corporate form vs OpenAI itself"
+ },
+ {
+ "left": "Android",
+ "right": "iOS and Android mobile devices",
+ "verdict": "different"
+ },
+ {
+ "left": "Android team at Google",
+ "right": "Google",
+ "verdict": "different"
+ },
+ {
+ "left": "Anduril",
+ "right": "Anduril Industries",
+ "verdict": "same",
+ "why": "Anduril is Anduril Industries"
+ },
+ {
+ "left": "Anthropic",
+ "right": "Anthropic",
+ "verdict": "same"
+ },
+ {
+ "left": "Anthropic",
+ "right": "Anthropic Institute",
+ "verdict": "different",
+ "why": "the institute is a subsidiary"
+ },
+ {
+ "left": "Anthropic",
+ "right": "Anthropic lease",
+ "verdict": "different"
+ },
+ {
+ "left": "APEC United States 2023",
+ "right": "United States",
+ "verdict": "different"
+ },
+ {
+ "left": "API",
+ "right": "API",
+ "verdict": "same",
+ "why": "OpenAI's API in both"
+ },
+ {
+ "left": "Apple",
+ "right": "Apple",
+ "verdict": "same"
+ },
+ {
+ "left": "Apple",
+ "right": "Apple Inc.",
+ "verdict": "same",
+ "why": "Apple is Apple Inc."
+ },
+ {
+ "left": "Apple",
+ "right": "Apple's terms of service",
+ "verdict": "different"
+ },
+ {
+ "left": "Arizona State University",
+ "right": "Arizona State University",
+ "verdict": "same"
+ },
+ {
+ "left": "artificial general intelligence (AGI)",
+ "right": "ensuring that artificial general intelligence (AGI) \"benefits all of humanity\"",
+ "verdict": "different"
+ },
+ {
+ "left": "Atari 2600 games",
+ "right": "Go, chess, shogi, and Atari 2600 games",
+ "verdict": "different"
+ },
+ {
+ "left": "attorneys general of California and Delaware",
+ "right": "California",
+ "verdict": "different"
+ },
+ {
+ "left": "attorneys general of California and Delaware",
+ "right": "Delaware",
+ "verdict": "different"
+ },
+ {
+ "left": "authors",
+ "right": "authors and media companies",
+ "verdict": "different"
+ },
+ {
+ "left": "authors",
+ "right": "authors and publishers",
+ "verdict": "different"
+ },
+ {
+ "left": "authors",
+ "right": "Authors Guild",
+ "verdict": "different"
+ },
+ {
+ "left": "authors",
+ "right": "The Authors Guild",
+ "verdict": "different"
+ },
+ {
+ "left": "autonomous cyberattack",
+ "right": "cyberattack",
+ "verdict": "same",
+ "why": "the 2026-07 autonomous cyberattack by OpenAI agents"
+ },
+ {
+ "left": "Axios",
+ "right": "Axios",
+ "verdict": "same"
+ },
+ {
+ "left": "Bard",
+ "right": "Bard",
+ "verdict": "same",
+ "why": "Bard, later Gemini"
+ },
+ {
+ "left": "board members",
+ "right": "board members of staging a coup",
+ "verdict": "different",
+ "why": "a garbled phrase vs the board members"
+ },
+ {
+ "left": "Books1",
+ "right": "Books1 and Books2 contained over 100,000 copyrighted books",
+ "verdict": "different"
+ },
+ {
+ "left": "Books1 and Books2 contained over 100,000 copyrighted books",
+ "right": "Books2",
+ "verdict": "different"
+ },
+ {
+ "left": "bots",
+ "right": "Dota 2 bots",
+ "verdict": "different"
+ },
+ {
+ "left": "Breakout",
+ "right": "Breakout",
+ "verdict": "same"
+ },
+ {
+ "left": "Bret Taylor",
+ "right": "Taylor",
+ "verdict": "same",
+ "why": "Taylor is Bret Taylor"
+ },
+ {
+ "left": "Brian Chesky",
+ "right": "Chesky",
+ "verdict": "same",
+ "why": "Chesky is Brian Chesky"
+ },
+ {
+ "left": "Broadcom",
+ "right": "Broadcom",
+ "verdict": "same"
+ },
+ {
+ "left": "Brockman",
+ "right": "Greg Brockman",
+ "verdict": "same",
+ "why": "Brockman is Greg Brockman"
+ },
+ {
+ "left": "California",
+ "right": "Palo Alto, California",
+ "verdict": "different"
+ },
+ {
+ "left": "California",
+ "right": "San Francisco, California",
+ "verdict": "different"
+ },
+ {
+ "left": "candidate high-probability moves",
+ "right": "high",
+ "verdict": "different"
+ },
+ {
+ "left": "ChatGPT",
+ "right": "ChatGPT",
+ "verdict": "same",
+ "why": "ChatGPT under two types"
+ },
+ {
+ "left": "ChatGPT",
+ "right": "ChatGPT agent",
+ "verdict": "different"
+ },
+ {
+ "left": "ChatGPT",
+ "right": "ChatGPT Android app",
+ "verdict": "different",
+ "why": "an app vs the product"
+ },
+ {
+ "left": "ChatGPT",
+ "right": "ChatGPT Atlas",
+ "verdict": "different"
+ },
+ {
+ "left": "ChatGPT",
+ "right": "ChatGPT iOS app",
+ "verdict": "different"
+ },
+ {
+ "left": "ChatGPT",
+ "right": "ChatGPT played a role in enabling a prolonged stalking campaign against her",
+ "verdict": "different"
+ },
+ {
+ "left": "ChatGPT",
+ "right": "ChatGPT Plus",
+ "verdict": "different"
+ },
+ {
+ "left": "ChatGPT",
+ "right": "ChatGPT Pro",
+ "verdict": "different"
+ },
+ {
+ "left": "ChatGPT",
+ "right": "ChatGPT Search",
+ "verdict": "different"
+ },
+ {
+ "left": "ChatGPT",
+ "right": "ChatGPT update",
+ "verdict": "different"
+ },
+ {
+ "left": "ChatGPT",
+ "right": "ChatGPT Windows app",
+ "verdict": "different"
+ },
+ {
+ "left": "ChatGPT",
+ "right": "ChatGPT Work",
+ "verdict": "different"
+ },
+ {
+ "left": "ChatGPT",
+ "right": "custom ChatGPT instances",
+ "verdict": "different"
+ },
+ {
+ "left": "ChatGPT",
+ "right": "dedicated task force on ChatGPT",
+ "verdict": "different"
+ },
+ {
+ "left": "ChatGPT",
+ "right": "feature of ChatGPT Plus",
+ "verdict": "different"
+ },
+ {
+ "left": "ChatGPT",
+ "right": "increased use of ChatGPT for China-origin influence operations",
+ "verdict": "different"
+ },
+ {
+ "left": "ChatGPT",
+ "right": "new sign-ups for ChatGPT Plus",
+ "verdict": "different"
+ },
+ {
+ "left": "ChatGPT",
+ "right": "non-work tasks account for more than 72 percent of all ChatGPT usage",
+ "verdict": "different"
+ },
+ {
+ "left": "ChatGPT",
+ "right": "study on how people use ChatGPT for everyday tasks",
+ "verdict": "different"
+ },
+ {
+ "left": "ChatGPT",
+ "right": "the risks of using ChatGPT for therapy",
+ "verdict": "different"
+ },
+ {
+ "left": "ChatGPT Atlas",
+ "right": "ChatGPT Atlas",
+ "verdict": "same"
+ },
+ {
+ "left": "ChatGPT Enterprise",
+ "right": "ChatGPT Enterprise",
+ "verdict": "same"
+ },
+ {
+ "left": "ChatGPT iOS app",
+ "right": "iOS app",
+ "verdict": "different",
+ "why": "Claude's iOS app vs ChatGPT's"
+ },
+ {
+ "left": "ChatGPT Plus",
+ "right": "ChatGPT Plus",
+ "verdict": "same"
+ },
+ {
+ "left": "chess",
+ "right": "Go, chess, shogi, and Atari 2600 games",
+ "verdict": "different"
+ },
+ {
+ "left": "Chinese government",
+ "right": "Chinese government",
+ "verdict": "same"
+ },
+ {
+ "left": "class action lawsuit",
+ "right": "lawsuit",
+ "verdict": "unknown",
+ "why": "\"lawsuit\" is a catch-all with three suits behind it"
+ },
+ {
+ "left": "class-action lawsuit",
+ "right": "lawsuit",
+ "verdict": "different",
+ "why": "the stalking suit against OpenAI vs the authors' class action against Anthropic"
+ },
+ {
+ "left": "class action lawsuit against OpenAI",
+ "right": "OpenAI",
+ "verdict": "different"
+ },
+ {
+ "left": "Claude",
+ "right": "Claude",
+ "verdict": "same",
+ "why": "Claude the LLM under two types"
+ },
+ {
+ "left": "Claude",
+ "right": "Claude 2",
+ "verdict": "different",
+ "why": "a version vs the model line"
+ },
+ {
+ "left": "Claude",
+ "right": "Claude 3.5 Sonnet",
+ "verdict": "different"
+ },
+ {
+ "left": "Claude",
+ "right": "Claude 3 Haiku",
+ "verdict": "different"
+ },
+ {
+ "left": "Claude",
+ "right": "Claude 3 Opus",
+ "verdict": "different"
+ },
+ {
+ "left": "Claude",
+ "right": "Claude 3 Sonnet",
+ "verdict": "different"
+ },
+ {
+ "left": "Claude",
+ "right": "Claude 4",
+ "verdict": "different"
+ },
+ {
+ "left": "Claude",
+ "right": "Claude 4 Opus",
+ "verdict": "different"
+ },
+ {
+ "left": "Claude",
+ "right": "Claude 4 Sonnet",
+ "verdict": "different"
+ },
+ {
+ "left": "Claude",
+ "right": "Claude Code",
+ "verdict": "different",
+ "why": "a product built on Claude"
+ },
+ {
+ "left": "Claude",
+ "right": "Claude Fable 5 and Mythos 5",
+ "verdict": "different"
+ },
+ {
+ "left": "Claude",
+ "right": "Claude Gov",
+ "verdict": "different"
+ },
+ {
+ "left": "Claude",
+ "right": "Claude Instant",
+ "verdict": "different"
+ },
+ {
+ "left": "Claude",
+ "right": "Claude Mythos",
+ "verdict": "different"
+ },
+ {
+ "left": "Claude",
+ "right": "Claude Mythos 5",
+ "verdict": "different"
+ },
+ {
+ "left": "Claude",
+ "right": "Claude Opus 4.6",
+ "verdict": "different"
+ },
+ {
+ "left": "Claude",
+ "right": "Claude series of large language models",
+ "verdict": "same",
+ "why": "the Claude series is the LLM Claude"
+ },
+ {
+ "left": "Claude 2",
+ "right": "Claude 2",
+ "verdict": "same"
+ },
+ {
+ "left": "Claude 3 Haiku",
+ "right": "Haiku",
+ "verdict": "different",
+ "why": "one model vs the tier"
+ },
+ {
+ "left": "Claude 4",
+ "right": "Claude 4 Opus",
+ "verdict": "different"
+ },
+ {
+ "left": "Claude 4",
+ "right": "Claude 4 Sonnet",
+ "verdict": "different"
+ },
+ {
+ "left": "Claude Fable 5 and Mythos 5",
+ "right": "Fable",
+ "verdict": "different"
+ },
+ {
+ "left": "Claude Fable 5 and Mythos 5",
+ "right": "Fable 5",
+ "verdict": "different",
+ "why": "two models vs one"
+ },
+ {
+ "left": "Claude Mythos",
+ "right": "Claude Mythos 5",
+ "verdict": "same",
+ "why": "Claude Mythos 5 is the Mythos model announced in April 2026"
+ },
+ {
+ "left": "Codeforces",
+ "right": "GitHub data and Codeforces problems and solutions",
+ "verdict": "different"
+ },
+ {
+ "left": "collaboration with the People's Republic of China",
+ "right": "People's Republic of China",
+ "verdict": "different"
+ },
+ {
+ "left": "companies using OpenAI",
+ "right": "OpenAI",
+ "verdict": "different"
+ },
+ {
+ "left": "competitiveness and safety concerns",
+ "right": "safety",
+ "verdict": "different"
+ },
+ {
+ "left": "concerns about AI safety and existential risk from artificial general intelligence",
+ "right": "safety",
+ "verdict": "different"
+ },
+ {
+ "left": "consortium of investors led by Elon Musk",
+ "right": "investors",
+ "verdict": "different"
+ },
+ {
+ "left": "constitution",
+ "right": "Constitutional AI",
+ "verdict": "different",
+ "why": "the framework vs the document"
+ },
+ {
+ "left": "Copilot",
+ "right": "Copilot Studio",
+ "verdict": "different"
+ },
+ {
+ "left": "Copilot",
+ "right": "Microsoft Copilot",
+ "verdict": "same",
+ "why": "Copilot is Microsoft Copilot"
+ },
+ {
+ "left": "copyright infringement",
+ "right": "copyright infringement lawsuit",
+ "verdict": "different"
+ },
+ {
+ "left": "copyright infringement",
+ "right": "lawsuit for copyright infringement",
+ "verdict": "different"
+ },
+ {
+ "left": "copyright infringement lawsuit",
+ "right": "lawsuit",
+ "verdict": "different"
+ },
+ {
+ "left": "critical thinking",
+ "right": "Thinking",
+ "verdict": "different"
+ },
+ {
+ "left": "C++ Standard Library sorting algorithms",
+ "right": "sorting algorithm",
+ "verdict": "different"
+ },
+ {
+ "left": "Dario Amodei",
+ "right": "Dario Amodei",
+ "verdict": "same"
+ },
+ {
+ "left": "Data Protection Act",
+ "right": "Royal Free hospital failed to comply with the Data Protection Act",
+ "verdict": "different"
+ },
+ {
+ "left": "data-sharing agreement between DeepMind and the Royal Free",
+ "right": "DeepMind",
+ "verdict": "different"
+ },
+ {
+ "left": "DeepMind",
+ "right": "DeepMind",
+ "verdict": "same"
+ },
+ {
+ "left": "DeepMind",
+ "right": "DeepMind Ethics and Society",
+ "verdict": "different"
+ },
+ {
+ "left": "DeepMind",
+ "right": "DeepMind Ethics & Society",
+ "verdict": "different"
+ },
+ {
+ "left": "DeepMind",
+ "right": "DeepMind Health",
+ "verdict": "different",
+ "why": "a division"
+ },
+ {
+ "left": "DeepMind",
+ "right": "DeepMind Health data sharing",
+ "verdict": "different"
+ },
+ {
+ "left": "DeepMind",
+ "right": "DeepMind Professors of machine learning",
+ "verdict": "different"
+ },
+ {
+ "left": "DeepMind",
+ "right": "DeepMind's initial algorithms",
+ "verdict": "different"
+ },
+ {
+ "left": "DeepMind",
+ "right": "Google DeepMind",
+ "verdict": "same",
+ "why": "DeepMind became Google DeepMind"
+ },
+ {
+ "left": "DeepMind Health",
+ "right": "DeepMind Health data sharing",
+ "verdict": "different"
+ },
+ {
+ "left": "deep reinforcement learning",
+ "right": "reinforcement learning",
+ "verdict": "different",
+ "why": "deep RL is a sub-technique"
+ },
+ {
+ "left": "Deep Research",
+ "right": "deep research agent",
+ "verdict": "same",
+ "why": "Deep Research is the deep research agent"
+ },
+ {
+ "left": "Delaware",
+ "right": "Delaware",
+ "verdict": "same"
+ },
+ {
+ "left": "Department of Defense",
+ "right": "Department of Defense (DoD)",
+ "verdict": "same"
+ },
+ {
+ "left": "Department of Defense",
+ "right": "United States Department of Defense",
+ "verdict": "same"
+ },
+ {
+ "left": "Department of Defense",
+ "right": "United States Department of Defense (DoD)",
+ "verdict": "same"
+ },
+ {
+ "left": "Department of Defense",
+ "right": "US Department of Defense",
+ "verdict": "same"
+ },
+ {
+ "left": "Department of Defense (DoD)",
+ "right": "United States Department of Defense (DoD)",
+ "verdict": "same"
+ },
+ {
+ "left": "DevDay event",
+ "right": "event",
+ "verdict": "different"
+ },
+ {
+ "left": "DGX-1 supercomputer",
+ "right": "supercomputer",
+ "verdict": "different",
+ "why": "Microsoft's supercomputer vs Nvidia's DGX-1"
+ },
+ {
+ "left": "DolphinGemma",
+ "right": "Gemma",
+ "verdict": "different"
+ },
+ {
+ "left": "domestic surveillance",
+ "right": "drop restrictions against mass domestic surveillance and fully autonomous weapons",
+ "verdict": "different"
+ },
+ {
+ "left": "Donald Trump",
+ "right": "President Donald Trump",
+ "verdict": "same"
+ },
+ {
+ "left": "drop restrictions against mass domestic surveillance and fully autonomous weapons",
+ "right": "fully autonomous weapons",
+ "verdict": "different"
+ },
+ {
+ "left": "education",
+ "right": "medical education",
+ "verdict": "different"
+ },
+ {
+ "left": "efforts",
+ "right": "Sam Altman's efforts",
+ "verdict": "different"
+ },
+ {
+ "left": "Emmett Shear",
+ "right": "Shear",
+ "verdict": "same",
+ "why": "Shear is Emmett Shear"
+ },
+ {
+ "left": "employees",
+ "right": "employees and other investors",
+ "verdict": "different"
+ },
+ {
+ "left": "employees",
+ "right": "first 52 employees",
+ "verdict": "different"
+ },
+ {
+ "left": "employees",
+ "right": "former employees",
+ "verdict": "different"
+ },
+ {
+ "left": "employees",
+ "right": "former employees from the agreement",
+ "verdict": "different"
+ },
+ {
+ "left": "employees",
+ "right": "many employees",
+ "verdict": "different"
+ },
+ {
+ "left": "employees",
+ "right": "some employees",
+ "verdict": "different"
+ },
+ {
+ "left": "employees and other investors",
+ "right": "investors",
+ "verdict": "different"
+ },
+ {
+ "left": "English",
+ "right": "Japanese to English",
+ "verdict": "different"
+ },
+ {
+ "left": "Eric Schmidt",
+ "right": "Eric Schmidt",
+ "verdict": "same"
+ },
+ {
+ "left": "ethics",
+ "right": "the ethics of its development",
+ "verdict": "different"
+ },
+ {
+ "left": "European General Data Protection Regulation",
+ "right": "violating the European General Data Protection Regulation",
+ "verdict": "different"
+ },
+ {
+ "left": "European Union",
+ "right": "European Union",
+ "verdict": "same"
+ },
+ {
+ "left": "European Union",
+ "right": "European Union documents",
+ "verdict": "different"
+ },
+ {
+ "left": "European Union",
+ "right": "push by the European Union and its member governments towards digital sovereignty",
+ "verdict": "different"
+ },
+ {
+ "left": "event",
+ "right": "prevent further misuse",
+ "verdict": "different"
+ },
+ {
+ "left": "event",
+ "right": "prevent harmful image generation",
+ "verdict": "different"
+ },
+ {
+ "left": "event",
+ "right": "prevent sexual abuse and violence",
+ "verdict": "different"
+ },
+ {
+ "left": "event",
+ "right": "recent events",
+ "verdict": "different"
+ },
+ {
+ "left": "existing investors",
+ "right": "investors",
+ "verdict": "different"
+ },
+ {
+ "left": "ExploitGym",
+ "right": "ExploitGym scorer",
+ "verdict": "different"
+ },
+ {
+ "left": "Facebook",
+ "right": "Facebook",
+ "verdict": "same"
+ },
+ {
+ "left": "federal judge",
+ "right": "judge",
+ "verdict": "unknown",
+ "why": "the bare \"judge\" has no context"
+ },
+ {
+ "left": "Federal Trade Commission",
+ "right": "US Federal Trade Commission (FTC)",
+ "verdict": "same"
+ },
+ {
+ "left": "Financial Times",
+ "right": "Time",
+ "verdict": "different",
+ "why": "Time vs the Financial Times"
+ },
+ {
+ "left": "Flow",
+ "right": "Stack Overflow",
+ "verdict": "different"
+ },
+ {
+ "left": "focus on safely developing a superintelligence",
+ "right": "superintelligence",
+ "verdict": "different"
+ },
+ {
+ "left": "former employees",
+ "right": "former employees from the agreement",
+ "verdict": "different"
+ },
+ {
+ "left": "France",
+ "right": "France",
+ "verdict": "same"
+ },
+ {
+ "left": "France",
+ "right": "France's National Assembly",
+ "verdict": "different"
+ },
+ {
+ "left": "free users",
+ "right": "users",
+ "verdict": "different"
+ },
+ {
+ "left": "fully autonomous weapons",
+ "right": "legally binding prohibitions on domestic mass surveillance or fully autonomous weapons",
+ "verdict": "different"
+ },
+ {
+ "left": "FunSearch",
+ "right": "MuZero, AlphaStar, AlphaGeometry, FunSearch, AlphaEvolve, AlphaDev, AlphaTensor",
+ "verdict": "different"
+ },
+ {
+ "left": "Gemini",
+ "right": "Gemini 2.0 Flash",
+ "verdict": "different"
+ },
+ {
+ "left": "Gemini",
+ "right": "Gemini 2.5",
+ "verdict": "different"
+ },
+ {
+ "left": "Gemini",
+ "right": "Gemini 3 Pro",
+ "verdict": "different"
+ },
+ {
+ "left": "Gemini",
+ "right": "Gemini Robotics",
+ "verdict": "different"
+ },
+ {
+ "left": "Gemini",
+ "right": "Gemini Robotics 1.5",
+ "verdict": "different"
+ },
+ {
+ "left": "Gemini",
+ "right": "Gemini Robotics-ER",
+ "verdict": "different"
+ },
+ {
+ "left": "Gemini",
+ "right": "Gemini Robotics ER-1.6",
+ "verdict": "different"
+ },
+ {
+ "left": "Gemini Robotics",
+ "right": "Gemini Robotics 1.5",
+ "verdict": "different"
+ },
+ {
+ "left": "Gemini Robotics",
+ "right": "Gemini Robotics-ER",
+ "verdict": "different"
+ },
+ {
+ "left": "Gemini Robotics",
+ "right": "Gemini Robotics ER-1.6",
+ "verdict": "different"
+ },
+ {
+ "left": "Gemma",
+ "right": "Gemma 2",
+ "verdict": "different"
+ },
+ {
+ "left": "Gemma",
+ "right": "Gemma 3",
+ "verdict": "different"
+ },
+ {
+ "left": "Gemma",
+ "right": "PaliGemma 2",
+ "verdict": "different"
+ },
+ {
+ "left": "Gemma",
+ "right": "PaliGemma 2 Mix",
+ "verdict": "different"
+ },
+ {
+ "left": "Gemma",
+ "right": "TxGemma",
+ "verdict": "different"
+ },
+ {
+ "left": "Gemma 2",
+ "right": "PaliGemma 2",
+ "verdict": "different"
+ },
+ {
+ "left": "Gemma 2",
+ "right": "PaliGemma 2 Mix",
+ "verdict": "different"
+ },
+ {
+ "left": "Genie",
+ "right": "Genie 2",
+ "verdict": "different"
+ },
+ {
+ "left": "Genie",
+ "right": "Genie 3",
+ "verdict": "different"
+ },
+ {
+ "left": "geometry problems",
+ "right": "METR",
+ "verdict": "different"
+ },
+ {
+ "left": "GitHub",
+ "right": "GitHub data and Codeforces problems and solutions",
+ "verdict": "different"
+ },
+ {
+ "left": "Go, chess, shogi, and Atari 2600 games",
+ "right": "shogi",
+ "verdict": "different"
+ },
+ {
+ "left": "Google",
+ "right": "Google",
+ "verdict": "same"
+ },
+ {
+ "left": "Google",
+ "right": "Google AI's Google Brain division",
+ "verdict": "different"
+ },
+ {
+ "left": "Google",
+ "right": "Google Cloud",
+ "verdict": "different",
+ "why": "Google Cloud is a division"
+ },
+ {
+ "left": "Google",
+ "right": "Google DeepMind",
+ "verdict": "different"
+ },
+ {
+ "left": "Google",
+ "right": "Google Play's personalized app recommendations",
+ "verdict": "different"
+ },
+ {
+ "left": "Google",
+ "right": "Google Search",
+ "verdict": "different"
+ },
+ {
+ "left": "Google",
+ "right": "Google services or projects",
+ "verdict": "different"
+ },
+ {
+ "left": "Google",
+ "right": "Google TPUs",
+ "verdict": "different"
+ },
+ {
+ "left": "Google",
+ "right": "users to make specific chats discoverable through search engines like Google",
+ "verdict": "different"
+ },
+ {
+ "left": "Google DeepMind",
+ "right": "Google DeepMind",
+ "verdict": "same"
+ },
+ {
+ "left": "GPT-3",
+ "right": "GPT-3.5",
+ "verdict": "different"
+ },
+ {
+ "left": "GPT-3",
+ "right": "training of GPT-3",
+ "verdict": "different"
+ },
+ {
+ "left": "GPT-4",
+ "right": "GPT-4",
+ "verdict": "same",
+ "why": "GPT-4 under two types"
+ },
+ {
+ "left": "GPT-4",
+ "right": "GPT-4.5",
+ "verdict": "different"
+ },
+ {
+ "left": "GPT-4",
+ "right": "GPT-4 answered 44% of prompts asking for copyrighted text",
+ "verdict": "different"
+ },
+ {
+ "left": "GPT-4",
+ "right": "GPT-4o",
+ "verdict": "different"
+ },
+ {
+ "left": "GPT-4",
+ "right": "GPT-4 Turbo platform",
+ "verdict": "different"
+ },
+ {
+ "left": "GPT-4 answered 44% of prompts asking for copyrighted text",
+ "right": "prompts",
+ "verdict": "different"
+ },
+ {
+ "left": "GPT-4 answered 44% of prompts asking for copyrighted text",
+ "right": "prompts asking for copyrighted text",
+ "verdict": "different"
+ },
+ {
+ "left": "GPT-5 mini",
+ "right": "mini",
+ "verdict": "different",
+ "why": "the generic mini tier vs one model"
+ },
+ {
+ "left": "GPT-5 nano",
+ "right": "nano",
+ "verdict": "different"
+ },
+ {
+ "left": "GPT Image",
+ "right": "images generated by GPT Image",
+ "verdict": "different"
+ },
+ {
+ "left": "hallucinations",
+ "right": "hallucinations",
+ "verdict": "same"
+ },
+ {
+ "left": "high",
+ "right": "Higher Education Advisory Board",
+ "verdict": "different"
+ },
+ {
+ "left": "high",
+ "right": "Highly Persistent Internal Model (HPIM)",
+ "verdict": "different"
+ },
+ {
+ "left": "how and when OpenAI reports potentially dangerous user behavior to authorities",
+ "right": "OpenAI",
+ "verdict": "different"
+ },
+ {
+ "left": "Hugging Face",
+ "right": "Hugging Face",
+ "verdict": "same"
+ },
+ {
+ "left": "ICO investigation",
+ "right": "investigation",
+ "verdict": "different"
+ },
+ {
+ "left": "increased use of ChatGPT for China-origin influence operations",
+ "right": "influence operations",
+ "verdict": "different"
+ },
+ {
+ "left": "Inflection AI",
+ "right": "Inflection AI, Inc.",
+ "verdict": "same",
+ "why": "Inflection AI is Inflection AI, Inc."
+ },
+ {
+ "left": "information to the company's senior management and board, particularly regarding safety issues",
+ "right": "safety",
+ "verdict": "different"
+ },
+ {
+ "left": "initial public offering",
+ "right": "initial public offering (IPO)",
+ "verdict": "same",
+ "why": "OpenAI's 2026 IPO"
+ },
+ {
+ "left": "Instacart",
+ "right": "Instacart",
+ "verdict": "same"
+ },
+ {
+ "left": "International Mathematical Olympiad",
+ "right": "International Mathematical Olympiad",
+ "verdict": "same"
+ },
+ {
+ "left": "interview",
+ "right": "Suchir Balaji interview",
+ "verdict": "same",
+ "why": "the NYT interview with Suchir Balaji"
+ },
+ {
+ "left": "investigation",
+ "right": "investigation into Suchir Balaji's death",
+ "verdict": "different"
+ },
+ {
+ "left": "investigation into Suchir Balaji's death",
+ "right": "Suchir Balaji",
+ "verdict": "different"
+ },
+ {
+ "left": "investors",
+ "right": "reimburse investors",
+ "verdict": "different"
+ },
+ {
+ "left": "investors",
+ "right": "tender offer for early-stage investors",
+ "verdict": "different"
+ },
+ {
+ "left": "Jakub Pachocki",
+ "right": "Pachocki",
+ "verdict": "same",
+ "why": "Pachocki is Jakub Pachocki"
+ },
+ {
+ "left": "Japanese",
+ "right": "Japanese to English",
+ "verdict": "different"
+ },
+ {
+ "left": "Jason Kwon",
+ "right": "Kwon",
+ "verdict": "same",
+ "why": "Kwon is Jason Kwon"
+ },
+ {
+ "left": "John Schulman",
+ "right": "John Schulman",
+ "verdict": "same"
+ },
+ {
+ "left": "judge",
+ "right": "U.S. District Judge Araceli Martínez-Olguín",
+ "verdict": "unknown",
+ "why": "the bare \"judge\" has no context"
+ },
+ {
+ "left": "Labs",
+ "right": "Meta Superintelligence Labs",
+ "verdict": "different"
+ },
+ {
+ "left": "large language model",
+ "right": "large language models",
+ "verdict": "same",
+ "why": "singular and plural of the same concept"
+ },
+ {
+ "left": "lawsuit",
+ "right": "lawsuit against OpenAI",
+ "verdict": "different"
+ },
+ {
+ "left": "lawsuit",
+ "right": "lawsuit against OpenAI and Altman",
+ "verdict": "different"
+ },
+ {
+ "left": "lawsuit",
+ "right": "lawsuit for copyright infringement",
+ "verdict": "different"
+ },
+ {
+ "left": "lawsuit against OpenAI",
+ "right": "OpenAI",
+ "verdict": "different"
+ },
+ {
+ "left": "lawsuit against OpenAI and Altman",
+ "right": "OpenAI",
+ "verdict": "different"
+ },
+ {
+ "left": "leaked letter from the National Data Guardian",
+ "right": "National Data Guardian",
+ "verdict": "different"
+ },
+ {
+ "left": "less training time",
+ "right": "Time",
+ "verdict": "different"
+ },
+ {
+ "left": "LLMs",
+ "right": "LLMs",
+ "verdict": "same"
+ },
+ {
+ "left": "LLMs",
+ "right": "LLMs can sometimes plan ahead",
+ "verdict": "different"
+ },
+ {
+ "left": "LLMs",
+ "right": "LLMs performance on copyrighted text",
+ "verdict": "different"
+ },
+ {
+ "left": "LLMs can sometimes plan ahead",
+ "right": "Time",
+ "verdict": "different"
+ },
+ {
+ "left": "London",
+ "right": "three London hospitals",
+ "verdict": "different"
+ },
+ {
+ "left": "lying",
+ "right": "lying and being manipulative in different situations",
+ "verdict": "different",
+ "why": "two allegations from different documents"
+ },
+ {
+ "left": "Lyria",
+ "right": "Lyria 3",
+ "verdict": "different"
+ },
+ {
+ "left": "Lyria",
+ "right": "Lyria 3 Pro",
+ "verdict": "different"
+ },
+ {
+ "left": "Lyria 3",
+ "right": "Lyria 3 Pro",
+ "verdict": "different"
+ },
+ {
+ "left": "March 2023",
+ "right": "March 2023 open letter",
+ "verdict": "different"
+ },
+ {
+ "left": "matrix multiplication",
+ "right": "matrix multiplication algorithm",
+ "verdict": "different",
+ "why": "the problem vs an algorithm for it"
+ },
+ {
+ "left": "matrix multiplication",
+ "right": "Multi",
+ "verdict": "different"
+ },
+ {
+ "left": "matrix multiplication algorithm",
+ "right": "Multi",
+ "verdict": "different"
+ },
+ {
+ "left": "Meta",
+ "right": "Meta",
+ "verdict": "same"
+ },
+ {
+ "left": "Meta",
+ "right": "Meta Platforms",
+ "verdict": "same",
+ "why": "Meta is Meta Platforms"
+ },
+ {
+ "left": "Meta",
+ "right": "Meta Superintelligence Labs",
+ "verdict": "different"
+ },
+ {
+ "left": "Meta Platforms",
+ "right": "Meta Platforms",
+ "verdict": "same"
+ },
+ {
+ "left": "METR",
+ "right": "MuZero, AlphaStar, AlphaGeometry, FunSearch, AlphaEvolve, AlphaDev, AlphaTensor",
+ "verdict": "different"
+ },
+ {
+ "left": "Microsoft",
+ "right": "Microsoft",
+ "verdict": "same"
+ },
+ {
+ "left": "Microsoft",
+ "right": "Microsoft AI",
+ "verdict": "different",
+ "why": "a division"
+ },
+ {
+ "left": "Microsoft",
+ "right": "Microsoft Azure",
+ "verdict": "different"
+ },
+ {
+ "left": "Microsoft",
+ "right": "Microsoft-Inflection deal",
+ "verdict": "different"
+ },
+ {
+ "left": "Microsoft",
+ "right": "Microsoft's relationship with OpenAI",
+ "verdict": "different"
+ },
+ {
+ "left": "Microsoft Azure",
+ "right": "Microsoft Azure",
+ "verdict": "same"
+ },
+ {
+ "left": "Microsoft's relationship with OpenAI",
+ "right": "OpenAI",
+ "verdict": "different"
+ },
+ {
+ "left": "Mira Murati",
+ "right": "Murati",
+ "verdict": "same",
+ "why": "Murati is Mira Murati"
+ },
+ {
+ "left": "MuZero",
+ "right": "MuZero",
+ "verdict": "same"
+ },
+ {
+ "left": "MuZero",
+ "right": "MuZero, AlphaStar, AlphaGeometry, FunSearch, AlphaEvolve, AlphaDev, AlphaTensor",
+ "verdict": "different"
+ },
+ {
+ "left": "Nadella",
+ "right": "Satya Nadella",
+ "verdict": "same",
+ "why": "Nadella is Satya Nadella"
+ },
+ {
+ "left": "New York Times",
+ "right": "The New York Times",
+ "verdict": "same",
+ "why": "New York Times with and without the article"
+ },
+ {
+ "left": "New York Times",
+ "right": "Time",
+ "verdict": "different"
+ },
+ {
+ "left": "non-disparagement agreement",
+ "right": "non-disparagement agreements",
+ "verdict": "same",
+ "why": "singular and plural"
+ },
+ {
+ "left": "OpenAI",
+ "right": "OpenAI",
+ "verdict": "same",
+ "why": "OpenAI under two types"
+ },
+ {
+ "left": "OpenAI",
+ "right": "OpenAI employees",
+ "verdict": "different"
+ },
+ {
+ "left": "OpenAI",
+ "right": "OpenAI engineers",
+ "verdict": "different"
+ },
+ {
+ "left": "OpenAI",
+ "right": "OpenAI Foundation",
+ "verdict": "different",
+ "why": "the nonprofit parent is its own legal entity"
+ },
+ {
+ "left": "OpenAI",
+ "right": "OpenAI Global, LLC",
+ "verdict": "different",
+ "why": "a subsidiary"
+ },
+ {
+ "left": "OpenAI",
+ "right": "OpenAI GP LLC",
+ "verdict": "different"
+ },
+ {
+ "left": "OpenAI",
+ "right": "OpenAI Group PBC",
+ "verdict": "different",
+ "why": "the for-profit subsidiary"
+ },
+ {
+ "left": "OpenAI",
+ "right": "OpenAI, Inc.",
+ "verdict": "different",
+ "why": "the nonprofit legal entity vs the organization as a whole"
+ },
+ {
+ "left": "OpenAI",
+ "right": "OpenAI investors",
+ "verdict": "different"
+ },
+ {
+ "left": "OpenAI",
+ "right": "OpenAI Ireland Ltd",
+ "verdict": "different",
+ "why": "a subsidiary"
+ },
+ {
+ "left": "OpenAI",
+ "right": "OpenAI LP",
+ "verdict": "different"
+ },
+ {
+ "left": "OpenAI",
+ "right": "OpenAI OpCo, LLC",
+ "verdict": "different",
+ "why": "a legal entity of OpenAI's, still its own record"
+ },
+ {
+ "left": "OpenAI",
+ "right": "OpenAI representative",
+ "verdict": "different"
+ },
+ {
+ "left": "OpenAI",
+ "right": "OpenAI's board of directors",
+ "verdict": "different"
+ },
+ {
+ "left": "OpenAI",
+ "right": "OpenAI's safety team",
+ "verdict": "different"
+ },
+ {
+ "left": "OpenAI",
+ "right": "over one hundred companies using OpenAI",
+ "verdict": "different"
+ },
+ {
+ "left": "OpenAI",
+ "right": "similarities between DeepSeek's model distillation approach and OpenAI's methodology",
+ "verdict": "different"
+ },
+ {
+ "left": "OpenAI",
+ "right": "two executives in OpenAI",
+ "verdict": "different"
+ },
+ {
+ "left": "OpenAI Global, LLC",
+ "right": "OpenAI Global, LLC",
+ "verdict": "same"
+ },
+ {
+ "left": "OpenAI, Inc.",
+ "right": "OpenAI, Inc.",
+ "verdict": "same"
+ },
+ {
+ "left": "Oracle",
+ "right": "Oracle",
+ "verdict": "same"
+ },
+ {
+ "left": "PaliGemma 2",
+ "right": "PaliGemma 2 Mix",
+ "verdict": "different"
+ },
+ {
+ "left": "Palo Alto, California",
+ "right": "Palo Alto, California",
+ "verdict": "same"
+ },
+ {
+ "left": "partnership",
+ "right": "Partnership on Artificial Intelligence to Benefit People and Society",
+ "verdict": "different"
+ },
+ {
+ "left": "Patronus AI",
+ "right": "Patronus AI",
+ "verdict": "same"
+ },
+ {
+ "left": "Pentagon",
+ "right": "The Pentagon",
+ "verdict": "same",
+ "why": "the Pentagon with and without the article"
+ },
+ {
+ "left": "policy-gradient reinforcement learning",
+ "right": "reinforcement learning",
+ "verdict": "different"
+ },
+ {
+ "left": "policy network",
+ "right": "predict winners of games played by the policy network against itself",
+ "verdict": "different"
+ },
+ {
+ "left": "Pong",
+ "right": "Pong",
+ "verdict": "same"
+ },
+ {
+ "left": "potential violations of YouTube's terms of service",
+ "right": "YouTube",
+ "verdict": "different"
+ },
+ {
+ "left": "prompts",
+ "right": "prompts asking for copyrighted text",
+ "verdict": "different"
+ },
+ {
+ "left": "prompts",
+ "right": "rejects violating prompts",
+ "verdict": "different"
+ },
+ {
+ "left": "prompts",
+ "right": "writing prompts",
+ "verdict": "different"
+ },
+ {
+ "left": "protein folding",
+ "right": "state of the art records on benchmark tests for protein folding prediction",
+ "verdict": "different"
+ },
+ {
+ "left": "psychological abuse from Sam Altman",
+ "right": "Sam Altman",
+ "verdict": "different"
+ },
+ {
+ "left": "public benefit corporation",
+ "right": "public benefit corporation (PBC)",
+ "verdict": "same"
+ },
+ {
+ "left": "Quake",
+ "right": "Quake III Arena",
+ "verdict": "different",
+ "why": "the franchise vs one title"
+ },
+ {
+ "left": "rationale for firing Sam Altman",
+ "right": "Sam Altman",
+ "verdict": "different"
+ },
+ {
+ "left": "Reddit",
+ "right": "Reddit",
+ "verdict": "same"
+ },
+ {
+ "left": "reinforcement learning",
+ "right": "reinforcement learning from human feedback (RLHF)",
+ "verdict": "different",
+ "why": "RLHF is a specific technique"
+ },
+ {
+ "left": "reinforcement learning",
+ "right": "reinforcement learning (RL)",
+ "verdict": "same"
+ },
+ {
+ "left": "reward model",
+ "right": "reward models",
+ "verdict": "same",
+ "why": "singular and plural"
+ },
+ {
+ "left": "Royal Free hospital",
+ "right": "Royal Free Hospital",
+ "verdict": "same"
+ },
+ {
+ "left": "Royal Free hospital",
+ "right": "Royal Free hospital failed to comply with the Data Protection Act",
+ "verdict": "different"
+ },
+ {
+ "left": "Royal Free London NHS Foundation Trust",
+ "right": "Royal Free London NHS Foundation Trust",
+ "verdict": "same"
+ },
+ {
+ "left": "safety",
+ "right": "technical safety research",
+ "verdict": "different"
+ },
+ {
+ "left": "Sama",
+ "right": "Sama",
+ "verdict": "same"
+ },
+ {
+ "left": "Sam Altman",
+ "right": "Sam Altman's efforts",
+ "verdict": "different"
+ },
+ {
+ "left": "Sam Altman",
+ "right": "they didn't support Sam Altman",
+ "verdict": "different"
+ },
+ {
+ "left": "San Francisco, California",
+ "right": "San Francisco, California",
+ "verdict": "same"
+ },
+ {
+ "left": "Science",
+ "right": "Science",
+ "verdict": "different",
+ "why": "the journal vs Anthropic's service"
+ },
+ {
+ "left": "Securities and Exchange Commission",
+ "right": "US Securities and Exchange Commission (SEC)",
+ "verdict": "same"
+ },
+ {
+ "left": "Sequoia Capital",
+ "right": "Sequoia Capital",
+ "verdict": "same"
+ },
+ {
+ "left": "share sale led by Thrive Capital",
+ "right": "Thrive",
+ "verdict": "different"
+ },
+ {
+ "left": "share sale led by Thrive Capital",
+ "right": "Thrive Capital",
+ "verdict": "different"
+ },
+ {
+ "left": "Sidor",
+ "right": "Szymon Sidor",
+ "verdict": "same",
+ "why": "Sidor is Szymon Sidor"
+ },
+ {
+ "left": "SoftBank",
+ "right": "SoftBank",
+ "verdict": "same"
+ },
+ {
+ "left": "SoftBank",
+ "right": "SoftBank Group-owned SB Energy",
+ "verdict": "different"
+ },
+ {
+ "left": "Software Applications Incorporated",
+ "right": "software_application",
+ "verdict": "different"
+ },
+ {
+ "left": "Sony Corporation",
+ "right": "Sony Corporation",
+ "verdict": "same"
+ },
+ {
+ "left": "Sora",
+ "right": "Sora app",
+ "verdict": "same",
+ "why": "the Sora app is the Sora platform"
+ },
+ {
+ "left": "Sora",
+ "right": "Sora model API access",
+ "verdict": "different"
+ },
+ {
+ "left": "Sora model",
+ "right": "Sora model API access",
+ "verdict": "different"
+ },
+ {
+ "left": "Space Invaders",
+ "right": "Space Invaders",
+ "verdict": "same"
+ },
+ {
+ "left": "StarCraft",
+ "right": "StarCraft II",
+ "verdict": "different",
+ "why": "StarCraft II is one title of the series"
+ },
+ {
+ "left": "strategic thinking",
+ "right": "Thinking",
+ "verdict": "different"
+ },
+ {
+ "left": "Suchir Balaji",
+ "right": "Suchir Balaji interview",
+ "verdict": "different"
+ },
+ {
+ "left": "summarization",
+ "right": "translation and summarization tasks",
+ "verdict": "different"
+ },
+ {
+ "left": "supervised learning",
+ "right": "supervised learning",
+ "verdict": "same"
+ },
+ {
+ "left": "Suzanne Adams",
+ "right": "Suzanne Adams's estate",
+ "verdict": "different"
+ },
+ {
+ "left": "Switzerland",
+ "right": "Switzerland",
+ "verdict": "same"
+ },
+ {
+ "left": "The Atlantic",
+ "right": "The Atlantic",
+ "verdict": "same"
+ },
+ {
+ "left": "The New York Times",
+ "right": "The New York Times",
+ "verdict": "same"
+ },
+ {
+ "left": "The New York Times",
+ "right": "Time",
+ "verdict": "different"
+ },
+ {
+ "left": "The Wall Street Journal",
+ "right": "The Wall Street Journal",
+ "verdict": "same"
+ },
+ {
+ "left": "The Wall Street Journal",
+ "right": "Wall Street",
+ "verdict": "different"
+ },
+ {
+ "left": "Thrive",
+ "right": "Thrive Capital",
+ "verdict": "same",
+ "why": "Thrive is Thrive Capital"
+ },
+ {
+ "left": "Thrive",
+ "right": "Thrive Capital's tender offer",
+ "verdict": "different"
+ },
+ {
+ "left": "Thrive Capital",
+ "right": "Thrive Capital's tender offer",
+ "verdict": "different"
+ },
+ {
+ "left": "Time",
+ "right": "Time",
+ "verdict": "unknown",
+ "why": "the bare \"Time\" has no facts"
+ },
+ {
+ "left": "Time",
+ "right": "Time magazine",
+ "verdict": "same",
+ "why": "Time is Time magazine"
+ },
+ {
+ "left": "training data",
+ "right": "training data from published books by destructive book scanning",
+ "verdict": "different"
+ },
+ {
+ "left": "Trump administration",
+ "right": "Trump administration",
+ "verdict": "same"
+ },
+ {
+ "left": "UK",
+ "right": "UK",
+ "verdict": "same"
+ },
+ {
+ "left": "United States",
+ "right": "United States",
+ "verdict": "same"
+ },
+ {
+ "left": "United States",
+ "right": "United States Congress",
+ "verdict": "different"
+ },
+ {
+ "left": "United States",
+ "right": "United States Department of Defense",
+ "verdict": "different"
+ },
+ {
+ "left": "United States",
+ "right": "United States Department of Defense (DoD)",
+ "verdict": "different"
+ },
+ {
+ "left": "University College London",
+ "right": "University College London, Department of Computer Science",
+ "verdict": "different",
+ "why": "a department vs the university"
+ },
+ {
+ "left": "users",
+ "right": "users to make specific chats discoverable through search engines like Google",
+ "verdict": "different"
+ },
+ {
+ "left": "WaveNet",
+ "right": "WaveNetEQ",
+ "verdict": "different"
+ },
+ {
+ "left": "Whisper",
+ "right": "Whisper",
+ "verdict": "same"
+ },
+ {
+ "left": "xAI",
+ "right": "xAI",
+ "verdict": "same"
+ }
+ ]
+}