World's first open-source enterprise world model.
Rust agent-memorybitemporalgraphragknowledge-baseknowledge-graphllmontologypgvectorpostgresqlragrustself-hostedsemantic-searchtemporal-knowledge-graphworld-model
A push that names its external_id keeps that id's document (#1268)
Szabo Bogdan4c5221ad6bdc
diff --git a/crates/utopia-server/src/api/sources_routes.rs b/crates/utopia-server/src/api/sources_routes.rsindex 2c786394..6e8cec73 100644--- a/crates/utopia-server/src/api/sources_routes.rs+++ b/crates/utopia-server/src/api/sources_routes.rs@@ -404,6 +404,16 @@ pub struct IngestBody {pub deleted: bool,}+/// 推送的回执:动作,重复时再加上占着这份内容的文档+fn action_reply(action: crate::ingest_sources::IngestAction) -> serde_json::Value {+ match action {+ crate::ingest_sources::IngestAction::Duplicate(id) => {+ json!({ "action": action_str(action), "duplicate_of": id })+ }+ _ => json!({ "action": action_str(action) }),+ }+}+fn action_str(action: crate::ingest_sources::IngestAction) -> &'static str {match action {crate::ingest_sources::IngestAction::Created => "created",@@ -411,6 +421,7 @@ fn action_str(action: crate::ingest_sources::IngestAction) -> &'static str {crate::ingest_sources::IngestAction::Moved => "moved",crate::ingest_sources::IngestAction::Unchanged => "unchanged",crate::ingest_sources::IngestAction::Tombstoned => "marked_missing",+ crate::ingest_sources::IngestAction::Duplicate(_) => "duplicate",}}@@ -444,7 +455,7 @@ pub async fn ingest().await.map_err(utopia_core::AppError::Other)?;- Ok(Json(json!({ "action": action_str(action) })))+ Ok(Json(action_reply(action)))}/// 推送失败的两种性质。**调用方发错了**和**我们这边没接住**得分开:前者记进@@ -474,13 +485,12 @@ async fn handle_push() -> Result<crate::ingest_sources::IngestAction, PushError> {let body: IngestBody = serde_json::from_slice(bytes).map_err(|e| PushError::Rejected(format!("Invalid JSON payload: {e}")))?;- let identity = body+ let declared = body.external_id.as_deref().map(str::trim)- .filter(|s| !s.is_empty())- .unwrap_or_else(|| body.filename.trim())- .to_string();+ .filter(|s| !s.is_empty());+ let identity = declared.unwrap_or_else(|| body.filename.trim()).to_string();if identity.is_empty() {return Err(PushError::Rejected("external_id or filename is required".into(),@@ -501,22 +511,31 @@ async fn handle_push("filename and content are required".into(),));}- let action = crate::ingest_sources::ingest_item(+ // 带了 external_id 身份就是它:不走「同内容 = 移动」,不把另一个 ID 的文档改成它+ let action = crate::ingest_sources::ingest_item_with_outcome(state,source.kb_id,source.id,&key,+ if declared.is_some() {+ crate::ingest_sources::Identity::Declared+ } else {+ crate::ingest_sources::Identity::Path+ },body.filename.trim(),"text/plain",body.content.as_bytes(),body.doc_time,).await- .map_err(|e| PushError::Failed(e.to_string()))?;- // 失而复得:曾被墓碑标记的身份再次正常推送,摘掉 missing 标记- utopia_store::documents::clear_missing_keys(&state.pool, source.id, &[key])- .await- .map_err(|e| PushError::Failed(e.to_string()))?;+ .map_err(|e| PushError::Failed(e.to_string()))?+ .action;+ // 失而复得:曾被墓碑标记的身份再次正常推送,摘掉 missing 标记;重复的推送什么也没落,不摘+ if !matches!(action, crate::ingest_sources::IngestAction::Duplicate(_)) {+ utopia_store::documents::clear_missing_keys(&state.pool, source.id, &[key])+ .await+ .map_err(|e| PushError::Failed(e.to_string()))?;+ }Ok(action)}@@ -554,14 +573,15 @@ pub async fn push(crate::ingest_sources::IngestAction::Updated| crate::ingest_sources::IngestAction::Moved => (0, 1),crate::ingest_sources::IngestAction::Unchanged- | crate::ingest_sources::IngestAction::Tombstoned => (0, 0),+ | crate::ingest_sources::IngestAction::Tombstoned+ | crate::ingest_sources::IngestAction::Duplicate(_) => (0, 0),};utopia_store::sources::finish_run(&state.pool, run, source.id, None, created, updated).await?;// 来源行同步反映"上一次推送":操作条/左栏状态点直接可用utopia_store::sources::finish_sync(&state.pool, source.id, None, created).await?;state.emit_source(source.kb_id);- Ok(Json(json!({ "action": action_str(action) })))+ Ok(Json(action_reply(action)))}Err(err) => {let msg = err.message().to_string();@@ -612,13 +632,14 @@ pub async fn push_statements(crate::ingest_sources::IngestAction::Updated| crate::ingest_sources::IngestAction::Moved => (0, 1),crate::ingest_sources::IngestAction::Unchanged- | crate::ingest_sources::IngestAction::Tombstoned => (0, 0),+ | crate::ingest_sources::IngestAction::Tombstoned+ | crate::ingest_sources::IngestAction::Duplicate(_) => (0, 0),};utopia_store::sources::finish_run(&state.pool, run, source.id, None, created, updated).await?;utopia_store::sources::finish_sync(&state.pool, source.id, None, created).await?;state.emit_source(source.kb_id);- Ok(Json(json!({ "action": action_str(action) })))+ Ok(Json(action_reply(action)))}Err(err) => {let msg = err.message().to_string();@@ -797,21 +818,25 @@ async fn handle_statements_push(return Ok(crate::ingest_sources::IngestAction::Tombstoned);};let filename = format!("{identity}.json");- let action = crate::ingest_sources::ingest_item(+ let action = crate::ingest_sources::ingest_item_with_outcome(state,source.kb_id,source.id,&key,+ crate::ingest_sources::Identity::Declared,&filename,"application/json",content.as_bytes(),body.doc_time,).await- .map_err(|e| PushError::Failed(e.to_string()))?;- utopia_store::documents::clear_missing_keys(&state.pool, source.id, &[key])- .await- .map_err(|e| PushError::Failed(e.to_string()))?;+ .map_err(|e| PushError::Failed(e.to_string()))?+ .action;+ if !matches!(action, crate::ingest_sources::IngestAction::Duplicate(_)) {+ utopia_store::documents::clear_missing_keys(&state.pool, source.id, &[key])+ .await+ .map_err(|e| PushError::Failed(e.to_string()))?;+ }Ok(action)}diff --git a/crates/utopia-server/src/api/sources_statements_tests.rs b/crates/utopia-server/src/api/sources_statements_tests.rsindex 47350f25..9d97827a 100644--- a/crates/utopia-server/src/api/sources_statements_tests.rs+++ b/crates/utopia-server/src/api/sources_statements_tests.rs@@ -183,8 +183,24 @@ impl Fixture {token: Option<&str>,body: Vec<u8>,) -> anyhow::Result<(StatusCode, Value)> {- let mut request = Request::post(format!("/api/v1/sources/{source}/statements"))- .header("Content-Type", "application/json");+ self.send(&format!("/api/v1/sources/{source}/statements"), token, body)+ .await+ }++ /// 推给 api 来源(`/ingest`),用它自己的密钥+ async fn push_api(&self, body: &Value) -> anyhow::Result<(StatusCode, Value)> {+ let path = format!("/api/v1/sources/{}/ingest", self.api_source);+ self.send(&path, Some(&self.api_token), body.to_string().into_bytes())+ .await+ }++ async fn send(+ &self,+ path: &str,+ token: Option<&str>,+ body: Vec<u8>,+ ) -> anyhow::Result<(StatusCode, Value)> {+ let mut request = Request::post(path).header("Content-Type", "application/json");if let Some(token) = token {request = request.header("Authorization", format!("Bearer {token}"));}@@ -577,3 +593,146 @@ async fn the_same_payload_under_a_new_identity_is_a_second_observation() -> anyh}f.cleanup().await}++/// api 推送带了 `external_id`,身份就是这个 ID:另一个 ID 推来同样的字节不是把第一篇+/// 改名过去(从前走「同内容出现在新路径 = 移动」,第一个 ID 就此消失),而是如实回一句+/// 重复了谁,两个 ID 里已有的那篇原样留着+#[tokio::test]+async fn a_new_id_with_the_same_content_does_not_take_the_first_ids_document() -> anyhow::Result<()>+{+ let Some(f) = Fixture::new().await? else {+ return Ok(());+ };+ let record = |id: &str| json!({ "filename": "record.txt", "content": "colour = blue", "external_id": id });+ let (status, body) = f.push_api(&record("a")).await?;+ assert_eq!(+ (status, body["action"].as_str()),+ (StatusCode::OK, Some("created")),+ "{body}"+ );+ let first = documents::find_by_external_key(&f.pool, f.api_source, "api:a")+ .await?+ .expect("a is stored");++ let (status, body) = f.push_api(&record("b")).await?;+ assert_eq!(status, StatusCode::OK, "{body}");+ assert_eq!(+ body,+ json!({ "action": "duplicate", "duplicate_of": first.id })+ );+ let kept = documents::find_by_external_key(&f.pool, f.api_source, "api:a").await?;+ assert_eq!(kept.map(|d| d.id), Some(first.id), "a keeps its document");+ f.cleanup().await+}++/// 同样的字节已经在库里别处(这里是一篇普通上传):从前回 `unchanged`、什么也没建,+/// 调用方以为它的 ID 已经落库+#[tokio::test]+async fn content_already_in_the_base_is_reported_as_a_duplicate() -> anyhow::Result<()> {+ let Some(f) = Fixture::new().await? else {+ return Ok(());+ };+ crate::ingest_sources::ingest_upload(+ &f.state,+ f.kb,+ "upload.txt",+ "text/plain",+ b"colour = green",+ None,+ )+ .await?;+ let upload: Uuid =+ sqlx::query_scalar("SELECT id FROM documents WHERE kb_id = $1 AND source_id IS NULL")+ .bind(f.kb)+ .fetch_one(&f.pool)+ .await?;++ let push = json!({ "filename": "record.txt", "content": "colour = green", "external_id": "c" });+ let (status, body) = f.push_api(&push).await?;+ assert_eq!(status, StatusCode::OK, "{body}");+ assert_eq!(+ body,+ json!({ "action": "duplicate", "duplicate_of": upload })+ );+ f.cleanup().await+}++/// 已有的 ID 改成另一篇文档的内容:从前撞上 `documents_kb_sha_idx`,推送以一条 SQL 错误+/// 失败、来源被标成同步失败;现在回重复了谁,这个 ID 的文档保持原来的内容+#[tokio::test]+async fn an_update_to_another_documents_content_is_a_duplicate_not_a_failure() -> anyhow::Result<()>+{+ let Some(f) = Fixture::new().await? else {+ return Ok(());+ };+ let push = |id: &str, content: &str| json!({ "filename": "record.txt", "content": content, "external_id": id });+ f.push_api(&push("a", "colour = blue")).await?;+ f.push_api(&push("b", "colour = red")).await?;+ let a = documents::find_by_external_key(&f.pool, f.api_source, "api:a")+ .await?+ .expect("a is stored");++ let (status, body) = f.push_api(&push("b", "colour = blue")).await?;+ assert_eq!(status, StatusCode::OK, "{body}");+ assert_eq!(body, json!({ "action": "duplicate", "duplicate_of": a.id }));+ let b = documents::find_by_external_key(&f.pool, f.api_source, "api:b")+ .await?+ .expect("b is still stored");+ assert_ne!(b.sha256, a.sha256, "b keeps its own content");+ f.cleanup().await+}++/// 拿 ID 当路径用的调用方这样改名:先墓碑旧 ID,再用同样的字节推新 ID。新 ID 接手那篇+/// 被墓碑标记的文档(同一篇、历史还在),旧 ID 不再留着一篇「不在来源中」的孤本+#[tokio::test]+async fn a_new_id_with_a_tombstoned_ids_content_takes_over_its_document() -> anyhow::Result<()> {+ let Some(f) = Fixture::new().await? else {+ return Ok(());+ };+ let record = |id: &str| json!({ "filename": "record.txt", "content": "colour = blue", "external_id": id });+ f.push_api(&record("a")).await?;+ let first = documents::find_by_external_key(&f.pool, f.api_source, "api:a")+ .await?+ .expect("a is stored");+ f.push_api(&json!({ "filename": "record.txt", "external_id": "a", "deleted": true }))+ .await?;++ let (status, body) = f.push_api(&record("b")).await?;+ assert_eq!(status, StatusCode::OK, "{body}");+ assert_eq!(body, json!({ "action": "moved" }));+ let b = documents::find_by_external_key(&f.pool, f.api_source, "api:b")+ .await?+ .expect("b is stored");+ assert_eq!((b.id, b.missing_since), (first.id, None));+ let a = documents::find_by_external_key(&f.pool, f.api_source, "api:a").await?;+ assert!(a.is_none(), "a is gone");+ f.cleanup().await+}++/// 回收站里的 ID 推来另一篇文档的内容:先答重复,回收站里那篇不被恢复+#[tokio::test]+async fn a_trashed_id_pushed_with_another_documents_content_stays_in_the_trash(+) -> anyhow::Result<()> {+ let Some(f) = Fixture::new().await? else {+ return Ok(());+ };+ let push = |id: &str, content: &str| json!({ "filename": "record.txt", "content": content, "external_id": id });+ f.push_api(&push("a", "colour = blue")).await?;+ f.push_api(&push("b", "colour = red")).await?;+ let a = documents::find_by_external_key(&f.pool, f.api_source, "api:a")+ .await?+ .expect("a is stored");+ let b = documents::find_by_external_key(&f.pool, f.api_source, "api:b")+ .await?+ .expect("b is stored");+ documents::delete(&f.pool, f.kb, a.id, None).await?;++ let (status, body) = f.push_api(&push("a", "colour = red")).await?;+ assert_eq!(status, StatusCode::OK, "{body}");+ assert_eq!(body, json!({ "action": "duplicate", "duplicate_of": b.id }));+ let a = documents::find_by_external_key(&f.pool, f.api_source, "api:a")+ .await?+ .expect("a is still stored");+ assert!(a.deleted_at.is_some(), "a stays in the trash");+ f.cleanup().await+}diff --git a/crates/utopia-server/src/ingest_sources.rs b/crates/utopia-server/src/ingest_sources.rsindex 62f81c69..17469be0 100644--- a/crates/utopia-server/src/ingest_sources.rs+++ b/crates/utopia-server/src/ingest_sources.rs@@ -153,6 +153,17 @@ pub enum IngestAction {Unchanged,/// 墓碑:标记"不在来源中"(不删除文档,删不删由用户在 UI 决定)Tombstoned,+ /// 库里已有一篇同内容的文档(`documents_kb_sha_idx`):这次什么也没建、没改,指向那一篇+ Duplicate(Uuid),+}++/// 身份从哪来。`Path`:来源按位置认文档,同内容出现在新位置就是移动/改名,历史跟着走;+/// `Declared`:调用方亲口给了 ID,身份就是它,绝不把另一个活着的 ID 的文档改成它;+/// 只有已被墓碑标记的那篇可以接手(改名)。+#[derive(Debug, Clone, Copy, PartialEq)]+pub enum Identity {+ Path,+ Declared,}#[derive(Debug, Clone, Copy)]@@ -231,6 +242,7 @@ pub async fn ingest_item(kb_id,source_id,external_key,+ Identity::Path,filename,mime,bytes,@@ -246,6 +258,7 @@ pub async fn ingest_item_with_outcome(kb_id: Uuid,source_id: Uuid,external_key: &str,+ identity: Identity,filename: &str,mime: &str,bytes: &[u8],@@ -281,6 +294,16 @@ pub async fn ingest_item_with_outcome(}if let Some(doc) = existing {+ // 新内容已被库里另一篇占着:先答重复,免得下面恢复回收站、重建索引之后才发现+ if doc.sha256 != sha256 {+ if let Some(holder) =+ utopia_store::documents::find_by_kb_sha(&state.pool, kb_id, &sha256).await?+ {+ return Ok(IngestOutcome {+ action: IngestAction::Duplicate(holder.id),+ });+ }+ }// Ordinary source reappearance retains upstream restore semantics,// including search and derivation repair, before comparing content.let revived = doc.deleted_at.is_some();@@ -312,7 +335,7 @@ pub async fn ingest_item_with_outcome(}// 变更:原地替换,旧版本入 document_versions,并与处理任务一起提交write_blob(state, &sha256, bytes).await?;- utopia_store::documents::replace_content_and_enqueue_processing(+ match utopia_store::documents::replace_content_and_enqueue_processing(&state.pool,doc.id,filename,@@ -321,17 +344,26 @@ pub async fn ingest_item_with_outcome(&sha256,doc_time,)- .await?;+ .await+ {+ Ok(()) => {}+ Err(utopia_core::AppError::Conflict(_)) => {+ return duplicate_of(state, kb_id, &sha256).await+ }+ Err(e) => return Err(e.into()),+ }state.emit_document(kb_id, doc.id);return Ok(IngestOutcome {action: IngestAction::Updated,});}- // 同内容出现在新路径:识别为移动/改名,不重跑管道- if let Some(doc) =- utopia_store::documents::find_by_source_sha(&state.pool, source_id, &sha256).await?- {+ // 同内容出现在新路径:识别为移动/改名,不重跑管道。声明的 ID 只接手已被墓碑标记的那篇+ // (先墓碑旧 ID、再推新 ID = 改名),活着的绝不改走+ let moved = utopia_store::documents::find_by_source_sha(&state.pool, source_id, &sha256)+ .await?+ .filter(|doc| identity == Identity::Path || doc.missing_since.is_some());+ if let Some(doc) = moved {utopia_store::documents::update_location(&state.pool, doc.id, filename, external_key).await?;state.emit_document(kb_id, doc.id);@@ -360,14 +392,23 @@ pub async fn ingest_item_with_outcome(action: IngestAction::Created,})}- // KB 内已有同内容:不伪造新的身份,保留幂等结果- Err(utopia_core::AppError::Conflict(_)) => Ok(IngestOutcome {- action: IngestAction::Unchanged,- }),+ // KB 内已有同内容:不伪造新的身份,如实说重复了谁+ Err(utopia_core::AppError::Conflict(_)) => duplicate_of(state, kb_id, &sha256).await,Err(e) => Err(e.into()),}}+async fn duplicate_of(+ state: &AppState,+ kb_id: Uuid,+ sha256: &str,+) -> anyhow::Result<IngestOutcome> {+ let holder = utopia_store::documents::find_by_kb_sha(&state.pool, kb_id, sha256).await?;+ Ok(IngestOutcome {+ action: holder.map_or(IngestAction::Unchanged, |d| IngestAction::Duplicate(d.id)),+ })+}+async fn sync_urls(state: &AppState, source: &Source) -> anyhow::Result<SyncStats> {let urls: Vec<String> = source.config["urls"].as_array()diff --git a/crates/utopia-store/src/documents.rs b/crates/utopia-store/src/documents.rsindex 3822fbad..45c12acf 100644--- a/crates/utopia-store/src/documents.rs+++ b/crates/utopia-store/src/documents.rs@@ -282,7 +282,13 @@ pub async fn replace_content_and_enqueue_processing(.bind(sha256).bind(doc_time).execute(&mut *tx)- .await?;+ .await+ .map_err(|e| match &e {+ sqlx::Error::Database(db) if db.is_unique_violation() => AppError::Conflict(format!(+ "File already exists (identical content): {filename}"+ )),+ _ => AppError::Db(e),+ })?;sqlx::query("INSERT INTO document_versions (id, document_id, version, sha256, size_bytes, doc_time)VALUES ($1, $2,@@ -736,6 +742,21 @@ pub async fn find_by_source_sha(.await?)}+/// 库里占着这份内容的那篇文档(`documents_kb_sha_idx` 管的范围:未清除的,含回收站里的)。+pub async fn find_by_kb_sha(+ pool: &PgPool,+ kb_id: Uuid,+ sha256: &str,+) -> AppResult<Option<Document>> {+ Ok(sqlx::query_as(+ "SELECT * FROM documents WHERE kb_id = $1 AND sha256 = $2 AND purged_at IS NULL",+ )+ .bind(kb_id)+ .bind(sha256)+ .fetch_optional(pool)+ .await?)+}+/// 迁移前的历史文档(无 external_key)按文件名认领——0008 之前建的行的一次性兜底。pub async fn find_legacy_by_filename(pool: &PgPool,diff --git a/docs/decisions/0013-a-source-should-hand-over-its-history.md b/docs/decisions/0013-a-source-should-hand-over-its-history.mdindex 766040d0..0fed62a3 100644--- a/docs/decisions/0013-a-source-should-hand-over-its-history.md+++ b/docs/decisions/0013-a-source-should-hand-over-its-history.md@@ -126,6 +126,7 @@ shape — a class with an IRI drawn as a circle is a picture that lies.- 2026-09-03: every connector's credentials stay on the server (#246). Until now only `auth_header` was stripped from responses; the object-storage, WebDAV and Notion keys went out to every viewer. The keys now live in one list, `SOURCE_SECRET_KEYS`, shared by the listing, the create / update responses and the update merge (blank or missing keeps the stored value, an explicit `null` removes it). Adding a connector means adding its keys there first.- 2026-09-03: the five connectors added under this record could not be created (#247): the store's hand-written `KINDS` allowlist stopped at seven kinds while the sync dispatcher and the UI knew twelve. The kinds now come from one enum, `SourceKind` in `utopia-core`; the allowlist is derived from it, the dispatcher matches it exhaustively, and a test compares the frontend's `web/src/sourceKinds.ts` against it, so the three can no longer drift apart.+- 2026-10-16: a push that names its `external_id` is identified by that id alone. The same bytes under a new id used to take the "same content at a new path is a move" branch, which re-keyed the existing document and silently dropped the first id. That branch now runs in full only for sources that identify by location (files, and an `api` push without `external_id`), where it keeps a renamed file's history. A push with `external_id` takes it over only from a document of the same source whose id was tombstoned, never from a live one: tombstoning the old id and then pushing the new id with the same bytes is how such a client renames, and the document keeps its history. A push whose content another document of the knowledge base already holds (any not purged, so `duplicate_of` can name a document in the trash), whether it is new or an update of an existing id, is answered `{"action": "duplicate", "duplicate_of": <document id>}`; it used to answer `unchanged` for a new id and fail the push with a database error for an update. One document per content per base (`documents_kb_sha_idx`) stands: storing the second id as its own document would need that index relaxed, which this revision does not do.## Status historydiff --git a/web/src/docs/ingest.md b/web/src/docs/ingest.mdindex d5cbaac4..e7bce050 100644--- a/web/src/docs/ingest.md+++ b/web/src/docs/ingest.md@@ -107,7 +107,9 @@ The response tells you what happened:{ "action": "created" }```-`created` · `updated` · `moved` (same content, new name) · `unchanged` · `marked_missing`.+`created` · `updated` · `moved` (same content, new name) · `unchanged` · `marked_missing` · `duplicate` (the base already holds this exact content as another document, named by `duplicate_of`, which may be a document in the trash; nothing was created or changed).++With `external_id`, the id is the identity: the same content under a new id never takes over a live id's document. To rename an id, tombstone the old one (`"deleted": true`), then push the new id with the same content; the new id takes over the document, with its history, and answers `moved`.Example with curl:@@ -156,7 +158,7 @@ Two things to know before wiring a system to it:- **Send events, not state.** A row that *is* the current state of a system belongs on a mounted database, read at query time. Pushing it here copies state into the ledger and the two drift.- **A new push under the same identity does not close the old statement.** The earlier one becomes stale and goes to review; deciding that an interval ended is done on the typed layer, by alignment and the temporal engine, not by the push.-The response is the API push's: `{ "action": "created" }` · `updated` · `unchanged` · `marked_missing`.+The response is the API push's: `{ "action": "created" }` · `updated` · `unchanged` · `marked_missing` · `duplicate` (with `duplicate_of`). It is never `moved`: the stored document carries its `external_id`, so two identities never share content.---