World's first open-source enterprise world model.
Rust agent-memorybitemporalgraphragknowledge-baseknowledge-graphllmontologypgvectorpostgresqlragrustself-hostedsemantic-searchtemporal-knowledge-graphworld-model
A MySQL-compatible warehouse can be mounted (#315)
WaylandYangd28655690b62
diff --git a/Cargo.lock b/Cargo.lockindex 7d1db369..fe1462ff 100644--- a/Cargo.lock+++ b/Cargo.lock@@ -370,6 +370,19 @@ version = "1.8.3"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "2af50177e190e07a26ab74f8b1efbfe2ef87da2116221318cb1c2e82baf7de06"+[[package]]+name = "bigdecimal"+version = "0.4.10"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "4d6867f1565b3aad85681f1015055b087fcfd840d6aeee6eee7f2da317603695"+dependencies = [+ "autocfg",+ "libm",+ "num-bigint",+ "num-integer",+ "num-traits",+]+[[package]]name = "bitflags"version = "2.13.1"@@ -4004,6 +4017,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "ee6798b1838b6a0f69c007c133b8df5866302197e404e8b6ee8ed3e3a5e68dc6"dependencies = ["base64 0.22.1",+ "bigdecimal","bytes","chrono","crc",@@ -4081,6 +4095,7 @@ checksum = "aa003f0038df784eb8fecbbac13affe3da23b45194bd57dba231c8f48199c526"dependencies = ["atoi","base64 0.22.1",+ "bigdecimal","bitflags","byteorder","bytes",@@ -4125,6 +4140,7 @@ checksum = "db58fcd5a53cf07c184b154801ff91347e4c30d17a3562a635ff028ad5deda46"dependencies = ["atoi","base64 0.22.1",+ "bigdecimal","bitflags","byteorder","chrono",@@ -4142,6 +4158,7 @@ dependencies = ["log","md-5 0.10.6","memchr",+ "num-bigint","once_cell","rand 0.8.8","serde",diff --git a/Cargo.toml b/Cargo.tomlindex 57db85a4..36c00feb 100644--- a/Cargo.toml+++ b/Cargo.toml@@ -26,7 +26,11 @@ axum = { version = "0.8", features = ["multipart"] }axum-extra = { version = "0.10", features = ["cookie"] }tower-http = { version = "0.6", features = ["cors", "trace", "fs"] }sqlx = { version = "0.8", default-features = false, features = [- "runtime-tokio", "tls-rustls", "postgres", "uuid", "chrono", "json", "migrate", "macros",+ # mysql 只为问数引擎而开(query_engine::mysql):一条线协议顺带覆盖+ # TiDB / OceanBase / Doris / StarRocks。本体的库始终是 postgres。+ # bigdecimal 是 DECIMAL 唯一的读法——驱动把 DECIMAL 同时挡在 f64 和 String+ # 之外("floating-point numbers have different semantics"),没有它金额列会整列变 null+ "runtime-tokio", "tls-rustls", "postgres", "mysql", "bigdecimal", "uuid", "chrono", "json", "migrate", "macros",] }serde = { version = "1", features = ["derive"] }serde_json = "1"diff --git a/README.md b/README.mdindex f57cfb23..5b236d8b 100644--- a/README.md+++ b/README.md@@ -61,7 +61,7 @@ One Rust binary and one Postgres. Full-text search is embedded in the binary, ve| **Entity resolution and review** | Duplicates are resolved in three stages: exact name or alias, embedding similarity, then a model's call on the doubtful pairs. Every merge can be undone. Uncertain cases go to a review queue: low-confidence extractions, suspected duplicates and cardinality conflicts. || **Reasoning and derivation** | Ontology axioms compile into rules: transitivity, symmetry, inverses and relation hierarchy derive new facts by forward chaining. Derivation is off by default, since a wrong axiom derives wrong facts. A derived fact is marked as such on the graph, carries validity and confidence like any other, and shows what it was derived from. When it contradicts an asserted fact, the asserted one stands. || **Conflict detection** | Three kinds of conflict, three sets of choices. A new fact that clashes with an older one: close the old, keep both, or reject the new. Data that breaks an axiom (self-loop, asymmetry, transitive cycle, cardinality): retract the fact, relax the axiom, or accept both. The ontology itself is checked first, because violations of a self-contradictory ontology are noise. |-| **Ontology-driven querying** | Mount a database on a base (Postgres, Trino for Iceberg / Delta Lake / Hive, Databricks, Snowflake) and chat can query it alongside the documents. The agent proposes how its tables map onto the ontology, and you confirm. The method behind it, [Ontology2SQL](https://github.com/deeplethe/ontology2sql), is state of the art on BIRD Mini-Dev for SQLite and PostgreSQL ([submission](https://github.com/bird-bench/bird-bench.github.io/pull/218)). |+| **Ontology-driven querying** | Mount a database on a base (Postgres, MySQL and the engines that speak its protocol, Trino for Iceberg / Delta Lake / Hive, Databricks, Snowflake) and chat can query it alongside the documents. The agent proposes how its tables map onto the ontology, and you confirm. The method behind it, [Ontology2SQL](https://github.com/deeplethe/ontology2sql), is state of the art on BIRD Mini-Dev for SQLite and PostgreSQL ([submission](https://github.com/bird-bench/bird-bench.github.io/pull/218)). || **Multi-user and permissions** | Each knowledge base has its own members and roles: owner, admin, editor and viewer. Open bases are readable by everyone in the deployment, restricted ones only by invitation. The first account registered becomes the system administrator. || **Decision ledger** | Confirming or rejecting a fact, merging or reverting an entity, rebuilding the graph: each leaves a record of who, when, and what the object looked like at the time. The ledger is append-only, and a record outlives its object, even the base it belonged to. || **[Decision intelligence (in development)](#roadmap)** | Record a decision, replay both what was understood and the course it took, and reason over overlaid scenarios. |@@ -105,7 +105,7 @@ cd web && pnpm install && pnpm dev- [ ] **Business rules**: rules written by people over an entity's attribute facts, a threshold or a category set, that classify it as a derived fact with the rule and the premises as its explanation ([#277](https://github.com/deeplethe/utopia/issues/277))- [ ] **Execution gate**: checking an agent's calls against ontology rules and symbolic logic- [ ] **MaxCompute**: mapping exploration and Ontology2SQL over Alibaba Cloud MaxCompute (Iceberg / Delta Lake via Trino, Databricks and Snowflake are in, awaiting a run against a real cluster)-- [ ] **More sources**: MySQL, ClickHouse and Doris drivers; a Feishu connector+- [ ] **More sources**: a ClickHouse driver; a Feishu connector- [ ] **Time to the moment**: an `instant` precision beside year / month / day, for sources that carry a real timestamp. Today a connector rounds it to a UTC day, which can shift an event across midnight by one day- [ ] **Agent memory over MCP**: episode writes, the retrieve endpoint, and the MCP server- [ ] **Enterprise**: OIDC SSO, backup and restore commands, benchmarks at 100k documentsdiff --git a/README.zh-CN.md b/README.zh-CN.mdindex 19166b25..d74bcbf2 100644--- a/README.zh-CN.md+++ b/README.zh-CN.md@@ -106,7 +106,7 @@ cd web && pnpm install && pnpm dev- [ ] **业务规则**:由人写下、作用于实体属性事实的规则(阈值、类别集合),把实体归类为一条带前提的派生事实,规则与前提就是它的解释([#277](https://github.com/deeplethe/utopia/issues/277))- [ ] **执行校验层**:对 Agent 的调用进行本体规则与符号逻辑校验- [ ] **问数与映射添加数据湖仓支持**:Iceberg / Delta Lake,以及 Databricks、Snowflake、MaxCompute 的映射探索与 Ontology2SQL 支持-- [ ] **更多数据源**:MySQL、ClickHouse、Doris 驱动,飞书连接器+- [ ] **更多数据源**:ClickHouse 驱动,飞书连接器- [ ] **精确到时刻**:在年 / 月 / 日之外加一档 `instant` 精度,给那些本来就带时间戳的来源——现在连接器按 UTC 截到天,跨午夜的事件会差一天- [ ] **MCP 上的 Agent 记忆**:补齐 episodes 写入、retrieve 端点与 MCP 服务器- [ ] **企业化**:OIDC SSO、备份恢复命令、10 万文档级别的性能基准diff --git a/crates/utopia-server/src/api/datasource_routes.rs b/crates/utopia-server/src/api/datasource_routes.rsindex a4404684..f1434cc1 100644--- a/crates/utopia-server/src/api/datasource_routes.rs+++ b/crates/utopia-server/src/api/datasource_routes.rs@@ -58,7 +58,7 @@ pub async fn create(utopia_core::AppError::invalid("unsupported_conn_scheme",format!(- "Connection string must start with one of: postgres://, trino://, databricks://, snowflake:// (engines: {})",+ "Connection string must start with one of: postgres://, mysql://, trino://, databricks://, snowflake:// (engines: {})",crate::query_engine::ENGINES.join(", ")),)diff --git a/crates/utopia-server/src/query_engine/mod.rs b/crates/utopia-server/src/query_engine/mod.rsindex 2bd17132..7e5cb6ce 100644--- a/crates/utopia-server/src/query_engine/mod.rs+++ b/crates/utopia-server/src/query_engine/mod.rs@@ -1,6 +1,8 @@//! 问数查询引擎:trait 接缝(BlobStore 同手法)+ 引擎无关的安全闸。//!-//! 引擎按协议族扩,不按产品名扩:postgres 线协议(`postgres.rs`)→ HTTP 族——+//! 引擎按协议族扩,不按产品名扩:postgres 线协议(`postgres.rs`)、MySQL 线协议+//! (`mysql.rs`,一条协议顺带覆盖 TiDB / OceanBase / Doris / StarRocks / MariaDB)+//! → HTTP 族——//! `trino.rs` 一个顶起 Iceberg / Delta / Hive 整个湖仓生态,`databricks.rs`、//! `snowflake.rs` 各走自家的 SQL REST API。挂载模型与注册表引擎无关,加引擎只放宽//! 一条 CHECK。连接串是唯一的输入:引擎由 scheme 决定([`engine_from_conn`]),@@ -16,12 +18,15 @@mod conn;mod databricks;+mod mysql;mod postgres;mod snowflake;mod trino;use sqlparser::ast::Statement;-use sqlparser::dialect::{DatabricksDialect, GenericDialect, PostgreSqlDialect, SnowflakeDialect};+use sqlparser::dialect::{+ DatabricksDialect, GenericDialect, MySqlDialect, PostgreSqlDialect, SnowflakeDialect,+};use sqlparser::parser::Parser;use std::time::Duration;@@ -33,7 +38,7 @@ pub(crate) const HTTP_REQUEST_TIMEOUT: Duration = Duration::from_secs(20);pub(crate) const HTTP_POLL_BUDGET: Duration = Duration::from_secs(30);/// 注册表里 `engine` 列的取值。迁移里的 CHECK 与这张表要一致-pub const ENGINES: &[&str] = &["postgres", "trino", "databricks", "snowflake"];+pub const ENGINES: &[&str] = &["postgres", "mysql", "trino", "databricks", "snowflake"];#[derive(Debug)]pub struct QueryResult {@@ -64,6 +69,9 @@ pub fn engine_from_conn(conn: &str) -> Option<&'static str> {let scheme = conn.trim().split("://").next()?.to_ascii_lowercase();match scheme.as_str() {"postgres" | "postgresql" => Some("postgres"),+ // 一条线协议顺带覆盖 TiDB / OceanBase / Doris / StarRocks——它们都说+ // MySQL 协议,连接串照写 mysql:// 即可+ "mysql" | "mariadb" => Some("mysql"),"trino" | "presto" => Some("trino"),"databricks" => Some("databricks"),"snowflake" => Some("snowflake"),@@ -75,6 +83,7 @@ pub fn engine_from_conn(conn: &str) -> Option<&'static str> {pub fn engine_for(engine: &str, conn: &str) -> anyhow::Result<Box<dyn QueryEngine>> {match engine {"postgres" => Ok(Box::new(postgres::PostgresEngine::new(conn))),+ "mysql" => Ok(Box::new(mysql::MysqlEngine::new(conn))),"trino" => Ok(Box::new(trino::TrinoEngine::new(conn::TrinoConn::parse(conn,)?))),@@ -98,6 +107,7 @@ pub fn guard_sql_for(engine: &str, sql: &str) -> anyhow::Result<String> {"databricks" => Parser::parse_sql(&DatabricksDialect {}, cleaned),"snowflake" => Parser::parse_sql(&SnowflakeDialect {}, cleaned),"trino" => Parser::parse_sql(&GenericDialect {}, cleaned),+ "mysql" => Parser::parse_sql(&MySqlDialect {}, cleaned),_ => Parser::parse_sql(&PostgreSqlDialect {}, cleaned),};let statements = parsed.map_err(|e| anyhow::anyhow!("SQL parse error: {e}"))?;@@ -292,7 +302,7 @@ mod tests {#[test]fn every_dialect_keeps_the_same_gate() {- for engine in ["postgres", "trino", "databricks", "snowflake"] {+ for engine in ["postgres", "mysql", "trino", "databricks", "snowflake"] {assert!(guard_sql_for(engine, "SELECT a FROM t WHERE b > 1").is_ok(),"{engine}"@@ -307,6 +317,8 @@ mod tests {assert!(guard_sql_for("databricks", "SELECT `region` FROM main.sales.orders").is_ok());assert!(guard_sql_for("snowflake", "SELECT amount::number FROM db.public.orders").is_ok());assert!(guard_sql_for("trino", "SELECT count(*) FROM hive.default.orders").is_ok());+ // MySQL 的反引号与 PG 方言不兼容,走自己的方言才过得去+ assert!(guard_sql_for("mysql", "SELECT `region` FROM `sales`.`orders`").is_ok());}#[test]@@ -323,7 +335,9 @@ mod tests {engine_from_conn("snowflake://:[email protected]/db"),Some("snowflake"));- assert_eq!(engine_from_conn("mysql://u@h/db"), None);+ assert_eq!(engine_from_conn("mysql://u:p@h:3306/db"), Some("mysql"));+ // 同一套协议的另一个写法,引擎里会被改写成 mysql:// 再交给驱动+ assert_eq!(engine_from_conn("mariadb://u@h/db"), Some("mysql"));assert_eq!(engine_from_conn("garbage"), None);}diff --git a/crates/utopia-server/src/query_engine/mysql.rs b/crates/utopia-server/src/query_engine/mysql.rsnew file mode 100644index 00000000..7c8b16fd--- /dev/null+++ b/crates/utopia-server/src/query_engine/mysql.rs@@ -0,0 +1,425 @@+//! MySQL 线协议族。**一条协议顺带覆盖一片**:TiDB、OceanBase、Doris、StarRocks+//! 都说这个协议,MariaDB 也是,所以这个文件的性价比在四个引擎里最高。+//!+//! 与 `postgres.rs` 的两处不同,都来自 MySQL 自己:+//!+//! - **没有 `row_to_json`。** PG 那边把整行交给库转成 JSON 文本,列序天然保留;+//! 这里只能逐列取值自己拼(同 HTTP 族的 `rows_to_json_lines`),于是多出一张+//! 类型映射表。那张表的判据是驱动的 `ColumnType::name` 与各类型的+//! `compatible`,不是 MySQL 手册:两处分歧会静默地把整列变成 null——+//! 无符号整数的类型名带 ` UNSIGNED` 后缀,而 `DECIMAL` 被同时挡在 `f64`+//! 与 `String` 之外,只有 `BigDecimal` 读得出来(见 `Cell`)+//! - **超时的写法有两种。** MySQL 是 `max_execution_time`(毫秒),MariaDB 是+//! `max_statement_time`(秒)。两个都试,都不认才报错:这一层挡的是全表扫描+//! 拖垮库,外包的 LIMIT 挡不住它(先扫完再截断),所以不能静默降级++use super::{+ coerce, rows_to_json_lines, truncate_rows, wrap_limit, QueryEngine, QueryResult, SchemaColumn,+ STATEMENT_TIMEOUT_SECS,+};+use sqlx::mysql::MySqlPoolOptions;+use sqlx::{Column, Row, TypeInfo};+use std::time::Duration;++pub struct MysqlEngine {+ conn: String,+}++impl MysqlEngine {+ pub fn new(conn: &str) -> Self {+ // sqlx 只认 mysql://,而 mariadb:// 是同一套协议的另一个写法+ let conn = match conn.strip_prefix("mariadb://") {+ Some(rest) => format!("mysql://{rest}"),+ None => conn.to_string(),+ };+ Self { conn }+ }++ async fn pool(&self) -> anyhow::Result<sqlx::MySqlPool> {+ Ok(MySqlPoolOptions::new()+ .max_connections(1)+ .acquire_timeout(Duration::from_secs(5))+ .connect(&self.conn)+ .await?)+ }+}++/// 列类型 → 按哪一档去读。+///+/// **分派与读取分开**,因为只有前者测得了:造一个 `MySqlRow` 需要真的连上+/// 服务器,而这张表恰恰是这个文件里最容易写错的地方。+#[derive(Debug, PartialEq, Eq)]+enum Cell {+ Bool,+ /// 有符号整数。驱动的 `i64` 明确排除 UNSIGNED,所以无符号另走一档+ Int,+ /// `BIGINT UNSIGNED` 一类。驱动给的类型名**带 UNSIGNED 后缀**,+ /// 认不出来就会掉进兜底档,而兜底读不出整数——整列变 null+ UnsignedInt,+ Float,+ /// `DECIMAL`。驱动把它同时挡在 `f64`("floating-point numbers have+ /// different semantics")和 `String`(不在字符串兼容表里)之外,+ /// 只能用 `BigDecimal` 读。金额列几乎都是这个类型,掉档的代价最大+ Decimal,+ Json,+ Date,+ Time,+ DateTime,+ Timestamp,+ /// 兜底:先字符串,再字节。整数不走这里——`42` 变成 `"42"` 之后,+ /// 模型对它的算术不一样+ Text,+}++/// 类型名 → 档位。名字取自驱动的 `ColumnType::name`,那张表是这个函数的唯一依据:+/// `TINYINT(1)` 报 `BOOLEAN`、`DECIMAL` 与 `NEWDECIMAL` 都报 `DECIMAL`、+/// 无符号整数带 ` UNSIGNED` 后缀。+fn cell_kind(type_name: &str) -> Cell {+ let ty = type_name.to_ascii_uppercase();+ // 后缀先判:`INT UNSIGNED` 落到 `INT` 那一档会用 i64 去读,驱动直接拒绝+ if let Some(base) = ty.strip_suffix(" UNSIGNED") {+ return match base {+ "TINYINT" | "SMALLINT" | "MEDIUMINT" | "INT" | "BIGINT" => Cell::UnsignedInt,+ "FLOAT" | "DOUBLE" => Cell::Float,+ "DECIMAL" => Cell::Decimal,+ _ => Cell::Text,+ };+ }+ match ty.as_str() {+ "BOOLEAN" | "BOOL" => Cell::Bool,+ "TINYINT" | "SMALLINT" | "MEDIUMINT" | "INT" | "BIGINT" => Cell::Int,+ "FLOAT" | "DOUBLE" => Cell::Float,+ "DECIMAL" => Cell::Decimal,+ "JSON" => Cell::Json,+ "DATE" => Cell::Date,+ "TIME" => Cell::Time,+ "DATETIME" => Cell::DateTime,+ "TIMESTAMP" => Cell::Timestamp,+ _ => Cell::Text,+ }+}++/// 一格的值 → JSON。取不出来就退字符串,再取不出来才 null。+fn cell_to_json(row: &sqlx::mysql::MySqlRow, i: usize, type_name: &str) -> serde_json::Value {+ use serde_json::Value;+ // 每个分支都取 `Option<T>`,NULL 在各自那一档里变成 JSON null——+ // 不做统一的前置探测:二进制列取 `Option<String>` 会报错而不是给 None,+ // 那种探测会把一个有值的 BLOB 判成空+ let ty = type_name.to_ascii_uppercase();+ match cell_kind(&ty) {+ Cell::Bool => row+ .try_get::<Option<bool>, _>(i)+ .map(|v| v.map_or(Value::Null, Value::Bool))+ .unwrap_or(Value::Null),+ Cell::Int => row+ .try_get::<Option<i64>, _>(i)+ .map(|v| v.map_or(Value::Null, |n| n.into()))+ .unwrap_or(Value::Null),+ Cell::UnsignedInt => row+ .try_get::<Option<u64>, _>(i)+ .map(|v| v.map_or(Value::Null, |n| n.into()))+ .unwrap_or(Value::Null),+ // BigDecimal → 文本 → coerce,与 Databricks / Snowflake 的 DECIMAL 同一条路。+ // 转数走 coerce 里的 f64,超出 f64 精度的值会留成字符串而不是变成一个近似数+ Cell::Decimal => row+ .try_get::<Option<sqlx::types::BigDecimal>, _>(i)+ .ok()+ .flatten()+ .map_or(Value::Null, |d| {+ coerce("DECIMAL", &Value::String(d.to_string()))+ }),+ Cell::Float => row+ .try_get::<Option<f64>, _>(i)+ .ok()+ .flatten()+ .and_then(serde_json::Number::from_f64)+ .map_or(Value::Null, Value::Number),+ Cell::Json => row+ .try_get::<Option<serde_json::Value>, _>(i)+ .ok()+ .flatten()+ .unwrap_or(Value::Null),+ Cell::Date => try_text(row, i, |r| {+ r.try_get::<Option<chrono::NaiveDate>, _>(i)+ .map(|v| v.map(|d| d.to_string()))+ }),+ Cell::Time => try_text(row, i, |r| {+ r.try_get::<Option<chrono::NaiveTime>, _>(i)+ .map(|v| v.map(|t| t.to_string()))+ }),+ Cell::DateTime => try_text(row, i, |r| {+ r.try_get::<Option<chrono::NaiveDateTime>, _>(i)+ .map(|v| v.map(|t| t.to_string()))+ }),+ Cell::Timestamp => try_text(row, i, |r| {+ r.try_get::<Option<chrono::DateTime<chrono::Utc>>, _>(i)+ .map(|v| v.map(|t| t.to_rfc3339()))+ }),+ Cell::Text => {+ // CHAR / VARCHAR / TEXT / ENUM / SET 走这里。仍然过一遍 coerce:+ // 服务器把数字列声明成字符串的情况不少见+ let raw = row+ .try_get::<Option<String>, _>(i)+ .ok()+ .flatten()+ .map(Value::String);+ match raw {+ Some(v) => coerce(&ty, &v),+ // 二进制列(BLOB / VARBINARY / GEOMETRY)不是合法 UTF-8,+ // 给个长度而不是塞一堆转义字节进模型的上下文+ None => match row.try_get::<Option<Vec<u8>>, _>(i) {+ Ok(Some(bytes)) => Value::String(format!("<{} bytes>", bytes.len())),+ _ => Value::Null,+ },+ }+ }+ }+}++/// 按类型读文本,读不出来退回通用字符串(时间列被服务器配置成字符串时会走到)。+fn try_text<F>(row: &sqlx::mysql::MySqlRow, i: usize, f: F) -> serde_json::Value+where+ F: Fn(&sqlx::mysql::MySqlRow) -> Result<Option<String>, sqlx::Error>,+{+ match f(row) {+ Ok(Some(s)) => serde_json::Value::String(s),+ Ok(None) => serde_json::Value::Null,+ Err(_) => row+ .try_get::<Option<String>, _>(i)+ .ok()+ .flatten()+ .map_or(serde_json::Value::Null, serde_json::Value::String),+ }+}++#[async_trait::async_trait]+impl QueryEngine for MysqlEngine {+ async fn test(&self) -> anyhow::Result<()> {+ let pool = self.pool().await?;+ sqlx::query("SELECT 1").execute(&pool).await?;+ pool.close().await;+ Ok(())+ }++ async fn fetch_schema(&self) -> anyhow::Result<Vec<SchemaColumn>> {+ let pool = self.pool().await?;+ // MySQL 的 schema 就是 database。系统库按名字排除——information_schema+ // 在这里是**要排除的对象**,与 PG 那边同名的概念不是一回事+ let rows: Vec<(String, String, String, String, Option<String>)> = sqlx::query_as(+ "SELECT table_schema, table_name, column_name, column_type, column_comment+ FROM information_schema.columns+ WHERE table_schema NOT IN+ ('information_schema', 'mysql', 'performance_schema', 'sys')+ ORDER BY table_schema, table_name, ordinal_position",+ )+ .fetch_all(&pool)+ .await?;+ pool.close().await;+ Ok(rows+ .into_iter()+ .map(|(schema, table, column, data_type, comment)| SchemaColumn {+ schema,+ table,+ column,+ data_type,+ // 没有注释时这一列是空串而不是 NULL,照抄会让每张表都挂一个空注释+ comment: comment.filter(|c| !c.trim().is_empty()),+ })+ .collect())+ }++ async fn execute(&self, sql: &str) -> anyhow::Result<QueryResult> {+ let pool = self.pool().await?;+ // 纵深防御第 3 层:会话只读 + 语句超时。只读在自动提交下逐条生效,+ // parser 万一漏网也写不进去+ sqlx::query("SET SESSION TRANSACTION READ ONLY")+ .execute(&pool)+ .await?;+ let millis = STATEMENT_TIMEOUT_SECS * 1000;+ let mysql_form = sqlx::query(&format!("SET SESSION max_execution_time = {millis}"))+ .execute(&pool)+ .await;+ if mysql_form.is_err() {+ // MariaDB:秒,且是浮点+ sqlx::query(&format!(+ "SET SESSION max_statement_time = {STATEMENT_TIMEOUT_SECS}"+ ))+ .execute(&pool)+ .await+ .map_err(|e| {+ anyhow::anyhow!("Could not set a statement timeout on this server: {e}")+ })?;+ }++ let fetched = sqlx::query(&wrap_limit(sql)).fetch_all(&pool).await?;+ pool.close().await;++ let (fetched, truncated) = truncate_rows(fetched);+ let Some(first) = fetched.first() else {+ return Ok(QueryResult {+ rows: Vec::new(),+ truncated,+ });+ };+ // 列名与类型只取一次:同一结果集每行的列都一样+ let columns: Vec<String> = first+ .columns()+ .iter()+ .map(|c| c.name().to_string())+ .collect();+ let types: Vec<String> = first+ .columns()+ .iter()+ .map(|c| c.type_info().name().to_string())+ .collect();+ let values: Vec<Vec<serde_json::Value>> = fetched+ .iter()+ .map(|row| {+ (0..columns.len())+ .map(|i| cell_to_json(row, i, &types[i]))+ .collect()+ })+ .collect();+ Ok(QueryResult {+ rows: rows_to_json_lines(&columns, &values),+ truncated,+ })+ }+}++#[cfg(test)]+mod tests {+ use super::{cell_kind, Cell, MysqlEngine, QueryEngine};++ /// 对着真服务器跑的那一档。没有 `UTOPIA_TEST_MYSQL_URL` 就跳过——+ /// 这三样(information_schema 的列名、两种超时写法、取值往返)是+ /// 类型表之外唯一测不到的部分,而它们只在真服务器上才有答案。+ ///+ /// 起一个来跑:+ /// `docker run -d -e MARIADB_ROOT_PASSWORD=pw -p 13306:3306 mariadb:11.4`+ /// 然后 `UTOPIA_TEST_MYSQL_URL=mysql://root:[email protected]:13306/sales`。+ /// 建表语句见 #316。+ fn live_url() -> Option<String> {+ std::env::var("UTOPIA_TEST_MYSQL_URL")+ .ok()+ .filter(|u| !u.trim().is_empty())+ }++ #[tokio::test]+ async fn a_live_server_answers_with_typed_values() {+ let Some(url) = live_url() else {+ return;+ };+ let engine = MysqlEngine::new(&url);+ engine.test().await.expect("SELECT 1");++ // information_schema 的列名与 PG 不同(column_type / column_comment),+ // 写错了不会报错,只会让 schema 文档少一半+ let schema = engine.fetch_schema().await.expect("schema");+ let amount = schema+ .iter()+ .find(|c| c.table == "orders" && c.column == "amount")+ .expect("orders.amount");+ assert!(+ amount.data_type.starts_with("decimal"),+ "column_type 要给出带精度的形态,拿到的是 {}",+ amount.data_type+ );+ assert_eq!(+ amount.comment.as_deref(),+ Some("Order total in CNY"),+ "注释要跟着列走"+ );+ assert!(+ schema.iter().all(|c| c.comment.as_deref() != Some("")),+ "空注释是空串不是 NULL,要过滤掉"+ );++ // 取值往返:每一种掉档都会在这里现形——数变成字符串,或者整列 null+ let r = engine+ .execute(+ "SELECT id, region, amount, qty, flag, placed_on FROM sales.orders ORDER BY id",+ )+ .await+ .expect("execute");+ assert_eq!(r.rows.len(), 3);+ let first: serde_json::Value = serde_json::from_str(&r.rows[0]).unwrap();+ assert_eq!(first["id"], serde_json::json!(1));+ assert_eq!(first["region"], serde_json::json!("east"));+ // DECIMAL:没有 BigDecimal 这一格就是 null+ assert_eq!(first["amount"], serde_json::json!(1234.56));+ // BIGINT UNSIGNED:i64 装不下,且驱动拒绝用 i64 读它+ assert_eq!(first["qty"], serde_json::json!(18446744073709551615u64));+ // TINYINT(1) 报的类型名是 BOOLEAN+ assert_eq!(first["flag"], serde_json::json!(true));+ assert_eq!(first["placed_on"], serde_json::json!("2023-06-01"));++ // 全 NULL 的那一行:每一格都该是 JSON null,而不是某个类型的零值+ let third: serde_json::Value = serde_json::from_str(&r.rows[2]).unwrap();+ for k in ["region", "amount", "qty", "flag", "placed_on"] {+ assert_eq!(third[k], serde_json::Value::Null, "{k} 该是 null");+ }++ // 写路径仍然被闸挡住(第 1 层),只读会话是第 3 层+ assert!(super::super::guard_sql_for("mysql", "DELETE FROM sales.orders").is_err());+ }++ #[test]+ fn mariadb_is_the_same_protocol_under_another_name() {+ // sqlx 只认 mysql://,而界面允许写 mariadb://——不改写的话驱动会以+ // 「未知 scheme」拒掉一个完全正常的连接串+ assert_eq!(+ MysqlEngine::new("mariadb://u:p@h:3306/db").conn,+ "mysql://u:p@h:3306/db"+ );+ assert_eq!(+ MysqlEngine::new("mysql://u:p@h:3306/db").conn,+ "mysql://u:p@h:3306/db"+ );+ // 只剥前缀:密码里出现同样的字符不该被动到+ assert_eq!(+ MysqlEngine::new("mysql://u:mariadb://x@h/db").conn,+ "mysql://u:mariadb://x@h/db"+ );+ }++ /// 这三个断言各对应一种「整列变 null」。名字来自驱动的 `ColumnType::name`——+ /// 那张表是判据,不是猜的。+ #[test]+ fn every_number_shape_lands_in_a_readable_slot() {+ for t in ["TINYINT", "SMALLINT", "MEDIUMINT", "INT", "BIGINT", "int"] {+ assert_eq!(cell_kind(t), Cell::Int, "{t}");+ }+ // 驱动的 i64 明确排除 UNSIGNED,而类型名带着后缀过来。+ // 少了这一档,一个 BIGINT UNSIGNED 列会掉进兜底,兜底读不出整数+ for t in [+ "TINYINT UNSIGNED",+ "SMALLINT UNSIGNED",+ "MEDIUMINT UNSIGNED",+ "INT UNSIGNED",+ "BIGINT UNSIGNED",+ ] {+ assert_eq!(cell_kind(t), Cell::UnsignedInt, "{t}");+ }+ // DECIMAL 被驱动同时挡在 f64 与 String 之外,只有 BigDecimal 读得出。+ // 金额列几乎都是这个类型,掉档的代价最大+ assert_eq!(cell_kind("DECIMAL"), Cell::Decimal);+ assert_eq!(cell_kind("DECIMAL UNSIGNED"), Cell::Decimal);+ assert_eq!(cell_kind("DOUBLE"), Cell::Float);+ assert_eq!(cell_kind("DOUBLE UNSIGNED"), Cell::Float);+ // TINYINT(1) 在这个驱动里就报 BOOLEAN,不是 TINYINT+ assert_eq!(cell_kind("BOOLEAN"), Cell::Bool);+ assert_eq!(cell_kind("VARCHAR"), Cell::Text);+ assert_eq!(cell_kind("BLOB"), Cell::Text);+ }++ #[test]+ fn each_time_type_keeps_its_own_shape() {+ // 四种时间各读各的。DATE 当成 DATETIME 读会报错退回字符串,+ // 拿到的就不是 2023-06-01 而是驱动的原始形态+ assert_eq!(cell_kind("DATE"), Cell::Date);+ assert_eq!(cell_kind("TIME"), Cell::Time);+ assert_eq!(cell_kind("DATETIME"), Cell::DateTime);+ assert_eq!(cell_kind("TIMESTAMP"), Cell::Timestamp);+ }+}diff --git a/docs/decisions/0018-the-lakehouse-is-one-protocol-away.md b/docs/decisions/0018-the-lakehouse-is-one-protocol-away.mdindex 91ccdd46..7a7a32bc 100644--- a/docs/decisions/0018-the-lakehouse-is-one-protocol-away.md+++ b/docs/decisions/0018-the-lakehouse-is-one-protocol-away.md@@ -1,6 +1,6 @@# 0018 · The lakehouse is one protocol away-- **Status**: implemented · `trino` / `databricks` / `snowflake` engines in `query_engine/` (migration `0021` widens the `engine` CHECK) · every engine is covered by protocol replays only (wiremock); **none has run against a real cluster** (#240, #241, #242) · MaxCompute is not done, see the last section+- **Status**: implemented · `trino` / `databricks` / `snowflake` engines in `query_engine/` (migration `0021` widens the `engine` CHECK) · every engine is covered by protocol replays only (wiremock); **none has run against a real cluster** (#240, #241, #242) · MaxCompute is not done, see the last section · the MySQL wire family this record skipped landed later in #303 (migration `0025`), so the order below is now "Postgres, MySQL, HTTP" as the `query_engine` header always had it- **Written**: 2026-09-03 (conventions in the [README](README.md))- **Related**: [0011](0011-a-mapping-is-not-a-fact.md) placed data sources at the deployment level and mounts at the base level; this record leaves that layer alone. [0016](0016-close-the-open-seams-before-cutting-new-ones.md) D4 put the MySQL wire protocol ahead of the lakehouse; the first section explains why the order flippeddiff --git a/migrations/0025_mysql_engine.sql b/migrations/0025_mysql_engine.sqlnew file mode 100644index 00000000..02595e5c--- /dev/null+++ b/migrations/0025_mysql_engine.sql@@ -0,0 +1,6 @@+-- MySQL 线协议进注册表(#303)。+-- 允许的名字与 `query_engine::ENGINES` 同一张表;挂载模型与引擎无关,+-- 加一个引擎仍然只放宽这一条 CHECK(同 0021)。+ALTER TABLE data_sources DROP CONSTRAINT data_sources_engine_check;+ALTER TABLE data_sources ADD CONSTRAINT data_sources_engine_check+ CHECK (engine IN ('postgres', 'mysql', 'trino', 'databricks', 'snowflake'));diff --git a/web/src/i18n/en.ts b/web/src/i18n/en.tsindex 8cea093e..2b740876 100644--- a/web/src/i18n/en.ts+++ b/web/src/i18n/en.ts@@ -951,9 +951,10 @@ export const en = {"Register connections here; each knowledge base mounts the ones it may query.",name: "Name",connString: "Connection string — the scheme picks the engine",- // 四种写法各一行;令牌放 password 位,Databricks 的路径就是控制台里的 httpPath+ // 每种写法各一行;令牌放 password 位,Databricks 的路径就是控制台里的 httpPathconnSchemes:"postgres://user:pass@host:5432/db\n" ++ "mysql://user:pass@host:3306/db (MariaDB, TiDB, OceanBase, Doris, StarRocks)\n" +"trino://user[:pass]@host:8080/catalog[/schema] (Iceberg, Delta Lake, Hive)\n" +"databricks://:TOKEN@host/sql/1.0/warehouses/ID?catalog=main\n" +"snowflake://:[email protected]/DB/SCHEMA?warehouse=WH",diff --git a/web/src/i18n/zh.ts b/web/src/i18n/zh.tsindex c100afe1..12484c98 100644--- a/web/src/i18n/zh.ts+++ b/web/src/i18n/zh.ts@@ -854,6 +854,7 @@ export const zh: Strings = {connString: "连接串,前缀决定引擎",connSchemes:"postgres://user:pass@host:5432/db\n" ++ "mysql://user:pass@host:3306/db (MariaDB、TiDB、OceanBase、Doris、StarRocks)\n" +"trino://user[:pass]@host:8080/catalog[/schema] (Iceberg、Delta Lake、Hive)\n" +"databricks://:TOKEN@host/sql/1.0/warehouses/ID?catalog=main\n" +"snowflake://:[email protected]/DB/SCHEMA?warehouse=WH",