Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.
TypeScript agentaiai-agentsall-in-onecrmcrm-systememaillinearmcpmessagingnotesnotionofficerustslackslack-alternativestartupstartup-toolstypescriptworkspace
fix(databases): return errors instead of panicking in the SQL engine and tools (#8948)
Wolf Mermelstein5af2febd4f6e
diff --git a/.agents/rules/rust/libraries.md b/.agents/rules/rust/libraries.mdindex 75fc7b4e26..ef6cb393ba 100644--- a/.agents/rules/rust/libraries.md+++ b/.agents/rules/rust/libraries.md@@ -23,9 +23,12 @@ them instead of their generic equivalents.map adapter errors with `rootcause::Report::new(e).into_dynamic()`. Binariesreturn `Result<(), rootcause::Report>`. Never add `anyhow` to a crate thatdoesn't use it (CS-46). Model: `crates/user_api_key`.-- **CS-57** No `Box<dyn Error>`, `Result<_, String>`, or `.unwrap()` /- `.expect()` outside tests and startup invariants; an `expect` states the- invariant it relies on.+- **CS-57** No panics outside tests and startup: no `.unwrap()`, `.expect()`,+ `unreachable!`, `panic!`, `todo!`, or unchecked indexing on a path a request,+ tool call, or message can reach. Return a typed error the caller can show+ instead, even for an "impossible" state. Never recover from a panic with+ `catch_unwind` or a `JoinError`; fix the code that panics. No+ `Box<dyn Error>` or `Result<_, String>` either.- **CS-58** Parse structured text with `nom` (and `logos` for tokenizing a realgrammar, as `crates/database_sql` does), not hand-rolled `split_once` /`strip_prefix` / index scanners.diff --git a/apps/web/src/lib/core/database-sql/generated/types.ts b/apps/web/src/lib/core/database-sql/generated/types.tsindex 9c9e079a0b..87edf45ebc 100644--- a/apps/web/src/lib/core/database-sql/generated/types.ts+++ b/apps/web/src/lib/core/database-sql/generated/types.ts@@ -1706,13 +1706,17 @@ export type RunError =/** The row. */row: RowId;} & {+ column?: never;expected?: never;fed?: never;+ invariant?: never;+ key?: never;limit?: never;message?: never;received?: never;request?: never;sent?: never;+ table?: never;what?: never;})/** An `UPDATE` or `DELETE` matched more rows than a statement reads. */@@ -1721,14 +1725,18 @@ export type RunError =/** The most rows a statement reads. */limit: number;} & {+ column?: never;expected?: never;fed?: never;+ invariant?: never;+ key?: never;message?: never;position?: never;received?: never;request?: never;row?: never;sent?: never;+ table?: never;what?: never;})/**@@ -1744,12 +1752,16 @@ export type RunError =/** What was fed. */fed: Answer;} & {+ column?: never;+ invariant?: never;+ key?: never;limit?: never;message?: never;position?: never;received?: never;row?: never;sent?: never;+ table?: never;what?: never;})/**@@ -1757,8 +1769,11 @@ export type RunError =* answer.*/| ({ stage: 'notAnsweredByBins' } & {+ column?: never;expected?: never;fed?: never;+ invariant?: never;+ key?: never;limit?: never;message?: never;position?: never;@@ -1766,6 +1781,7 @@ export type RunError =request?: never;row?: never;sent?: never;+ table?: never;what?: never;})/** The sink answered a different number of results than ops sent. */@@ -1774,14 +1790,18 @@ export type RunError =/** The results that came back. */received: number;} & {+ column?: never;expected?: never;fed?: never;+ invariant?: never;+ key?: never;limit?: never;message?: never;position?: never;request?: never;row?: never;sent?: never;+ table?: never;what?: never;})/** The sink answered an op with a result of another kind. */@@ -1792,13 +1812,17 @@ export type RunError =/** The result that came back. */received: OpResultKind;} & {+ column?: never;expected?: never;fed?: never;+ invariant?: never;+ key?: never;limit?: never;message?: never;position?: never;request?: never;row?: never;+ table?: never;what?: never;})/** A feed quoted a request the engine is not waiting on. */@@ -1809,6 +1833,9 @@ export type RunError =/** The id fed. */fed: number;} & {+ column?: never;+ invariant?: never;+ key?: never;limit?: never;message?: never;position?: never;@@ -1816,6 +1843,7 @@ export type RunError =request?: never;row?: never;sent?: never;+ table?: never;what?: never;})/** A feed arrived when nothing was outstanding. */@@ -1824,7 +1852,10 @@ export type RunError =/** The id fed. */fed: number;} & {+ column?: never;expected?: never;+ invariant?: never;+ key?: never;limit?: never;message?: never;position?: never;@@ -1832,6 +1863,7 @@ export type RunError =request?: never;row?: never;sent?: never;+ table?: never;what?: never;})/** A value handed to the engine is not the shape it reads. */@@ -1842,14 +1874,18 @@ export type RunError =/** Why it could not be read. */message: string;} & {+ column?: never;expected?: never;fed?: never;+ invariant?: never;+ key?: never;limit?: never;position?: never;received?: never;request?: never;row?: never;sent?: never;+ table?: never;})/** A value the engine hands back could not be written out. */| ({@@ -1857,20 +1893,27 @@ export type RunError =/** Why it could not be written. */message: string;} & {+ column?: never;expected?: never;fed?: never;+ invariant?: never;+ key?: never;limit?: never;position?: never;received?: never;request?: never;row?: never;sent?: never;+ table?: never;what?: never;})/** The first step was asked for twice. */| ({ stage: 'alreadyStarted' } & {+ column?: never;expected?: never;fed?: never;+ invariant?: never;+ key?: never;limit?: never;message?: never;position?: never;@@ -1878,6 +1921,117 @@ export type RunError =request?: never;row?: never;sent?: never;+ table?: never;+ what?: never;+ })+ /**+ * The statement names a table its catalog does not have: it was+ * resolved against another catalog.+ */+ | ({+ stage: 'tableNotInCatalog';+ /** The table. */+ table: TableId;+ } & {+ column?: never;+ expected?: never;+ fed?: never;+ invariant?: never;+ key?: never;+ limit?: never;+ message?: never;+ position?: never;+ received?: never;+ request?: never;+ row?: never;+ sent?: never;+ what?: never;+ })+ /**+ * The statement names a column its table does not have: it was+ * resolved against another catalog.+ */+ | ({+ stage: 'columnNotInTable';+ /** The table. */+ table: TableId;+ /** The column's definition. */+ column: string;+ } & {+ expected?: never;+ fed?: never;+ invariant?: never;+ key?: never;+ limit?: never;+ message?: never;+ position?: never;+ received?: never;+ request?: never;+ row?: never;+ sent?: never;+ what?: never;+ })+ /** A select item names a key nothing in the statement binds. */+ | ({+ stage: 'unboundColumn';+ /** The key. */+ key: string;+ } & {+ column?: never;+ expected?: never;+ fed?: never;+ invariant?: never;+ limit?: never;+ message?: never;+ position?: never;+ received?: never;+ request?: never;+ row?: never;+ sent?: never;+ table?: never;+ what?: never;+ })+ /** A value for a column is not of the column's type. */+ | ({+ stage: 'mismatchedValue';+ /** The column's name. */+ column: string;+ } & {+ expected?: never;+ fed?: never;+ invariant?: never;+ key?: never;+ limit?: never;+ message?: never;+ position?: never;+ received?: never;+ request?: never;+ row?: never;+ sent?: never;+ table?: never;+ what?: never;+ })+ /**+ * The engine broke one of its own invariants: a bug in the engine, not+ * a problem with the statement.+ */+ | ({+ stage: 'internal';+ /** The invariant broken. */+ invariant: string;+ } & {+ column?: never;+ expected?: never;+ fed?: never;+ key?: never;+ limit?: never;+ message?: never;+ position?: never;+ received?: never;+ request?: never;+ row?: never;+ sent?: never;+ table?: never;what?: never;});diff --git a/crates/database_sql/src/engine.rs b/crates/database_sql/src/engine.rsindex c5fdd9c85d..9b76755096 100644--- a/crates/database_sql/src/engine.rs+++ b/crates/database_sql/src/engine.rs@@ -31,7 +31,7 @@ use crate::resolve::{column_key, compile, row_id_key, row_position_key,};use crate::run::{Answer, Outcome, OutcomeColumn, PAGE_LIMIT, Page, ROW_CAP, RunError, describe};-use crate::split::{GqlQuery, KeyHint, MAX_KEY_HINT_VALUES, Plan, Shape, split};+use crate::split::{GqlQuery, KeyHint, MAX_KEY_HINT_VALUES, Plan, RelationPlan, Shape, split};use crate::write::{self, Sent};/// What the driver does next.@@ -84,8 +84,6 @@ pub struct Engine {read: Option<Read>,/// What the statement writes once its read is done.then: Option<Then>,- /// The ops sent, whose results make the outcome.- sent: Option<Sent>,requests: Requests,}@@ -93,10 +91,19 @@ pub struct Engine {/// it asked for.#[derive(Debug, Clone, Default)]struct Requests {- outstanding: Option<(u32, Answer)>,+ outstanding: Option<(u32, Awaiting)>,next: u32,}+/// What the outstanding request waits on.+#[derive(Debug, Clone)]+enum Awaiting {+ Page,+ Bins,+ /// The results of the ops sent, read back as `Sent` says.+ OpResults(Sent),+}+/// A read's progress.#[derive(Debug, Clone)]struct Read {@@ -124,39 +131,39 @@ impl Engine {pub fn start(catalog: &Catalog, sql: &str) -> Result<(Engine, Step), RunError> {let mut engine = Engine::new(catalog);let step = match compile(catalog, sql)? {- Query::Select(select) => engine.begin_read(select),+ Query::Select(select) => engine.begin_read(select)?,Query::Insert(insert) => {- let op = write::insert(catalog, &insert);- engine.send(insert.table, Sent::Rows, op)+ let op = write::insert(catalog, &insert)?;+ engine.send(insert.table, Sent::Rows, op)?}Query::AlterColumnType(alter) => {- let op = write::alter(catalog, &alter);+ let op = write::alter(catalog, &alter)?;let sent = Sent::Column {table: alter.table,column: alter.column,to: alter.to,};- engine.send(alter.table, sent, op)+ engine.send(alter.table, sent, op)?}Query::Update(update) => {let read = update.read.clone();engine.then = Some(Then::Update(update));- engine.begin_read(read)+ engine.begin_read(read)?}Query::Delete(delete) => {let read = delete.read.clone();engine.then = Some(Then::Delete(delete));- engine.begin_read(read)+ engine.begin_read(read)?}};Ok((engine, step))}/// Plan a resolved `SELECT` and ask for its first fetch.- pub fn from_select(catalog: &Catalog, select: SelectQuery) -> (Engine, Step) {+ pub fn from_select(catalog: &Catalog, select: SelectQuery) -> Result<(Engine, Step), RunError> {let mut engine = Engine::new(catalog);- let step = engine.begin_read(select);- (engine, step)+ let step = engine.begin_read(select)?;+ Ok((engine, step))}fn new(catalog: &Catalog) -> Engine {@@ -164,7 +171,6 @@ impl Engine {catalog: catalog.clone(),read: None,then: None,- sent: None,requests: Requests::default(),}}@@ -173,10 +179,13 @@ impl Engine {/// property definition, as a source reads them; the engine keys a/// joined relation's cells itself (see [`column_key`]).pub fn feed_page(&mut self, request_id: u32, page: Page) -> Result<Step, RunError> {- self.requests.answer(request_id, Answer::Page)?;- let read = reading(&mut self.read);- let keyed = keyed(&self.catalog, read, page.rows);- let rows = &mut read.fetched[read.current];+ match self.requests.answer(request_id)? {+ Awaiting::Page => {}+ awaiting => return Err(awaiting.refuse(request_id, Answer::Page)),+ }+ let read = reading(&mut self.read)?;+ let keyed = keyed(&self.catalog, read, page.rows)?;+ let rows = read.current_rows()?;rows.extend(keyed);if rows.len() >= ROW_CAP {rows.truncate(ROW_CAP);@@ -184,15 +193,18 @@ impl Engine {return self.finish_relation();}match page.next {- Some(cursor) => Ok(request(read, &mut self.requests, Some(cursor))),+ Some(cursor) => request(read, &mut self.requests, Some(cursor)),None => self.finish_relation(),}}/// Take the bins of the outstanding request.pub fn feed_bins(&mut self, request_id: u32, bins: Vec<Bin>) -> Result<Step, RunError> {- self.requests.answer(request_id, Answer::Bins)?;- let read = reading(&mut self.read);+ match self.requests.answer(request_id)? {+ Awaiting::Bins => {}+ awaiting => return Err(awaiting.refuse(request_id, Answer::Bins)),+ }+ let read = reading(&mut self.read)?;let rows = fold_bins(&self.catalog, &read.plan, bins)?;let outcome = read.outcome(rows, Vec::new());self.finish_read(outcome)@@ -200,23 +212,24 @@ impl Engine {/// Take the results of the outstanding ops, one per op.pub fn feed_ops(&mut self, request_id: u32, results: Vec<OpResult>) -> Result<Step, RunError> {- self.requests.answer(request_id, Answer::OpResults)?;- let sent = self- .sent- .take()- .expect("op results are only asked for once ops are sent");- Ok(Step::Done(write::outcome(&sent, &results)?))+ match self.requests.answer(request_id)? {+ Awaiting::OpResults(sent) => Ok(Step::Done(write::outcome(&sent, &results)?)),+ awaiting => Err(awaiting.refuse(request_id, Answer::OpResults)),+ }}/// Plan a read and ask for its first fetch.- fn begin_read(&mut self, select: SelectQuery) -> Step {+ fn begin_read(&mut self, select: SelectQuery) -> Result<Step, RunError> {+ for relation in &select.relations {+ table(&self.catalog, relation.table)?;+ }let columns = describe(&self.catalog,&select.items,&select.labels,&select.bindings,&select.relations,- );+ )?;let plan = split(&self.catalog, select);let read = self.read.insert(Read {columns,@@ -232,16 +245,17 @@ impl Engine {/// A relation is complete: give it its row ids and positions, then move/// on or fold.fn finish_relation(&mut self) -> Result<Step, RunError> {- let read = reading(&mut self.read);- let table = read.plan.relations[read.current].relation.table;+ let read = reading(&mut self.read)?;+ let table = read.current_relation()?.relation.table;let bound = |key: Uuid| binding(&read.plan.bindings, key).is_some();let (id_key, position_key) = (row_id_key(table), row_position_key(table));let (ids, positions) = (bound(id_key), bound(position_key));// Sources list rows in any order (Soup: newest first); the fold sees// them in table order, as the server's row store keeps them, so an// unordered result or its groups come out alike on both.- read.fetched[read.current].sort_by(table_order);- for row in &mut read.fetched[read.current] {+ let rows = read.current_rows()?;+ rows.sort_by(table_order);+ for row in rows {if ids {row.cells.insert(id_key, Cell::Row(row.id));}@@ -252,7 +266,7 @@ impl Engine {}read.current += 1;if read.current < read.plan.relations.len() {- return Ok(begin_relation(read, &mut self.requests));+ return begin_relation(read, &mut self.requests);}let fetched = std::mem::take(&mut read.fetched);let (rows, row_ids) = fold_relations(&self.catalog, &read.plan, fetched);@@ -287,46 +301,40 @@ impl Engine {});}let (table, op) = match &then {- Then::Update(update) => (update.table, write::update(&self.catalog, update, &found)),+ Then::Update(update) => (update.table, write::update(&self.catalog, update, &found)?),Then::Delete(delete) => (delete.table, write::delete(delete.table, &found)),};- Ok(match op {+ match op {Some(op) => self.send(table, Sent::Rows, op),- None => Step::Done(Outcome::default()),- })+ None => Ok(Step::Done(Outcome::default())),+ }}/// Ask for one op to be applied to the database `table` belongs to.- fn send(&mut self, table_id: TableId, sent: Sent, op: DatabaseOp) -> Step {- let database = table(&self.catalog, table_id).database_id;- self.sent = Some(sent);- Step::Ops {- id: self.requests.issue(Answer::OpResults),+ fn send(&mut self, table_id: TableId, sent: Sent, op: DatabaseOp) -> Result<Step, RunError> {+ let database = table(&self.catalog, table_id)?.database_id;+ Ok(Step::Ops {+ id: self.requests.issue(Awaiting::OpResults(sent)),database,ops: vec![op],- }+ })}}impl Requests {- /// A new request's id, now the one outstanding, answered by `answer`.- fn issue(&mut self, answer: Answer) -> u32 {+ /// A new request's id, now the one outstanding, waiting on `awaiting`.+ fn issue(&mut self, awaiting: Awaiting) -> u32 {let id = self.next;self.next += 1;- self.outstanding = Some((id, answer));+ self.outstanding = Some((id, awaiting));id}- /// A feed of `fed` for `request_id`, which must be the outstanding- /// request and ask for it. The request is spent either way.- fn answer(&mut self, request_id: u32, fed: Answer) -> Result<(), RunError> {+ /// What `request_id`, which must be the outstanding request, waits on.+ /// The request is spent either way.+ fn answer(&mut self, request_id: u32) -> Result<Awaiting, RunError> {match self.outstanding.take() {- Some((expected, wanted)) if expected == request_id && wanted == fed => Ok(()),- Some((expected, wanted)) if expected == request_id => Err(RunError::WrongAnswer {- request: request_id,- expected: wanted,- fed,- }),+ Some((expected, awaiting)) if expected == request_id => Ok(awaiting),Some((expected, _)) => Err(RunError::WrongRequest {expected,fed: request_id,@@ -336,7 +344,41 @@ impl Requests {}}+impl Awaiting {+ /// The refusal of `fed` as the answer to `request`, which waits on this.+ fn refuse(&self, request: u32, fed: Answer) -> RunError {+ RunError::WrongAnswer {+ request,+ expected: match self {+ Awaiting::Page => Answer::Page,+ Awaiting::Bins => Answer::Bins,+ Awaiting::OpResults(_) => Answer::OpResults,+ },+ fed,+ }+ }+}+impl Read {+ /// The plan of the relation being fetched.+ fn current_relation(&self) -> Result<&RelationPlan, RunError> {+ self.plan+ .relations+ .get(self.current)+ .ok_or(RunError::Internal {+ invariant: "the relation being fetched is in the plan",+ })+ }++ /// The rows of the relation being fetched, so far.+ fn current_rows(&mut self) -> Result<&mut Vec<Row>, RunError> {+ self.fetched+ .get_mut(self.current)+ .ok_or(RunError::Internal {+ invariant: "the relation being fetched has begun",+ })+ }+/// The rows the current relation needs: for the `FROM` table, the rows/// its `WHERE` names by id; for a joined one, the values it is joined on./// `None` past [`MAX_KEY_HINT_VALUES`].@@ -376,7 +418,9 @@ impl Read {let owner = binding(&self.plan.bindings, left)?;let mut seen: HashSet<CellKey> = HashSet::new();let mut values: Vec<Cell> = Vec::new();- for cell in self.fetched[owner.relation]+ for cell in self+ .fetched+ .get(owner.relation)?.iter().filter_map(|row| row.cells.get(&left)){@@ -414,17 +458,17 @@ impl Read {/// Start fetching the current relation, with the join's key hint filled/// in.-fn begin_relation(read: &mut Read, requests: &mut Requests) -> Step {+fn begin_relation(read: &mut Read, requests: &mut Requests) -> Result<Step, RunError> {read.fetched.push(Vec::new());- let mut query = read.plan.relations[read.current].query.clone();+ let mut query = read.current_relation()?.query.clone();if let GqlQuery::GroupSoup { .. } = query {- return Step::Bins(Request {- id: requests.issue(Answer::Bins),+ return Ok(Step::Bins(Request {+ id: requests.issue(Awaiting::Bins),query,needs: Vec::new(),cursor: None,limit: 0,- });+ }));}if let Some(hint) = read.key_hint() {match &mut query {@@ -449,31 +493,32 @@ fn begin_relation(read: &mut Read, requests: &mut Requests) -> Step {}/// The next page of the current relation.-fn request(read: &Read, requests: &mut Requests, cursor: Option<String>) -> Step {- let relation = &read.plan.relations[read.current];- let fetched = read.fetched[read.current].len();- Step::Fetch(Request {- id: requests.issue(Answer::Page),- query: read- .query- .clone()- .expect("a relation's query is set when it starts"),+fn request(read: &Read, requests: &mut Requests, cursor: Option<String>) -> Result<Step, RunError> {+ let relation = read.current_relation()?;+ let fetched = read.fetched.get(read.current).map_or(0, Vec::len);+ let query = read.query.clone().ok_or(RunError::Internal {+ invariant: "a relation's query is set when it starts",+ })?;+ Ok(Step::Fetch(Request {+ id: requests.issue(Awaiting::Page),+ query,needs: relation.needs.clone(),cursor,- limit: (ROW_CAP - fetched).min(PAGE_LIMIT),- })+ limit: ROW_CAP.saturating_sub(fetched).min(PAGE_LIMIT),+ }))}/// A joined relation's cells under the keys the plan uses for it: a cell/// keyed by one of its table's definitions moves to that column's key for/// this relation. The `FROM` relation's keys are its definitions already,/// and a cell keyed some other way is kept as is.-fn keyed(catalog: &Catalog, read: &Read, rows: Vec<Row>) -> Vec<Row> {+fn keyed(catalog: &Catalog, read: &Read, rows: Vec<Row>) -> Result<Vec<Row>, RunError> {if read.current == 0 {- return rows;+ return Ok(rows);}- let table = table(catalog, read.plan.relations[read.current].relation.table);- rows.into_iter()+ let table = table(catalog, read.current_relation()?.relation.table)?;+ Ok(rows+ .into_iter().map(|row| Row {id: row.id,position: row.position,@@ -489,16 +534,16 @@ fn keyed(catalog: &Catalog, read: &Read, rows: Vec<Row>) -> Vec<Row> {}).collect(),})- .collect()+ .collect())}/// A table the statement reads or writes.-fn table(catalog: &Catalog, id: TableId) -> &Table {+fn table(catalog: &Catalog, id: TableId) -> Result<&Table, RunError> {catalog.tables.iter().find(|table| table.id == id)- .expect("the engine resolved its statement against this catalog")+ .ok_or(RunError::TableNotInCatalog { table: id })}/// The rows of `table` a `WHERE row_id = …` or `WHERE row_id IN (…)` names.@@ -530,9 +575,10 @@ fn named_rows(table: TableId, filter: Option<&Filter>) -> Vec<RowId> {}/// The read in flight; pages and bins are only asked for while reading.-fn reading(read: &mut Option<Read>) -> &mut Read {- read.as_mut()- .expect("pages and bins are only asked for while reading")+fn reading(read: &mut Option<Read>) -> Result<&mut Read, RunError> {+ read.as_mut().ok_or(RunError::Internal {+ invariant: "pages and bins are only asked for while reading",+ })}/// Position, then id, as `database_rows` orders a table; rows without adiff --git a/crates/database_sql/src/engine/test.rs b/crates/database_sql/src/engine/test.rsindex 36bebee867..d3faaccbc8 100644--- a/crates/database_sql/src/engine/test.rs+++ b/crates/database_sql/src/engine/test.rs@@ -1054,6 +1054,17 @@ fn a_feed_of_the_wrong_kind_is_refused() {);}+#[test]+fn a_select_resolved_against_another_catalog_is_refused() {+ let Query::Select(select) = compile(&catalog(), "SELECT name FROM crm.deals").unwrap() else {+ panic!("expected a select");+ };+ assert_eq!(+ Engine::from_select(&issues_catalog(), select).map(|(_, step)| step),+ Err(RunError::TableNotInCatalog { table: DEALS })+ );+}+#[test]fn a_join_past_the_hint_limit_fetches_the_joined_relation_whole() {let assigned = |count: usize| -> Vec<Row> {diff --git a/crates/database_sql/src/fold.rs b/crates/database_sql/src/fold.rsindex 69f0d05b90..637163c496 100644--- a/crates/database_sql/src/fold.rs+++ b/crates/database_sql/src/fold.rs@@ -140,11 +140,12 @@ pub fn fold_relations(/// Give the rows of relation `index` the cells of its table's derived/// columns the plan reads, computed from their other cells.fn derive(catalog: &Catalog, plan: &Plan, index: usize, rows: &mut [Row]) {- let Some(table) = catalog- .tables- .iter()- .find(|table| table.id == plan.relations[index].relation.table)- else {+ let Some(table) = plan.relations.get(index).and_then(|relation| {+ catalog+ .tables+ .iter()+ .find(|table| table.id == relation.relation.table)+ }) else {return;};let key = |definition: Uuid| column_key(index, definition);@@ -184,7 +185,7 @@ fn fold_joined(catalog: &Catalog, plan: &Plan, rows: Vec<Row>) -> (Table, Vec<Romatch &plan.shape {Shape::Rows(columns) => {let mut rows = rows;- sort::rows(catalog, plan, &mut rows);+ sort::rows(catalog, plan, columns, &mut rows);let projected = rows.into_iter().map(|row| {let cells: Vec<Option<Cell>> = columns.iter()diff --git a/crates/database_sql/src/fold/sort.rs b/crates/database_sql/src/fold/sort.rsindex 3808de314b..72c1897db7 100644--- a/crates/database_sql/src/fold/sort.rs+++ b/crates/database_sql/src/fold/sort.rs@@ -17,17 +17,19 @@ use super::{Cell, Row, group_order_index};/// Each option's place in its column's declared order.type OptionRanks = HashMap<OptionId, usize>;-/// Sort fetched rows by column keys. A key on an item index cannot occur-/// here: a row shape has no aggregates to refer to.-pub fn rows(catalog: &Catalog, plan: &Plan, rows: &mut [Row]) {+/// Sort fetched rows of a row shape selecting `columns`. A key on an item+/// index (`ORDER BY 2`, or an `AS` label) sorts by the column that item+/// selects; resolve has checked the index is in range.+pub fn rows(catalog: &Catalog, plan: &Plan, columns: &[Uuid], rows: &mut [Row]) {let keys: Vec<(Uuid, Direction, OptionRanks)> = plan.order_by.iter()- .map(|order| {- let OrderKey::Column(column) = order.key else {- unreachable!("row shapes order by columns only");+ .filter_map(|order| {+ let column = match order.key {+ OrderKey::Column(column) => column,+ OrderKey::Item(index) => *columns.get(index)?,};- (column, order.direction, option_ranks(catalog, plan, column))+ Some((column, order.direction, option_ranks(catalog, plan, column)))}).collect();rows.sort_by(|left, right| {diff --git a/crates/database_sql/src/resolve/filter.rs b/crates/database_sql/src/resolve/filter.rsindex 257efe79e7..b861321646 100644--- a/crates/database_sql/src/resolve/filter.rs+++ b/crates/database_sql/src/resolve/filter.rs@@ -2,9 +2,9 @@//! the column's kind.use chrono::{DateTime, NaiveDate, Utc};-use models_databases::DateRange;+use models_databases::{DateRange, OptionId};-use crate::catalog::{Column, ColumnKind, EntityKind};+use crate::catalog::{Column, ColumnKind, EntityKind, SelectOption};use crate::parse::{ComparisonOperator, Condition, Literal};use super::names::Scope;@@ -193,29 +193,34 @@ pub fn typed_cell(column: &Column, lit: Literal) -> Result<Value, ResolveError>Literal::List(elements) => elements,single => vec![single],};- if !column.kind.is_multi() {- let count = elements.len();- return match <[Literal; 1]>::try_from(elements) {- Ok([single]) => typed_one(column, single),- Err(_) => Err(ResolveError::ListOnSingleValued {- column: column.name.clone(),- count,- }),- };- }- let mut options = Vec::new();- let mut entities = Vec::new();- for element in elements {- match typed_one(column, element)? {- Value::Option(id) => options.push(id),- Value::Entity(id) => entities.push(id),- other => unreachable!("multi-valued columns are select or entity: {other:?}"),+ match &column.kind {+ ColumnKind::Entity {+ multi: true,+ target,+ } => elements+ .into_iter()+ .map(|element| typed_entity(column, *target, element))+ .collect::<Result<_, _>>()+ .map(Value::Entities),+ ColumnKind::Select {+ multi: true,+ options,+ } => elements+ .into_iter()+ .map(|element| typed_option(column, options, element))+ .collect::<Result<_, _>>()+ .map(Value::Options),+ _ => {+ let count = elements.len();+ match <[Literal; 1]>::try_from(elements) {+ Ok([single]) => typed_one(column, single),+ Err(_) => Err(ResolveError::ListOnSingleValued {+ column: column.name.clone(),+ count,+ }),+ }}}- Ok(match column.kind {- ColumnKind::Entity { .. } => Value::Entities(entities),- _ => Value::Options(options),- })}fn typed_one(column: &Column, lit: Literal) -> Result<Value, ResolveError> {@@ -240,35 +245,50 @@ fn typed_one(column: &Column, lit: Literal) -> Result<Value, ResolveError> {.map(Value::DateRange).ok_or_else(|| mismatch("date range", RANGE_HINT)),(ColumnKind::DateRange, _) => Err(mismatch("date range", RANGE_HINT)),- (ColumnKind::Select { options, .. }, Literal::Text(label)) => options- .iter()- .find(|option| option.label.eq_ignore_ascii_case(&label))- .map(|option| Value::Option(option.id))- .ok_or_else(|| ResolveError::UnknownOption {- column: column.name.clone(),- label,- options: options.iter().map(|option| option.label.clone()).collect(),- }),- (ColumnKind::Select { .. }, _) => {- Err(mismatch("select", "compare it to a quoted option label"))+ (ColumnKind::Select { options, .. }, lit) => {+ typed_option(column, options, lit).map(Value::Option)}- (- ColumnKind::Entity {- target: EntityKind::Row,- ..- },- Literal::Text(id),- ) => match id.parse::<models_databases::RowId>() {- Ok(_) => Ok(Value::Entity(id)),+ (ColumnKind::Entity { target, .. }, lit) => {+ typed_entity(column, *target, lit).map(Value::Entity)+ }+ }+}++fn typed_option(+ column: &Column,+ options: &[SelectOption],+ lit: Literal,+) -> Result<OptionId, ResolveError> {+ let Literal::Text(label) = lit else {+ return Err(ResolveError::TypeMismatch {+ column: column.name.clone(),+ expected: "select",+ hint: "compare it to a quoted option label",+ });+ };+ options+ .iter()+ .find(|option| option.label.eq_ignore_ascii_case(&label))+ .map(|option| option.id)+ .ok_or_else(|| ResolveError::UnknownOption {+ column: column.name.clone(),+ label,+ options: options.iter().map(|option| option.label.clone()).collect(),+ })+}++fn typed_entity(column: &Column, target: EntityKind, lit: Literal) -> Result<String, ResolveError> {+ match (target, lit) {+ (EntityKind::Row, Literal::Text(id)) => match id.parse::<models_databases::RowId>() {+ Ok(_) => Ok(id),Err(_) => Err(ResolveError::RowIdNotAnId { written: id }),},- (ColumnKind::Entity { .. }, Literal::Text(id)) if is_entity_id(&id) => {- Ok(Value::Entity(id))- }- (ColumnKind::Entity { .. }, _) => Err(mismatch(- "entity",- "give an id like 'macro|[email protected]', not a name",- )),+ (_, Literal::Text(id)) if is_entity_id(&id) => Ok(id),+ _ => Err(ResolveError::TypeMismatch {+ column: column.name.clone(),+ expected: "entity",+ hint: "give an id like 'macro|[email protected]', not a name",+ }),}}diff --git a/crates/database_sql/src/run.rs b/crates/database_sql/src/run.rsindex 4da8d0314d..e566f15a7f 100644--- a/crates/database_sql/src/run.rs+++ b/crates/database_sql/src/run.rs@@ -199,6 +199,45 @@ pub enum RunError {/// The first step was asked for twice.#[error("the query has already started")]AlreadyStarted,+ /// The statement names a table its catalog does not have: it was+ /// resolved against another catalog.+ #[error("table {table} is not in the catalog the statement was resolved against")]+ TableNotInCatalog {+ /// The table.+ table: TableId,+ },+ /// The statement names a column its table does not have: it was+ /// resolved against another catalog.+ #[error(+ "table {table} has no column {column}; the statement was resolved against another catalog"+ )]+ ColumnNotInTable {+ /// The table.+ table: TableId,+ /// The column's definition.+ column: Uuid,+ },+ /// A select item names a key nothing in the statement binds.+ #[error("select item {key} is bound to no column of the statement")]+ UnboundColumn {+ /// The key.+ key: Uuid,+ },+ /// A value for a column is not of the column's type.+ #[error("a value written to \"{column}\" is not of that column's type")]+ MismatchedValue {+ /// The column's name.+ column: String,+ },+ /// The engine broke one of its own invariants: a bug in the engine, not+ /// a problem with the statement.+ #[error(+ "the query engine failed internally ({invariant}); this is a bug, not a problem with the statement"+ )]+ Internal {+ /// The invariant broken.+ invariant: &'static str,+ },}/// A failure as it crosses the wasm boundary: the typed error, and the words@@ -561,7 +600,10 @@ pub async fn run<Source: RowSource, Sink: OpsSink>(/// The columns a resolved `SELECT` returns, named and typed as its outcome/// will be.-pub fn result_columns(catalog: &Catalog, select: &SelectQuery) -> Vec<OutcomeColumn> {+pub fn result_columns(+ catalog: &Catalog,+ select: &SelectQuery,+) -> Result<Vec<OutcomeColumn>, RunError> {describe(catalog,&select.items,@@ -578,23 +620,24 @@ pub(crate) fn describe(labels: &[(usize, String)],bindings: &[Binding],relations: &[Relation],-) -> Vec<OutcomeColumn> {+) -> Result<Vec<OutcomeColumn>, RunError> {let column = |key: Uuid| {column_of(catalog, bindings, relations, key).cloned().or_else(|| virtual_column_of(bindings, relations, key))- .expect("select items are bound in the scope")+ .ok_or(RunError::UnboundColumn { key })};let mut columns: Vec<OutcomeColumn> = items.iter().map(|item| match item {SelectItem::Column(key) => {- let column = column(*key);+ let column = column(*key)?;let bound = binding(bindings, *key);let row_table = bound- .map(|bound| relations[bound.relation].table)+ .and_then(|bound| relations.get(bound.relation))+ .map(|relation| relation.table).filter(|table| *key == row_id_key(*table));- OutcomeColumn {+ Ok(OutcomeColumn {column: bound.and_then(|bound| bound.column),table: row_table,kind: match column.kind {@@ -608,23 +651,23 @@ pub(crate) fn describe(ColumnKind::Entity { .. } => OutcomeKind::Entity,},name: column.name,- }+ })}SelectItem::Aggregate {function,column: None,- } => OutcomeColumn {+ } => Ok(OutcomeColumn {name: format!("{}(*)", function.name()),column: None,kind: OutcomeKind::Number,table: None,- },+ }),SelectItem::Aggregate {function,column: Some(key),} => {- let column = column(*key);- OutcomeColumn {+ let column = column(*key)?;+ Ok(OutcomeColumn {name: format!("{}({})", function.name(), column.name),column: None,kind: match (function, &column.kind) {@@ -634,14 +677,14 @@ pub(crate) fn describe(_ => OutcomeKind::Number,},table: None,- }+ })}})- .collect();+ .collect::<Result<_, RunError>>()?;for (index, label) in labels {if let Some(column) = columns.get_mut(*index) {column.name = label.clone();}}- columns+ Ok(columns)}diff --git a/crates/database_sql/src/split.rs b/crates/database_sql/src/split.rsindex e3f27a131b..4bc550c699 100644--- a/crates/database_sql/src/split.rs+++ b/crates/database_sql/src/split.rs@@ -277,10 +277,11 @@ pub fn column_of<'catalog>() -> Option<&'catalog Column> {let binding = binding(bindings, key)?;let definition = binding.column?;+ let table_id = relations.get(binding.relation)?.as_ref().table;catalog.tables.iter()- .find(|table| table.id == relations[binding.relation].as_ref().table)?+ .find(|table| table.id == table_id)?.columns.iter().find(|column| column.id == definition)@@ -297,7 +298,7 @@ pub fn virtual_column_of(if binding.column.is_some() {return None;}- crate::resolve::virtual_column(relations[binding.relation].table, key)+ crate::resolve::virtual_column(relations.get(binding.relation)?.table, key)}impl AsRef<Relation> for RelationPlan {diff --git a/crates/database_sql/src/split/pushdown.rs b/crates/database_sql/src/split/pushdown.rsindex 36037d70c7..e3862e1df8 100644--- a/crates/database_sql/src/split/pushdown.rs+++ b/crates/database_sql/src/split/pushdown.rs@@ -96,7 +96,7 @@ fn pushable(}});let relation = relation?;- if spans || relations[relation].source != TableSource::Database {+ if spans || relations.get(relation)?.source != TableSource::Database {return None;}Some((relation, push(filter, bindings)?))diff --git a/crates/database_sql/src/view.rs b/crates/database_sql/src/view.rsindex 61df62f044..91ed364d8c 100644--- a/crates/database_sql/src/view.rs+++ b/crates/database_sql/src/view.rs@@ -44,11 +44,11 @@ fn schema_column(column: &Column) -> SchemaColumn {)}-/// The column a checked view names by placement.-fn placed(table: &Table, placement: ColumnId) -> &Column {+/// The column a view names by placement.+fn placed(table: &Table, placement: ColumnId) -> Result<&Column, ViewProblem> {table.columns.iter().find(|column| column.placement == placement)- .expect("the view was checked against this table")+ .ok_or(ViewProblem::UnknownColumn { column: placement })}diff --git a/crates/database_sql/src/view/board.rs b/crates/database_sql/src/view/board.rsindex 8966835a58..88bf15cb17 100644--- a/crates/database_sql/src/view/board.rs+++ b/crates/database_sql/src/view/board.rs@@ -3,6 +3,8 @@#[cfg(test)]mod test;+use std::collections::HashMap;+use models_databases::RowId;use models_databases::position::Position;use models_databases::views::{@@ -56,7 +58,7 @@ pub fn board(else {return Err(ViewProblem::NotABoard);};- let grouping = placed(table, *group_by);+ let grouping = placed(table, *group_by)?;let cell_index = outcome.columns.iter()@@ -69,21 +71,23 @@ pub fn board(.collect();let order = lane_order(&grouping.kind, listed, &card_lanes);- let mut cards: Vec<Vec<RowId>> = vec![Vec::new(); order.len()];- for (row, lane) in outcome.row_ids.iter().zip(&card_lanes) {- let lane = order- .iter()- .position(|key| key == lane)- .or_else(|| order.iter().position(|key| *key == LaneKey::None))- .expect("every board has the lane of empty cells");- cards[lane].push(*row);+ // A card in no lane of the order (an option since removed) shows in the+ // lane of empty cells, which every order has.+ let mut cards: HashMap<LaneKey, Vec<RowId>> = HashMap::new();+ for (row, lane) in outcome.row_ids.iter().zip(card_lanes) {+ let lane = if order.contains(&lane) {+ lane+ } else {+ LaneKey::None+ };+ cards.entry(lane).or_default().push(*row);}let sorted = !view.query.sort.is_empty();let lanes = order.into_iter()- .zip(cards)- .map(|(key, cards)| {+ .map(|key| {+ let cards = cards.remove(&key).unwrap_or_default();let cards = if sorted {cards} else {diff --git a/crates/database_sql/src/view/compile.rs b/crates/database_sql/src/view/compile.rsindex b7811d9784..15e059d97b 100644--- a/crates/database_sql/src/view/compile.rs+++ b/crates/database_sql/src/view/compile.rs@@ -9,7 +9,7 @@ use models_databases::views::{TextOperator, ViewProblem,};-use crate::catalog::{Catalog, Table};+use crate::catalog::{Catalog, Column, Table};use crate::resolve::{Binding, ComparisonOperator, Direction, Filter, Order, OrderKey, Relation, SelectItem,SelectQuery, Value, row_position_key,@@ -23,11 +23,14 @@ const LIKE_ESCAPE: char = '\\';/// The `SELECT * … WHERE … ORDER BY …, row_position` a view shows, exactly as/// [`resolve`](crate::resolve::resolve) would bind it.pub fn compile_view(view: &DatabaseView, catalog: &Catalog) -> Result<SelectQuery, ViewProblem> {- Ok(compile_checked(view, checked_table(view, catalog)?))+ compile_checked(view, checked_table(view, catalog)?)}/// [`compile_view`] once the view has checked out against `table`.-pub(super) fn compile_checked(view: &DatabaseView, table: &Table) -> SelectQuery {+pub(super) fn compile_checked(+ view: &DatabaseView,+ table: &Table,+) -> Result<SelectQuery, ViewProblem> {let position = row_position_key(table.id);let mut bindings: Vec<Binding> = table.columns@@ -47,19 +50,25 @@ pub(super) fn compile_checked(view: &DatabaseView, table: &Table) -> SelectQuery.query.sort.iter()- .map(|key| Order {- key: OrderKey::Column(placed(table, key.column).id),- direction: match key.direction {- SortDirection::Ascending => Direction::Ascending,- SortDirection::Descending => Direction::Descending,- },+ .map(|key| {+ Ok(Order {+ key: OrderKey::Column(placed(table, key.column)?.id),+ direction: match key.direction {+ SortDirection::Ascending => Direction::Ascending,+ SortDirection::Descending => Direction::Descending,+ },+ })})- .chain(std::iter::once(Order {+ .chain(std::iter::once(Ok(Order {key: OrderKey::Column(position),direction: Direction::Ascending,- }))- .collect();- SelectQuery {+ })))+ .collect::<Result<_, ViewProblem>>()?;+ let where_ = match &view.query.filter {+ Some(group) => group_filter(table, group)?,+ None => None,+ };+ Ok(SelectQuery {distinct: false,relations: vec![Relation {table: table.id,@@ -73,34 +82,30 @@ pub(super) fn compile_checked(view: &DatabaseView, table: &Table) -> SelectQuery.map(|column| SelectItem::Column(column.id)).collect(),labels: vec![],- where_: view- .query- .filter- .as_ref()- .and_then(|group| group_filter(table, group)),+ where_,group_by: None,order_by,limit: None,offset: None,bindings,- }+ })}/// A group's filter; `None` when it keeps every row.-fn group_filter(table: &Table, group: &FilterGroup) -> Option<Filter> {+fn group_filter(table: &Table, group: &FilterGroup) -> Result<Option<Filter>, ViewProblem> {let mut parts = Vec::new();for node in &group.conditions {let part = match node {- FilterNode::Condition(condition) => Some(condition_filter(table, condition)),- FilterNode::Group(group) => group_filter(table, group),+ FilterNode::Condition(condition) => Some(condition_filter(table, condition)?),+ FilterNode::Group(group) => group_filter(table, group)?,};match (part, group.conjunction) {(Some(part), _) => parts.push(part),(None, Conjunction::And) => {}- (None, Conjunction::Or) => return None,+ (None, Conjunction::Or) => return Ok(None),}}- joined(group.conjunction, parts)+ Ok(joined(group.conjunction, parts))}/// The parts under one conjunction: one part stands alone, none keeps every@@ -114,8 +119,9 @@ fn joined(conjunction: Conjunction, mut parts: Vec<Filter>) -> Option<Filter> {}}-fn condition_filter(table: &Table, condition: &FilterCondition) -> Filter {- let column = placed(table, condition.column).id;+fn condition_filter(table: &Table, condition: &FilterCondition) -> Result<Filter, ViewProblem> {+ let placed = placed(table, condition.column)?;+ let column = placed.id;let compare = |operator, value| Filter::Comparison {column,operator,@@ -136,7 +142,7 @@ fn condition_filter(table: &Table, condition: &FilterCondition) -> Filter {escape: Some(LIKE_ESCAPE),negated,};- match &condition.test {+ let filter = match &condition.test {FilterTest::Presence { operator } => Filter::IsNull {column,negated: *operator == PresenceOperator::IsNotEmpty,@@ -208,27 +214,33 @@ fn condition_filter(table: &Table, condition: &FilterCondition) -> Filter {or_empty(compare(ComparisonOperator::Equal, Value::Bool(false)))}FilterTest::Options { operator, options } => set_filter(- column,+ placed,*operator,options.iter().map(|option| Value::Option(*option)).collect(),- ),+ )?,FilterTest::Entities { operator, entities } => set_filter(- column,+ placed,*operator,entities.iter().map(|entity| Value::Entity(entity.clone())).collect(),- ),- }+ )?,+ };+ Ok(filter)}/// A set test. An absent multi-valued cell has no member, so `NOT HAS`/// already keeps it; only `NOT IN` needs the empty cell named.-fn set_filter(column: uuid::Uuid, operator: SetOperator, values: Vec<Value>) -> Filter {+fn set_filter(+ placed: &Column,+ operator: SetOperator,+ values: Vec<Value>,+) -> Result<Filter, ViewProblem> {+ let column = placed.id;let has = |negated| {values.iter()@@ -241,14 +253,14 @@ fn set_filter(column: uuid::Uuid, operator: SetOperator, values: Vec<Value>) ->};let members = match operator {SetOperator::IsAnyOf => {- return Filter::In {+ return Ok(Filter::In {column,values,negated: false,- };+ });}SetOperator::IsNoneOf => {- return Filter::Or(vec![+ return Ok(Filter::Or(vec![Filter::In {column,values,@@ -258,13 +270,15 @@ fn set_filter(column: uuid::Uuid, operator: SetOperator, values: Vec<Value>) ->column,negated: false,},- ]);+ ]));}SetOperator::HasAny => joined(Conjunction::Or, has(false)),SetOperator::HasAll => joined(Conjunction::And, has(false)),SetOperator::HasNone => joined(Conjunction::And, has(true)),};- members.expect("the view check refuses a set test naming nothing")+ members.ok_or_else(|| ViewProblem::NothingToMatch {+ column: placed.name.clone(),+ })}/// Text matched literally inside a `LIKE` pattern.diff --git a/crates/database_sql/src/wasm.rs b/crates/database_sql/src/wasm.rsindex 4c884d1d70..ce617b2e1f 100644--- a/crates/database_sql/src/wasm.rs+++ b/crates/database_sql/src/wasm.rs@@ -117,7 +117,9 @@ pub fn run_view(catalog: JsValue, view: JsValue) -> Result<Query, JsValue> {let catalog: Catalog = read(Input::Catalog, catalog)?;let view: DatabaseView = read(Input::View, view)?;let select = view::compile_view(&view, &catalog).map_err(|problem| thrown(problem.into()))?;- Ok(Query::held(Engine::from_select(&catalog, select)))+ Engine::from_select(&catalog, select)+ .map(Query::held)+ .map_err(thrown)}/// The `Board` a board view makes of `outcome`, what its `runView` query@@ -227,11 +229,13 @@ fn read<Value: DeserializeOwned>(what: Input, value: JsValue) -> Result<Value, J})}-/// The error as the value a driver catches.+/// The error as the value a driver catches; a plain `Error` with its+/// message should it not serialize.fn thrown(error: RunError) -> JsValue {- EngineError::from(error)+ let error = EngineError::from(error);+ error.serialize(&Serializer::json_compatible())- .expect("an EngineError is strings, numbers and ids, which always serialize")+ .unwrap_or_else(|failure| JsError::new(&format!("{} ({failure})", error.message)).into())}/// Plain objects and arrays, as JSON would give them: maps become objectsdiff --git a/crates/database_sql/src/write.rs b/crates/database_sql/src/write.rsindex e6ba24ea3a..f711e88155 100644--- a/crates/database_sql/src/write.rs+++ b/crates/database_sql/src/write.rs@@ -30,9 +30,9 @@ pub(crate) enum Sent {}/// `INSERT`: every row in one op. Labels name options the column has.-pub(crate) fn insert(catalog: &Catalog, query: &InsertQuery) -> DatabaseOp {- let table = table(catalog, query.table);- DatabaseOp::Rows {+pub(crate) fn insert(catalog: &Catalog, query: &InsertQuery) -> Result<DatabaseOp, RunError> {+ let table = table(catalog, query.table)?;+ Ok(DatabaseOp::Rows {table: query.table,change: RowsChange::Insert {rows: query@@ -42,17 +42,17 @@ pub(crate) fn insert(catalog: &Catalog, query: &InsertQuery) -> DatabaseOp {cells.iter().map(|(definition, value)| {- let column = column(table, *definition);- CellWrite {+ let column = column(table, *definition)?;+ Ok(CellWrite {column: column.placement,- value: cell_value(column, Some(value)),- }+ value: cell_value(column, Some(value))?,+ })}).collect()})- .collect(),+ .collect::<Result<_, RunError>>()?,},- }+ })}/// `UPDATE` of the rows its read found: the same cells for every row when@@ -62,46 +62,49 @@ pub(crate) fn update(catalog: &Catalog,query: &UpdateQuery,found: &Outcome,-) -> Option<DatabaseOp> {+) -> Result<Option<DatabaseOp>, RunError> {if found.row_ids.is_empty() {- return None;+ return Ok(None);}- let table = table(catalog, query.table);+ let table = table(catalog, query.table)?;let uniform = query.assignments.iter().all(|assignment| matches!(assignment.value, Assigned::Value(_)));- let cells = |row: &[Option<Cell>]| -> Vec<CellWrite> {+ let cells = |row: &[Option<Cell>]| -> Result<Vec<CellWrite>, RunError> {query.assignments.iter().map(|assignment| {- let target = column(table, assignment.column);+ let target = column(table, assignment.column)?;let value = match &assignment.value {- Assigned::Value(value) => cell_value(target, value.as_ref()),+ Assigned::Value(value) => cell_value(target, value.as_ref())?,Assigned::Column(definition) => {- let item = query+ let copied = query.read.items.iter().position(|item| {*item == SelectItem::Column(column_key(0, *definition))})- .expect("the read selects every copied column");- copied_value(target, column(table, *definition), row[item].as_ref())+ .and_then(|item| row.get(item))+ .ok_or(RunError::Internal {+ invariant: "an update's read selects every copied column",+ })?;+ copied_value(target, column(table, *definition)?, copied.as_ref())?}};- CellWrite {+ Ok(CellWrite {column: target.placement,value,- }+ })}).collect()};let changes = if uniform {RowChanges::Uniform {rows: found.row_ids.clone(),- cells: cells(&[]),+ cells: cells(&[])?,}} else {RowChanges::PerRow {@@ -109,17 +112,19 @@ pub(crate) fn update(.row_ids.iter().zip(&found.rows)- .map(|(row, values)| RowChange {- row: *row,- cells: cells(values),+ .map(|(row, values)| {+ Ok(RowChange {+ row: *row,+ cells: cells(values)?,+ })})- .collect(),+ .collect::<Result<_, RunError>>()?,}};- Some(DatabaseOp::Rows {+ Ok(Some(DatabaseOp::Rows {table: query.table,change: RowsChange::Update { changes },- })+ }))}/// `DELETE` of the rows its read found; `None` when it found nothing.@@ -133,12 +138,15 @@ pub(crate) fn delete(table: TableId, found: &Outcome) -> Option<DatabaseOp> {}/// `ALTER COLUMN … TYPE`.-pub(crate) fn alter(catalog: &Catalog, query: &AlterColumnTypeQuery) -> DatabaseOp {- DatabaseOp::Column {+pub(crate) fn alter(+ catalog: &Catalog,+ query: &AlterColumnTypeQuery,+) -> Result<DatabaseOp, RunError> {+ Ok(DatabaseOp::Column {table: query.table,- column: column(table(catalog, query.table), query.column).placement,+ column: column(table(catalog, query.table)?, query.column)?.placement,change: ColumnChange::ChangeType { to: query.to },- }+ })}/// The outcome the results of what was sent make.@@ -194,48 +202,55 @@ pub(crate) fn outcome(sent: &Sent, results: &[OpResult]) -> Result<Outcome, RunE}}-fn table(catalog: &Catalog, id: TableId) -> &Table {+fn table(catalog: &Catalog, id: TableId) -> Result<&Table, RunError> {catalog.tables.iter().find(|table| table.id == id)- .expect("the statement was resolved against this catalog")+ .ok_or(RunError::TableNotInCatalog { table: id })}-fn column(table: &Table, definition: Uuid) -> &Column {+fn column(table: &Table, definition: Uuid) -> Result<&Column, RunError> {table.columns.iter().find(|column| column.id == definition)- .expect("the statement was resolved against this catalog")+ .ok_or(RunError::ColumnNotInTable {+ table: table.id,+ column: definition,+ })}/// A typed literal as the cell value it writes; `None` clears the cell.-fn cell_value(column: &Column, value: Option<&Value>) -> CellValue {+fn cell_value(column: &Column, value: Option<&Value>) -> Result<CellValue, RunError> {let Some(value) = value else {- return CellValue::Clear;+ return Ok(CellValue::Clear);};- match value {+ Ok(match value {Value::Text(text) if column.kind == ColumnKind::Link => CellValue::Link(vec![text.clone()]),Value::Text(text) => CellValue::Text(text.clone()),Value::Number(number) => CellValue::Number(*number),Value::Bool(checked) => CellValue::Boolean(*checked),Value::Date(date) => CellValue::Date(*date),Value::DateRange(range) => CellValue::DateRange(*range),- Value::Option(id) => options(column, std::slice::from_ref(id)),- Value::Options(ids) => options(column, ids),- Value::Entity(id) => references(column, std::slice::from_ref(id)),- Value::Entities(ids) => references(column, ids),- }+ Value::Option(id) => options(column, std::slice::from_ref(id))?,+ Value::Options(ids) => options(column, ids)?,+ Value::Entity(id) => references(column, std::slice::from_ref(id))?,+ Value::Entities(ids) => references(column, ids)?,+ })}/// Another column's cell, as the value it writes into `target`; the resolver/// allows only columns of the same kind.-fn copied_value(target: &Column, source: &Column, cell: Option<&Cell>) -> CellValue {+fn copied_value(+ target: &Column,+ source: &Column,+ cell: Option<&Cell>,+) -> Result<CellValue, RunError> {let Some(cell) = cell else {- return CellValue::Clear;+ return Ok(CellValue::Clear);};- match cell {+ Ok(match cell {Cell::Text(text) if target.kind == ColumnKind::Link => {CellValue::Link(text.split_whitespace().map(str::to_owned).collect())}@@ -244,21 +259,23 @@ fn copied_value(target: &Column, source: &Column, cell: Option<&Cell>) -> CellVaCell::Bool(checked) => CellValue::Boolean(*checked),Cell::Date(date) => CellValue::Date(*date),Cell::DateRange(range) => CellValue::DateRange(*range),- Cell::Options(ids) => options(source, ids),- Cell::Entities(ids) => references(target, ids),- Cell::Row(id) => references(target, &[id.to_string()]),- }+ Cell::Options(ids) => options(source, ids)?,+ Cell::Entities(ids) => references(target, ids)?,+ Cell::Row(id) => references(target, &[id.to_string()])?,+ })}/// Options by label, the way a statement names them.-fn options(column: &Column, ids: &[OptionId]) -> CellValue {+fn options(column: &Column, ids: &[OptionId]) -> Result<CellValue, RunError> {let ColumnKind::Select {options: labels, ..} = &column.kindelse {- unreachable!("only select columns hold options")+ return Err(RunError::MismatchedValue {+ column: column.name.clone(),+ });};- CellValue::Options(+ Ok(CellValue::Options(ids.iter().map(|id| {labels@@ -269,15 +286,17 @@ fn options(column: &Column, ids: &[OptionId]) -> CellValue {})}).collect(),- )+ ))}/// Entity references, or related rows for a relation.-fn references(column: &Column, ids: &[String]) -> CellValue {+fn references(column: &Column, ids: &[String]) -> Result<CellValue, RunError> {let ColumnKind::Entity { target, .. } = column.kind else {- unreachable!("only entity columns hold references")+ return Err(RunError::MismatchedValue {+ column: column.name.clone(),+ });};- match OpEntityKind::try_from(target) {+ Ok(match OpEntityKind::try_from(target) {Ok(entity_type) => CellValue::Entities(ids.iter().map(|id| EntityRef {@@ -291,5 +310,5 @@ fn references(column: &Column, ids: &[String]) -> CellValue {.filter_map(|id| id.parse::<RowId>().ok()).collect(),),- }+ })}diff --git a/crates/databases/src/domain/service/column_types.rs b/crates/databases/src/domain/service/column_types.rsindex 04050059b8..7b22abf166 100644--- a/crates/databases/src/domain/service/column_types.rs+++ b/crates/databases/src/domain/service/column_types.rs@@ -146,22 +146,46 @@ impl<'a> Converter<'a> {if values.len() > 1 && !self.target_is_multi() {return Err(Misfit::SeveralValues);}- let cell = if takes_options(self.target.data_type) {- ConvertedCell::Options(- values- .into_iter()- .map(|value| self.label(value))- .collect::<Result<_, _>>()?,- )- } else if self.target.data_type == DataType::Link {- ConvertedCell::Value(PropertyValue::Link(- values.into_iter().map(url).collect::<Result<_, _>>()?,- ))- } else {- let value = values.into_iter().next().ok_or(Misfit::NotOption)?;- ConvertedCell::Value(self.scalar(value)?)+ let value = match self.target.data_type {+ DataType::SelectString | DataType::SelectNumber | DataType::Tag => {+ return Ok(ConvertedCell::Options(+ values+ .into_iter()+ .map(|value| self.label(value))+ .collect::<Result<_, _>>()?,+ ));+ }+ DataType::Link => {+ PropertyValue::Link(values.into_iter().map(url).collect::<Result<_, _>>()?)+ }+ DataType::String => text(single(values)?)+ .map(PropertyValue::Str)+ .ok_or(Misfit::NotText)?,+ DataType::Number => number(single(values)?).map(PropertyValue::Num)?,+ DataType::Boolean => match single(values)? {+ PropertyValue::Bool(value) => PropertyValue::Bool(value),+ PropertyValue::Str(value) if value == "true" || value == "false" => {+ PropertyValue::Bool(value == "true")+ }+ _ => return Err(Misfit::NotCheckbox),+ },+ DataType::Date => match single(values)? {+ PropertyValue::Date(value) => PropertyValue::Date(value),+ PropertyValue::Str(value) => date(&value).map(PropertyValue::Date)?,+ _ => return Err(Misfit::NotDate),+ },+ DataType::DateRange => match single(values)? {+ PropertyValue::DateRange(range) => PropertyValue::DateRange(range),+ PropertyValue::Date(date) => PropertyValue::DateRange(DateRange::at(date)),+ PropertyValue::Str(value) => date_range(&value).map(PropertyValue::DateRange)?,+ _ => return Err(Misfit::NotDateRange),+ },+ DataType::Entity => {+ single(values)?;+ return Err(Misfit::OtherReference);+ }};- Ok(cell)+ Ok(ConvertedCell::Value(value))}fn convert_references(@@ -237,36 +261,6 @@ impl<'a> Converter<'a> {Ok(label)}- /// One value as a text, number, checkbox or date cell.- fn scalar(&self, value: PropertyValue) -> Result<PropertyValue, Misfit> {- match self.target.data_type {- DataType::String => text(value).map(PropertyValue::Str).ok_or(Misfit::NotText),- DataType::Number => number(value).map(PropertyValue::Num),- DataType::Boolean => match value {- PropertyValue::Bool(value) => Ok(PropertyValue::Bool(value)),- PropertyValue::Str(value) if value == "true" || value == "false" => {- Ok(PropertyValue::Bool(value == "true"))- }- _ => Err(Misfit::NotCheckbox),- },- DataType::Date => match value {- PropertyValue::Date(value) => Ok(PropertyValue::Date(value)),- PropertyValue::Str(value) => date(&value).map(PropertyValue::Date),- _ => Err(Misfit::NotDate),- },- DataType::DateRange => match value {- PropertyValue::DateRange(range) => Ok(PropertyValue::DateRange(range)),- PropertyValue::Date(date) => Ok(PropertyValue::DateRange(DateRange::at(date))),- PropertyValue::Str(value) => date_range(&value).map(PropertyValue::DateRange),- _ => Err(Misfit::NotDateRange),- },- DataType::Entity => Err(Misfit::OtherReference),- DataType::Link | DataType::SelectString | DataType::SelectNumber | DataType::Tag => {- unreachable!("options and links are converted by convert")- }- }- }-/// A cell's value as a refusal quotes it.fn example(&self, value: &PropertyValue) -> String {let text = match value {@@ -299,6 +293,11 @@ pub(super) fn is_empty(value: &PropertyValue) -> bool {}/// A number, written exactly: no padding, rounding or leading zeros.+/// The one value a scalar type takes.+fn single(values: Vec<PropertyValue>) -> Result<PropertyValue, Misfit> {+ values.into_iter().next().ok_or(Misfit::NotOption)+}+fn number(value: PropertyValue) -> Result<f64, Misfit> {match value {PropertyValue::Num(value) if value.is_finite() => Ok(value),diff --git a/crates/databases/src/domain/service/storage.rs b/crates/databases/src/domain/service/storage.rsindex 9a5201ae6d..44937f57ae 100644--- a/crates/databases/src/domain/service/storage.rs+++ b/crates/databases/src/domain/service/storage.rs@@ -173,7 +173,10 @@ where.take(PAGE_SIZE as usize).copied().collect();- let next = (start + page.len() < ordered.len()).then(|| *page.last().unwrap());+ let next = page+ .last()+ .copied()+ .filter(|_| start + page.len() < ordered.len());let rows = page.into_iter().map(|row_id| {diff --git a/crates/databases/src/domain/starter.rs b/crates/databases/src/domain/starter.rsindex efa350d45d..a84aba5003 100644--- a/crates/databases/src/domain/starter.rs+++ b/crates/databases/src/domain/starter.rs@@ -3,7 +3,7 @@use chrono::{DateTime, Utc};use macro_event_broker::MacroEventBroker;use models_databases::OptionId;-use models_databases::position::{Position, PositionError, keys_between};+use models_databases::position::{Position, PositionError, key_between};use models_databases::views::{Lane, LaneKey, ViewLayout, ViewQuery};use serde::Serialize;@@ -77,9 +77,8 @@ impl StarterBlueprint {stage_options: &[OptionId],now: DateTime<Utc>,) -> Result<[DatabaseView; 2], PositionError> {- let [table_position, board_position] = keys_between(None, None, 2)?- .try_into()- .expect("two keys were asked for");+ let board_position = key_between(None, None)?;+ let table_position = key_between(None, Some(&board_position))?;let view = |name: &str, position: Position, layout: ViewLayout| DatabaseView {id: ViewId::new(),database_id: self.database_id,diff --git a/crates/databases_sql/src/service.rs b/crates/databases_sql/src/service.rsindex 921e6232b3..2938f51d03 100644--- a/crates/databases_sql/src/service.rs+++ b/crates/databases_sql/src/service.rs@@ -230,7 +230,7 @@ where};if let Some(chart) = chart {check_chart(- &database_sql::result_columns(catalog.catalog(), &select),+ &database_sql::result_columns(catalog.catalog(), &select)?,chart,)?;}diff --git a/crates/databases_sql/src/service/schema.rs b/crates/databases_sql/src/service/schema.rsindex b10ecd5af6..6acd903518 100644--- a/crates/databases_sql/src/service/schema.rs+++ b/crates/databases_sql/src/service/schema.rs@@ -142,33 +142,34 @@ impl<if self.read_only {return Err(refused("This SQL surface is read-only."));}- if let SchemaStatement::CreateDatabase(name) = command {- let created = self- .databases- .create_database(CreateDatabase {- name: name.0,- owner_id: viewer.user_id,- acting_bot: viewer.acting_bot,- })- .await- .map_err(schema_error)?;- return Ok(outcome(- created.id,- format!(- "Created database \"{}\". Call DescribeDatabase for its starter table and column IDs.",- created.name- ),- HashMap::new(),- ));- }- let catalog = self.catalog(&viewer, request.scope).await?;+ // Read only by the statements that change an existing database.+ let catalog = self.catalog(&viewer, request.scope);let mut ops = Vec::new();let (database_id, summary, rename) = match command {- SchemaStatement::CreateDatabase(_) => unreachable!("handled before catalog read"),+ SchemaStatement::CreateDatabase(name) => {+ let created = self+ .databases+ .create_database(CreateDatabase {+ name: name.0,+ owner_id: viewer.user_id.clone(),+ acting_bot: viewer.acting_bot,+ })+ .await+ .map_err(schema_error)?;+ return Ok(outcome(+ created.id,+ format!(+ "Created database \"{}\". Call DescribeDatabase for its starter table and column IDs.",+ created.name+ ),+ HashMap::new(),+ ));+ }SchemaStatement::RenameDatabase {database: name,name: new_name,} => {+ let catalog = catalog.await?;let database = database(&catalog, Some(&name), request.scope)?;(database.database.id,@@ -180,6 +181,7 @@ impl<database: name,tables,} => {+ let catalog = catalog.await?;let database = database(&catalog, Some(&name), request.scope)?;let selected =ViewerCatalog::new(vec![database.clone()], Some(database.database.id));@@ -203,6 +205,7 @@ impl<table: name,columns,} => {+ let catalog = catalog.await?;let database = database(&catalog, name.database.as_ref(), request.scope)?;let table = TableId::new();ops.push(DatabaseOp::Table {@@ -245,6 +248,7 @@ impl<)}SchemaStatement::DropTable(name) => {+ let catalog = catalog.await?;let table = table(catalog.catalog(), &name)?;ops.push(DatabaseOp::Table {table: table.id,@@ -260,6 +264,7 @@ impl<table: name,change,} => {+ let catalog = catalog.await?;let table = table(catalog.catalog(), &name)?;let op = match change {SchemaChange::Rename(name) => DatabaseOp::Table {diff --git a/crates/databases_sql/src/toolset.rs b/crates/databases_sql/src/toolset.rsindex 87e8e5e3f5..e7804aba48 100644--- a/crates/databases_sql/src/toolset.rs+++ b/crates/databases_sql/src/toolset.rs@@ -119,7 +119,12 @@ fn sql_error(error: SqlError) -> ToolCallError {establish that the user's table is missing.")}- SqlError::Run(RunError::NoSuchRow { .. } | RunError::TooManyRows { .. }) => {+ SqlError::Run(+ RunError::NoSuchRow { .. }+ | RunError::TooManyRows { .. }+ | RunError::MismatchedValue { .. }+ | RunError::Internal { .. },+ ) => {format!("SQL error: {error}")}SqlError::Run(@@ -131,7 +136,10 @@ fn sql_error(error: SqlError) -> ToolCallError {| RunError::NothingOutstanding { .. }| RunError::Unreadable { .. }| RunError::Unwritable { .. }- | RunError::AlreadyStarted,+ | RunError::AlreadyStarted+ | RunError::TableNotInCatalog { .. }+ | RunError::ColumnNotInTable { .. }+ | RunError::UnboundColumn { .. },)| SqlError::WrittenTableNotInCatalog { .. }| SqlError::AlteredColumnNotInCatalog { .. }diff --git a/crates/databases_sql/src/toolset/save_database_query.rs b/crates/databases_sql/src/toolset/save_database_query.rsindex 30a350ff8f..1b319051c6 100644--- a/crates/databases_sql/src/toolset/save_database_query.rs+++ b/crates/databases_sql/src/toolset/save_database_query.rs@@ -126,7 +126,7 @@ fn query_block_markdown(prompt: &str,display_mode: QueryDatabaseDisplay,chart: Option<&ToolChart>,-) -> String {+) -> Result<String, ToolCallError> {let json = serde_json::to_string(&QueryBlock {query_id,database_id,@@ -135,10 +135,18 @@ fn query_block_markdown(display_mode,chart,})- .expect("a query block always serializes");+ .map_err(|error| ToolCallError {+ description: format!(+ "The query was saved as {query_id}, but its block could not be written: {error}"+ ),+ internal_error: error.into(),+ })?;// `<` only occurs inside strings; escaping it keeps a title such as// "</m-db-query>" from closing the tag early.- format!("<m-db-query>{}</m-db-query>", json.replace('<', "\\u003c"))+ Ok(format!(+ "<m-db-query>{}</m-db-query>",+ json.replace('<', "\\u003c")+ ))}fn invalid(description: impl Into<String>) -> ToolCallError {@@ -253,7 +261,7 @@ whereprompt,self.display_mode,self.chart.as_ref(),- ),+ )?,})}}