A reactive notebook for Python — run reproducible experiments, query with SQL, execute as a script, deploy as an app, and version with git. Stored as pure Python. All in a modern, AI-native editor.
Python artificial-intelligencedagdata-sciencedata-visualizationdataflowdeveloper-toolsmachine-learningnotebookspipelinepythonreactivesqlweb-app
feat: lint pattern to extract and process logs (capture invalid sql) (#6407)
Dylan Madisetti11f64b3eb078
diff --git a/development_docs/adding_lint_rules.md b/development_docs/adding_lint_rules.mdindex 375010883a..7104f62525 100644--- a/development_docs/adding_lint_rules.md+++ b/development_docs/adding_lint_rules.md@@ -309,7 +309,11 @@ Bad: `"Multiple definition error"`### Fixability-TODO: Right now, the only fixable errors are via re-serialization.+Safely fixable errors are applied by default via re-serialization. However,+rules may implement unsafe fixes (mutating the notebook structure) that require+the `--unsafe-fixes` flag. To do this, implement an `async def+apply_unsafe_fixes(self, notebook, diagnostics) -> Notebook` method in your+rule class, and inherit from `UnsafeFixRule` instead of `LintRule`.## Common Patterns@@ -388,6 +392,22 @@ Before submitting your rule:- [ ] Error messages are clear and actionable- [ ] Performance is reasonable for large notebooks-## Example: Simple Rule Implementation+## Examples-Here's a complete example of a simple rule that checks for syntax errors: https://github.com/marimo-team/marimo/pull/6384+### Simple Rule Implementation++Example of a simple rule that checks for syntax errors: https://github.com/marimo-team/marimo/pull/6384++### Rule Implementation with `--unsafe-fixes`++Some rules may have "fixes" that mutate the notebook structure.+An example of a rule that mutates the notebook structure (i.e. an unsafe fix) by removing empty cells: https://github.com/marimo-team/marimo/pull/6398++### Rule Implementation with Log Context++Some parsing issues result in log warnings or errors.+An example of a rule that hooks into issued logs can be found here:++**Note**: Resist intentionally adding log statements such that they trigger+lint rules. Log statements should be added only when they provide useful+context, on notebook startup.diff --git a/docs/guides/lint_rules/index.md b/docs/guides/lint_rules/index.mdindex 804a97a174..aee7f35473 100644--- a/docs/guides/lint_rules/index.md+++ b/docs/guides/lint_rules/index.md@@ -43,6 +43,8 @@ These are style and formatting issues.| [MF002](rules/parse_stdout.md) | parse-stdout | Parse captured stdout during notebook loading | ❌ || [MF003](rules/parse_stderr.md) | parse-stderr | Parse captured stderr during notebook loading | ❌ || [MF004](rules/empty_cells.md) | empty-cells | Empty cells that can be safely removed. | ⚠️ |+| [MF005](rules/sql_parse_error.md) | sql-parse-error | SQL parsing errors during dependency analysis | ❌ |+| [MF006](rules/misc_log_capture.md) | misc-log-capture | Miscellaneous log messages during processing | ❌ |## Legenddiff --git a/docs/guides/lint_rules/rules/misc_log_capture.md b/docs/guides/lint_rules/rules/misc_log_capture.mdnew file mode 100644index 0000000000..36bc73bfab--- /dev/null+++ b/docs/guides/lint_rules/rules/misc_log_capture.md@@ -0,0 +1,30 @@+# MF006: misc-log-capture++✨ **Formatting** ❌ Not Fixable++MF006: Miscellaneous log messages during processing.++## What it does++Captures warning and error level log messages that aren't handled by+other specific log rules and creates diagnostics to surface them.++## Why is this bad?++Unhandled log messages may indicate:+- Unexpected issues during notebook processing+- Configuration problems+- Library warnings that affect execution+- Performance or resource issues++## Examples++**Triggered by:**+- General warnings from imported libraries+- Configuration issues+- Unexpected errors during processing++## References++- [Understanding Errors](https://docs.marimo.io/guides/understanding_errors/)+diff --git a/docs/guides/lint_rules/rules/sql_parse_error.md b/docs/guides/lint_rules/rules/sql_parse_error.mdnew file mode 100644index 0000000000..eb7c05bdfd--- /dev/null+++ b/docs/guides/lint_rules/rules/sql_parse_error.md@@ -0,0 +1,31 @@+# MF005: sql-parse-error++✨ **Formatting** ❌ Not Fixable++MF005: SQL parsing errors during dependency analysis.++## What it does++Captures SQL parsing error logs and creates diagnostics pointing to+problematic SQL statements in cells.++## Why is this bad?++SQL parsing failures can lead to:+- Incorrect dependency analysis for SQL-using cells+- Missing dataframe references in dependency graph+- Reduced effectiveness of reactive execution+- Potential runtime errors when SQL is executed++## Examples++**Triggered by:**+- Invalid SQL syntax in cell code+- Unsupported SQL dialects or extensions+- Complex SQL that exceeds parser capabilities++## References++- [Understanding Errors](https://docs.marimo.io/guides/understanding_errors/)+- [SQL Support](https://docs.marimo.io/guides/sql/)+diff --git a/examples/sql/connect_to_motherduck.py b/examples/sql/connect_to_motherduck.pyindex 3df6b80c2c..7a003c2251 100644--- a/examples/sql/connect_to_motherduck.py+++ b/examples/sql/connect_to_motherduck.py@@ -212,7 +212,7 @@ def _(MONTHS, duckdb, mo):"""SELECT DISTINCT type as 'HN Type'FROM sample_data.hn.hacker_news- WHERE score NOT NULL AND descendants NOT NULL+ WHERE score IS NOT NULL AND descendants IS NOT NULLLIMIT 10;""").df()@@ -259,8 +259,7 @@ def _(hn_type_select, mo, month_list):ANDMONTH(timestamp) in ({month_list})AND- descendants NOT NULl-+ descendants IS NOT NULL)SELECTdiff --git a/examples/third_party/motherduck/embeddings/embeddings_explorer.py b/examples/third_party/motherduck/embeddings/embeddings_explorer.pyindex 83bb75549c..15825bcdee 100644--- a/examples/third_party/motherduck/embeddings/embeddings_explorer.py+++ b/examples/third_party/motherduck/embeddings/embeddings_explorer.py@@ -16,7 +16,7 @@import marimo-__generated_with = "0.15.5"+__generated_with = "0.16.0"app = marimo.App(width="medium")with app.setup:@@ -24,5 +24,9 @@ with app.setup:# https://motherduck.com/blog/MotherDuck-Visualize-Embeddings-Marimo/import marimo as mo++++if __name__ == "__main__":app.run()diff --git a/marimo/_ast/sql_utils.py b/marimo/_ast/sql_utils.pyindex c1749f14d2..52b06cd981 100644--- a/marimo/_ast/sql_utils.py+++ b/marimo/_ast/sql_utils.py@@ -1,6 +1,8 @@# Copyright 2025 Marimo. All rights reserved.-from typing import Literal, Optional, Union+import ast+import re+from typing import Callable, Literal, Optional, TypedDict, Unionfrom marimo import _loggersfrom marimo._dependencies.dependencies import DependencyManager@@ -17,6 +19,20 @@ SQLGLOT_DIALECTS = Literal[]+class SQLErrorMetadata(TypedDict):+ """Structured metadata for SQL parsing errors."""++ lint_rule: str+ error_type: str+ clean_message: str # Just the meaningful error without SQL trace+ node_lineno: int+ node_col_offset: int+ sql_statement: str # Truncated if needed+ sql_line: Optional[int] # 0-based line within SQL+ sql_col: Optional[int] # 0-based column within SQL+ context: str++def classify_sql_statement(sql_statement: str, dialect: Optional[SQLGLOT_DIALECTS] = None) -> Union[SQL_TYPE, Literal["unknown"]]:@@ -33,7 +49,14 @@ def classify_sql_statement(with _loggers.suppress_warnings_logs("sqlglot"):expression_list = parse(sql_statement, dialect=dialect)except ParseError as e:- LOGGER.debug(f"Unable to parse SQL. Error: {e}")+ log_sql_error(+ LOGGER.debug,+ message="Failed to parse SQL statement for classification.",+ exception=e,+ rule_code="MF005",+ node=None,+ sql_content=sql_statement,+ )return "unknown"for expression in expression_list:@@ -52,3 +75,45 @@ def classify_sql_statement(return "DQL"return "unknown"+++def log_sql_error(+ logger: Callable[..., None],+ *,+ message: str,+ exception: BaseException,+ rule_code: str,+ node: Optional[ast.expr] = None,+ sql_content: str = "",+ context: str = "",+) -> None:+ """Utility to log SQL-related errors with consistent metadata."""+ # Parse SQL position from exception message if available+ sql_line = None+ sql_col = None++ exception_msg = str(exception)+ line_col_match = re.search(r"Line (\d+), Col: (\d+)", exception_msg)+ if line_col_match:+ sql_line = int(line_col_match.group(1)) - 1 # Convert to 0-based+ sql_col = int(line_col_match.group(2)) - 1 # Convert to 0-based++ # Truncate long SQL content+ truncated_sql = sql_content+ if sql_content and len(sql_content) > 200:+ truncated_sql = sql_content[:200] + "..."++ # Create metadata using TypedDict+ metadata: SQLErrorMetadata = {+ "lint_rule": rule_code,+ "error_type": type(exception).__name__,+ "clean_message": exception_msg.split("\n", 1)[0],+ "node_lineno": node.lineno if node else 0,+ "node_col_offset": node.col_offset if node else 0,+ "sql_statement": truncated_sql,+ "sql_line": sql_line,+ "sql_col": sql_col,+ "context": context,+ }++ logger(message, exception, extra=metadata)diff --git a/marimo/_ast/sql_visitor.py b/marimo/_ast/sql_visitor.pyindex 9084cfc0ae..bd13cc982e 100644--- a/marimo/_ast/sql_visitor.py+++ b/marimo/_ast/sql_visitor.py@@ -250,6 +250,7 @@ def find_sql_defs(sql_statement: str) -> SQLDefs:LOGGER.warning("Unexpected number of parts in CREATE TABLE: %s",parts,+ extra={"parts": parts},)if is_table:@@ -497,7 +498,6 @@ def find_sql_refs(sql_statement: str) -> set[SQLRef]:DependencyManager.sqlglot.require(why="SQL parsing")from sqlglot import exp, parse- from sqlglot.errors import ParseErrorfrom sqlglot.optimizer.scope import build_scopedef get_ref_from_table(table: exp.Table) -> Optional[SQLRef]:@@ -507,7 +507,6 @@ def find_sql_refs(sql_statement: str) -> set[SQLRef]:catalog_name = table.catalog or Noneif table_name is None:- LOGGER.warning("Table name cannot be found in the SQL statement")return None# Check if the table name looks like a URL or has a file extension.@@ -520,12 +519,9 @@ def find_sql_refs(sql_statement: str) -> set[SQLRef]:table=table_name, schema=schema_name, catalog=catalog_name)- try:- with _loggers.suppress_warnings_logs("sqlglot"):- expression_list = parse(sql_statement, dialect="duckdb")- except ParseError as e:- LOGGER.error(f"Unable to parse SQL. Error: {e}")- return set()+ # May raise a ParseError+ with _loggers.suppress_warnings_logs("sqlglot"):+ expression_list = parse(sql_statement, dialect="duckdb")refs: set[SQLRef] = set()diff --git a/marimo/_ast/visitor.py b/marimo/_ast/visitor.pyindex a03ca4b6d2..bb2951692c 100644--- a/marimo/_ast/visitor.py+++ b/marimo/_ast/visitor.py@@ -12,6 +12,7 @@ from uuid import uuid4from marimo import _loggersfrom marimo._ast.errors import ImportStarError+from marimo._ast.sql_utils import log_sql_errorfrom marimo._ast.sql_visitor import (SQLDefs,SQLKind,@@ -26,8 +27,10 @@ from marimo._utils.strings import standardize_annotation_quotesLOGGER = _loggers.marimo_logger()+Name = str+Language = Literal["python", "sql"]@@ -632,6 +635,18 @@ class ScopedVisitor(ast.NodeVisitor):and sql):import duckdb # type: ignore[import-not-found,import-untyped,unused-ignore] # noqa: E501++ # Import ParseError outside try block so we can except it+ # Use a Union approach to handle both cases+ ParseErrorType: type[Exception]+ if DependencyManager.sqlglot.has():+ from sqlglot.errors import ParseError as SQLGLOTParseError++ ParseErrorType = SQLGLOTParseError+ else:+ # Fallback when sqlglot is not available+ ParseErrorType = Exception+# TODO: Handle other SQL languages# TODO: Get the engine so we can differentiate tables in diff engines@@ -655,7 +670,14 @@ class ScopedVisitor(ast.NodeVisitor):# We catch base exceptions because we don't want to# fail due to bugs in duckdb -- users code should# be saveable no matter what- LOGGER.warning("Unexpected duckdb error %s", e)+ log_sql_error(+ LOGGER.warning,+ message=f"Unexpected duckdb error {e}",+ exception=e,+ node=node,+ rule_code="MF005",+ sql_content=sql,+ )self.generic_visit(node)return node@@ -667,7 +689,15 @@ class ScopedVisitor(ast.NodeVisitor):except duckdb.ProgrammingError:sql_defs = SQLDefs()except BaseException as e:- LOGGER.warning("Unexpected duckdb error %s", e)+ log_sql_error(+ LOGGER.warning,+ message=f"Unexpected duckdb error {e}",+ exception=e,+ node=node,+ rule_code="MF005",+ sql_content=sql,+ context="sql_defs_extraction",+ )sql_defs = SQLDefs()defined_names = set()@@ -699,13 +729,23 @@ class ScopedVisitor(ast.NodeVisitor):sql_refs: set[SQLRef] = set()try:+ # Take resultssql_refs = find_sql_refs_cached(statement.query)- except (duckdb.ProgrammingError, duckdb.IOException):- LOGGER.debug(- "Error parsing SQL statement: %s", statement.query+ except (+ duckdb.ProgrammingError,+ duckdb.IOException,+ ParseErrorType,+ BaseException,+ ) as e:+ # Use first_arg (SQL string node) for accurate positioning+ log_sql_error(+ LOGGER.error,+ message=f"Error parsing SQL statement: {e}",+ exception=e,+ node=first_arg,+ rule_code="MF005",+ sql_content=statement.query,)- except BaseException as e:- LOGGER.warning("Unexpected duckdb error %s", e)for ref in sql_refs:name = ref.qualified_namediff --git a/marimo/_lint/context.py b/marimo/_lint/context.pyindex d6b8065a55..e593358079 100644--- a/marimo/_lint/context.py+++ b/marimo/_lint/context.py@@ -1,14 +1,19 @@-# Copyright 2024 Marimo. All rights reserved.+# Copyright 2025 Marimo. All rights reserved.from __future__ import annotationsimport asyncioimport heapq+import logging # noqa: TC003import threadingfrom typing import TYPE_CHECKING-from marimo._ast.load import load_notebook_ir+from marimo._ast.names import SETUP_CELL_NAME++# Note: load_notebook_ir not used - we do manual compilation for per-cell log capturefrom marimo._lint.diagnostic import Diagnostic, Severity-from marimo._schemas.serialization import NotebookSerialization+from marimo._loggers import capture_output+from marimo._schemas.serialization import CellDef, NotebookSerialization+from marimo._types.ids import CellId_tif TYPE_CHECKING:from marimo._lint.rules.base import LintRule@@ -30,6 +35,7 @@ class LintContext:notebook: NotebookSerialization,stderr: str = "",stdout: str = "",+ logs: list[logging.LogRecord] | None = None,):self.notebook = notebookself._diagnostics: list[tuple[int, int, Diagnostic]] = []@@ -43,6 +49,13 @@ class LintContext:self.stderr = stderrself.stdout = stdout+ self._log_records = logs or []+ self._logs_by_rule: dict[str, list[logging.LogRecord]] = {}+ self._logs_by_cell_and_rule: dict[+ str, dict[str, list[logging.LogRecord]]+ ] = {}++ self._errors: dict[str, list[tuple[Exception, CellDef]]] = {}def _get_diagnostics_lock(self) -> asyncio.Lock:"""Get the diagnostics lock, creating it if needed."""@@ -105,6 +118,45 @@ class LintContext:return new_diagnostics+ def _group_initial_logs(self) -> None:+ """Group initial log records by rule code."""+ for record in self._log_records:+ # Check if record has lint_rule metadata+ lint_rule = getattr(record, "lint_rule", None)+ if hasattr(record, "__dict__") and "lint_rule" in record.__dict__:+ lint_rule = record.__dict__["lint_rule"]++ # Default to MF006 (misc) if no specific rule+ rule_code = lint_rule if lint_rule else "MF006"++ if rule_code not in self._logs_by_rule:+ self._logs_by_rule[rule_code] = []+ self._logs_by_rule[rule_code].append(record)++ def _enhance_cell_logs(+ self,+ cell_logs: list[logging.LogRecord],+ cell_id: str,+ cell_lineno: int,+ ) -> None:+ """Enhance log records with cell information and store globally."""+ for record in cell_logs:+ # Add cell information to the log record+ if hasattr(record, "__dict__"):+ record.__dict__["cell_id"] = cell_id+ record.__dict__["cell_lineno"] = cell_lineno++ lint_rule = getattr(record, "lint_rule", None)+ if hasattr(record, "__dict__") and "lint_rule" in record.__dict__:+ lint_rule = record.__dict__["lint_rule"]++ rule_code = lint_rule if lint_rule else "MF006"+ if rule_code not in self._logs_by_rule:+ self._logs_by_rule[rule_code] = []+ self._logs_by_rule[rule_code].append(record)++ self._log_records.extend(cell_logs)+def get_graph(self) -> DirectedGraph:"""Get the dependency graph, constructing it once and caching."""if self._graph is not None:@@ -115,13 +167,75 @@ class LintContext:if self._graph is not None:return self._graph- # Construct the graph- app = load_notebook_ir(self.notebook)+ # Group any initial logs+ self._group_initial_logs()++ # Manually compile the graph with per-cell log capture+ from marimo._ast.app import App, InternalApp+ from marimo._ast.cell import CellConfig+ from marimo._ast.cell_manager import CellManager+ from marimo._ast.compiler import ir_cell_factory+ from marimo._schemas.serialization import UnparsableCell++ # Create the app+ app = App(+ **self.notebook.app.options, _filename=self.notebook.filename+ )self._graph = app._graph+ # Process each cell individually to capture logs per-cell+ for i, cell in enumerate(self.notebook.cells):+ if isinstance(cell, UnparsableCell):+ app._unparsable_cell(cell.code, **cell.options)+ continue++ # Capture logs during this specific cell's compilation+ with capture_output() as (_, _, cell_logs):+ # Call ir_cell_factory directly with proper exception handling+ if cell.name == SETUP_CELL_NAME:+ cell_id = CellId_t(SETUP_CELL_NAME)+ else:+ cell_id = app._cell_manager.create_cell_id()+ filename = self.notebook.filename+ cell_config = CellConfig.from_dict(cell.options)++ try:+ compiled_cell = ir_cell_factory(+ cell, cell_id=cell_id, filename=filename+ )+ compiled_cell._cell.configure(cell_config)+ # Register the successfully compiled cell+ app._cell_manager._register_cell(+ compiled_cell, InternalApp(app)+ )+ except SyntaxError as e:+ # Handle syntax errors like register_ir_cell does+ app._cell_manager.unparsable = True+ app._cell_manager.register_cell(+ cell_id=cell_id,+ code=cell.code,+ config=cell_config,+ name=cell.name,+ cell=None,+ )+ self._errors.setdefault("SyntaxError", []).append(+ (e, cell)+ )+ except Exception as e:+ self._errors.setdefault("unhandled", []).append(+ (e, cell)+ )++ # Enhance logs with cell information and store globally+ simplified_cell_id = f"cell-{i}"+ self._enhance_cell_logs(+ cell_logs, simplified_cell_id, cell.lineno+ )+# Initialize the app to register cells in the graph- for cell_id, cell in app._cell_manager.valid_cells():- self._graph.register_cell(cell_id, cell._cell)+ cell_manager: CellManager = app._cell_manager+ for cell_id, cell_impl in cell_manager.valid_cells():+ self._graph.register_cell(cell_id, cell_impl._cell)return self._graph@@ -169,3 +283,32 @@ class RuleContext:def stderr(self) -> str:"""Access to the captured stderr."""return self.global_context.stderr++ def get_errors(self, key: str) -> list[tuple[Exception, CellDef]]:+ return self.global_context._errors.get(key, [])++ def get_logs(+ self, rule_code: str | None = None+ ) -> list[logging.LogRecord]:+ """Get log records for a specific rule or all logs if no rule specified."""+ if rule_code is None:+ return self.global_context._log_records++ return self.global_context._logs_by_rule.get(rule_code, [])++ def get_logs_for_cell(+ self, cell_id: str, rule_code: str | None = None+ ) -> list[logging.LogRecord]:+ """Get log records for a specific cell and rule."""+ if cell_id not in self.global_context._logs_by_cell_and_rule:+ return []++ cell_logs = self.global_context._logs_by_cell_and_rule[cell_id]+ if rule_code is None:+ # Return all logs for this cell+ all_logs = []+ for logs in cell_logs.values():+ all_logs.extend(logs)+ return all_logs++ return cell_logs.get(rule_code, [])diff --git a/marimo/_lint/linter.py b/marimo/_lint/linter.pyindex 79abf05c9f..3ce31e4342 100644--- a/marimo/_lint/linter.py+++ b/marimo/_lint/linter.py@@ -1,9 +1,7 @@-# Copyright 2024 Marimo. All rights reserved.+# Copyright 2025 Marimo. All rights reserved.from __future__ import annotationsimport asyncio-import contextlib-import ioimport refrom dataclasses import dataclass, fieldfrom pathlib import Path@@ -15,6 +13,7 @@ from marimo._cli.print import redfrom marimo._convert.converters import MarimoConvertfrom marimo._lint.diagnostic import Diagnostic, Severityfrom marimo._lint.rule_engine import EarlyStoppingConfig, RuleEngine+from marimo._loggers import capture_outputfrom marimo._schemas.serialization import NotebookSerializationif TYPE_CHECKING:@@ -110,14 +109,8 @@ class Linter:)return file_status- captured_stdout = io.StringIO()- captured_stderr = io.StringIO()-try:- with (- contextlib.redirect_stdout(captured_stdout),- contextlib.redirect_stderr(captured_stderr),- ):+ with capture_output() as (stdout, stderr, logs):load_result = get_notebook_status(file_path)except SyntaxError as e:# Handle syntax errors in notebooks@@ -170,8 +163,9 @@ class Linter:await self.rule_engine.check_notebook(load_result.notebook,# Add parsing rule if there's captured output- stdout=captured_stdout.getvalue().strip(),- stderr=captured_stderr.getvalue().strip(),+ stdout=stdout.getvalue().strip(),+ stderr=stderr.getvalue().strip(),+ logs=logs,))except Exception as e:@@ -242,10 +236,11 @@ class Linter:"""Generate file contents from notebook serialization."""converter = MarimoConvert.from_ir(notebook)- if filename.endswith((".md", ".qmd")):- return converter.to_markdown(filename)- else:- return converter.to_py()+ with capture_output():+ if filename.endswith((".md", ".qmd")):+ return converter.to_markdown(filename)+ else:+ return converter.to_py()@staticmethoddef _generate_file_contents(file_status: FileStatus) -> str:diff --git a/marimo/_lint/rule_engine.py b/marimo/_lint/rule_engine.pyindex eb5732b380..27c1ec68ab 100644--- a/marimo/_lint/rule_engine.py+++ b/marimo/_lint/rule_engine.py@@ -10,6 +10,7 @@ from marimo._lint.rules import RULE_CODESfrom marimo._schemas.serialization import NotebookSerializationif TYPE_CHECKING:+ import loggingfrom collections.abc import AsyncIteratorfrom marimo._lint.diagnostic import Diagnostic@@ -67,9 +68,10 @@ class RuleEngine:notebook: NotebookSerialization,stdout: str = "",stderr: str = "",+ logs: list[logging.LogRecord] | None = None,) -> AsyncIterator[Diagnostic]:"""Check notebook and yield diagnostics as they become available."""- ctx = LintContext(notebook, stdout, stderr)+ ctx = LintContext(notebook, stdout, stderr, logs)# Create tasks for all rules with their completion trackingpending_tasks = {@@ -115,11 +117,12 @@ class RuleEngine:notebook: NotebookSerialization,stdout: str = "",stderr: str = "",+ logs: list[logging.LogRecord] | None = None,) -> list[Diagnostic]:"""Check notebook for all lint rule violations using async execution."""diagnostics = []async for diagnostic in self.check_notebook_streaming(- notebook, stdout, stderr+ notebook, stdout, stderr, logs):diagnostics.append(diagnostic)return diagnosticsdiff --git a/marimo/_lint/rules/breaking/syntax_error.py b/marimo/_lint/rules/breaking/syntax_error.pyindex 3da7d48bd6..c23638dd8e 100644--- a/marimo/_lint/rules/breaking/syntax_error.py+++ b/marimo/_lint/rules/breaking/syntax_error.py@@ -3,14 +3,9 @@ from __future__ import annotationsfrom typing import TYPE_CHECKING-from marimo._ast.compiler import (- ir_cell_factory,-)from marimo._ast.errors import ImportStarErrorfrom marimo._lint.diagnostic import Diagnostic, Severityfrom marimo._lint.rules.base import LintRule-from marimo._schemas.serialization import UnparsableCell-from marimo._types.ids import CellId_tif TYPE_CHECKING:from marimo._lint.context import RuleContext@@ -98,35 +93,36 @@ class SyntaxErrorRule(LintRule):async def check(self, ctx: RuleContext) -> None:"""Check for syntax errors during compilation."""- for cell in ctx.notebook.cells:- if not isinstance(cell, UnparsableCell):- try:- ir_cell_factory(- cell,- cell_id=CellId_t("Hbol"),- filename=ctx.notebook.filename,- )- # Technically a SyntaxError subclass- except ImportStarError as e:- line, column = _handle_import_star_error(e, cell)- await ctx.add_diagnostic(- Diagnostic(- message=IMPORT_STAR_ERROR_MESSAGE,- line=line,- column=column,- fix=IMPORT_STAR_HINT,- )+ for e, cell in ctx.get_errors("SyntaxError"):+ if isinstance(e, ImportStarError):+ line, column = _handle_import_star_error(e, cell)+ await ctx.add_diagnostic(+ Diagnostic(+ message=IMPORT_STAR_ERROR_MESSAGE,+ line=line,+ column=column,+ fix=IMPORT_STAR_HINT,)- except SyntaxError as e:+ )+ else:+ # Handle SyntaxError specifically+ if isinstance(e, SyntaxError):message = f"{e.msg}"- await ctx.add_diagnostic(- Diagnostic(- message=message,- line=cell.lineno + (e.lineno or 1) - 1,- column=(e.offset or 1),- fix=_get_known_hints(message),- )+ line = cell.lineno + (e.lineno or 1) - 1+ column = e.offset or 1+ else:+ # For other exceptions, use string representation+ message = str(e)+ line = cell.lineno+ column = 1+ await ctx.add_diagnostic(+ Diagnostic(+ message=message,+ line=line,+ column=column,+ fix=_get_known_hints(message),)+ )def _handle_import_star_error(diff --git a/marimo/_lint/rules/formatting/__init__.py b/marimo/_lint/rules/formatting/__init__.pyindex 0d618646b4..977c612def 100644--- a/marimo/_lint/rules/formatting/__init__.py+++ b/marimo/_lint/rules/formatting/__init__.py@@ -2,13 +2,20 @@from marimo._lint.rules.base import LintRule, UnsafeFixRulefrom marimo._lint.rules.formatting.empty_cells import EmptyCellRulefrom marimo._lint.rules.formatting.general import GeneralFormattingRule-from marimo._lint.rules.formatting.parsing import StderrRule, StdoutRule+from marimo._lint.rules.formatting.parsing import (+ MiscLogRule,+ SqlParseRule,+ StderrRule,+ StdoutRule,+)FORMATTING_RULE_CODES: dict[str, type[LintRule]] = {"MF001": GeneralFormattingRule,"MF002": StdoutRule,"MF003": StderrRule,"MF004": EmptyCellRule,+ "MF005": SqlParseRule,+ "MF006": MiscLogRule,}__all__ = [@@ -17,5 +24,7 @@ __all__ = ["StdoutRule","StderrRule","EmptyCellRule",+ "SqlParseRule",+ "MiscLogRule","UnsafeFixRule",]diff --git a/marimo/_lint/rules/formatting/parsing.py b/marimo/_lint/rules/formatting/parsing.pyindex 6a1fb4e8b3..02d57dbd26 100644--- a/marimo/_lint/rules/formatting/parsing.py+++ b/marimo/_lint/rules/formatting/parsing.py@@ -1,6 +1,7 @@-# Copyright 2024 Marimo. All rights reserved.+# Copyright 2025 Marimo. All rights reserved.from __future__ import annotations+import loggingimport refrom typing import TYPE_CHECKING@@ -155,3 +156,144 @@ class StderrRule(LintRule):column=0,))+++class SqlParseRule(LintRule):+ """MF005: SQL parsing errors during dependency analysis.++ This rule processes log messages captured when marimo encounters errors+ while parsing SQL statements in notebook cells. SQL parsing is used for+ dependency analysis and dataframe tracking.++ ## What it does++ Captures SQL parsing error logs and creates diagnostics pointing to+ problematic SQL statements in cells.++ ## Why is this bad?++ SQL parsing failures can lead to:+ - Incorrect dependency analysis for SQL-using cells+ - Missing dataframe references in dependency graph+ - Reduced effectiveness of reactive execution+ - Potential runtime errors when SQL is executed++ ## Examples++ **Triggered by:**+ - Invalid SQL syntax in cell code+ - Unsupported SQL dialects or extensions+ - Complex SQL that exceeds parser capabilities++ ## References++ - [Understanding Errors](https://docs.marimo.io/guides/understanding_errors/)+ - [SQL Support](https://docs.marimo.io/guides/sql/)+ """++ code = "MF005"+ name = "sql-parse-error"+ description = "SQL parsing errors during dependency analysis"+ severity = Severity.FORMATTING+ fixable = False++ async def check(self, ctx: RuleContext) -> None:+ """Process SQL parsing error logs."""+ logs = ctx.get_logs(self.code)++ for record in logs:+ # Extract metadata from log record - ONLY use extra_data+ extra_data = getattr(record, "__dict__", {})++ # Use clean message from metadata (without SQL trace)+ message = extra_data.get("clean_message", "SQL parsing error")++ # Calculate line position using cell information+ cell_lineno = extra_data.get(+ "cell_lineno", 0+ ) # Cell start line in notebook+ node_lineno = extra_data.get(+ "node_lineno", 1+ ) # Node line within cell+ sql_line = extra_data.get("sql_line") # SQL line within SQL string++ # Start with cell position + node position within cell+ line = cell_lineno + node_lineno - 1 # Convert to 0-based++ # Add SQL line offset if available+ if sql_line is not None:+ line += sql_line++ # Use SQL column if available, otherwise node column+ sql_col = extra_data.get("sql_col")+ col = (+ sql_col+ if sql_col is not None+ else extra_data.get("node_col_offset", 0)+ )++ await ctx.add_diagnostic(+ Diagnostic(+ message=message,+ line=line,+ cell_id=None,+ column=col,+ )+ )+++class MiscLogRule(LintRule):+ """MF006: Miscellaneous log messages during processing.++ This rule processes log messages that don't have a specific rule assigned+ but may still be relevant for understanding notebook health and potential+ issues during processing.++ ## What it does++ Captures warning and error level log messages that aren't handled by+ other specific log rules and creates diagnostics to surface them.++ ## Why is this bad?++ Unhandled log messages may indicate:+ - Unexpected issues during notebook processing+ - Configuration problems+ - Library warnings that affect execution+ - Performance or resource issues++ ## Examples++ **Triggered by:**+ - General warnings from imported libraries+ - Configuration issues+ - Unexpected errors during processing++ ## References++ - [Understanding Errors](https://docs.marimo.io/guides/understanding_errors/)+ """++ code = "MF006"+ name = "misc-log-capture"+ description = "Miscellaneous log messages during processing"+ severity = Severity.FORMATTING+ fixable = False++ async def check(self, ctx: RuleContext) -> None:+ """Process miscellaneous log messages."""+ logs = ctx.get_logs(self.code)++ for record in logs:+ # Only process WARNING and ERROR level logs to avoid noise+ if record.levelno < logging.WARNING:+ continue++ await ctx.add_diagnostic(+ Diagnostic(+ message=record.getMessage(),+ line=0, # Misc logs don't have meaningful line positioning+ cell_id=None,+ column=0,+ )+ )diff --git a/marimo/_loggers.py b/marimo/_loggers.pyindex 8a958afbb1..51079b4d1e 100644--- a/marimo/_loggers.py+++ b/marimo/_loggers.py@@ -1,6 +1,7 @@# Copyright 2024 Marimo. All rights reserved.from __future__ import annotations+import ioimport loggingfrom contextlib import contextmanagerfrom logging.handlers import TimedRotatingFileHandler@@ -9,7 +10,7 @@ from typing import TYPE_CHECKING, Optionalfrom marimo._utils.log_formatter import LogFormatterif TYPE_CHECKING:- from collections.abc import Generator+ from collections.abc import Generator, Iteratorfrom pathlib import Path# This file manages and creates loggers used throughout marimo.@@ -182,6 +183,49 @@ def _file_handler() -> logging.FileHandler:return file_handler+class ListHandler(logging.Handler):+ """Log handler that captures records in a list for processing."""++ def __init__(self, level: int = logging.NOTSET) -> None:+ super().__init__(level)+ self.records: list[logging.LogRecord] = []++ def emit(self, record: logging.LogRecord) -> None:+ self.records.append(record)+++@contextmanager+def capture_output(+ *, log_level: int = logging.DEBUG+) -> Iterator[tuple[io.StringIO, io.StringIO, list[logging.LogRecord]]]:+ """+ Capture stdout, stderr, and log records during execution.++ Yields: (stdout_buf, stderr_buf, log_records)+ - log_records are raw LogRecord objects with full metadata (name, level, pathname, lineno, created, etc.).+ - Log emission to existing handlers is silenced inside the context.+ """+ import contextlib++ logger = marimo_logger()+ out, err = io.StringIO(), io.StringIO()+ h: ListHandler = ListHandler(log_level)++ # Snapshot and suppress existing handlers+ old_handlers, old_propagate = logger.handlers[:], logger.propagate+ logger.addHandler(h)+ logger.handlers = [h]+ logger.propagate = False++ try:+ with contextlib.redirect_stdout(out), contextlib.redirect_stderr(err):+ yield out, err, h.records+ finally:+ logger.removeHandler(h)+ logger.handlers = old_handlers+ logger.propagate = old_propagate++@contextmanagerdef suppress_warnings_logs(name: str) -> Generator[None, None, None]:"""Suppress logs for a given logger."""diff --git a/mkdocs.yml b/mkdocs.ymlindex 0a9a01adcf..b979736ab1 100644--- a/mkdocs.yml+++ b/mkdocs.yml@@ -105,6 +105,8 @@ nav:- Parse stdout: guides/lint_rules/rules/parse_stdout.md- Parse stderr: guides/lint_rules/rules/parse_stderr.md- Empty cells: guides/lint_rules/rules/empty_cells.md+ - SQL parse error: guides/lint_rules/rules/sql_parse_error.md+ - Misc parse log: guides/lint_rules/rules/misc_log_capture.md- SQL, dataframes, and plots:- SQL, dataframes, and plots: guides/working_with_data/index.md- DataFrames: guides/working_with_data/dataframes.mddiff --git a/tests/_ast/test_sql_visitor.py b/tests/_ast/test_sql_visitor.pyindex 30eed88aa6..cb2023fe8b 100644--- a/tests/_ast/test_sql_visitor.py+++ b/tests/_ast/test_sql_visitor.py@@ -788,7 +788,10 @@ class TestFindSQLRefs:def test_invalid_sql(self) -> None:sql = "SELECT * FROM"- assert find_sql_refs(sql) == set()+ from sqlglot import ParseError++ with pytest.raises(ParseError):+ find_sql_refs(sql)def test_dml_with_subquery(self) -> None:# firstdiff --git a/tests/_lint/snapshots/sql_parsing_errors.txt b/tests/_lint/snapshots/sql_parsing_errors.txtnew file mode 100644index 0000000000..f591190f4b--- /dev/null+++ b/tests/_lint/snapshots/sql_parsing_errors.txt@@ -0,0 +1,6 @@+warning[sql-parse-error]: Expecting ). Line 19, Col: 21.+ --> tests/_lint/test_files/sql_parsing_errors.py:36:20+ 36 | AND+ 37 | descendants NOT NULL+ | ^+ 38 | )\ No newline at end of filediff --git a/tests/_lint/test_files/sql_parsing_errors.py b/tests/_lint/test_files/sql_parsing_errors.pynew file mode 100644index 0000000000..91e0ec1d36--- /dev/null+++ b/tests/_lint/test_files/sql_parsing_errors.py@@ -0,0 +1,56 @@+# Copyright 2025 Marimo. All rights reserved.+"""Test file with SQL parsing errors to test log rules positioning."""++import marimo++__generated_with = "0.8.0"+app = marimo.App()+++def _():+ import marimo as mo+ return mo,+++def _(mo):+ # This should trigger an MF005 SQL parsing error due to trailing comma+ result = mo.sql(f"""+ WITH ranked_stories AS (+ SELECT+ title,+ score,+ type,+ descendants,+ YEAR(timestamp) AS year,+ MONTH(timestamp) AS month,+ ROW_NUMBER()+ OVER (PARTITION BY YEAR(timestamp), MONTH(timestamp) ORDER BY score DESC)+ AS rn+ FROM sample_data.hn.hacker_news+ WHERE+ type = 'story'+ AND+ MONTH(timestamp) in (null)+ AND+ descendants NOT NULL+ )++ SELECT+ month,+ score,+ type,+ title,+ hn_url,+ descendants as nb_comments,+ year,+ FROM ranked_stories+ WHERE rn = 1+ ORDER BY year, month;+ """)+ return result,+++if __name__ == "__main__":+ app.run()diff --git a/tests/_lint/test_log_rules.py b/tests/_lint/test_log_rules.pynew file mode 100644index 0000000000..8e9006acaa--- /dev/null+++ b/tests/_lint/test_log_rules.py@@ -0,0 +1,158 @@+# Copyright 2025 Marimo. All rights reserved.+from __future__ import annotations++import logging++from marimo._lint.context import LintContext, RuleContext+from marimo._lint.rules.formatting.parsing import (+ MiscLogRule,+ SqlParseRule,+)+from marimo._schemas.serialization import NotebookSerialization+++class TestLogRules:+ """Test log message rules and grouping."""++ def test_log_grouping_async_safe(self):+ """Test that log grouping is async-safe and works correctly."""+ # Create mock log records with different rule targets+ record1 = logging.LogRecord(+ name="marimo",+ level=logging.ERROR,+ pathname="/test.py",+ lineno=10,+ msg="SQL parse error",+ args=(),+ exc_info=None,+ )+ record1.__dict__["lint_rule"] = "MF005"++ record2 = logging.LogRecord(+ name="marimo",+ level=logging.WARNING,+ pathname="/test.py",+ lineno=30,+ msg="General warning",+ args=(),+ exc_info=None,+ )+ # No lint_rule specified - should go to MF006++ # Create minimal notebook+ notebook = NotebookSerialization(+ filename="test.py", cells=[], app=None+ )++ # Test with initial logs+ ctx = LintContext(notebook, logs=[record1, record2])+ ctx._group_initial_logs()++ # Verify logs are correctly grouped+ assert len(ctx._logs_by_rule.get("MF005", [])) == 1+ assert len(ctx._logs_by_rule.get("MF006", [])) == 1+ assert ctx._logs_by_rule["MF005"][0].getMessage() == "SQL parse error"+ assert ctx._logs_by_rule["MF006"][0].getMessage() == "General warning"++ async def test_sql_parse_rule(self):+ """Test SqlParseRule processes MF005 logs correctly."""+ record = logging.LogRecord(+ name="marimo",+ level=logging.ERROR,+ pathname="/test.py",+ lineno=20,+ msg="SQL parsing error",+ args=(),+ exc_info=None,+ )+ record.__dict__["lint_rule"] = "MF005"++ notebook = NotebookSerialization(+ filename="test.py", cells=[], app=None+ )+ ctx = LintContext(notebook, logs=[record])+ ctx._group_initial_logs()++ rule = SqlParseRule()+ rule_ctx = RuleContext(ctx, rule)+ await rule.check(rule_ctx)++ diagnostics = await ctx.get_diagnostics()+ assert len(diagnostics) == 1+ assert diagnostics[0].code == "MF005"+ assert diagnostics[0].message == "SQL parsing error"++ async def test_misc_log_rule(self):+ """Test MiscLogRule processes unspecified logs correctly."""+ # WARNING level - should be processed+ record1 = logging.LogRecord(+ name="marimo",+ level=logging.WARNING,+ pathname="/test.py",+ lineno=10,+ msg="General warning",+ args=(),+ exc_info=None,+ )++ # DEBUG level - should be ignored+ record2 = logging.LogRecord(+ name="marimo",+ level=logging.DEBUG,+ pathname="/test.py",+ lineno=20,+ msg="Debug message",+ args=(),+ exc_info=None,+ )++ notebook = NotebookSerialization(+ filename="test.py", cells=[], app=None+ )+ ctx = LintContext(notebook, logs=[record1, record2])+ ctx._group_initial_logs()++ rule = MiscLogRule()+ rule_ctx = RuleContext(ctx, rule)+ await rule.check(rule_ctx)++ diagnostics = await ctx.get_diagnostics()+ # Only WARNING should create a diagnostic+ assert len(diagnostics) == 1+ assert diagnostics[0].code == "MF006"+ assert diagnostics[0].message == "General warning"++ def test_rule_context_get_logs(self):+ """Test RuleContext.get_logs method."""+ record1 = logging.LogRecord(+ name="marimo",+ level=logging.ERROR,+ pathname="/test.py",+ lineno=10,+ msg="SQL parse error",+ args=(),+ exc_info=None,+ )+ record1.__dict__["lint_rule"] = "MF005"++ notebook = NotebookSerialization(+ filename="test.py", cells=[], app=None+ )+ ctx = LintContext(notebook, logs=[record1])+ ctx._group_initial_logs()++ rule = SqlParseRule()+ rule_ctx = RuleContext(ctx, rule)++ # Test getting specific rule logs+ mf005_logs = rule_ctx.get_logs("MF005")+ assert len(mf005_logs) == 1+ assert mf005_logs[0].getMessage() == "SQL parse error"++ # Test getting non-existent rule logs+ empty_logs = rule_ctx.get_logs("MF999")+ assert len(empty_logs) == 0++ # Test getting all logs+ all_logs = rule_ctx.get_logs(None)+ assert len(all_logs) == 1diff --git a/tests/_lint/test_sql_log_rules_snapshot.py b/tests/_lint/test_sql_log_rules_snapshot.pynew file mode 100644index 0000000000..fafa0f100f--- /dev/null+++ b/tests/_lint/test_sql_log_rules_snapshot.py@@ -0,0 +1,36 @@+# Copyright 2025 Marimo. All rights reserved.+"""Snapshot tests for SQL log message lint rules."""++import pytest++from marimo._ast.parse import parse_notebook+from tests._lint.utils import lint_notebook+from tests.mocks import snapshotter++snapshot = snapshotter(__file__)+++# Only run this in 3.12 since the formatting may differ in other versions[email protected]("sys.version_info < (3, 13)")+def test_sql_parsing_errors_snapshot():+ """Test snapshot for SQL parsing log errors with positioning."""+ file = "tests/_lint/test_files/sql_parsing_errors.py"+ with open(file) as f:+ code = f.read()++ notebook = parse_notebook(code, filepath=file)+ errors = lint_notebook(notebook)++ log_errors = [+ error for error in errors if error.code in ("MF005", "MF006")+ ]++ # Format errors for snapshot+ error_output = []+ for error in log_errors:+ error_output.append(error.format())++ if not error_output:+ error_output = ["No SQL log errors found"]++ snapshot("sql_parsing_errors.txt", "\n".join(error_output))