Skip to content

A reactive notebook for Python — run reproducible experiments, query with SQL, execute as a script, deploy as an app, and version with git. Stored as pure Python. All in a modern, AI-native editor.

feat: lint pattern to extract and process logs (capture invalid sql) (#6407)

Dylan Madisetti11f64b3eb078
diff --git a/development_docs/adding_lint_rules.md b/development_docs/adding_lint_rules.md
index 375010883a..7104f62525 100644
--- a/development_docs/adding_lint_rules.md
+++ b/development_docs/adding_lint_rules.md
@@ -309,7 +309,11 @@ Bad: `"Multiple definition error"`
### Fixability
-TODO: Right now, the only fixable errors are via re-serialization.
+Safely fixable errors are applied by default via re-serialization. However,
+rules may implement unsafe fixes (mutating the notebook structure) that require
+the `--unsafe-fixes` flag. To do this, implement an `async def
+apply_unsafe_fixes(self, notebook, diagnostics) -> Notebook` method in your
+rule class, and inherit from `UnsafeFixRule` instead of `LintRule`.
## Common Patterns
@@ -388,6 +392,22 @@ Before submitting your rule:
- [ ] Error messages are clear and actionable
- [ ] Performance is reasonable for large notebooks
-## Example: Simple Rule Implementation
+## Examples
-Here's a complete example of a simple rule that checks for syntax errors: https://github.com/marimo-team/marimo/pull/6384
+### Simple Rule Implementation
+
+Example of a simple rule that checks for syntax errors: https://github.com/marimo-team/marimo/pull/6384
+
+### Rule Implementation with `--unsafe-fixes`
+
+Some rules may have "fixes" that mutate the notebook structure.
+An example of a rule that mutates the notebook structure (i.e. an unsafe fix) by removing empty cells: https://github.com/marimo-team/marimo/pull/6398
+
+### Rule Implementation with Log Context
+
+Some parsing issues result in log warnings or errors.
+An example of a rule that hooks into issued logs can be found here:
+
+**Note**: Resist intentionally adding log statements such that they trigger
+lint rules. Log statements should be added only when they provide useful
+context, on notebook startup.
diff --git a/docs/guides/lint_rules/index.md b/docs/guides/lint_rules/index.md
index 804a97a174..aee7f35473 100644
--- a/docs/guides/lint_rules/index.md
+++ b/docs/guides/lint_rules/index.md
@@ -43,6 +43,8 @@ These are style and formatting issues.
| [MF002](rules/parse_stdout.md) | parse-stdout | Parse captured stdout during notebook loading | ❌ |
| [MF003](rules/parse_stderr.md) | parse-stderr | Parse captured stderr during notebook loading | ❌ |
| [MF004](rules/empty_cells.md) | empty-cells | Empty cells that can be safely removed. | ⚠️ |
+| [MF005](rules/sql_parse_error.md) | sql-parse-error | SQL parsing errors during dependency analysis | ❌ |
+| [MF006](rules/misc_log_capture.md) | misc-log-capture | Miscellaneous log messages during processing | ❌ |
## Legend
diff --git a/docs/guides/lint_rules/rules/misc_log_capture.md b/docs/guides/lint_rules/rules/misc_log_capture.md
new file mode 100644
index 0000000000..36bc73bfab
--- /dev/null
+++ b/docs/guides/lint_rules/rules/misc_log_capture.md
@@ -0,0 +1,30 @@
+# MF006: misc-log-capture
+
+✨ **Formatting** ❌ Not Fixable
+
+MF006: Miscellaneous log messages during processing.
+
+## What it does
+
+Captures warning and error level log messages that aren't handled by
+other specific log rules and creates diagnostics to surface them.
+
+## Why is this bad?
+
+Unhandled log messages may indicate:
+- Unexpected issues during notebook processing
+- Configuration problems
+- Library warnings that affect execution
+- Performance or resource issues
+
+## Examples
+
+**Triggered by:**
+- General warnings from imported libraries
+- Configuration issues
+- Unexpected errors during processing
+
+## References
+
+- [Understanding Errors](https://docs.marimo.io/guides/understanding_errors/)
+
diff --git a/docs/guides/lint_rules/rules/sql_parse_error.md b/docs/guides/lint_rules/rules/sql_parse_error.md
new file mode 100644
index 0000000000..eb7c05bdfd
--- /dev/null
+++ b/docs/guides/lint_rules/rules/sql_parse_error.md
@@ -0,0 +1,31 @@
+# MF005: sql-parse-error
+
+✨ **Formatting** ❌ Not Fixable
+
+MF005: SQL parsing errors during dependency analysis.
+
+## What it does
+
+Captures SQL parsing error logs and creates diagnostics pointing to
+problematic SQL statements in cells.
+
+## Why is this bad?
+
+SQL parsing failures can lead to:
+- Incorrect dependency analysis for SQL-using cells
+- Missing dataframe references in dependency graph
+- Reduced effectiveness of reactive execution
+- Potential runtime errors when SQL is executed
+
+## Examples
+
+**Triggered by:**
+- Invalid SQL syntax in cell code
+- Unsupported SQL dialects or extensions
+- Complex SQL that exceeds parser capabilities
+
+## References
+
+- [Understanding Errors](https://docs.marimo.io/guides/understanding_errors/)
+- [SQL Support](https://docs.marimo.io/guides/sql/)
+
diff --git a/examples/sql/connect_to_motherduck.py b/examples/sql/connect_to_motherduck.py
index 3df6b80c2c..7a003c2251 100644
--- a/examples/sql/connect_to_motherduck.py
+++ b/examples/sql/connect_to_motherduck.py
@@ -212,7 +212,7 @@ def _(MONTHS, duckdb, mo):
"""
SELECT DISTINCT type as 'HN Type'
FROM sample_data.hn.hacker_news
- WHERE score NOT NULL AND descendants NOT NULL
+ WHERE score IS NOT NULL AND descendants IS NOT NULL
LIMIT 10;
"""
).df()
@@ -259,8 +259,7 @@ def _(hn_type_select, mo, month_list):
AND
MONTH(timestamp) in ({month_list})
AND
- descendants NOT NULl
-
+ descendants IS NOT NULL
)
SELECT
diff --git a/examples/third_party/motherduck/embeddings/embeddings_explorer.py b/examples/third_party/motherduck/embeddings/embeddings_explorer.py
index 83bb75549c..15825bcdee 100644
--- a/examples/third_party/motherduck/embeddings/embeddings_explorer.py
+++ b/examples/third_party/motherduck/embeddings/embeddings_explorer.py
@@ -16,7 +16,7 @@
import marimo
-__generated_with = "0.15.5"
+__generated_with = "0.16.0"
app = marimo.App(width="medium")
with app.setup:
@@ -24,5 +24,9 @@ with app.setup:
# https://motherduck.com/blog/MotherDuck-Visualize-Embeddings-Marimo/
import marimo as mo
+
+
+
+
if __name__ == "__main__":
app.run()
diff --git a/marimo/_ast/sql_utils.py b/marimo/_ast/sql_utils.py
index c1749f14d2..52b06cd981 100644
--- a/marimo/_ast/sql_utils.py
+++ b/marimo/_ast/sql_utils.py
@@ -1,6 +1,8 @@
# Copyright 2025 Marimo. All rights reserved.
-from typing import Literal, Optional, Union
+import ast
+import re
+from typing import Callable, Literal, Optional, TypedDict, Union
from marimo import _loggers
from marimo._dependencies.dependencies import DependencyManager
@@ -17,6 +19,20 @@ SQLGLOT_DIALECTS = Literal[
]
+class SQLErrorMetadata(TypedDict):
+ """Structured metadata for SQL parsing errors."""
+
+ lint_rule: str
+ error_type: str
+ clean_message: str # Just the meaningful error without SQL trace
+ node_lineno: int
+ node_col_offset: int
+ sql_statement: str # Truncated if needed
+ sql_line: Optional[int] # 0-based line within SQL
+ sql_col: Optional[int] # 0-based column within SQL
+ context: str
+
+
def classify_sql_statement(
sql_statement: str, dialect: Optional[SQLGLOT_DIALECTS] = None
) -> Union[SQL_TYPE, Literal["unknown"]]:
@@ -33,7 +49,14 @@ def classify_sql_statement(
with _loggers.suppress_warnings_logs("sqlglot"):
expression_list = parse(sql_statement, dialect=dialect)
except ParseError as e:
- LOGGER.debug(f"Unable to parse SQL. Error: {e}")
+ log_sql_error(
+ LOGGER.debug,
+ message="Failed to parse SQL statement for classification.",
+ exception=e,
+ rule_code="MF005",
+ node=None,
+ sql_content=sql_statement,
+ )
return "unknown"
for expression in expression_list:
@@ -52,3 +75,45 @@ def classify_sql_statement(
return "DQL"
return "unknown"
+
+
+def log_sql_error(
+ logger: Callable[..., None],
+ *,
+ message: str,
+ exception: BaseException,
+ rule_code: str,
+ node: Optional[ast.expr] = None,
+ sql_content: str = "",
+ context: str = "",
+) -> None:
+ """Utility to log SQL-related errors with consistent metadata."""
+ # Parse SQL position from exception message if available
+ sql_line = None
+ sql_col = None
+
+ exception_msg = str(exception)
+ line_col_match = re.search(r"Line (\d+), Col: (\d+)", exception_msg)
+ if line_col_match:
+ sql_line = int(line_col_match.group(1)) - 1 # Convert to 0-based
+ sql_col = int(line_col_match.group(2)) - 1 # Convert to 0-based
+
+ # Truncate long SQL content
+ truncated_sql = sql_content
+ if sql_content and len(sql_content) > 200:
+ truncated_sql = sql_content[:200] + "..."
+
+ # Create metadata using TypedDict
+ metadata: SQLErrorMetadata = {
+ "lint_rule": rule_code,
+ "error_type": type(exception).__name__,
+ "clean_message": exception_msg.split("\n", 1)[0],
+ "node_lineno": node.lineno if node else 0,
+ "node_col_offset": node.col_offset if node else 0,
+ "sql_statement": truncated_sql,
+ "sql_line": sql_line,
+ "sql_col": sql_col,
+ "context": context,
+ }
+
+ logger(message, exception, extra=metadata)
diff --git a/marimo/_ast/sql_visitor.py b/marimo/_ast/sql_visitor.py
index 9084cfc0ae..bd13cc982e 100644
--- a/marimo/_ast/sql_visitor.py
+++ b/marimo/_ast/sql_visitor.py
@@ -250,6 +250,7 @@ def find_sql_defs(sql_statement: str) -> SQLDefs:
LOGGER.warning(
"Unexpected number of parts in CREATE TABLE: %s",
parts,
+ extra={"parts": parts},
)
if is_table:
@@ -497,7 +498,6 @@ def find_sql_refs(sql_statement: str) -> set[SQLRef]:
DependencyManager.sqlglot.require(why="SQL parsing")
from sqlglot import exp, parse
- from sqlglot.errors import ParseError
from sqlglot.optimizer.scope import build_scope
def get_ref_from_table(table: exp.Table) -> Optional[SQLRef]:
@@ -507,7 +507,6 @@ def find_sql_refs(sql_statement: str) -> set[SQLRef]:
catalog_name = table.catalog or None
if table_name is None:
- LOGGER.warning("Table name cannot be found in the SQL statement")
return None
# Check if the table name looks like a URL or has a file extension.
@@ -520,12 +519,9 @@ def find_sql_refs(sql_statement: str) -> set[SQLRef]:
table=table_name, schema=schema_name, catalog=catalog_name
)
- try:
- with _loggers.suppress_warnings_logs("sqlglot"):
- expression_list = parse(sql_statement, dialect="duckdb")
- except ParseError as e:
- LOGGER.error(f"Unable to parse SQL. Error: {e}")
- return set()
+ # May raise a ParseError
+ with _loggers.suppress_warnings_logs("sqlglot"):
+ expression_list = parse(sql_statement, dialect="duckdb")
refs: set[SQLRef] = set()
diff --git a/marimo/_ast/visitor.py b/marimo/_ast/visitor.py
index a03ca4b6d2..bb2951692c 100644
--- a/marimo/_ast/visitor.py
+++ b/marimo/_ast/visitor.py
@@ -12,6 +12,7 @@ from uuid import uuid4
from marimo import _loggers
from marimo._ast.errors import ImportStarError
+from marimo._ast.sql_utils import log_sql_error
from marimo._ast.sql_visitor import (
SQLDefs,
SQLKind,
@@ -26,8 +27,10 @@ from marimo._utils.strings import standardize_annotation_quotes
LOGGER = _loggers.marimo_logger()
+
Name = str
+
Language = Literal["python", "sql"]
@@ -632,6 +635,18 @@ class ScopedVisitor(ast.NodeVisitor):
and sql
):
import duckdb # type: ignore[import-not-found,import-untyped,unused-ignore] # noqa: E501
+
+ # Import ParseError outside try block so we can except it
+ # Use a Union approach to handle both cases
+ ParseErrorType: type[Exception]
+ if DependencyManager.sqlglot.has():
+ from sqlglot.errors import ParseError as SQLGLOTParseError
+
+ ParseErrorType = SQLGLOTParseError
+ else:
+ # Fallback when sqlglot is not available
+ ParseErrorType = Exception
+
# TODO: Handle other SQL languages
# TODO: Get the engine so we can differentiate tables in diff engines
@@ -655,7 +670,14 @@ class ScopedVisitor(ast.NodeVisitor):
# We catch base exceptions because we don't want to
# fail due to bugs in duckdb -- users code should
# be saveable no matter what
- LOGGER.warning("Unexpected duckdb error %s", e)
+ log_sql_error(
+ LOGGER.warning,
+ message=f"Unexpected duckdb error {e}",
+ exception=e,
+ node=node,
+ rule_code="MF005",
+ sql_content=sql,
+ )
self.generic_visit(node)
return node
@@ -667,7 +689,15 @@ class ScopedVisitor(ast.NodeVisitor):
except duckdb.ProgrammingError:
sql_defs = SQLDefs()
except BaseException as e:
- LOGGER.warning("Unexpected duckdb error %s", e)
+ log_sql_error(
+ LOGGER.warning,
+ message=f"Unexpected duckdb error {e}",
+ exception=e,
+ node=node,
+ rule_code="MF005",
+ sql_content=sql,
+ context="sql_defs_extraction",
+ )
sql_defs = SQLDefs()
defined_names = set()
@@ -699,13 +729,23 @@ class ScopedVisitor(ast.NodeVisitor):
sql_refs: set[SQLRef] = set()
try:
+ # Take results
sql_refs = find_sql_refs_cached(statement.query)
- except (duckdb.ProgrammingError, duckdb.IOException):
- LOGGER.debug(
- "Error parsing SQL statement: %s", statement.query
+ except (
+ duckdb.ProgrammingError,
+ duckdb.IOException,
+ ParseErrorType,
+ BaseException,
+ ) as e:
+ # Use first_arg (SQL string node) for accurate positioning
+ log_sql_error(
+ LOGGER.error,
+ message=f"Error parsing SQL statement: {e}",
+ exception=e,
+ node=first_arg,
+ rule_code="MF005",
+ sql_content=statement.query,
)
- except BaseException as e:
- LOGGER.warning("Unexpected duckdb error %s", e)
for ref in sql_refs:
name = ref.qualified_name
diff --git a/marimo/_lint/context.py b/marimo/_lint/context.py
index d6b8065a55..e593358079 100644
--- a/marimo/_lint/context.py
+++ b/marimo/_lint/context.py
@@ -1,14 +1,19 @@
-# Copyright 2024 Marimo. All rights reserved.
+# Copyright 2025 Marimo. All rights reserved.
from __future__ import annotations
import asyncio
import heapq
+import logging # noqa: TC003
import threading
from typing import TYPE_CHECKING
-from marimo._ast.load import load_notebook_ir
+from marimo._ast.names import SETUP_CELL_NAME
+
+# Note: load_notebook_ir not used - we do manual compilation for per-cell log capture
from marimo._lint.diagnostic import Diagnostic, Severity
-from marimo._schemas.serialization import NotebookSerialization
+from marimo._loggers import capture_output
+from marimo._schemas.serialization import CellDef, NotebookSerialization
+from marimo._types.ids import CellId_t
if TYPE_CHECKING:
from marimo._lint.rules.base import LintRule
@@ -30,6 +35,7 @@ class LintContext:
notebook: NotebookSerialization,
stderr: str = "",
stdout: str = "",
+ logs: list[logging.LogRecord] | None = None,
):
self.notebook = notebook
self._diagnostics: list[tuple[int, int, Diagnostic]] = []
@@ -43,6 +49,13 @@ class LintContext:
self.stderr = stderr
self.stdout = stdout
+ self._log_records = logs or []
+ self._logs_by_rule: dict[str, list[logging.LogRecord]] = {}
+ self._logs_by_cell_and_rule: dict[
+ str, dict[str, list[logging.LogRecord]]
+ ] = {}
+
+ self._errors: dict[str, list[tuple[Exception, CellDef]]] = {}
def _get_diagnostics_lock(self) -> asyncio.Lock:
"""Get the diagnostics lock, creating it if needed."""
@@ -105,6 +118,45 @@ class LintContext:
return new_diagnostics
+ def _group_initial_logs(self) -> None:
+ """Group initial log records by rule code."""
+ for record in self._log_records:
+ # Check if record has lint_rule metadata
+ lint_rule = getattr(record, "lint_rule", None)
+ if hasattr(record, "__dict__") and "lint_rule" in record.__dict__:
+ lint_rule = record.__dict__["lint_rule"]
+
+ # Default to MF006 (misc) if no specific rule
+ rule_code = lint_rule if lint_rule else "MF006"
+
+ if rule_code not in self._logs_by_rule:
+ self._logs_by_rule[rule_code] = []
+ self._logs_by_rule[rule_code].append(record)
+
+ def _enhance_cell_logs(
+ self,
+ cell_logs: list[logging.LogRecord],
+ cell_id: str,
+ cell_lineno: int,
+ ) -> None:
+ """Enhance log records with cell information and store globally."""
+ for record in cell_logs:
+ # Add cell information to the log record
+ if hasattr(record, "__dict__"):
+ record.__dict__["cell_id"] = cell_id
+ record.__dict__["cell_lineno"] = cell_lineno
+
+ lint_rule = getattr(record, "lint_rule", None)
+ if hasattr(record, "__dict__") and "lint_rule" in record.__dict__:
+ lint_rule = record.__dict__["lint_rule"]
+
+ rule_code = lint_rule if lint_rule else "MF006"
+ if rule_code not in self._logs_by_rule:
+ self._logs_by_rule[rule_code] = []
+ self._logs_by_rule[rule_code].append(record)
+
+ self._log_records.extend(cell_logs)
+
def get_graph(self) -> DirectedGraph:
"""Get the dependency graph, constructing it once and caching."""
if self._graph is not None:
@@ -115,13 +167,75 @@ class LintContext:
if self._graph is not None:
return self._graph
- # Construct the graph
- app = load_notebook_ir(self.notebook)
+ # Group any initial logs
+ self._group_initial_logs()
+
+ # Manually compile the graph with per-cell log capture
+ from marimo._ast.app import App, InternalApp
+ from marimo._ast.cell import CellConfig
+ from marimo._ast.cell_manager import CellManager
+ from marimo._ast.compiler import ir_cell_factory
+ from marimo._schemas.serialization import UnparsableCell
+
+ # Create the app
+ app = App(
+ **self.notebook.app.options, _filename=self.notebook.filename
+ )
self._graph = app._graph
+ # Process each cell individually to capture logs per-cell
+ for i, cell in enumerate(self.notebook.cells):
+ if isinstance(cell, UnparsableCell):
+ app._unparsable_cell(cell.code, **cell.options)
+ continue
+
+ # Capture logs during this specific cell's compilation
+ with capture_output() as (_, _, cell_logs):
+ # Call ir_cell_factory directly with proper exception handling
+ if cell.name == SETUP_CELL_NAME:
+ cell_id = CellId_t(SETUP_CELL_NAME)
+ else:
+ cell_id = app._cell_manager.create_cell_id()
+ filename = self.notebook.filename
+ cell_config = CellConfig.from_dict(cell.options)
+
+ try:
+ compiled_cell = ir_cell_factory(
+ cell, cell_id=cell_id, filename=filename
+ )
+ compiled_cell._cell.configure(cell_config)
+ # Register the successfully compiled cell
+ app._cell_manager._register_cell(
+ compiled_cell, InternalApp(app)
+ )
+ except SyntaxError as e:
+ # Handle syntax errors like register_ir_cell does
+ app._cell_manager.unparsable = True
+ app._cell_manager.register_cell(
+ cell_id=cell_id,
+ code=cell.code,
+ config=cell_config,
+ name=cell.name,
+ cell=None,
+ )
+ self._errors.setdefault("SyntaxError", []).append(
+ (e, cell)
+ )
+ except Exception as e:
+ self._errors.setdefault("unhandled", []).append(
+ (e, cell)
+ )
+
+ # Enhance logs with cell information and store globally
+ simplified_cell_id = f"cell-{i}"
+ self._enhance_cell_logs(
+ cell_logs, simplified_cell_id, cell.lineno
+ )
+
# Initialize the app to register cells in the graph
- for cell_id, cell in app._cell_manager.valid_cells():
- self._graph.register_cell(cell_id, cell._cell)
+ cell_manager: CellManager = app._cell_manager
+ for cell_id, cell_impl in cell_manager.valid_cells():
+ self._graph.register_cell(cell_id, cell_impl._cell)
return self._graph
@@ -169,3 +283,32 @@ class RuleContext:
def stderr(self) -> str:
"""Access to the captured stderr."""
return self.global_context.stderr
+
+ def get_errors(self, key: str) -> list[tuple[Exception, CellDef]]:
+ return self.global_context._errors.get(key, [])
+
+ def get_logs(
+ self, rule_code: str | None = None
+ ) -> list[logging.LogRecord]:
+ """Get log records for a specific rule or all logs if no rule specified."""
+ if rule_code is None:
+ return self.global_context._log_records
+
+ return self.global_context._logs_by_rule.get(rule_code, [])
+
+ def get_logs_for_cell(
+ self, cell_id: str, rule_code: str | None = None
+ ) -> list[logging.LogRecord]:
+ """Get log records for a specific cell and rule."""
+ if cell_id not in self.global_context._logs_by_cell_and_rule:
+ return []
+
+ cell_logs = self.global_context._logs_by_cell_and_rule[cell_id]
+ if rule_code is None:
+ # Return all logs for this cell
+ all_logs = []
+ for logs in cell_logs.values():
+ all_logs.extend(logs)
+ return all_logs
+
+ return cell_logs.get(rule_code, [])
diff --git a/marimo/_lint/linter.py b/marimo/_lint/linter.py
index 79abf05c9f..3ce31e4342 100644
--- a/marimo/_lint/linter.py
+++ b/marimo/_lint/linter.py
@@ -1,9 +1,7 @@
-# Copyright 2024 Marimo. All rights reserved.
+# Copyright 2025 Marimo. All rights reserved.
from __future__ import annotations
import asyncio
-import contextlib
-import io
import re
from dataclasses import dataclass, field
from pathlib import Path
@@ -15,6 +13,7 @@ from marimo._cli.print import red
from marimo._convert.converters import MarimoConvert
from marimo._lint.diagnostic import Diagnostic, Severity
from marimo._lint.rule_engine import EarlyStoppingConfig, RuleEngine
+from marimo._loggers import capture_output
from marimo._schemas.serialization import NotebookSerialization
if TYPE_CHECKING:
@@ -110,14 +109,8 @@ class Linter:
)
return file_status
- captured_stdout = io.StringIO()
- captured_stderr = io.StringIO()
-
try:
- with (
- contextlib.redirect_stdout(captured_stdout),
- contextlib.redirect_stderr(captured_stderr),
- ):
+ with capture_output() as (stdout, stderr, logs):
load_result = get_notebook_status(file_path)
except SyntaxError as e:
# Handle syntax errors in notebooks
@@ -170,8 +163,9 @@ class Linter:
await self.rule_engine.check_notebook(
load_result.notebook,
# Add parsing rule if there's captured output
- stdout=captured_stdout.getvalue().strip(),
- stderr=captured_stderr.getvalue().strip(),
+ stdout=stdout.getvalue().strip(),
+ stderr=stderr.getvalue().strip(),
+ logs=logs,
)
)
except Exception as e:
@@ -242,10 +236,11 @@ class Linter:
"""Generate file contents from notebook serialization."""
converter = MarimoConvert.from_ir(notebook)
- if filename.endswith((".md", ".qmd")):
- return converter.to_markdown(filename)
- else:
- return converter.to_py()
+ with capture_output():
+ if filename.endswith((".md", ".qmd")):
+ return converter.to_markdown(filename)
+ else:
+ return converter.to_py()
@staticmethod
def _generate_file_contents(file_status: FileStatus) -> str:
diff --git a/marimo/_lint/rule_engine.py b/marimo/_lint/rule_engine.py
index eb5732b380..27c1ec68ab 100644
--- a/marimo/_lint/rule_engine.py
+++ b/marimo/_lint/rule_engine.py
@@ -10,6 +10,7 @@ from marimo._lint.rules import RULE_CODES
from marimo._schemas.serialization import NotebookSerialization
if TYPE_CHECKING:
+ import logging
from collections.abc import AsyncIterator
from marimo._lint.diagnostic import Diagnostic
@@ -67,9 +68,10 @@ class RuleEngine:
notebook: NotebookSerialization,
stdout: str = "",
stderr: str = "",
+ logs: list[logging.LogRecord] | None = None,
) -> AsyncIterator[Diagnostic]:
"""Check notebook and yield diagnostics as they become available."""
- ctx = LintContext(notebook, stdout, stderr)
+ ctx = LintContext(notebook, stdout, stderr, logs)
# Create tasks for all rules with their completion tracking
pending_tasks = {
@@ -115,11 +117,12 @@ class RuleEngine:
notebook: NotebookSerialization,
stdout: str = "",
stderr: str = "",
+ logs: list[logging.LogRecord] | None = None,
) -> list[Diagnostic]:
"""Check notebook for all lint rule violations using async execution."""
diagnostics = []
async for diagnostic in self.check_notebook_streaming(
- notebook, stdout, stderr
+ notebook, stdout, stderr, logs
):
diagnostics.append(diagnostic)
return diagnostics
diff --git a/marimo/_lint/rules/breaking/syntax_error.py b/marimo/_lint/rules/breaking/syntax_error.py
index 3da7d48bd6..c23638dd8e 100644
--- a/marimo/_lint/rules/breaking/syntax_error.py
+++ b/marimo/_lint/rules/breaking/syntax_error.py
@@ -3,14 +3,9 @@ from __future__ import annotations
from typing import TYPE_CHECKING
-from marimo._ast.compiler import (
- ir_cell_factory,
-)
from marimo._ast.errors import ImportStarError
from marimo._lint.diagnostic import Diagnostic, Severity
from marimo._lint.rules.base import LintRule
-from marimo._schemas.serialization import UnparsableCell
-from marimo._types.ids import CellId_t
if TYPE_CHECKING:
from marimo._lint.context import RuleContext
@@ -98,35 +93,36 @@ class SyntaxErrorRule(LintRule):
async def check(self, ctx: RuleContext) -> None:
"""Check for syntax errors during compilation."""
- for cell in ctx.notebook.cells:
- if not isinstance(cell, UnparsableCell):
- try:
- ir_cell_factory(
- cell,
- cell_id=CellId_t("Hbol"),
- filename=ctx.notebook.filename,
- )
- # Technically a SyntaxError subclass
- except ImportStarError as e:
- line, column = _handle_import_star_error(e, cell)
- await ctx.add_diagnostic(
- Diagnostic(
- message=IMPORT_STAR_ERROR_MESSAGE,
- line=line,
- column=column,
- fix=IMPORT_STAR_HINT,
- )
+ for e, cell in ctx.get_errors("SyntaxError"):
+ if isinstance(e, ImportStarError):
+ line, column = _handle_import_star_error(e, cell)
+ await ctx.add_diagnostic(
+ Diagnostic(
+ message=IMPORT_STAR_ERROR_MESSAGE,
+ line=line,
+ column=column,
+ fix=IMPORT_STAR_HINT,
)
- except SyntaxError as e:
+ )
+ else:
+ # Handle SyntaxError specifically
+ if isinstance(e, SyntaxError):
message = f"{e.msg}"
- await ctx.add_diagnostic(
- Diagnostic(
- message=message,
- line=cell.lineno + (e.lineno or 1) - 1,
- column=(e.offset or 1),
- fix=_get_known_hints(message),
- )
+ line = cell.lineno + (e.lineno or 1) - 1
+ column = e.offset or 1
+ else:
+ # For other exceptions, use string representation
+ message = str(e)
+ line = cell.lineno
+ column = 1
+ await ctx.add_diagnostic(
+ Diagnostic(
+ message=message,
+ line=line,
+ column=column,
+ fix=_get_known_hints(message),
)
+ )
def _handle_import_star_error(
diff --git a/marimo/_lint/rules/formatting/__init__.py b/marimo/_lint/rules/formatting/__init__.py
index 0d618646b4..977c612def 100644
--- a/marimo/_lint/rules/formatting/__init__.py
+++ b/marimo/_lint/rules/formatting/__init__.py
@@ -2,13 +2,20 @@
from marimo._lint.rules.base import LintRule, UnsafeFixRule
from marimo._lint.rules.formatting.empty_cells import EmptyCellRule
from marimo._lint.rules.formatting.general import GeneralFormattingRule
-from marimo._lint.rules.formatting.parsing import StderrRule, StdoutRule
+from marimo._lint.rules.formatting.parsing import (
+ MiscLogRule,
+ SqlParseRule,
+ StderrRule,
+ StdoutRule,
+)
FORMATTING_RULE_CODES: dict[str, type[LintRule]] = {
"MF001": GeneralFormattingRule,
"MF002": StdoutRule,
"MF003": StderrRule,
"MF004": EmptyCellRule,
+ "MF005": SqlParseRule,
+ "MF006": MiscLogRule,
}
__all__ = [
@@ -17,5 +24,7 @@ __all__ = [
"StdoutRule",
"StderrRule",
"EmptyCellRule",
+ "SqlParseRule",
+ "MiscLogRule",
"UnsafeFixRule",
]
diff --git a/marimo/_lint/rules/formatting/parsing.py b/marimo/_lint/rules/formatting/parsing.py
index 6a1fb4e8b3..02d57dbd26 100644
--- a/marimo/_lint/rules/formatting/parsing.py
+++ b/marimo/_lint/rules/formatting/parsing.py
@@ -1,6 +1,7 @@
-# Copyright 2024 Marimo. All rights reserved.
+# Copyright 2025 Marimo. All rights reserved.
from __future__ import annotations
+import logging
import re
from typing import TYPE_CHECKING
@@ -155,3 +156,144 @@ class StderrRule(LintRule):
column=0,
)
)
+
+
+class SqlParseRule(LintRule):
+ """MF005: SQL parsing errors during dependency analysis.
+
+ This rule processes log messages captured when marimo encounters errors
+ while parsing SQL statements in notebook cells. SQL parsing is used for
+ dependency analysis and dataframe tracking.
+
+ ## What it does
+
+ Captures SQL parsing error logs and creates diagnostics pointing to
+ problematic SQL statements in cells.
+
+ ## Why is this bad?
+
+ SQL parsing failures can lead to:
+ - Incorrect dependency analysis for SQL-using cells
+ - Missing dataframe references in dependency graph
+ - Reduced effectiveness of reactive execution
+ - Potential runtime errors when SQL is executed
+
+ ## Examples
+
+ **Triggered by:**
+ - Invalid SQL syntax in cell code
+ - Unsupported SQL dialects or extensions
+ - Complex SQL that exceeds parser capabilities
+
+ ## References
+
+ - [Understanding Errors](https://docs.marimo.io/guides/understanding_errors/)
+ - [SQL Support](https://docs.marimo.io/guides/sql/)
+ """
+
+ code = "MF005"
+ name = "sql-parse-error"
+ description = "SQL parsing errors during dependency analysis"
+ severity = Severity.FORMATTING
+ fixable = False
+
+ async def check(self, ctx: RuleContext) -> None:
+ """Process SQL parsing error logs."""
+ logs = ctx.get_logs(self.code)
+
+ for record in logs:
+ # Extract metadata from log record - ONLY use extra_data
+ extra_data = getattr(record, "__dict__", {})
+
+ # Use clean message from metadata (without SQL trace)
+ message = extra_data.get("clean_message", "SQL parsing error")
+
+ # Calculate line position using cell information
+ cell_lineno = extra_data.get(
+ "cell_lineno", 0
+ ) # Cell start line in notebook
+ node_lineno = extra_data.get(
+ "node_lineno", 1
+ ) # Node line within cell
+ sql_line = extra_data.get("sql_line") # SQL line within SQL string
+
+ # Start with cell position + node position within cell
+ line = cell_lineno + node_lineno - 1 # Convert to 0-based
+
+ # Add SQL line offset if available
+ if sql_line is not None:
+ line += sql_line
+
+ # Use SQL column if available, otherwise node column
+ sql_col = extra_data.get("sql_col")
+ col = (
+ sql_col
+ if sql_col is not None
+ else extra_data.get("node_col_offset", 0)
+ )
+
+ await ctx.add_diagnostic(
+ Diagnostic(
+ message=message,
+ line=line,
+ cell_id=None,
+ column=col,
+ )
+ )
+
+
+class MiscLogRule(LintRule):
+ """MF006: Miscellaneous log messages during processing.
+
+ This rule processes log messages that don't have a specific rule assigned
+ but may still be relevant for understanding notebook health and potential
+ issues during processing.
+
+ ## What it does
+
+ Captures warning and error level log messages that aren't handled by
+ other specific log rules and creates diagnostics to surface them.
+
+ ## Why is this bad?
+
+ Unhandled log messages may indicate:
+ - Unexpected issues during notebook processing
+ - Configuration problems
+ - Library warnings that affect execution
+ - Performance or resource issues
+
+ ## Examples
+
+ **Triggered by:**
+ - General warnings from imported libraries
+ - Configuration issues
+ - Unexpected errors during processing
+
+ ## References
+
+ - [Understanding Errors](https://docs.marimo.io/guides/understanding_errors/)
+ """
+
+ code = "MF006"
+ name = "misc-log-capture"
+ description = "Miscellaneous log messages during processing"
+ severity = Severity.FORMATTING
+ fixable = False
+
+ async def check(self, ctx: RuleContext) -> None:
+ """Process miscellaneous log messages."""
+ logs = ctx.get_logs(self.code)
+
+ for record in logs:
+ # Only process WARNING and ERROR level logs to avoid noise
+ if record.levelno < logging.WARNING:
+ continue
+
+ await ctx.add_diagnostic(
+ Diagnostic(
+ message=record.getMessage(),
+ line=0, # Misc logs don't have meaningful line positioning
+ cell_id=None,
+ column=0,
+ )
+ )
diff --git a/marimo/_loggers.py b/marimo/_loggers.py
index 8a958afbb1..51079b4d1e 100644
--- a/marimo/_loggers.py
+++ b/marimo/_loggers.py
@@ -1,6 +1,7 @@
# Copyright 2024 Marimo. All rights reserved.
from __future__ import annotations
+import io
import logging
from contextlib import contextmanager
from logging.handlers import TimedRotatingFileHandler
@@ -9,7 +10,7 @@ from typing import TYPE_CHECKING, Optional
from marimo._utils.log_formatter import LogFormatter
if TYPE_CHECKING:
- from collections.abc import Generator
+ from collections.abc import Generator, Iterator
from pathlib import Path
# This file manages and creates loggers used throughout marimo.
@@ -182,6 +183,49 @@ def _file_handler() -> logging.FileHandler:
return file_handler
+class ListHandler(logging.Handler):
+ """Log handler that captures records in a list for processing."""
+
+ def __init__(self, level: int = logging.NOTSET) -> None:
+ super().__init__(level)
+ self.records: list[logging.LogRecord] = []
+
+ def emit(self, record: logging.LogRecord) -> None:
+ self.records.append(record)
+
+
+@contextmanager
+def capture_output(
+ *, log_level: int = logging.DEBUG
+) -> Iterator[tuple[io.StringIO, io.StringIO, list[logging.LogRecord]]]:
+ """
+ Capture stdout, stderr, and log records during execution.
+
+ Yields: (stdout_buf, stderr_buf, log_records)
+ - log_records are raw LogRecord objects with full metadata (name, level, pathname, lineno, created, etc.).
+ - Log emission to existing handlers is silenced inside the context.
+ """
+ import contextlib
+
+ logger = marimo_logger()
+ out, err = io.StringIO(), io.StringIO()
+ h: ListHandler = ListHandler(log_level)
+
+ # Snapshot and suppress existing handlers
+ old_handlers, old_propagate = logger.handlers[:], logger.propagate
+ logger.addHandler(h)
+ logger.handlers = [h]
+ logger.propagate = False
+
+ try:
+ with contextlib.redirect_stdout(out), contextlib.redirect_stderr(err):
+ yield out, err, h.records
+ finally:
+ logger.removeHandler(h)
+ logger.handlers = old_handlers
+ logger.propagate = old_propagate
+
+
@contextmanager
def suppress_warnings_logs(name: str) -> Generator[None, None, None]:
"""Suppress logs for a given logger."""
diff --git a/mkdocs.yml b/mkdocs.yml
index 0a9a01adcf..b979736ab1 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -105,6 +105,8 @@ nav:
- Parse stdout: guides/lint_rules/rules/parse_stdout.md
- Parse stderr: guides/lint_rules/rules/parse_stderr.md
- Empty cells: guides/lint_rules/rules/empty_cells.md
+ - SQL parse error: guides/lint_rules/rules/sql_parse_error.md
+ - Misc parse log: guides/lint_rules/rules/misc_log_capture.md
- SQL, dataframes, and plots:
- SQL, dataframes, and plots: guides/working_with_data/index.md
- DataFrames: guides/working_with_data/dataframes.md
diff --git a/tests/_ast/test_sql_visitor.py b/tests/_ast/test_sql_visitor.py
index 30eed88aa6..cb2023fe8b 100644
--- a/tests/_ast/test_sql_visitor.py
+++ b/tests/_ast/test_sql_visitor.py
@@ -788,7 +788,10 @@ class TestFindSQLRefs:
def test_invalid_sql(self) -> None:
sql = "SELECT * FROM"
- assert find_sql_refs(sql) == set()
+ from sqlglot import ParseError
+
+ with pytest.raises(ParseError):
+ find_sql_refs(sql)
def test_dml_with_subquery(self) -> None:
# first
diff --git a/tests/_lint/snapshots/sql_parsing_errors.txt b/tests/_lint/snapshots/sql_parsing_errors.txt
new file mode 100644
index 0000000000..f591190f4b
--- /dev/null
+++ b/tests/_lint/snapshots/sql_parsing_errors.txt
@@ -0,0 +1,6 @@
+warning[sql-parse-error]: Expecting ). Line 19, Col: 21.
+ --> tests/_lint/test_files/sql_parsing_errors.py:36:20
+ 36 | AND
+ 37 | descendants NOT NULL
+ | ^
+ 38 | )
\ No newline at end of file
diff --git a/tests/_lint/test_files/sql_parsing_errors.py b/tests/_lint/test_files/sql_parsing_errors.py
new file mode 100644
index 0000000000..91e0ec1d36
--- /dev/null
+++ b/tests/_lint/test_files/sql_parsing_errors.py
@@ -0,0 +1,56 @@
+# Copyright 2025 Marimo. All rights reserved.
+"""Test file with SQL parsing errors to test log rules positioning."""
+
+import marimo
+
+__generated_with = "0.8.0"
+app = marimo.App()
+
+
+def _():
+ import marimo as mo
+ return mo,
+
+
+def _(mo):
+ # This should trigger an MF005 SQL parsing error due to trailing comma
+ result = mo.sql(f"""
+ WITH ranked_stories AS (
+ SELECT
+ title,
+ score,
+ type,
+ descendants,
+ YEAR(timestamp) AS year,
+ MONTH(timestamp) AS month,
+ ROW_NUMBER()
+ OVER (PARTITION BY YEAR(timestamp), MONTH(timestamp) ORDER BY score DESC)
+ AS rn
+ FROM sample_data.hn.hacker_news
+ WHERE
+ type = 'story'
+ AND
+ MONTH(timestamp) in (null)
+ AND
+ descendants NOT NULL
+ )
+
+ SELECT
+ month,
+ score,
+ type,
+ title,
+ hn_url,
+ descendants as nb_comments,
+ year,
+ FROM ranked_stories
+ WHERE rn = 1
+ ORDER BY year, month;
+ """)
+ return result,
+
+
+if __name__ == "__main__":
+ app.run()
diff --git a/tests/_lint/test_log_rules.py b/tests/_lint/test_log_rules.py
new file mode 100644
index 0000000000..8e9006acaa
--- /dev/null
+++ b/tests/_lint/test_log_rules.py
@@ -0,0 +1,158 @@
+# Copyright 2025 Marimo. All rights reserved.
+from __future__ import annotations
+
+import logging
+
+from marimo._lint.context import LintContext, RuleContext
+from marimo._lint.rules.formatting.parsing import (
+ MiscLogRule,
+ SqlParseRule,
+)
+from marimo._schemas.serialization import NotebookSerialization
+
+
+class TestLogRules:
+ """Test log message rules and grouping."""
+
+ def test_log_grouping_async_safe(self):
+ """Test that log grouping is async-safe and works correctly."""
+ # Create mock log records with different rule targets
+ record1 = logging.LogRecord(
+ name="marimo",
+ level=logging.ERROR,
+ pathname="/test.py",
+ lineno=10,
+ msg="SQL parse error",
+ args=(),
+ exc_info=None,
+ )
+ record1.__dict__["lint_rule"] = "MF005"
+
+ record2 = logging.LogRecord(
+ name="marimo",
+ level=logging.WARNING,
+ pathname="/test.py",
+ lineno=30,
+ msg="General warning",
+ args=(),
+ exc_info=None,
+ )
+ # No lint_rule specified - should go to MF006
+
+ # Create minimal notebook
+ notebook = NotebookSerialization(
+ filename="test.py", cells=[], app=None
+ )
+
+ # Test with initial logs
+ ctx = LintContext(notebook, logs=[record1, record2])
+ ctx._group_initial_logs()
+
+ # Verify logs are correctly grouped
+ assert len(ctx._logs_by_rule.get("MF005", [])) == 1
+ assert len(ctx._logs_by_rule.get("MF006", [])) == 1
+ assert ctx._logs_by_rule["MF005"][0].getMessage() == "SQL parse error"
+ assert ctx._logs_by_rule["MF006"][0].getMessage() == "General warning"
+
+ async def test_sql_parse_rule(self):
+ """Test SqlParseRule processes MF005 logs correctly."""
+ record = logging.LogRecord(
+ name="marimo",
+ level=logging.ERROR,
+ pathname="/test.py",
+ lineno=20,
+ msg="SQL parsing error",
+ args=(),
+ exc_info=None,
+ )
+ record.__dict__["lint_rule"] = "MF005"
+
+ notebook = NotebookSerialization(
+ filename="test.py", cells=[], app=None
+ )
+ ctx = LintContext(notebook, logs=[record])
+ ctx._group_initial_logs()
+
+ rule = SqlParseRule()
+ rule_ctx = RuleContext(ctx, rule)
+ await rule.check(rule_ctx)
+
+ diagnostics = await ctx.get_diagnostics()
+ assert len(diagnostics) == 1
+ assert diagnostics[0].code == "MF005"
+ assert diagnostics[0].message == "SQL parsing error"
+
+ async def test_misc_log_rule(self):
+ """Test MiscLogRule processes unspecified logs correctly."""
+ # WARNING level - should be processed
+ record1 = logging.LogRecord(
+ name="marimo",
+ level=logging.WARNING,
+ pathname="/test.py",
+ lineno=10,
+ msg="General warning",
+ args=(),
+ exc_info=None,
+ )
+
+ # DEBUG level - should be ignored
+ record2 = logging.LogRecord(
+ name="marimo",
+ level=logging.DEBUG,
+ pathname="/test.py",
+ lineno=20,
+ msg="Debug message",
+ args=(),
+ exc_info=None,
+ )
+
+ notebook = NotebookSerialization(
+ filename="test.py", cells=[], app=None
+ )
+ ctx = LintContext(notebook, logs=[record1, record2])
+ ctx._group_initial_logs()
+
+ rule = MiscLogRule()
+ rule_ctx = RuleContext(ctx, rule)
+ await rule.check(rule_ctx)
+
+ diagnostics = await ctx.get_diagnostics()
+ # Only WARNING should create a diagnostic
+ assert len(diagnostics) == 1
+ assert diagnostics[0].code == "MF006"
+ assert diagnostics[0].message == "General warning"
+
+ def test_rule_context_get_logs(self):
+ """Test RuleContext.get_logs method."""
+ record1 = logging.LogRecord(
+ name="marimo",
+ level=logging.ERROR,
+ pathname="/test.py",
+ lineno=10,
+ msg="SQL parse error",
+ args=(),
+ exc_info=None,
+ )
+ record1.__dict__["lint_rule"] = "MF005"
+
+ notebook = NotebookSerialization(
+ filename="test.py", cells=[], app=None
+ )
+ ctx = LintContext(notebook, logs=[record1])
+ ctx._group_initial_logs()
+
+ rule = SqlParseRule()
+ rule_ctx = RuleContext(ctx, rule)
+
+ # Test getting specific rule logs
+ mf005_logs = rule_ctx.get_logs("MF005")
+ assert len(mf005_logs) == 1
+ assert mf005_logs[0].getMessage() == "SQL parse error"
+
+ # Test getting non-existent rule logs
+ empty_logs = rule_ctx.get_logs("MF999")
+ assert len(empty_logs) == 0
+
+ # Test getting all logs
+ all_logs = rule_ctx.get_logs(None)
+ assert len(all_logs) == 1
diff --git a/tests/_lint/test_sql_log_rules_snapshot.py b/tests/_lint/test_sql_log_rules_snapshot.py
new file mode 100644
index 0000000000..fafa0f100f
--- /dev/null
+++ b/tests/_lint/test_sql_log_rules_snapshot.py
@@ -0,0 +1,36 @@
+# Copyright 2025 Marimo. All rights reserved.
+"""Snapshot tests for SQL log message lint rules."""
+
+import pytest
+
+from marimo._ast.parse import parse_notebook
+from tests._lint.utils import lint_notebook
+from tests.mocks import snapshotter
+
+snapshot = snapshotter(__file__)
+
+
+# Only run this in 3.12 since the formatting may differ in other versions
[email protected]("sys.version_info < (3, 13)")
+def test_sql_parsing_errors_snapshot():
+ """Test snapshot for SQL parsing log errors with positioning."""
+ file = "tests/_lint/test_files/sql_parsing_errors.py"
+ with open(file) as f:
+ code = f.read()
+
+ notebook = parse_notebook(code, filepath=file)
+ errors = lint_notebook(notebook)
+
+ log_errors = [
+ error for error in errors if error.code in ("MF005", "MF006")
+ ]
+
+ # Format errors for snapshot
+ error_output = []
+ for error in log_errors:
+ error_output.append(error.format())
+
+ if not error_output:
+ error_output = ["No SQL log errors found"]
+
+ snapshot("sql_parsing_errors.txt", "\n".join(error_output))