Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

Move all eight sqlglot declarations to the 30 line, and give the forward matrix leg somewhere to point (#856)

Beenu Aroraca272d8243e1
diff --git a/.github/workflows/check-openapi.yml b/.github/workflows/check-openapi.yml
index c07ec8382..49d26a091 100644
--- a/.github/workflows/check-openapi.yml
+++ b/.github/workflows/check-openapi.yml
@@ -27,7 +27,7 @@ env:
opentelemetry-exporter-otlp-proto-grpc
"opentelemetry-instrumentation-fastapi"
"strawberry-graphql[fastapi]" httpx email-validator "PyJWT>=2.8,<3"
- "sqlglot>=23,<27"
+ "sqlglot>=30,<31"
jobs:
openapi-check:
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index aa35c6e91..651e12fc5 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -609,7 +609,7 @@ jobs:
"opentelemetry-instrumentation-fastapi"
"strawberry-graphql[fastapi]"
neo4j redis celery httpx aiofiles email-validator "PyJWT>=2.8,<3"
- "sqlglot>=23,<27" aiosqlite
+ "sqlglot>=30,<31" aiosqlite
"croniter>=2.0,<7.0"
pytest pytest-asyncio
steps:
diff --git a/.github/workflows/cross-tenant-rbac.yml b/.github/workflows/cross-tenant-rbac.yml
index 81b3c9ff9..a309f7802 100644
--- a/.github/workflows/cross-tenant-rbac.yml
+++ b/.github/workflows/cross-tenant-rbac.yml
@@ -86,7 +86,7 @@ jobs:
"opentelemetry-instrumentation-fastapi"
"strawberry-graphql[fastapi]"
neo4j redis celery httpx aiofiles email-validator "PyJWT>=2.8,<3"
- "sqlglot>=23,<27" aiosqlite
+ "sqlglot>=30,<31" aiosqlite
pytest pytest-asyncio
steps:
diff --git a/.github/workflows/integration.yml b/.github/workflows/integration.yml
index ae849258e..37237ced3 100644
--- a/.github/workflows/integration.yml
+++ b/.github/workflows/integration.yml
@@ -409,7 +409,7 @@ jobs:
"opentelemetry-instrumentation-fastapi"
"strawberry-graphql[fastapi]"
neo4j redis celery httpx aiofiles email-validator "PyJWT>=2.8,<3"
- "sqlglot>=23,<27" aiosqlite "croniter>=2.0,<7.0"
+ "sqlglot>=30,<31" aiosqlite "croniter>=2.0,<7.0"
- name: Apply full migration chain on fresh Postgres (strict)
working-directory: services/api
@@ -870,7 +870,7 @@ jobs:
"opentelemetry-instrumentation-fastapi"
"strawberry-graphql[fastapi]"
neo4j redis celery httpx aiofiles email-validator "PyJWT>=2.8,<3"
- "sqlglot>=23,<27" aiosqlite "croniter>=2.0,<7.0"
+ "sqlglot>=30,<31" aiosqlite "croniter>=2.0,<7.0"
- name: Record HEAD sha
id: head
diff --git a/.github/workflows/isolation-live.yml b/.github/workflows/isolation-live.yml
index d45e9b016..fb291f9fb 100644
--- a/.github/workflows/isolation-live.yml
+++ b/.github/workflows/isolation-live.yml
@@ -44,13 +44,22 @@ jobs:
- name: Install clients + test deps
# sqlglot MUST be pinned to the same range the API service ships
- # (`>=23,<27`): the ClickHouse test runs the production
- # lake_sql.rewrite_for_tenant, and sqlglot >= 27 silently stops
- # injecting the tenant WHERE predicate (the tenant-isolation break this
- # very gate exists to catch). test_lake_sql.py guards the pin itself.
+ # (`>=30,<31`): the ClickHouse test runs the production
+ # lake_sql.rewrite_for_tenant, so a version this job resolves and the
+ # image does not is a different isolation guarantee than the one under
+ # test. `scripts/check_sqlglot_pin.py` is what keeps the two equal.
+ #
+ # The historical break is why the pin is enforced rather than
+ # suggested: sqlglot 27 renamed the SELECT's FROM argument key, the
+ # table walk found nothing, and every single-table query came back
+ # with no tenant predicate, no allowlist check and no table-function
+ # ban — reported as success. `lake_sql._from_clause` now resolves the
+ # clause by node type rather than by key name, so that specific
+ # rename cannot recur; the agreement requirement stands because the
+ # next AST change will not announce itself either.
run: >-
pip install --quiet
- pytest pytest-asyncio structlog "sqlglot>=23,<27"
+ pytest pytest-asyncio structlog "sqlglot>=30,<31"
neo4j "redis>=5,<6" clickhouse-driver "kafka-python>=2,<3"
# Single-container stores run as `services:`-style `docker run` so we can
diff --git a/.github/workflows/lake-isolation.yml b/.github/workflows/lake-isolation.yml
index 273bb7a15..070e21c6c 100644
--- a/.github/workflows/lake-isolation.yml
+++ b/.github/workflows/lake-isolation.yml
@@ -22,9 +22,39 @@ name: Lake tenant isolation
# range, so two builds of one commit cannot disagree about the
# version of the parser that enforces isolation.
# 2. The rewriter suite runs against the pinned range *and* against the
-# next major. The second leg is forward-compatibility coverage: we
-# ship the pinned range, but a future bump must fail loudly here
-# rather than quietly downgrade isolation in production.
+# newest sqlglot pip will resolve. The second leg is forward-compatibility
+# coverage: we ship the pinned range, but a future bump must fail loudly
+# here rather than quietly downgrade isolation in production.
+#
+# Why the forward leg is unbounded above rather than naming the next major.
+# It used to read `>=27,<31`, which worked only while the shipped range sat
+# below 27. With the pin on the 30 line there is no sqlglot 31 to name, and
+# the three ways of writing that leg are all worse than leaving the ceiling
+# off:
+#
+# `>=31` pip cannot resolve it, so the leg is red every run for a
+# reason that has nothing to do with the change under test.
+# `>=30,<31` identical to the shipped leg. Two legs, one question, and
+# the matrix reports two greens for one piece of evidence.
+# `>=27,<30` tests majors *below* the pin. That is regression coverage
+# for versions nothing installs, and it removes the only
+# thing standing between a future major and production —
+# which is precisely the hole 27 came through.
+#
+# Unbounded resolves the newest release on every run, so the leg arms itself
+# the day sqlglot 31 is published, with no edit and no one remembering. Until
+# then it resolves inside the shipped range, and the `Forward coverage`
+# step below says so as a warning annotation rather than letting a duplicate
+# green pass for evidence. `scripts/check_sqlglot_pin.py` enforces the
+# structural half — the leg must carry no upper bound and must not float
+# below the shipped floor — because that is the part that can rot silently.
+# Whether upstream has published a higher major is not this repository's to
+# assert.
+#
+# The cost is stated rather than hidden: this leg depends on the outside
+# world, so an upstream release can turn it red without a local change. That
+# is the job. The workflow is path-filtered to the rewriter, its tests and
+# its pins, so the people it interrupts are the ones who need to know.
on:
push:
@@ -72,11 +102,12 @@ jobs:
matrix:
include:
# The range the service ships and CI installs everywhere else.
- - sqlglot: '>=23,<27'
+ - sqlglot: '>=30,<31'
label: shipped
- # The next major. Not shipped; proves a bump cannot silently
- # turn the isolation guarantee off the way 27 did.
- - sqlglot: '>=27,<31'
+ # Whatever is newest. Not shipped; proves a bump cannot silently
+ # turn the isolation guarantee off the way 27 did. No ceiling, so
+ # it starts testing the next major the day one exists.
+ - sqlglot: '>=30'
label: forward-compat
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
@@ -84,9 +115,34 @@ jobs:
with:
python-version: '3.11'
- name: Install
- run: pip install --quiet pytest "sqlglot${{ matrix.sqlglot }}"
+ run: pip install --quiet --upgrade pytest "sqlglot${{ matrix.sqlglot }}"
- name: Record resolved version
run: python -c "import sqlglot; print('resolved sqlglot', sqlglot.__version__)"
+ - name: Forward coverage — is this leg above the shipped ceiling?
+ if: matrix.label == 'forward-compat'
+ # A leg that resolves back inside the shipped range is re-running the
+ # shipped version. The suite below is still worth running, but the
+ # result is not forward coverage and must not read as though it were.
+ run: |
+ python - <<'PY'
+ import sqlglot
+
+ # Kept equal to the `<N` bound of the shipped range by
+ # `scripts/check_sqlglot_pin.py`, so this number cannot drift away
+ # from the pin it describes and start reporting the wrong verdict.
+ SHIPPED_CEILING = 31
+
+ major = int(sqlglot.__version__.split(".")[0])
+ if major >= SHIPPED_CEILING:
+ print(f"forward coverage: sqlglot {sqlglot.__version__} is above the shipped ceiling {SHIPPED_CEILING}")
+ else:
+ print(
+ f"::warning title=No forward coverage today::sqlglot {sqlglot.__version__} is the newest "
+ f"release and it is inside the shipped range, so this leg is re-running the shipped "
+ f"version rather than proving a future major is safe. It becomes real forward coverage "
+ f"with no edit to this file once {SHIPPED_CEILING}.0.0 is published."
+ )
+ PY
- name: Rewriter + fail-closed suites
working-directory: services/api
run: python -m pytest tests/test_lake_sql.py tests/test_lake_sql_fail_closed.py -q
diff --git a/.github/workflows/reproducible-builds.yml b/.github/workflows/reproducible-builds.yml
index 2b2a70a72..75d3b53b2 100644
--- a/.github/workflows/reproducible-builds.yml
+++ b/.github/workflows/reproducible-builds.yml
@@ -173,7 +173,7 @@ jobs:
"opentelemetry-instrumentation-fastapi"
"strawberry-graphql[fastapi]"
neo4j redis celery httpx aiofiles email-validator
- "sqlglot>=23,<27" aiosqlite "croniter>=2.0,<7.0"
+ "sqlglot>=30,<31" aiosqlite "croniter>=2.0,<7.0"
pytest pytest-asyncio
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 0a4efcb7a..2c93b7851 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -9,6 +9,58 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Changed
+- **`sqlglot` moved to the 30 line across all eight declarations, and the
+ forward-compatibility matrix leg was rewritten so it can still reach past
+ the pin.** The bump itself was never the work: a single-path proposal moved
+ 1 of 8 declarations and left six workflows at `>=23,<27`, which
+ `scripts/check_sqlglot_pin.py` fails closed on — correctly, since that gate
+ exists because sqlglot 27 renamed the SELECT's FROM argument key and
+ silently stripped the tenant predicate, the table allowlist and the `url()`
+ exfiltration ban from every single-table query while reporting success. All
+ eight now read `>=30,<31` (`services/api/pyproject.toml` plus `ci.yml`,
+ `integration.yml` ×2, `isolation-live.yml`, `cross-tenant-rbac.yml`,
+ `check-openapi.yml`, `reproducible-builds.yml`), and
+ `services/api/poetry.lock` resolves 30.19.0 inside it.
+
+ One major wide rather than four. The previous range permitted 23–26 and
+ exactly one of those was ever exercised, so three majors were installable
+ and untested; 27, 28 and 29 are likewise unexercised, and a range states
+ what may be installed rather than what upstream has released.
+
+ Measured before moving, on 64 adversarial cases written against the
+ documented policy rather than reusing the module's fixtures — DDL, DML,
+ admin verbs, multi-statement input, seven ClickHouse table functions
+ including ones hidden in subqueries and CTE bodies, non-allowlisted tables
+ reached through JOIN, subquery, UNION arm and CTE, and five forged
+ `tenant_id` predicates. 26.33.0 and 30.19.0 return the same disposition on
+ all 64 (23 rewritten, 33 `LakeSqlForbiddenError`, 8 `LakeSqlSyntaxError`)
+ and the rendered SQL of all 23 rewritten cases is byte-identical. Two cases
+ differ in wording only and both remain refusals: `DETACH TABLE` now reaches
+ the statement-kind check instead of failing to parse, and `KILL`'s class
+ moved to `sqlglot.expressions.ddl`, which the parse error quotes. The
+ in-tree suites pass 58/58 on each major.
+
+ The forward leg is the part that needed a decision. It read `>=27,<31`,
+ which only reached past the pin while the pin sat below 27; there is no
+ sqlglot 31, so with the pin on the 30 line every way of naming a ceiling is
+ worse than leaving it off — `>=31` cannot be installed and reds the job
+ forever, `>=30,<31` duplicates the shipped leg and reports two greens for
+ one piece of evidence, and retargeting to `>=27,<30` tests majors below the
+ pin, which is regression coverage for versions nothing installs and leaves
+ nothing between the next major and production. The leg is now unbounded
+ above, so it resolves whatever is newest and arms itself the day a higher
+ major is published with no edit. Because that means it re-runs the shipped
+ version today, a `Forward coverage` step emits a warning annotation saying
+ so rather than letting a duplicate green pass for evidence, and
+ `check_sqlglot_pin.py` gained the structural half in both directions: the
+ forward leg must carry no upper bound and must not float below the shipped
+ floor, and the step's `SHIPPED_CEILING` must equal the shipped range's `<N`
+ bound so a stale copy cannot report the wrong verdict. Seven injected
+ violations in `--self-test`, which runs before the gate. What is
+ deliberately not asserted is that a higher major exists: upstream's release
+ schedule is not this repository's to require, and a gate red until sqlglot
+ ships 31 would be red for a reason no change here could fix.
+
- **Four root-workspace dependency bumps landed as one lockfile change.**
`cytoscape` 3.34.0 → 3.34.3, `lucide-react` 1.28.0 → 1.48.0,
`@modelcontextprotocol/sdk` 1.29.0 → 1.30.1 and `tsx` 4.23.1 → 4.23.15 all
diff --git a/scripts/check_sqlglot_pin.py b/scripts/check_sqlglot_pin.py
index 572afbd6e..dab00ec6c 100644
--- a/scripts/check_sqlglot_pin.py
+++ b/scripts/check_sqlglot_pin.py
@@ -31,9 +31,7 @@ from pathlib import Path
# test loads it by path with importlib. gate_toolkit sits beside it either way.
sys.path.insert(0, str(Path(__file__).resolve().parent))
-from gate_toolkit import repo_root, self_test_if_requested
-
-self_test_if_requested(__file__)
+from gate_toolkit import SELF_TEST_FLAG, repo_root, self_test_main
REPO_ROOT = repo_root()
# Every file that installs sqlglot for a build or a test run. A new one must
@@ -50,12 +48,40 @@ DECLARING_FILES = (
)
# `lake-isolation.yml` installs sqlglot from a build matrix, deliberately
-# including a version outside the shipped range so a future bump cannot
-# silently turn isolation off. It therefore cannot declare a single pin. What
-# it *must* do is test the version we ship, so the matrix leg labelled
-# `shipped` is checked against the agreed pin below.
+# reaching past the shipped range so a future bump cannot silently turn
+# isolation off. It therefore cannot declare a single pin, and is checked on
+# its own terms instead of being exempted:
+#
+# `shipped` must equal the agreed pin, or the matrix proves a version
+# nothing installs still works.
+# `forward-compat` must carry no upper bound, and must not float below the
+# shipped floor.
+#
+# The second rule is the one with history behind it. That leg used to read
+# `>=27,<31`, which only reached past the pin while the pin sat below 27.
+# With the pin on the 30 line there is no sqlglot 31 to name, so the leg is
+# written unbounded and resolves whatever is newest: it re-runs the shipped
+# version today and arms itself the day a higher major is published, with no
+# edit and nobody remembering. A ceiling would take that away silently, which
+# is exactly how 27 reached production — every install path pinned below the
+# version that broke isolation, so nothing ever ran on it.
+#
+# What is deliberately *not* asserted here is that a higher major exists.
+# Upstream's release schedule is not this repository's to require, and a gate
+# that failed until sqlglot shipped 31 would be red for a reason no change in
+# this tree could fix. The workflow's own `Forward coverage` step reports
+# which of the two states it is in on every run.
MATRIX_WORKFLOW = ".github/workflows/lake-isolation.yml"
-_SHIPPED_MATRIX_LEG = re.compile(r"""-\s*sqlglot:\s*["'](?P<spec>[^"']+)["']\s*\n\s*label:\s*shipped""")
+_MATRIX_LEG = re.compile(r"""-\s*sqlglot:\s*["'](?P<spec>[^"']+)["']\s*\n\s*label:\s*(?P<label>[\w-]+)""")
+
+# The ceiling the workflow's `Forward coverage` step compares against. It is a
+# second copy of the shipped range's `<N` bound, so it is checked against the
+# first rather than trusted: a stale copy would report "above the shipped
+# ceiling" about a version that is inside it.
+_WORKFLOW_CEILING = re.compile(r"""^\s*SHIPPED_CEILING\s*=\s*(?P<major>\d+)\s*$""", re.MULTILINE)
+
+SHIPPED_LABEL = "shipped"
+FORWARD_LABEL = "forward-compat"
# `services/api/Dockerfile` used to be on the list above, because its pip
# fallback carried its own copy of the dependency list. That fallback is gone:
@@ -96,6 +122,87 @@ def satisfies(version: str, spec: str) -> bool:
return False
return True
+
+def _bound(spec: str, operators: tuple[str, ...]) -> tuple[int, ...] | None:
+ """The version named by the first clause using one of ``operators``."""
+ for clause in spec.replace(" ", "").split(","):
+ match = re.fullmatch(r"(?P<op>[<>=!]+)(?P<ver>[0-9][0-9.]*)", clause)
+ if match and match.group("op") in operators:
+ return _release(match.group("ver"))
+ return None
+
+
+def upper_bound(spec: str) -> tuple[int, ...] | None:
+ """The ceiling a specifier declares, or None when it has none."""
+ return _bound(spec, ("<", "<="))
+
+
+def lower_bound(spec: str) -> tuple[int, ...] | None:
+ """The floor a specifier declares, or None when it has none."""
+ return _bound(spec, (">=", ">"))
+
+
+def matrix_problems(text: str, agreed: str) -> list[str]:
+ """Everything wrong with the version matrix in ``lake-isolation.yml``.
+
+ Pure, taking the workflow text rather than reading it, so ``--self-test``
+ can inject each violation and require this function to report it. A rule
+ that only ever runs against a tree already satisfying it is a rule nobody
+ has seen fail.
+ """
+ legs = {match.group("label"): match.group("spec") for match in _MATRIX_LEG.finditer(text)}
+ problems: list[str] = []
+
+ shipped = legs.get(SHIPPED_LABEL)
+ if shipped is None:
+ problems.append(f"{MATRIX_WORKFLOW} has no matrix leg labelled `{SHIPPED_LABEL}`")
+ elif normalise(shipped) != agreed:
+ problems.append(
+ f"{MATRIX_WORKFLOW} tests the shipped rewriter against {shipped}, but the declared pin is {agreed}"
+ )
+
+ forward = legs.get(FORWARD_LABEL)
+ if forward is None:
+ problems.append(
+ f"{MATRIX_WORKFLOW} has no matrix leg labelled `{FORWARD_LABEL}` — the rewriter would only ever "
+ f"be exercised on the version already shipping, which is the state sqlglot 27 reached production in"
+ )
+ return problems
+
+ ceiling = upper_bound(forward)
+ if ceiling is not None:
+ problems.append(
+ f"{MATRIX_WORKFLOW}'s `{FORWARD_LABEL}` leg is bounded above ({forward}), so it can never resolve a "
+ f"version past the pin. That is the shape of the original break: every install path pinned below the "
+ f"major that turned isolation off, so nothing ran on it. Leave the ceiling off"
+ )
+ if shipped is not None:
+ floor, shipped_floor = lower_bound(forward), lower_bound(shipped)
+ if floor is not None and shipped_floor is not None and floor < shipped_floor:
+ problems.append(
+ f"{MATRIX_WORKFLOW}'s `{FORWARD_LABEL}` leg floors at {forward}, below the shipped floor "
+ f"{shipped}. Retargeting it downwards turns forward coverage into regression coverage for "
+ f"versions nothing installs, and leaves nothing between the next major and production"
+ )
+ shipped_ceiling = upper_bound(shipped)
+ declared = _WORKFLOW_CEILING.search(text)
+ if shipped_ceiling is None:
+ problems.append(f"{MATRIX_WORKFLOW}'s `{SHIPPED_LABEL}` leg {shipped} has no upper bound to compare against")
+ elif declared is None:
+ problems.append(
+ f"{MATRIX_WORKFLOW} declares no `SHIPPED_CEILING`, so the `Forward coverage` step cannot say "
+ f"whether the leg reached past the pin — and a leg that cannot report that is a leg that "
+ f"reports a duplicate green as evidence"
+ )
+ elif int(declared.group("major")) != shipped_ceiling[0]:
+ problems.append(
+ f"{MATRIX_WORKFLOW}'s `Forward coverage` step compares against SHIPPED_CEILING="
+ f"{declared.group('major')} while the shipped leg is {shipped} — the step would report "
+ f"forward coverage for a version inside the shipped range, or deny it for one above"
+ )
+ return problems
+
+
# Where we deliberately do not look: prose, and the archived prototype subtree.
SKIP_PREFIXES = ("plans/", "apps/docs/", "docs/", "scripts/check_sqlglot_pin.py")
@@ -227,13 +334,10 @@ def main() -> int:
if not matrix_path.exists():
problems.append(f"{MATRIX_WORKFLOW} is missing — the rewriter would no longer be tested across sqlglot majors")
elif agreed is not None:
- leg = _SHIPPED_MATRIX_LEG.search(matrix_path.read_text(encoding="utf-8"))
- if leg is None:
- problems.append(f"{MATRIX_WORKFLOW} has no matrix leg labelled `shipped`")
- elif normalise(leg.group("spec")) != agreed:
- problems.append(f"{MATRIX_WORKFLOW} tests the shipped rewriter against {leg.group('spec')}, but the declared pin is {agreed}")
+ problems += matrix_problems(matrix_path.read_text(encoding="utf-8"), agreed)
if problems:
+ print(f"check_sqlglot_pin: root {REPO_ROOT}")
print("check_sqlglot_pin: FAIL")
for problem in problems:
print(f" - {problem}")
@@ -241,12 +345,68 @@ def main() -> int:
return 1
only = next(iter(seen))
+ print(f"check_sqlglot_pin: root {REPO_ROOT}")
print(
f"check_sqlglot_pin: OK — {len(DECLARING_FILES)} install paths all declare {only}, "
- f"and {RESOLVED_FILE} resolves {locked_version} inside it"
+ f"{RESOLVED_FILE} resolves {locked_version} inside it, and {MATRIX_WORKFLOW} tests "
+ f"that range plus everything above it"
)
return 0
+# ── Self-test ────────────────────────────────────────────────────────────────
+#
+# The empty-tree refusal comes from the shared body. What is added here is the
+# matrix rule, because it is new and because a rule whose only exercise is a
+# tree that already satisfies it has never been observed to fail.
+_MATRIX_FIXTURE = """
+ include:
+ - sqlglot: '>=30,<31'
+ label: shipped
+ - sqlglot: '>=30'
+ label: forward-compat
+ steps:
+ - run: |
+ SHIPPED_CEILING = 31
+"""
+
+
+def _matrix_self_test() -> list[tuple[str, bool]]:
+ agreed = normalise(">=30,<31")
+
+ def detects(text: str, fragment: str) -> bool:
+ return any(fragment in problem for problem in matrix_problems(text, agreed))
+
+ return [
+ ("the unperturbed matrix reports nothing", not matrix_problems(_MATRIX_FIXTURE, agreed)),
+ (
+ "a shipped leg that is not the agreed pin is reported",
+ detects(_MATRIX_FIXTURE.replace("'>=30,<31'", "'>=23,<27'"), "but the declared pin is"),
+ ),
+ (
+ "a forward leg bounded above is reported",
+ detects(_MATRIX_FIXTURE.replace("- sqlglot: '>=30'", "- sqlglot: '>=30,<31'"), "is bounded above"),
+ ),
+ (
+ "a forward leg retargeted below the shipped floor is reported",
+ detects(_MATRIX_FIXTURE.replace("- sqlglot: '>=30'", "- sqlglot: '>=27'"), "below the shipped floor"),
+ ),
+ (
+ "a missing forward leg is reported",
+ detects(_MATRIX_FIXTURE.replace("label: forward-compat", "label: spare"), "has no matrix leg labelled"),
+ ),
+ (
+ "a SHIPPED_CEILING that has drifted from the shipped leg is reported",
+ detects(_MATRIX_FIXTURE.replace("SHIPPED_CEILING = 31", "SHIPPED_CEILING = 27"), "SHIPPED_CEILING="),
+ ),
+ (
+ "a deleted SHIPPED_CEILING is reported",
+ detects(_MATRIX_FIXTURE.replace("SHIPPED_CEILING = 31", "pass"), "declares no `SHIPPED_CEILING`"),
+ ),
+ ]
+
+
if __name__ == "__main__":
+ if SELF_TEST_FLAG in sys.argv[1:]:
+ sys.exit(self_test_main(Path(__file__).name, extra=_matrix_self_test()))
sys.exit(main())
diff --git a/services/api/poetry.lock b/services/api/poetry.lock
index 5f6986675..fe4588ebe 100644
--- a/services/api/poetry.lock
+++ b/services/api/poetry.lock
@@ -3223,19 +3223,20 @@ sqlcipher = ["sqlcipher3_binary"]
[[package]]
name = "sqlglot"
-version = "26.33.0"
+version = "30.19.0"
description = "An easily customizable SQL parser and transpiler"
optional = false
-python-versions = ">=3.8"
+python-versions = ">=3.9"
groups = ["main"]
files = [
- {file = "sqlglot-26.33.0-py3-none-any.whl", hash = "sha256:031cee20c0c796a83d26d079a47fdce667604df430598c7eabfa4e4dfd147033"},
- {file = "sqlglot-26.33.0.tar.gz", hash = "sha256:2817278779fa51d6def43aa0d70690b93a25c83eb18ec97130fdaf707abc0d73"},
+ {file = "sqlglot-30.19.0-py3-none-any.whl", hash = "sha256:b4500fcced395359feea582f09d2d4433bf1f36362a3a9d0d6d5ebbc1f84242e"},
+ {file = "sqlglot-30.19.0.tar.gz", hash = "sha256:6fcfbe93c5ee76e07faadf0930dd119eebb545c4f317f8e4f205f177305d8b08"},
]
[package.extras]
-dev = ["duckdb (>=0.6)", "maturin (>=1.4,<2.0)", "mypy", "pandas", "pandas-stubs", "pdoc", "pre-commit", "pyperf", "python-dateutil", "pytz", "ruff (==0.7.2)", "types-python-dateutil", "types-pytz", "typing_extensions"]
-rs = ["sqlglotrs (==0.6.1)"]
+c = ["sqlglotc (==30.19.0) ; python_version >= \"3.10\""]
+dev = ["duckdb (>=0.6)", "mypy ; python_version < \"3.10\"", "pandas", "pandas-stubs", "pdoc", "pre-commit", "pyperf", "python-dateutil", "pytz", "ruff (==0.15.6)", "setuptools_scm", "sqlglot-mypy (>=2.3.0.post2) ; python_version >= \"3.10\"", "types-python-dateutil", "types-pytz", "typing_extensions"]
+rs = ["sqlglotc (==30.19.0) ; python_version >= \"3.10\"", "sqlglotrs (==0.13.0)"]
[[package]]
name = "starlette"
@@ -4004,4 +4005,4 @@ test = ["pytest"]
[metadata]
lock-version = "2.1"
python-versions = "^3.11"
-content-hash = "19f13aee663fa5073c7e53f528a739c5a9a7239a00c371a454b43c68b9694b7c"
+content-hash = "157d0d696f4d6f866d2258ce317919c1234ea227405d661e389eeb5ec70ecdc3"
diff --git a/services/api/pyproject.toml b/services/api/pyproject.toml
index 2e1594bd8..6380cba13 100644
--- a/services/api/pyproject.toml
+++ b/services/api/pyproject.toml
@@ -65,7 +65,13 @@ clickhouse-driver = "^0.2.8"
# two different isolation guarantees. This line read `<31.0.0` while the
# Dockerfile and CI read `<27`, and on 27+ the rewriter returned unscoped
# SQL and reported success.
-sqlglot = ">=23,<27"
+#
+# One major wide, deliberately. The previous `>=23,<27` permitted four majors
+# of which exactly one was ever exercised, so three of them were installable
+# and untested. The rewriter suites and an adversarial corpus were run on the
+# 30 line; 27, 28 and 29 were not, and a range is a statement about what may
+# be installed rather than about what upstream has released.
+sqlglot = ">=30,<31"
boto3 = "^1.34.0"
neo4j = ">=5.19,<7.0"
pysigma = "^0.11.0"
diff --git a/tests/isolation/README.md b/tests/isolation/README.md
index f44a92049..3db64b9cd 100644
--- a/tests/isolation/README.md
+++ b/tests/isolation/README.md
@@ -151,7 +151,12 @@ rewriter returned every single-table query unscoped *and* unfiltered by the
allowlist while reporting success. Two gates in
`.github/workflows/lake-isolation.yml` close that off — `check_sqlglot_pin.py`
requires all seven install paths to declare one identical range, and the
-rewriter suites run against the shipped range and the next major. The rewriter
+rewriter suites run against the shipped range and against whatever sqlglot
+publishes above it. The second leg carries no upper bound on purpose, so it
+begins testing the next major the day one exists rather than when somebody
+remembers to widen it; while no such release exists it re-runs the shipped
+version and says so in a warning annotation rather than passing a duplicate
+green off as forward coverage. The rewriter
also audits its own output now and raises `LakeSqlIsolationError` rather than
returning SQL it cannot prove is scoped; `services/api/tests/test_lake_sql_fail_closed.py`
blinds the table walk deliberately to assert that refusal on any sqlglot
diff --git a/tests/isolation/stores.py b/tests/isolation/stores.py
index 1b68e20c3..c82d91c69 100644
--- a/tests/isolation/stores.py
+++ b/tests/isolation/stores.py
@@ -45,7 +45,8 @@ STORES: tuple[StoreCoverage, ...] = (
"container_gated",
"lake_sql.rewrite_for_tenant injects tenant predicate and refuses (LakeSqlIsolationError) when it cannot prove "
"it did; live-replay test_live_stores.py::test_clickhouse_lake_query_as_A_excludes_B (isolation-live.yml); "
- "rewriter run against two sqlglot majors + pin agreement gate (lake-isolation.yml)",
+ "rewriter run against the shipped sqlglot range and against the newest release above it "
+ "+ pin agreement gate (lake-isolation.yml)",
),
StoreCoverage(
"redis",