Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
Move all eight sqlglot declarations to the 30 line, and give the forward matrix leg somewhere to point (#856)
Beenu Aroraca272d8243e1
diff --git a/.github/workflows/check-openapi.yml b/.github/workflows/check-openapi.ymlindex c07ec8382..49d26a091 100644--- a/.github/workflows/check-openapi.yml+++ b/.github/workflows/check-openapi.yml@@ -27,7 +27,7 @@ env:opentelemetry-exporter-otlp-proto-grpc"opentelemetry-instrumentation-fastapi""strawberry-graphql[fastapi]" httpx email-validator "PyJWT>=2.8,<3"- "sqlglot>=23,<27"+ "sqlglot>=30,<31"jobs:openapi-check:diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlindex aa35c6e91..651e12fc5 100644--- a/.github/workflows/ci.yml+++ b/.github/workflows/ci.yml@@ -609,7 +609,7 @@ jobs:"opentelemetry-instrumentation-fastapi""strawberry-graphql[fastapi]"neo4j redis celery httpx aiofiles email-validator "PyJWT>=2.8,<3"- "sqlglot>=23,<27" aiosqlite+ "sqlglot>=30,<31" aiosqlite"croniter>=2.0,<7.0"pytest pytest-asynciosteps:diff --git a/.github/workflows/cross-tenant-rbac.yml b/.github/workflows/cross-tenant-rbac.ymlindex 81b3c9ff9..a309f7802 100644--- a/.github/workflows/cross-tenant-rbac.yml+++ b/.github/workflows/cross-tenant-rbac.yml@@ -86,7 +86,7 @@ jobs:"opentelemetry-instrumentation-fastapi""strawberry-graphql[fastapi]"neo4j redis celery httpx aiofiles email-validator "PyJWT>=2.8,<3"- "sqlglot>=23,<27" aiosqlite+ "sqlglot>=30,<31" aiosqlitepytest pytest-asynciosteps:diff --git a/.github/workflows/integration.yml b/.github/workflows/integration.ymlindex ae849258e..37237ced3 100644--- a/.github/workflows/integration.yml+++ b/.github/workflows/integration.yml@@ -409,7 +409,7 @@ jobs:"opentelemetry-instrumentation-fastapi""strawberry-graphql[fastapi]"neo4j redis celery httpx aiofiles email-validator "PyJWT>=2.8,<3"- "sqlglot>=23,<27" aiosqlite "croniter>=2.0,<7.0"+ "sqlglot>=30,<31" aiosqlite "croniter>=2.0,<7.0"- name: Apply full migration chain on fresh Postgres (strict)working-directory: services/api@@ -870,7 +870,7 @@ jobs:"opentelemetry-instrumentation-fastapi""strawberry-graphql[fastapi]"neo4j redis celery httpx aiofiles email-validator "PyJWT>=2.8,<3"- "sqlglot>=23,<27" aiosqlite "croniter>=2.0,<7.0"+ "sqlglot>=30,<31" aiosqlite "croniter>=2.0,<7.0"- name: Record HEAD shaid: headdiff --git a/.github/workflows/isolation-live.yml b/.github/workflows/isolation-live.ymlindex d45e9b016..fb291f9fb 100644--- a/.github/workflows/isolation-live.yml+++ b/.github/workflows/isolation-live.yml@@ -44,13 +44,22 @@ jobs:- name: Install clients + test deps# sqlglot MUST be pinned to the same range the API service ships- # (`>=23,<27`): the ClickHouse test runs the production- # lake_sql.rewrite_for_tenant, and sqlglot >= 27 silently stops- # injecting the tenant WHERE predicate (the tenant-isolation break this- # very gate exists to catch). test_lake_sql.py guards the pin itself.+ # (`>=30,<31`): the ClickHouse test runs the production+ # lake_sql.rewrite_for_tenant, so a version this job resolves and the+ # image does not is a different isolation guarantee than the one under+ # test. `scripts/check_sqlglot_pin.py` is what keeps the two equal.+ #+ # The historical break is why the pin is enforced rather than+ # suggested: sqlglot 27 renamed the SELECT's FROM argument key, the+ # table walk found nothing, and every single-table query came back+ # with no tenant predicate, no allowlist check and no table-function+ # ban — reported as success. `lake_sql._from_clause` now resolves the+ # clause by node type rather than by key name, so that specific+ # rename cannot recur; the agreement requirement stands because the+ # next AST change will not announce itself either.run: >-pip install --quiet- pytest pytest-asyncio structlog "sqlglot>=23,<27"+ pytest pytest-asyncio structlog "sqlglot>=30,<31"neo4j "redis>=5,<6" clickhouse-driver "kafka-python>=2,<3"# Single-container stores run as `services:`-style `docker run` so we candiff --git a/.github/workflows/lake-isolation.yml b/.github/workflows/lake-isolation.ymlindex 273bb7a15..070e21c6c 100644--- a/.github/workflows/lake-isolation.yml+++ b/.github/workflows/lake-isolation.yml@@ -22,9 +22,39 @@ name: Lake tenant isolation# range, so two builds of one commit cannot disagree about the# version of the parser that enforces isolation.# 2. The rewriter suite runs against the pinned range *and* against the-# next major. The second leg is forward-compatibility coverage: we-# ship the pinned range, but a future bump must fail loudly here-# rather than quietly downgrade isolation in production.+# newest sqlglot pip will resolve. The second leg is forward-compatibility+# coverage: we ship the pinned range, but a future bump must fail loudly+# here rather than quietly downgrade isolation in production.+#+# Why the forward leg is unbounded above rather than naming the next major.+# It used to read `>=27,<31`, which worked only while the shipped range sat+# below 27. With the pin on the 30 line there is no sqlglot 31 to name, and+# the three ways of writing that leg are all worse than leaving the ceiling+# off:+#+# `>=31` pip cannot resolve it, so the leg is red every run for a+# reason that has nothing to do with the change under test.+# `>=30,<31` identical to the shipped leg. Two legs, one question, and+# the matrix reports two greens for one piece of evidence.+# `>=27,<30` tests majors *below* the pin. That is regression coverage+# for versions nothing installs, and it removes the only+# thing standing between a future major and production —+# which is precisely the hole 27 came through.+#+# Unbounded resolves the newest release on every run, so the leg arms itself+# the day sqlglot 31 is published, with no edit and no one remembering. Until+# then it resolves inside the shipped range, and the `Forward coverage`+# step below says so as a warning annotation rather than letting a duplicate+# green pass for evidence. `scripts/check_sqlglot_pin.py` enforces the+# structural half — the leg must carry no upper bound and must not float+# below the shipped floor — because that is the part that can rot silently.+# Whether upstream has published a higher major is not this repository's to+# assert.+#+# The cost is stated rather than hidden: this leg depends on the outside+# world, so an upstream release can turn it red without a local change. That+# is the job. The workflow is path-filtered to the rewriter, its tests and+# its pins, so the people it interrupts are the ones who need to know.on:push:@@ -72,11 +102,12 @@ jobs:matrix:include:# The range the service ships and CI installs everywhere else.- - sqlglot: '>=23,<27'+ - sqlglot: '>=30,<31'label: shipped- # The next major. Not shipped; proves a bump cannot silently- # turn the isolation guarantee off the way 27 did.- - sqlglot: '>=27,<31'+ # Whatever is newest. Not shipped; proves a bump cannot silently+ # turn the isolation guarantee off the way 27 did. No ceiling, so+ # it starts testing the next major the day one exists.+ - sqlglot: '>=30'label: forward-compatsteps:- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7@@ -84,9 +115,34 @@ jobs:with:python-version: '3.11'- name: Install- run: pip install --quiet pytest "sqlglot${{ matrix.sqlglot }}"+ run: pip install --quiet --upgrade pytest "sqlglot${{ matrix.sqlglot }}"- name: Record resolved versionrun: python -c "import sqlglot; print('resolved sqlglot', sqlglot.__version__)"+ - name: Forward coverage — is this leg above the shipped ceiling?+ if: matrix.label == 'forward-compat'+ # A leg that resolves back inside the shipped range is re-running the+ # shipped version. The suite below is still worth running, but the+ # result is not forward coverage and must not read as though it were.+ run: |+ python - <<'PY'+ import sqlglot++ # Kept equal to the `<N` bound of the shipped range by+ # `scripts/check_sqlglot_pin.py`, so this number cannot drift away+ # from the pin it describes and start reporting the wrong verdict.+ SHIPPED_CEILING = 31++ major = int(sqlglot.__version__.split(".")[0])+ if major >= SHIPPED_CEILING:+ print(f"forward coverage: sqlglot {sqlglot.__version__} is above the shipped ceiling {SHIPPED_CEILING}")+ else:+ print(+ f"::warning title=No forward coverage today::sqlglot {sqlglot.__version__} is the newest "+ f"release and it is inside the shipped range, so this leg is re-running the shipped "+ f"version rather than proving a future major is safe. It becomes real forward coverage "+ f"with no edit to this file once {SHIPPED_CEILING}.0.0 is published."+ )+ PY- name: Rewriter + fail-closed suitesworking-directory: services/apirun: python -m pytest tests/test_lake_sql.py tests/test_lake_sql_fail_closed.py -qdiff --git a/.github/workflows/reproducible-builds.yml b/.github/workflows/reproducible-builds.ymlindex 2b2a70a72..75d3b53b2 100644--- a/.github/workflows/reproducible-builds.yml+++ b/.github/workflows/reproducible-builds.yml@@ -173,7 +173,7 @@ jobs:"opentelemetry-instrumentation-fastapi""strawberry-graphql[fastapi]"neo4j redis celery httpx aiofiles email-validator- "sqlglot>=23,<27" aiosqlite "croniter>=2.0,<7.0"+ "sqlglot>=30,<31" aiosqlite "croniter>=2.0,<7.0"pytest pytest-asynciosteps:- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 0a4efcb7a..2c93b7851 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -9,6 +9,58 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0### Changed+- **`sqlglot` moved to the 30 line across all eight declarations, and the+ forward-compatibility matrix leg was rewritten so it can still reach past+ the pin.** The bump itself was never the work: a single-path proposal moved+ 1 of 8 declarations and left six workflows at `>=23,<27`, which+ `scripts/check_sqlglot_pin.py` fails closed on — correctly, since that gate+ exists because sqlglot 27 renamed the SELECT's FROM argument key and+ silently stripped the tenant predicate, the table allowlist and the `url()`+ exfiltration ban from every single-table query while reporting success. All+ eight now read `>=30,<31` (`services/api/pyproject.toml` plus `ci.yml`,+ `integration.yml` ×2, `isolation-live.yml`, `cross-tenant-rbac.yml`,+ `check-openapi.yml`, `reproducible-builds.yml`), and+ `services/api/poetry.lock` resolves 30.19.0 inside it.++ One major wide rather than four. The previous range permitted 23–26 and+ exactly one of those was ever exercised, so three majors were installable+ and untested; 27, 28 and 29 are likewise unexercised, and a range states+ what may be installed rather than what upstream has released.++ Measured before moving, on 64 adversarial cases written against the+ documented policy rather than reusing the module's fixtures — DDL, DML,+ admin verbs, multi-statement input, seven ClickHouse table functions+ including ones hidden in subqueries and CTE bodies, non-allowlisted tables+ reached through JOIN, subquery, UNION arm and CTE, and five forged+ `tenant_id` predicates. 26.33.0 and 30.19.0 return the same disposition on+ all 64 (23 rewritten, 33 `LakeSqlForbiddenError`, 8 `LakeSqlSyntaxError`)+ and the rendered SQL of all 23 rewritten cases is byte-identical. Two cases+ differ in wording only and both remain refusals: `DETACH TABLE` now reaches+ the statement-kind check instead of failing to parse, and `KILL`'s class+ moved to `sqlglot.expressions.ddl`, which the parse error quotes. The+ in-tree suites pass 58/58 on each major.++ The forward leg is the part that needed a decision. It read `>=27,<31`,+ which only reached past the pin while the pin sat below 27; there is no+ sqlglot 31, so with the pin on the 30 line every way of naming a ceiling is+ worse than leaving it off — `>=31` cannot be installed and reds the job+ forever, `>=30,<31` duplicates the shipped leg and reports two greens for+ one piece of evidence, and retargeting to `>=27,<30` tests majors below the+ pin, which is regression coverage for versions nothing installs and leaves+ nothing between the next major and production. The leg is now unbounded+ above, so it resolves whatever is newest and arms itself the day a higher+ major is published with no edit. Because that means it re-runs the shipped+ version today, a `Forward coverage` step emits a warning annotation saying+ so rather than letting a duplicate green pass for evidence, and+ `check_sqlglot_pin.py` gained the structural half in both directions: the+ forward leg must carry no upper bound and must not float below the shipped+ floor, and the step's `SHIPPED_CEILING` must equal the shipped range's `<N`+ bound so a stale copy cannot report the wrong verdict. Seven injected+ violations in `--self-test`, which runs before the gate. What is+ deliberately not asserted is that a higher major exists: upstream's release+ schedule is not this repository's to require, and a gate red until sqlglot+ ships 31 would be red for a reason no change here could fix.+- **Four root-workspace dependency bumps landed as one lockfile change.**`cytoscape` 3.34.0 → 3.34.3, `lucide-react` 1.28.0 → 1.48.0,`@modelcontextprotocol/sdk` 1.29.0 → 1.30.1 and `tsx` 4.23.1 → 4.23.15 alldiff --git a/scripts/check_sqlglot_pin.py b/scripts/check_sqlglot_pin.pyindex 572afbd6e..dab00ec6c 100644--- a/scripts/check_sqlglot_pin.py+++ b/scripts/check_sqlglot_pin.py@@ -31,9 +31,7 @@ from pathlib import Path# test loads it by path with importlib. gate_toolkit sits beside it either way.sys.path.insert(0, str(Path(__file__).resolve().parent))-from gate_toolkit import repo_root, self_test_if_requested--self_test_if_requested(__file__)+from gate_toolkit import SELF_TEST_FLAG, repo_root, self_test_mainREPO_ROOT = repo_root()# Every file that installs sqlglot for a build or a test run. A new one must@@ -50,12 +48,40 @@ DECLARING_FILES = ()# `lake-isolation.yml` installs sqlglot from a build matrix, deliberately-# including a version outside the shipped range so a future bump cannot-# silently turn isolation off. It therefore cannot declare a single pin. What-# it *must* do is test the version we ship, so the matrix leg labelled-# `shipped` is checked against the agreed pin below.+# reaching past the shipped range so a future bump cannot silently turn+# isolation off. It therefore cannot declare a single pin, and is checked on+# its own terms instead of being exempted:+#+# `shipped` must equal the agreed pin, or the matrix proves a version+# nothing installs still works.+# `forward-compat` must carry no upper bound, and must not float below the+# shipped floor.+#+# The second rule is the one with history behind it. That leg used to read+# `>=27,<31`, which only reached past the pin while the pin sat below 27.+# With the pin on the 30 line there is no sqlglot 31 to name, so the leg is+# written unbounded and resolves whatever is newest: it re-runs the shipped+# version today and arms itself the day a higher major is published, with no+# edit and nobody remembering. A ceiling would take that away silently, which+# is exactly how 27 reached production — every install path pinned below the+# version that broke isolation, so nothing ever ran on it.+#+# What is deliberately *not* asserted here is that a higher major exists.+# Upstream's release schedule is not this repository's to require, and a gate+# that failed until sqlglot shipped 31 would be red for a reason no change in+# this tree could fix. The workflow's own `Forward coverage` step reports+# which of the two states it is in on every run.MATRIX_WORKFLOW = ".github/workflows/lake-isolation.yml"-_SHIPPED_MATRIX_LEG = re.compile(r"""-\s*sqlglot:\s*["'](?P<spec>[^"']+)["']\s*\n\s*label:\s*shipped""")+_MATRIX_LEG = re.compile(r"""-\s*sqlglot:\s*["'](?P<spec>[^"']+)["']\s*\n\s*label:\s*(?P<label>[\w-]+)""")++# The ceiling the workflow's `Forward coverage` step compares against. It is a+# second copy of the shipped range's `<N` bound, so it is checked against the+# first rather than trusted: a stale copy would report "above the shipped+# ceiling" about a version that is inside it.+_WORKFLOW_CEILING = re.compile(r"""^\s*SHIPPED_CEILING\s*=\s*(?P<major>\d+)\s*$""", re.MULTILINE)++SHIPPED_LABEL = "shipped"+FORWARD_LABEL = "forward-compat"# `services/api/Dockerfile` used to be on the list above, because its pip# fallback carried its own copy of the dependency list. That fallback is gone:@@ -96,6 +122,87 @@ def satisfies(version: str, spec: str) -> bool:return Falsereturn True++def _bound(spec: str, operators: tuple[str, ...]) -> tuple[int, ...] | None:+ """The version named by the first clause using one of ``operators``."""+ for clause in spec.replace(" ", "").split(","):+ match = re.fullmatch(r"(?P<op>[<>=!]+)(?P<ver>[0-9][0-9.]*)", clause)+ if match and match.group("op") in operators:+ return _release(match.group("ver"))+ return None+++def upper_bound(spec: str) -> tuple[int, ...] | None:+ """The ceiling a specifier declares, or None when it has none."""+ return _bound(spec, ("<", "<="))+++def lower_bound(spec: str) -> tuple[int, ...] | None:+ """The floor a specifier declares, or None when it has none."""+ return _bound(spec, (">=", ">"))+++def matrix_problems(text: str, agreed: str) -> list[str]:+ """Everything wrong with the version matrix in ``lake-isolation.yml``.++ Pure, taking the workflow text rather than reading it, so ``--self-test``+ can inject each violation and require this function to report it. A rule+ that only ever runs against a tree already satisfying it is a rule nobody+ has seen fail.+ """+ legs = {match.group("label"): match.group("spec") for match in _MATRIX_LEG.finditer(text)}+ problems: list[str] = []++ shipped = legs.get(SHIPPED_LABEL)+ if shipped is None:+ problems.append(f"{MATRIX_WORKFLOW} has no matrix leg labelled `{SHIPPED_LABEL}`")+ elif normalise(shipped) != agreed:+ problems.append(+ f"{MATRIX_WORKFLOW} tests the shipped rewriter against {shipped}, but the declared pin is {agreed}"+ )++ forward = legs.get(FORWARD_LABEL)+ if forward is None:+ problems.append(+ f"{MATRIX_WORKFLOW} has no matrix leg labelled `{FORWARD_LABEL}` — the rewriter would only ever "+ f"be exercised on the version already shipping, which is the state sqlglot 27 reached production in"+ )+ return problems++ ceiling = upper_bound(forward)+ if ceiling is not None:+ problems.append(+ f"{MATRIX_WORKFLOW}'s `{FORWARD_LABEL}` leg is bounded above ({forward}), so it can never resolve a "+ f"version past the pin. That is the shape of the original break: every install path pinned below the "+ f"major that turned isolation off, so nothing ran on it. Leave the ceiling off"+ )+ if shipped is not None:+ floor, shipped_floor = lower_bound(forward), lower_bound(shipped)+ if floor is not None and shipped_floor is not None and floor < shipped_floor:+ problems.append(+ f"{MATRIX_WORKFLOW}'s `{FORWARD_LABEL}` leg floors at {forward}, below the shipped floor "+ f"{shipped}. Retargeting it downwards turns forward coverage into regression coverage for "+ f"versions nothing installs, and leaves nothing between the next major and production"+ )+ shipped_ceiling = upper_bound(shipped)+ declared = _WORKFLOW_CEILING.search(text)+ if shipped_ceiling is None:+ problems.append(f"{MATRIX_WORKFLOW}'s `{SHIPPED_LABEL}` leg {shipped} has no upper bound to compare against")+ elif declared is None:+ problems.append(+ f"{MATRIX_WORKFLOW} declares no `SHIPPED_CEILING`, so the `Forward coverage` step cannot say "+ f"whether the leg reached past the pin — and a leg that cannot report that is a leg that "+ f"reports a duplicate green as evidence"+ )+ elif int(declared.group("major")) != shipped_ceiling[0]:+ problems.append(+ f"{MATRIX_WORKFLOW}'s `Forward coverage` step compares against SHIPPED_CEILING="+ f"{declared.group('major')} while the shipped leg is {shipped} — the step would report "+ f"forward coverage for a version inside the shipped range, or deny it for one above"+ )+ return problems++# Where we deliberately do not look: prose, and the archived prototype subtree.SKIP_PREFIXES = ("plans/", "apps/docs/", "docs/", "scripts/check_sqlglot_pin.py")@@ -227,13 +334,10 @@ def main() -> int:if not matrix_path.exists():problems.append(f"{MATRIX_WORKFLOW} is missing — the rewriter would no longer be tested across sqlglot majors")elif agreed is not None:- leg = _SHIPPED_MATRIX_LEG.search(matrix_path.read_text(encoding="utf-8"))- if leg is None:- problems.append(f"{MATRIX_WORKFLOW} has no matrix leg labelled `shipped`")- elif normalise(leg.group("spec")) != agreed:- problems.append(f"{MATRIX_WORKFLOW} tests the shipped rewriter against {leg.group('spec')}, but the declared pin is {agreed}")+ problems += matrix_problems(matrix_path.read_text(encoding="utf-8"), agreed)if problems:+ print(f"check_sqlglot_pin: root {REPO_ROOT}")print("check_sqlglot_pin: FAIL")for problem in problems:print(f" - {problem}")@@ -241,12 +345,68 @@ def main() -> int:return 1only = next(iter(seen))+ print(f"check_sqlglot_pin: root {REPO_ROOT}")print(f"check_sqlglot_pin: OK — {len(DECLARING_FILES)} install paths all declare {only}, "- f"and {RESOLVED_FILE} resolves {locked_version} inside it"+ f"{RESOLVED_FILE} resolves {locked_version} inside it, and {MATRIX_WORKFLOW} tests "+ f"that range plus everything above it")return 0+# ── Self-test ────────────────────────────────────────────────────────────────+#+# The empty-tree refusal comes from the shared body. What is added here is the+# matrix rule, because it is new and because a rule whose only exercise is a+# tree that already satisfies it has never been observed to fail.+_MATRIX_FIXTURE = """+ include:+ - sqlglot: '>=30,<31'+ label: shipped+ - sqlglot: '>=30'+ label: forward-compat+ steps:+ - run: |+ SHIPPED_CEILING = 31+"""+++def _matrix_self_test() -> list[tuple[str, bool]]:+ agreed = normalise(">=30,<31")++ def detects(text: str, fragment: str) -> bool:+ return any(fragment in problem for problem in matrix_problems(text, agreed))++ return [+ ("the unperturbed matrix reports nothing", not matrix_problems(_MATRIX_FIXTURE, agreed)),+ (+ "a shipped leg that is not the agreed pin is reported",+ detects(_MATRIX_FIXTURE.replace("'>=30,<31'", "'>=23,<27'"), "but the declared pin is"),+ ),+ (+ "a forward leg bounded above is reported",+ detects(_MATRIX_FIXTURE.replace("- sqlglot: '>=30'", "- sqlglot: '>=30,<31'"), "is bounded above"),+ ),+ (+ "a forward leg retargeted below the shipped floor is reported",+ detects(_MATRIX_FIXTURE.replace("- sqlglot: '>=30'", "- sqlglot: '>=27'"), "below the shipped floor"),+ ),+ (+ "a missing forward leg is reported",+ detects(_MATRIX_FIXTURE.replace("label: forward-compat", "label: spare"), "has no matrix leg labelled"),+ ),+ (+ "a SHIPPED_CEILING that has drifted from the shipped leg is reported",+ detects(_MATRIX_FIXTURE.replace("SHIPPED_CEILING = 31", "SHIPPED_CEILING = 27"), "SHIPPED_CEILING="),+ ),+ (+ "a deleted SHIPPED_CEILING is reported",+ detects(_MATRIX_FIXTURE.replace("SHIPPED_CEILING = 31", "pass"), "declares no `SHIPPED_CEILING`"),+ ),+ ]++if __name__ == "__main__":+ if SELF_TEST_FLAG in sys.argv[1:]:+ sys.exit(self_test_main(Path(__file__).name, extra=_matrix_self_test()))sys.exit(main())diff --git a/services/api/poetry.lock b/services/api/poetry.lockindex 5f6986675..fe4588ebe 100644--- a/services/api/poetry.lock+++ b/services/api/poetry.lock@@ -3223,19 +3223,20 @@ sqlcipher = ["sqlcipher3_binary"][[package]]name = "sqlglot"-version = "26.33.0"+version = "30.19.0"description = "An easily customizable SQL parser and transpiler"optional = false-python-versions = ">=3.8"+python-versions = ">=3.9"groups = ["main"]files = [- {file = "sqlglot-26.33.0-py3-none-any.whl", hash = "sha256:031cee20c0c796a83d26d079a47fdce667604df430598c7eabfa4e4dfd147033"},- {file = "sqlglot-26.33.0.tar.gz", hash = "sha256:2817278779fa51d6def43aa0d70690b93a25c83eb18ec97130fdaf707abc0d73"},+ {file = "sqlglot-30.19.0-py3-none-any.whl", hash = "sha256:b4500fcced395359feea582f09d2d4433bf1f36362a3a9d0d6d5ebbc1f84242e"},+ {file = "sqlglot-30.19.0.tar.gz", hash = "sha256:6fcfbe93c5ee76e07faadf0930dd119eebb545c4f317f8e4f205f177305d8b08"},][package.extras]-dev = ["duckdb (>=0.6)", "maturin (>=1.4,<2.0)", "mypy", "pandas", "pandas-stubs", "pdoc", "pre-commit", "pyperf", "python-dateutil", "pytz", "ruff (==0.7.2)", "types-python-dateutil", "types-pytz", "typing_extensions"]-rs = ["sqlglotrs (==0.6.1)"]+c = ["sqlglotc (==30.19.0) ; python_version >= \"3.10\""]+dev = ["duckdb (>=0.6)", "mypy ; python_version < \"3.10\"", "pandas", "pandas-stubs", "pdoc", "pre-commit", "pyperf", "python-dateutil", "pytz", "ruff (==0.15.6)", "setuptools_scm", "sqlglot-mypy (>=2.3.0.post2) ; python_version >= \"3.10\"", "types-python-dateutil", "types-pytz", "typing_extensions"]+rs = ["sqlglotc (==30.19.0) ; python_version >= \"3.10\"", "sqlglotrs (==0.13.0)"][[package]]name = "starlette"@@ -4004,4 +4005,4 @@ test = ["pytest"][metadata]lock-version = "2.1"python-versions = "^3.11"-content-hash = "19f13aee663fa5073c7e53f528a739c5a9a7239a00c371a454b43c68b9694b7c"+content-hash = "157d0d696f4d6f866d2258ce317919c1234ea227405d661e389eeb5ec70ecdc3"diff --git a/services/api/pyproject.toml b/services/api/pyproject.tomlindex 2e1594bd8..6380cba13 100644--- a/services/api/pyproject.toml+++ b/services/api/pyproject.toml@@ -65,7 +65,13 @@ clickhouse-driver = "^0.2.8"# two different isolation guarantees. This line read `<31.0.0` while the# Dockerfile and CI read `<27`, and on 27+ the rewriter returned unscoped# SQL and reported success.-sqlglot = ">=23,<27"+#+# One major wide, deliberately. The previous `>=23,<27` permitted four majors+# of which exactly one was ever exercised, so three of them were installable+# and untested. The rewriter suites and an adversarial corpus were run on the+# 30 line; 27, 28 and 29 were not, and a range is a statement about what may+# be installed rather than about what upstream has released.+sqlglot = ">=30,<31"boto3 = "^1.34.0"neo4j = ">=5.19,<7.0"pysigma = "^0.11.0"diff --git a/tests/isolation/README.md b/tests/isolation/README.mdindex f44a92049..3db64b9cd 100644--- a/tests/isolation/README.md+++ b/tests/isolation/README.md@@ -151,7 +151,12 @@ rewriter returned every single-table query unscoped *and* unfiltered by theallowlist while reporting success. Two gates in`.github/workflows/lake-isolation.yml` close that off — `check_sqlglot_pin.py`requires all seven install paths to declare one identical range, and the-rewriter suites run against the shipped range and the next major. The rewriter+rewriter suites run against the shipped range and against whatever sqlglot+publishes above it. The second leg carries no upper bound on purpose, so it+begins testing the next major the day one exists rather than when somebody+remembers to widen it; while no such release exists it re-runs the shipped+version and says so in a warning annotation rather than passing a duplicate+green off as forward coverage. The rewriteralso audits its own output now and raises `LakeSqlIsolationError` rather thanreturning SQL it cannot prove is scoped; `services/api/tests/test_lake_sql_fail_closed.py`blinds the table walk deliberately to assert that refusal on any sqlglotdiff --git a/tests/isolation/stores.py b/tests/isolation/stores.pyindex 1b68e20c3..c82d91c69 100644--- a/tests/isolation/stores.py+++ b/tests/isolation/stores.py@@ -45,7 +45,8 @@ STORES: tuple[StoreCoverage, ...] = ("container_gated","lake_sql.rewrite_for_tenant injects tenant predicate and refuses (LakeSqlIsolationError) when it cannot prove ""it did; live-replay test_live_stores.py::test_clickhouse_lake_query_as_A_excludes_B (isolation-live.yml); "- "rewriter run against two sqlglot majors + pin agreement gate (lake-isolation.yml)",+ "rewriter run against the shipped sqlglot range and against the newest release above it "+ "+ pin agreement gate (lake-isolation.yml)",),StoreCoverage("redis",