Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
feat(costs): show a tenant's projected storage $/mo beside its LLM $/mo (6b) (#1019)
Beenu Arora1eac31a3d034
diff --git a/apps/docs/docs/operations/managed-instance.md b/apps/docs/docs/operations/managed-instance.mdindex 59094c161..ce961242f 100644--- a/apps/docs/docs/operations/managed-instance.md+++ b/apps/docs/docs/operations/managed-instance.md@@ -153,13 +153,46 @@ honest unit economic looks like for a SOC product, and we'd rather dothat with a small cohort of design partners than with a public pricesheet that we have to walk back later.-What we **do** care about is making LLM cost visible from day one. The+What we **do** care about is making cost visible from day one. The[LLM cost dashboard](https://tryaisoc.com/costs) (rolled out as WS-H1 ofthe v1.0 buyer-value plan) tracks every model call with token counts,provider, and run cost, scoped per tenant. When the beta exits andpricing lands, that dashboard is what we'll bill against — socustomers can see the bill being built in real-time today.+### Storage, and why its number is shaped differently++LLM spend and storage spend are both on that page and they are **not the+same kind of number**, so the console keeps them apart and never adds them+together.++LLM cost is measured: the gateway reports what a call cost, and where it+cannot, the figure is re-priced from a published list price and labelled an+estimate.++Storage cost is **projected**. There is no per-tenant storage invoice to+read, so the console measures the one thing it can — the uncompressed bytes+your events occupied in the event lake over the window — and runs the+committed storage cost model over it+([ADR-0005](https://github.com/beenuar/AiSOC/blob/main/docs/decisions/0005-storage-consolidation.md),+worked example in `docs/decisions/storage-cost-model.json`,+drift-gated by `.github/workflows/perf.yml`). At the model's reference+scenario of 1 TB/day that comes out at about **$902/month**, or **$30 per+raw TB ingested**, across a hot/warm/cold ladder at reference list prices.++Three things follow, and the panel states each of them:++- It is a model, not your provider's bill. Verify against your own region+ and negotiated rates before planning against it.+- The model's retention shape (30 days hot / 60 warm / 275 cold) is the+ sizing scenario, not necessarily what your deployment runs — the shipped+ ClickHouse tiering SQL is two tiers over 90 days, and per-tenant retention+ policies override both.+- The event lake runs in the `full` profile. On a CORE deployment there is+ no ingest volume to measure, so the panel reads **"Not measured"** with the+ reason. It does not read `$0.00`, because a confident zero for a tenant+ nobody measured is indistinguishable from free storage.+## Operator runbook (for the AiSOC community)If you're operating the managed instance yourself, the relevantdiff --git a/apps/web/src/components/admin/CostDashboardView.test.tsx b/apps/web/src/components/admin/CostDashboardView.test.tsxindex 73565b452..0bc9f73ea 100644--- a/apps/web/src/components/admin/CostDashboardView.test.tsx+++ b/apps/web/src/components/admin/CostDashboardView.test.tsx@@ -117,6 +117,22 @@ function fixture(overrides: Partial<CostDashboard> = {}): CostDashboard {unpriced_tokens: 0,savings_usd: 0,},+ storage: {+ measured: true,+ unmeasured_reason: null,+ raw_bytes_measured: 3_000_000_000,+ events_measured: 412_000,+ raw_tb_per_day: 0.0001,+ monthly_usd: 0.09,+ usd_per_raw_tb_ingested: 30.07,+ tiers: [+ { tier: 'hot_block', retention_days: 30, resident_gb: 0.38, rate_usd_per_gb_month: 0.08, monthly_usd: 0.03 },+ { tier: 'warm_object', retention_days: 60, resident_gb: 0.75, rate_usd_per_gb_month: 0.023, monthly_usd: 0.02 },+ { tier: 'cold_archive', retention_days: 275, resident_gb: 3.44, rate_usd_per_gb_month: 0.0125, monthly_usd: 0.04 },+ ],+ compression_ratio: 8.0,+ disclaimer: 'Projected from your measured ingest volume using the committed storage cost model.',+ },...overrides,};}@@ -343,6 +359,40 @@ describe('CostDashboardView — a window nothing measured', () => {// The single most important assertion in this file: a page rendering// 1,287 real LLM calls must not print a currency amount it did not// measure. Not "$0.00", not "<$0.01", not a default-rate estimate.+ //+ // The storage panel is removed from the text first, and only that panel.+ // It is a projection over its own, separate measurement — the bytes the+ // tenant's events occupied in the lake — so it can legitimately carry a+ // figure in a window where no LLM call was priced. Excluding it by+ // node rather than by loosening the pattern keeps this assertion exactly+ // as strict as it was for every LLM surface, and the storage panel's own+ // honesty (labelled a model; "Not measured" rather than $0.00 when the+ // lake is absent) is asserted in its own describe block below.+ screen.getByTestId('storage-panel').remove();++ expect(document.body.textContent).not.toMatch(/\$\d/);+ });++ it('still says not measured for storage when the lake is absent', () => {+ const payload = unmeasured();+ payload.storage = {+ measured: false,+ unmeasured_reason: 'no lake on this deployment',+ raw_bytes_measured: null,+ events_measured: null,+ raw_tb_per_day: null,+ monthly_usd: null,+ usd_per_raw_tb_ingested: null,+ tiers: [],+ compression_ratio: 8.0,+ disclaimer: 'Projected from your measured ingest volume.',+ };+ seed(30, payload);+ render(<CostDashboardView />);++ // With both halves unmeasured, the original whole-body guarantee holds+ // with nothing excluded — which is what makes the exclusion above an+ // exemption for a measured figure, not a hole.expect(document.body.textContent).not.toMatch(/\$\d/);});@@ -425,3 +475,52 @@ describe('CostDashboardView — estimates are labelled', () => {).toBeInTheDocument();});});++/**+ * ADR-0005 / 6b. The storage figure is a model run over one measurement, and+ * it sits beside a measured LLM spend. Two things must hold on every render:+ * it is never presented as billed, and an absent lake reads as "not measured"+ * rather than as $0.00 — a confident zero for a tenant nobody measured is how+ * a deployment comes to look like it stores data for free.+ */+describe('CostDashboardView — storage is a projection, not a bill', () => {+ it('labels the projection and never calls it billed', () => {+ seed(30, fixture());+ render(<CostDashboardView />);++ const panel = screen.getByTestId('storage-panel');+ expect(panel).toHaveAttribute('data-storage', 'projected');+ expect(within(panel).getByText('Modelled, not billed')).toBeInTheDocument();+ expect(within(panel).getByText(/projected at full retention/)).toBeInTheDocument();+ expect(within(panel).getByTestId('storage-tiers')).toBeInTheDocument();+ });++ it('says not measured, with the reason, when the lake is absent', () => {+ const base = fixture();+ seed(30, {+ ...base,+ storage: {+ measured: false,+ unmeasured_reason:+ 'Ingest volume is counted in the ClickHouse event lake, which runs in the `full` profile.',+ raw_bytes_measured: null,+ events_measured: null,+ raw_tb_per_day: null,+ monthly_usd: null,+ usd_per_raw_tb_ingested: null,+ tiers: [],+ compression_ratio: 8.0,+ disclaimer: 'Projected from your measured ingest volume.',+ },+ });+ render(<CostDashboardView />);++ const panel = screen.getByTestId('storage-panel');+ expect(panel).toHaveAttribute('data-storage', 'not-measured');+ expect(within(panel).getByText('Not measured')).toBeInTheDocument();+ expect(within(panel).getByText(/full. profile/)).toBeInTheDocument();+ // The failure this guards: an unmeasured projection rendering as money.+ expect(within(panel).queryByText(/\$0\.00/)).not.toBeInTheDocument();+ expect(within(panel).queryByTestId('storage-tiers')).not.toBeInTheDocument();+ });+});diff --git a/apps/web/src/components/admin/CostDashboardView.tsx b/apps/web/src/components/admin/CostDashboardView.tsxindex ed47d2cb7..c4b9e335e 100644--- a/apps/web/src/components/admin/CostDashboardView.tsx+++ b/apps/web/src/components/admin/CostDashboardView.tsx@@ -198,6 +198,7 @@ function DashboardBody({ data }: { data: CostDashboard }) {<ModelTable data={data} /><CasesAndActions data={data} /><ByokPanel data={data} />+ <StoragePanel data={data} /></>);}@@ -529,6 +530,102 @@ function ByokPanel({ data }: { data: CostDashboard }) {);}+function fmtGb(gb: number): string {+ if (gb >= 1000) return `${(gb / 1000).toFixed(2)} TB`;+ if (gb >= 1) return `${gb.toFixed(1)} GB`;+ return `${(gb * 1000).toFixed(0)} MB`;+}++/**+ * ADR-0005 / 6b — storage $/mo beside LLM $/mo.+ *+ * Kept visually distinct from the LLM figures above and never added to them.+ * The spend above is measured or imputed from a real call; this is a model+ * run over one measurement, so the two answer different questions and a+ * single total would label one number two ways.+ *+ * When the lake is absent there is no measurement, and the panel says so with+ * the reason rather than rendering a zero — the same contract the measured /+ * estimated / unpriced trichotomy holds for spend.+ */+function StoragePanel({ data }: { data: CostDashboard }) {+ const s = data.storage;++ if (!s.measured) {+ return (+ <section+ className="rounded-xl border border-gray-700 bg-gray-900 p-5"+ data-testid="storage-panel"+ data-storage="not-measured"+ >+ <p className="text-xs uppercase tracking-wide text-gray-400">Storage cost</p>+ <p className="mt-1 text-lg font-semibold text-white">Not measured</p>+ <p className="mt-3 text-sm text-gray-400">{s.unmeasured_reason}</p>+ </section>+ );+ }++ return (+ <section+ className="rounded-xl border border-gray-700 bg-gray-900 p-5"+ data-testid="storage-panel"+ data-storage="projected"+ >+ <div className="flex flex-wrap items-start justify-between gap-3">+ <div>+ <p className="text-xs uppercase tracking-wide text-gray-400">Storage cost</p>+ <p className="mt-1 text-lg font-semibold text-white">+ ~{fmtUsd(s.monthly_usd ?? 0)}/mo projected at full retention+ </p>+ </div>+ <span className="rounded-full bg-amber-500/20 px-3 py-1 text-xs font-medium text-amber-200">+ Modelled, not billed+ </span>+ </div>+ <dl className="mt-4 grid grid-cols-1 gap-4 text-sm sm:grid-cols-3">+ <Stat+ label="Ingest measured"+ value={fmtGb((s.raw_bytes_measured ?? 0) / 1e9)}+ hint={`${fmtNumber(s.events_measured ?? 0)} events in this window`}+ />+ <Stat+ label="Daily rate"+ value={`${(s.raw_tb_per_day ?? 0).toFixed(4)} TB/day`}+ hint={`compressed ~${s.compression_ratio}x in the lake`}+ />+ <Stat+ label="Per raw TB ingested"+ value={`~${fmtUsd(s.usd_per_raw_tb_ingested ?? 0)}`}+ hint="reference list prices"+ />+ </dl>+ <table className="mt-4 w-full text-left text-sm" data-testid="storage-tiers">+ <thead>+ <tr className="border-b border-gray-800 text-xs uppercase tracking-wide text-gray-500">+ <th className="py-2 font-medium">Tier</th>+ <th className="py-2 text-right font-medium">Retention</th>+ <th className="py-2 text-right font-medium">Resident</th>+ <th className="py-2 text-right font-medium">$/GB-mo</th>+ <th className="py-2 text-right font-medium">$/mo</th>+ </tr>+ </thead>+ <tbody>+ {s.tiers.map((t) => (+ <tr key={t.tier} className="border-b border-gray-900 last:border-0">+ <td className="py-2 text-gray-300">{t.tier.replace('_', ' ')}</td>+ <td className="py-2 text-right text-gray-400">{t.retention_days}d</td>+ <td className="py-2 text-right text-gray-400">{fmtGb(t.resident_gb)}</td>+ <td className="py-2 text-right text-gray-400">{t.rate_usd_per_gb_month}</td>+ <td className="py-2 text-right text-gray-300">{fmtUsd(t.monthly_usd)}</td>+ </tr>+ ))}+ </tbody>+ </table>+ <p className="mt-4 text-xs text-gray-500">{s.disclaimer}</p>+ </section>+ );+}+// ── Local primitives (mirror ExecutiveDigest for visual consistency) ───────function Card({diff --git a/apps/web/src/lib/api.ts b/apps/web/src/lib/api.tsindex 7a5a00692..a0bfb9ba5 100644--- a/apps/web/src/lib/api.ts+++ b/apps/web/src/lib/api.ts@@ -5812,6 +5812,38 @@ export interface ByokSavings {savings_usd: number;}+export interface StorageTierProjection {+ tier: string;+ retention_days: number;+ resident_gb: number;+ rate_usd_per_gb_month: number;+ monthly_usd: number;+}++/**+ * ADR-0005 / 6b — projected storage cost beside the measured LLM spend.+ *+ * A projection, never a bill: it runs the committed storage cost model over+ * one measurement (the uncompressed bytes this tenant's events occupied in+ * the lake) at reference list prices. `measured` is the field that matters —+ * when it is false every money field is `null`, not `0`, because the lake+ * runs in the `full` profile and a confident zero for a tenant nobody+ * measured reads as free storage.+ */+export interface StorageCostProjection {+ measured: boolean;+ /** Why there is no projection. Null when there is one. */+ unmeasured_reason: string | null;+ raw_bytes_measured: number | null;+ events_measured: number | null;+ raw_tb_per_day: number | null;+ monthly_usd: number | null;+ usd_per_raw_tb_ingested: number | null;+ tiers: StorageTierProjection[];+ compression_ratio: number;+ disclaimer: string;+}+export interface CostDashboard {tenant_id: string;period: DashboardPeriod;@@ -5821,6 +5853,7 @@ export interface CostDashboard {top_cases: TopCostCase[];action_counts: ActionCount[];byok_savings: ByokSavings;+ storage: StorageCostProjection;}/**diff --git a/docs/openapi.yaml b/docs/openapi.yamlindex 16e790322..eeb13dcb1 100644--- a/docs/openapi.yaml+++ b/docs/openapi.yaml@@ -20926,12 +20926,15 @@ components:title: Action Countsbyok_savings:$ref: '#/components/schemas/ByokSavings'+ storage:+ $ref: '#/components/schemas/StorageCostProjection'type: objectrequired:- tenant_id- period- headline- byok_savings+ - storagetitle: CostDashboarddescription: Top-level deterministic snapshot.CostHeadline:@@ -31506,6 +31509,94 @@ components:- capabilitytitle: StepDispatchRequestdescription: One playbook step, expressed as the verb it names.+ StorageCostProjection:+ properties:+ measured:+ type: boolean+ title: Measured+ unmeasured_reason:+ anyOf:+ - type: string+ - type: 'null'+ title: Unmeasured Reason+ raw_bytes_measured:+ anyOf:+ - type: integer+ - type: 'null'+ title: Raw Bytes Measured+ events_measured:+ anyOf:+ - type: integer+ - type: 'null'+ title: Events Measured+ raw_tb_per_day:+ anyOf:+ - type: number+ - type: 'null'+ title: Raw Tb Per Day+ monthly_usd:+ anyOf:+ - type: number+ - type: 'null'+ title: Monthly Usd+ usd_per_raw_tb_ingested:+ anyOf:+ - type: number+ - type: 'null'+ title: Usd Per Raw Tb Ingested+ tiers:+ items:+ $ref: '#/components/schemas/StorageTierProjection'+ type: array+ title: Tiers+ compression_ratio:+ type: number+ title: Compression Ratio+ default: 8.0+ disclaimer:+ type: string+ title: Disclaimer+ default: Projected from your measured ingest volume using the committed+ storage cost model (docs/decisions/storage-cost-model.json) at reference+ list prices. This is a model, not your provider's bill — verify against+ your own region and negotiated rates.+ type: object+ required:+ - measured+ title: StorageCostProjection+ description: 'Projected storage cost for a tenant, or an honest statement that+ there is none.+++ ``measured`` is the field that matters. When it is false every money field++ is ``None`` — not ``0.0`` — and ``unmeasured_reason`` says why.'+ StorageTierProjection:+ properties:+ tier:+ type: string+ title: Tier+ retention_days:+ type: integer+ title: Retention Days+ resident_gb:+ type: number+ title: Resident Gb+ rate_usd_per_gb_month:+ type: number+ title: Rate Usd Per Gb Month+ monthly_usd:+ type: number+ title: Monthly Usd+ type: object+ required:+ - tier+ - retention_days+ - resident_gb+ - rate_usd_per_gb_month+ - monthly_usd+ title: StorageTierProjection+ description: One retention tier's share of the projected monthly storage cost.SubmissionResponse:properties:id:diff --git a/scripts/storage_cost_model.py b/scripts/storage_cost_model.pyindex 5f33cb96a..e9b2f40ac 100644--- a/scripts/storage_cost_model.py+++ b/scripts/storage_cost_model.py@@ -23,6 +23,8 @@ Usage:from __future__ import annotationsimport argparse+import ast+import contextlibimport jsonimport sysfrom pathlib import Path@@ -105,10 +107,63 @@ def _print_table(result: dict) -> None:print(f" ({result['disclaimer']})")+#: The API service's copy of this model. It cannot import this module — the+#: container has neither ``scripts/`` nor a git checkout, and ``repo_root()``+#: above shells out to git at import — so the constants are mirrored there,+#: the same constraint that put a second LLM price table in+#: ``cost_dashboard._PUBLIC_PRICING``. A second copy of a number is a second+#: place for it to drift, so ``--check`` reads it back. Without this, 6b would+#: have closed "the model has no consumer" by creating a consumer that could+#: silently start quoting different prices.+CONSUMER = "services/api/app/services/storage_cost.py"++_CONSUMER_CONSTANTS = {+ "RATE_CARD_USD_PER_GB_MONTH": lambda: RATE_CARD_USD_PER_GB_MONTH,+ "RETENTION_DAYS": lambda: SCENARIO["retention_days"],+ "COMPRESSION_RATIO": lambda: SCENARIO["compression_ratio"],+}+++def check_consumer() -> list[str]:+ """Read the API's mirrored constants back and report any disagreement.++ Parsed rather than imported, for the same reason the API cannot import+ this file: the gate runs in a dep-light lint job and must not pull in+ FastAPI, pydantic and the ClickHouse driver to compare three literals.+ """+ path = ROOT / CONSUMER+ if not path.exists():+ return [f"{CONSUMER} is missing — the model's only consumer is gone, so this gate now checks a constant nothing reads"]++ tree = ast.parse(path.read_text(encoding="utf-8"))+ found: dict[str, object] = {}+ for node in tree.body:+ target: str | None = None+ value: ast.expr | None = None+ if isinstance(node, ast.Assign) and len(node.targets) == 1 and isinstance(node.targets[0], ast.Name):+ target, value = node.targets[0].id, node.value+ elif isinstance(node, ast.AnnAssign) and isinstance(node.target, ast.Name):+ target, value = node.target.id, node.value+ if target in _CONSUMER_CONSTANTS and value is not None:+ with contextlib.suppress(ValueError):+ found[target] = ast.literal_eval(value)++ problems: list[str] = []+ for name, expected in _CONSUMER_CONSTANTS.items():+ if name not in found:+ problems.append(f"{CONSUMER} no longer declares {name}; the mirror cannot be checked, so it is not a mirror")+ elif found[name] != expected():+ problems.append(+ f"{CONSUMER}:{name} is {found[name]!r}, this model says {expected()!r} — "+ "the console would quote a price this model does not"+ )+ return problems++def main() -> int:parser = argparse.ArgumentParser(description=__doc__)parser.add_argument("--json", dest="as_json", action="store_true")- parser.add_argument("--check", action="store_true", help="fail if committed JSON is stale")+ parser.add_argument("--check", action="store_true", help="fail if committed JSON or the API's mirrored constants are stale")args = parser.parse_args()result = compute(SCENARIO, RATE_CARD_USD_PER_GB_MONTH)@@ -125,7 +180,13 @@ def main() -> int:file=sys.stderr,)return 1- print(f"OK: storage cost model current — ${result['usd_per_raw_tb_ingested']}/raw-TB")+ drift = check_consumer()+ if drift:+ print(f"ERROR: the model and {CONSUMER} disagree:", file=sys.stderr)+ for problem in drift:+ print(f" - {problem}", file=sys.stderr)+ return 1+ print(f"OK: storage cost model current — ${result['usd_per_raw_tb_ingested']}/raw-TB, and {CONSUMER} mirrors it")return 0if args.as_json:diff --git a/services/api/app/services/cost_dashboard.py b/services/api/app/services/cost_dashboard.pyindex 692ad68ae..fb824a0f9 100644--- a/services/api/app/services/cost_dashboard.py+++ b/services/api/app/services/cost_dashboard.py@@ -69,6 +69,13 @@ from pydantic import BaseModel, Fieldfrom sqlalchemy import textfrom sqlalchemy.ext.asyncio import AsyncSession+from app.services.storage_cost import (+ NO_LAKE_REASON,+ StorageCostProjection,+ measure_and_project,+ not_measured,+)+# ---------------------------------------------------------------------------# Public list pricing for BYOK imputation.#@@ -280,6 +287,11 @@ class CostDashboard(BaseModel):top_cases: list[TopCostCase] = Field(default_factory=list)action_counts: list[ActionCount] = Field(default_factory=list)byok_savings: ByokSavings+ #: Projected storage cost beside the measured LLM spend (ADR-0005 / 6b).+ #: Deliberately a sibling of the LLM figures and never summed with them:+ #: one number cannot be labelled two ways, and adding a model to a+ #: measurement is how the model stops looking like one.+ storage: StorageCostProjection# ---------------------------------------------------------------------------@@ -336,6 +348,10 @@ class DashboardInputs:cost_rows: list[CostRow] = field(default_factory=list)audit_rows: list[AuditRow] = field(default_factory=list)llm: LlmContext = field(default_factory=lambda: LlmContext(provider="none", is_local=False))+ #: ADR-0005 / 6b. Storage is a projection over a measured ingest volume,+ #: never a measured spend, so it arrives already labelled and the default+ #: is "not measured" rather than an empty projection reading zero.+ storage: StorageCostProjection = field(default_factory=lambda: not_measured(NO_LAKE_REASON))# ---------------------------------------------------------------------------@@ -611,6 +627,7 @@ def build_dashboard_from_rows(inputs: DashboardInputs) -> CostDashboard:top_cases=_top_cases(inputs.cost_rows),action_counts=_action_counts(inputs.audit_rows),byok_savings=_byok_savings(inputs.cost_rows, inputs.llm),+ storage=inputs.storage,)@@ -738,6 +755,7 @@ async def build_cost_dashboard(cost_rows = await _fetch_cost_rows(db, tenant_id, start, end)audit_rows = await _fetch_audit_rows(db, tenant_id, start, end)+ storage = await measure_and_project(tenant_id, start=start, end=end, window_days=window_days)inputs = DashboardInputs(tenant_id=tenant_id,@@ -749,6 +767,7 @@ async def build_cost_dashboard(provider=llm_provider or "unknown",is_local=bool(is_local),),+ storage=storage,)return build_dashboard_from_rows(inputs)@@ -765,6 +784,7 @@ __all__ = ["DashboardPeriod","LlmContext","ModelBreakdown",+ "StorageCostProjection","TopCostCase","build_cost_dashboard","build_dashboard_from_rows",diff --git a/services/api/app/services/storage_cost.py b/services/api/app/services/storage_cost.pynew file mode 100644index 000000000..d00986650--- /dev/null+++ b/services/api/app/services/storage_cost.py@@ -0,0 +1,253 @@+"""A tenant's storage $/mo, beside its LLM $/mo (ADR-0005 follow-up, 6b).++Phase 6 computed a storage cost model and gated it. Both halves were real:+``scripts/storage_cost_model.py`` computes ≈$902/mo and ≈$30 per raw TB at+1 TB/day, ``docs/decisions/storage-cost-model.json`` is the committed worked+example, and ``.github/workflows/perf.yml`` fails when the two diverge.++Nothing read it. ADR-0005 asked for the tenant's storage $/mo to show next to+its LLM $/mo, and outside the ADR, the CHANGELOG and the perf workflow the+model's only readers were two ClickHouse tiering files citing it in a comment.+A cost model whose only consumer is the gate that checks the cost model is a+well-tested constant.++What this is, and what it is not+--------------------------------+This is a **projection**, not a bill. It takes one measurement — the+uncompressed bytes a tenant's events occupied in the lake over the window —+and runs the committed model's arithmetic over it at **reference list+prices**. It is not that tenant's spend, it is not their provider's invoice,+and it is not this deployment's actual retention shape (the shipped tiering+SQL is two tiers over 90 days; the model's scenario is three over 365). Every+one of those distinctions is carried on the wire so the console can say it,+because a modelled number rendered beside a measured one without a label is+how a guess stops looking like one — the defect the rest of+``cost_dashboard`` exists to prevent.++Absence is not zero+-------------------+The lake runs in the ``full`` profile. On a CORE deployment there is no+ClickHouse, so there is no measurement, and the projection is reported as+**not measured with the reason** rather than as ``$0.00``. This is the same+contract ``usage_metering.UNMEASURED`` already holds for ``events_ingested``,+and for the same reason: a confident zero for a tenant nobody measured reads+as free storage.++Why the rate card is duplicated here+------------------------------------+``scripts/storage_cost_model.py`` calls ``repo_root()`` at module scope, which+shells out to git. The API container has neither ``scripts/`` nor a checkout,+so it cannot be imported — the same constraint that put a second copy of the+LLM price table in ``cost_dashboard._PUBLIC_PRICING``. A second copy of a+number is a second place for it to drift, so ``storage_cost_model.py --check``+now reads these constants back and fails when they disagree. The duplication+is deliberate; the divergence is gated.+"""++from __future__ import annotations++import logging+import uuid+from datetime import datetime++from pydantic import BaseModel, Field++logger = logging.getLogger("aisoc.storage_cost")++# ---------------------------------------------------------------------------+# The model. Mirrored from scripts/storage_cost_model.py and gated against it+# by `python3 scripts/storage_cost_model.py --check`.+# ---------------------------------------------------------------------------++#: USD per GB-month, by tier. Reference list prices.+RATE_CARD_USD_PER_GB_MONTH: dict[str, float] = {+ "hot_block": 0.080, # gp3-class block storage backing the ClickHouse hot tier+ "warm_object": 0.023, # standard object storage+ "cold_archive": 0.0125, # infrequent-access / archive object storage+}++#: Days resident in each tier. Sums to 365.+RETENTION_DAYS: dict[str, int] = {"hot_block": 30, "warm_object": 60, "cold_archive": 275}++#: What the lake's ZSTD(3) columns achieve on OCSF-shaped JSON.+COMPRESSION_RATIO: float = 8.0++_GB_PER_TB = 1000+_BYTES_PER_GB = 1_000_000_000++#: Carried on every projection so a reader cannot take it for a bill.+DISCLAIMER = (+ "Projected from your measured ingest volume using the committed storage cost model "+ "(docs/decisions/storage-cost-model.json) at reference list prices. This is a model, "+ "not your provider's bill — verify against your own region and negotiated rates."+)++#: Said out loud when the lake is absent, rather than reporting a zero.+NO_LAKE_REASON = (+ "Ingest volume is counted in the ClickHouse event lake, which runs in the `full` profile. "+ "Not measured on a deployment without it, so no storage projection is possible."+)+++class StorageTierProjection(BaseModel):+ """One retention tier's share of the projected monthly storage cost."""++ tier: str+ retention_days: int+ resident_gb: float+ rate_usd_per_gb_month: float+ monthly_usd: float+++class StorageCostProjection(BaseModel):+ """Projected storage cost for a tenant, or an honest statement that there is none.++ ``measured`` is the field that matters. When it is false every money field+ is ``None`` — not ``0.0`` — and ``unmeasured_reason`` says why.+ """++ measured: bool+ #: Why there is no projection. ``None`` when there is one.+ unmeasured_reason: str | None = None++ #: The one measurement this rests on: uncompressed bytes the tenant's+ #: events occupied over the window, and how many events that was.+ raw_bytes_measured: int | None = None+ events_measured: int | None = None++ #: The measurement, expressed as the model's input.+ raw_tb_per_day: float | None = None++ #: The projection.+ monthly_usd: float | None = None+ usd_per_raw_tb_ingested: float | None = None+ tiers: list[StorageTierProjection] = Field(default_factory=list)++ #: Model parameters, echoed so the console can show what produced the number.+ compression_ratio: float = COMPRESSION_RATIO+ disclaimer: str = DISCLAIMER+++def project(raw_bytes: int, *, window_days: int, events: int | None = None) -> StorageCostProjection:+ """Run the committed model over a measured ingest volume.++ Pure. ``raw_bytes`` is uncompressed bytes observed over ``window_days``;+ the arithmetic below is the same as ``storage_cost_model.compute`` with+ the scenario's fixed 1 TB/day replaced by the tenant's own rate.+ """+ days = max(window_days, 1)+ raw_tb_per_day = (raw_bytes / _BYTES_PER_GB) / _GB_PER_TB / days++ stored_gb_per_day = (raw_tb_per_day * _GB_PER_TB) / COMPRESSION_RATIO+ tiers: list[StorageTierProjection] = []+ total_monthly = 0.0+ for tier, retention in RETENTION_DAYS.items():+ resident_gb = stored_gb_per_day * retention+ rate = RATE_CARD_USD_PER_GB_MONTH[tier]+ monthly = resident_gb * rate+ total_monthly += monthly+ tiers.append(+ StorageTierProjection(+ tier=tier,+ retention_days=retention,+ resident_gb=round(resident_gb, 2),+ rate_usd_per_gb_month=rate,+ monthly_usd=round(monthly, 2),+ )+ )++ raw_tb_per_month = raw_tb_per_day * 30+ # A tenant that ingested nothing has a real, measured zero — it is not the+ # same as a tenant nobody measured, and `measured` is what tells them apart.+ per_tb = round(total_monthly / raw_tb_per_month, 2) if raw_tb_per_month else 0.0++ return StorageCostProjection(+ measured=True,+ raw_bytes_measured=raw_bytes,+ events_measured=events,+ raw_tb_per_day=round(raw_tb_per_day, 6),+ monthly_usd=round(total_monthly, 2),+ usd_per_raw_tb_ingested=per_tb,+ tiers=tiers,+ )+++def not_measured(reason: str) -> StorageCostProjection:+ """A projection that could not be made, carrying why."""+ return StorageCostProjection(measured=False, unmeasured_reason=reason)+++#: The tenant predicate is bound as a query parameter rather than formatted+#: in. ``rewrite_for_tenant`` exists for untrusted operator SQL; this SQL is+#: the API's own, and the isolation lesson from ``_events_of_interest`` was+#: that handing API-authored SQL to the rewriter and trusting the predicate+#: survived is how every tenant came to see the same global figure.+_VOLUME_SQL = """+SELECT count() AS events,+ sum(length(raw_payload) + length(ocsf_json)) AS raw_bytes+FROM aisoc.raw_events+WHERE tenant_id = %(tenant_id)s+ AND ingest_time >= %(start_at)s+ AND ingest_time < %(end_at)s+"""+++async def measure_and_project(+ tenant_id: uuid.UUID,+ *,+ start: datetime,+ end: datetime,+ window_days: int,+) -> StorageCostProjection:+ """Measure the tenant's ingest volume and project its storage cost.++ Never raises: an unreachable or unconfigured lake is reported as *not+ measured* with the reason, because a storage panel that 500s the whole+ cost dashboard would trade a missing number for a missing page.+ """+ # Imported here rather than at module scope, deliberately. Everything+ # above this function is the model itself — constants and arithmetic over+ # them — and importing the lake pulls in `app.core.config`, the settings+ # stack and the ClickHouse driver. `cost_dashboard`'s own test suite is+ # pure by design (rows in, dashboard out, no database), and a module-level+ # import here would have made the pure builder untestable without a+ # driver, which is the coupling the split in `cost_dashboard` exists to+ # avoid. The same reasoning as `retro_hunt_consumer._build_consumer`.+ from app.db.clickhouse import ( # noqa: PLC0415+ LakeQueryError,+ LakeQueryNotConfiguredError,+ execute_lake_query,+ )++ try:+ result = await execute_lake_query(+ _VOLUME_SQL,+ timeout_seconds=10.0,+ params={"tenant_id": str(tenant_id), "start_at": start, "end_at": end},+ )+ except LakeQueryNotConfiguredError:+ return not_measured(NO_LAKE_REASON)+ except LakeQueryError as exc:+ logger.warning("storage_cost lake query failed: %s", str(exc).replace("\r", "").replace("\n", " ")[:200])+ return not_measured(f"The event lake did not answer, so ingest volume was not measured this run ({type(exc).__name__}).")++ if not result.rows or result.rows[0][1] is None:+ return not_measured("The event lake returned no rows for this tenant and window, so there is nothing to project from.")++ events = int(result.rows[0][0] or 0)+ raw_bytes = int(result.rows[0][1] or 0)+ return project(raw_bytes, window_days=window_days, events=events)+++__all__ = [+ "COMPRESSION_RATIO",+ "DISCLAIMER",+ "NO_LAKE_REASON",+ "RATE_CARD_USD_PER_GB_MONTH",+ "RETENTION_DAYS",+ "StorageCostProjection",+ "StorageTierProjection",+ "measure_and_project",+ "not_measured",+ "project",+]diff --git a/services/api/tests/test_storage_cost.py b/services/api/tests/test_storage_cost.pynew file mode 100644index 000000000..8ce20c059--- /dev/null+++ b/services/api/tests/test_storage_cost.py@@ -0,0 +1,159 @@+"""The storage projection beside the LLM spend (ADR-0005 / 6b).++Two properties carry the honesty rule here and both are asserted below.++A projection is not a bill. The number comes from running the committed model+over one measurement at reference list prices, so it must arrive labelled and+must never be summed into the measured LLM spend.++An absence is not a zero. The lake runs in the ``full`` profile, so on a CORE+deployment there is no ingest volume to measure — and a confident ``$0.00``+for a tenant nobody measured reads as free storage. That is the same contract+``usage_metering.UNMEASURED`` already holds for ``events_ingested``.+"""++from __future__ import annotations++import json+import uuid+from pathlib import Path++import pytest+from app.services.storage_cost import (+ COMPRESSION_RATIO,+ RATE_CARD_USD_PER_GB_MONTH,+ RETENTION_DAYS,+ StorageCostProjection,+ measure_and_project,+ not_measured,+ project,+)++REPO_ROOT = Path(__file__).resolve().parents[3]+COMMITTED_MODEL = REPO_ROOT / "docs" / "decisions" / "storage-cost-model.json"++_BYTES_PER_TB = 1_000_000_000_000+++class TestItReproducesTheCommittedModel:+ """The whole point of 6b is that the console quotes *this* model."""++ def test_one_tb_per_day_reproduces_the_committed_worked_example(self) -> None:+ committed = json.loads(COMMITTED_MODEL.read_text(encoding="utf-8"))+ # 30 days at exactly the scenario's 1 TB/day.+ result = project(_BYTES_PER_TB * 30, window_days=30, events=1)++ assert result.monthly_usd == pytest.approx(committed["total_monthly_usd"], abs=0.02)+ assert result.usd_per_raw_tb_ingested == pytest.approx(committed["usd_per_raw_tb_ingested"], abs=0.02)++ def test_each_tier_reproduces_the_committed_tier(self) -> None:+ committed = json.loads(COMMITTED_MODEL.read_text(encoding="utf-8"))+ result = project(_BYTES_PER_TB * 30, window_days=30)++ by_tier = {t.tier: t for t in result.tiers}+ assert set(by_tier) == set(committed["tiers"])+ for name, expected in committed["tiers"].items():+ got = by_tier[name]+ assert got.retention_days == expected["retention_days"]+ assert got.rate_usd_per_gb_month == expected["rate_usd_per_gb_month"]+ assert got.resident_gb == pytest.approx(expected["resident_gb"], rel=0.001)+ assert got.monthly_usd == pytest.approx(expected["monthly_usd"], abs=0.02)++ def test_the_constants_match_the_committed_rate_card(self) -> None:+ """Belt to the drift gate's braces.++ ``scripts/storage_cost_model.py --check`` parses these constants out of+ the module. This asserts the same thing from the other side, so a+ change that somehow satisfied the parser still fails here.+ """+ committed = json.loads(COMMITTED_MODEL.read_text(encoding="utf-8"))+ assert RATE_CARD_USD_PER_GB_MONTH == committed["rate_card_usd_per_gb_month"]+ assert RETENTION_DAYS == committed["scenario"]["retention_days"]+ assert COMPRESSION_RATIO == committed["scenario"]["compression_ratio"]+++class TestAbsenceIsNotZero:+ def test_not_measured_carries_the_reason_and_no_money(self) -> None:+ result = not_measured("no lake on this deployment")++ assert result.measured is False+ assert result.unmeasured_reason == "no lake on this deployment"+ # Every money field must be None. A zero here is the defect.+ assert result.monthly_usd is None+ assert result.usd_per_raw_tb_ingested is None+ assert result.raw_bytes_measured is None+ assert result.tiers == []++ @pytest.mark.asyncio+ async def test_an_unconfigured_lake_reports_not_measured_rather_than_zero(self, monkeypatch: pytest.MonkeyPatch) -> None:+ from app.db import clickhouse++ async def _refuse(*_args: object, **_kwargs: object) -> None:+ raise clickhouse.LakeQueryNotConfiguredError("CLICKHOUSE_HOST is not configured")++ monkeypatch.setattr(clickhouse, "execute_lake_query", _refuse)+ result = await measure_and_project(uuid.uuid4(), start=_t(0), end=_t(30), window_days=30)++ assert result.measured is False+ assert result.monthly_usd is None+ assert "full" in (result.unmeasured_reason or "")++ @pytest.mark.asyncio+ async def test_an_unreachable_lake_degrades_rather_than_raising(self, monkeypatch: pytest.MonkeyPatch) -> None:+ """A missing storage number must not take the whole cost page with it."""+ from app.db import clickhouse++ async def _fail(*_args: object, **_kwargs: object) -> None:+ raise clickhouse.LakeQueryError("connection refused")++ monkeypatch.setattr(clickhouse, "execute_lake_query", _fail)+ result = await measure_and_project(uuid.uuid4(), start=_t(0), end=_t(30), window_days=30)++ assert result.measured is False+ assert result.monthly_usd is None++ def test_a_tenant_that_ingested_nothing_is_measured_zero_not_unmeasured(self) -> None:+ """The distinction the `measured` flag exists to carry.++ Ingesting nothing is a fact about the tenant. Having no lake is a fact+ about the deployment. Reporting both as ``$0.00`` would lose the only+ difference that matters to someone reading the number.+ """+ result = project(0, window_days=30, events=0)++ assert result.measured is True+ assert result.monthly_usd == 0.0+ assert result.raw_bytes_measured == 0+++class TestItIsLabelledAsAModel:+ def test_every_projection_carries_the_disclaimer(self) -> None:+ result = project(_BYTES_PER_TB, window_days=30)++ assert "not your provider's bill" in result.disclaimer+ assert "reference list prices" in result.disclaimer++ def test_no_field_is_named_so_it_reads_as_measured_spend(self) -> None:+ """``check_cost_provenance.py`` reconciles any field ending ``cost_usd``+ against a call count. A projection has no calls, so it deliberately+ does not use that suffix — the naming is the label."""+ fields = set(StorageCostProjection.model_fields)++ assert not any(f.endswith("cost_usd") for f in fields), fields+ assert "measured" in fields and "unmeasured_reason" in fields+++class TestItIsNotSummedIntoTheLlmSpend:+ def test_the_dashboard_exposes_storage_as_a_sibling_not_a_total(self) -> None:+ from app.services.cost_dashboard import CostDashboard, CostHeadline++ assert "storage" in CostDashboard.model_fields+ # If a future edit adds storage into the headline, this fails — which+ # is the point. One number cannot be labelled two ways.+ assert not any("storage" in f for f in CostHeadline.model_fields)+++def _t(day: int):+ from datetime import UTC, datetime, timedelta++ return datetime(2026, 5, 1, tzinfo=UTC) + timedelta(days=day)