Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

feat(costs): show a tenant's projected storage $/mo beside its LLM $/mo (6b) (#1019)

Beenu Arora1eac31a3d034
diff --git a/apps/docs/docs/operations/managed-instance.md b/apps/docs/docs/operations/managed-instance.md
index 59094c161..ce961242f 100644
--- a/apps/docs/docs/operations/managed-instance.md
+++ b/apps/docs/docs/operations/managed-instance.md
@@ -153,13 +153,46 @@ honest unit economic looks like for a SOC product, and we'd rather do
that with a small cohort of design partners than with a public price
sheet that we have to walk back later.
-What we **do** care about is making LLM cost visible from day one. The
+What we **do** care about is making cost visible from day one. The
[LLM cost dashboard](https://tryaisoc.com/costs) (rolled out as WS-H1 of
the v1.0 buyer-value plan) tracks every model call with token counts,
provider, and run cost, scoped per tenant. When the beta exits and
pricing lands, that dashboard is what we'll bill against — so
customers can see the bill being built in real-time today.
+### Storage, and why its number is shaped differently
+
+LLM spend and storage spend are both on that page and they are **not the
+same kind of number**, so the console keeps them apart and never adds them
+together.
+
+LLM cost is measured: the gateway reports what a call cost, and where it
+cannot, the figure is re-priced from a published list price and labelled an
+estimate.
+
+Storage cost is **projected**. There is no per-tenant storage invoice to
+read, so the console measures the one thing it can — the uncompressed bytes
+your events occupied in the event lake over the window — and runs the
+committed storage cost model over it
+([ADR-0005](https://github.com/beenuar/AiSOC/blob/main/docs/decisions/0005-storage-consolidation.md),
+worked example in `docs/decisions/storage-cost-model.json`,
+drift-gated by `.github/workflows/perf.yml`). At the model's reference
+scenario of 1 TB/day that comes out at about **$902/month**, or **$30 per
+raw TB ingested**, across a hot/warm/cold ladder at reference list prices.
+
+Three things follow, and the panel states each of them:
+
+- It is a model, not your provider's bill. Verify against your own region
+ and negotiated rates before planning against it.
+- The model's retention shape (30 days hot / 60 warm / 275 cold) is the
+ sizing scenario, not necessarily what your deployment runs — the shipped
+ ClickHouse tiering SQL is two tiers over 90 days, and per-tenant retention
+ policies override both.
+- The event lake runs in the `full` profile. On a CORE deployment there is
+ no ingest volume to measure, so the panel reads **"Not measured"** with the
+ reason. It does not read `$0.00`, because a confident zero for a tenant
+ nobody measured is indistinguishable from free storage.
+
## Operator runbook (for the AiSOC community)
If you're operating the managed instance yourself, the relevant
diff --git a/apps/web/src/components/admin/CostDashboardView.test.tsx b/apps/web/src/components/admin/CostDashboardView.test.tsx
index 73565b452..0bc9f73ea 100644
--- a/apps/web/src/components/admin/CostDashboardView.test.tsx
+++ b/apps/web/src/components/admin/CostDashboardView.test.tsx
@@ -117,6 +117,22 @@ function fixture(overrides: Partial<CostDashboard> = {}): CostDashboard {
unpriced_tokens: 0,
savings_usd: 0,
},
+ storage: {
+ measured: true,
+ unmeasured_reason: null,
+ raw_bytes_measured: 3_000_000_000,
+ events_measured: 412_000,
+ raw_tb_per_day: 0.0001,
+ monthly_usd: 0.09,
+ usd_per_raw_tb_ingested: 30.07,
+ tiers: [
+ { tier: 'hot_block', retention_days: 30, resident_gb: 0.38, rate_usd_per_gb_month: 0.08, monthly_usd: 0.03 },
+ { tier: 'warm_object', retention_days: 60, resident_gb: 0.75, rate_usd_per_gb_month: 0.023, monthly_usd: 0.02 },
+ { tier: 'cold_archive', retention_days: 275, resident_gb: 3.44, rate_usd_per_gb_month: 0.0125, monthly_usd: 0.04 },
+ ],
+ compression_ratio: 8.0,
+ disclaimer: 'Projected from your measured ingest volume using the committed storage cost model.',
+ },
...overrides,
};
}
@@ -343,6 +359,40 @@ describe('CostDashboardView — a window nothing measured', () => {
// The single most important assertion in this file: a page rendering
// 1,287 real LLM calls must not print a currency amount it did not
// measure. Not "$0.00", not "<$0.01", not a default-rate estimate.
+ //
+ // The storage panel is removed from the text first, and only that panel.
+ // It is a projection over its own, separate measurement — the bytes the
+ // tenant's events occupied in the lake — so it can legitimately carry a
+ // figure in a window where no LLM call was priced. Excluding it by
+ // node rather than by loosening the pattern keeps this assertion exactly
+ // as strict as it was for every LLM surface, and the storage panel's own
+ // honesty (labelled a model; "Not measured" rather than $0.00 when the
+ // lake is absent) is asserted in its own describe block below.
+ screen.getByTestId('storage-panel').remove();
+
+ expect(document.body.textContent).not.toMatch(/\$\d/);
+ });
+
+ it('still says not measured for storage when the lake is absent', () => {
+ const payload = unmeasured();
+ payload.storage = {
+ measured: false,
+ unmeasured_reason: 'no lake on this deployment',
+ raw_bytes_measured: null,
+ events_measured: null,
+ raw_tb_per_day: null,
+ monthly_usd: null,
+ usd_per_raw_tb_ingested: null,
+ tiers: [],
+ compression_ratio: 8.0,
+ disclaimer: 'Projected from your measured ingest volume.',
+ };
+ seed(30, payload);
+ render(<CostDashboardView />);
+
+ // With both halves unmeasured, the original whole-body guarantee holds
+ // with nothing excluded — which is what makes the exclusion above an
+ // exemption for a measured figure, not a hole.
expect(document.body.textContent).not.toMatch(/\$\d/);
});
@@ -425,3 +475,52 @@ describe('CostDashboardView — estimates are labelled', () => {
).toBeInTheDocument();
});
});
+
+/**
+ * ADR-0005 / 6b. The storage figure is a model run over one measurement, and
+ * it sits beside a measured LLM spend. Two things must hold on every render:
+ * it is never presented as billed, and an absent lake reads as "not measured"
+ * rather than as $0.00 — a confident zero for a tenant nobody measured is how
+ * a deployment comes to look like it stores data for free.
+ */
+describe('CostDashboardView — storage is a projection, not a bill', () => {
+ it('labels the projection and never calls it billed', () => {
+ seed(30, fixture());
+ render(<CostDashboardView />);
+
+ const panel = screen.getByTestId('storage-panel');
+ expect(panel).toHaveAttribute('data-storage', 'projected');
+ expect(within(panel).getByText('Modelled, not billed')).toBeInTheDocument();
+ expect(within(panel).getByText(/projected at full retention/)).toBeInTheDocument();
+ expect(within(panel).getByTestId('storage-tiers')).toBeInTheDocument();
+ });
+
+ it('says not measured, with the reason, when the lake is absent', () => {
+ const base = fixture();
+ seed(30, {
+ ...base,
+ storage: {
+ measured: false,
+ unmeasured_reason:
+ 'Ingest volume is counted in the ClickHouse event lake, which runs in the `full` profile.',
+ raw_bytes_measured: null,
+ events_measured: null,
+ raw_tb_per_day: null,
+ monthly_usd: null,
+ usd_per_raw_tb_ingested: null,
+ tiers: [],
+ compression_ratio: 8.0,
+ disclaimer: 'Projected from your measured ingest volume.',
+ },
+ });
+ render(<CostDashboardView />);
+
+ const panel = screen.getByTestId('storage-panel');
+ expect(panel).toHaveAttribute('data-storage', 'not-measured');
+ expect(within(panel).getByText('Not measured')).toBeInTheDocument();
+ expect(within(panel).getByText(/full. profile/)).toBeInTheDocument();
+ // The failure this guards: an unmeasured projection rendering as money.
+ expect(within(panel).queryByText(/\$0\.00/)).not.toBeInTheDocument();
+ expect(within(panel).queryByTestId('storage-tiers')).not.toBeInTheDocument();
+ });
+});
diff --git a/apps/web/src/components/admin/CostDashboardView.tsx b/apps/web/src/components/admin/CostDashboardView.tsx
index ed47d2cb7..c4b9e335e 100644
--- a/apps/web/src/components/admin/CostDashboardView.tsx
+++ b/apps/web/src/components/admin/CostDashboardView.tsx
@@ -198,6 +198,7 @@ function DashboardBody({ data }: { data: CostDashboard }) {
<ModelTable data={data} />
<CasesAndActions data={data} />
<ByokPanel data={data} />
+ <StoragePanel data={data} />
</>
);
}
@@ -529,6 +530,102 @@ function ByokPanel({ data }: { data: CostDashboard }) {
);
}
+function fmtGb(gb: number): string {
+ if (gb >= 1000) return `${(gb / 1000).toFixed(2)} TB`;
+ if (gb >= 1) return `${gb.toFixed(1)} GB`;
+ return `${(gb * 1000).toFixed(0)} MB`;
+}
+
+/**
+ * ADR-0005 / 6b — storage $/mo beside LLM $/mo.
+ *
+ * Kept visually distinct from the LLM figures above and never added to them.
+ * The spend above is measured or imputed from a real call; this is a model
+ * run over one measurement, so the two answer different questions and a
+ * single total would label one number two ways.
+ *
+ * When the lake is absent there is no measurement, and the panel says so with
+ * the reason rather than rendering a zero — the same contract the measured /
+ * estimated / unpriced trichotomy holds for spend.
+ */
+function StoragePanel({ data }: { data: CostDashboard }) {
+ const s = data.storage;
+
+ if (!s.measured) {
+ return (
+ <section
+ className="rounded-xl border border-gray-700 bg-gray-900 p-5"
+ data-testid="storage-panel"
+ data-storage="not-measured"
+ >
+ <p className="text-xs uppercase tracking-wide text-gray-400">Storage cost</p>
+ <p className="mt-1 text-lg font-semibold text-white">Not measured</p>
+ <p className="mt-3 text-sm text-gray-400">{s.unmeasured_reason}</p>
+ </section>
+ );
+ }
+
+ return (
+ <section
+ className="rounded-xl border border-gray-700 bg-gray-900 p-5"
+ data-testid="storage-panel"
+ data-storage="projected"
+ >
+ <div className="flex flex-wrap items-start justify-between gap-3">
+ <div>
+ <p className="text-xs uppercase tracking-wide text-gray-400">Storage cost</p>
+ <p className="mt-1 text-lg font-semibold text-white">
+ ~{fmtUsd(s.monthly_usd ?? 0)}/mo projected at full retention
+ </p>
+ </div>
+ <span className="rounded-full bg-amber-500/20 px-3 py-1 text-xs font-medium text-amber-200">
+ Modelled, not billed
+ </span>
+ </div>
+ <dl className="mt-4 grid grid-cols-1 gap-4 text-sm sm:grid-cols-3">
+ <Stat
+ label="Ingest measured"
+ value={fmtGb((s.raw_bytes_measured ?? 0) / 1e9)}
+ hint={`${fmtNumber(s.events_measured ?? 0)} events in this window`}
+ />
+ <Stat
+ label="Daily rate"
+ value={`${(s.raw_tb_per_day ?? 0).toFixed(4)} TB/day`}
+ hint={`compressed ~${s.compression_ratio}x in the lake`}
+ />
+ <Stat
+ label="Per raw TB ingested"
+ value={`~${fmtUsd(s.usd_per_raw_tb_ingested ?? 0)}`}
+ hint="reference list prices"
+ />
+ </dl>
+ <table className="mt-4 w-full text-left text-sm" data-testid="storage-tiers">
+ <thead>
+ <tr className="border-b border-gray-800 text-xs uppercase tracking-wide text-gray-500">
+ <th className="py-2 font-medium">Tier</th>
+ <th className="py-2 text-right font-medium">Retention</th>
+ <th className="py-2 text-right font-medium">Resident</th>
+ <th className="py-2 text-right font-medium">$/GB-mo</th>
+ <th className="py-2 text-right font-medium">$/mo</th>
+ </tr>
+ </thead>
+ <tbody>
+ {s.tiers.map((t) => (
+ <tr key={t.tier} className="border-b border-gray-900 last:border-0">
+ <td className="py-2 text-gray-300">{t.tier.replace('_', ' ')}</td>
+ <td className="py-2 text-right text-gray-400">{t.retention_days}d</td>
+ <td className="py-2 text-right text-gray-400">{fmtGb(t.resident_gb)}</td>
+ <td className="py-2 text-right text-gray-400">{t.rate_usd_per_gb_month}</td>
+ <td className="py-2 text-right text-gray-300">{fmtUsd(t.monthly_usd)}</td>
+ </tr>
+ ))}
+ </tbody>
+ </table>
+ <p className="mt-4 text-xs text-gray-500">{s.disclaimer}</p>
+ </section>
+ );
+}
+
// ── Local primitives (mirror ExecutiveDigest for visual consistency) ───────
function Card({
diff --git a/apps/web/src/lib/api.ts b/apps/web/src/lib/api.ts
index 7a5a00692..a0bfb9ba5 100644
--- a/apps/web/src/lib/api.ts
+++ b/apps/web/src/lib/api.ts
@@ -5812,6 +5812,38 @@ export interface ByokSavings {
savings_usd: number;
}
+export interface StorageTierProjection {
+ tier: string;
+ retention_days: number;
+ resident_gb: number;
+ rate_usd_per_gb_month: number;
+ monthly_usd: number;
+}
+
+/**
+ * ADR-0005 / 6b — projected storage cost beside the measured LLM spend.
+ *
+ * A projection, never a bill: it runs the committed storage cost model over
+ * one measurement (the uncompressed bytes this tenant's events occupied in
+ * the lake) at reference list prices. `measured` is the field that matters —
+ * when it is false every money field is `null`, not `0`, because the lake
+ * runs in the `full` profile and a confident zero for a tenant nobody
+ * measured reads as free storage.
+ */
+export interface StorageCostProjection {
+ measured: boolean;
+ /** Why there is no projection. Null when there is one. */
+ unmeasured_reason: string | null;
+ raw_bytes_measured: number | null;
+ events_measured: number | null;
+ raw_tb_per_day: number | null;
+ monthly_usd: number | null;
+ usd_per_raw_tb_ingested: number | null;
+ tiers: StorageTierProjection[];
+ compression_ratio: number;
+ disclaimer: string;
+}
+
export interface CostDashboard {
tenant_id: string;
period: DashboardPeriod;
@@ -5821,6 +5853,7 @@ export interface CostDashboard {
top_cases: TopCostCase[];
action_counts: ActionCount[];
byok_savings: ByokSavings;
+ storage: StorageCostProjection;
}
/**
diff --git a/docs/openapi.yaml b/docs/openapi.yaml
index 16e790322..eeb13dcb1 100644
--- a/docs/openapi.yaml
+++ b/docs/openapi.yaml
@@ -20926,12 +20926,15 @@ components:
title: Action Counts
byok_savings:
$ref: '#/components/schemas/ByokSavings'
+ storage:
+ $ref: '#/components/schemas/StorageCostProjection'
type: object
required:
- tenant_id
- period
- headline
- byok_savings
+ - storage
title: CostDashboard
description: Top-level deterministic snapshot.
CostHeadline:
@@ -31506,6 +31509,94 @@ components:
- capability
title: StepDispatchRequest
description: One playbook step, expressed as the verb it names.
+ StorageCostProjection:
+ properties:
+ measured:
+ type: boolean
+ title: Measured
+ unmeasured_reason:
+ anyOf:
+ - type: string
+ - type: 'null'
+ title: Unmeasured Reason
+ raw_bytes_measured:
+ anyOf:
+ - type: integer
+ - type: 'null'
+ title: Raw Bytes Measured
+ events_measured:
+ anyOf:
+ - type: integer
+ - type: 'null'
+ title: Events Measured
+ raw_tb_per_day:
+ anyOf:
+ - type: number
+ - type: 'null'
+ title: Raw Tb Per Day
+ monthly_usd:
+ anyOf:
+ - type: number
+ - type: 'null'
+ title: Monthly Usd
+ usd_per_raw_tb_ingested:
+ anyOf:
+ - type: number
+ - type: 'null'
+ title: Usd Per Raw Tb Ingested
+ tiers:
+ items:
+ $ref: '#/components/schemas/StorageTierProjection'
+ type: array
+ title: Tiers
+ compression_ratio:
+ type: number
+ title: Compression Ratio
+ default: 8.0
+ disclaimer:
+ type: string
+ title: Disclaimer
+ default: Projected from your measured ingest volume using the committed
+ storage cost model (docs/decisions/storage-cost-model.json) at reference
+ list prices. This is a model, not your provider's bill — verify against
+ your own region and negotiated rates.
+ type: object
+ required:
+ - measured
+ title: StorageCostProjection
+ description: 'Projected storage cost for a tenant, or an honest statement that
+ there is none.
+
+
+ ``measured`` is the field that matters. When it is false every money field
+
+ is ``None`` — not ``0.0`` — and ``unmeasured_reason`` says why.'
+ StorageTierProjection:
+ properties:
+ tier:
+ type: string
+ title: Tier
+ retention_days:
+ type: integer
+ title: Retention Days
+ resident_gb:
+ type: number
+ title: Resident Gb
+ rate_usd_per_gb_month:
+ type: number
+ title: Rate Usd Per Gb Month
+ monthly_usd:
+ type: number
+ title: Monthly Usd
+ type: object
+ required:
+ - tier
+ - retention_days
+ - resident_gb
+ - rate_usd_per_gb_month
+ - monthly_usd
+ title: StorageTierProjection
+ description: One retention tier's share of the projected monthly storage cost.
SubmissionResponse:
properties:
id:
diff --git a/scripts/storage_cost_model.py b/scripts/storage_cost_model.py
index 5f33cb96a..e9b2f40ac 100644
--- a/scripts/storage_cost_model.py
+++ b/scripts/storage_cost_model.py
@@ -23,6 +23,8 @@ Usage:
from __future__ import annotations
import argparse
+import ast
+import contextlib
import json
import sys
from pathlib import Path
@@ -105,10 +107,63 @@ def _print_table(result: dict) -> None:
print(f" ({result['disclaimer']})")
+#: The API service's copy of this model. It cannot import this module — the
+#: container has neither ``scripts/`` nor a git checkout, and ``repo_root()``
+#: above shells out to git at import — so the constants are mirrored there,
+#: the same constraint that put a second LLM price table in
+#: ``cost_dashboard._PUBLIC_PRICING``. A second copy of a number is a second
+#: place for it to drift, so ``--check`` reads it back. Without this, 6b would
+#: have closed "the model has no consumer" by creating a consumer that could
+#: silently start quoting different prices.
+CONSUMER = "services/api/app/services/storage_cost.py"
+
+_CONSUMER_CONSTANTS = {
+ "RATE_CARD_USD_PER_GB_MONTH": lambda: RATE_CARD_USD_PER_GB_MONTH,
+ "RETENTION_DAYS": lambda: SCENARIO["retention_days"],
+ "COMPRESSION_RATIO": lambda: SCENARIO["compression_ratio"],
+}
+
+
+def check_consumer() -> list[str]:
+ """Read the API's mirrored constants back and report any disagreement.
+
+ Parsed rather than imported, for the same reason the API cannot import
+ this file: the gate runs in a dep-light lint job and must not pull in
+ FastAPI, pydantic and the ClickHouse driver to compare three literals.
+ """
+ path = ROOT / CONSUMER
+ if not path.exists():
+ return [f"{CONSUMER} is missing — the model's only consumer is gone, so this gate now checks a constant nothing reads"]
+
+ tree = ast.parse(path.read_text(encoding="utf-8"))
+ found: dict[str, object] = {}
+ for node in tree.body:
+ target: str | None = None
+ value: ast.expr | None = None
+ if isinstance(node, ast.Assign) and len(node.targets) == 1 and isinstance(node.targets[0], ast.Name):
+ target, value = node.targets[0].id, node.value
+ elif isinstance(node, ast.AnnAssign) and isinstance(node.target, ast.Name):
+ target, value = node.target.id, node.value
+ if target in _CONSUMER_CONSTANTS and value is not None:
+ with contextlib.suppress(ValueError):
+ found[target] = ast.literal_eval(value)
+
+ problems: list[str] = []
+ for name, expected in _CONSUMER_CONSTANTS.items():
+ if name not in found:
+ problems.append(f"{CONSUMER} no longer declares {name}; the mirror cannot be checked, so it is not a mirror")
+ elif found[name] != expected():
+ problems.append(
+ f"{CONSUMER}:{name} is {found[name]!r}, this model says {expected()!r} — "
+ "the console would quote a price this model does not"
+ )
+ return problems
+
+
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--json", dest="as_json", action="store_true")
- parser.add_argument("--check", action="store_true", help="fail if committed JSON is stale")
+ parser.add_argument("--check", action="store_true", help="fail if committed JSON or the API's mirrored constants are stale")
args = parser.parse_args()
result = compute(SCENARIO, RATE_CARD_USD_PER_GB_MONTH)
@@ -125,7 +180,13 @@ def main() -> int:
file=sys.stderr,
)
return 1
- print(f"OK: storage cost model current — ${result['usd_per_raw_tb_ingested']}/raw-TB")
+ drift = check_consumer()
+ if drift:
+ print(f"ERROR: the model and {CONSUMER} disagree:", file=sys.stderr)
+ for problem in drift:
+ print(f" - {problem}", file=sys.stderr)
+ return 1
+ print(f"OK: storage cost model current — ${result['usd_per_raw_tb_ingested']}/raw-TB, and {CONSUMER} mirrors it")
return 0
if args.as_json:
diff --git a/services/api/app/services/cost_dashboard.py b/services/api/app/services/cost_dashboard.py
index 692ad68ae..fb824a0f9 100644
--- a/services/api/app/services/cost_dashboard.py
+++ b/services/api/app/services/cost_dashboard.py
@@ -69,6 +69,13 @@ from pydantic import BaseModel, Field
from sqlalchemy import text
from sqlalchemy.ext.asyncio import AsyncSession
+from app.services.storage_cost import (
+ NO_LAKE_REASON,
+ StorageCostProjection,
+ measure_and_project,
+ not_measured,
+)
+
# ---------------------------------------------------------------------------
# Public list pricing for BYOK imputation.
#
@@ -280,6 +287,11 @@ class CostDashboard(BaseModel):
top_cases: list[TopCostCase] = Field(default_factory=list)
action_counts: list[ActionCount] = Field(default_factory=list)
byok_savings: ByokSavings
+ #: Projected storage cost beside the measured LLM spend (ADR-0005 / 6b).
+ #: Deliberately a sibling of the LLM figures and never summed with them:
+ #: one number cannot be labelled two ways, and adding a model to a
+ #: measurement is how the model stops looking like one.
+ storage: StorageCostProjection
# ---------------------------------------------------------------------------
@@ -336,6 +348,10 @@ class DashboardInputs:
cost_rows: list[CostRow] = field(default_factory=list)
audit_rows: list[AuditRow] = field(default_factory=list)
llm: LlmContext = field(default_factory=lambda: LlmContext(provider="none", is_local=False))
+ #: ADR-0005 / 6b. Storage is a projection over a measured ingest volume,
+ #: never a measured spend, so it arrives already labelled and the default
+ #: is "not measured" rather than an empty projection reading zero.
+ storage: StorageCostProjection = field(default_factory=lambda: not_measured(NO_LAKE_REASON))
# ---------------------------------------------------------------------------
@@ -611,6 +627,7 @@ def build_dashboard_from_rows(inputs: DashboardInputs) -> CostDashboard:
top_cases=_top_cases(inputs.cost_rows),
action_counts=_action_counts(inputs.audit_rows),
byok_savings=_byok_savings(inputs.cost_rows, inputs.llm),
+ storage=inputs.storage,
)
@@ -738,6 +755,7 @@ async def build_cost_dashboard(
cost_rows = await _fetch_cost_rows(db, tenant_id, start, end)
audit_rows = await _fetch_audit_rows(db, tenant_id, start, end)
+ storage = await measure_and_project(tenant_id, start=start, end=end, window_days=window_days)
inputs = DashboardInputs(
tenant_id=tenant_id,
@@ -749,6 +767,7 @@ async def build_cost_dashboard(
provider=llm_provider or "unknown",
is_local=bool(is_local),
),
+ storage=storage,
)
return build_dashboard_from_rows(inputs)
@@ -765,6 +784,7 @@ __all__ = [
"DashboardPeriod",
"LlmContext",
"ModelBreakdown",
+ "StorageCostProjection",
"TopCostCase",
"build_cost_dashboard",
"build_dashboard_from_rows",
diff --git a/services/api/app/services/storage_cost.py b/services/api/app/services/storage_cost.py
new file mode 100644
index 000000000..d00986650
--- /dev/null
+++ b/services/api/app/services/storage_cost.py
@@ -0,0 +1,253 @@
+"""A tenant's storage $/mo, beside its LLM $/mo (ADR-0005 follow-up, 6b).
+
+Phase 6 computed a storage cost model and gated it. Both halves were real:
+``scripts/storage_cost_model.py`` computes ≈$902/mo and ≈$30 per raw TB at
+1 TB/day, ``docs/decisions/storage-cost-model.json`` is the committed worked
+example, and ``.github/workflows/perf.yml`` fails when the two diverge.
+
+Nothing read it. ADR-0005 asked for the tenant's storage $/mo to show next to
+its LLM $/mo, and outside the ADR, the CHANGELOG and the perf workflow the
+model's only readers were two ClickHouse tiering files citing it in a comment.
+A cost model whose only consumer is the gate that checks the cost model is a
+well-tested constant.
+
+What this is, and what it is not
+--------------------------------
+This is a **projection**, not a bill. It takes one measurement — the
+uncompressed bytes a tenant's events occupied in the lake over the window —
+and runs the committed model's arithmetic over it at **reference list
+prices**. It is not that tenant's spend, it is not their provider's invoice,
+and it is not this deployment's actual retention shape (the shipped tiering
+SQL is two tiers over 90 days; the model's scenario is three over 365). Every
+one of those distinctions is carried on the wire so the console can say it,
+because a modelled number rendered beside a measured one without a label is
+how a guess stops looking like one — the defect the rest of
+``cost_dashboard`` exists to prevent.
+
+Absence is not zero
+-------------------
+The lake runs in the ``full`` profile. On a CORE deployment there is no
+ClickHouse, so there is no measurement, and the projection is reported as
+**not measured with the reason** rather than as ``$0.00``. This is the same
+contract ``usage_metering.UNMEASURED`` already holds for ``events_ingested``,
+and for the same reason: a confident zero for a tenant nobody measured reads
+as free storage.
+
+Why the rate card is duplicated here
+------------------------------------
+``scripts/storage_cost_model.py`` calls ``repo_root()`` at module scope, which
+shells out to git. The API container has neither ``scripts/`` nor a checkout,
+so it cannot be imported — the same constraint that put a second copy of the
+LLM price table in ``cost_dashboard._PUBLIC_PRICING``. A second copy of a
+number is a second place for it to drift, so ``storage_cost_model.py --check``
+now reads these constants back and fails when they disagree. The duplication
+is deliberate; the divergence is gated.
+"""
+
+from __future__ import annotations
+
+import logging
+import uuid
+from datetime import datetime
+
+from pydantic import BaseModel, Field
+
+logger = logging.getLogger("aisoc.storage_cost")
+
+# ---------------------------------------------------------------------------
+# The model. Mirrored from scripts/storage_cost_model.py and gated against it
+# by `python3 scripts/storage_cost_model.py --check`.
+# ---------------------------------------------------------------------------
+
+#: USD per GB-month, by tier. Reference list prices.
+RATE_CARD_USD_PER_GB_MONTH: dict[str, float] = {
+ "hot_block": 0.080, # gp3-class block storage backing the ClickHouse hot tier
+ "warm_object": 0.023, # standard object storage
+ "cold_archive": 0.0125, # infrequent-access / archive object storage
+}
+
+#: Days resident in each tier. Sums to 365.
+RETENTION_DAYS: dict[str, int] = {"hot_block": 30, "warm_object": 60, "cold_archive": 275}
+
+#: What the lake's ZSTD(3) columns achieve on OCSF-shaped JSON.
+COMPRESSION_RATIO: float = 8.0
+
+_GB_PER_TB = 1000
+_BYTES_PER_GB = 1_000_000_000
+
+#: Carried on every projection so a reader cannot take it for a bill.
+DISCLAIMER = (
+ "Projected from your measured ingest volume using the committed storage cost model "
+ "(docs/decisions/storage-cost-model.json) at reference list prices. This is a model, "
+ "not your provider's bill — verify against your own region and negotiated rates."
+)
+
+#: Said out loud when the lake is absent, rather than reporting a zero.
+NO_LAKE_REASON = (
+ "Ingest volume is counted in the ClickHouse event lake, which runs in the `full` profile. "
+ "Not measured on a deployment without it, so no storage projection is possible."
+)
+
+
+class StorageTierProjection(BaseModel):
+ """One retention tier's share of the projected monthly storage cost."""
+
+ tier: str
+ retention_days: int
+ resident_gb: float
+ rate_usd_per_gb_month: float
+ monthly_usd: float
+
+
+class StorageCostProjection(BaseModel):
+ """Projected storage cost for a tenant, or an honest statement that there is none.
+
+ ``measured`` is the field that matters. When it is false every money field
+ is ``None`` — not ``0.0`` — and ``unmeasured_reason`` says why.
+ """
+
+ measured: bool
+ #: Why there is no projection. ``None`` when there is one.
+ unmeasured_reason: str | None = None
+
+ #: The one measurement this rests on: uncompressed bytes the tenant's
+ #: events occupied over the window, and how many events that was.
+ raw_bytes_measured: int | None = None
+ events_measured: int | None = None
+
+ #: The measurement, expressed as the model's input.
+ raw_tb_per_day: float | None = None
+
+ #: The projection.
+ monthly_usd: float | None = None
+ usd_per_raw_tb_ingested: float | None = None
+ tiers: list[StorageTierProjection] = Field(default_factory=list)
+
+ #: Model parameters, echoed so the console can show what produced the number.
+ compression_ratio: float = COMPRESSION_RATIO
+ disclaimer: str = DISCLAIMER
+
+
+def project(raw_bytes: int, *, window_days: int, events: int | None = None) -> StorageCostProjection:
+ """Run the committed model over a measured ingest volume.
+
+ Pure. ``raw_bytes`` is uncompressed bytes observed over ``window_days``;
+ the arithmetic below is the same as ``storage_cost_model.compute`` with
+ the scenario's fixed 1 TB/day replaced by the tenant's own rate.
+ """
+ days = max(window_days, 1)
+ raw_tb_per_day = (raw_bytes / _BYTES_PER_GB) / _GB_PER_TB / days
+
+ stored_gb_per_day = (raw_tb_per_day * _GB_PER_TB) / COMPRESSION_RATIO
+ tiers: list[StorageTierProjection] = []
+ total_monthly = 0.0
+ for tier, retention in RETENTION_DAYS.items():
+ resident_gb = stored_gb_per_day * retention
+ rate = RATE_CARD_USD_PER_GB_MONTH[tier]
+ monthly = resident_gb * rate
+ total_monthly += monthly
+ tiers.append(
+ StorageTierProjection(
+ tier=tier,
+ retention_days=retention,
+ resident_gb=round(resident_gb, 2),
+ rate_usd_per_gb_month=rate,
+ monthly_usd=round(monthly, 2),
+ )
+ )
+
+ raw_tb_per_month = raw_tb_per_day * 30
+ # A tenant that ingested nothing has a real, measured zero — it is not the
+ # same as a tenant nobody measured, and `measured` is what tells them apart.
+ per_tb = round(total_monthly / raw_tb_per_month, 2) if raw_tb_per_month else 0.0
+
+ return StorageCostProjection(
+ measured=True,
+ raw_bytes_measured=raw_bytes,
+ events_measured=events,
+ raw_tb_per_day=round(raw_tb_per_day, 6),
+ monthly_usd=round(total_monthly, 2),
+ usd_per_raw_tb_ingested=per_tb,
+ tiers=tiers,
+ )
+
+
+def not_measured(reason: str) -> StorageCostProjection:
+ """A projection that could not be made, carrying why."""
+ return StorageCostProjection(measured=False, unmeasured_reason=reason)
+
+
+#: The tenant predicate is bound as a query parameter rather than formatted
+#: in. ``rewrite_for_tenant`` exists for untrusted operator SQL; this SQL is
+#: the API's own, and the isolation lesson from ``_events_of_interest`` was
+#: that handing API-authored SQL to the rewriter and trusting the predicate
+#: survived is how every tenant came to see the same global figure.
+_VOLUME_SQL = """
+SELECT count() AS events,
+ sum(length(raw_payload) + length(ocsf_json)) AS raw_bytes
+FROM aisoc.raw_events
+WHERE tenant_id = %(tenant_id)s
+ AND ingest_time >= %(start_at)s
+ AND ingest_time < %(end_at)s
+"""
+
+
+async def measure_and_project(
+ tenant_id: uuid.UUID,
+ *,
+ start: datetime,
+ end: datetime,
+ window_days: int,
+) -> StorageCostProjection:
+ """Measure the tenant's ingest volume and project its storage cost.
+
+ Never raises: an unreachable or unconfigured lake is reported as *not
+ measured* with the reason, because a storage panel that 500s the whole
+ cost dashboard would trade a missing number for a missing page.
+ """
+ # Imported here rather than at module scope, deliberately. Everything
+ # above this function is the model itself — constants and arithmetic over
+ # them — and importing the lake pulls in `app.core.config`, the settings
+ # stack and the ClickHouse driver. `cost_dashboard`'s own test suite is
+ # pure by design (rows in, dashboard out, no database), and a module-level
+ # import here would have made the pure builder untestable without a
+ # driver, which is the coupling the split in `cost_dashboard` exists to
+ # avoid. The same reasoning as `retro_hunt_consumer._build_consumer`.
+ from app.db.clickhouse import ( # noqa: PLC0415
+ LakeQueryError,
+ LakeQueryNotConfiguredError,
+ execute_lake_query,
+ )
+
+ try:
+ result = await execute_lake_query(
+ _VOLUME_SQL,
+ timeout_seconds=10.0,
+ params={"tenant_id": str(tenant_id), "start_at": start, "end_at": end},
+ )
+ except LakeQueryNotConfiguredError:
+ return not_measured(NO_LAKE_REASON)
+ except LakeQueryError as exc:
+ logger.warning("storage_cost lake query failed: %s", str(exc).replace("\r", "").replace("\n", " ")[:200])
+ return not_measured(f"The event lake did not answer, so ingest volume was not measured this run ({type(exc).__name__}).")
+
+ if not result.rows or result.rows[0][1] is None:
+ return not_measured("The event lake returned no rows for this tenant and window, so there is nothing to project from.")
+
+ events = int(result.rows[0][0] or 0)
+ raw_bytes = int(result.rows[0][1] or 0)
+ return project(raw_bytes, window_days=window_days, events=events)
+
+
+__all__ = [
+ "COMPRESSION_RATIO",
+ "DISCLAIMER",
+ "NO_LAKE_REASON",
+ "RATE_CARD_USD_PER_GB_MONTH",
+ "RETENTION_DAYS",
+ "StorageCostProjection",
+ "StorageTierProjection",
+ "measure_and_project",
+ "not_measured",
+ "project",
+]
diff --git a/services/api/tests/test_storage_cost.py b/services/api/tests/test_storage_cost.py
new file mode 100644
index 000000000..8ce20c059
--- /dev/null
+++ b/services/api/tests/test_storage_cost.py
@@ -0,0 +1,159 @@
+"""The storage projection beside the LLM spend (ADR-0005 / 6b).
+
+Two properties carry the honesty rule here and both are asserted below.
+
+A projection is not a bill. The number comes from running the committed model
+over one measurement at reference list prices, so it must arrive labelled and
+must never be summed into the measured LLM spend.
+
+An absence is not a zero. The lake runs in the ``full`` profile, so on a CORE
+deployment there is no ingest volume to measure — and a confident ``$0.00``
+for a tenant nobody measured reads as free storage. That is the same contract
+``usage_metering.UNMEASURED`` already holds for ``events_ingested``.
+"""
+
+from __future__ import annotations
+
+import json
+import uuid
+from pathlib import Path
+
+import pytest
+from app.services.storage_cost import (
+ COMPRESSION_RATIO,
+ RATE_CARD_USD_PER_GB_MONTH,
+ RETENTION_DAYS,
+ StorageCostProjection,
+ measure_and_project,
+ not_measured,
+ project,
+)
+
+REPO_ROOT = Path(__file__).resolve().parents[3]
+COMMITTED_MODEL = REPO_ROOT / "docs" / "decisions" / "storage-cost-model.json"
+
+_BYTES_PER_TB = 1_000_000_000_000
+
+
+class TestItReproducesTheCommittedModel:
+ """The whole point of 6b is that the console quotes *this* model."""
+
+ def test_one_tb_per_day_reproduces_the_committed_worked_example(self) -> None:
+ committed = json.loads(COMMITTED_MODEL.read_text(encoding="utf-8"))
+ # 30 days at exactly the scenario's 1 TB/day.
+ result = project(_BYTES_PER_TB * 30, window_days=30, events=1)
+
+ assert result.monthly_usd == pytest.approx(committed["total_monthly_usd"], abs=0.02)
+ assert result.usd_per_raw_tb_ingested == pytest.approx(committed["usd_per_raw_tb_ingested"], abs=0.02)
+
+ def test_each_tier_reproduces_the_committed_tier(self) -> None:
+ committed = json.loads(COMMITTED_MODEL.read_text(encoding="utf-8"))
+ result = project(_BYTES_PER_TB * 30, window_days=30)
+
+ by_tier = {t.tier: t for t in result.tiers}
+ assert set(by_tier) == set(committed["tiers"])
+ for name, expected in committed["tiers"].items():
+ got = by_tier[name]
+ assert got.retention_days == expected["retention_days"]
+ assert got.rate_usd_per_gb_month == expected["rate_usd_per_gb_month"]
+ assert got.resident_gb == pytest.approx(expected["resident_gb"], rel=0.001)
+ assert got.monthly_usd == pytest.approx(expected["monthly_usd"], abs=0.02)
+
+ def test_the_constants_match_the_committed_rate_card(self) -> None:
+ """Belt to the drift gate's braces.
+
+ ``scripts/storage_cost_model.py --check`` parses these constants out of
+ the module. This asserts the same thing from the other side, so a
+ change that somehow satisfied the parser still fails here.
+ """
+ committed = json.loads(COMMITTED_MODEL.read_text(encoding="utf-8"))
+ assert RATE_CARD_USD_PER_GB_MONTH == committed["rate_card_usd_per_gb_month"]
+ assert RETENTION_DAYS == committed["scenario"]["retention_days"]
+ assert COMPRESSION_RATIO == committed["scenario"]["compression_ratio"]
+
+
+class TestAbsenceIsNotZero:
+ def test_not_measured_carries_the_reason_and_no_money(self) -> None:
+ result = not_measured("no lake on this deployment")
+
+ assert result.measured is False
+ assert result.unmeasured_reason == "no lake on this deployment"
+ # Every money field must be None. A zero here is the defect.
+ assert result.monthly_usd is None
+ assert result.usd_per_raw_tb_ingested is None
+ assert result.raw_bytes_measured is None
+ assert result.tiers == []
+
+ @pytest.mark.asyncio
+ async def test_an_unconfigured_lake_reports_not_measured_rather_than_zero(self, monkeypatch: pytest.MonkeyPatch) -> None:
+ from app.db import clickhouse
+
+ async def _refuse(*_args: object, **_kwargs: object) -> None:
+ raise clickhouse.LakeQueryNotConfiguredError("CLICKHOUSE_HOST is not configured")
+
+ monkeypatch.setattr(clickhouse, "execute_lake_query", _refuse)
+ result = await measure_and_project(uuid.uuid4(), start=_t(0), end=_t(30), window_days=30)
+
+ assert result.measured is False
+ assert result.monthly_usd is None
+ assert "full" in (result.unmeasured_reason or "")
+
+ @pytest.mark.asyncio
+ async def test_an_unreachable_lake_degrades_rather_than_raising(self, monkeypatch: pytest.MonkeyPatch) -> None:
+ """A missing storage number must not take the whole cost page with it."""
+ from app.db import clickhouse
+
+ async def _fail(*_args: object, **_kwargs: object) -> None:
+ raise clickhouse.LakeQueryError("connection refused")
+
+ monkeypatch.setattr(clickhouse, "execute_lake_query", _fail)
+ result = await measure_and_project(uuid.uuid4(), start=_t(0), end=_t(30), window_days=30)
+
+ assert result.measured is False
+ assert result.monthly_usd is None
+
+ def test_a_tenant_that_ingested_nothing_is_measured_zero_not_unmeasured(self) -> None:
+ """The distinction the `measured` flag exists to carry.
+
+ Ingesting nothing is a fact about the tenant. Having no lake is a fact
+ about the deployment. Reporting both as ``$0.00`` would lose the only
+ difference that matters to someone reading the number.
+ """
+ result = project(0, window_days=30, events=0)
+
+ assert result.measured is True
+ assert result.monthly_usd == 0.0
+ assert result.raw_bytes_measured == 0
+
+
+class TestItIsLabelledAsAModel:
+ def test_every_projection_carries_the_disclaimer(self) -> None:
+ result = project(_BYTES_PER_TB, window_days=30)
+
+ assert "not your provider's bill" in result.disclaimer
+ assert "reference list prices" in result.disclaimer
+
+ def test_no_field_is_named_so_it_reads_as_measured_spend(self) -> None:
+ """``check_cost_provenance.py`` reconciles any field ending ``cost_usd``
+ against a call count. A projection has no calls, so it deliberately
+ does not use that suffix — the naming is the label."""
+ fields = set(StorageCostProjection.model_fields)
+
+ assert not any(f.endswith("cost_usd") for f in fields), fields
+ assert "measured" in fields and "unmeasured_reason" in fields
+
+
+class TestItIsNotSummedIntoTheLlmSpend:
+ def test_the_dashboard_exposes_storage_as_a_sibling_not_a_total(self) -> None:
+ from app.services.cost_dashboard import CostDashboard, CostHeadline
+
+ assert "storage" in CostDashboard.model_fields
+ # If a future edit adds storage into the headline, this fails — which
+ # is the point. One number cannot be labelled two ways.
+ assert not any("storage" in f for f in CostHeadline.model_fields)
+
+
+def _t(day: int):
+ from datetime import UTC, datetime, timedelta
+
+ return datetime(2026, 5, 1, tzinfo=UTC) + timedelta(days=day)