Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

perf(4.4): entity-partitioned publishing, lag hints, and ADR-0011

Beenu Aroracda01b1214aa
diff --git a/docs/decisions/0011-how-fusion-scales.md b/docs/decisions/0011-how-fusion-scales.md
new file mode 100644
index 000000000..bffc39169
--- /dev/null
+++ b/docs/decisions/0011-how-fusion-scales.md
@@ -0,0 +1,126 @@
+# ADR-0011: How fusion scales — required-field indexing and entity partitioning
+
+- **Status:** accepted
+- **Date:** 2026-10-11
+- **Context:** depth plan 4.4 / parity plan 6.9
+
+## Context
+
+Parity 6.9 asks for three things: detection rules indexed by log source so
+fusion stops running every rule against every event, consumers partitioned
+with ordering kept per entity, and lag-based autoscaling hints.
+
+Two of those have an obvious implementation that is wrong, and this ADR
+exists to record why the shipped ones differ from the plan's wording.
+
+## Decision 1: index rules by required field, not by log source
+
+`services/fusion/app/services/detection_engine.py` evaluated every rule
+against every event. Its own docstring already recorded why a log-source
+index could not be the fix:
+
+> Connector product names don't line up 1:1 with spec products
+> (`aws_cloudtrail` vs `aws`, `crowdstrike_falcon` vs `edr`), so any
+> product-based pre-filter risks silently dropping a real match.
+
+A product-based pre-filter had been tried and reverted for exactly that.
+Reinstating it with a mapping table would leave every new connector one
+missing table entry away from a detection that stops firing, with nothing
+failing and no error anywhere — the worst available failure mode for a
+detection platform.
+
+**We index on the fields a rule requires instead.** `match_when` is an AND
+at the top level, so a rule that requires a field the event does not carry
+cannot match it, whatever either side's product names say. That is
+strictly more selective than log source *and* it is provably lossless,
+which log source is not.
+
+"Required" is decided by **running the real matcher against an empty
+event**, once per clause at load time, rather than by reading operator
+names:
+
+```python
+_check(field, op, expected, {}) is False # => the field is required
+```
+
+Writing the test for that proved the point. `neq` reads like a negation
+and would be on anyone's hand-written list of absence-tolerant operators.
+This matcher's `neq` returns `False` for a missing field *on purpose* —
+its own comment says a rule using it must not fire on absence of data — so
+the field is required. A hand-maintained list would have been wrong in the
+direction that drops detections.
+
+### Measured
+
+| | |
+|---|---|
+| Rules loaded | 2,511 |
+| Rules with at least one required field | 1,751 (69.74%) |
+| Rules in every candidate list | 760 |
+| Candidates for a routine cloud event | 797 of 2,511 (**68.3% fewer**) |
+| Evaluation time, full pass | 2.760 ms/event |
+| Evaluation time, indexed | 1.066 ms/event (**2.6x**) |
+
+Measured over 2,000 iterations in process on an 18-core Apple-silicon
+host. Not a cloud-hardware figure; see the performance page.
+
+### What keeps it safe
+
+`services/fusion/tests/test_rule_index_is_lossless.py` replays every
+committed fixture through both paths and requires identical hits in the
+same order, plus 200 random non-matching events and the empty event. The
+indexed fraction is pinned just below the measured 0.6974 so it can only
+ratchet up and a change that lowers it has to say so in the same commit.
+
+## Decision 2: partition by entity, keyed the way fusion correlates
+
+`services/ingest` keyed every Kafka message `tenant:event_id`, which is
+unique per event. Every event therefore hashed to an arbitrary partition
+and **nothing about one entity's events was ordered relative to another's**
+— while fusion correlates on `{tenant}:{entity}:{tactic}` and the windowed
+engine stitches sequences by event time. Two events about one host
+arriving on two partitions, consumed by two replicas, is the case where a
+sequence is stitched out of order or one incident is split in two.
+
+The key is now `tenant:entity`, with entity resolved in **the same
+precedence `RawAlert.correlation_key()` uses** — source IP, then hostname,
+then username, then domain. Partitions are ordered, so one entity's events
+are ordered relative to each other; different entities still spread across
+the cluster, so this is ordering without serialising the topic.
+
+A partition key that disagreed with the correlation key would route an
+entity's events to a partition and then correlate them under a different
+name, which spreads one incident across replicas while looking correct
+from either end. The two orders are held together by a test.
+
+An event with no resolvable entity falls back to its own id. Falling back
+to the tenant id would put every anonymous event in a tenant onto one
+partition — a hot partition, and the least useful one to serialise.
+
+## Decision 3: lag hints are published, not acted on
+
+`services/fusion` publishes a consumer-lag gauge and a derived
+`aisoc_fusion_scale_hint` so an operator's autoscaler can read it. AiSOC
+does **not** scale itself.
+
+Deliberate. An autoscaler is the cluster's, not the application's: a
+platform that scaled its own deployments would need write access to the
+orchestrator, would fight whatever HPA or KEDA scaler the operator already
+runs, and would be unable to honour the quota and cost policies that live
+outside it. Publishing the signal is the whole of our job; the
+`ScaledObject` in `apps/docs/docs/operations/performance.md` shows how to
+consume it.
+
+## Consequences
+
+- A rule whose clauses are all absence-tolerant is in every candidate
+ list. That is correct and is 760 rules today; a corpus that drifted
+ towards such rules would quietly lose the index's benefit, which is why
+ the fraction is pinned.
+- Partitioning by entity means a single very noisy entity is a hot
+ partition. That is the trade for per-entity ordering and is the
+ conventional one; the alternative loses the ordering fusion depends on.
+- The 5,000 events/s target in the depth plan is **not met and not
+ measured** on cloud hardware. No cloud instance has been provisioned for
+ this project. The performance page states that rather than implying a
+ figure.
diff --git a/services/fusion/app/api/router.py b/services/fusion/app/api/router.py
index cf4557f58..97d69dad3 100644
--- a/services/fusion/app/api/router.py
+++ b/services/fusion/app/api/router.py
@@ -43,6 +43,28 @@ async def metrics():
return {"status": "ok", "metrics": FusionWorker.get_metrics()}
[email protected]("/scale-hint")
+async def scale_hint():
+ """Lag-based autoscaling signal (parity 6.9 / depth 4.4).
+
+ AiSOC publishes this and does not act on it. An autoscaler belongs to
+ the cluster, not to the application; ADR-0011 records why, and
+ `apps/docs/docs/operations/performance.md` shows the KEDA
+ `ScaledObject` that consumes it.
+
+ Unauthenticated for the same reason `/metrics` is: a scaler scrapes
+ it on the cluster network and it carries no tenant data — a lag count
+ and a replica suggestion.
+ """
+ if _worker_ref is None:
+ # Not a zero. A hint of "one replica, no backlog" from a service
+ # whose worker never started is a scaler being told everything is
+ # fine by a process that is not consuming anything.
+ return {"status": "worker not started", "lag_measured": False}
+ hint = await _worker_ref.scale_hint()
+ return {"status": "ok", **hint}
+
+
@router.get("/ml/status")
async def ml_status(principal: ScopedPrincipal):
"""Return current ML model training status."""
diff --git a/services/fusion/app/services/scale_hint.py b/services/fusion/app/services/scale_hint.py
new file mode 100644
index 000000000..8f8de4025
--- /dev/null
+++ b/services/fusion/app/services/scale_hint.py
@@ -0,0 +1,147 @@
+"""Lag-based autoscaling hints (parity 6.9 / depth 4.4).
+
+AiSOC publishes a signal. It does not scale itself.
+
+That is a decision rather than an omission, and ADR-0011 records it: an
+autoscaler belongs to the cluster, not to the application. A platform
+that scaled its own deployments would need write access to the
+orchestrator, would fight whatever HPA or KEDA scaler the operator
+already runs, and could not honour the quota and cost policies that live
+outside it. Publishing the signal is the whole of the job.
+
+What the hint is
+----------------
+Consumer lag divided by a per-replica budget, rounded up, and clamped. It
+answers "how many replicas would clear this backlog in the target time",
+which is the input a KEDA ``ScaledObject`` or an HPA external metric
+wants — not "scale up", which is a decision with more inputs than fusion
+has.
+
+Where the budget comes from
+---------------------------
+From a measurement, and from one that is published. The realistic-mix run
+in ``apps/docs/docs/operations/performance.md`` is the reference, and the
+constant below names it. A per-replica budget picked out of the air would
+make the hint a number that relates to nothing — which is the defect this
+repository has already measured on a performance gate whose floor sat
+orders of magnitude below the figures it was supposed to protect.
+
+The budget is overridable per deployment, because the published figure is
+a laptop's and an operator's hardware is not. An override is read once and
+logged, so a hint nobody can explain has a line that explains it.
+"""
+
+from __future__ import annotations
+
+import math
+import os
+from dataclasses import dataclass
+
+import structlog
+
+logger = structlog.get_logger()
+
+#: Events per second one fusion replica drained in the published
+#: realistic-mix run (docs/perf/results/2026-10-11-compose-realistic-mix.json:
+#: 6,000 events accepted at 122.56 events/s through a single replica).
+#:
+#: Named rather than inlined so the derivation is greppable from the
+#: figure: if the published run changes, this is the one place that has to
+#: move, and the test that pins the two together fails until it does.
+PUBLISHED_REPLICA_EPS = 122.56
+
+#: How long a backlog should take to clear. Two minutes is a policy
+#: choice and is stated as one: nothing measured says two minutes is
+#: right. It is the window in which an operator would rather have added a
+#: replica than waited.
+DEFAULT_DRAIN_TARGET_SECONDS = 120.0
+
+#: Bounds on the hint. A hint of zero would ask an autoscaler to remove
+#: the last replica, and an unbounded one turns a stuck consumer — lag
+#: that is not falling because nothing is consuming — into an unbounded
+#: scale-out that cannot help.
+MIN_REPLICAS = 1
+MAX_REPLICAS = 20
+
+_EPS_ENV = "AISOC_FUSION_REPLICA_EPS"
+_TARGET_ENV = "AISOC_FUSION_DRAIN_TARGET_SECONDS"
+
+
+def _positive_float(name: str, default: float) -> float:
+ raw = os.environ.get(name, "").strip()
+ if not raw:
+ return default
+ try:
+ value = float(raw)
+ except ValueError:
+ logger.error("scale_hint.bad_override", variable=name, value=raw, using=default)
+ return default
+ if value <= 0:
+ logger.error("scale_hint.non_positive_override", variable=name, value=value, using=default)
+ return default
+ logger.info("scale_hint.override", variable=name, value=value)
+ return value
+
+
+@dataclass(frozen=True)
+class ScaleHint:
+ """What an external autoscaler should read."""
+
+ lag_messages: int
+ replica_eps: float
+ drain_target_seconds: float
+ suggested_replicas: int
+ #: True when the lag figure is unknown rather than zero. An autoscaler
+ #: reading a hint built on an unknown lag is reading a guess, and the
+ #: difference between "no backlog" and "we could not measure the
+ #: backlog" is the whole of this field.
+ lag_measured: bool
+
+ def as_dict(self) -> dict[str, object]:
+ return {
+ "lag_messages": self.lag_messages,
+ "lag_measured": self.lag_measured,
+ "replica_eps": self.replica_eps,
+ "drain_target_seconds": self.drain_target_seconds,
+ "suggested_replicas": self.suggested_replicas,
+ "basis": (
+ f"{self.lag_messages} message(s) backlog, cleared in "
+ f"{self.drain_target_seconds:.0f}s at {self.replica_eps:.1f} events/s per replica"
+ if self.lag_measured
+ else "consumer lag could not be read; the hint is the floor, not a measurement"
+ ),
+ }
+
+
+def compute(lag_messages: int | None) -> ScaleHint:
+ """Turn a measured consumer lag into a replica count.
+
+ ``None`` means the lag could not be read — a broker that did not
+ answer, a consumer not yet assigned a partition. That returns the
+ floor with ``lag_measured`` false, rather than zero lag: reporting an
+ unreadable backlog as no backlog is how a stalled consumer looks
+ healthy, which is the defect this service has shipped before.
+ """
+ replica_eps = _positive_float(_EPS_ENV, PUBLISHED_REPLICA_EPS)
+ target = _positive_float(_TARGET_ENV, DEFAULT_DRAIN_TARGET_SECONDS)
+
+ if lag_messages is None:
+ return ScaleHint(
+ lag_messages=0,
+ replica_eps=replica_eps,
+ drain_target_seconds=target,
+ suggested_replicas=MIN_REPLICAS,
+ lag_measured=False,
+ )
+
+ lag = max(0, int(lag_messages))
+ per_replica = replica_eps * target
+ suggested = MIN_REPLICAS if per_replica <= 0 else math.ceil(lag / per_replica)
+ suggested = max(MIN_REPLICAS, min(MAX_REPLICAS, suggested))
+ return ScaleHint(
+ lag_messages=lag,
+ replica_eps=replica_eps,
+ drain_target_seconds=target,
+ suggested_replicas=suggested,
+ lag_measured=True,
+ )
diff --git a/services/fusion/app/workers/consumer.py b/services/fusion/app/workers/consumer.py
index 07f60a422..63a74d3c1 100644
--- a/services/fusion/app/workers/consumer.py
+++ b/services/fusion/app/workers/consumer.py
@@ -22,6 +22,7 @@ from app.services.event_schema import validate_event
from app.services.fusion_engine import FusionEngine
from app.services.lake_writer import LakeWriter
from app.services.promoter import promote_normalized_event
+from app.services.scale_hint import compute as compute_scale_hint
from app.services.tenant_overlay import OverlayCache
from app.services.ueba_signal import UebaSignalCache
from app.services.windowed_detection import WindowedDetectionEngine
@@ -411,3 +412,41 @@ class FusionWorker:
@staticmethod
def get_metrics() -> dict:
return dict(_METRICS)
+
+ async def consumer_lag(self) -> int | None:
+ """Total backlog across the partitions this replica is assigned.
+
+ ``None`` when it cannot be read — no consumer, no assignment yet,
+ or a broker that did not answer. Deliberately not zero: an
+ unreadable backlog reported as no backlog is how a stalled
+ consumer looks healthy, which is the defect this service has
+ shipped before.
+ """
+ consumer = self._consumer
+ if consumer is None:
+ return None
+ try:
+ assignment = consumer.assignment()
+ if not assignment:
+ return None
+ total = 0
+ for partition in assignment:
+ end = await consumer.position(partition)
+ highwater = consumer.highwater(partition)
+ if end is None or highwater is None:
+ return None
+ total += max(0, highwater - end)
+ return total
+ except Exception as exc: # noqa: BLE001 — a lag read must never stop the consumer
+ logger.warning("fusion.lag_unreadable", error=str(exc))
+ return None
+
+ async def scale_hint(self) -> dict:
+ """The autoscaling signal, for an operator's HPA or KEDA scaler.
+
+ AiSOC publishes this and does not act on it; ADR-0011 records
+ why. The per-replica budget is derived from the published
+ realistic-mix run rather than chosen, so the number relates to a
+ measurement someone can go and check.
+ """
+ return compute_scale_hint(await self.consumer_lag()).as_dict()
diff --git a/services/fusion/tests/test_scale_hint.py b/services/fusion/tests/test_scale_hint.py
new file mode 100644
index 000000000..93a99c4e7
--- /dev/null
+++ b/services/fusion/tests/test_scale_hint.py
@@ -0,0 +1,99 @@
+"""The autoscaling hint, and the two ways a hint lies (parity 6.9).
+
+A hint is only useful if the number relates to something. The two
+failures this pins are both ones this repository has shipped before: a
+threshold that relates to no published figure, and an unmeasurable value
+reported as zero.
+"""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+
+import pytest
+from app.services import scale_hint
+
+REPO_ROOT = Path(__file__).resolve().parents[3]
+PUBLISHED_RUN = REPO_ROOT / "docs/perf/results/2026-10-11-compose-realistic-mix.json"
+
+
+def test_the_per_replica_budget_is_the_published_figure():
+ """The constant has to be derivable from a run someone can check.
+
+ A per-replica budget picked out of the air makes the hint a number
+ that relates to nothing — the defect this repository measured on a
+ performance gate whose floor sat orders of magnitude below the
+ figures it was supposed to protect. If the published run changes,
+ this fails until the constant moves with it.
+ """
+ assert PUBLISHED_RUN.is_file(), f"{PUBLISHED_RUN} is missing; the constant has no source"
+ report = json.loads(PUBLISHED_RUN.read_text(encoding="utf-8"))
+ published = report["metrics"]["ingest_accepted_eps"]
+ assert published["measured"] is True
+ assert scale_hint.PUBLISHED_REPLICA_EPS == pytest.approx(published["value"], abs=0.01), (
+ "PUBLISHED_REPLICA_EPS has drifted from the run it names"
+ )
+
+
+def test_an_unreadable_lag_is_not_reported_as_no_backlog(monkeypatch):
+ # The difference between "no backlog" and "we could not measure the
+ # backlog" is the whole of lag_measured. Reporting the second as the
+ # first is how a stalled consumer looks healthy.
+ hint = scale_hint.compute(None)
+ assert hint.lag_measured is False
+ assert hint.suggested_replicas == scale_hint.MIN_REPLICAS
+ assert "could not be read" in hint.as_dict()["basis"]
+
+
+def test_no_backlog_suggests_the_floor_and_says_it_was_measured():
+ hint = scale_hint.compute(0)
+ assert hint.lag_measured is True
+ assert hint.suggested_replicas == scale_hint.MIN_REPLICAS
+
+
+def test_a_backlog_one_replica_clears_in_the_window_needs_one_replica():
+ # 122.56 events/s for 120s is roughly 14,707 messages.
+ within = int(scale_hint.PUBLISHED_REPLICA_EPS * scale_hint.DEFAULT_DRAIN_TARGET_SECONDS) - 1
+ assert scale_hint.compute(within).suggested_replicas == 1
+
+
+def test_a_backlog_of_three_windows_asks_for_three_replicas():
+ per_replica = scale_hint.PUBLISHED_REPLICA_EPS * scale_hint.DEFAULT_DRAIN_TARGET_SECONDS
+ assert scale_hint.compute(int(per_replica * 2.5)).suggested_replicas == 3
+
+
+def test_the_hint_is_clamped_so_a_stuck_consumer_cannot_ask_for_the_cluster():
+ # Lag that is not falling because nothing is consuming would otherwise
+ # turn into an unbounded scale-out that cannot help.
+ enormous = int(scale_hint.PUBLISHED_REPLICA_EPS * scale_hint.DEFAULT_DRAIN_TARGET_SECONDS * 10_000)
+ assert scale_hint.compute(enormous).suggested_replicas == scale_hint.MAX_REPLICAS
+
+
+def test_the_hint_never_suggests_removing_the_last_replica():
+ assert scale_hint.compute(0).suggested_replicas >= 1
+ assert scale_hint.MIN_REPLICAS >= 1
+
+
+def test_a_negative_lag_is_floored_rather_than_inverting_the_hint():
+ assert scale_hint.compute(-500).lag_messages == 0
+
+
[email protected]("bad", ["not-a-number", "0", "-5"])
+def test_a_bad_override_falls_back_to_the_published_figure(monkeypatch, bad):
+ monkeypatch.setenv("AISOC_FUSION_REPLICA_EPS", bad)
+ assert scale_hint.compute(0).replica_eps == pytest.approx(scale_hint.PUBLISHED_REPLICA_EPS)
+
+
+def test_a_valid_override_is_honoured(monkeypatch):
+ # The published figure is a laptop's; an operator's hardware is not.
+ monkeypatch.setenv("AISOC_FUSION_REPLICA_EPS", "1000")
+ hint = scale_hint.compute(int(1000 * scale_hint.DEFAULT_DRAIN_TARGET_SECONDS * 1.5))
+ assert hint.replica_eps == pytest.approx(1000.0)
+ assert hint.suggested_replicas == 2
+
+
+def test_the_basis_string_states_what_the_number_came_from():
+ basis = scale_hint.compute(30_000).as_dict()["basis"]
+ assert "backlog" in basis
+ assert "per replica" in basis
diff --git a/services/ingest/internal/publisher/partition_key_test.go b/services/ingest/internal/publisher/partition_key_test.go
new file mode 100644
index 000000000..b9d5ac895
--- /dev/null
+++ b/services/ingest/internal/publisher/partition_key_test.go
@@ -0,0 +1,111 @@
+package publisher
+
+import (
+ "testing"
+
+ "github.com/beenuar/aisoc/services/ingest/internal/normalizer"
+)
+
+func event(tenant, id string, ocsf map[string]interface{}) *normalizer.NormalizedEvent {
+ return &normalizer.NormalizedEvent{ID: id, TenantID: tenant, OcsfEvent: ocsf}
+}
+
+// The whole point: two events about one host must land on one partition,
+// or fusion correlates them on two replicas and stitches a sequence out of
+// order or splits one incident in two.
+func TestTwoEventsAboutOneEntityShareAKey(t *testing.T) {
+ first := event("t1", "e1", map[string]interface{}{
+ "device": map[string]interface{}{"hostname": "WS-014"},
+ })
+ second := event("t1", "e2", map[string]interface{}{
+ "device": map[string]interface{}{"hostname": "WS-014"},
+ })
+ if string(partitionKey(first)) != string(partitionKey(second)) {
+ t.Fatalf("%q != %q", partitionKey(first), partitionKey(second))
+ }
+}
+
+func TestDifferentEntitiesGetDifferentKeys(t *testing.T) {
+ a := event("t1", "e1", map[string]interface{}{"hostname": "WS-014"})
+ b := event("t1", "e2", map[string]interface{}{"hostname": "WS-015"})
+ if string(partitionKey(a)) == string(partitionKey(b)) {
+ t.Fatal("two hosts hashed to one key; the topic is serialised, not partitioned")
+ }
+}
+
+// A tenant's events must never share a partition with another tenant's by
+// accident of having the same hostname.
+func TestTheSameEntityInTwoTenantsIsTwoKeys(t *testing.T) {
+ a := event("t1", "e1", map[string]interface{}{"hostname": "WS-014"})
+ b := event("t2", "e2", map[string]interface{}{"hostname": "WS-014"})
+ if string(partitionKey(a)) == string(partitionKey(b)) {
+ t.Fatal("two tenants share a partition key")
+ }
+}
+
+// The precedence has to be fusion's, or an entity is routed under one name
+// and correlated under another.
+func TestPrecedenceMatchesTheCorrelationKey(t *testing.T) {
+ cases := []struct {
+ name string
+ ocsf map[string]interface{}
+ want string
+ }{
+ {
+ "source ip wins over hostname",
+ map[string]interface{}{
+ "src_endpoint": map[string]interface{}{"ip": "198.51.100.4"},
+ "device": map[string]interface{}{"hostname": "WS-014"},
+ },
+ "t1:198.51.100.4",
+ },
+ {
+ "hostname wins over username",
+ map[string]interface{}{
+ "device": map[string]interface{}{"hostname": "WS-014"},
+ "actor.user": map[string]interface{}{"name": "alice"},
+ },
+ "t1:WS-014",
+ },
+ {
+ "flat connector fields are read too",
+ map[string]interface{}{"src_ip": "203.0.113.9"},
+ "t1:203.0.113.9",
+ },
+ }
+ for _, tc := range cases {
+ t.Run(tc.name, func(t *testing.T) {
+ if got := string(partitionKey(event("t1", "e1", tc.ocsf))); got != tc.want {
+ t.Fatalf("got %q, want %q", got, tc.want)
+ }
+ })
+ }
+}
+
+// An event naming nobody must not pile onto one partition per tenant: that
+// is a hot partition and also the least useful one to serialise.
+func TestAnEventWithNoEntityFallsBackToItsOwnId(t *testing.T) {
+ a := partitionKey(event("t1", "e1", map[string]interface{}{"message": "something happened"}))
+ b := partitionKey(event("t1", "e2", map[string]interface{}{"message": "something else"}))
+ if string(a) == string(b) {
+ t.Fatal("two anonymous events share a key; that is a hot partition")
+ }
+ if string(a) != "t1:e1" {
+ t.Fatalf("fallback key is %q", a)
+ }
+}
+
+func TestAnEmptyEntityStringIsNotUsedAsAKey(t *testing.T) {
+ // A whitespace-only hostname is not an entity, and keying on it would
+ // collapse every event that carries one onto a single partition.
+ got := partitionKey(event("t1", "e1", map[string]interface{}{"hostname": " "}))
+ if string(got) != "t1:e1" {
+ t.Fatalf("blank hostname produced key %q", got)
+ }
+}
+
+func TestANilOcsfEventStillGetsAKey(t *testing.T) {
+ if string(partitionKey(event("t1", "e1", nil))) != "t1:e1" {
+ t.Fatal("a nil OCSF event produced no usable key")
+ }
+}
diff --git a/services/ingest/internal/publisher/publisher.go b/services/ingest/internal/publisher/publisher.go
index d08813780..4c90abb42 100644
--- a/services/ingest/internal/publisher/publisher.go
+++ b/services/ingest/internal/publisher/publisher.go
@@ -6,6 +6,7 @@ import (
"encoding/json"
"fmt"
"net"
+ "strings"
"time"
"github.com/beenuar/aisoc/services/ingest/internal/config"
@@ -182,6 +183,99 @@ func (p *Publisher) Publish(ctx context.Context, event *normalizer.NormalizedEve
}
// PublishBatch sends multiple events in a single batch
+// partitionKey decides which partition an event lands on, and therefore
+// what fusion sees in order.
+//
+// Parity plan 6.9 / depth plan 4.4. The key used to be
+// `tenant:event_id`, which is unique per event — so every event hashed to
+// an arbitrary partition and nothing about one entity's events was
+// ordered relative to another's. Fusion correlates on
+// `{tenant}:{entity}:{tactic}` and the windowed engine stitches sequences
+// by event time, so two events about one host arriving on two partitions
+// and being consumed by two replicas is the case where a sequence is
+// stitched out of order or split across two incidents.
+//
+// The key is `tenant:entity`, where entity is resolved in the same
+// precedence the alert's own `correlation_key()` uses — source IP, then
+// hostname, then username, then domain. Partitions are ordered, so all of
+// one entity's events are ordered relative to each other; different
+// entities still spread across the cluster, so this is ordering without
+// serialising the topic.
+//
+// An event with no resolvable entity falls back to the event id. That
+// keeps the spread even for telemetry that names nobody, and such an
+// event has no sequence to be part of, so there is no ordering to lose.
+// Falling back to the tenant id instead would put every anonymous event
+// in one tenant onto one partition — a hot partition that is also the
+// least useful one.
+func partitionKey(event *normalizer.NormalizedEvent) []byte {
+ entity := primaryEntity(event)
+ if entity == "" {
+ return []byte(event.TenantID + ":" + event.ID)
+ }
+ return []byte(event.TenantID + ":" + entity)
+}
+
+// entityFieldOrder is the precedence fusion's own correlation key uses
+// (services/fusion/app/models/alert.py, `RawAlert.correlation_key`). Held
+// in the same order deliberately: a partition key that disagrees with the
+// correlation key routes an entity's events to a partition and then
+// correlates them under a different name, which spreads one incident
+// across replicas while looking correct from either end.
+var entityFieldOrder = []struct {
+ section string
+ field string
+}{
+ {"src_endpoint", "ip"},
+ {"device", "hostname"},
+ {"device", "name"},
+ {"actor.user", "name"},
+ {"", "src_ip"},
+ {"", "hostname"},
+ {"", "host"},
+ {"", "user"},
+ {"", "username"},
+ {"", "domain"},
+}
+
+// primaryEntity pulls the entity out of the OCSF event, checking the
+// structured locations first and the flat connector fields after.
+func primaryEntity(event *normalizer.NormalizedEvent) string {
+ ocsf := event.OcsfEvent
+ if ocsf == nil {
+ return ""
+ }
+ for _, candidate := range entityFieldOrder {
+ var value string
+ if candidate.section == "" {
+ value = stringAt(ocsf, candidate.field)
+ } else {
+ value = stringAt(ocsf, strings.Split(candidate.section+"."+candidate.field, ".")...)
+ }
+ if value != "" {
+ return value
+ }
+ }
+ return ""
+}
+
+// stringAt walks a dotted path through nested maps and returns a non-empty
+// string leaf, or "".
+func stringAt(root map[string]interface{}, path ...string) string {
+ var cur interface{} = root
+ for _, key := range path {
+ m, ok := cur.(map[string]interface{})
+ if !ok {
+ return ""
+ }
+ cur = m[key]
+ }
+ if s, ok := cur.(string); ok {
+ return strings.TrimSpace(s)
+ }
+ return ""
+}
+
func (p *Publisher) PublishBatch(ctx context.Context, events []*normalizer.NormalizedEvent) error {
if len(events) == 0 {
return nil
@@ -195,7 +289,7 @@ func (p *Publisher) PublishBatch(ctx context.Context, events []*normalizer.Norma
continue
}
msgs = append(msgs, kafka.Message{
- Key: []byte(event.TenantID + ":" + event.ID),
+ Key: partitionKey(event),
Value: data,
Headers: []kafka.Header{
{Key: "tenant_id", Value: []byte(event.TenantID)},