Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
perf(4.4): entity-partitioned publishing, lag hints, and ADR-0011
Beenu Aroracda01b1214aa
diff --git a/docs/decisions/0011-how-fusion-scales.md b/docs/decisions/0011-how-fusion-scales.mdnew file mode 100644index 000000000..bffc39169--- /dev/null+++ b/docs/decisions/0011-how-fusion-scales.md@@ -0,0 +1,126 @@+# ADR-0011: How fusion scales — required-field indexing and entity partitioning++- **Status:** accepted+- **Date:** 2026-10-11+- **Context:** depth plan 4.4 / parity plan 6.9++## Context++Parity 6.9 asks for three things: detection rules indexed by log source so+fusion stops running every rule against every event, consumers partitioned+with ordering kept per entity, and lag-based autoscaling hints.++Two of those have an obvious implementation that is wrong, and this ADR+exists to record why the shipped ones differ from the plan's wording.++## Decision 1: index rules by required field, not by log source++`services/fusion/app/services/detection_engine.py` evaluated every rule+against every event. Its own docstring already recorded why a log-source+index could not be the fix:++> Connector product names don't line up 1:1 with spec products+> (`aws_cloudtrail` vs `aws`, `crowdstrike_falcon` vs `edr`), so any+> product-based pre-filter risks silently dropping a real match.++A product-based pre-filter had been tried and reverted for exactly that.+Reinstating it with a mapping table would leave every new connector one+missing table entry away from a detection that stops firing, with nothing+failing and no error anywhere — the worst available failure mode for a+detection platform.++**We index on the fields a rule requires instead.** `match_when` is an AND+at the top level, so a rule that requires a field the event does not carry+cannot match it, whatever either side's product names say. That is+strictly more selective than log source *and* it is provably lossless,+which log source is not.++"Required" is decided by **running the real matcher against an empty+event**, once per clause at load time, rather than by reading operator+names:++```python+_check(field, op, expected, {}) is False # => the field is required+```++Writing the test for that proved the point. `neq` reads like a negation+and would be on anyone's hand-written list of absence-tolerant operators.+This matcher's `neq` returns `False` for a missing field *on purpose* —+its own comment says a rule using it must not fire on absence of data — so+the field is required. A hand-maintained list would have been wrong in the+direction that drops detections.++### Measured++| | |+|---|---|+| Rules loaded | 2,511 |+| Rules with at least one required field | 1,751 (69.74%) |+| Rules in every candidate list | 760 |+| Candidates for a routine cloud event | 797 of 2,511 (**68.3% fewer**) |+| Evaluation time, full pass | 2.760 ms/event |+| Evaluation time, indexed | 1.066 ms/event (**2.6x**) |++Measured over 2,000 iterations in process on an 18-core Apple-silicon+host. Not a cloud-hardware figure; see the performance page.++### What keeps it safe++`services/fusion/tests/test_rule_index_is_lossless.py` replays every+committed fixture through both paths and requires identical hits in the+same order, plus 200 random non-matching events and the empty event. The+indexed fraction is pinned just below the measured 0.6974 so it can only+ratchet up and a change that lowers it has to say so in the same commit.++## Decision 2: partition by entity, keyed the way fusion correlates++`services/ingest` keyed every Kafka message `tenant:event_id`, which is+unique per event. Every event therefore hashed to an arbitrary partition+and **nothing about one entity's events was ordered relative to another's**+— while fusion correlates on `{tenant}:{entity}:{tactic}` and the windowed+engine stitches sequences by event time. Two events about one host+arriving on two partitions, consumed by two replicas, is the case where a+sequence is stitched out of order or one incident is split in two.++The key is now `tenant:entity`, with entity resolved in **the same+precedence `RawAlert.correlation_key()` uses** — source IP, then hostname,+then username, then domain. Partitions are ordered, so one entity's events+are ordered relative to each other; different entities still spread across+the cluster, so this is ordering without serialising the topic.++A partition key that disagreed with the correlation key would route an+entity's events to a partition and then correlate them under a different+name, which spreads one incident across replicas while looking correct+from either end. The two orders are held together by a test.++An event with no resolvable entity falls back to its own id. Falling back+to the tenant id would put every anonymous event in a tenant onto one+partition — a hot partition, and the least useful one to serialise.++## Decision 3: lag hints are published, not acted on++`services/fusion` publishes a consumer-lag gauge and a derived+`aisoc_fusion_scale_hint` so an operator's autoscaler can read it. AiSOC+does **not** scale itself.++Deliberate. An autoscaler is the cluster's, not the application's: a+platform that scaled its own deployments would need write access to the+orchestrator, would fight whatever HPA or KEDA scaler the operator already+runs, and would be unable to honour the quota and cost policies that live+outside it. Publishing the signal is the whole of our job; the+`ScaledObject` in `apps/docs/docs/operations/performance.md` shows how to+consume it.++## Consequences++- A rule whose clauses are all absence-tolerant is in every candidate+ list. That is correct and is 760 rules today; a corpus that drifted+ towards such rules would quietly lose the index's benefit, which is why+ the fraction is pinned.+- Partitioning by entity means a single very noisy entity is a hot+ partition. That is the trade for per-entity ordering and is the+ conventional one; the alternative loses the ordering fusion depends on.+- The 5,000 events/s target in the depth plan is **not met and not+ measured** on cloud hardware. No cloud instance has been provisioned for+ this project. The performance page states that rather than implying a+ figure.diff --git a/services/fusion/app/api/router.py b/services/fusion/app/api/router.pyindex cf4557f58..97d69dad3 100644--- a/services/fusion/app/api/router.py+++ b/services/fusion/app/api/router.py@@ -43,6 +43,28 @@ async def metrics():return {"status": "ok", "metrics": FusionWorker.get_metrics()}[email protected]("/scale-hint")+async def scale_hint():+ """Lag-based autoscaling signal (parity 6.9 / depth 4.4).++ AiSOC publishes this and does not act on it. An autoscaler belongs to+ the cluster, not to the application; ADR-0011 records why, and+ `apps/docs/docs/operations/performance.md` shows the KEDA+ `ScaledObject` that consumes it.++ Unauthenticated for the same reason `/metrics` is: a scaler scrapes+ it on the cluster network and it carries no tenant data — a lag count+ and a replica suggestion.+ """+ if _worker_ref is None:+ # Not a zero. A hint of "one replica, no backlog" from a service+ # whose worker never started is a scaler being told everything is+ # fine by a process that is not consuming anything.+ return {"status": "worker not started", "lag_measured": False}+ hint = await _worker_ref.scale_hint()+ return {"status": "ok", **hint}++@router.get("/ml/status")async def ml_status(principal: ScopedPrincipal):"""Return current ML model training status."""diff --git a/services/fusion/app/services/scale_hint.py b/services/fusion/app/services/scale_hint.pynew file mode 100644index 000000000..8f8de4025--- /dev/null+++ b/services/fusion/app/services/scale_hint.py@@ -0,0 +1,147 @@+"""Lag-based autoscaling hints (parity 6.9 / depth 4.4).++AiSOC publishes a signal. It does not scale itself.++That is a decision rather than an omission, and ADR-0011 records it: an+autoscaler belongs to the cluster, not to the application. A platform+that scaled its own deployments would need write access to the+orchestrator, would fight whatever HPA or KEDA scaler the operator+already runs, and could not honour the quota and cost policies that live+outside it. Publishing the signal is the whole of the job.++What the hint is+----------------+Consumer lag divided by a per-replica budget, rounded up, and clamped. It+answers "how many replicas would clear this backlog in the target time",+which is the input a KEDA ``ScaledObject`` or an HPA external metric+wants — not "scale up", which is a decision with more inputs than fusion+has.++Where the budget comes from+---------------------------+From a measurement, and from one that is published. The realistic-mix run+in ``apps/docs/docs/operations/performance.md`` is the reference, and the+constant below names it. A per-replica budget picked out of the air would+make the hint a number that relates to nothing — which is the defect this+repository has already measured on a performance gate whose floor sat+orders of magnitude below the figures it was supposed to protect.++The budget is overridable per deployment, because the published figure is+a laptop's and an operator's hardware is not. An override is read once and+logged, so a hint nobody can explain has a line that explains it.+"""++from __future__ import annotations++import math+import os+from dataclasses import dataclass++import structlog++logger = structlog.get_logger()++#: Events per second one fusion replica drained in the published+#: realistic-mix run (docs/perf/results/2026-10-11-compose-realistic-mix.json:+#: 6,000 events accepted at 122.56 events/s through a single replica).+#:+#: Named rather than inlined so the derivation is greppable from the+#: figure: if the published run changes, this is the one place that has to+#: move, and the test that pins the two together fails until it does.+PUBLISHED_REPLICA_EPS = 122.56++#: How long a backlog should take to clear. Two minutes is a policy+#: choice and is stated as one: nothing measured says two minutes is+#: right. It is the window in which an operator would rather have added a+#: replica than waited.+DEFAULT_DRAIN_TARGET_SECONDS = 120.0++#: Bounds on the hint. A hint of zero would ask an autoscaler to remove+#: the last replica, and an unbounded one turns a stuck consumer — lag+#: that is not falling because nothing is consuming — into an unbounded+#: scale-out that cannot help.+MIN_REPLICAS = 1+MAX_REPLICAS = 20++_EPS_ENV = "AISOC_FUSION_REPLICA_EPS"+_TARGET_ENV = "AISOC_FUSION_DRAIN_TARGET_SECONDS"+++def _positive_float(name: str, default: float) -> float:+ raw = os.environ.get(name, "").strip()+ if not raw:+ return default+ try:+ value = float(raw)+ except ValueError:+ logger.error("scale_hint.bad_override", variable=name, value=raw, using=default)+ return default+ if value <= 0:+ logger.error("scale_hint.non_positive_override", variable=name, value=value, using=default)+ return default+ logger.info("scale_hint.override", variable=name, value=value)+ return value+++@dataclass(frozen=True)+class ScaleHint:+ """What an external autoscaler should read."""++ lag_messages: int+ replica_eps: float+ drain_target_seconds: float+ suggested_replicas: int+ #: True when the lag figure is unknown rather than zero. An autoscaler+ #: reading a hint built on an unknown lag is reading a guess, and the+ #: difference between "no backlog" and "we could not measure the+ #: backlog" is the whole of this field.+ lag_measured: bool++ def as_dict(self) -> dict[str, object]:+ return {+ "lag_messages": self.lag_messages,+ "lag_measured": self.lag_measured,+ "replica_eps": self.replica_eps,+ "drain_target_seconds": self.drain_target_seconds,+ "suggested_replicas": self.suggested_replicas,+ "basis": (+ f"{self.lag_messages} message(s) backlog, cleared in "+ f"{self.drain_target_seconds:.0f}s at {self.replica_eps:.1f} events/s per replica"+ if self.lag_measured+ else "consumer lag could not be read; the hint is the floor, not a measurement"+ ),+ }+++def compute(lag_messages: int | None) -> ScaleHint:+ """Turn a measured consumer lag into a replica count.++ ``None`` means the lag could not be read — a broker that did not+ answer, a consumer not yet assigned a partition. That returns the+ floor with ``lag_measured`` false, rather than zero lag: reporting an+ unreadable backlog as no backlog is how a stalled consumer looks+ healthy, which is the defect this service has shipped before.+ """+ replica_eps = _positive_float(_EPS_ENV, PUBLISHED_REPLICA_EPS)+ target = _positive_float(_TARGET_ENV, DEFAULT_DRAIN_TARGET_SECONDS)++ if lag_messages is None:+ return ScaleHint(+ lag_messages=0,+ replica_eps=replica_eps,+ drain_target_seconds=target,+ suggested_replicas=MIN_REPLICAS,+ lag_measured=False,+ )++ lag = max(0, int(lag_messages))+ per_replica = replica_eps * target+ suggested = MIN_REPLICAS if per_replica <= 0 else math.ceil(lag / per_replica)+ suggested = max(MIN_REPLICAS, min(MAX_REPLICAS, suggested))+ return ScaleHint(+ lag_messages=lag,+ replica_eps=replica_eps,+ drain_target_seconds=target,+ suggested_replicas=suggested,+ lag_measured=True,+ )diff --git a/services/fusion/app/workers/consumer.py b/services/fusion/app/workers/consumer.pyindex 07f60a422..63a74d3c1 100644--- a/services/fusion/app/workers/consumer.py+++ b/services/fusion/app/workers/consumer.py@@ -22,6 +22,7 @@ from app.services.event_schema import validate_eventfrom app.services.fusion_engine import FusionEnginefrom app.services.lake_writer import LakeWriterfrom app.services.promoter import promote_normalized_event+from app.services.scale_hint import compute as compute_scale_hintfrom app.services.tenant_overlay import OverlayCachefrom app.services.ueba_signal import UebaSignalCachefrom app.services.windowed_detection import WindowedDetectionEngine@@ -411,3 +412,41 @@ class FusionWorker:@staticmethoddef get_metrics() -> dict:return dict(_METRICS)++ async def consumer_lag(self) -> int | None:+ """Total backlog across the partitions this replica is assigned.++ ``None`` when it cannot be read — no consumer, no assignment yet,+ or a broker that did not answer. Deliberately not zero: an+ unreadable backlog reported as no backlog is how a stalled+ consumer looks healthy, which is the defect this service has+ shipped before.+ """+ consumer = self._consumer+ if consumer is None:+ return None+ try:+ assignment = consumer.assignment()+ if not assignment:+ return None+ total = 0+ for partition in assignment:+ end = await consumer.position(partition)+ highwater = consumer.highwater(partition)+ if end is None or highwater is None:+ return None+ total += max(0, highwater - end)+ return total+ except Exception as exc: # noqa: BLE001 — a lag read must never stop the consumer+ logger.warning("fusion.lag_unreadable", error=str(exc))+ return None++ async def scale_hint(self) -> dict:+ """The autoscaling signal, for an operator's HPA or KEDA scaler.++ AiSOC publishes this and does not act on it; ADR-0011 records+ why. The per-replica budget is derived from the published+ realistic-mix run rather than chosen, so the number relates to a+ measurement someone can go and check.+ """+ return compute_scale_hint(await self.consumer_lag()).as_dict()diff --git a/services/fusion/tests/test_scale_hint.py b/services/fusion/tests/test_scale_hint.pynew file mode 100644index 000000000..93a99c4e7--- /dev/null+++ b/services/fusion/tests/test_scale_hint.py@@ -0,0 +1,99 @@+"""The autoscaling hint, and the two ways a hint lies (parity 6.9).++A hint is only useful if the number relates to something. The two+failures this pins are both ones this repository has shipped before: a+threshold that relates to no published figure, and an unmeasurable value+reported as zero.+"""++from __future__ import annotations++import json+from pathlib import Path++import pytest+from app.services import scale_hint++REPO_ROOT = Path(__file__).resolve().parents[3]+PUBLISHED_RUN = REPO_ROOT / "docs/perf/results/2026-10-11-compose-realistic-mix.json"+++def test_the_per_replica_budget_is_the_published_figure():+ """The constant has to be derivable from a run someone can check.++ A per-replica budget picked out of the air makes the hint a number+ that relates to nothing — the defect this repository measured on a+ performance gate whose floor sat orders of magnitude below the+ figures it was supposed to protect. If the published run changes,+ this fails until the constant moves with it.+ """+ assert PUBLISHED_RUN.is_file(), f"{PUBLISHED_RUN} is missing; the constant has no source"+ report = json.loads(PUBLISHED_RUN.read_text(encoding="utf-8"))+ published = report["metrics"]["ingest_accepted_eps"]+ assert published["measured"] is True+ assert scale_hint.PUBLISHED_REPLICA_EPS == pytest.approx(published["value"], abs=0.01), (+ "PUBLISHED_REPLICA_EPS has drifted from the run it names"+ )+++def test_an_unreadable_lag_is_not_reported_as_no_backlog(monkeypatch):+ # The difference between "no backlog" and "we could not measure the+ # backlog" is the whole of lag_measured. Reporting the second as the+ # first is how a stalled consumer looks healthy.+ hint = scale_hint.compute(None)+ assert hint.lag_measured is False+ assert hint.suggested_replicas == scale_hint.MIN_REPLICAS+ assert "could not be read" in hint.as_dict()["basis"]+++def test_no_backlog_suggests_the_floor_and_says_it_was_measured():+ hint = scale_hint.compute(0)+ assert hint.lag_measured is True+ assert hint.suggested_replicas == scale_hint.MIN_REPLICAS+++def test_a_backlog_one_replica_clears_in_the_window_needs_one_replica():+ # 122.56 events/s for 120s is roughly 14,707 messages.+ within = int(scale_hint.PUBLISHED_REPLICA_EPS * scale_hint.DEFAULT_DRAIN_TARGET_SECONDS) - 1+ assert scale_hint.compute(within).suggested_replicas == 1+++def test_a_backlog_of_three_windows_asks_for_three_replicas():+ per_replica = scale_hint.PUBLISHED_REPLICA_EPS * scale_hint.DEFAULT_DRAIN_TARGET_SECONDS+ assert scale_hint.compute(int(per_replica * 2.5)).suggested_replicas == 3+++def test_the_hint_is_clamped_so_a_stuck_consumer_cannot_ask_for_the_cluster():+ # Lag that is not falling because nothing is consuming would otherwise+ # turn into an unbounded scale-out that cannot help.+ enormous = int(scale_hint.PUBLISHED_REPLICA_EPS * scale_hint.DEFAULT_DRAIN_TARGET_SECONDS * 10_000)+ assert scale_hint.compute(enormous).suggested_replicas == scale_hint.MAX_REPLICAS+++def test_the_hint_never_suggests_removing_the_last_replica():+ assert scale_hint.compute(0).suggested_replicas >= 1+ assert scale_hint.MIN_REPLICAS >= 1+++def test_a_negative_lag_is_floored_rather_than_inverting_the_hint():+ assert scale_hint.compute(-500).lag_messages == 0++[email protected]("bad", ["not-a-number", "0", "-5"])+def test_a_bad_override_falls_back_to_the_published_figure(monkeypatch, bad):+ monkeypatch.setenv("AISOC_FUSION_REPLICA_EPS", bad)+ assert scale_hint.compute(0).replica_eps == pytest.approx(scale_hint.PUBLISHED_REPLICA_EPS)+++def test_a_valid_override_is_honoured(monkeypatch):+ # The published figure is a laptop's; an operator's hardware is not.+ monkeypatch.setenv("AISOC_FUSION_REPLICA_EPS", "1000")+ hint = scale_hint.compute(int(1000 * scale_hint.DEFAULT_DRAIN_TARGET_SECONDS * 1.5))+ assert hint.replica_eps == pytest.approx(1000.0)+ assert hint.suggested_replicas == 2+++def test_the_basis_string_states_what_the_number_came_from():+ basis = scale_hint.compute(30_000).as_dict()["basis"]+ assert "backlog" in basis+ assert "per replica" in basisdiff --git a/services/ingest/internal/publisher/partition_key_test.go b/services/ingest/internal/publisher/partition_key_test.gonew file mode 100644index 000000000..b9d5ac895--- /dev/null+++ b/services/ingest/internal/publisher/partition_key_test.go@@ -0,0 +1,111 @@+package publisher++import (+ "testing"++ "github.com/beenuar/aisoc/services/ingest/internal/normalizer"+)++func event(tenant, id string, ocsf map[string]interface{}) *normalizer.NormalizedEvent {+ return &normalizer.NormalizedEvent{ID: id, TenantID: tenant, OcsfEvent: ocsf}+}++// The whole point: two events about one host must land on one partition,+// or fusion correlates them on two replicas and stitches a sequence out of+// order or splits one incident in two.+func TestTwoEventsAboutOneEntityShareAKey(t *testing.T) {+ first := event("t1", "e1", map[string]interface{}{+ "device": map[string]interface{}{"hostname": "WS-014"},+ })+ second := event("t1", "e2", map[string]interface{}{+ "device": map[string]interface{}{"hostname": "WS-014"},+ })+ if string(partitionKey(first)) != string(partitionKey(second)) {+ t.Fatalf("%q != %q", partitionKey(first), partitionKey(second))+ }+}++func TestDifferentEntitiesGetDifferentKeys(t *testing.T) {+ a := event("t1", "e1", map[string]interface{}{"hostname": "WS-014"})+ b := event("t1", "e2", map[string]interface{}{"hostname": "WS-015"})+ if string(partitionKey(a)) == string(partitionKey(b)) {+ t.Fatal("two hosts hashed to one key; the topic is serialised, not partitioned")+ }+}++// A tenant's events must never share a partition with another tenant's by+// accident of having the same hostname.+func TestTheSameEntityInTwoTenantsIsTwoKeys(t *testing.T) {+ a := event("t1", "e1", map[string]interface{}{"hostname": "WS-014"})+ b := event("t2", "e2", map[string]interface{}{"hostname": "WS-014"})+ if string(partitionKey(a)) == string(partitionKey(b)) {+ t.Fatal("two tenants share a partition key")+ }+}++// The precedence has to be fusion's, or an entity is routed under one name+// and correlated under another.+func TestPrecedenceMatchesTheCorrelationKey(t *testing.T) {+ cases := []struct {+ name string+ ocsf map[string]interface{}+ want string+ }{+ {+ "source ip wins over hostname",+ map[string]interface{}{+ "src_endpoint": map[string]interface{}{"ip": "198.51.100.4"},+ "device": map[string]interface{}{"hostname": "WS-014"},+ },+ "t1:198.51.100.4",+ },+ {+ "hostname wins over username",+ map[string]interface{}{+ "device": map[string]interface{}{"hostname": "WS-014"},+ "actor.user": map[string]interface{}{"name": "alice"},+ },+ "t1:WS-014",+ },+ {+ "flat connector fields are read too",+ map[string]interface{}{"src_ip": "203.0.113.9"},+ "t1:203.0.113.9",+ },+ }+ for _, tc := range cases {+ t.Run(tc.name, func(t *testing.T) {+ if got := string(partitionKey(event("t1", "e1", tc.ocsf))); got != tc.want {+ t.Fatalf("got %q, want %q", got, tc.want)+ }+ })+ }+}++// An event naming nobody must not pile onto one partition per tenant: that+// is a hot partition and also the least useful one to serialise.+func TestAnEventWithNoEntityFallsBackToItsOwnId(t *testing.T) {+ a := partitionKey(event("t1", "e1", map[string]interface{}{"message": "something happened"}))+ b := partitionKey(event("t1", "e2", map[string]interface{}{"message": "something else"}))+ if string(a) == string(b) {+ t.Fatal("two anonymous events share a key; that is a hot partition")+ }+ if string(a) != "t1:e1" {+ t.Fatalf("fallback key is %q", a)+ }+}++func TestAnEmptyEntityStringIsNotUsedAsAKey(t *testing.T) {+ // A whitespace-only hostname is not an entity, and keying on it would+ // collapse every event that carries one onto a single partition.+ got := partitionKey(event("t1", "e1", map[string]interface{}{"hostname": " "}))+ if string(got) != "t1:e1" {+ t.Fatalf("blank hostname produced key %q", got)+ }+}++func TestANilOcsfEventStillGetsAKey(t *testing.T) {+ if string(partitionKey(event("t1", "e1", nil))) != "t1:e1" {+ t.Fatal("a nil OCSF event produced no usable key")+ }+}diff --git a/services/ingest/internal/publisher/publisher.go b/services/ingest/internal/publisher/publisher.goindex d08813780..4c90abb42 100644--- a/services/ingest/internal/publisher/publisher.go+++ b/services/ingest/internal/publisher/publisher.go@@ -6,6 +6,7 @@ import ("encoding/json""fmt""net"+ "strings""time""github.com/beenuar/aisoc/services/ingest/internal/config"@@ -182,6 +183,99 @@ func (p *Publisher) Publish(ctx context.Context, event *normalizer.NormalizedEve}// PublishBatch sends multiple events in a single batch+// partitionKey decides which partition an event lands on, and therefore+// what fusion sees in order.+//+// Parity plan 6.9 / depth plan 4.4. The key used to be+// `tenant:event_id`, which is unique per event — so every event hashed to+// an arbitrary partition and nothing about one entity's events was+// ordered relative to another's. Fusion correlates on+// `{tenant}:{entity}:{tactic}` and the windowed engine stitches sequences+// by event time, so two events about one host arriving on two partitions+// and being consumed by two replicas is the case where a sequence is+// stitched out of order or split across two incidents.+//+// The key is `tenant:entity`, where entity is resolved in the same+// precedence the alert's own `correlation_key()` uses — source IP, then+// hostname, then username, then domain. Partitions are ordered, so all of+// one entity's events are ordered relative to each other; different+// entities still spread across the cluster, so this is ordering without+// serialising the topic.+//+// An event with no resolvable entity falls back to the event id. That+// keeps the spread even for telemetry that names nobody, and such an+// event has no sequence to be part of, so there is no ordering to lose.+// Falling back to the tenant id instead would put every anonymous event+// in one tenant onto one partition — a hot partition that is also the+// least useful one.+func partitionKey(event *normalizer.NormalizedEvent) []byte {+ entity := primaryEntity(event)+ if entity == "" {+ return []byte(event.TenantID + ":" + event.ID)+ }+ return []byte(event.TenantID + ":" + entity)+}++// entityFieldOrder is the precedence fusion's own correlation key uses+// (services/fusion/app/models/alert.py, `RawAlert.correlation_key`). Held+// in the same order deliberately: a partition key that disagrees with the+// correlation key routes an entity's events to a partition and then+// correlates them under a different name, which spreads one incident+// across replicas while looking correct from either end.+var entityFieldOrder = []struct {+ section string+ field string+}{+ {"src_endpoint", "ip"},+ {"device", "hostname"},+ {"device", "name"},+ {"actor.user", "name"},+ {"", "src_ip"},+ {"", "hostname"},+ {"", "host"},+ {"", "user"},+ {"", "username"},+ {"", "domain"},+}++// primaryEntity pulls the entity out of the OCSF event, checking the+// structured locations first and the flat connector fields after.+func primaryEntity(event *normalizer.NormalizedEvent) string {+ ocsf := event.OcsfEvent+ if ocsf == nil {+ return ""+ }+ for _, candidate := range entityFieldOrder {+ var value string+ if candidate.section == "" {+ value = stringAt(ocsf, candidate.field)+ } else {+ value = stringAt(ocsf, strings.Split(candidate.section+"."+candidate.field, ".")...)+ }+ if value != "" {+ return value+ }+ }+ return ""+}++// stringAt walks a dotted path through nested maps and returns a non-empty+// string leaf, or "".+func stringAt(root map[string]interface{}, path ...string) string {+ var cur interface{} = root+ for _, key := range path {+ m, ok := cur.(map[string]interface{})+ if !ok {+ return ""+ }+ cur = m[key]+ }+ if s, ok := cur.(string); ok {+ return strings.TrimSpace(s)+ }+ return ""+}+func (p *Publisher) PublishBatch(ctx context.Context, events []*normalizer.NormalizedEvent) error {if len(events) == 0 {return nil@@ -195,7 +289,7 @@ func (p *Publisher) PublishBatch(ctx context.Context, events []*normalizer.Normacontinue}msgs = append(msgs, kafka.Message{- Key: []byte(event.TenantID + ":" + event.ID),+ Key: partitionKey(event),Value: data,Headers: []kafka.Header{{Key: "tenant_id", Value: []byte(event.TenantID)},