Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
depth 4.2: a syslog listener for RFC 5424, RFC 3164, CEF and LEEF (#1225)
Beenu Arorae3a4d7ea6793
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlindex 5d826a733..38616927e 100644--- a/.github/workflows/ci.yml+++ b/.github/workflows/ci.yml@@ -649,6 +649,20 @@ jobs:run: python3 scripts/check_connector_profiles.py --self-test- name: Connector profile drift gaterun: python3 scripts/check_connector_profiles.py+ # The syslog listener is the one input path an appliance can reach on+ # its own: every other push route needs something in front of it that+ # already speaks HTTP. This asserts it parses all four wire formats,+ # is started by the service rather than merely compiled, is named by+ # /readyz, takes its tenant from a minted token and from nothing a+ # sender controls, and projects through templates that ship and can+ # be minted. Self-test first, on the same tree: it injects one+ # violation per rule — including each of the three Go syntaxes a+ # tenant could leak through, because the first version of this gate+ # knew only one of them and a runtime negative control walked past it.+ - name: Syslog listener gate — self-test+ run: python3 scripts/check_syslog_listener.py --self-test+ - name: Syslog listener gate+ run: python3 scripts/check_syslog_listener.py# Every connector above reads an API that answers a time window, so the# window bounds what a poll can return. The three cloud-native# collection paths do not: an SQS queue after an outage, a Pub/Subdiff --git a/CHANGELOG.md b/CHANGELOG.mdindex 2e13fec36..5ae5364db 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -44,6 +44,35 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0### Added+- **An appliance that speaks only syslog can now reach AiSOC on its own.**+ The one syslog path that existed, `POST /v1/inbox/cef`, requires something+ in front of it that already speaks HTTP — so the firewall, the proxy, the+ legacy IDS and the printer still could not reach AiSOC, only a forwarder+ could. `services/ingest` now accepts syslog over UDP and TCP and reads all+ four wire formats the long tail speaks: RFC 5424, RFC 3164, CEF and LEEF.++ The tenant comes from a minted ingest token in the service's own+ configuration and from nowhere else. Syslog carries no authenticated+ principal and no header, so anything read off the wire is+ sender-controlled: a tenant taken from a hostname or a structured-data+ parameter would be a cross-tenant write requiring no attacker effort at+ all. One listener therefore serves one tenant, which the setup guide says+ rather than leaving it to be discovered.++ A line matching none of the four formats is kept as an unstructured+ message with its text intact, never reported as a structured record with+ invented fields — a mis-split line puts half the message in the hostname+ column and an analyst reading that record sees a host that does not+ exist. Plain syslog lands at OCSF class 1007 in category 1 rather than at+ 2001: category 2 is promoted unconditionally, and filing a firewall's+ routine daemon chatter as a Security Finding would turn every line into an+ alert. CEF and LEEF do land at 2001, because a device emitting one of+ those has already decided the line is a security event.++ `/readyz` names the listener, what it bound, how many messages it has+ received and what it last failed on. A receiver is legitimately silent for+ hours, so a listener that failed to bind and one nobody is sending to are+ indistinguishable unless readiness says which.- **Cloud telemetry collected the way an estate actually emits it.** Everycloud connector before this read an API that answers a time window, whichis the small-account shape: `cloudtrail:LookupEvents` is capped at twodiff --git a/DEPTH_PROGRESS.md b/DEPTH_PROGRESS.mdindex f7531a588..158830c01 100644--- a/DEPTH_PROGRESS.md+++ b/DEPTH_PROGRESS.md@@ -379,6 +379,18 @@ than absent.## Phase 4: Collection and scale+- [ ] **4.1** Cloud-native collection (S3+SQS org trail, Pub/Sub, Event Hubs)+- [~] **4.2** Standard inputs (syslog, OTLP, Kafka, TAXII 2.1). **One of+ four bullets closed.** The syslog listener is done: `services/ingest`+ accepts UDP and TCP and reads RFC 5424, RFC 3164, CEF and LEEF, with the+ tenant taken from a minted token because syslog carries nothing a sender+ does not control. Gated by `scripts/check_syslog_listener.py` (self-test:+ one injected violation per rule, including each of the three Go syntaxes+ a tenant could leak through) wired into `ci.yml :: python-lint`. 33 new+ Go tests. **Three bullets are open and were not attempted**, named here+ rather than left to be inferred: an OTLP logs receiver, a Kafka input for+ customer topics, and a TAXII 2.1 server over the tenant IOC store. Each+ reproduces trivially — nothing in the tree implements any of them.- [x] **4.1** Cloud-native collection (S3+SQS org trail, Pub/Sub, Event Hubs).Three connectors (`aws_cloudtrail_s3`, `gcp_pubsub`, `azure_event_hubs`),each resumable and each declaring a bounded `collection_budget` that says@@ -446,6 +458,7 @@ than absent.|---|---|---|| 2026-10-07 | 0.1 | This file created at base commit `1b8bc2d4`. || 2026-10-07 | 0.2 | Every figure re-derived. Two matched exactly (detections, unreachable families); executor arms measured 74 against a captured 73; the cloud/identity/SaaS/code figure measured 395 against a captured 461 on a grouping the plan does not pin, recorded above. Two measurement caveats found: executable and quarantined overlap by 1,724 rules, and the "69% Windows" figure does not reproduce from the index. |+| 2026-10-09 | 4.2 | Reproduced: `check_syslog_listener.py` against `origin/main` reported "no Go sources under services/ingest/internal/syslog", and the only syslog path was `POST /v1/inbox/cef`, which needs a forwarder that already speaks HTTP. Implemented `services/ingest/internal/syslog` (UDP + TCP, both RFC 6587 framings, four wire formats, token-derived tenant, batched publish, `/readyz` subscription). Five negative controls recorded in the PR; one of them found a hole in the new gate itself, which reported OK while the tenant came from `msg.Hostname` through a struct field — the detector now knows all three Go syntaxes and the self-test exercises each. Writing the tests also found a shutdown deadlock: `Stop` waited on goroutines watching a context the caller had not cancelled yet. Ingest suite green, `internal/syslog` 33 cases. || 2026-10-07 | 0.3 | `make up` and `make smoke` (10/10) pass. Injection suite and load-harness baselines committed. `make up-full` deferred (D2) and hosted model rows blocked (D3). Fixing D1 was a precondition for the load-harness baseline. || 2026-10-09 | 3.3 | Identity privilege (18 rules) and a per-tenant first-seen store (2) built; 5 rules were never unreachable (gate operator drift, D9); 18 retired with a reason (D10). `MAX_UNREACHABLE` 45 → 2, the two remaining both needing the 3.4 baseline. Two matcher operators three shipped rules already used were implemented. || 2026-10-09 | 3.2 | Ordered and unordered sequences in the windowed engine; all four translatable Sigma correlation types compile, seven refusal reasons recorded. No upstream correlation rule exists in this tree to import (D8), so the corpus is first-party and labelled as such. |diff --git a/apps/docs/docs/connectors/syslog-listener.md b/apps/docs/docs/connectors/syslog-listener.mdnew file mode 100644index 000000000..22f4c2284--- /dev/null+++ b/apps/docs/docs/connectors/syslog-listener.md@@ -0,0 +1,137 @@+---+id: syslog-listener+title: Syslog listener (RFC 5424, RFC 3164, CEF, LEEF)+sidebar_label: Syslog listener+---++Firewalls, proxies, load balancers, legacy IDS and a great deal of custom+gear emit syslog and will never grow a webhook. The ingest service accepts+syslog directly over UDP and TCP, so the appliance reaches AiSOC itself+rather than needing a forwarder in front of it.++Four wire formats are read:++| Format | What it is | How it is recognised |+|---|---|---|+| **RFC 5424** | The current standard: version digit, RFC 3339 timestamp, structured data | the `1 ` version digit after the priority |+| **RFC 3164** | The one most gear emits; no year, no zone, optional hostname | the three-letter month |+| **CEF** | ArcSight key/value, carried inside a syslog line | a `CEF:` anchor |+| **LEEF** | IBM QRadar key/value, with a delimiter the header may redefine | a `LEEF:` anchor |++A line matching none of them is kept as an unstructured message with its+text intact. It is never reported as a structured record with invented+fields — a mis-split line puts half the message in the hostname column, and+an analyst reading that record sees a host that does not exist.++## One listener, one tenant++The tenant comes from a minted ingest token you put in the service's+configuration, and from nowhere else.++That is not a limitation to work around, it is the point. Syslog carries no+authenticated principal and no header, so **anything read off the wire is+sender-controlled**: a tenant taken from the hostname, the application name+or a structured-data parameter would be a cross-tenant write that needs no+attacker effort at all. A multi-tenant deployment runs one listener port per+tenant.++## Setting it up++**1. Mint a token.** Use the `syslog` template:++```bash+curl -sS -X POST https://your-aisoc/api/v1/inbox/tokens \+ -H "Authorization: Bearer $AISOC_API_TOKEN" \+ -H 'Content-Type: application/json' \+ -d '{"template_id": "syslog", "label": "edge firewalls"}'+```++The response carries the token once. One token covers all four formats: a+single appliance commonly emits RFC 3164 for its daemons and CEF for its+security events, and making you mint two tokens and run two ports for one+device would be a configuration no vendor documents.++**2. Turn the listener on.** In the ingest service's environment:++```bash+AISOC_SYSLOG_ENABLED=true+AISOC_SYSLOG_TOKEN=<the token from step 1>+AISOC_SYSLOG_UDP_ADDR=0.0.0.0:5514+AISOC_SYSLOG_TCP_ADDR=0.0.0.0:5514+```++Port **5514**, not 514, is the default on purpose: 514 is privileged, so+binding it needs either `CAP_NET_BIND_SERVICE` on the container or a+published port mapping `514` to `5514` on the host. Publish the UDP and TCP+ports in your compose file or Service, and remember that a UDP port+published by Docker needs `/udp` spelled out.++Set `AISOC_SYSLOG_UDP_ADDR` or `AISOC_SYSLOG_TCP_ADDR` to an empty string to+run only one transport. Setting both to empty is refused rather than+accepted as a no-op: a listener bound to nothing looks identical to a+listener nobody is sending to.++**3. Point the device at it.** On most Linux senders:++```+*.* @@aisoc-ingest.internal:5514 # TCP+*.* @aisoc-ingest.internal:5514 # UDP+```++**4. Check `/readyz`.** The listener appears by name under+`subscriptions`, with what it bound, how many messages it has received and+what it last failed on:++```json+{+ "subscriptions": {+ "syslog": {+ "attached": true,+ "not_resolving": false,+ "detail": "listening on udp 0.0.0.0:5514, tcp 0.0.0.0:5514; received 1842, rejected 0"+ }+ }+}+```++A receiver is legitimately silent for hours, so "no messages" proves nothing+either way — which is why readiness reports the counters rather than only a+boolean.++## What happens to a message++| Parsed as | Template | OCSF class |+|---|---|---|+| RFC 5424, RFC 3164, unstructured | `syslog` | 1007 Process Activity (category 1) |+| CEF | `cef-syslog` | 2001 Security Finding (category 2) |+| LEEF | `leef-syslog` | 2001 Security Finding (category 2) |++The split matters. Category 2 is promoted to an alert unconditionally, so+filing a firewall's routine daemon chatter as a Security Finding would turn+every line into an alert — exactly the noise the platform exists to remove.+A plain syslog line becomes an alert when a detection rule fires on it. A+CEF or LEEF line is promoted because a device emitting one of those has+already decided the line is a security event.++The original line is kept verbatim on every event as `raw_data`. Every+parse here is a lossy summary, and an appliance emitting something the+parser misreads is itself the finding.++## Limits worth knowing before you rely on it++- **TCP framing**: both of RFC 6587's framings are read — octet-counting+ (`123 <34>1 ...`) and newline-delimited. Octet-counting is the one that+ survives a message containing a newline; prefer it if your sender offers+ it.+- **A failed publish loses the message.** Syslog has no acknowledgement to+ withhold, so if Kafka is unreachable the batch is dropped, counted on+ `aisoc_ingest_syslog_rejected_total` and named in the `/readyz` detail.+ Nothing can make that lossless at this layer; use the HTTP inbox routes+ for a source where loss is unacceptable.+- **RFC 3164 timestamps carry no year.** The receiver's year is filled in,+ with a day of slack so a line arriving just after midnight on 1 January is+ dated to the year it was written rather than to the next one. The original+ text stays in `raw_data`.+- **Connections are capped** (`AISOC_SYSLOG_MAX_CONNECTIONS`, default 256)+ and a connection over the cap is refused immediately rather than held+ open, so a sender finds out instead of believing it is delivering.diff --git a/apps/docs/sidebars.ts b/apps/docs/sidebars.tsindex d2d3fd6c7..f271c728c 100644--- a/apps/docs/sidebars.ts+++ b/apps/docs/sidebars.ts@@ -84,6 +84,7 @@ const sidebars: SidebarsConfig = {"connectors/endpoint-decision-matrix","connectors/api-coverage","connectors/universal-capture",+ "connectors/syslog-listener","connectors/osquery-extensions","connectors/abnormal_security","connectors/ai-gateway",diff --git a/scripts/check_syslog_listener.py b/scripts/check_syslog_listener.pynew file mode 100644index 000000000..ef266e140--- /dev/null+++ b/scripts/check_syslog_listener.py@@ -0,0 +1,341 @@+#!/usr/bin/env python3+"""The syslog listener has to be reachable, wired, and tenant-safe.++Depth plan 4.2. Before this, AiSOC's only syslog path was+``POST /v1/inbox/cef``, which needs something in front of it that already+speaks HTTP — so the appliance could not reach AiSOC, only a forwarder+could. The listener closes that, and three of its properties are the kind+this repository keeps rediscovering the absence of:++**It is wired.** A package that parses four formats perfectly and is+constructed by nothing is indistinguishable from a feature until someone+traces the call graph. ``main.go`` has to construct it *and* start it.++**Readiness names it.** A receiver is legitimately silent for hours, so+"no messages" proves nothing either way — a listener that failed to bind+and one nobody is sending to look identical unless readiness says which.++**The tenant comes from a credential.** Syslog carries no authenticated+principal and no header, so anything read off the wire is sender-+controlled. A tenant taken from a hostname, an app name or a structured-+data parameter is a cross-tenant write that needs no attacker effort at+all. The sink must read it from the resolved token and from nowhere else.++Plus the mechanical half: all four formats are actually distinguished, and+both templates the listener projects through exist and are mintable.++Parsed with ``ast`` and plain text rather than by running Go, because this+runs in the lint job where no Go toolchain is installed — and a gate that+skips when it cannot find a compiler is a gate that silently stops+checking.++Usage:+ python3 scripts/check_syslog_listener.py+ python3 scripts/check_syslog_listener.py --self-test+"""++from __future__ import annotations++import argparse+import ast+import re+import shutil+import sys+import tempfile+from pathlib import Path++sys.path.insert(0, str(Path(__file__).resolve().parent))++from gate_toolkit import TREE_SHAPES, refuses_an_empty_tree, repo_root # noqa: E402++REPO_ROOT = repo_root()++SYSLOG_PKG = Path("services/ingest/internal/syslog")+MAIN_GO = Path("services/ingest/main.go")+TEMPLATE_DIR = Path("services/ingest/internal/normalizer/templates")+INBOX_ENDPOINT = Path("services/api/app/api/v1/endpoints/inbox.py")++#: The four the plan names, with the constant each is declared as. A parser+#: that collapses two of them into one is the defect: a LEEF 2.0 line read+#: with CEF's rules yields a record that looks structurally fine and has+#: every attribute in one key.+REQUIRED_FORMATS: dict[str, str] = {+ "RFC 5424": "FormatRFC5424",+ "RFC 3164": "FormatRFC3164",+ "CEF": "FormatCEF",+ "LEEF": "FormatLEEF",+}++#: Templates the listener projects through. Named here rather than derived,+#: because deriving them from the sink would make the gate agree with+#: whatever the sink happens to say.+REQUIRED_TEMPLATES = ("syslog", "cef-syslog", "leef-syslog")++#: Fields a sender controls. None of them may reach the tenant.+SENDER_CONTROLLED = ("Hostname", "AppName", "ProcID", "MsgID", "Message", "StructuredData", "Extension", "Raw")+++def _read(root: Path, rel: Path) -> str:+ path = root / rel+ try:+ return path.read_text(encoding="utf-8")+ except OSError:+ return ""+++def _go_sources(root: Path) -> str:+ """Every non-test Go file in the syslog package, concatenated."""+ directory = root / SYSLOG_PKG+ if not directory.is_dir():+ return ""+ parts = [p.read_text(encoding="utf-8", errors="replace") for p in sorted(directory.glob("*.go")) if not p.name.endswith("_test.go")]+ return "\n".join(parts)+++def _strip_go_comments(source: str) -> str:+ """Comments out, so a gate does not match its own explanation.++ The listener's own comments quote the field names this gate forbids —+ they say *why* the tenant does not come from them — and a checker that+ cannot tell code from prose about the code fires on the documentation.+ """+ source = re.sub(r"/\*.*?\*/", "", source, flags=re.S)+ return "\n".join(re.sub(r"//.*$", "", line) for line in source.splitlines())+++def _mintable_ids(root: Path) -> set[str]:+ source = _read(root, INBOX_ENDPOINT)+ if not source:+ return set()+ try:+ tree = ast.parse(source)+ except SyntaxError:+ return set()+ for node in ast.walk(tree):+ if isinstance(node, ast.Assign):+ targets: list[ast.expr] = list(node.targets)+ elif isinstance(node, ast.AnnAssign):+ targets = [node.target]+ else:+ continue+ for target in targets:+ if isinstance(target, ast.Name) and target.id == "ALLOWED_TEMPLATE_IDS" and isinstance(node.value, ast.Tuple):+ return {e.value for e in node.value.elts if isinstance(e, ast.Constant) and isinstance(e.value, str)}+ return set()+++def evaluate(root: Path) -> list[str]:+ failures: list[str] = []++ sources = _go_sources(root)+ if not sources.strip():+ failures.append(+ f"no Go sources under {SYSLOG_PKG}. The only syslog path is then POST /v1/inbox/cef, which needs a "+ f"forwarder that already speaks HTTP — so an appliance cannot reach AiSOC, only something in front of it can."+ )+ return failures++ code = _strip_go_comments(sources)++ # 1. All four formats are distinguished. The constant existing is not+ # enough — it is declared in one file and assigned in another, and a+ # parser that stopped *reaching* a branch still mentions its name.+ for label, constant in REQUIRED_FORMATS.items():+ if not re.search(rf"Format\s*=\s*{constant}\b", code):+ failures.append(+ f"{label} is never assigned ({constant} appears in no `Format = ...` in {SYSLOG_PKG}). "+ f"A parser that collapses two of the four formats produces records that look structurally "+ f"fine and are wrong in every field."+ )++ # 2. Wired into the service, constructed *and* started.+ main_go = _strip_go_comments(_read(root, MAIN_GO))+ if not main_go:+ failures.append(f"{MAIN_GO} is not readable, so nothing can be said about whether the listener runs.")+ else:+ if "syslog.New(" not in main_go:+ failures.append(+ f"{MAIN_GO} never calls syslog.New. A package that parses four formats perfectly and is "+ f"constructed by nothing is indistinguishable from a feature until someone traces the call graph."+ )+ if not re.search(r"syslogListener\.Start\(", main_go):+ failures.append(f"{MAIN_GO} constructs the listener but never starts it, so no socket is ever read.")+ if not re.search(r"RegisterSubscription\(\s*\"syslog\"", main_go):+ failures.append(+ f"{MAIN_GO} does not register the listener with /readyz. A receiver is legitimately silent for "+ f"hours, so a listener that failed to bind and one nobody is sending to are indistinguishable "+ f"unless readiness names it."+ )++ # 3. The tenant comes from the credential.+ sink = _strip_go_comments(_read(root, SYSLOG_PKG / "sink.go"))+ if not sink:+ failures.append(f"{SYSLOG_PKG}/sink.go is missing, so there is nothing resolving a tenant for the listener.")+ else:+ if "tok.TenantID" not in sink:+ failures.append(+ f"{SYSLOG_PKG}/sink.go does not read the tenant from the resolved token. Syslog carries no "+ f"authenticated principal and no header, so the token is the only thing a sender does not control."+ )+ # Three syntaxes, because Go offers three and a detector that knows+ # one reports OK over the other two. This gate shipped knowing only+ # the first: a runtime negative control that moved the tenant to+ # `msg.Hostname` through a struct field and a call argument passed+ # it cleanly while the published event carried the sender's value.+ tenant_sinks = (+ # tenantID := msg.Hostname / tenantID = msg.Hostname+ r"[Tt]enant\w*\s*:?=\s*[^\n]*\bmsg\.{field}\b",+ # TenantID: msg.Hostname (a struct literal field)+ r"TenantID\s*:\s*[^\n,]*\bmsg\.{field}\b",+ # tmpl.Apply(fields, msg.Hostname, ...) — the tenant is Apply's+ # second argument, so anything derived from the message there is+ # the tenant whatever the local variable is called.+ r"\.Apply\([^\n]*\bmsg\.{field}\b",+ )+ for field in SENDER_CONTROLLED:+ for pattern in tenant_sinks:+ if re.search(pattern.format(field=field), sink):+ failures.append(+ f"{SYSLOG_PKG}/sink.go derives the tenant from msg.{field}, which the sender controls. "+ f"That is a cross-tenant write requiring no attacker effort at all."+ )+ break++ # 4. The templates it projects through exist and are mintable.+ mintable = _mintable_ids(root)+ for template in REQUIRED_TEMPLATES:+ if not (root / TEMPLATE_DIR / f"{template}.yaml").is_file():+ failures.append(+ f"{TEMPLATE_DIR}/{template}.yaml is missing, so messages of that format cannot be normalised "+ f"and the listener refuses the whole batch rather than filing them under the wrong vendor."+ )+ if mintable and template not in mintable:+ failures.append(+ f"'{template}' is not in ALLOWED_TEMPLATE_IDS ({INBOX_ENDPOINT}), so no token can be minted for it "+ f"and an operator following the setup guide gets a 400."+ )++ return failures+++def _self_test() -> int:+ results: list[tuple[str, bool]] = []++ baseline = evaluate(REPO_ROOT)+ results.append(("the undisturbed tree is clean, so a caught case means something", not baseline))+ for line in baseline[:6]:+ print(f" {line}")++ cases = (+ ("a parser that stopped distinguishing LEEF from CEF", "FORMAT"),+ ("a listener nothing constructs", "UNWIRED"),+ ("a listener constructed but never started", "UNSTARTED"),+ ("a listener readiness does not name", "UNPROBED"),+ ("a tenant assigned from a sender-controlled field", "TENANT_ASSIGN"),+ ("a tenant set through a struct literal field", "TENANT_STRUCT"),+ ("a tenant passed straight into the template's tenant argument", "TENANT_ARG"),+ ("a template the listener projects through that nothing ships", "TEMPLATE"),+ ("a template no token can be minted for", "UNMINTABLE"),+ )++ for label, kind in cases:+ tmp = Path(tempfile.mkdtemp(prefix="aisoc-syslog-selftest-"))+ try:+ for rel in (SYSLOG_PKG, TEMPLATE_DIR, MAIN_GO.parent, INBOX_ENDPOINT.parent):+ src = REPO_ROOT / rel+ if src.is_dir():+ shutil.copytree(src, tmp / rel, dirs_exist_ok=True)+ kv = tmp / SYSLOG_PKG / "kv.go"+ main_go = tmp / MAIN_GO+ sink = tmp / SYSLOG_PKG / "sink.go"++ if kind == "FORMAT" and kv.is_file():+ kv.write_text(kv.read_text(encoding="utf-8").replace("FormatLEEF", "FormatCEF"), encoding="utf-8")+ elif kind == "UNWIRED" and main_go.is_file():+ main_go.write_text(main_go.read_text(encoding="utf-8").replace("syslog.New(", "syslogDisabled("), encoding="utf-8")+ elif kind == "UNSTARTED" and main_go.is_file():+ main_go.write_text(+ main_go.read_text(encoding="utf-8").replace("syslogListener.Start(ctx)", "_ = syslogListener"), encoding="utf-8"+ )+ elif kind == "UNPROBED" and main_go.is_file():+ main_go.write_text(+ main_go.read_text(encoding="utf-8").replace('h.RegisterSubscription("syslog"', 'noop("syslog"'),+ encoding="utf-8",+ )+ elif kind == "TENANT_ASSIGN" and sink.is_file():+ sink.write_text(+ sink.read_text(encoding="utf-8").replace("tenantID := tok.TenantID.String()", "tenantID := msg.Hostname"),+ encoding="utf-8",+ )+ elif kind == "TENANT_STRUCT" and sink.is_file():+ sink.write_text(+ sink.read_text(encoding="utf-8").replace("TenantID: tenantID,", "TenantID: msg.Hostname,"),+ encoding="utf-8",+ )+ elif kind == "TENANT_ARG" and sink.is_file():+ sink.write_text(+ sink.read_text(encoding="utf-8").replace(+ "tmpl.Apply(msg.Fields(), tenantID, connectorRef, receivedAt)",+ "tmpl.Apply(msg.Fields(), msg.Hostname, connectorRef, receivedAt)",+ ),+ encoding="utf-8",+ )+ elif kind == "TEMPLATE":+ (tmp / TEMPLATE_DIR / "leef-syslog.yaml").unlink(missing_ok=True)+ elif kind == "UNMINTABLE":+ endpoint = tmp / INBOX_ENDPOINT+ endpoint.write_text(endpoint.read_text(encoding="utf-8").replace(' "leef-syslog",\n', "", 1), encoding="utf-8")++ results.append((f"catches {label}", bool(evaluate(tmp))))+ finally:+ shutil.rmtree(tmp, ignore_errors=True)++ details: list[str] = []+ for shape in TREE_SHAPES:+ refused, detail = refuses_an_empty_tree(Path(__file__).name, shape=shape)+ results.append((f"refuses a {shape} tree rather than reporting it clean", refused))+ details.append(f"{shape}: {detail}")++ ok = True+ for description, passed in results:+ ok &= passed+ print(f" {'PASS' if passed else 'FAIL'} {description}")+ for line in "\n".join(details).splitlines():+ print(f" {line}")+ print()+ if not ok:+ print(f"{Path(__file__).name}: self-test FAILED")+ return 1+ print(f"{Path(__file__).name}: self-test OK")+ return 0+++def main() -> int:+ parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)+ parser.add_argument("--self-test", action="store_true", help="inject one violation per rule and require each to be caught")+ args = parser.parse_args()+ if args.self_test:+ return _self_test()++ if not (REPO_ROOT / MAIN_GO).is_file():+ print(f"check_syslog_listener: {MAIN_GO} is not there — nothing to render a verdict about.", file=sys.stderr)+ return 2++ failures = evaluate(REPO_ROOT)+ if failures:+ print("The syslog listener (depth plan 4.2) is incomplete:\n", file=sys.stderr)+ for line in failures:+ print(f" - {line}", file=sys.stderr)+ print(f"\n{len(failures)} problem(s).", file=sys.stderr)+ return 1++ print(+ f"OK — the syslog listener reads {len(REQUIRED_FORMATS)} wire formats, is started by the service, "+ f"is named by /readyz, takes its tenant from a minted token, and projects through "+ f"{len(REQUIRED_TEMPLATES)} shipped templates."+ )+ return 0+++if __name__ == "__main__":+ raise SystemExit(main())diff --git a/scripts/generate_connector_docs.py b/scripts/generate_connector_docs.pyindex 0abd79c7a..315175c60 100644--- a/scripts/generate_connector_docs.py+++ b/scripts/generate_connector_docs.py@@ -411,6 +411,12 @@ SIDEBAR_PREAMBLE = ("connectors/endpoint-decision-matrix","connectors/api-coverage","connectors/universal-capture",+ # A transport rather than a connector: the syslog listener in+ # services/ingest serves every appliance that speaks RFC 5424, RFC 3164,+ # CEF or LEEF, so there is no schema() for the generated list below to+ # pick up and the page has to be named here or it drops out of+ # navigation on the next regeneration.+ "connectors/syslog-listener",# Not a connector, so the generated list below does not cover it."connectors/osquery-extensions",)diff --git a/services/api/app/api/v1/endpoints/inbox.py b/services/api/app/api/v1/endpoints/inbox.pyindex 6f8bdb048..4a9001824 100644--- a/services/api/app/api/v1/endpoints/inbox.py+++ b/services/api/app/api/v1/endpoints/inbox.py@@ -76,6 +76,15 @@ ALLOWED_TEMPLATE_IDS: tuple[str, ...] = ("cloudflare-logpush",# Universal-capture sidecars"cef-syslog",+ # Depth plan 4.2 — the syslog listener in services/ingest. One token+ # covers all four wire formats: a single appliance commonly emits+ # RFC 3164 for its daemons and CEF for its security events, and making+ # the operator mint two tokens and run two ports for one device would+ # be a configuration no vendor documents. The listener picks the+ # template per message; `leef-syslog` is reachable through it and is+ # mintable in its own right for a forwarder that posts over HTTP.+ "syslog",+ "leef-syslog","splunk-hec","email-forwarded",# AI estate. Both templates shipped with the feature and both are what@@ -292,6 +301,20 @@ _TEMPLATE_CATALOG: dict[str, dict[str, str]] = {"description": "ArcSight Common Event Format over HTTPS.","category": "siem",},+ "syslog": {+ "label": "Syslog (RFC 5424 / RFC 3164)",+ "description": (+ "Mint this for the syslog listener in the ingest service, which accepts UDP and TCP on a port you "+ "choose and reads RFC 5424, RFC 3164, CEF and LEEF. The token is where the tenant comes from: syslog "+ "carries no header and no authenticated principal, so one listener serves one tenant."+ ),+ "category": "siem",+ },+ "leef-syslog": {+ "label": "Syslog / LEEF",+ "description": "IBM QRadar Log Event Extended Format, over the syslog listener or posted over HTTPS.",+ "category": "siem",+ },"splunk-hec": {"label": "Splunk HEC-compatible","description": (diff --git a/services/ingest/internal/config/config.go b/services/ingest/internal/config/config.goindex 994df4ac4..6143bfad0 100644--- a/services/ingest/internal/config/config.go+++ b/services/ingest/internal/config/config.go@@ -69,6 +69,25 @@ type Config struct {VulnKafkaTopic string // topic for VULNERABILITY_MATCH eventsNvdAPIKey string // optional NVD API key for higher rate limits+ // Depth plan 4.2 — the syslog listener.+ //+ // Off by default: binding a port that nothing sends to is harmless but+ // misleading, and 514 in particular needs either a capability or a+ // published container port, which an operator has to decide.+ //+ // SyslogToken is a minted ingest token, and it is the *only* place the+ // tenant can come from. Syslog carries no authenticated principal and+ // no header, so anything read off the wire is sender-controlled — a+ // tenant taken from a syslog field would be a cross-tenant write with+ // no attacker effort at all. One listener therefore serves one tenant.+ SyslogEnabled bool+ SyslogUDPAddr string+ SyslogTCPAddr string+ SyslogToken string+ SyslogBatchSize int+ SyslogFlushMs int+ SyslogMaxConnections int+// Workstream 6 — universal capture push paths.// InboxEnabled toggles the /v1/inbox/* routes. Off by default in// development if no DATABASE_DSN is set, since the inbox store needs@@ -220,9 +239,16 @@ func Load() (*Config, error) {NvdAPIKey: getEnv("NVD_API_KEY", ""),// Universal capture (Workstream 6).- InboxEnabled: getEnv("INBOX_ENABLED", "true") == "true",- InboxTemplatesDir: getEnv("INBOX_TEMPLATES_DIR", "/app/templates"),- InboxMaxBodyBytes: int64(mustGetEnvInt("INBOX_MAX_BODY_BYTES", 10*1024*1024)),+ SyslogEnabled: getEnv("AISOC_SYSLOG_ENABLED", "false") == "true",+ SyslogUDPAddr: getEnv("AISOC_SYSLOG_UDP_ADDR", "0.0.0.0:5514"),+ SyslogTCPAddr: getEnv("AISOC_SYSLOG_TCP_ADDR", "0.0.0.0:5514"),+ SyslogToken: getEnv("AISOC_SYSLOG_TOKEN", ""),+ SyslogBatchSize: mustGetEnvInt("AISOC_SYSLOG_BATCH_SIZE", 500),+ SyslogFlushMs: mustGetEnvInt("AISOC_SYSLOG_FLUSH_MS", 2000),+ SyslogMaxConnections: mustGetEnvInt("AISOC_SYSLOG_MAX_CONNECTIONS", 256),+ InboxEnabled: getEnv("INBOX_ENABLED", "true") == "true",+ InboxTemplatesDir: getEnv("INBOX_TEMPLATES_DIR", "/app/templates"),+ InboxMaxBodyBytes: int64(mustGetEnvInt("INBOX_MAX_BODY_BYTES", 10*1024*1024)),// Defaults sized to be invisible to a well-behaved vendor and// firmly bounding to a runaway one: 50 req/s sustained with a// 200 burst, and 5k events/s with a 20k burst (one large batchdiff --git a/services/ingest/internal/normalizer/templates/leef-syslog.yaml b/services/ingest/internal/normalizer/templates/leef-syslog.yamlnew file mode 100644index 000000000..2cf018822--- /dev/null+++ b/services/ingest/internal/normalizer/templates/leef-syslog.yaml@@ -0,0 +1,61 @@+# Log Event Extended Format (LEEF) over syslog.+#+# QRadar's answer to the same problem CEF solves, and different from it in+# the one way that matters to a parser: LEEF 2.0 lets the header redefine+# the attribute delimiter, so a caret-delimited line read with the default+# tab lands every pair in a single key. The listener's parser reads that+# field; this template only sees the result.+#+# LEEF:Version|Vendor|Product|Version|EventID|key=value<TAB>key=value+# LEEF:2.0|Vendor|Product|Version|EventID|<delimiter>|key=value...+#+# Reference:+# https://www.ibm.com/docs/en/dsm?topic=overview-leef-event-components+id: leef-syslog+vendor_name: LEEF+product_name: LEEF Syslog+class_uid: 2001+class_name: Security Finding+activity_id: 1+field_map:+ DeviceVendor: metadata.product.vendor_name+ DeviceProduct: metadata.product.name+ DeviceVersion: metadata.product.version+ EventID: finding.uid+ cat: finding.types+ src: src_endpoint.ip+ dst: dst_endpoint.ip+ srcPort: src_endpoint.port+ dstPort: dst_endpoint.port+ srcPostNATPort: src_endpoint.svc_name+ usrName: actor.user.name+ identSrc: src_endpoint.hostname+ identHostName: device.hostname+ proto: connection_info.protocol_name+ action: activity_name+ msg: message+ policy: finding.title+ raw: raw_data+severity_field: sev+# LEEF's `sev` is 1-10, where 1 is least severe — the same scale CEF uses+# and the opposite of syslog's. Mapped to the five OCSF tiers the rest of+# the platform speaks.+severity_map:+ "1": 1+ "2": 1+ "3": 2+ "4": 2+ "5": 3+ "6": 3+ "7": 4+ "8": 4+ "9": 5+ "10": 6+ Low: 2+ Medium: 3+ High: 4+ Critical: 6+time_field: devTime+message_field: msg+constants:+ source_type: leef-syslogdiff --git a/services/ingest/internal/normalizer/templates/syslog.yaml b/services/ingest/internal/normalizer/templates/syslog.yamlnew file mode 100644index 000000000..a011d0b4f--- /dev/null+++ b/services/ingest/internal/normalizer/templates/syslog.yaml@@ -0,0 +1,58 @@+# RFC 5424 / RFC 3164 syslog.+#+# Used by the syslog listener (UDP and TCP) in services/ingest for lines+# that carry no vendor key/value payload. CEF and LEEF lines are projected+# through cef-syslog.yaml and leef-syslog.yaml instead, because those carry+# a vendor, a product and a severity of their own and projecting them+# through this template would throw all three away.+#+# Class 1007 Process Activity, in category 1 (System Activity). A bare+# syslog line is a daemon saying something about itself, and nothing more+# specific is honest without knowing the daemon — the per-event catalogue+# is what reaches a narrower class later.+#+# It is deliberately NOT 2001 Security Finding, which is what every other+# push template here uses. The promoter promotes category 2+# unconditionally, so filing a firewall's routine daemon chatter as a+# finding would turn every line into an alert — precisely the noise this+# platform exists to remove. A detection rule is what turns one of these+# into an alert. CEF and LEEF lines do land at 2001, because a device+# emitting those has already decided the line is a security event.+#+# References:+# https://datatracker.ietf.org/doc/html/rfc5424+# https://datatracker.ietf.org/doc/html/rfc3164+id: syslog+vendor_name: Syslog+product_name: Syslog+class_uid: 1007+class_name: Process Activity+activity_id: 1+field_map:+ hostname: device.hostname+ app_name: actor.process.name+ proc_id: actor.process.pid+ msg_id: metadata.event_code+ message: message+ facility: metadata.labels.facility+ format: metadata.labels.syslog_format+ raw: raw_data+severity_field: severity+# The syslog ladder runs most-severe-first and OCSF's runs the other way,+# so this is a reversal rather than a rescale. emerg/alert/crit all reach+# OCSF 5 (Critical): collapsing a device's own top three tiers into one is+# a loss, but into `high` would be a second one, and 5 is the tier each of+# them means.+severity_map:+ emerg: 6+ alert: 5+ crit: 5+ err: 4+ warning: 3+ notice: 2+ info: 1+ debug: 1+time_field: timestamp+message_field: message+constants:+ source_type: syslogdiff --git a/services/ingest/internal/syslog/kv.go b/services/ingest/internal/syslog/kv.gonew file mode 100644index 000000000..cd9690c75--- /dev/null+++ b/services/ingest/internal/syslog/kv.go@@ -0,0 +1,199 @@+package syslog++import (+ "strconv"+ "strings"+)++// CEF and LEEF are two vendors' answers to the same problem — carrying+// key/value pairs inside a syslog message — and they differ in exactly the+// way that makes one parser for both wrong: CEF separates pairs with+// spaces and escapes `=` inside values, while LEEF separates them with a+// delimiter the header may *redefine*, most often a tab.+//+// `inbox/cef.go` already parses CEF for the HTTP route. This is a second+// implementation only in the sense that it reads the same format; it is+// here because the listener has no HTTP request to hang the inbox handler+// off, and the two are pinned against each other by a test rather than+// left to drift.++const cefHeaderFields = 7++// parseCEF fills msg.Extension from a CEF payload and sets the format.+//+// CEF:Version|DeviceVendor|DeviceProduct|DeviceVersion|SignatureID|Name|Severity|Extension+//+// A pipe inside a header field is escaped as `\|`, which a plain Split+// would cut on — so the split honours the escape.+func parseCEF(msg *Message, payload string) {+ idx := strings.Index(payload, "CEF:")+ if idx < 0 {+ return+ }+ body := payload[idx+len("CEF:"):]+ fields := splitEscaped(body, '|', cefHeaderFields)+ if len(fields) < cefHeaderFields {+ // Not a CEF line after all. Left as whatever the syslog header+ // parse made of it rather than reported as CEF with empty fields,+ // because a record labelled CEF with no vendor reads as a product+ // bug rather than as a malformed line.+ return+ }+ msg.Format = FormatCEF+ for key, value := range map[string]string{+ "CEFVersion": fields[0],+ "DeviceVendor": unescapeCEF(fields[1]),+ "DeviceProduct": unescapeCEF(fields[2]),+ "DeviceVersion": unescapeCEF(fields[3]),+ "SignatureID": unescapeCEF(fields[4]),+ "Name": unescapeCEF(fields[5]),+ "Severity": unescapeCEF(fields[6]),+ } {+ msg.Extension[key] = value+ }+ if len(fields) > cefHeaderFields {+ for key, value := range parseCEFExtension(fields[cefHeaderFields]) {+ msg.Extension[key] = value+ }+ }+ if name := msg.Extension["Name"]; name != "" && msg.Message == payload {+ msg.Message = name+ }+ if text := msg.Extension["msg"]; text != "" {+ msg.Message = text+ }+}++// parseCEFExtension reads the `k=v k=v` tail.+//+// The hard part is that values may contain spaces: `msg=connection refused+// src=10.0.0.1`. A split on spaces loses the message. So the scan finds the+// *next* unescaped `=`, walks back to the space before its key, and ends+// the previous value there — which is the only way to read the format+// without a schema.+func parseCEFExtension(ext string) map[string]string {+ out := map[string]string{}+ if strings.TrimSpace(ext) == "" {+ return out+ }++ type span struct {+ key string+ start int+ }+ var spans []span+ for i := 0; i < len(ext); i++ {+ if ext[i] != '=' || (i > 0 && ext[i-1] == '\\') {+ continue+ }+ keyStart := strings.LastIndexByte(ext[:i], ' ') + 1+ key := ext[keyStart:i]+ if key == "" {+ continue+ }+ spans = append(spans, span{key: key, start: i + 1})+ }+ for i, s := range spans {+ end := len(ext)+ if i+1 < len(spans) {+ // Back up over the next key and the space before it.+ end = spans[i+1].start - len(spans[i+1].key) - 2+ if end < s.start {+ end = s.start+ }+ }+ out[s.key] = unescapeCEF(strings.TrimSpace(ext[s.start:end]))+ }+ return out+}++// parseLEEF fills msg.Extension from a LEEF payload.+//+// LEEF:1.0|Vendor|Product|Version|EventID|key=value<TAB>key=value+// LEEF:2.0|Vendor|Product|Version|EventID|<delimiter>|key=value...+//+// The 2.0 delimiter field is the part that catches people: it may be a+// literal character, or `xHH` / `x09` naming one by hex code. Assuming tab+// means every pair in a caret-delimited line lands in one key.+func parseLEEF(msg *Message, payload string) {+ idx := strings.Index(payload, "LEEF:")+ if idx < 0 {+ return+ }+ body := payload[idx+len("LEEF:"):]+ fields := splitEscaped(body, '|', 7)+ if len(fields) < 5 {+ return+ }+ msg.Format = FormatLEEF+ version := fields[0]+ msg.Extension["LEEFVersion"] = version+ msg.Extension["DeviceVendor"] = fields[1]+ msg.Extension["DeviceProduct"] = fields[2]+ msg.Extension["DeviceVersion"] = fields[3]+ msg.Extension["EventID"] = fields[4]++ delimiter := "\t"+ attrs := ""+ switch {+ case strings.HasPrefix(version, "2") && len(fields) >= 7:+ delimiter = resolveLEEFDelimiter(fields[5])+ attrs = fields[6]+ case len(fields) >= 6:+ attrs = fields[5]+ }++ for _, pair := range strings.Split(attrs, delimiter) {+ key, value, found := strings.Cut(pair, "=")+ if !found {+ continue+ }+ key = strings.TrimSpace(key)+ if key != "" {+ msg.Extension[key] = strings.TrimSpace(value)+ }+ }+ if text := msg.Extension["msg"]; text != "" {+ msg.Message = text+ }+}++func resolveLEEFDelimiter(field string) string {+ field = strings.TrimSpace(field)+ if field == "" {+ return "\t"+ }+ if (strings.HasPrefix(field, "x") || strings.HasPrefix(field, "0x")) && len(field) >= 3 {+ hex := strings.TrimPrefix(strings.TrimPrefix(field, "0x"), "x")+ if code, err := strconv.ParseUint(hex, 16, 8); err == nil {+ return string(rune(code))+ }+ }+ return field[:1]+}++// splitEscaped splits on sep, honouring a backslash escape, into at most+// `limit` fields with the remainder in the last.+func splitEscaped(s string, sep byte, limit int) []string {+ var out []string+ var current strings.Builder+ for i := 0; i < len(s); i++ {+ if s[i] == '\\' && i+1 < len(s) && s[i+1] == sep {+ current.WriteByte(sep)+ i+++ continue+ }+ if s[i] == sep && len(out) < limit {+ out = append(out, current.String())+ current.Reset()+ continue+ }+ current.WriteByte(s[i])+ }+ out = append(out, current.String())+ return out+}++func unescapeCEF(value string) string {+ return strings.NewReplacer(`\=`, `=`, `\|`, `|`, `\\`, `\`, `\n`, "\n", `\r`, "\r").Replace(value)+}diff --git a/services/ingest/internal/syslog/listener.go b/services/ingest/internal/syslog/listener.gonew file mode 100644index 000000000..ad96c8eb8--- /dev/null+++ b/services/ingest/internal/syslog/listener.go@@ -0,0 +1,459 @@+package syslog++import (+ "bufio"+ "context"+ "errors"+ "fmt"+ "io"+ "net"+ "strconv"+ "strings"+ "sync"+ "time"++ "github.com/prometheus/client_golang/prometheus"+ "github.com/prometheus/client_golang/prometheus/promauto"+ "github.com/rs/zerolog/log"+)++// Sink receives parsed messages. An interface rather than the publisher so+// the listener can be tested without Kafka — and so the thing under test is+// the framing and the parse, which is where the defects are.+type Sink interface {+ Deliver(ctx context.Context, messages []*Message) error+}++// Config is one listener's worth of settings.+type Config struct {+ // UDPAddr and TCPAddr are host:port. An empty one disables that+ // transport; both empty is a configuration error rather than a silent+ // no-op, because a listener bound to nothing looks identical to a+ // listener nobody is sending to.+ UDPAddr string+ TCPAddr string++ // BatchSize and FlushInterval bound how long a message waits before it+ // is published. Syslog arrives one line at a time and publishing one+ // Kafka record per line is what makes a syslog receiver fall over, so+ // lines are batched — but a quiet source must not have its one message+ // held indefinitely, which is what the interval is for.+ BatchSize int+ FlushInterval time.Duration++ // MaxConnections caps concurrent TCP sessions. Without it, anything+ // that can open sockets can exhaust the process's file descriptors and+ // take the HTTP ingest path down with it.+ MaxConnections int++ // ReadTimeout closes a TCP connection that has sent nothing for this+ // long. A forwarder that disappears without a FIN otherwise holds its+ // slot until the process restarts.+ ReadTimeout time.Duration+}++func (c Config) withDefaults() Config {+ if c.BatchSize <= 0 {+ c.BatchSize = 500+ }+ if c.FlushInterval <= 0 {+ c.FlushInterval = 2 * time.Second+ }+ if c.MaxConnections <= 0 {+ c.MaxConnections = 256+ }+ if c.ReadTimeout <= 0 {+ c.ReadTimeout = 5 * time.Minute+ }+ return c+}++var (+ syslogMessages = promauto.NewCounterVec(+ prometheus.CounterOpts{+ Name: "aisoc_ingest_syslog_messages_total",+ Help: "Syslog messages received, by transport and parsed format.",+ },+ []string{"transport", "format"},+ )+ syslogRejected = promauto.NewCounterVec(+ prometheus.CounterOpts{+ Name: "aisoc_ingest_syslog_rejected_total",+ Help: "Syslog lines that could not be accepted, by transport and reason.",+ },+ []string{"transport", "reason"},+ )+ syslogConnections = promauto.NewGauge(+ prometheus.GaugeOpts{+ Name: "aisoc_ingest_syslog_tcp_connections",+ Help: "Open syslog TCP connections.",+ },+ )+)++// Listener accepts syslog over UDP and TCP and hands parsed messages to a+// Sink in batches.+type Listener struct {+ cfg Config+ sink Sink++ udp *net.UDPConn+ tcp net.Listener++ mu sync.Mutex+ pending []*Message+ health State+ wg sync.WaitGroup+ connSlot chan struct{}+ // done is closed by Stop. The flush loop watches it as well as the+ // caller's context, because Stop waits on the goroutines and a loop+ // that only watches a context the caller has not cancelled yet never+ // returns — a deadlock on the shutdown path, which is the one path+ // nobody exercises until production.+ done chan struct{}+ stopOnce sync.Once+}++// State is what readiness reports. A listener that is bound and a listener+// that failed to bind are indistinguishable from outside unless the state+// is named, and "nothing has arrived" is the normal condition for a syslog+// receiver, so silence proves nothing either way.+type State struct {+ UDPBound bool+ TCPBound bool+ UDPAddr string+ TCPAddr string+ LastError string+ Received uint64+ Rejected uint64+ LastDelivery time.Time+}++// New binds the configured transports. Binding is done here rather than in+// Start so a port already in use is a startup error an operator sees,+// not a goroutine that exits into a log line nobody reads.+func New(cfg Config, sink Sink) (*Listener, error) {+ cfg = cfg.withDefaults()+ if cfg.UDPAddr == "" && cfg.TCPAddr == "" {+ return nil, errors.New("syslog: neither UDP nor TCP address configured; set at least one or leave the listener disabled")+ }+ if sink == nil {+ return nil, errors.New("syslog: no sink")+ }++ l := &Listener{cfg: cfg, sink: sink, connSlot: make(chan struct{}, cfg.MaxConnections), done: make(chan struct{})}++ if cfg.UDPAddr != "" {+ addr, err := net.ResolveUDPAddr("udp", cfg.UDPAddr)+ if err != nil {+ return nil, fmt.Errorf("syslog: resolving UDP address %q: %w", cfg.UDPAddr, err)+ }+ conn, err := net.ListenUDP("udp", addr)+ if err != nil {+ return nil, fmt.Errorf("syslog: binding UDP %q: %w (a port below 1024 needs a capability or a published container port)", cfg.UDPAddr, err)+ }+ l.udp = conn+ l.health.UDPBound = true+ l.health.UDPAddr = conn.LocalAddr().String()+ }++ if cfg.TCPAddr != "" {+ conn, err := net.Listen("tcp", cfg.TCPAddr)+ if err != nil {+ if l.udp != nil {+ _ = l.udp.Close()+ }+ return nil, fmt.Errorf("syslog: binding TCP %q: %w (a port below 1024 needs a capability or a published container port)", cfg.TCPAddr, err)+ }+ l.tcp = conn+ l.health.TCPBound = true+ l.health.TCPAddr = conn.Addr().String()+ }+ return l, nil+}++// Addrs reports what was actually bound, which is not what was configured+// when the port was `:0`.+func (l *Listener) Addrs() (udp string, tcp string) {+ return l.health.UDPAddr, l.health.TCPAddr+}++// Health is a snapshot for the readiness endpoint.+func (l *Listener) Health() State {+ l.mu.Lock()+ defer l.mu.Unlock()+ return l.health+}++// Start runs until ctx is cancelled.+func (l *Listener) Start(ctx context.Context) {+ if l.udp != nil {+ l.wg.Add(1)+ go func() {+ defer l.wg.Done()+ l.readUDP(ctx)+ }()+ }+ if l.tcp != nil {+ l.wg.Add(1)+ go func() {+ defer l.wg.Done()+ l.acceptTCP(ctx)+ }()+ }+ l.wg.Add(1)+ go func() {+ defer l.wg.Done()+ l.flushLoop(ctx)+ }()+}++// Stop closes the sockets and drains what is already buffered.+func (l *Listener) Stop(ctx context.Context) {+ l.stopOnce.Do(func() { close(l.done) })+ if l.udp != nil {+ _ = l.udp.Close()+ }+ if l.tcp != nil {+ _ = l.tcp.Close()+ }+ l.wg.Wait()+ l.flush(ctx)+}++func (l *Listener) readUDP(ctx context.Context) {+ buf := make([]byte, MaxLineBytes)+ for {+ if ctx.Err() != nil || l.stopping() {+ return+ }+ n, _, err := l.udp.ReadFromUDP(buf)+ if err != nil {+ if isClosed(err) || ctx.Err() != nil || l.stopping() {+ return+ }+ // A read error on a bound UDP socket is almost always+ // transient (an ICMP port-unreachable from a previous send).+ // Logged rather than fatal, and the loop continues, because+ // exiting here detaches the listener while the socket stays+ // bound — the shape that makes a dead consumer look idle.+ log.Warn().Err(err).Str("transport", "udp").Msg("syslog: read error, continuing")+ syslogRejected.WithLabelValues("udp", "read_error").Inc()+ continue+ }+ // One datagram is one message. A datagram carrying several+ // newline-separated lines is non-standard but common from+ // relays, so it is split.+ for _, line := range strings.Split(string(buf[:n]), "\n") {+ l.ingest(ctx, "udp", line)+ }+ }+}++func (l *Listener) acceptTCP(ctx context.Context) {+ for {+ conn, err := l.tcp.Accept()+ if err != nil {+ if isClosed(err) || ctx.Err() != nil || l.stopping() {+ return+ }+ log.Warn().Err(err).Msg("syslog: accept error, continuing")+ continue+ }+ select {+ case l.connSlot <- struct{}{}:+ default:+ // At the connection cap. Refused immediately and counted,+ // because silently holding the socket open would look to the+ // forwarder like a working receiver that never acknowledges.+ syslogRejected.WithLabelValues("tcp", "too_many_connections").Inc()+ log.Warn().Int("max", l.cfg.MaxConnections).Msg("syslog: refusing connection, at the configured cap")+ _ = conn.Close()+ continue+ }+ syslogConnections.Inc()+ l.wg.Add(1)+ go func() {+ defer l.wg.Done()+ defer func() { <-l.connSlot; syslogConnections.Dec() }()+ l.readTCPConn(ctx, conn)+ }()+ }+}++// readTCPConn reads one session, handling both framings RFC 6587 defines.+//+// Octet-counting (`123 <34>1 ...`) is the one the RFC recommends and the+// one that works when a message contains a newline. Non-transparent framing+// (LF-delimited) is what most senders actually use. A receiver has to read+// both, and it decides per message by looking at whether the next thing on+// the wire is a digit run followed by a space.+func (l *Listener) readTCPConn(ctx context.Context, conn net.Conn) {+ defer func() { _ = conn.Close() }()+ reader := bufio.NewReaderSize(conn, 64*1024)++ for {+ if ctx.Err() != nil || l.stopping() {+ return+ }+ if err := conn.SetReadDeadline(time.Now().Add(l.cfg.ReadTimeout)); err != nil {+ return+ }++ line, err := l.readFrame(reader)+ if err != nil {+ if !errors.Is(err, io.EOF) && !isClosed(err) && ctx.Err() == nil {+ var netErr net.Error+ if errors.As(err, &netErr) && netErr.Timeout() {+ log.Debug().Msg("syslog: closing an idle TCP connection")+ } else {+ syslogRejected.WithLabelValues("tcp", "frame_error").Inc()+ log.Warn().Err(err).Msg("syslog: framing error, closing connection")+ }+ }+ return+ }+ l.ingest(ctx, "tcp", line)+ }+}++func (l *Listener) readFrame(reader *bufio.Reader) (string, error) {+ peeked, err := reader.Peek(1)+ if err != nil {+ return "", err+ }+ if peeked[0] < '0' || peeked[0] > '9' {+ line, err := reader.ReadString('\n')+ if err != nil && line == "" {+ return "", err+ }+ return strings.TrimRight(line, "\r\n"), nil+ }++ // Octet-counting: MSG-LEN SP SYSLOG-MSG.+ var digits strings.Builder+ for {+ b, err := reader.ReadByte()+ if err != nil {+ return "", err+ }+ if b == ' ' {+ break+ }+ if b < '0' || b > '9' {+ // Not a length prefix after all — a message that happens to+ // start with a digit. Put the byte back and read to newline.+ if err := reader.UnreadByte(); err != nil {+ return "", err+ }+ rest, err := reader.ReadString('\n')+ if err != nil && rest == "" {+ return "", err+ }+ return digits.String() + strings.TrimRight(rest, "\r\n"), nil+ }+ digits.WriteByte(b)+ if digits.Len() > 10 {+ return "", fmt.Errorf("syslog: octet count %q is not a plausible length", digits.String())+ }+ }+ length, err := strconv.Atoi(digits.String())+ if err != nil || length < 0 {+ return "", fmt.Errorf("syslog: unreadable octet count %q", digits.String())+ }+ if length > MaxLineBytes {+ return "", fmt.Errorf("syslog: framed message of %d bytes is over the %d cap", length, MaxLineBytes)+ }+ buf := make([]byte, length)+ if _, err := io.ReadFull(reader, buf); err != nil {+ return "", err+ }+ return string(buf), nil+}++func (l *Listener) ingest(ctx context.Context, transport, line string) {+ msg, err := Parse(line)+ if err != nil {+ if !errors.Is(err, ErrEmpty) {+ syslogRejected.WithLabelValues(transport, "parse_error").Inc()+ l.mu.Lock()+ l.health.Rejected+++ l.health.LastError = err.Error()+ l.mu.Unlock()+ }+ return+ }+ syslogMessages.WithLabelValues(transport, string(msg.Format)).Inc()++ l.mu.Lock()+ l.pending = append(l.pending, msg)+ l.health.Received+++ full := len(l.pending) >= l.cfg.BatchSize+ l.mu.Unlock()++ if full {+ l.flush(ctx)+ }+}++func (l *Listener) flushLoop(ctx context.Context) {+ ticker := time.NewTicker(l.cfg.FlushInterval)+ defer ticker.Stop()+ for {+ select {+ case <-ctx.Done():+ return+ case <-l.done:+ return+ case <-ticker.C:+ l.flush(ctx)+ }+ }+}++// Flush publishes whatever is buffered. Exported so a test can drive the+// boundary directly rather than sleeping on the interval.+func (l *Listener) Flush(ctx context.Context) { l.flush(ctx) }++func (l *Listener) flush(ctx context.Context) {+ l.mu.Lock()+ if len(l.pending) == 0 {+ l.mu.Unlock()+ return+ }+ batch := l.pending+ l.pending = nil+ l.mu.Unlock()++ if err := l.sink.Deliver(ctx, batch); err != nil {+ // Counted and named. Not re-queued: syslog is a fire-and-forget+ // transport with no acknowledgement to withhold, so a sender has+ // already moved on and holding the batch would only grow the+ // buffer until the process dies. Saying so is the honest part —+ // the alternative is a receiver that loses messages silently.+ syslogRejected.WithLabelValues("sink", "publish_error").Add(float64(len(batch)))+ l.mu.Lock()+ l.health.Rejected += uint64(len(batch))+ l.health.LastError = err.Error()+ l.mu.Unlock()+ log.Error().Err(err).Int("dropped", len(batch)).+ Msg("syslog: publish failed; these messages are lost because syslog has no acknowledgement to withhold")+ return+ }+ l.mu.Lock()+ l.health.LastDelivery = time.Now().UTC()+ l.mu.Unlock()+}++func (l *Listener) stopping() bool {+ select {+ case <-l.done:+ return true+ default:+ return false+ }+}++func isClosed(err error) bool {+ return errors.Is(err, net.ErrClosed) || errors.Is(err, io.EOF)+}diff --git a/services/ingest/internal/syslog/listener_test.go b/services/ingest/internal/syslog/listener_test.gonew file mode 100644index 000000000..7559d2c42--- /dev/null+++ b/services/ingest/internal/syslog/listener_test.go@@ -0,0 +1,325 @@+package syslog++import (+ "context"+ "errors"+ "fmt"+ "net"+ "strings"+ "sync"+ "testing"+ "time"+)++// recordingSink captures what the listener delivers. Deliberately less+// capable than the real one: it cannot invent a tenant, cannot normalise,+// and fails when told to — which is the only way to see what the listener+// does with a failed publish.+type recordingSink struct {+ mu sync.Mutex+ messages []*Message+ fail error+ delivers int+}++func (s *recordingSink) Deliver(_ context.Context, messages []*Message) error {+ s.mu.Lock()+ defer s.mu.Unlock()+ s.delivers+++ if s.fail != nil {+ return s.fail+ }+ s.messages = append(s.messages, messages...)+ return nil+}++func (s *recordingSink) snapshot() []*Message {+ s.mu.Lock()+ defer s.mu.Unlock()+ return append([]*Message(nil), s.messages...)+}++// disabled is the empty address, named so a test reads as "this transport+// is off" rather than as a typo.+const disabled = ""++func newTestListener(t *testing.T, sink Sink, cfg Config) *Listener {+ t.Helper()+ if cfg.UDPAddr == "" && cfg.TCPAddr == "" {+ cfg.UDPAddr = "127.0.0.1:0"+ cfg.TCPAddr = "127.0.0.1:0"+ }+ if cfg.FlushInterval == 0 {+ cfg.FlushInterval = 20 * time.Millisecond+ }+ l, err := New(cfg, sink)+ if err != nil {+ t.Fatalf("New: %v", err)+ }+ return l+}++func waitFor(t *testing.T, condition func() bool) {+ t.Helper()+ deadline := time.Now().Add(3 * time.Second)+ for time.Now().Before(deadline) {+ if condition() {+ return+ }+ time.Sleep(5 * time.Millisecond)+ }+ t.Fatal("condition was not met within 3s")+}++func TestABindFailureIsAStartupErrorNotASilentNoOp(t *testing.T) {+ // A listener that failed to bind and a listener nobody is sending to+ // are indistinguishable from outside, which is why binding happens in+ // New and reports rather than in a goroutine that exits into a log.+ first := newTestListener(t, &recordingSink{}, Config{TCPAddr: "127.0.0.1:0", UDPAddr: disabled})+ _, tcp := first.Addrs()+ defer first.Stop(context.Background())++ _, err := New(Config{TCPAddr: tcp}, &recordingSink{})+ if err == nil {+ t.Fatal("binding an address already in use should be an error")+ }+ if !strings.Contains(err.Error(), "binding TCP") {+ t.Errorf("error should name the transport and the address: %v", err)+ }+}++func TestNoTransportAtAllIsRefused(t *testing.T) {+ if _, err := New(Config{}, &recordingSink{}); err == nil {+ t.Fatal("a listener bound to nothing should be refused, not started")+ }+}++func TestUDPDatagramsAreParsedAndDelivered(t *testing.T) {+ sink := &recordingSink{}+ l := newTestListener(t, sink, Config{UDPAddr: "127.0.0.1:0", TCPAddr: disabled})+ udp, _ := l.Addrs()+ ctx, cancel := context.WithCancel(context.Background())+ defer cancel()+ l.Start(ctx)+ defer l.Stop(context.Background())++ conn, err := net.Dial("udp", udp)+ if err != nil {+ t.Fatalf("dial: %v", err)+ }+ defer func() { _ = conn.Close() }()+ if _, err := conn.Write([]byte("<34>Oct 11 22:14:15 mymachine su: failed for lonvick")); err != nil {+ t.Fatalf("write: %v", err)+ }++ waitFor(t, func() bool { return len(sink.snapshot()) == 1 })+ got := sink.snapshot()[0]+ if got.Hostname != "mymachine" || got.AppName != "su" {+ t.Errorf("parsed wrong: %#v", got)+ }+}++func TestTCPReadsNonTransparentFraming(t *testing.T) {+ // LF-delimited. Not what RFC 6587 recommends, and what almost every+ // sender actually does.+ sink := &recordingSink{}+ l := newTestListener(t, sink, Config{TCPAddr: "127.0.0.1:0", UDPAddr: disabled})+ _, tcp := l.Addrs()+ ctx, cancel := context.WithCancel(context.Background())+ defer cancel()+ l.Start(ctx)+ defer l.Stop(context.Background())++ conn, err := net.Dial("tcp", tcp)+ if err != nil {+ t.Fatalf("dial: %v", err)+ }+ defer func() { _ = conn.Close() }()+ _, _ = conn.Write([]byte("<13>Oct 11 22:14:15 web01 sshd[1]: one\n<13>Oct 11 22:14:16 web01 sshd[2]: two\n"))++ waitFor(t, func() bool { return len(sink.snapshot()) == 2 })+ if got := sink.snapshot()[1].ProcID; got != "2" {+ t.Errorf("second message pid = %q, want 2", got)+ }+}++func TestTCPReadsOctetCounting(t *testing.T) {+ // RFC 6587 §3.4.1. The framing that exists because a message may+ // legitimately contain a newline — which an LF-framed reader splits+ // into two records, one of them nonsense.+ sink := &recordingSink{}+ l := newTestListener(t, sink, Config{TCPAddr: "127.0.0.1:0", UDPAddr: disabled})+ _, tcp := l.Addrs()+ ctx, cancel := context.WithCancel(context.Background())+ defer cancel()+ l.Start(ctx)+ defer l.Stop(context.Background())++ conn, err := net.Dial("tcp", tcp)+ if err != nil {+ t.Fatalf("dial: %v", err)+ }+ defer func() { _ = conn.Close() }()+ payload := "<13>Oct 11 22:14:15 web01 app: line one\nline two"+ _, _ = conn.Write([]byte(fmt.Sprintf("%d %s", len(payload), payload)))++ waitFor(t, func() bool { return len(sink.snapshot()) == 1 })+ if got := sink.snapshot()[0].Message; got != "line one\nline two" {+ t.Errorf("message = %q; octet-counted framing must not split on the newline", got)+ }+}++func TestAMessageStartingWithADigitIsNotMistakenForALengthPrefix(t *testing.T) {+ // The framing is decided per message by peeking at one byte, so a+ // non-framed line whose first character is a digit has to fall back to+ // LF framing rather than consuming the rest of the stream as payload.+ sink := &recordingSink{}+ l := newTestListener(t, sink, Config{TCPAddr: "127.0.0.1:0", UDPAddr: disabled})+ _, tcp := l.Addrs()+ ctx, cancel := context.WithCancel(context.Background())+ defer cancel()+ l.Start(ctx)+ defer l.Stop(context.Background())++ conn, err := net.Dial("tcp", tcp)+ if err != nil {+ t.Fatalf("dial: %v", err)+ }+ defer func() { _ = conn.Close() }()+ _, _ = conn.Write([]byte("2026-10-08 something happened\n"))++ waitFor(t, func() bool { return len(sink.snapshot()) == 1 })+ if got := sink.snapshot()[0].Message; got != "2026-10-08 something happened" {+ t.Errorf("message = %q", got)+ }+}++func TestAnImplausibleOctetCountClosesTheConnection(t *testing.T) {+ // The length comes off the wire. A reader that trusts it allocates+ // whatever a hostile or broken sender asks for.+ sink := &recordingSink{}+ l := newTestListener(t, sink, Config{TCPAddr: "127.0.0.1:0", UDPAddr: disabled})+ _, tcp := l.Addrs()+ ctx, cancel := context.WithCancel(context.Background())+ defer cancel()+ l.Start(ctx)+ defer l.Stop(context.Background())++ conn, err := net.Dial("tcp", tcp)+ if err != nil {+ t.Fatalf("dial: %v", err)+ }+ defer func() { _ = conn.Close() }()+ _, _ = conn.Write([]byte("99999999 hello"))++ // The connection is closed rather than the allocation made.+ _ = conn.SetReadDeadline(time.Now().Add(2 * time.Second))+ buf := make([]byte, 1)+ if _, err := conn.Read(buf); err == nil {+ t.Error("the connection should have been closed")+ }+ if len(sink.snapshot()) != 0 {+ t.Error("nothing should have been delivered")+ }+}++func TestTheBatchIsFlushedOnSizeAndOnTime(t *testing.T) {+ // Publishing one Kafka record per syslog line is what makes a receiver+ // fall over; holding a quiet source's single message forever is the+ // other failure. Both are tested because fixing one usually breaks the+ // other.+ sink := &recordingSink{}+ l := newTestListener(t, sink, Config{UDPAddr: "127.0.0.1:0", TCPAddr: disabled, BatchSize: 3, FlushInterval: 30 * time.Millisecond})+ udp, _ := l.Addrs()+ ctx, cancel := context.WithCancel(context.Background())+ defer cancel()+ l.Start(ctx)+ defer l.Stop(context.Background())++ conn, err := net.Dial("udp", udp)+ if err != nil {+ t.Fatalf("dial: %v", err)+ }+ defer func() { _ = conn.Close() }()+ for i := 0; i < 3; i++ {+ _, _ = conn.Write([]byte(fmt.Sprintf("<13>Oct 11 22:14:1%d web01 app: n%d", i, i)))+ }+ waitFor(t, func() bool { return len(sink.snapshot()) == 3 })++ // One more, on its own: it must not wait for a fourth that never comes.+ _, _ = conn.Write([]byte("<13>Oct 11 22:14:19 web01 app: lonely"))+ waitFor(t, func() bool { return len(sink.snapshot()) == 4 })+}++func TestAFailedPublishIsCountedRatherThanSilent(t *testing.T) {+ // Syslog has no acknowledgement to withhold, so a failed publish is a+ // lost message no matter what this does. What it must not do is lose it+ // quietly: the rejected counter and the readiness detail are the only+ // evidence an operator will ever get.+ sink := &recordingSink{fail: errors.New("kafka unreachable")}+ l := newTestListener(t, sink, Config{UDPAddr: "127.0.0.1:0", TCPAddr: disabled, BatchSize: 1})+ udp, _ := l.Addrs()+ ctx, cancel := context.WithCancel(context.Background())+ defer cancel()+ l.Start(ctx)+ defer l.Stop(context.Background())++ conn, err := net.Dial("udp", udp)+ if err != nil {+ t.Fatalf("dial: %v", err)+ }+ defer func() { _ = conn.Close() }()+ _, _ = conn.Write([]byte("<13>Oct 11 22:14:15 web01 app: hello"))++ waitFor(t, func() bool { return l.Health().Rejected > 0 })+ if detail := l.Health().LastError; !strings.Contains(detail, "kafka unreachable") {+ t.Errorf("readiness detail = %q; it must name the failure", detail)+ }+}++func TestReadinessNamesWhatIsBound(t *testing.T) {+ // "attached" alone is a lie of omission for a receiver that is+ // legitimately silent for hours.+ l := newTestListener(t, &recordingSink{}, Config{UDPAddr: "127.0.0.1:0", TCPAddr: "127.0.0.1:0"})+ defer l.Stop(context.Background())++ state := l.Health()+ if !state.UDPBound || !state.TCPBound {+ t.Fatalf("both transports should report bound: %#v", state)+ }+ if state.UDPAddr == "" || state.TCPAddr == "" {+ t.Error("readiness should report the address actually bound, which is not the one configured when the port was :0")+ }+}++func TestTheConnectionCapIsEnforced(t *testing.T) {+ // Without it, anything that can open sockets exhausts the process's+ // file descriptors and takes the HTTP ingest path down with it.+ sink := &recordingSink{}+ l := newTestListener(t, sink, Config{TCPAddr: "127.0.0.1:0", UDPAddr: disabled, MaxConnections: 1})+ _, tcp := l.Addrs()+ ctx, cancel := context.WithCancel(context.Background())+ defer cancel()+ l.Start(ctx)+ defer l.Stop(context.Background())++ first, err := net.Dial("tcp", tcp)+ if err != nil {+ t.Fatalf("dial: %v", err)+ }+ defer func() { _ = first.Close() }()+ // Give the accept loop time to take the only slot.+ _, _ = first.Write([]byte("<13>Oct 11 22:14:15 web01 app: one\n"))+ waitFor(t, func() bool { return len(sink.snapshot()) == 1 })++ second, err := net.Dial("tcp", tcp)+ if err != nil {+ t.Fatalf("dial: %v", err)+ }+ defer func() { _ = second.Close() }()+ _ = second.SetReadDeadline(time.Now().Add(2 * time.Second))+ buf := make([]byte, 1)+ if _, err := second.Read(buf); err == nil {+ t.Error("the second connection should have been refused immediately rather than held open")+ }+}diff --git a/services/ingest/internal/syslog/parse.go b/services/ingest/internal/syslog/parse.gonew file mode 100644index 000000000..49a694578--- /dev/null+++ b/services/ingest/internal/syslog/parse.go@@ -0,0 +1,498 @@+// Package syslog reads the four wire formats the long tail of security+// appliances speaks, and nothing else.+//+// Firewalls, proxies, load balancers, legacy IDS, printers and a great deal+// of custom gear emit syslog and will never grow a webhook. Before this,+// AiSOC's only syslog path was `POST /v1/inbox/cef`, which requires a+// forwarder in front of it that can already speak HTTP — so the appliance+// still could not reach AiSOC, something in front of it had to.+//+// Four formats, and the distinction between them is not cosmetic:+//+// RFC 5424 the current standard: a version digit, an RFC 3339 timestamp,+// and structured data. Unambiguous to parse.+// RFC 3164 the one most gear actually emits, and the one that cannot be+// parsed reliably: the timestamp carries no year and no zone,+// the hostname is optional, and the tag is delimited by a colon+// that may also appear in the message.+// CEF ArcSight's key/value format, carried *inside* a syslog line.+// LEEF QRadar's, likewise, with a delimiter the header may redefine.+//+// A parser that guesses between them is worse than one that refuses: a+// mis-split RFC 3164 line puts half the message in the hostname column, and+// an analyst reading that record sees a host that does not exist. So the+// format is decided by markers the formats themselves define — a version+// digit after the priority, a `CEF:` or `LEEF:` anchor — and anything that+// matches none of them is returned as an unstructured message with its text+// intact, never as a structured record with invented fields.+package syslog++import (+ "errors"+ "fmt"+ "strconv"+ "strings"+ "time"+)++// Format names the wire shape one line turned out to be.+type Format string++const (+ FormatRFC5424 Format = "rfc5424"+ FormatRFC3164 Format = "rfc3164"+ FormatCEF Format = "cef"+ FormatLEEF Format = "leef"+ // FormatUnstructured is a line with no recognisable framing. It is a+ // result, not an error: a device emitting bare text is still telling+ // you something, and dropping it loses the only copy.+ FormatUnstructured Format = "unstructured"+)++// ErrEmpty is returned for a line with nothing in it. Distinguished from a+// parse failure because an empty UDP datagram is a keepalive, not a fault,+// and counting it as malformed makes the malformed-line metric useless.+var ErrEmpty = errors.New("empty syslog line")++// MaxLineBytes caps one message. RFC 5424 requires a receiver to accept at+// least 2048 octets and permits more; 64 KiB is well above what any+// appliance emits and well below what a hostile sender could use to exhaust+// a connection's buffer.+const MaxLineBytes = 64 * 1024++// : The eight syslog severities, lowest number = most severe. Mapped onto+// : AiSOC's five tiers by the template's severity_map; kept here as the+// : vendor-native value so nothing downstream has to re-derive it.+var severityNames = [8]string{"emerg", "alert", "crit", "err", "warning", "notice", "info", "debug"}++var facilityNames = [24]string{+ "kern", "user", "mail", "daemon", "auth", "syslog", "lpr", "news",+ "uucp", "cron", "authpriv", "ftp", "ntp", "audit", "alert", "clock",+ "local0", "local1", "local2", "local3", "local4", "local5", "local6", "local7",+}++// Message is one parsed line. Every field is optional except Format and+// Message itself, because every field *is* optional in at least one of the+// four formats.+type Message struct {+ Format Format+ Priority int+ Facility string+ Severity string+ // SeverityCode is the syslog 0-7 value. Kept beside the name because a+ // template maps on the string and a human reads the number, and+ // re-deriving either from the other at three call sites is how they+ // come to disagree.+ SeverityCode int+ Timestamp string+ Hostname string+ AppName string+ ProcID string+ MsgID string+ // StructuredData is RFC 5424's SD-ELEMENT list, flattened to+ // "[email protected]" keys so a template can address one.+ StructuredData map[string]string+ Message string+ // Extension carries the CEF or LEEF key/value pairs and header fields.+ Extension map[string]string+ // Raw is the line exactly as it arrived. Kept because every parse here+ // is a lossy summary and an appliance emitting something this parser+ // misreads is itself the finding.+ Raw string+}++// Fields flattens the message into the shape an inbox template addresses.+func (m *Message) Fields() map[string]any {+ out := make(map[string]any, 12+len(m.Extension)+len(m.StructuredData))+ out["format"] = string(m.Format)+ out["priority"] = m.Priority+ out["facility"] = m.Facility+ out["severity"] = m.Severity+ out["severity_code"] = m.SeverityCode+ out["message"] = m.Message+ out["raw"] = m.Raw+ for key, value := range map[string]string{+ "timestamp": m.Timestamp,+ "hostname": m.Hostname,+ "app_name": m.AppName,+ "proc_id": m.ProcID,+ "msg_id": m.MsgID,+ } {+ if value != "" {+ out[key] = value+ }+ }+ for key, value := range m.StructuredData {+ out["sd."+key] = value+ }+ // CEF and LEEF keys are merged at the top level because that is where+ // the existing cef-syslog template addresses them (`src`, `suser`,+ // `act`). Nesting them would mean two templates for one format.+ for key, value := range m.Extension {+ out[key] = value+ }+ return out+}++// Parse reads one line. It never returns both a message and an error.+func Parse(line string) (*Message, error) {+ trimmed := strings.TrimRight(strings.TrimSpace(line), "\x00")+ if trimmed == "" {+ return nil, ErrEmpty+ }+ if len(trimmed) > MaxLineBytes {+ return nil, fmt.Errorf("syslog line is %d bytes, over the %d cap", len(trimmed), MaxLineBytes)+ }++ msg := &Message{Raw: line, StructuredData: map[string]string{}, Extension: map[string]string{}}+ rest := trimmed++ // <PRI> is optional on the wire — a device writing straight to a TCP+ // socket often omits it — so its absence is not a parse failure.+ if strings.HasPrefix(rest, "<") {+ if end := strings.Index(rest, ">"); end > 1 && end <= 4 {+ if pri, err := strconv.Atoi(rest[1:end]); err == nil && pri >= 0 && pri < 192 {+ msg.Priority = pri+ msg.SeverityCode = pri % 8+ msg.Severity = severityNames[pri%8]+ msg.Facility = facilityNames[pri/8]+ rest = rest[end+1:]+ }+ }+ } else {+ // No priority at all. `user.notice` is what the RFC names as the+ // default, and recording it explicitly beats leaving a zero that+ // reads as `kern.emerg` — the most severe value there is.+ msg.Priority = 13+ msg.SeverityCode = 5+ msg.Severity = severityNames[5]+ msg.Facility = facilityNames[1]+ }++ switch {+ case isRFC5424(rest):+ parseRFC5424(msg, rest)+ case anchorAt(rest, "LEEF:") >= 0:+ // Checked before RFC 3164 because a LEEF line usually carries a+ // 3164 header *and* a LEEF payload, and the payload is the part+ // worth having. The split is at the anchor rather than after the+ // header parse: the 3164 tag reader would otherwise take `LEEF`+ // for an application name and eat the anchor with it.+ at := anchorAt(rest, "LEEF:")+ parseRFC3164Header(msg, rest[:at])+ msg.Message = rest[at:]+ parseLEEF(msg, rest[at:])+ case anchorAt(rest, "CEF:") >= 0:+ at := anchorAt(rest, "CEF:")+ parseRFC3164Header(msg, rest[:at])+ msg.Message = rest[at:]+ parseCEF(msg, rest[at:])+ case looksLikeRFC3164(rest):+ parseRFC3164Header(msg, rest)+ msg.Format = FormatRFC3164+ default:+ msg.Format = FormatUnstructured+ msg.Message = rest+ }++ return msg, nil+}++// anchorAt finds a CEF or LEEF anchor, or -1. `LEEF:` is checked before+// `CEF:` by the caller because the two substrings do not overlap but a+// vendor name containing "CEF:" inside a message would; nothing here tries+// to be clever about that, because the anchor is the only marker either+// format defines and inventing a second one would change which lines parse.+func anchorAt(rest, anchor string) int {+ return strings.Index(rest, anchor)+}++// isRFC5424 looks for the version digit the format requires immediately+// after the priority. `1 ` is the only version ever assigned, and matching+// it is what keeps a 3164 line whose message happens to start with a digit+// from being read as 5424.+func isRFC5424(rest string) bool {+ return strings.HasPrefix(rest, "1 ")+}++func parseRFC5424(msg *Message, rest string) {+ msg.Format = FormatRFC5424+ // VERSION SP TIMESTAMP SP HOSTNAME SP APP-NAME SP PROCID SP MSGID SP SD [SP MSG]+ parts := strings.SplitN(rest, " ", 7)+ if len(parts) < 7 {+ // A header this short is still readable; what is missing is the+ // structured data and message, not the identity fields.+ parts = append(parts, make([]string, 7-len(parts))...)+ }+ nilable := func(v string) string {+ if v == "-" {+ return ""+ }+ return v+ }+ msg.Timestamp = normaliseTimestamp(nilable(parts[1]))+ msg.Hostname = nilable(parts[2])+ msg.AppName = nilable(parts[3])+ msg.ProcID = nilable(parts[4])+ msg.MsgID = nilable(parts[5])++ tail := parts[6]+ sd, message := splitStructuredData(tail)+ for key, value := range sd {+ msg.StructuredData[key] = value+ }+ message = strings.TrimPrefix(message, "\ufeff") // RFC 5424's optional BOM+ msg.Message = message++ // A 5424 line can still carry CEF or LEEF in its message.+ switch {+ case strings.Contains(message, "LEEF:"):+ parseLEEF(msg, message)+ case strings.Contains(message, "CEF:"):+ parseCEF(msg, message)+ }+}++// splitStructuredData separates the SD-ELEMENT list from the free-text+// message. Bracket counting rather than a regular expression, because a+// parameter value may contain `]` escaped as `\]` and a regex that ignores+// that truncates the structured data at the first one.+func splitStructuredData(tail string) (map[string]string, string) {+ sd := map[string]string{}+ if tail == "" {+ return sd, ""+ }+ if strings.HasPrefix(tail, "-") {+ return sd, strings.TrimSpace(tail[1:])+ }+ if !strings.HasPrefix(tail, "[") {+ return sd, tail+ }++ // Brackets inside a quoted PARAM-VALUE are data, not structure: rsyslog's+ // own `origin` element ships an `x-info` URL and operators routinely put+ // a bracketed note in one. Counting those as nesting leaves the scan at+ // depth 1 forever, and the whole SD block then falls through into the+ // message — silently, and the message still looks plausible.+ depth := 0+ escaped := false+ inQuotes := false+ end := -1+ for i := 0; i < len(tail) && end < 0; i++ {+ switch {+ case escaped:+ escaped = false+ case tail[i] == '\\':+ escaped = true+ case tail[i] == '"':+ inQuotes = !inQuotes+ case inQuotes:+ // Data.+ case tail[i] == '[':+ depth+++ case tail[i] == ']':+ depth--+ // Several SD-ELEMENTs may run together with no separator, so+ // the list only ends when the next byte is not another opening+ // bracket.+ if depth == 0 && (i+1 >= len(tail) || tail[i+1] != '[') {+ end = i+ }+ }+ }+ if end < 0 {+ // Unterminated: keep the whole thing as the message rather than+ // inventing a boundary.+ return sd, tail+ }+ for _, element := range splitElements(tail[:end+1]) {+ id, params := parseSDElement(element)+ for key, value := range params {+ sd[id+"."+key] = value+ }+ }+ return sd, strings.TrimSpace(tail[end+1:])+}++func splitElements(block string) []string {+ // Quote-aware for the same reason the scan above is: a bracket inside a+ // PARAM-VALUE is data. Without it this returns nothing for an element+ // carrying a bracketed value, and the caller then reports a line with+ // no structured data at all rather than failing.+ var out []string+ depth := 0+ escaped := false+ inQuotes := false+ start := 0+ for i := 0; i < len(block); i++ {+ switch {+ case escaped:+ escaped = false+ case block[i] == '\\':+ escaped = true+ case block[i] == '"':+ inQuotes = !inQuotes+ case inQuotes:+ // Data.+ case block[i] == '[':+ if depth == 0 {+ start = i+ }+ depth+++ case block[i] == ']':+ depth--+ if depth == 0 {+ out = append(out, block[start:i+1])+ }+ }+ }+ return out+}++func parseSDElement(element string) (string, map[string]string) {+ element = strings.TrimSuffix(strings.TrimPrefix(element, "["), "]")+ params := map[string]string{}+ fields := splitQuoted(element)+ if len(fields) == 0 {+ return "", params+ }+ id := fields[0]+ for _, field := range fields[1:] {+ key, value, found := strings.Cut(field, "=")+ if !found {+ continue+ }+ value = strings.TrimSuffix(strings.TrimPrefix(value, `"`), `"`)+ params[key] = unescapeSD(value)+ }+ return id, params+}++// splitQuoted splits on spaces that are not inside a quoted value.+func splitQuoted(s string) []string {+ var out []string+ var current strings.Builder+ inQuotes := false+ escaped := false+ for _, r := range s {+ switch {+ case escaped:+ current.WriteRune(r)+ escaped = false+ case r == '\\':+ current.WriteRune(r)+ escaped = true+ case r == '"':+ inQuotes = !inQuotes+ current.WriteRune(r)+ case r == ' ' && !inQuotes:+ if current.Len() > 0 {+ out = append(out, current.String())+ current.Reset()+ }+ default:+ current.WriteRune(r)+ }+ }+ if current.Len() > 0 {+ out = append(out, current.String())+ }+ return out+}++func unescapeSD(value string) string {+ replacer := strings.NewReplacer(`\"`, `"`, `\\`, `\`, `\]`, `]`)+ return replacer.Replace(value)+}++// looksLikeRFC3164 checks for the three-letter month the format opens with.+// Deliberately narrow: the alternative is "anything with a colon in it",+// which splits ordinary prose into a hostname and a message.+func looksLikeRFC3164(rest string) bool {+ if len(rest) < 15 {+ return false+ }+ months := "JanFebMarAprMayJunJulAugSepOctNovDec"+ return strings.Contains(months, rest[:3]) && rest[3] == ' '+}++func parseRFC3164Header(msg *Message, rest string) {+ if msg.Format == "" {+ msg.Format = FormatRFC3164+ }+ msg.Message = rest++ rest = strings.TrimSpace(rest)+ if !looksLikeRFC3164(rest) {+ return+ }+ // "MMM dd hh:mm:ss " is fixed-width at 16 octets including the trailing+ // space, which is the one thing about 3164 that *is* unambiguous.+ if len(rest) < 16 {+ return+ }+ msg.Timestamp = normaliseTimestamp(rest[:15])+ remainder := strings.TrimLeft(rest[15:], " ")++ host, tail, found := strings.Cut(remainder, " ")+ if !found {+ // Everything after the timestamp and nothing else. RFC 3164 puts+ // the hostname there, so reading it as the message would drop the+ // one identity field the format carries — which is the field a+ // responder needs most.+ msg.Hostname = remainder+ msg.Message = ""+ return+ }+ msg.Hostname = host+ msg.Message = tail++ // TAG[PID]: — the colon is only a delimiter when it arrives within the+ // 32 octets the RFC allows a tag, and only when what precedes it has no+ // space in it. Without both conditions, "Error: connection refused"+ // becomes an app name.+ if idx := strings.Index(tail, ":"); idx > 0 && idx <= 32 && !strings.Contains(tail[:idx], " ") {+ tag := tail[:idx]+ if open := strings.Index(tag, "["); open > 0 && strings.HasSuffix(tag, "]") {+ msg.AppName = tag[:open]+ msg.ProcID = tag[open+1 : len(tag)-1]+ } else {+ msg.AppName = tag+ }+ msg.Message = strings.TrimLeft(tail[idx+1:], " ")+ }+}++// normaliseTimestamp returns RFC 3339 where the input carries enough+// information, and the input unchanged where it does not.+//+// RFC 3164's timestamp has no year and no zone. Stamping the current year+// on it is the conventional move and it is wrong twice a year — at a new+// year boundary, and for any device whose clock is ahead — so the year is+// filled from the receiver's clock and the *original* stays in `raw`,+// which is the only honest option available.+func normaliseTimestamp(value string) string {+ value = strings.TrimSpace(value)+ if value == "" {+ return ""+ }+ for _, layout := range []string{time.RFC3339Nano, time.RFC3339} {+ if t, err := time.Parse(layout, value); err == nil {+ return t.UTC().Format(time.RFC3339Nano)+ }+ }+ for _, layout := range []string{"Jan _2 15:04:05", "Jan 2 15:04:05"} {+ if t, err := time.Parse(layout, value); err == nil {+ now := time.Now().UTC()+ stamped := time.Date(now.Year(), t.Month(), t.Day(), t.Hour(), t.Minute(), t.Second(), 0, time.UTC)+ // A December log read in January is last year's, not next+ // year's. One day of slack absorbs clock skew without+ // re-dating an ordinary same-day message.+ if stamped.Sub(now) > 24*time.Hour {+ stamped = stamped.AddDate(-1, 0, 0)+ }+ return stamped.Format(time.RFC3339Nano)+ }+ }+ return value+}diff --git a/services/ingest/internal/syslog/parse_test.go b/services/ingest/internal/syslog/parse_test.gonew file mode 100644index 000000000..b798736b1--- /dev/null+++ b/services/ingest/internal/syslog/parse_test.go@@ -0,0 +1,288 @@+package syslog++import (+ "strings"+ "testing"+ "time"+)++// Every line below is a wire sample from the format's own specification or+// from a vendor's published example, not one this parser produced. A sample+// built from the parser's output only proves the parser agrees with itself,+// and the defect this family has is mis-splitting a line nobody wrote down.++func TestRFC5424IsParsedIntoItsNamedFields(t *testing.T) {+ // RFC 5424 §6.5, Example 2, with the structured data from Example 3.+ line := "<165>1 2026-10-08T22:14:15.003Z mymachine.example.com evntslog 1234 ID47 " ++ `[exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"] ` + "\ufeff" + "An application event log entry..."++ msg, err := Parse(line)+ if err != nil {+ t.Fatalf("Parse: %v", err)+ }+ if msg.Format != FormatRFC5424 {+ t.Fatalf("format = %q, want rfc5424", msg.Format)+ }+ if msg.Hostname != "mymachine.example.com" || msg.AppName != "evntslog" || msg.ProcID != "1234" || msg.MsgID != "ID47" {+ t.Errorf("header fields wrong: host=%q app=%q proc=%q msgid=%q", msg.Hostname, msg.AppName, msg.ProcID, msg.MsgID)+ }+ if msg.Severity != "notice" || msg.Facility != "local4" {+ t.Errorf("priority 165 should be local4.notice, got %s.%s", msg.Facility, msg.Severity)+ }+ if got := msg.StructuredData["[email protected]"]; got != "1011" {+ t.Errorf("structured data eventID = %q, want 1011", got)+ }+ if !strings.HasPrefix(msg.Message, "An application event log entry") {+ t.Errorf("message = %q; the BOM should be stripped and nothing else", msg.Message)+ }+}++func TestAStructuredDataValueMayContainTheClosingBracket(t *testing.T) {+ // The reason the split counts brackets instead of using a regex: a+ // parameter value may contain an escaped `]`, and a reader that stops at+ // the first one truncates the structured data and puts the remainder in+ // the message.+ line := `<34>1 2026-10-08T22:14:15Z host app - - [origin software="rsyslogd" x-info="see [docs\]"] the message`++ msg, err := Parse(line)+ if err != nil {+ t.Fatalf("Parse: %v", err)+ }+ if got := msg.StructuredData["origin.x-info"]; got != "see [docs]" {+ t.Errorf("x-info = %q, want %q", got, "see [docs]")+ }+ if msg.Message != "the message" {+ t.Errorf("message = %q, want %q", msg.Message, "the message")+ }+}++func TestRFC3164IsParsedWithoutSwallowingTheMessage(t *testing.T) {+ // RFC 3164 §5.4, Example 1.+ line := `<34>Oct 11 22:14:15 mymachine su: 'su root' failed for lonvick on /dev/pts/8`++ msg, err := Parse(line)+ if err != nil {+ t.Fatalf("Parse: %v", err)+ }+ if msg.Format != FormatRFC3164 {+ t.Fatalf("format = %q, want rfc3164", msg.Format)+ }+ if msg.Hostname != "mymachine" || msg.AppName != "su" {+ t.Errorf("host = %q, app = %q", msg.Hostname, msg.AppName)+ }+ if msg.Message != "'su root' failed for lonvick on /dev/pts/8" {+ t.Errorf("message = %q", msg.Message)+ }+ if msg.Severity != "crit" {+ t.Errorf("priority 34 should be auth.crit, got %s.%s", msg.Facility, msg.Severity)+ }+}++func TestATagWithAPidIsSplit(t *testing.T) {+ msg, err := Parse(`<13>Oct 11 22:14:15 web01 sshd[4242]: Accepted publickey for root from 203.0.113.9`)+ if err != nil {+ t.Fatalf("Parse: %v", err)+ }+ if msg.AppName != "sshd" || msg.ProcID != "4242" {+ t.Errorf("app = %q, pid = %q", msg.AppName, msg.ProcID)+ }+}++func TestAColonInProseIsNotATag(t *testing.T) {+ // The defect this guards: "Error: connection refused" read as an app+ // name called "Error". An analyst then sees a process that does not+ // exist, which is worse than seeing no process at all.+ msg, err := Parse(`<13>Oct 11 22:14:15 web01 the database said: connection refused`)+ if err != nil {+ t.Fatalf("Parse: %v", err)+ }+ if msg.AppName != "" {+ t.Errorf("app name = %q; prose containing a colon is not a tag", msg.AppName)+ }+ if msg.Message != "the database said: connection refused" {+ t.Errorf("message = %q", msg.Message)+ }+}++func TestAnUnrecognisableLineKeepsItsText(t *testing.T) {+ // A device emitting bare text is still telling you something. Returning+ // an error here would drop the only copy.+ msg, err := Parse("a bare line from some appliance")+ if err != nil {+ t.Fatalf("Parse: %v", err)+ }+ if msg.Format != FormatUnstructured {+ t.Errorf("format = %q, want unstructured", msg.Format)+ }+ if msg.Message != "a bare line from some appliance" {+ t.Errorf("message = %q", msg.Message)+ }+}++func TestAMissingPriorityDefaultsToNoticeRatherThanEmergency(t *testing.T) {+ // A zero priority decodes as kern.emerg — the most severe value there+ // is. A device that simply omitted the field would page someone.+ msg, err := Parse("Oct 11 22:14:15 web01 sshd: hello")+ if err != nil {+ t.Fatalf("Parse: %v", err)+ }+ if msg.Severity != "notice" {+ t.Errorf("severity = %q, want notice (RFC 3164's stated default), not %q", msg.Severity, "emerg")+ }+}++func TestAnEmptyLineIsNotAParseFailure(t *testing.T) {+ // A zero-length UDP datagram is a keepalive. Counting it as malformed+ // makes the malformed-line metric useless.+ if _, err := Parse(" "); err != ErrEmpty {+ t.Errorf("err = %v, want ErrEmpty", err)+ }+}++func TestAnOversizedLineIsRefusedRatherThanTruncated(t *testing.T) {+ if _, err := Parse(strings.Repeat("x", MaxLineBytes+1)); err == nil {+ t.Error("an oversized line should be refused, not silently truncated")+ }+}++func TestCEFInsideASyslogLineIsParsed(t *testing.T) {+ // The header example from the CEF implementation standard, with a+ // syslog header in front of it the way a real forwarder sends it.+ line := `<134>Oct 11 22:14:15 fw01 CEF:0|Security|threatmanager|1.0|100|worm successfully stopped|10|src=10.0.0.1 dst=2.1.2.2 spt=1232`++ msg, err := Parse(line)+ if err != nil {+ t.Fatalf("Parse: %v", err)+ }+ if msg.Format != FormatCEF {+ t.Fatalf("format = %q, want cef", msg.Format)+ }+ for key, want := range map[string]string{+ "DeviceVendor": "Security",+ "DeviceProduct": "threatmanager",+ "SignatureID": "100",+ "Name": "worm successfully stopped",+ "Severity": "10",+ "src": "10.0.0.1",+ "dst": "2.1.2.2",+ "spt": "1232",+ } {+ if got := msg.Extension[key]; got != want {+ t.Errorf("%s = %q, want %q", key, got, want)+ }+ }+ if msg.Hostname != "fw01" {+ t.Errorf("the syslog header in front of the CEF payload should still parse; hostname = %q", msg.Hostname)+ }+}++func TestACEFValueMayContainSpaces(t *testing.T) {+ // This is the whole difficulty of the extension format: `msg=connection+ // refused src=10.0.0.1` has to yield two pairs, not four tokens. A+ // split on spaces loses the message, which is the field an analyst+ // reads first.+ line := `CEF:0|v|p|1.0|1|Name|5|msg=connection refused by policy src=10.0.0.1 suser=alice`++ msg, err := Parse(line)+ if err != nil {+ t.Fatalf("Parse: %v", err)+ }+ if got := msg.Extension["msg"]; got != "connection refused by policy" {+ t.Errorf("msg = %q, want %q", got, "connection refused by policy")+ }+ if msg.Extension["src"] != "10.0.0.1" || msg.Extension["suser"] != "alice" {+ t.Errorf("later pairs lost: src=%q suser=%q", msg.Extension["src"], msg.Extension["suser"])+ }+}++func TestAnEscapedPipeStaysInsideItsCEFHeaderField(t *testing.T) {+ msg, err := Parse(`CEF:0|Vendor\|Inc|product|1.0|42|a name|3|src=10.0.0.1`)+ if err != nil {+ t.Fatalf("Parse: %v", err)+ }+ if got := msg.Extension["DeviceVendor"]; got != "Vendor|Inc" {+ t.Errorf("DeviceVendor = %q, want %q", got, "Vendor|Inc")+ }+ if msg.Extension["SignatureID"] != "42" {+ t.Errorf("an escaped pipe shifted every later field: SignatureID = %q", msg.Extension["SignatureID"])+ }+}++func TestLEEF1UsesTabsByDefault(t *testing.T) {+ line := "<134>Oct 11 22:14:15 qr01 LEEF:1.0|Lancope|StealthWatch|1.0|41|src=192.0.2.1\tdst=172.16.0.1\tsev=5"++ msg, err := Parse(line)+ if err != nil {+ t.Fatalf("Parse: %v", err)+ }+ if msg.Format != FormatLEEF {+ t.Fatalf("format = %q, want leef", msg.Format)+ }+ if msg.Extension["src"] != "192.0.2.1" || msg.Extension["sev"] != "5" || msg.Extension["EventID"] != "41" {+ t.Errorf("attributes wrong: %#v", msg.Extension)+ }+}++func TestLEEF2HonoursTheDelimiterTheHeaderDeclares(t *testing.T) {+ // The reason LEEF needs its own parser rather than a tab split: a+ // caret-delimited line read as tab-delimited puts every attribute into+ // one key, and the record still looks structurally fine.+ line := `LEEF:2.0|Vendor|Product|1.0|4711|^|src=10.0.0.1^dst=10.0.0.2^sev=8`++ msg, err := Parse(line)+ if err != nil {+ t.Fatalf("Parse: %v", err)+ }+ if msg.Extension["src"] != "10.0.0.1" || msg.Extension["dst"] != "10.0.0.2" || msg.Extension["sev"] != "8" {+ t.Errorf("the declared ^ delimiter was ignored: %#v", msg.Extension)+ }+}++func TestLEEF2AcceptsAHexDelimiter(t *testing.T) {+ line := "LEEF:2.0|Vendor|Product|1.0|4711|x09|src=10.0.0.1\tsev=2"++ msg, err := Parse(line)+ if err != nil {+ t.Fatalf("Parse: %v", err)+ }+ if msg.Extension["src"] != "10.0.0.1" || msg.Extension["sev"] != "2" {+ t.Errorf("x09 should resolve to a tab: %#v", msg.Extension)+ }+}++func TestA3164TimestampIsGivenAYearWithoutJumpingForward(t *testing.T) {+ // RFC 3164 carries no year. Stamping the current one is the+ // conventional move and it is wrong at a year boundary: a December log+ // read in January is last year's. One day of slack absorbs clock skew+ // without re-dating an ordinary same-day message.+ future := time.Now().UTC().AddDate(0, 0, 3)+ line := "<13>" + future.Format("Jan _2 15:04:05") + " host app: hello"++ msg, err := Parse(line)+ if err != nil {+ t.Fatalf("Parse: %v", err)+ }+ parsed, err := time.Parse(time.RFC3339Nano, msg.Timestamp)+ if err != nil {+ t.Fatalf("timestamp %q is not RFC 3339: %v", msg.Timestamp, err)+ }+ if parsed.After(time.Now().UTC().Add(24 * time.Hour)) {+ t.Errorf("timestamp %s is in the future; a year-boundary line was dated forward", msg.Timestamp)+ }+}++func TestTheRawLineIsAlwaysKept(t *testing.T) {+ // Every parse here is a lossy summary, and an appliance emitting+ // something this parser misreads is itself the finding.+ line := `<34>Oct 11 22:14:15 mymachine su: something`+ msg, err := Parse(line)+ if err != nil {+ t.Fatalf("Parse: %v", err)+ }+ if msg.Raw != line {+ t.Errorf("raw = %q, want the line exactly as it arrived", msg.Raw)+ }+ if got := msg.Fields()["raw"]; got != line {+ t.Errorf("Fields()[raw] = %q", got)+ }+}diff --git a/services/ingest/internal/syslog/sink.go b/services/ingest/internal/syslog/sink.gonew file mode 100644index 000000000..dccc34690--- /dev/null+++ b/services/ingest/internal/syslog/sink.go@@ -0,0 +1,127 @@+package syslog++import (+ "context"+ "fmt"+ "time"++ "github.com/beenuar/aisoc/services/ingest/internal/inbox"+ "github.com/beenuar/aisoc/services/ingest/internal/normalizer"+ "github.com/google/uuid"+)++// TokenResolver is the half of inbox.Store this package needs. Narrowed to+// one method so a test can supply one without a Postgres pool.+type TokenResolver interface {+ Resolve(ctx context.Context, token string) (*inbox.Token, error)+}++// Publisher is the half of publisher.Publisher this package needs.+type Publisher interface {+ PublishBatch(ctx context.Context, events []*normalizer.NormalizedEvent) error+}++// TemplateSource resolves a template id to a template.+type TemplateSource interface {+ Get(id string) (*inbox.Template, error)+}++// RequiredTemplateID is the template a syslog token must be minted with.+//+// One token covers all four formats deliberately. A single appliance+// commonly emits RFC 3164 for its own daemons and CEF for its security+// events, and making the operator mint two tokens and run two ports for one+// device would be a configuration no vendor's documentation describes.+const RequiredTemplateID = "syslog"++// : Which OCSF template each parsed format is projected through. CEF and+// : LEEF carry vendor, product and severity of their own, so projecting them+// : through the plain syslog template would throw that away and file an+// : ArcSight finding as an anonymous log line.+var templateForFormat = map[Format]string{+ FormatRFC5424: "syslog",+ FormatRFC3164: "syslog",+ FormatUnstructured: "syslog",+ FormatCEF: "cef-syslog",+ FormatLEEF: "leef-syslog",+}++// PublishingSink turns parsed messages into OCSF events on Kafka.+//+// The tenant comes from the configured token and from nowhere else. That is+// the whole reason this is a token and not a field: syslog has no+// authenticated principal and no header, so anything taken off the wire is+// caller-controlled, and a tenant a sender can set is a cross-tenant write+// waiting for someone to notice. One listener therefore serves one tenant,+// and a multi-tenant deployment runs one port per tenant — stated in the+// setup guide rather than left to be discovered.+type PublishingSink struct {+ tokens TokenResolver+ templates TemplateSource+ pub Publisher+ token string+}++// NewPublishingSink wires the sink. The token is not resolved here: a+// resolver needs a context and a database, and failing construction because+// Postgres was slow to start would take the whole ingest service down with+// it. It is resolved on the first delivery instead, where a failure is a+// counted, named error.+func NewPublishingSink(tokens TokenResolver, templates TemplateSource, pub Publisher, token string) (*PublishingSink, error) {+ if tokens == nil || templates == nil || pub == nil {+ return nil, fmt.Errorf("syslog: sink needs a token resolver, a template source and a publisher")+ }+ if token == "" {+ return nil, fmt.Errorf("syslog: no ingest token configured, so there is no tenant to attribute messages to")+ }+ return &PublishingSink{tokens: tokens, templates: templates, pub: pub, token: token}, nil+}++// Deliver projects a batch and publishes it.+func (s *PublishingSink) Deliver(ctx context.Context, messages []*Message) error {+ if len(messages) == 0 {+ return nil+ }+ tok, err := s.tokens.Resolve(ctx, s.token)+ if err != nil {+ // Named rather than generic: a revoked token and an unreachable+ // database are different problems and only one of them is fixed by+ // waiting.+ return fmt.Errorf("syslog: resolving the configured ingest token: %w", err)+ }+ if tok.TemplateID != RequiredTemplateID {+ return fmt.Errorf(+ "syslog: the configured token was minted with template %q; the listener needs one minted with %q",+ tok.TemplateID, RequiredTemplateID,+ )+ }++ receivedAt := time.Now().UTC().Format(time.RFC3339Nano)+ tenantID := tok.TenantID.String()+ events := make([]*normalizer.NormalizedEvent, 0, len(messages))++ for _, msg := range messages {+ templateID := templateForFormat[msg.Format]+ if templateID == "" {+ templateID = RequiredTemplateID+ }+ tmpl, err := s.templates.Get(templateID)+ if err != nil {+ // Falling back to the plain template would file a CEF finding+ // as an anonymous log line, which is worse than refusing: the+ // record would look complete.+ return fmt.Errorf("syslog: this build has no %q template, so %s messages cannot be normalised", templateID, msg.Format)+ }+ connectorRef := "syslog:" + templateID+ ocsf := tmpl.Apply(msg.Fields(), tenantID, connectorRef, receivedAt)+ events = append(events, &normalizer.NormalizedEvent{+ ID: uuid.NewString(),+ ConnectorID: connectorRef,+ TenantID: tenantID,+ OcsfEvent: ocsf,+ NormalizationVersion: "syslog/1.0",+ })+ }++ return s.pub.PublishBatch(ctx, events)+}diff --git a/services/ingest/internal/syslog/sink_test.go b/services/ingest/internal/syslog/sink_test.gonew file mode 100644index 000000000..1ffc49244--- /dev/null+++ b/services/ingest/internal/syslog/sink_test.go@@ -0,0 +1,219 @@+package syslog++import (+ "context"+ "errors"+ "strings"+ "testing"++ "github.com/beenuar/aisoc/services/ingest/internal/inbox"+ "github.com/beenuar/aisoc/services/ingest/internal/normalizer"+ "github.com/google/uuid"+)++type fakeTokens struct {+ token *inbox.Token+ err error+ asked []string+}++func (f *fakeTokens) Resolve(_ context.Context, token string) (*inbox.Token, error) {+ f.asked = append(f.asked, token)+ if f.err != nil {+ return nil, f.err+ }+ return f.token, nil+}++type capturingPublisher struct {+ batches [][]*normalizer.NormalizedEvent+ err error+}++func (p *capturingPublisher) PublishBatch(_ context.Context, events []*normalizer.NormalizedEvent) error {+ if p.err != nil {+ return p.err+ }+ p.batches = append(p.batches, events)+ return nil+}++// realTemplates loads the templates that ship in the binary, so these tests+// exercise the YAML the service actually applies rather than a stub whose+// field map agrees with whatever the test expects.+func realTemplates(t *testing.T) *inbox.Registry {+ t.Helper()+ r := inbox.NewRegistry()+ if err := r.LoadEmbedded(); err != nil {+ t.Fatalf("LoadEmbedded: %v", err)+ }+ return r+}++func tenantToken(template string) *inbox.Token {+ return &inbox.Token{+ Token: "tok_abc",+ TenantID: uuid.MustParse("3fa85f64-5717-4562-b3fc-2c963f66afa6"),+ TemplateID: template,+ Label: "edge firewalls",+ }+}++func TestTheTenantComesFromTheTokenAndNowhereElse(t *testing.T) {+ // The reason the listener takes a token rather than reading a tenant+ // off the wire: syslog has no authenticated principal and no header, so+ // a tenant a sender could set would be a cross-tenant write requiring+ // no attacker effort at all.+ tokens := &fakeTokens{token: tenantToken(RequiredTemplateID)}+ pub := &capturingPublisher{}+ sink, err := NewPublishingSink(tokens, realTemplates(t), pub, "tok_abc")+ if err != nil {+ t.Fatalf("NewPublishingSink: %v", err)+ }++ // A sender doing its best to claim another tenant, in every field it+ // controls: the hostname, the app name, the message and a structured+ // data parameter.+ msg, err := Parse(`<34>1 2026-10-08T22:14:15Z 00000000-0000-0000-0000-000000000999 tenant_id - - ` ++ `[meta tenant_id="00000000-0000-0000-0000-000000000999"] tenant_id=00000000-0000-0000-0000-000000000999`)+ if err != nil {+ t.Fatalf("Parse: %v", err)+ }+ if err := sink.Deliver(context.Background(), []*Message{msg}); err != nil {+ t.Fatalf("Deliver: %v", err)+ }++ if len(pub.batches) != 1 || len(pub.batches[0]) != 1 {+ t.Fatalf("expected one event, got %#v", pub.batches)+ }+ got := pub.batches[0][0]+ if got.TenantID != "3fa85f64-5717-4562-b3fc-2c963f66afa6" {+ t.Errorf("tenant = %q; it must come from the token, not from the message", got.TenantID)+ }+ metadata, _ := got.OcsfEvent["metadata"].(map[string]any)+ if metadata["tenant_uid"] != "3fa85f64-5717-4562-b3fc-2c963f66afa6" {+ t.Errorf("metadata.tenant_uid = %v", metadata["tenant_uid"])+ }+}++func TestEachFormatIsProjectedThroughItsOwnTemplate(t *testing.T) {+ // A CEF line carries a vendor, a product and a severity of its own.+ // Projecting it through the plain syslog template would throw all three+ // away and file an ArcSight finding as an anonymous log line — which+ // still looks like a complete record.+ tokens := &fakeTokens{token: tenantToken(RequiredTemplateID)}+ pub := &capturingPublisher{}+ sink, _ := NewPublishingSink(tokens, realTemplates(t), pub, "tok_abc")++ plain, _ := Parse("<34>Oct 11 22:14:15 mymachine su: failed")+ cef, _ := Parse("<134>Oct 11 22:14:15 fw01 CEF:0|Security|threatmanager|1.0|100|worm stopped|10|src=10.0.0.1")+ leef, _ := Parse("<134>Oct 11 22:14:15 qr01 LEEF:1.0|Lancope|StealthWatch|1.0|41|src=192.0.2.1\tsev=8")++ if err := sink.Deliver(context.Background(), []*Message{plain, cef, leef}); err != nil {+ t.Fatalf("Deliver: %v", err)+ }+ events := pub.batches[0]+ if len(events) != 3 {+ t.Fatalf("want 3 events, got %d", len(events))+ }++ // The plain line stays in category 1: the promoter promotes category 2+ // unconditionally, so filing routine daemon chatter as a finding would+ // make every line an alert.+ if got := events[0].OcsfEvent["class_uid"]; got != 1007 {+ t.Errorf("plain syslog class_uid = %v, want 1007", got)+ }+ if got := events[1].OcsfEvent["class_uid"]; got != 2001 {+ t.Errorf("CEF class_uid = %v, want 2001", got)+ }+ if got := events[2].OcsfEvent["class_uid"]; got != 2001 {+ t.Errorf("LEEF class_uid = %v, want 2001", got)+ }+ if events[1].ConnectorID != "syslog:cef-syslog" || events[2].ConnectorID != "syslog:leef-syslog" {+ t.Errorf("connector refs: %q, %q", events[1].ConnectorID, events[2].ConnectorID)+ }++ metadata, _ := events[1].OcsfEvent["metadata"].(map[string]any)+ product, _ := metadata["product"].(map[string]any)+ if product["vendor_name"] != "Security" {+ t.Errorf("the CEF vendor was lost: %v", product)+ }+}++func TestAVendorSeverityReachesTheOCSFLadder(t *testing.T) {+ tokens := &fakeTokens{token: tenantToken(RequiredTemplateID)}+ pub := &capturingPublisher{}+ sink, _ := NewPublishingSink(tokens, realTemplates(t), pub, "tok_abc")++ // syslog `crit` and CEF severity 10 both mean the top of their own+ // ladder. Neither may be collapsed into the tier below it.+ critical, _ := Parse("<34>Oct 11 22:14:15 mymachine su: failed")+ cef, _ := Parse("CEF:0|v|p|1.0|1|Name|10|src=10.0.0.1")+ if err := sink.Deliver(context.Background(), []*Message{critical, cef}); err != nil {+ t.Fatalf("Deliver: %v", err)+ }+ events := pub.batches[0]+ if got := events[0].OcsfEvent["severity_id"]; got != 5 {+ t.Errorf("syslog crit severity_id = %v, want 5", got)+ }+ if got := events[1].OcsfEvent["severity_id"]; got != 6 {+ t.Errorf("CEF severity 10 severity_id = %v, want 6", got)+ }+}++func TestATokenMintedWithTheWrongTemplateIsRefused(t *testing.T) {+ // Not silently reinterpreted. A token minted for a different vendor's+ // webhook would otherwise publish syslog under that vendor's name.+ tokens := &fakeTokens{token: tenantToken("pagerduty")}+ pub := &capturingPublisher{}+ sink, _ := NewPublishingSink(tokens, realTemplates(t), pub, "tok_abc")++ msg, _ := Parse("<34>Oct 11 22:14:15 mymachine su: failed")+ err := sink.Deliver(context.Background(), []*Message{msg})+ if err == nil {+ t.Fatal("a token minted with the wrong template should be refused")+ }+ if !strings.Contains(err.Error(), RequiredTemplateID) {+ t.Errorf("the error should name the template the listener needs: %v", err)+ }+ if len(pub.batches) != 0 {+ t.Error("nothing should have been published")+ }+}++func TestARevokedTokenIsNamedRatherThanGeneric(t *testing.T) {+ // A revoked token and an unreachable database are different problems+ // and only one of them is fixed by waiting.+ tokens := &fakeTokens{err: errors.New("token revoked")}+ pub := &capturingPublisher{}+ sink, _ := NewPublishingSink(tokens, realTemplates(t), pub, "tok_abc")++ msg, _ := Parse("<34>Oct 11 22:14:15 mymachine su: failed")+ err := sink.Deliver(context.Background(), []*Message{msg})+ if err == nil || !strings.Contains(err.Error(), "token revoked") {+ t.Errorf("err = %v; it should carry the resolver's own reason", err)+ }+}++func TestASinkWithNoTokenIsRefusedAtConstruction(t *testing.T) {+ // Starting a listener with no token would bind a port and attribute+ // every message to nobody.+ if _, err := NewPublishingSink(&fakeTokens{}, realTemplates(t), &capturingPublisher{}, ""); err == nil {+ t.Fatal("a sink with no token should be refused")+ }+}++func TestTheRawLineSurvivesIntoTheEvent(t *testing.T) {+ tokens := &fakeTokens{token: tenantToken(RequiredTemplateID)}+ pub := &capturingPublisher{}+ sink, _ := NewPublishingSink(tokens, realTemplates(t), pub, "tok_abc")++ line := "<34>Oct 11 22:14:15 mymachine su: 'su root' failed for lonvick"+ msg, _ := Parse(line)+ if err := sink.Deliver(context.Background(), []*Message{msg}); err != nil {+ t.Fatalf("Deliver: %v", err)+ }+ if got := pub.batches[0][0].OcsfEvent["raw_data"]; got != line {+ t.Errorf("raw_data = %v; every parse here is a lossy summary and the original is the only appeal", got)+ }+}diff --git a/services/ingest/main.go b/services/ingest/main.goindex 1e090321d..f456c7f6a 100644--- a/services/ingest/main.go+++ b/services/ingest/main.go@@ -9,6 +9,7 @@ import ("net/http""os""os/signal"+ "strings""syscall""time"@@ -23,6 +24,7 @@ import ("github.com/beenuar/aisoc/services/ingest/internal/normalizer""github.com/beenuar/aisoc/services/ingest/internal/publisher""github.com/beenuar/aisoc/services/ingest/internal/server"+ "github.com/beenuar/aisoc/services/ingest/internal/syslog""github.com/beenuar/aisoc/services/ingest/internal/telemetry""github.com/jackc/pgx/v5/pgxpool""github.com/rs/zerolog"@@ -296,6 +298,74 @@ func main() {log.Info().Msg("graph_ws: disabled (AISOC_GRAPH_WS_ENABLED!=true)")}+ // Depth plan 4.2 — the syslog listener. Off unless the operator asks+ // for it, because binding 514 needs a decision they have to make.+ //+ // Needs the same two things the inbox needs (a token store to resolve+ // the tenant, a template registry to normalise) plus a publisher, and+ // refuses to start rather than listening without them: a bound socket+ // that drops everything is indistinguishable from a working one.+ var syslogListener *syslog.Listener+ if cfg.SyslogEnabled {+ switch {+ case tokenStore == nil:+ log.Error().Msg("syslog: not started — DATABASE_DSN is unset, so the configured ingest token cannot be resolved to a tenant")+ case cfg.SyslogToken == "":+ log.Error().Msg("syslog: not started — AISOC_SYSLOG_TOKEN is unset, so there is no tenant to attribute messages to. " ++ "Mint one with POST /api/v1/inbox/tokens using the 'syslog' template")+ default:+ registry := inbox.NewRegistry()+ if err := registry.LoadEmbedded(); err != nil {+ log.Error().Err(err).Msg("syslog: not started — embedded templates failed to load")+ break+ }+ if err := registry.Load(cfg.InboxTemplatesDir); err != nil {+ log.Warn().Err(err).Str("dir", cfg.InboxTemplatesDir).Msg("syslog: template overrides on disk could not be read")+ }+ sink, err := syslog.NewPublishingSink(tokenStore, registry, pub, cfg.SyslogToken)+ if err != nil {+ log.Error().Err(err).Msg("syslog: not started")+ break+ }+ listener, err := syslog.New(syslog.Config{+ UDPAddr: cfg.SyslogUDPAddr,+ TCPAddr: cfg.SyslogTCPAddr,+ BatchSize: cfg.SyslogBatchSize,+ FlushInterval: time.Duration(cfg.SyslogFlushMs) * time.Millisecond,+ MaxConnections: cfg.SyslogMaxConnections,+ }, sink)+ if err != nil {+ // Not fatal: the HTTP ingest path is this service's+ // primary job and must not refuse traffic because 514 was+ // taken. Loud, though — the operator asked for a listener+ // and does not have one.+ log.Error().Err(err).Msg("syslog: listener disabled")+ break+ }+ syslogListener = listener+ udpAddr, tcpAddr := listener.Addrs()+ // /readyz names this listener and its state. A receiver nobody+ // is sending to and a receiver that failed to bind look the+ // same from outside unless readiness says which.+ h.RegisterSubscription("syslog", func() handler.SubscriptionStatus {+ state := listener.Health()+ bound := state.UDPBound || state.TCPBound+ return handler.SubscriptionStatus{+ Attached: bound,+ NotResolving: !bound,+ Detail: syslogDetail(state),+ }+ })+ log.Info().+ Str("udp", udpAddr).+ Str("tcp", tcpAddr).+ Int("batch_size", cfg.SyslogBatchSize).+ Msg("syslog: listener accepting RFC 5424, RFC 3164, CEF and LEEF")+ }+ } else {+ log.Info().Msg("syslog: disabled (AISOC_SYSLOG_ENABLED!=true)")+ }+srv := server.New(cfg, h, inboxHandler, graphWSServer)// Graceful shutdown@@ -307,6 +377,18 @@ func main() {defer graphWSBroker.Stop()}+ if syslogListener != nil {+ syslogListener.Start(ctx)+ defer func() {+ // Given its own short deadline rather than ctx: ctx is already+ // cancelled by the time deferred shutdown runs, and a drain on+ // a cancelled context publishes nothing.+ drainCtx, drainCancel := context.WithTimeout(context.Background(), 5*time.Second)+ syslogListener.Stop(drainCtx)+ drainCancel()+ }()+ }+// Drain VulnMatches → Kafka in a background goroutineif norm.VulnMatches != nil {go func() {@@ -352,3 +434,27 @@ func init() {// Set version in health check_ = fmt.Sprintf("aisoc-ingest/%s", Version)}++// syslogDetail turns listener state into the one sentence /readyz shows.+//+// "attached" alone would be a lie of omission for a receiver: a syslog+// listener is legitimately silent for hours, so the detail carries what it+// has bound, what it has received and what it last failed on — which is+// what separates "nothing is being sent" from "everything is being dropped".+func syslogDetail(state syslog.State) string {+ if !state.UDPBound && !state.TCPBound {+ return "no transport bound"+ }+ bound := make([]string, 0, 2)+ if state.UDPBound {+ bound = append(bound, "udp "+state.UDPAddr)+ }+ if state.TCPBound {+ bound = append(bound, "tcp "+state.TCPAddr)+ }+ detail := fmt.Sprintf("listening on %s; received %d, rejected %d", strings.Join(bound, ", "), state.Received, state.Rejected)+ if state.LastError != "" {+ detail += "; last error: " + state.LastError+ }+ return detail+}