Skip to content

LordOS — a free, fast and secure GNU/Linux distribution based on Debian, with its own desktop environment: display server and Wayland compositor in C, a React UI engine on QuickJS, and private native apps.

lord-netperm: count each app's traffic in the kernel, TCP and UDP (QUIC)

Federico Pereirad804bdfd417d
diff --git a/docs/FIREWALL.md b/docs/FIREWALL.md
index 6a7a500..5b9fe91 100644
--- a/docs/FIREWALL.md
+++ b/docs/FIREWALL.md
@@ -547,7 +547,7 @@ with gcc 14 on Linux.
```
server/src/netperm/np_shared.h map types and the decision (np_decide), identical in BPF and C
-server/src/netperm/np.bpf.c the 7 cgroup BPF programs (restricted C, clang -target bpf)
+server/src/netperm/np.bpf.c the cgroup BPF programs (restricted C, clang -target bpf)
server/src/netperm/bpf2c.c build tool: clang's .o → build/gen/np_bpf.h
server/src/netperm/bpfsys.{c,h} direct bpf() syscall: maps, load, bpf_link, ring
server/src/netperm/npcompile.{c,h} rules → map entries (pure C, with a test)
@@ -605,6 +605,7 @@ tests/netperm_bench.c, netperm_measure.sh measurements
| `cgroup/sock_create` | "no network": the app can't even create an AF_INET/AF_INET6 socket, so it can't ping either |
| `cgroup/dev` | "no camera", "no USB", "no screen capture" (`/dev/fb*`) and "no microphone" (ALSA capture nodes): device `open()`/`mknod()` (see [Linux apps](#linux-apps)) |
| `cgroup/lsm_socket_create` | "no Bluetooth": `socket(AF_BLUETOOTH)`, a per-cgroup LSM program (`BPF_LSM_CGROUP`) |
+| `cgroup/skb_ingress`, `skb_egress` | nothing: **traffic accounting** for the Monitor (bytes per app and per remote address, see below). They always let the packet through |
Each program does the following:
@@ -678,7 +679,8 @@ is denied by the union all the same. The decision helper (`np_decide`) is unchan
| `np_union` | hash | `user@<uid>.service` id → strict union of the user's restricted apps |
| `np_lpm4`, `np_lpm6` | LPM trie | (app index, network) → ordered list of rules |
| `np_seen` | LRU | (cgroup, address, port, protocol, connection direction, verdict) → last time reported |
-| `np_counts` | hash | cgroup id → allowed, denied |
+| `np_counts` | hash | cgroup id → allowed, denied, bytes received, bytes sent |
+| `np_flows` | LRU (4096) | (cgroup id, remote address) → bytes received, bytes sent |
| `np_events` | ring buffer | new connections (and device or Bluetooth denials) to the service |
| `np_snd` | hash | every ALSA minor in `/sys/class/sound` → 2 for a capture PCM, 1 for the rest |
@@ -687,8 +689,28 @@ is denied by the union all the same. The decision helper (`np_decide`) is unchan
inflated: getaddrinfo does a UDP connect for each address to sort them (RFC 6724).
The service gets one notice per new connection, through the ring, only if that same connection
-wasn't reported in the last 5 s. Nothing is per packet: no hook runs per packet and no packet is
-processed in user space. The exception is the DNS responses of the source without resolved, below.
+wasn't reported in the last 5 s. No decision is per packet and no packet is processed in user
+space (the exception is the DNS responses of the source without resolved, below).
+
+**Traffic accounting (0.8.1).** The Monitor used to add up `tcp_info` of the sockets open at that
+moment, so UDP — and with it QUIC (HTTP/3, video streaming) — counted nothing and a closed
+connection took its bytes with it: a browser playing a video showed a few KB. Now two
+`cgroup_skb` programs (ingress and egress) count every packet of the session's apps:
+
+- the app is the socket's cgroup at the scope level (`bpf_skb_ancestor_cgroup_id`), the same id
+ as the other programs; they only **add** to an `np_counts` entry that `lord-netperm` creates when
+ the scope appears, so nothing outside an app (login screen, the system) is counted or stored;
+- the bytes are the IP length of each skb (one GRO/GSO segment, so a fast TCP flow is a few
+ thousand calls per second, not one per wire packet); the remote address comes from the IP header
+ (source going in, destination going out) into the `np_flows` LRU;
+- loopback (`ifindex` 1) isn't counted, so the totals compare with the interfaces' counters;
+- they are attached only to `lordos.service`, not to `user@<uid>.service` (`np_attach_user`), and
+ are optional: a kernel that refuses them leaves the firewall working and the Monitor falls back
+ to `tcp_info` (`STATUS` says `"bytes": false`).
+
+Every second `lord-netperm` reads `np_counts` (one lookup per app) and, only if bytes moved, the
+whole `np_flows` table in one `BPF_MAP_LOOKUP_BATCH` call, works out the rates and sends `COUNTS`
+(below). With nothing moving it reads no table and sends nothing.
### From rules to maps (`npcompile.c`)
@@ -789,9 +811,13 @@ connect to the IPs of `permitido.com` even if another one requested them.
It reads it when the session appears and on every `RELOAD`.
- **From lord-netperm to lord-server:**
- - `STATUS`: `{bpf, dns, error}`;
+ - `STATUS`: `{bpf, devices, bluetooth, bytes, dns, error}` (`bytes`: the accounting programs are loaded);
- `EVENT`: `{app, v, dir, proto, addr, port, by, rule, flag, record}`, a new connection;
- - `COUNTS`: `{app: [allowed, denied]}`, every second, only if it changed.
+ - `COUNTS`: `{app: [allowed, denied, rx, tx, rxRate, txRate, [[addr, rx, tx, rxRate, txRate], ...]]}`,
+ every second, only if something changed. `rx`/`tx` are the app's bytes since its scope
+ appeared, the rates are bytes per second over the last second, and the list holds the app's
+ 64 busiest addresses (those active in the last 2 minutes). Readers that only know
+ `[allowed, denied]` keep working.
If nobody reads, notices are dropped. Counters are sent again when they change.
@@ -910,13 +936,45 @@ over TCP on 127.0.0.1.
- **The per-connection cost** shows in UDP, because its connect does no handshake: about
**0.5 µs**. In TCP it stays within loopback noise.
-- **Throughput doesn't change:** the differences are within run-to-run noise. No hook runs per
- packet.
+- **Throughput doesn't change:** the differences are within run-to-run noise. (These loopback
+ numbers predate the accounting programs of 0.8.1, which return at once on loopback.)
- **The compositor:** lord-server's main thread, with the Monitor open on Network (which calls
`lord.fw.poll` once per second), used 3.20 % of a core with lord-netperm on, 3.20 % off and
3.13 % on again. 30 s each. Nothing of the firewall runs in the frame: `poll` reads a socket that
doesn't block. Per-frame times weren't measured separately.
+### Measurements of the traffic accounting (0.8.1, VM 118, same CPU, kernel 6.12)
+
+**Accuracy**, against `/proc/net/dev` (eth0, which also counts the 14-byte Ethernet header and
+any other traffic of the machine) while only the app moved data:
+
+| App (its own scope) | Counted by the app | eth0 | Ratio |
+|---|---|---|---|
+| `curl --http3-only`, a 20 MB file over QUIC | 20.31 MB received | 20.55 MB | 98.8 % |
+| Chromium playing a video on a video site (QUIC, 60 s) | 4.54 MB received | 4.60 MB | 98.7 % |
+| `iperf3 -u -b 400M -l 1200`, 8 s, to a veth peer | 409,385,865 B sent | — | exact: 400,050,000 B payload + 333,375 × 28 B of IP/UDP headers |
+
+Before, the same QUIC download showed "—" (0 B): UDP had no `tcp_info`.
+
+**Cost**, `iperf3` through a veth pair into a private network namespace (eth0 untouched), the
+client inside an app scope; system CPU as % of one core over 15 s, two runs with the accounting
+programs:
+
+| | 0.8.0 `lord-netperm` | 0.8.1 (accounting) |
+|---|---|---|
+| TCP, ~24 Gbit/s (GSO/GRO: few, large skbs) | 162.0 / 161.6 % | 164.5 / 163.1 %, 162.4 / 162.3 % |
+| UDP 1 Gbit/s, 1200 B datagrams (104,000 packets/s), sending | 91.5 % | 94.7 %, 100.2 % |
+| UDP 1 Gbit/s, 1200 B datagrams, receiving | 100.2 % | 108.7 %, 106.9 % |
+
+- TCP: no measurable difference (throughput 22–25 Gbit/s run to run, either way).
+- UDP at 104,000 packets per second: about 5 % of one core more, roughly 0.5 µs per packet (two
+ map lookups, two atomic adds, one header read). A QUIC video at a few MB/s is a few thousand
+ packets per second: well under 0.1 %.
+- `lord-netperm` itself: 0.00–0.01 s of CPU per 15 s run (one counter read per app per second,
+ plus one batch read of `np_flows` only while bytes move).
+- Script: `bench-monred-118.sh` in the scratchpad (veth `npb0`/`npb1`, namespace `npb`, both
+ removed afterwards); `iperf3` was installed on VM 118 for this (`VM-SETUP.md`).
+
### Known limits
- **App scopes are root's now** (was: "Scopes belong to the user"). `lord-sessiond` creates the
diff --git a/sdk/monitor.js b/sdk/monitor.js
index 714c452..0233841 100644
--- a/sdk/monitor.js
+++ b/sdk/monitor.js
@@ -99,7 +99,8 @@ export function useMonitor({ procs = false, conns = false, geo = true, everyMs =
// ---- per-app firewall (lord.fw, docs/FIREWALL.md) ----
// lord-netperm (root) applies the rules in the kernel and counts each app's new connections;
-// lord-server passes them to JS. firewallState: { status, counts: { app: [allowed, denied] },
+// lord-server passes them to JS. firewallState: { status, counts: { app: [allowed, denied, rx, tx,
+// rxRate, txRate, [[addr, rx, tx, rxRate, txRate], ...]] } (the bytes when status.bytes),
// events: [the last 200 new connections], rev }. rev goes up when a rule changes.
const fwApi = () => (typeof lord !== 'undefined' && lord.fw) || null;
export const firewallState = createStore({ status: null, counts: {}, events: [], rev: 0 });
diff --git a/server/src/netperm/bpfsys.c b/server/src/netperm/bpfsys.c
index 4ef03bd..a32b3cb 100644
--- a/server/src/netperm/bpfsys.c
+++ b/server/src/netperm/bpfsys.c
@@ -11,6 +11,8 @@
#include <sys/syscall.h>
#include <unistd.h>
+#define ENOTSUPP_FALLBACK 524 /* the kernel's internal ENOTSUPP, which BPF calls may return */
+
static int sys_bpf(int cmd, union bpf_attr *a) {
return (int)syscall(__NR_bpf, cmd, a, sizeof *a);
}
@@ -67,6 +69,49 @@ int bpfs_map_delete(int fd, const void *key) {
return sys_bpf(BPF_MAP_DELETE_ELEM, &a);
}
+int bpfs_map_dump(int fd, void *keys, uint32_t ksz, void *vals, uint32_t vsz, uint32_t max) {
+ if (!max) return 0;
+ union bpf_attr a;
+ uint32_t got = 0;
+ uint64_t batch = 0; /* the kernel's cursor (u32 for hash maps; 8 bytes is enough) */
+ int first = 1, batch_ok = 1;
+ while (batch_ok && got < max) {
+ memset(&a, 0, sizeof a);
+ a.batch.map_fd = (uint32_t)fd;
+ a.batch.in_batch = first ? 0 : (uint64_t)(uintptr_t)&batch;
+ a.batch.out_batch = (uint64_t)(uintptr_t)&batch;
+ a.batch.keys = (uint64_t)(uintptr_t)((unsigned char *)keys + (size_t)got * ksz);
+ a.batch.values = (uint64_t)(uintptr_t)((unsigned char *)vals + (size_t)got * vsz);
+ a.batch.count = max - got;
+ int r = sys_bpf(BPF_MAP_LOOKUP_BATCH, &a);
+ if (r < 0 && errno != ENOENT) {
+ if (first && (errno == EINVAL || errno == ENOTSUPP_FALLBACK || errno == EOPNOTSUPP)) { batch_ok = 0; break; }
+ return got ? (int)got : -1;
+ }
+ got += a.batch.count;
+ first = 0;
+ if (r < 0) return (int)got; /* ENOENT: that was the last batch */
+ }
+ if (batch_ok) return (int)got;
+ /* without batches: key by key */
+ unsigned char cur[64], next[64];
+ if (ksz > sizeof cur) { errno = EINVAL; return -1; }
+ const void *prev = NULL;
+ while (got < max) {
+ memset(&a, 0, sizeof a);
+ a.map_fd = (uint32_t)fd;
+ a.key = (uint64_t)(uintptr_t)prev;
+ a.next_key = (uint64_t)(uintptr_t)next;
+ if (sys_bpf(BPF_MAP_GET_NEXT_KEY, &a)) break;
+ memcpy(cur, next, ksz);
+ prev = cur;
+ if (bpfs_map_lookup(fd, cur, (unsigned char *)vals + (size_t)got * vsz)) continue;
+ memcpy((unsigned char *)keys + (size_t)got * ksz, cur, ksz);
+ got++;
+ }
+ return (int)got;
+}
+
int bpfs_prog_load(uint32_t type, uint32_t attach, const NpProgDesc *d, int (*map_fd)(const char *), char *log, size_t loglen) {
return bpfs_prog_load_btf(type, attach, 0, "LordOS", d, map_fd, log, loglen);
}
diff --git a/server/src/netperm/bpfsys.h b/server/src/netperm/bpfsys.h
index add4cb7..a7af47c 100644
--- a/server/src/netperm/bpfsys.h
+++ b/server/src/netperm/bpfsys.h
@@ -29,6 +29,10 @@ int bpfs_map_create(uint32_t type, uint32_t key, uint32_t val, uint32_t max, ui
int bpfs_map_update(int fd, const void *key, const void *val, uint64_t flags);
int bpfs_map_lookup(int fd, const void *key, void *val);
int bpfs_map_delete(int fd, const void *key);
+/* Reads up to max entries of a hash map (keys of ksz bytes into keys, values of vsz into vals),
+ * in one BPF_MAP_LOOKUP_BATCH call when the kernel has it (5.6+), otherwise key by key. Returns how
+ * many it read, or -1. A map that changes meanwhile may give an entry twice or miss a new one. */
+int bpfs_map_dump(int fd, void *keys, uint32_t ksz, void *vals, uint32_t vsz, uint32_t max);
/* Loads a program with its relocations resolved (map_fd(name) gives the fd, -1 if it doesn't
* exist). log (optional) receives the verifier log if it fails. Returns the fd or -1. */
diff --git a/server/src/netperm/netperm.c b/server/src/netperm/netperm.c
index aec7363..e2caa87 100644
--- a/server/src/netperm/netperm.c
+++ b/server/src/netperm/netperm.c
@@ -69,6 +69,8 @@ typedef struct {
int has_domains, seen;
NpApp app_v; /* the last thing written to np_apps */
uint64_t allowed, denied; /* kernel counters */
+ uint64_t rx, tx; /* bytes (np_counts) */
+ double rrx, rtx; /* bytes per second over the last tick */
} Scope;
typedef struct {
@@ -101,7 +103,10 @@ typedef struct {
static struct {
int bpf_ok;
char bpf_err[200]; /* a code line (errcode.h): the firewall page shows it translated */
- int m_np_cfg, m_np_apps, m_np_union, m_np_lpm4, m_np_lpm6, m_np_seen, m_np_counts, m_np_events, m_np_snd;
+ int m_np_cfg, m_np_apps, m_np_union, m_np_lpm4, m_np_lpm6, m_np_seen, m_np_counts, m_np_events, m_np_snd, m_np_flows;
+ int acct_ok; /* the traffic accounting programs loaded */
+ FwMap flows; /* NpFlowKey → FlowSt: the previous reading of np_flows */
+ long long counts_ns; /* when push_counts last read the counters */
uint32_t snd[NP_MAX_SND]; /* the ALSA minors in np_snd */
int nsnd;
int dev_ok, bt_ok; /* the device / Bluetooth programs loaded */
@@ -211,6 +216,7 @@ static int map_fd(const char *name) {
if (!strcmp(name, "np_counts")) return G.m_np_counts;
if (!strcmp(name, "np_events")) return G.m_np_events;
if (!strcmp(name, "np_snd")) return G.m_np_snd;
+ if (!strcmp(name, "np_flows")) return G.m_np_flows;
return -1;
}
#endif
@@ -286,11 +292,21 @@ static void sync_scope(Session *s, Scope *sc) {
sc->has_domains = c.has_domains;
sc->app_v = c.app;
if (bpfs_map_update(G.m_np_apps, &sc->cgid, &c.app, BPF_ANY)) logf_("np_apps: %s", strerror(errno));
+ /* the counters' entry exists from the start: the accounting programs only add to existing
+ * entries, so they never store anything for a cgroup that isn't an app */
+ NpCount zero = { 0, 0, 0, 0 };
+ bpfs_map_update(G.m_np_counts, &sc->cgid, &zero, BPF_NOEXIST);
np_compiled_free(&c);
}
static void drop_scope(Scope *sc) {
if (!G.bpf_ok) return;
+ FWMAP_EACH(&G.flows, e) { /* its addresses' byte counters */
+ const NpFlowKey *k = (const NpFlowKey *)(const void *)e->key;
+ if (k->cgid != sc->cgid) continue;
+ bpfs_map_delete(G.m_np_flows, k);
+ free(fwmap_take(&G.flows, e->key, e->klen));
+ }
bpfs_map_delete(G.m_np_apps, &sc->cgid);
bpfs_map_delete(G.m_np_counts, &sc->cgid);
prune(sc->idx, 0);
@@ -355,6 +371,7 @@ static void sync_user_scope(Session *s) {
s->user_cg = 0;
if (!id || ufd < 0) { if (ufd >= 0) close(ufd); return; }
for (int i = 0; i < G.nprogs && s->nulinks < (int)(sizeof s->ulinks / sizeof s->ulinks[0]); i++) {
+ if (!np_attach_user(G.attach[i])) continue; /* accounting: only the app scopes */
int l = bpfs_link_create(G.progs[i], ufd, G.attach[i]);
if (l < 0) { logf_("uid %u: could not attach to user@: %s", s->uid, strerror(errno)); drop_user_links(s); close(ufd); return; }
s->ulinks[s->nulinks++] = l;
@@ -553,8 +570,9 @@ static void send_uid(unsigned uid, int type, const char *json, size_t n, int dro
static void status_json(char *b, size_t n) {
char err[260];
json_quote(err, sizeof err, G.bpf_ok ? "" : G.bpf_err);
- snprintf(b, n, "{\"bpf\":%s,\"devices\":%s,\"bluetooth\":%s,\"dns\":\"%s\",\"error\":%s}", G.bpf_ok ? "true" : "false",
- G.bpf_ok && G.dev_ok ? "true" : "false", G.bpf_ok && G.bt_ok ? "true" : "false", G.dns ? npdns_source(G.dns) : "none", err);
+ snprintf(b, n, "{\"bpf\":%s,\"devices\":%s,\"bluetooth\":%s,\"bytes\":%s,\"dns\":\"%s\",\"error\":%s}", G.bpf_ok ? "true" : "false",
+ G.bpf_ok && G.dev_ok ? "true" : "false", G.bpf_ok && G.bt_ok ? "true" : "false", G.bpf_ok && G.acct_ok == 2 ? "true" : "false",
+ G.dns ? npdns_source(G.dns) : "none", err);
}
static void accept_client(void) {
@@ -673,28 +691,141 @@ static void on_event(void *u, const void *data, uint32_t len) {
ev.flag ? " " : "", ev.flag && flag ? flag : "");
}
+/* ---- traffic per address (np_flows) ------------------------------------------------------- */
+
+typedef struct {
+ uint64_t rx, tx; /* the last reading */
+ double rrx, rtx; /* bytes per second over the last tick */
+ long long active; /* tick of the last change */
+ long long tick; /* tick of the last reading that saw it */
+} FlowSt;
+
+#define FLOW_KEEP_TICKS 120 /* an address idle for 2 min stops being sent (its bytes stay in the app's total) */
+#define FLOWS_PER_APP 64 /* the busiest addresses of each app, per COUNTS message */
+
+typedef struct { const NpFlowKey *k; const FlowSt *f; } FlowRef;
+
+static int flow_cmp(const void *a, const void *b) {
+ const FlowSt *x = ((const FlowRef *)a)->f, *y = ((const FlowRef *)b)->f;
+ double rx = x->rrx + x->rtx, ry = y->rrx + y->rtx;
+ if (rx != ry) return rx < ry ? 1 : -1;
+ uint64_t bx = x->rx + x->tx, by = y->rx + y->tx;
+ return bx < by ? 1 : bx > by ? -1 : 0;
+}
+
+/* Rereads np_flows (one batch call) and updates each address's rate. dt in seconds. */
+static void read_flows(double dt) {
+ static NpFlowKey keys[NP_MAX_FLOWS];
+ static NpFlow vals[NP_MAX_FLOWS];
+ int n = bpfs_map_dump(G.m_np_flows, keys, sizeof keys[0], vals, sizeof vals[0], NP_MAX_FLOWS);
+ for (int i = 0; i < n; i++) {
+ void **slot = fwmap_slot(&G.flows, &keys[i], sizeof keys[i], 1);
+ if (!slot) break;
+ FlowSt *f = *slot;
+ if (!f && !(f = *slot = calloc(1, sizeof *f))) break; /* new in the kernel since the last dump: all its bytes are new */
+ if (f->tick == G.tick) continue; /* the same entry twice in one dump */
+ uint64_t drx = vals[i].rx >= f->rx ? vals[i].rx - f->rx : vals[i].rx; /* LRU reuse: started over */
+ uint64_t dtx = vals[i].tx >= f->tx ? vals[i].tx - f->tx : vals[i].tx;
+ f->rrx = dt > 0 ? (double)drx / dt : 0;
+ f->rtx = dt > 0 ? (double)dtx / dt : 0;
+ if (drx || dtx || !f->active) f->active = G.tick;
+ f->rx = vals[i].rx;
+ f->tx = vals[i].tx;
+ f->tick = G.tick;
+ }
+ /* the table mirrors the kernel's: what the LRU evicted (or a closed scope left) goes */
+ FWMAP_EACH(&G.flows, e) {
+ FlowSt *f = e->val;
+ if (f->tick != G.tick) free(fwmap_take(&G.flows, e->key, e->klen));
+ }
+}
+
+/* No bytes moved this second: every address is idle (no need to read the kernel's table). */
+static void idle_flows(void) {
+ FWMAP_EACH(&G.flows, e) {
+ FlowSt *f = e->val;
+ f->rrx = f->rtx = 0;
+ }
+}
+
+/* [["addr",rx,tx,rxRate,txRate],...] of one app, busiest first. */
+static size_t flows_json(uint64_t cgid, char *j, size_t cap) {
+ static FlowRef refs[NP_MAX_FLOWS];
+ int n = 0;
+ FWMAP_EACH(&G.flows, e) {
+ const NpFlowKey *k = (const NpFlowKey *)(const void *)e->key;
+ const FlowSt *f = e->val;
+ if (k->cgid != cgid || n >= NP_MAX_FLOWS || G.tick - f->active > FLOW_KEEP_TICKS) continue;
+ refs[n].k = k;
+ refs[n++].f = e->val;
+ }
+ qsort(refs, (size_t)n, sizeof refs[0], flow_cmp);
+ size_t w = 0;
+ w += (size_t)snprintf(j + w, cap - w, "[");
+ for (int i = 0; i < n && i < FLOWS_PER_APP && w + 120 < cap; i++) {
+ char addr[INET6_ADDRSTRLEN] = "";
+ inet_ntop(refs[i].k->fam == 4 ? AF_INET : AF_INET6, refs[i].k->addr, addr, sizeof addr);
+ w += (size_t)snprintf(j + w, cap - w, "%s[\"%s\",%llu,%llu,%.0f,%.0f]", i ? "," : "", addr,
+ (unsigned long long)refs[i].f->rx, (unsigned long long)refs[i].f->tx, refs[i].f->rrx, refs[i].f->rtx);
+ }
+ w += (size_t)snprintf(j + w, cap - w, "]");
+ return w;
+}
+
+/* Every second: {"app":[allowed,denied,rx,tx,rxRate,txRate,[[addr,rx,tx,rxRate,txRate],...]],...}
+ * to each user with something new (the rates are bytes per second over the last second). With
+ * no traffic and no new connections nothing is sent. */
static void push_counts(void) {
+ struct timespec ts;
+ clock_gettime(CLOCK_MONOTONIC, &ts);
+ long long now = (long long)ts.tv_sec * 1000000000ll + ts.tv_nsec;
+ double dt = G.counts_ns ? (double)(now - G.counts_ns) / 1e9 : 0;
+ G.counts_ns = now;
+ int bytes_moved = 0;
for (int i = 0; i < G.nsess; i++) {
Session *s = &G.sess[i];
- int changed = s->dirty_counts;
for (int k = 0; k < s->nsc; k++) {
- NpCount c = { 0, 0 };
- if (G.bpf_ok) bpfs_map_lookup(G.m_np_counts, &s->sc[k].cgid, &c);
- if (c.allowed != s->sc[k].allowed || c.denied != s->sc[k].denied) changed = 1;
- s->sc[k].allowed = c.allowed;
- s->sc[k].denied = c.denied;
+ Scope *sc = &s->sc[k];
+ NpCount c = { 0, 0, 0, 0 };
+ if (G.bpf_ok) bpfs_map_lookup(G.m_np_counts, &sc->cgid, &c);
+ if (c.allowed != sc->allowed || c.denied != sc->denied) s->dirty_counts = 1;
+ double rrx = dt > 0 && c.rx >= sc->rx ? (double)(c.rx - sc->rx) / dt : 0;
+ double rtx = dt > 0 && c.tx >= sc->tx ? (double)(c.tx - sc->tx) / dt : 0;
+ if (c.rx != sc->rx || c.tx != sc->tx || (rrx == 0) != (sc->rrx == 0) || (rtx == 0) != (sc->rtx == 0)) { s->dirty_counts = 1; bytes_moved = 1; }
+ sc->allowed = c.allowed;
+ sc->denied = c.denied;
+ sc->rx = c.rx;
+ sc->tx = c.tx;
+ sc->rrx = rrx;
+ sc->rtx = rtx;
}
- if (!changed) continue;
+ }
+ /* the per-address table only while bytes move (one batch call per second) */
+ if (G.bpf_ok && G.acct_ok == 2) {
+ if (bytes_moved) read_flows(dt);
+ else idle_flows();
+ }
+ for (int i = 0; i < G.nsess; i++) {
+ Session *s = &G.sess[i];
+ if (!s->dirty_counts) continue;
+ int has_client = 0;
+ for (int c = 0; c < G.ncl && !has_client; c++) has_client = G.cl[c].uid == s->uid;
+ if (!has_client) continue; /* stays dirty until someone connects */
s->dirty_counts = 0;
- size_t cap = 64 + (size_t)s->nsc * (FW_APP_MAX * 2 + 48), w = 0;
+ size_t per = FW_APP_MAX * 2 + 160 + (size_t)FLOWS_PER_APP * 120;
+ size_t cap = 64 + (size_t)s->nsc * per, w = 0;
char *j = malloc(cap);
if (!j) continue;
j[w++] = '{';
for (int k = 0; k < s->nsc; k++) {
+ Scope *sc = &s->sc[k];
char app[FW_APP_MAX * 2 + 4];
- json_quote(app, sizeof app, s->sc[k].app);
- w += (size_t)snprintf(j + w, cap - w, "%s%s:[%llu,%llu]", w > 1 ? "," : "", app,
- (unsigned long long)s->sc[k].allowed, (unsigned long long)s->sc[k].denied);
+ json_quote(app, sizeof app, sc->app);
+ w += (size_t)snprintf(j + w, cap - w, "%s%s:[%llu,%llu,%llu,%llu,%.0f,%.0f,", w > 1 ? "," : "", app,
+ (unsigned long long)sc->allowed, (unsigned long long)sc->denied,
+ (unsigned long long)sc->rx, (unsigned long long)sc->tx, sc->rrx, sc->rtx);
+ w += flows_json(sc->cgid, j + w, cap - w - 2);
+ j[w++] = ']';
}
j[w++] = '}';
send_uid(s->uid, NP_MSG_COUNTS, j, w, 0);
@@ -761,7 +892,7 @@ static int bpf_setup(void) {
#define NP_MKMAP(n, t, k, v, m, f) G.m_##n = bpfs_map_create(t, k, v, m, f, #n);
NP_MAPS(NP_MKMAP)
#undef NP_MKMAP
- if (G.m_np_snd < 0 || G.m_np_cfg < 0 || G.m_np_apps < 0 || G.m_np_union < 0 || G.m_np_lpm4 < 0 || G.m_np_lpm6 < 0 || G.m_np_seen < 0 || G.m_np_counts < 0 || G.m_np_events < 0) {
+ if (G.m_np_flows < 0 || G.m_np_snd < 0 || G.m_np_cfg < 0 || G.m_np_apps < 0 || G.m_np_union < 0 || G.m_np_lpm4 < 0 || G.m_np_lpm6 < 0 || G.m_np_seen < 0 || G.m_np_counts < 0 || G.m_np_events < 0) {
lerr_set(G.bpf_err, sizeof G.bpf_err, "fw.bpfMaps", "errno", lerr_errno(errno), NULL);
return -1;
}
@@ -798,6 +929,7 @@ static int bpf_setup(void) {
}
if (want[w].attach == BPF_CGROUP_DEVICE) G.dev_ok = 1;
if (want[w].attach == BPF_LSM_CGROUP) G.bt_ok = 1;
+ if (want[w].attach == BPF_CGROUP_INET_INGRESS || want[w].attach == BPF_CGROUP_INET_EGRESS) G.acct_ok++;
G.attach[np] = want[w].attach;
G.progs[np++] = fd;
}
@@ -805,8 +937,8 @@ static int bpf_setup(void) {
NpCfg cfg = { SCOPE_LEVEL, 1, USER_LEVEL, hidraw_major() };
uint32_t zero = 0;
if (bpfs_map_update(G.m_np_cfg, &zero, &cfg, BPF_ANY)) { lerr_set(G.bpf_err, sizeof G.bpf_err, "fw.bpfConfig", "errno", lerr_errno(errno), NULL); return -1; }
- logf_("%d BPF programs loaded (devices: %s, Bluetooth sockets: %s, hidraw major %u)", np, G.dev_ok ? "yes" : "no",
- G.bt_ok ? "yes" : "no", cfg.hidraw_major);
+ logf_("%d BPF programs loaded (devices: %s, Bluetooth sockets: %s, traffic accounting: %s, hidraw major %u)", np, G.dev_ok ? "yes" : "no",
+ G.bt_ok ? "yes" : "no", G.acct_ok == 2 ? "yes" : "no", cfg.hidraw_major);
return 0;
#endif
}
diff --git a/server/src/netperm/np.bpf.c b/server/src/netperm/np.bpf.c
index 0e63582..b84a231 100644
--- a/server/src/netperm/np.bpf.c
+++ b/server/src/netperm/np.bpf.c
@@ -16,6 +16,8 @@
* to be a non-capture one); every other device stays allowed
* cgroup/lsm_socket_create "no Bluetooth": socket(AF_BLUETOOTH) (a per-cgroup LSM program,
* BPF_LSM_CGROUP: the inet sock_create hook never sees that family)
+ * cgroup/skb_ingress, egress traffic accounting for the Monitor: bytes per app (np_counts) and
+ * per (app, remote address) (np_flows). They never drop a packet.
*
* The same programs are also attached to the user@<uid>.service of each LordOS session, to
* close the `systemd-run --user` escape (the process ends up there, outside lordos.service). A
@@ -29,7 +31,12 @@
* 0 (the kernel returns EPERM to the app). It counts the connection and notifies lord-netperm over
* the ring buffer only if it's new (np_seen, 5 s). Nothing per packet: only per connection, bind or
* sendto. A UDP connect() counts only the first time within 5 s, like a sendto: getaddrinfo does one
- * per address to sort them (RFC 6724) and they aren't real connections. */
+ * per address to sort them (RFC 6724) and they aren't real connections.
+ *
+ * The one exception is traffic accounting (skb_ingress / skb_egress), which runs per packet
+ * (per GRO/GSO segment): only for the sessions' apps, two map lookups and two atomic adds, and it
+ * always lets the packet through. It's what counts UDP (QUIC) and the bytes of connections that
+ * already closed, which the sockets' own counters (tcp_info) can't. */
#define NP_BPF 1
#include <linux/bpf.h>
#include "netperm/np_shared.h"
@@ -42,6 +49,8 @@ static long (*bpf_map_update_elem)(void *map, const void *key, const void *value
static unsigned long long (*bpf_ktime_get_ns)(void) = (void *)5;
static unsigned long long (*bpf_get_current_ancestor_cgroup_id)(int level) = (void *)123;
static long (*bpf_ringbuf_output)(void *ringbuf, void *data, unsigned long long size, unsigned long long flags) = (void *)130;
+static long (*bpf_skb_load_bytes)(const void *skb, unsigned int offset, void *to, unsigned int len) = (void *)26;
+static unsigned long long (*bpf_skb_ancestor_cgroup_id)(void *skb, int level) = (void *)83;
/* the maps: only their names (lord-netperm defines them, netperm.c) */
char np_cfg[4] SEC("maps");
@@ -53,6 +62,7 @@ char np_seen[4] SEC("maps");
char np_counts[4] SEC("maps");
char np_events[4] SEC("maps");
char np_snd[4] SEC("maps");
+char np_flows[4] SEC("maps");
#define AF_INET 2
#define AF_INET6 10
@@ -87,7 +97,7 @@ NP_INLINE NpApp *find_app(NpCfg *cfg, np_u64 *cgid) {
NP_INLINE void count(np_u64 cgid, int deny) {
NpCount *c = bpf_map_lookup_elem(&np_counts, &cgid);
if (!c) {
- NpCount z = { 0, 0 };
+ NpCount z = { 0, 0, 0, 0 };
bpf_map_update_elem(&np_counts, &cgid, &z, BPF_NOEXIST);
c = bpf_map_lookup_elem(&np_counts, &cgid);
if (!c) return;
@@ -327,3 +337,49 @@ SEC("cgroup/lsm_socket_create") int np_bt_socket(unsigned long long *ctx) {
report_dev(cgid, NP_D_BT, 0, 0, NP_R_NO_BLUETOOTH);
return 0;
}
+
+/* ---- traffic accounting (the Monitor's bytes per app and per address) ---- */
+
+#define LOOPBACK_IFINDEX 1 /* "lo" is always ifindex 1 (in every network namespace) */
+
+/* Adds the packet's bytes to its app (the socket's cgroup, the scope at cfg->level) and to the
+ * (app, remote address) pair. Only for apps lord-netperm knows: it creates their np_counts entry
+ * when the scope appears, so nothing else (the login screen, the system) is counted or stored.
+ * Loopback isn't counted. The skb starts at the IP header in both directions. */
+NP_INLINE void account(struct __sk_buff *skb, int in) {
+ if (skb->ifindex == LOOPBACK_IFINDEX) return;
+ np_u32 zero = 0;
+ NpCfg *cfg = bpf_map_lookup_elem(&np_cfg, &zero);
+ if (!cfg) return;
+ np_u64 cgid = bpf_skb_ancestor_cgroup_id(skb, (int)cfg->level);
+ if (!cgid) return;
+ NpCount *c = bpf_map_lookup_elem(&np_counts, &cgid);
+ if (!c) return;
+ NpFlowKey k;
+ __builtin_memset(&k, 0, sizeof k);
+ k.cgid = cgid;
+ np_u8 v = 0;
+ if (bpf_skb_load_bytes(skb, 0, &v, 1)) return;
+ if (v >> 4 == 4) {
+ if (bpf_skb_load_bytes(skb, in ? 12 : 16, k.addr, 4)) return; /* source when it comes in, destination when it goes out */
+ k.fam = 4;
+ } else if (v >> 4 == 6) {
+ if (bpf_skb_load_bytes(skb, in ? 8 : 24, k.addr, 16)) return;
+ k.fam = 6;
+ } else return;
+ np_u64 len = skb->len;
+ if (in) __sync_fetch_and_add(&c->rx, len);
+ else __sync_fetch_and_add(&c->tx, len);
+ NpFlow *f = bpf_map_lookup_elem(&np_flows, &k);
+ if (!f) {
+ NpFlow z = { 0, 0 };
+ bpf_map_update_elem(&np_flows, &k, &z, BPF_NOEXIST);
+ f = bpf_map_lookup_elem(&np_flows, &k);
+ if (!f) return;
+ }
+ if (in) __sync_fetch_and_add(&f->rx, len);
+ else __sync_fetch_and_add(&f->tx, len);
+}
+
+SEC("cgroup/skb_ingress") int np_skb_in(struct __sk_buff *skb) { account(skb, 1); return 1; }
+SEC("cgroup/skb_egress") int np_skb_out(struct __sk_buff *skb) { account(skb, 0); return 1; }
diff --git a/server/src/netperm/np_shared.h b/server/src/netperm/np_shared.h
index 8d26b4b..50a6c92 100644
--- a/server/src/netperm/np_shared.h
+++ b/server/src/netperm/np_shared.h
@@ -13,7 +13,8 @@
* np_lpm4 LPM (app idx, IPv4/n) → NpList network and domain rules, ordered
* np_lpm6 LPM (app idx, IPv6/n) → NpList
* np_seen LRU NpSeenKey → u64 ns to report only new connections
- * np_counts HASH cgroup id → NpCount allowed / denied
+ * np_counts HASH cgroup id → NpCount allowed / denied, bytes received / sent
+ * np_flows LRU NpFlowKey → NpFlow bytes received / sent per (app, remote address)
* np_events RINGBUF NpEvent towards lord-netperm
* np_snd HASH ALSA minor (u32) → u32 every ALSA node in /sys/class/sound: NP_SND_CAPTURE for the
* capture PCMs (/dev/snd/pcmC*D*c), NP_SND_OTHER for the rest
@@ -47,6 +48,7 @@ typedef uint64_t np_u64;
#define NP_MAX_APPS 8192 /* cgroups with a policy */
#define NP_MAX_LPM 65536 /* entries of each trie */
#define NP_MAX_SEEN 16384
+#define NP_MAX_FLOWS 4096 /* (app, remote address) byte counters; the least recently used go first */
#define NP_RINGBUF (256u << 10)
#define NP_REPEAT_NS (5ull * 1000000000ull) /* a repeated connection is reported again after 5 s */
@@ -125,7 +127,19 @@ typedef struct {
np_u8 pad[2];
} NpSeenKey;
-typedef struct { np_u64 allowed, denied; } NpCount;
+/* allowed / denied: connections (connect, bind, the first sendto in 5 s). rx / tx: IP bytes
+ * received and sent by the app's sockets (cgroup/skb_ingress and skb_egress, every protocol:
+ * TCP, UDP and so QUIC), loopback excluded so they compare with the interfaces' counters. */
+typedef struct { np_u64 allowed, denied, rx, tx; } NpCount;
+
+typedef struct {
+ np_u64 cgid;
+ np_u8 addr[16]; /* the remote address; IPv4 (also IPv4-mapped IPv6) in the first 4 */
+ np_u8 fam; /* 4 or 6 */
+ np_u8 pad[7];
+} NpFlowKey;
+
+typedef struct { np_u64 rx, tx; } NpFlow;
typedef struct {
np_u64 cgid;
diff --git a/server/src/netperm/npload.h b/server/src/netperm/npload.h
index 86d2285..b973a95 100644
--- a/server/src/netperm/npload.h
+++ b/server/src/netperm/npload.h
@@ -22,11 +22,14 @@
X(np_seen, BPF_MAP_TYPE_LRU_HASH, sizeof(NpSeenKey), 8, NP_MAX_SEEN, 0) \
X(np_counts, BPF_MAP_TYPE_HASH, 8, sizeof(NpCount), NP_MAX_APPS, 0) \
X(np_events, BPF_MAP_TYPE_RINGBUF, 0, 0, NP_RINGBUF, 0) \
- X(np_snd, BPF_MAP_TYPE_HASH, 4, 4, NP_MAX_SND, 0)
+ X(np_snd, BPF_MAP_TYPE_HASH, 4, 4, NP_MAX_SND, 0) \
+ X(np_flows, BPF_MAP_TYPE_LRU_HASH, sizeof(NpFlowKey), sizeof(NpFlow), NP_MAX_FLOWS, 0)
/* Each program: its section, kind and attach point. btf: the kernel function an LSM program
- * hooks (its id comes from /sys/kernel/btf/vmlinux). opt: 1 = the device and Bluetooth
- * programs; if the kernel refuses one (no bpf LSM, an old kernel) the network ones still work. */
+ * hooks (its id comes from /sys/kernel/btf/vmlinux). opt: 1 = the device, Bluetooth and traffic
+ * accounting programs; if the kernel refuses one (no bpf LSM, an old kernel) the network ones
+ * still work. The accounting programs (BPF_CGROUP_INET_INGRESS / EGRESS) are attached only to
+ * lordos.service, not to user@<uid>.service (np_attach_user). */
typedef struct {
const char *sec;
uint32_t type, attach;
@@ -44,9 +47,14 @@ static const NpProgSpec np_prog_specs[] = {
{ "cgroup/sock_create", BPF_PROG_TYPE_CGROUP_SOCK, BPF_CGROUP_INET_SOCK_CREATE, NULL, 0 },
{ "cgroup/dev", BPF_PROG_TYPE_CGROUP_DEVICE, BPF_CGROUP_DEVICE, NULL, 1 },
{ "cgroup/lsm_socket_create", BPF_PROG_TYPE_LSM, BPF_LSM_CGROUP, "bpf_lsm_socket_create", 1 },
+ /* traffic accounting for the Monitor (bytes per app and per remote address); never denies */
+ { "cgroup/skb_ingress", BPF_PROG_TYPE_CGROUP_SKB, BPF_CGROUP_INET_INGRESS, NULL, 1 },
+ { "cgroup/skb_egress", BPF_PROG_TYPE_CGROUP_SKB, BPF_CGROUP_INET_EGRESS, NULL, 1 },
};
#define NP_NSPECS (int)(sizeof np_prog_specs / sizeof np_prog_specs[0])
+static inline int np_attach_user(uint32_t attach) { return attach != BPF_CGROUP_INET_INGRESS && attach != BPF_CGROUP_INET_EGRESS; }
+
/* The license the kernel sees: LSM programs must declare a GPL-compatible one (see bpfsys.h). */
static inline const char *np_prog_license(const NpProgSpec *p) { return p->type == BPF_PROG_TYPE_LSM ? "Dual MIT/GPL" : "LordOS"; }
diff --git a/server/src/netperm/npproto.h b/server/src/netperm/npproto.h
index 8cdf788..605c8fc 100644
--- a/server/src/netperm/npproto.h
+++ b/server/src/netperm/npproto.h
@@ -13,13 +13,19 @@
* of fw/rules.c. Any other message, or one longer than NP_IN_MAX bytes, closes the connection.
*
* From lord-netperm to lord-server (compact JSON in the data):
- * NP_MSG_STATUS {"bpf":true,"dns":"resolved","error":""} on connect and when it changes
+ * NP_MSG_STATUS {"bpf":true,"devices":true,"bluetooth":true,"bytes":true,"dns":"resolved","error":""}
+ * on connect and when it changes
* NP_MSG_EVENT {"app":"desktop:x","v":"deny","dir":"out","proto":"tcp","addr":"1.2.3.4",
* "port":443,"by":"rule","rule":7,"flag":"","record":0,
* "hosts":["example.com"]} a new connection; hosts:
* up to 4 names the resolver
* gave for addr
- * NP_MSG_COUNTS {"desktop:x":[allowed,denied],...} every second, if it changed */
+ * NP_MSG_COUNTS {"desktop:x":[allowed,denied,rx,tx,rxRate,txRate,
+ * [["1.2.3.4",rx,tx,rxRate,txRate],...]],...}
+ * every second, if it changed:
+ * connections, bytes (the
+ * cgroup_skb accounting) and
+ * the busiest addresses */
#ifndef LORD_NP_PROTO_H
#define LORD_NP_PROTO_H
diff --git a/server/src/ui/js_fw.c b/server/src/ui/js_fw.c
index 4da08d7..23876c6 100644
--- a/server/src/ui/js_fw.c
+++ b/server/src/ui/js_fw.c
@@ -21,7 +21,9 @@
* addRule({ app, dir, dst, port, proto, action }) → id removeRule(id)
* updateRule(id, { dir, dst, port, proto, action }) → id same app; the old rule comes back if
* the new one can't be added
- * poll() { counts: { app: [allowed, denied] }, events: [...] }
+ * poll() { counts: { app: [allowed, denied, rx, tx, rxRate, txRate, flows] },
+ * events: [...] } (the bytes when lord-netperm's accounting is loaded,
+ * see npproto.h)
* blocks(app, restriction) true if the app is restricted in it (sdk/notify.js asks before showing
* a built-in app's notification)
*