Skip to content

armgit / xerj

Public Imported from GitHub
0

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

fix(http2): raise the small-DATA-frame budget with the window (h2 0.4.19) (#908)

xerj-org01ef96293700
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 638dd0ae7..35f578a3c 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -234,6 +234,73 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
separate work: it has to prove the #825 union/sum contract with tombstones
present, and is not attempted here.
+- **A legitimate authenticated request whose body arrived in small HTTP/2 DATA
+ frames could be cut off with `GOAWAY ENHANCE_YOUR_CALM`**
+ ([#485](https://github.com/xerj-org/xerj/issues/485)). The rc.18 `h2` bump for
+ RUSTSEC-2026-0258 — described at length in that release's notes below — added
+ a per-connection budget for small DATA frames: a frame under 256 bytes
+ consumes `256 - payload_len`, a frame of 256 or more replenishes, and the
+ refund for a small frame lands only when the application *reads* it. In `h2`
+ 0.4.16 that budget was a fixed 25,600 bytes however large a window the server
+ had advertised, while hyper advertises a 1 MiB HTTP/2 connection window — so a
+ well-behaved client could have 40× the budget in flight before the handler was
+ ever scheduled. Any client that chunks small (a streaming JSON encoder, a
+ per-document flush, a gRPC stream of small messages) lost its whole
+ *connection* on a request that would otherwise have returned 200.
+
+ `h2` is now floored at 0.4.19 in `[workspace.dependencies]`
+ (`engine/Cargo.toml`), where the budget is half the configured connection
+ window with 25,600 bytes as the minimum — 512 KiB against hyper's 1 MiB
+ window — and the empty-DATA-frame flood the advisory was actually filed about
+ is capped by its own counter (100 frames per connection) rather than out of
+ the byte budget. The mitigation is not disabled or weakened here, and no
+ XERJ-side window or budget knob was changed; the floor sits in the workspace
+ manifest so `cargo update` cannot resolve back to a fixed-budget release.
+
+ Measured on this box, same harness on both sides (`axum::serve`, hyper 1.9.0,
+ a handler that reads the whole body, one DATA frame per chunk queued
+ back-to-back with no pause), bisecting for the largest body that completes.
+ The boundary is not a constant — it is a race between arriving charges and
+ the refunds a reading handler returns — so these are the min–max of three
+ bisections each, not thresholds to design against:
+
+ | DATA frame size | before (h2 0.4.16) | after (h2 0.4.19) |
+ |---|---|---|
+ | 64 B | 9.3–16.5 KB (145–258 frames) | 262 KB–6.4 MB (4,095–99,793 frames) |
+ | 100 B | 18.9–27.2 KB (189–272 frames) | 410 KB–4.4 MB (4,095–43,561 frames) |
+ | 128 B | 29.3–36.6 KB (229–286 frames) | 2.1–4.7 MB (16,127–36,628 frames) |
+ | 200 B | 141–419 KB (703–2,094 frames) | no cut-off up to 52 MB |
+ | 256 B | no cut-off up to 8 MB | no cut-off (never charges) |
+
+ Where the safe line now sits, exactly: with a budget of half the window, a
+ frame of **171 bytes or more** cannot exhaust it even if the application never
+ reads a byte — at most `window / size` frames can be in flight, each charging
+ `256 - size`, and `(256 - size) ≤ size / 2` from 171 up. That is why the
+ 200 B row has no ceiling at all. Below 171 bytes there is still a ceiling and
+ it still depends on how fast the handler drains; it is now one to two orders
+ of magnitude further out. A frame of 256 bytes or more never charges, before
+ or after. The rc.18 note advising clients to chunk at 256 bytes or above is
+ no longer needed for ordinary bodies.
+
+ Deliberately abusive traffic is still refused, and that is asserted rather
+ than assumed: a flood of empty DATA frames still ends in `GOAWAY
+ ENHANCE_YOUR_CALM` (upstream's 100-frame cap, unchanged), and a flood of
+ 1-byte frames is still cut off — on this harness the client saw the
+ connection go at roughly frame 4,600 instead of roughly frame 100.
+
+ What was measured where. The table above comes from a standalone harness
+ pinned to this workspace's own axum/hyper/h2 versions, not from the `xerj`
+ binary. The binary itself is covered by
+ `engine/crates/xerj-server/tests/http2_small_data_frames.rs`, which spawns the
+ real server with authentication on and drives a 40 KB authenticated `_bulk`
+ at 100 bytes per DATA frame (401 frames) through the ES-compat listener over
+ prior-knowledge h2c, asserting a 200 with every document indexed. Against the
+ unfixed dependency that same test is answered
+ `GOAWAY ENHANCE_YOUR_CALM (too_many_data_frames)` mid-body, which is how it
+ was checked. A second test holds the empty-frame guard in place. The TLS
+ listener and the gRPC listener sit on the same `h2` and take the same fix,
+ but neither is exercised by these tests.
+
- **`xerj autoindex --no-graph` no longer aborts with `Access denied
(os error 5)` on Windows**
([#482](https://github.com/xerj-org/xerj/issues/482)). The durable-snapshot
@@ -971,7 +1038,9 @@ silently reattributed.
drains — measurements on two machines differed in both directions, so treat
the hazard as real and the frequency as unpredictable rather than as a rate.
Tracked as [#485](https://github.com/xerj-org/xerj/issues/485).
- If you control the client, chunk at 256 bytes or above.
+ If you control the client, chunk at 256 bytes or above. **On this release
+ only** — see the Unreleased entry for #485 above, where the `h2` floor moves
+ to 0.4.19 and the budget starts scaling with the connection window.
XERJ has such a path: `auth_middleware` is an axum layer, so a request that
fails authentication is answered `401` without the handler ever extracting its
diff --git a/engine/Cargo.lock b/engine/Cargo.lock
index 62de03471..f828b86c3 100644
--- a/engine/Cargo.lock
+++ b/engine/Cargo.lock
@@ -1961,9 +1961,9 @@ checksum = "b40ca9252762c466af32d0b1002e91e4e1bc5398f77455e55474deb466355ff5"
[[package]]
name = "h2"
-version = "0.4.16"
+version = "0.4.19"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a9f37a958b41b3b19ee2707c06439c0e9e547e847223eb791ecb0cb821c65e27"
+checksum = "ef8e5e5a340588f4452631496976cf8636d4a7ecf600239fdc27615d2530bc16"
dependencies = [
"atomic-waker",
"bytes",
@@ -6961,6 +6961,7 @@ dependencies = [
"axum-server",
"chrono",
"flate2",
+ "h2",
"jemalloc_pprof",
"libc",
"memchr",
diff --git a/engine/Cargo.toml b/engine/Cargo.toml
index e56082f3f..039393ec3 100644
--- a/engine/Cargo.toml
+++ b/engine/Cargo.toml
@@ -31,6 +31,19 @@ authors = ["xerj Contributors"]
tokio = { version = "1", features = ["full"] }
# HTTP
axum = { version = "0.7", features = ["macros"] }
+# `h2` is the HTTP/2 implementation under every listener (hyper for REST +
+# ES-compat, tonic for gRPC) and under `reqwest` on the client side. Nothing in
+# this workspace calls it directly outside the `xerj-server` regression test,
+# but the floor is declared here on purpose: 0.4.16-0.4.18 size the small-DATA-
+# frame budget (the RUSTSEC-2026-0258 mitigation) at a fixed 25,600 bytes,
+# which a legitimate authenticated request chunked below ~200 bytes exhausts —
+# the client is answered `GOAWAY ENHANCE_YOUR_CALM` (#485). 0.4.19 scales the
+# budget with the configured connection window instead (half of it, floor
+# 25,600), so hyper's 1 MiB server window yields 512 KiB of headroom while the
+# empty-DATA-frame vector the advisory was filed for stays capped at 100 frames
+# by a separate counter. Declaring the floor here means `cargo update` cannot
+# resolve back to a version with the fixed budget.
+h2 = "0.4.19"
tower = "0.4"
tower-http = { version = "0.5", features = ["cors", "trace", "limit", "decompression-gzip"] }
# Serialization
diff --git a/engine/crates/xerj-server/Cargo.toml b/engine/crates/xerj-server/Cargo.toml
index 3f8f0d786..1561fbcd9 100644
--- a/engine/crates/xerj-server/Cargo.toml
+++ b/engine/crates/xerj-server/Cargo.toml
@@ -97,5 +97,10 @@ protox = { workspace = true }
# HTTPS round-trip test against the in-process rustls listener.
reqwest = { workspace = true }
tempfile = { workspace = true }
+# HTTP/2 *as a client*, so `tests/http2_small_data_frames.rs` can choose the
+# DATA-frame size — the one thing no ordinary HTTP client exposes, and the
+# whole subject of #485. It also pins the workspace floor at the h2 release
+# whose small-frame budget scales with the connection window.
+h2 = { workspace = true }
flate2 = "1"
tokio-stream = "0.1"
diff --git a/engine/crates/xerj-server/tests/http2_small_data_frames.rs b/engine/crates/xerj-server/tests/http2_small_data_frames.rs
new file mode 100644
index 000000000..ce675e916
--- /dev/null
+++ b/engine/crates/xerj-server/tests/http2_small_data_frames.rs
@@ -0,0 +1,353 @@
+//! Issue #485: a legitimate, authenticated request whose body arrives in small
+//! HTTP/2 DATA frames must not be answered `GOAWAY ENHANCE_YOUR_CALM`.
+//!
+//! `h2` 0.4.16 shipped the RUSTSEC-2026-0258 mitigation as a per-connection
+//! byte budget: a DATA frame under 256 bytes consumes `256 - payload_len` of
+//! it, a frame of 256 or more replenishes, and the budget is a fixed 25,600
+//! bytes (`DEFAULT_DATA_FRAME_OVERHEAD_THRESHOLD * 100`). The charge lands when
+//! the frame arrives and the refund only when the application reads it, so a
+//! client that chunks small — a streaming JSON encoder, a per-document flush, a
+//! gRPC stream of small messages — drains the whole budget before the handler
+//! is scheduled and has its *connection* closed. Run against the pre-fix
+//! dependency, the first test below is answered `GOAWAY ENHANCE_YOUR_CALM
+//! (too_many_data_frames)` part-way through a 40 KB body.
+//!
+//! Two things make this a process-level test rather than a unit test:
+//!
+//! 1. The budget is enforced inside `h2`, below hyper, below axum. Nothing in
+//! XERJ's own code can be called to observe it — only a real HTTP/2 peer
+//! sending real DATA frames at a real listener can.
+//! 2. The frame size is what matters, and no ordinary HTTP client lets you
+//! choose it. These tests therefore speak HTTP/2 directly (`h2` as a client,
+//! prior-knowledge h2c against the ES-compat listener) so every `send_data`
+//! is exactly one DATA frame of exactly the size asked for.
+//!
+//! The pair is deliberate. The first test is the regression — the traffic that
+//! must work. The second is the guard on the fix: the empty-DATA-frame flood
+//! the advisory was actually filed about must still be cut off, so raising the
+//! headroom for legitimate clients cannot be mistaken for switching the
+//! protection off.
+
+use std::io::Read;
+use std::net::TcpListener;
+use std::process::{Child, Command, Stdio};
+use std::sync::{Arc, Mutex};
+use std::time::{Duration, Instant};
+
+use axum::body::Bytes;
+use axum::http::Request;
+use tokio::net::TcpStream;
+
+/// Fixed key so the request below is unambiguously *authenticated*: the issue
+/// is about traffic that would otherwise have succeeded, not about a 401 path.
+const ADMIN_KEY: &str = "issue485-admin-key-not-a-secret";
+
+/// 100 bytes — the size named in the issue, and well inside the band that the
+/// fixed 25,600-byte budget rejects (`256 - 100 = 156` charged per frame).
+const CHUNK: usize = 100;
+
+/// TOML-safe rendering of a temp path (Windows backslashes would be escape
+/// sequences inside a basic string; forward slashes work everywhere).
+fn toml_path(p: &std::path::Path) -> String {
+ p.to_string_lossy().replace('\\', "/")
+}
+
+/// Three distinct free ports, held simultaneously so they cannot collide, then
+/// released for the child. `Config::validate` requires all three to differ.
+fn three_free_ports() -> (u16, u16, u16) {
+ let held: Vec<TcpListener> = (0..3)
+ .map(|_| TcpListener::bind("127.0.0.1:0").unwrap())
+ .collect();
+ let p: Vec<u16> = held
+ .iter()
+ .map(|l| l.local_addr().unwrap().port())
+ .collect();
+ (p[0], p[1], p[2])
+}
+
+/// Kills the child on the way out, including when a test panics — a leaked
+/// server would hold its data dir and its ports for the rest of the run.
+struct Server {
+ child: Child,
+ es_port: u16,
+ _dir: tempfile::TempDir,
+ output: Arc<Mutex<String>>,
+}
+
+impl Drop for Server {
+ fn drop(&mut self) {
+ let _ = self.child.kill();
+ let _ = self.child.wait();
+ }
+}
+
+fn drain_into(mut r: impl Read + Send + 'static, buf: Arc<Mutex<String>>) {
+ std::thread::spawn(move || {
+ let mut chunk = [0u8; 4096];
+ while let Ok(n) = r.read(&mut chunk) {
+ if n == 0 {
+ break;
+ }
+ buf.lock()
+ .unwrap()
+ .push_str(&String::from_utf8_lossy(&chunk[..n]));
+ }
+ });
+}
+
+/// Boot the real binary with authentication ON and wait until the ES-compat
+/// listener answers its readiness probe.
+async fn boot() -> Server {
+ let dir = tempfile::tempdir().unwrap();
+ let (rest, grpc, es) = three_free_ports();
+ let config = format!(
+ r#"
+[server]
+bind_address = "127.0.0.1"
+rest_port = {rest}
+grpc_port = {grpc}
+es_compat_port = {es}
+data_dir = "{data}"
+
+[auth]
+enabled = true
+admin_api_key = "{ADMIN_KEY}"
+
+[limits]
+disk_flood_stage_percent = 0
+"#,
+ data = toml_path(&dir.path().join("data")),
+ );
+ let config_path = dir.path().join("xerj.toml");
+ std::fs::write(&config_path, config).unwrap();
+
+ let mut child = Command::new(env!("CARGO_BIN_EXE_xerj"))
+ .arg("--config")
+ .arg(&config_path)
+ .stdout(Stdio::piped())
+ .stderr(Stdio::piped())
+ .spawn()
+ .expect("spawn xerj");
+
+ let output = Arc::new(Mutex::new(String::new()));
+ drain_into(child.stdout.take().unwrap(), output.clone());
+ drain_into(child.stderr.take().unwrap(), output.clone());
+
+ let server = Server {
+ child,
+ es_port: es,
+ _dir: dir,
+ output,
+ };
+
+ // `/health/ready` is auth-exempt and mounted on the ES-compat router.
+ let client = reqwest::Client::new();
+ let url = format!("http://127.0.0.1:{es}/health/ready");
+ let deadline = Instant::now() + Duration::from_secs(180);
+ loop {
+ if let Ok(r) = client.get(&url).send().await {
+ if r.status().is_success() {
+ return server;
+ }
+ }
+ assert!(
+ Instant::now() < deadline,
+ "xerj never became ready on :{es}\n{}",
+ server.output.lock().unwrap()
+ );
+ tokio::time::sleep(Duration::from_millis(100)).await;
+ }
+}
+
+/// Prior-knowledge h2c handshake against the ES-compat listener, plus a handle
+/// that resolves to the connection-level error (the `GOAWAY`, when there is
+/// one) once the connection ends.
+async fn connect(
+ port: u16,
+) -> (
+ h2::client::SendRequest<Bytes>,
+ tokio::sync::oneshot::Receiver<Option<String>>,
+) {
+ let tcp = TcpStream::connect(("127.0.0.1", port))
+ .await
+ .expect("connect to the ES-compat listener");
+ tcp.set_nodelay(true).unwrap();
+ let (client, connection) = h2::client::handshake(tcp)
+ .await
+ .expect("HTTP/2 handshake — the listener must speak prior-knowledge h2c");
+ let (tx, rx) = tokio::sync::oneshot::channel();
+ tokio::spawn(async move {
+ let _ = tx.send(match connection.await {
+ Ok(()) => None,
+ Err(e) => Some(format!("{e} (reason: {:?})", e.reason())),
+ });
+ });
+ (client.ready().await.expect("connection ready"), rx)
+}
+
+/// The connection-level error, or a note that the connection outlived the
+/// wait. Bounded so a server that resets the *stream* without ending the
+/// connection reports that instead of hanging the test.
+async fn connection_error(rx: tokio::sync::oneshot::Receiver<Option<String>>) -> String {
+ match tokio::time::timeout(Duration::from_secs(30), rx).await {
+ Ok(Ok(Some(e))) => e,
+ Ok(Ok(None)) => "the connection closed cleanly".into(),
+ Ok(Err(_)) => "the connection task vanished".into(),
+ Err(_) => "the connection was still open 30s later".into(),
+ }
+}
+
+/// An authenticated `_bulk` whose body is chunked at 100 bytes across ~400 DATA
+/// frames must return 200.
+///
+/// Sizing, both ways. The pre-fix budget is a fixed 25,600 bytes and a 100-byte
+/// frame charges 156, so it is spent by frame 165: against `h2` 0.4.16 this
+/// exact request is answered `GOAWAY ENHANCE_YOUR_CALM` mid-body, which is what
+/// was watched happen before the fix went in. The post-fix budget is half the
+/// 1 MiB connection window, 524,288 bytes, and ~400 frames charge at most
+/// 62,556 — an eighth of it, so the pass has real margin rather than sitting on
+/// a boundary. 40 KB is meanwhile an unremarkable `_bulk` by any other measure.
+#[tokio::test]
+async fn authenticated_bulk_chunked_at_100_bytes_completes() {
+ let server = boot().await;
+
+ let mut body = String::new();
+ let mut docs = 0;
+ while body.len() < 40_000 {
+ body.push_str("{\"index\":{\"_index\":\"issue485\"}}\n");
+ body.push_str(&format!(
+ "{{\"n\":{docs},\"text\":\"a legitimate document that the client happens to stream\"}}\n"
+ ));
+ docs += 1;
+ }
+ let body = Bytes::from(body);
+ let frames = body.len().div_ceil(CHUNK);
+
+ let (mut client, conn_err) = connect(server.es_port).await;
+ let request = Request::builder()
+ .method("POST")
+ .uri(format!("http://127.0.0.1:{}/_bulk", server.es_port))
+ .header("content-type", "application/x-ndjson")
+ .header("authorization", format!("ApiKey {ADMIN_KEY}"))
+ .body(())
+ .unwrap();
+ let (response, mut send) = client.send_request(request, false).unwrap();
+ send.reserve_capacity(body.len() + 1);
+
+ // One `send_data` per DATA frame, queued back-to-back with nothing awaited
+ // in between: the whole body is handed to the connection before it is
+ // driven, so it goes out as one burst of ~400 small DATA frames. That is
+ // the shape the issue reports — "ordinary socket writes and no artificial
+ // pauses" — and it is load-bearing here. Awaiting between frames (a
+ // `yield_now`, a sleep, a round trip) hands the server time to read and
+ // refund each frame, and the budget is then never exhausted: with a yield
+ // per frame this exact request passes on the *unfixed* stack too. A test
+ // written that way proves nothing, so do not add one back.
+ for (i, chunk) in body.chunks(CHUNK).enumerate() {
+ if send
+ .send_data(Bytes::copy_from_slice(chunk), false)
+ .is_err()
+ {
+ let goaway = connection_error(conn_err).await;
+ panic!(
+ "the server cut off an authenticated {}-byte _bulk at frame {i}/{frames} \
+ (chunked at {CHUNK} bytes): {goaway}",
+ body.len()
+ );
+ }
+ }
+ send.send_data(Bytes::new(), true).expect("end of stream");
+
+ let response = match response.await {
+ Ok(r) => r,
+ Err(e) => {
+ let goaway = connection_error(conn_err).await;
+ panic!(
+ "no response to an authenticated {}-byte _bulk chunked at {CHUNK} bytes \
+ ({frames} DATA frames): {e}; connection: {goaway}",
+ body.len()
+ );
+ }
+ };
+ assert_eq!(
+ response.status(),
+ 200,
+ "authenticated _bulk chunked at {CHUNK} bytes answered {}",
+ response.status()
+ );
+
+ let mut payload = Vec::new();
+ let mut stream = response.into_body();
+ while let Some(chunk) = stream.data().await {
+ let chunk = chunk.expect("read the response body");
+ stream.flow_control().release_capacity(chunk.len()).ok();
+ payload.extend_from_slice(&chunk);
+ }
+ let json: serde_json::Value = serde_json::from_slice(&payload).expect("bulk response is JSON");
+ assert_eq!(json["errors"], false, "{json}");
+ assert_eq!(
+ json["items"].as_array().map(|a| a.len()),
+ Some(docs),
+ "every document in the chunked body must have been indexed: {json}"
+ );
+}
+
+/// The guard on the fix: the flood RUSTSEC-2026-0258 was filed about — DATA
+/// frames with an empty payload, which cost the server memory and buy the
+/// client nothing — must still end in `GOAWAY ENHANCE_YOUR_CALM`.
+///
+/// Upstream caps these with a separate counter (100 frames) rather than out of
+/// the byte budget, which is why the budget could be raised for real traffic
+/// without reopening this. A failure here means the protection is gone, not
+/// that the fix is incomplete.
+#[tokio::test]
+async fn empty_data_frame_flood_is_still_refused() {
+ let server = boot().await;
+
+ let (mut client, conn_err) = connect(server.es_port).await;
+ let request = Request::builder()
+ .method("POST")
+ .uri(format!("http://127.0.0.1:{}/_bulk", server.es_port))
+ .header("content-type", "application/x-ndjson")
+ .header("authorization", format!("ApiKey {ADMIN_KEY}"))
+ .body(())
+ .unwrap();
+ let (_response, mut send) = client.send_request(request, false).unwrap();
+ send.reserve_capacity(1);
+
+ // Upstream's cap is 100 such frames. Yielding after each one lets the
+ // connection task write it and read what comes back, so the client learns
+ // it has been cut off within a few frames of the server deciding; without
+ // that it would simply queue all 1000 and find out at the end.
+ let mut cut_off_at = None;
+ for i in 0..1000 {
+ if send.send_data(Bytes::new(), false).is_err() {
+ cut_off_at = Some(i);
+ break;
+ }
+ tokio::task::yield_now().await;
+ }
+ let i = cut_off_at.expect("1000 empty DATA frames were accepted — the flood guard is gone");
+ assert!(
+ i < 500,
+ "the empty-frame flood ran to frame {i}; upstream's cap is 100 and the client should \
+ learn of it well before this"
+ );
+
+ // The teardown itself is the assertion. Which *description* of it the
+ // client gets is a race — the server sends `GOAWAY ENHANCE_YOUR_CALM` and
+ // then closes, and a client still writing into that socket often takes the
+ // `EPIPE` first — so the reason is only checked when the client managed to
+ // read it. What must never happen is a clean close or no close at all.
+ let goaway = connection_error(conn_err).await;
+ assert!(
+ !goaway.contains("closed cleanly") && !goaway.contains("still open"),
+ "the empty-frame flood ended without the connection being torn down: {goaway}"
+ );
+ if goaway.contains("reason: Some(") {
+ assert!(
+ goaway.contains("ENHANCE_YOUR_CALM"),
+ "empty-frame flood was stopped at frame {i} for the wrong reason: {goaway}"
+ );
+ }
+}