AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
fix(http2): raise the small-DATA-frame budget with the window (h2 0.4.19) (#908)
xerj-org01ef96293700
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 638dd0ae7..35f578a3c 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -234,6 +234,73 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0separate work: it has to prove the #825 union/sum contract with tombstonespresent, and is not attempted here.+- **A legitimate authenticated request whose body arrived in small HTTP/2 DATA+ frames could be cut off with `GOAWAY ENHANCE_YOUR_CALM`**+ ([#485](https://github.com/xerj-org/xerj/issues/485)). The rc.18 `h2` bump for+ RUSTSEC-2026-0258 — described at length in that release's notes below — added+ a per-connection budget for small DATA frames: a frame under 256 bytes+ consumes `256 - payload_len`, a frame of 256 or more replenishes, and the+ refund for a small frame lands only when the application *reads* it. In `h2`+ 0.4.16 that budget was a fixed 25,600 bytes however large a window the server+ had advertised, while hyper advertises a 1 MiB HTTP/2 connection window — so a+ well-behaved client could have 40× the budget in flight before the handler was+ ever scheduled. Any client that chunks small (a streaming JSON encoder, a+ per-document flush, a gRPC stream of small messages) lost its whole+ *connection* on a request that would otherwise have returned 200.++ `h2` is now floored at 0.4.19 in `[workspace.dependencies]`+ (`engine/Cargo.toml`), where the budget is half the configured connection+ window with 25,600 bytes as the minimum — 512 KiB against hyper's 1 MiB+ window — and the empty-DATA-frame flood the advisory was actually filed about+ is capped by its own counter (100 frames per connection) rather than out of+ the byte budget. The mitigation is not disabled or weakened here, and no+ XERJ-side window or budget knob was changed; the floor sits in the workspace+ manifest so `cargo update` cannot resolve back to a fixed-budget release.++ Measured on this box, same harness on both sides (`axum::serve`, hyper 1.9.0,+ a handler that reads the whole body, one DATA frame per chunk queued+ back-to-back with no pause), bisecting for the largest body that completes.+ The boundary is not a constant — it is a race between arriving charges and+ the refunds a reading handler returns — so these are the min–max of three+ bisections each, not thresholds to design against:++ | DATA frame size | before (h2 0.4.16) | after (h2 0.4.19) |+ |---|---|---|+ | 64 B | 9.3–16.5 KB (145–258 frames) | 262 KB–6.4 MB (4,095–99,793 frames) |+ | 100 B | 18.9–27.2 KB (189–272 frames) | 410 KB–4.4 MB (4,095–43,561 frames) |+ | 128 B | 29.3–36.6 KB (229–286 frames) | 2.1–4.7 MB (16,127–36,628 frames) |+ | 200 B | 141–419 KB (703–2,094 frames) | no cut-off up to 52 MB |+ | 256 B | no cut-off up to 8 MB | no cut-off (never charges) |++ Where the safe line now sits, exactly: with a budget of half the window, a+ frame of **171 bytes or more** cannot exhaust it even if the application never+ reads a byte — at most `window / size` frames can be in flight, each charging+ `256 - size`, and `(256 - size) ≤ size / 2` from 171 up. That is why the+ 200 B row has no ceiling at all. Below 171 bytes there is still a ceiling and+ it still depends on how fast the handler drains; it is now one to two orders+ of magnitude further out. A frame of 256 bytes or more never charges, before+ or after. The rc.18 note advising clients to chunk at 256 bytes or above is+ no longer needed for ordinary bodies.++ Deliberately abusive traffic is still refused, and that is asserted rather+ than assumed: a flood of empty DATA frames still ends in `GOAWAY+ ENHANCE_YOUR_CALM` (upstream's 100-frame cap, unchanged), and a flood of+ 1-byte frames is still cut off — on this harness the client saw the+ connection go at roughly frame 4,600 instead of roughly frame 100.++ What was measured where. The table above comes from a standalone harness+ pinned to this workspace's own axum/hyper/h2 versions, not from the `xerj`+ binary. The binary itself is covered by+ `engine/crates/xerj-server/tests/http2_small_data_frames.rs`, which spawns the+ real server with authentication on and drives a 40 KB authenticated `_bulk`+ at 100 bytes per DATA frame (401 frames) through the ES-compat listener over+ prior-knowledge h2c, asserting a 200 with every document indexed. Against the+ unfixed dependency that same test is answered+ `GOAWAY ENHANCE_YOUR_CALM (too_many_data_frames)` mid-body, which is how it+ was checked. A second test holds the empty-frame guard in place. The TLS+ listener and the gRPC listener sit on the same `h2` and take the same fix,+ but neither is exercised by these tests.+- **`xerj autoindex --no-graph` no longer aborts with `Access denied(os error 5)` on Windows**([#482](https://github.com/xerj-org/xerj/issues/482)). The durable-snapshot@@ -971,7 +1038,9 @@ silently reattributed.drains — measurements on two machines differed in both directions, so treatthe hazard as real and the frequency as unpredictable rather than as a rate.Tracked as [#485](https://github.com/xerj-org/xerj/issues/485).- If you control the client, chunk at 256 bytes or above.+ If you control the client, chunk at 256 bytes or above. **On this release+ only** — see the Unreleased entry for #485 above, where the `h2` floor moves+ to 0.4.19 and the budget starts scaling with the connection window.XERJ has such a path: `auth_middleware` is an axum layer, so a request thatfails authentication is answered `401` without the handler ever extracting itsdiff --git a/engine/Cargo.lock b/engine/Cargo.lockindex 62de03471..f828b86c3 100644--- a/engine/Cargo.lock+++ b/engine/Cargo.lock@@ -1961,9 +1961,9 @@ checksum = "b40ca9252762c466af32d0b1002e91e4e1bc5398f77455e55474deb466355ff5"[[package]]name = "h2"-version = "0.4.16"+version = "0.4.19"source = "registry+https://github.com/rust-lang/crates.io-index"-checksum = "a9f37a958b41b3b19ee2707c06439c0e9e547e847223eb791ecb0cb821c65e27"+checksum = "ef8e5e5a340588f4452631496976cf8636d4a7ecf600239fdc27615d2530bc16"dependencies = ["atomic-waker","bytes",@@ -6961,6 +6961,7 @@ dependencies = ["axum-server","chrono","flate2",+ "h2","jemalloc_pprof","libc","memchr",diff --git a/engine/Cargo.toml b/engine/Cargo.tomlindex e56082f3f..039393ec3 100644--- a/engine/Cargo.toml+++ b/engine/Cargo.toml@@ -31,6 +31,19 @@ authors = ["xerj Contributors"]tokio = { version = "1", features = ["full"] }# HTTPaxum = { version = "0.7", features = ["macros"] }+# `h2` is the HTTP/2 implementation under every listener (hyper for REST ++# ES-compat, tonic for gRPC) and under `reqwest` on the client side. Nothing in+# this workspace calls it directly outside the `xerj-server` regression test,+# but the floor is declared here on purpose: 0.4.16-0.4.18 size the small-DATA-+# frame budget (the RUSTSEC-2026-0258 mitigation) at a fixed 25,600 bytes,+# which a legitimate authenticated request chunked below ~200 bytes exhausts —+# the client is answered `GOAWAY ENHANCE_YOUR_CALM` (#485). 0.4.19 scales the+# budget with the configured connection window instead (half of it, floor+# 25,600), so hyper's 1 MiB server window yields 512 KiB of headroom while the+# empty-DATA-frame vector the advisory was filed for stays capped at 100 frames+# by a separate counter. Declaring the floor here means `cargo update` cannot+# resolve back to a version with the fixed budget.+h2 = "0.4.19"tower = "0.4"tower-http = { version = "0.5", features = ["cors", "trace", "limit", "decompression-gzip"] }# Serializationdiff --git a/engine/crates/xerj-server/Cargo.toml b/engine/crates/xerj-server/Cargo.tomlindex 3f8f0d786..1561fbcd9 100644--- a/engine/crates/xerj-server/Cargo.toml+++ b/engine/crates/xerj-server/Cargo.toml@@ -97,5 +97,10 @@ protox = { workspace = true }# HTTPS round-trip test against the in-process rustls listener.reqwest = { workspace = true }tempfile = { workspace = true }+# HTTP/2 *as a client*, so `tests/http2_small_data_frames.rs` can choose the+# DATA-frame size — the one thing no ordinary HTTP client exposes, and the+# whole subject of #485. It also pins the workspace floor at the h2 release+# whose small-frame budget scales with the connection window.+h2 = { workspace = true }flate2 = "1"tokio-stream = "0.1"diff --git a/engine/crates/xerj-server/tests/http2_small_data_frames.rs b/engine/crates/xerj-server/tests/http2_small_data_frames.rsnew file mode 100644index 000000000..ce675e916--- /dev/null+++ b/engine/crates/xerj-server/tests/http2_small_data_frames.rs@@ -0,0 +1,353 @@+//! Issue #485: a legitimate, authenticated request whose body arrives in small+//! HTTP/2 DATA frames must not be answered `GOAWAY ENHANCE_YOUR_CALM`.+//!+//! `h2` 0.4.16 shipped the RUSTSEC-2026-0258 mitigation as a per-connection+//! byte budget: a DATA frame under 256 bytes consumes `256 - payload_len` of+//! it, a frame of 256 or more replenishes, and the budget is a fixed 25,600+//! bytes (`DEFAULT_DATA_FRAME_OVERHEAD_THRESHOLD * 100`). The charge lands when+//! the frame arrives and the refund only when the application reads it, so a+//! client that chunks small — a streaming JSON encoder, a per-document flush, a+//! gRPC stream of small messages — drains the whole budget before the handler+//! is scheduled and has its *connection* closed. Run against the pre-fix+//! dependency, the first test below is answered `GOAWAY ENHANCE_YOUR_CALM+//! (too_many_data_frames)` part-way through a 40 KB body.+//!+//! Two things make this a process-level test rather than a unit test:+//!+//! 1. The budget is enforced inside `h2`, below hyper, below axum. Nothing in+//! XERJ's own code can be called to observe it — only a real HTTP/2 peer+//! sending real DATA frames at a real listener can.+//! 2. The frame size is what matters, and no ordinary HTTP client lets you+//! choose it. These tests therefore speak HTTP/2 directly (`h2` as a client,+//! prior-knowledge h2c against the ES-compat listener) so every `send_data`+//! is exactly one DATA frame of exactly the size asked for.+//!+//! The pair is deliberate. The first test is the regression — the traffic that+//! must work. The second is the guard on the fix: the empty-DATA-frame flood+//! the advisory was actually filed about must still be cut off, so raising the+//! headroom for legitimate clients cannot be mistaken for switching the+//! protection off.++use std::io::Read;+use std::net::TcpListener;+use std::process::{Child, Command, Stdio};+use std::sync::{Arc, Mutex};+use std::time::{Duration, Instant};++use axum::body::Bytes;+use axum::http::Request;+use tokio::net::TcpStream;++/// Fixed key so the request below is unambiguously *authenticated*: the issue+/// is about traffic that would otherwise have succeeded, not about a 401 path.+const ADMIN_KEY: &str = "issue485-admin-key-not-a-secret";++/// 100 bytes — the size named in the issue, and well inside the band that the+/// fixed 25,600-byte budget rejects (`256 - 100 = 156` charged per frame).+const CHUNK: usize = 100;++/// TOML-safe rendering of a temp path (Windows backslashes would be escape+/// sequences inside a basic string; forward slashes work everywhere).+fn toml_path(p: &std::path::Path) -> String {+ p.to_string_lossy().replace('\\', "/")+}++/// Three distinct free ports, held simultaneously so they cannot collide, then+/// released for the child. `Config::validate` requires all three to differ.+fn three_free_ports() -> (u16, u16, u16) {+ let held: Vec<TcpListener> = (0..3)+ .map(|_| TcpListener::bind("127.0.0.1:0").unwrap())+ .collect();+ let p: Vec<u16> = held+ .iter()+ .map(|l| l.local_addr().unwrap().port())+ .collect();+ (p[0], p[1], p[2])+}++/// Kills the child on the way out, including when a test panics — a leaked+/// server would hold its data dir and its ports for the rest of the run.+struct Server {+ child: Child,+ es_port: u16,+ _dir: tempfile::TempDir,+ output: Arc<Mutex<String>>,+}++impl Drop for Server {+ fn drop(&mut self) {+ let _ = self.child.kill();+ let _ = self.child.wait();+ }+}++fn drain_into(mut r: impl Read + Send + 'static, buf: Arc<Mutex<String>>) {+ std::thread::spawn(move || {+ let mut chunk = [0u8; 4096];+ while let Ok(n) = r.read(&mut chunk) {+ if n == 0 {+ break;+ }+ buf.lock()+ .unwrap()+ .push_str(&String::from_utf8_lossy(&chunk[..n]));+ }+ });+}++/// Boot the real binary with authentication ON and wait until the ES-compat+/// listener answers its readiness probe.+async fn boot() -> Server {+ let dir = tempfile::tempdir().unwrap();+ let (rest, grpc, es) = three_free_ports();+ let config = format!(+ r#"+[server]+bind_address = "127.0.0.1"+rest_port = {rest}+grpc_port = {grpc}+es_compat_port = {es}+data_dir = "{data}"++[auth]+enabled = true+admin_api_key = "{ADMIN_KEY}"++[limits]+disk_flood_stage_percent = 0+"#,+ data = toml_path(&dir.path().join("data")),+ );+ let config_path = dir.path().join("xerj.toml");+ std::fs::write(&config_path, config).unwrap();++ let mut child = Command::new(env!("CARGO_BIN_EXE_xerj"))+ .arg("--config")+ .arg(&config_path)+ .stdout(Stdio::piped())+ .stderr(Stdio::piped())+ .spawn()+ .expect("spawn xerj");++ let output = Arc::new(Mutex::new(String::new()));+ drain_into(child.stdout.take().unwrap(), output.clone());+ drain_into(child.stderr.take().unwrap(), output.clone());++ let server = Server {+ child,+ es_port: es,+ _dir: dir,+ output,+ };++ // `/health/ready` is auth-exempt and mounted on the ES-compat router.+ let client = reqwest::Client::new();+ let url = format!("http://127.0.0.1:{es}/health/ready");+ let deadline = Instant::now() + Duration::from_secs(180);+ loop {+ if let Ok(r) = client.get(&url).send().await {+ if r.status().is_success() {+ return server;+ }+ }+ assert!(+ Instant::now() < deadline,+ "xerj never became ready on :{es}\n{}",+ server.output.lock().unwrap()+ );+ tokio::time::sleep(Duration::from_millis(100)).await;+ }+}++/// Prior-knowledge h2c handshake against the ES-compat listener, plus a handle+/// that resolves to the connection-level error (the `GOAWAY`, when there is+/// one) once the connection ends.+async fn connect(+ port: u16,+) -> (+ h2::client::SendRequest<Bytes>,+ tokio::sync::oneshot::Receiver<Option<String>>,+) {+ let tcp = TcpStream::connect(("127.0.0.1", port))+ .await+ .expect("connect to the ES-compat listener");+ tcp.set_nodelay(true).unwrap();+ let (client, connection) = h2::client::handshake(tcp)+ .await+ .expect("HTTP/2 handshake — the listener must speak prior-knowledge h2c");+ let (tx, rx) = tokio::sync::oneshot::channel();+ tokio::spawn(async move {+ let _ = tx.send(match connection.await {+ Ok(()) => None,+ Err(e) => Some(format!("{e} (reason: {:?})", e.reason())),+ });+ });+ (client.ready().await.expect("connection ready"), rx)+}++/// The connection-level error, or a note that the connection outlived the+/// wait. Bounded so a server that resets the *stream* without ending the+/// connection reports that instead of hanging the test.+async fn connection_error(rx: tokio::sync::oneshot::Receiver<Option<String>>) -> String {+ match tokio::time::timeout(Duration::from_secs(30), rx).await {+ Ok(Ok(Some(e))) => e,+ Ok(Ok(None)) => "the connection closed cleanly".into(),+ Ok(Err(_)) => "the connection task vanished".into(),+ Err(_) => "the connection was still open 30s later".into(),+ }+}++/// An authenticated `_bulk` whose body is chunked at 100 bytes across ~400 DATA+/// frames must return 200.+///+/// Sizing, both ways. The pre-fix budget is a fixed 25,600 bytes and a 100-byte+/// frame charges 156, so it is spent by frame 165: against `h2` 0.4.16 this+/// exact request is answered `GOAWAY ENHANCE_YOUR_CALM` mid-body, which is what+/// was watched happen before the fix went in. The post-fix budget is half the+/// 1 MiB connection window, 524,288 bytes, and ~400 frames charge at most+/// 62,556 — an eighth of it, so the pass has real margin rather than sitting on+/// a boundary. 40 KB is meanwhile an unremarkable `_bulk` by any other measure.+#[tokio::test]+async fn authenticated_bulk_chunked_at_100_bytes_completes() {+ let server = boot().await;++ let mut body = String::new();+ let mut docs = 0;+ while body.len() < 40_000 {+ body.push_str("{\"index\":{\"_index\":\"issue485\"}}\n");+ body.push_str(&format!(+ "{{\"n\":{docs},\"text\":\"a legitimate document that the client happens to stream\"}}\n"+ ));+ docs += 1;+ }+ let body = Bytes::from(body);+ let frames = body.len().div_ceil(CHUNK);++ let (mut client, conn_err) = connect(server.es_port).await;+ let request = Request::builder()+ .method("POST")+ .uri(format!("http://127.0.0.1:{}/_bulk", server.es_port))+ .header("content-type", "application/x-ndjson")+ .header("authorization", format!("ApiKey {ADMIN_KEY}"))+ .body(())+ .unwrap();+ let (response, mut send) = client.send_request(request, false).unwrap();+ send.reserve_capacity(body.len() + 1);++ // One `send_data` per DATA frame, queued back-to-back with nothing awaited+ // in between: the whole body is handed to the connection before it is+ // driven, so it goes out as one burst of ~400 small DATA frames. That is+ // the shape the issue reports — "ordinary socket writes and no artificial+ // pauses" — and it is load-bearing here. Awaiting between frames (a+ // `yield_now`, a sleep, a round trip) hands the server time to read and+ // refund each frame, and the budget is then never exhausted: with a yield+ // per frame this exact request passes on the *unfixed* stack too. A test+ // written that way proves nothing, so do not add one back.+ for (i, chunk) in body.chunks(CHUNK).enumerate() {+ if send+ .send_data(Bytes::copy_from_slice(chunk), false)+ .is_err()+ {+ let goaway = connection_error(conn_err).await;+ panic!(+ "the server cut off an authenticated {}-byte _bulk at frame {i}/{frames} \+ (chunked at {CHUNK} bytes): {goaway}",+ body.len()+ );+ }+ }+ send.send_data(Bytes::new(), true).expect("end of stream");++ let response = match response.await {+ Ok(r) => r,+ Err(e) => {+ let goaway = connection_error(conn_err).await;+ panic!(+ "no response to an authenticated {}-byte _bulk chunked at {CHUNK} bytes \+ ({frames} DATA frames): {e}; connection: {goaway}",+ body.len()+ );+ }+ };+ assert_eq!(+ response.status(),+ 200,+ "authenticated _bulk chunked at {CHUNK} bytes answered {}",+ response.status()+ );++ let mut payload = Vec::new();+ let mut stream = response.into_body();+ while let Some(chunk) = stream.data().await {+ let chunk = chunk.expect("read the response body");+ stream.flow_control().release_capacity(chunk.len()).ok();+ payload.extend_from_slice(&chunk);+ }+ let json: serde_json::Value = serde_json::from_slice(&payload).expect("bulk response is JSON");+ assert_eq!(json["errors"], false, "{json}");+ assert_eq!(+ json["items"].as_array().map(|a| a.len()),+ Some(docs),+ "every document in the chunked body must have been indexed: {json}"+ );+}++/// The guard on the fix: the flood RUSTSEC-2026-0258 was filed about — DATA+/// frames with an empty payload, which cost the server memory and buy the+/// client nothing — must still end in `GOAWAY ENHANCE_YOUR_CALM`.+///+/// Upstream caps these with a separate counter (100 frames) rather than out of+/// the byte budget, which is why the budget could be raised for real traffic+/// without reopening this. A failure here means the protection is gone, not+/// that the fix is incomplete.+#[tokio::test]+async fn empty_data_frame_flood_is_still_refused() {+ let server = boot().await;++ let (mut client, conn_err) = connect(server.es_port).await;+ let request = Request::builder()+ .method("POST")+ .uri(format!("http://127.0.0.1:{}/_bulk", server.es_port))+ .header("content-type", "application/x-ndjson")+ .header("authorization", format!("ApiKey {ADMIN_KEY}"))+ .body(())+ .unwrap();+ let (_response, mut send) = client.send_request(request, false).unwrap();+ send.reserve_capacity(1);++ // Upstream's cap is 100 such frames. Yielding after each one lets the+ // connection task write it and read what comes back, so the client learns+ // it has been cut off within a few frames of the server deciding; without+ // that it would simply queue all 1000 and find out at the end.+ let mut cut_off_at = None;+ for i in 0..1000 {+ if send.send_data(Bytes::new(), false).is_err() {+ cut_off_at = Some(i);+ break;+ }+ tokio::task::yield_now().await;+ }+ let i = cut_off_at.expect("1000 empty DATA frames were accepted — the flood guard is gone");+ assert!(+ i < 500,+ "the empty-frame flood ran to frame {i}; upstream's cap is 100 and the client should \+ learn of it well before this"+ );++ // The teardown itself is the assertion. Which *description* of it the+ // client gets is a race — the server sends `GOAWAY ENHANCE_YOUR_CALM` and+ // then closes, and a client still writing into that socket often takes the+ // `EPIPE` first — so the reason is only checked when the client managed to+ // read it. What must never happen is a clean close or no close at all.+ let goaway = connection_error(conn_err).await;+ assert!(+ !goaway.contains("closed cleanly") && !goaway.contains("still open"),+ "the empty-frame flood ended without the connection being torn down: {goaway}"+ );+ if goaway.contains("reason: Some(") {+ assert!(+ goaway.contains("ENHANCE_YOUR_CALM"),+ "empty-frame flood was stopped at frame {i} for the wrong reason: {goaway}"+ );+ }+}