Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.
Python agentaianthropicclaude-codecompressioncontext-engineeringcontext-windowcursorfastapilangchainllmmcpopenaiprompt-engineeringproxypythonragtoken-optimizationtokenstypescript
fix(code): stop TS export duplication + comment displacement (#1906)
Hafiz Ismailadf8fed9bd75
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 718cd844c..0f017f9ce 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -105,6 +105,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0* **learn:** fix three Windows-specific failures in `headroom learn --verbosity` and CLI-backed analysis ([#1624](https://github.com/headroomlabs-ai/headroom/issues/1624)). `verbosity.py` read transcripts and profiles with the platform-default text codec instead of UTF-8, so non-ASCII content raised a silently-caught `UnicodeDecodeError`, producing `Sessions: 0, human turns: 0` for every project. `_greedy_path_decode` listed a directory's children with `is_dir()` inline in the same expression as `iterdir()`, so a single `PermissionError` on an inaccessible sibling (e.g. the `AppData\Local\Temporary Internet Files` junction present on most Windows profiles) aborted the whole listing and silently mis-decoded any project path that walked through it, causing `--project <path>` to report "No matching project" or resolve the wrong directory. `_call_cli_llm` launched CLI backends via `Popen`/`run`, which use `CreateProcess` on Windows and don't apply the shell's `PATHEXT` extension search, so an npm-installed `.cmd` shim (e.g. `claude`, `codex`) raised `FileNotFoundError` even though it was on `PATH`; a `shutil.which`-based retry now resolves the shim.* **proxy:** The Anthropic Messages route (`POST /v1/messages`) now honors the `x-headroom-base-url` per-request upstream override. It previously ignored the header and always forwarded to `api.anthropic.com`, so clients that speak the Anthropic Messages wire format while authenticating against a non-Anthropic gateway (e.g. OpenCode Zen) were rejected upstream with `401 invalid x-api-key`. The route now forwards to `<x-headroom-base-url>/v1/messages`, consistent with the OpenAI-compatible and passthrough routes ([#1760](https://github.com/headroomlabs-ai/headroom/issues/1760)).* **proxy:** the savings store now fsyncs its parent directory after the atomic rename, so the most recent `proxy_savings.json` write survives a power-loss or crash. `_save_locked` fsynced the temp file's contents but never the directory entry the rename created, leaving the rename itself non-durable on POSIX. Best-effort — a no-op on Windows and virtual filesystems where directory fsync is unsupported.+- **code:** fix two `CodeAwareCompressor` AST-reassembly bugs: an exported JS/TS function or class (`export function foo() {`) produced a duplicated `export export` keyword and invalid syntax, because line-based node slicing (used to preserve indentation) pulled in the preceding `export` sibling's text on top of the `export_statement` handler's own prefix reconstruction. Separately, in every supported language, a doc comment immediately above a top-level function, class, or type was detached from its declaration during extraction and re-emitted in a cluster at the end of the compressed output instead of staying attached to what it documents.## [0.31.0](https://github.com/headroomlabs-ai/headroom/compare/v0.30.0...v0.31.0) (2026-07-09)diff --git a/headroom/transforms/code_compressor.py b/headroom/transforms/code_compressor.pyindex 0a3e0c0d9..8f006ea27 100644--- a/headroom/transforms/code_compressor.py+++ b/headroom/transforms/code_compressor.py@@ -1235,18 +1235,21 @@ class CodeAwareCompressor(Transform):# Package declarations (Go, Java)if lang_config.package_node and node_type == lang_config.package_node:- structure.imports.insert(0, _get_node_text(node, code))+ leading = _get_leading_comment_text(node, code, captured_byte_ranges)+ structure.imports.insert(0, leading + _get_node_text(node, code))captured_byte_ranges.append((node.start_byte, node.end_byte))return# Import statementsif node_type in lang_config.import_nodes:- structure.imports.append(_get_node_text(node, code))+ leading = _get_leading_comment_text(node, code, captured_byte_ranges)+ structure.imports.append(leading + _get_node_text(node, code))captured_byte_ranges.append((node.start_byte, node.end_byte))return# Export statements (JS/TS) — may contain functions or re-exportsif node_type == "export_statement":+ leading = _get_leading_comment_text(node, code, captured_byte_ranges)text = _get_node_text(node, code)# Check if this export wraps a function or classhas_func_or_class = False@@ -1263,16 +1266,17 @@ class CodeAwareCompressor(Transform):export_prefix = _slice_code_bytes(code, node.start_byte, child.start_byte)export_suffix = _slice_code_bytes(code, child.end_byte, node.end_byte)structure.function_signatures.append(- export_prefix + compressed + export_suffix+ leading + export_prefix + compressed + export_suffix)breakif not has_func_or_class:- structure.imports.append(text)+ structure.imports.append(leading + text)captured_byte_ranges.append((node.start_byte, node.end_byte))return# Decorated definitions (Python)if lang_config.decorator_node and node_type == lang_config.decorator_node:+ leading = _get_leading_comment_text(node, code, captured_byte_ranges)decorator_text = []definition_compressed = Nonefor child in node.children:@@ -1287,7 +1291,7 @@ class CodeAwareCompressor(Transform):child, code, language, lang_config, body_limits, analysis)if decorator_text and definition_compressed:- full_def = "\n".join(decorator_text) + "\n" + definition_compressed+ full_def = leading + "\n".join(decorator_text) + "\n" + definition_compressed# Route to correct list based on inner definition typefor child in node.children:if child.type in lang_config.class_nodes:@@ -1296,25 +1300,27 @@ class CodeAwareCompressor(Transform):else:structure.function_signatures.append(full_def)elif definition_compressed:- structure.function_signatures.append(definition_compressed)+ structure.function_signatures.append(leading + definition_compressed)captured_byte_ranges.append((node.start_byte, node.end_byte))return# Function/method definitionsif node_type in lang_config.function_nodes:+ leading = _get_leading_comment_text(node, code, captured_byte_ranges)compressed = self._compress_function_ast(node, code, language, lang_config, body_limits, analysis)- structure.function_signatures.append(compressed)+ structure.function_signatures.append(leading + compressed)captured_byte_ranges.append((node.start_byte, node.end_byte))return# Class definitions — compress each method individuallyif node_type in lang_config.class_nodes:+ leading = _get_leading_comment_text(node, code, captured_byte_ranges)compressed = self._compress_class_ast(node, code, language, lang_config, body_limits, analysis)- structure.class_definitions.append(compressed)+ structure.class_definitions.append(leading + compressed)captured_byte_ranges.append((node.start_byte, node.end_byte))trailing_semicolon = _get_same_line_trailing_semicolon(node)if trailing_semicolon is not None:@@ -1325,7 +1331,8 @@ class CodeAwareCompressor(Transform):# Type definitionsif node_type in lang_config.type_nodes:- structure.type_definitions.append(_get_node_text(node, code))+ leading = _get_leading_comment_text(node, code, captured_byte_ranges)+ structure.type_definitions.append(leading + _get_node_text(node, code))captured_byte_ranges.append((node.start_byte, node.end_byte))return@@ -1374,8 +1381,7 @@ class CodeAwareCompressor(Transform):# (methods inside classes).code_lines = code.split("\n")start_row = node.start_point[0]- end_row = node.end_point[0]- node_lines = code_lines[start_row : end_row + 1]+ node_lines = _get_node_lines(node, code_lines)node_text = "\n".join(node_lines)func_name = _get_definition_name(node)@@ -1640,8 +1646,7 @@ class CodeAwareCompressor(Transform):# Use line-based extraction to preserve indentationcode_lines = code.split("\n")start_row = node.start_point[0]- end_row = node.end_point[0]- node_lines = code_lines[start_row : end_row + 1]+ node_lines = _get_node_lines(node, code_lines)node_text = "\n".join(node_lines)# Find the class/member container. For some languages this is not the@@ -2029,6 +2034,58 @@ def _get_node_text(node: Any, code: str) -> str:return _slice_code_bytes(code, node.start_byte, node.end_byte)+_COMMENT_NODE_TYPES = frozenset({"comment", "line_comment", "block_comment"})+++def _get_leading_comment_text(+ node: Any, code: str, captured_byte_ranges: list[tuple[int, int]]+) -> str:+ """Collect contiguous doc-comment siblings immediately preceding a node.++ Doc comments are top-level siblings of the declaration they document, not+ children of it. Left uncaptured, they fall through to the leftover+ top-level sweep and get grouped separately from the declarations they+ document instead of staying attached. Only comments with no blank line+ before the node (or the next comment) are treated as attached.+ """+ comments: list[Any] = []+ sibling = getattr(node, "prev_sibling", None)+ anchor_row = node.start_point[0]+ while (+ sibling is not None+ and sibling.type in _COMMENT_NODE_TYPES+ and (anchor_row - sibling.end_point[0] <= 1)+ ):+ comments.append(sibling)+ anchor_row = sibling.start_point[0]+ sibling = getattr(sibling, "prev_sibling", None)+ if not comments:+ return ""+ comments.reverse()+ captured_byte_ranges.extend((c.start_byte, c.end_byte) for c in comments)+ return "\n".join(_get_node_text(c, code) for c in comments) + "\n"+++def _get_node_lines(node: Any, code_lines: list[str]) -> list[str]:+ """Line-based slice of a node's source, preserving original indentation.++ Line-based (not byte-offset) slicing is used deliberately so leading+ whitespace survives for indented nested definitions (e.g. methods inside+ a class). But when a node shares its first line with a preceding sibling+ (e.g. the `export` keyword in `export function foo() {`), a naive+ full-line slice pulls in that sibling's text too — and callers that+ reconstruct the wrapper (re-adding the sibling text themselves) end up+ duplicating it. Trim the sibling prefix from the first line when it's not+ pure whitespace; keep the whole line (indentation intact) otherwise.+ """+ start_row, start_col = node.start_point+ end_row = node.end_point[0]+ node_lines = list(code_lines[start_row : end_row + 1])+ if node_lines and node_lines[0][:start_col].strip():+ node_lines[0] = node_lines[0][start_col:]+ return node_lines++def _get_same_line_trailing_semicolon(node: Any) -> Any | None:"""Return a trailing semicolon sibling that belongs to this declaration."""next_sibling = getattr(node, "next_sibling", None)diff --git a/tests/test_code_aware_brace_comment_regressions.py b/tests/test_code_aware_brace_comment_regressions.pynew file mode 100644index 000000000..cee8f5c5a--- /dev/null+++ b/tests/test_code_aware_brace_comment_regressions.py@@ -0,0 +1,125 @@+"""Regression tests for two CodeAwareCompressor AST-reassembly bugs found+while investigating a reported Go brace-duplication issue.++1. `export` keyword duplication (TS/JS): `_compress_function_ast` and+ `_compress_class_ast` use LINE-based slicing (not byte-offset) to+ preserve indentation for nested definitions. When a function/class shares+ its first line with a preceding sibling — e.g. the `export` keyword in+ `export function foo() {`, a sibling of the function inside+ `export_statement`, not part of the function node itself — a naive+ full-line slice pulled that sibling's text in too. The `export_statement`+ handler then re-prepended the same `export` text, producing+ `export export function foo() {` (invalid syntax, silently discarded by+ the `_verify_syntax` fallback).+2. Doc-comment displacement (all languages): doc comments are top-level+ siblings of the declaration they document, not children of it. Left+ unattached during AST extraction, they fell through to a "leftover+ top-level code" bucket that `_assemble_compressed` emits as one block+ after every function signature — detaching every doc comment from what it+ documents and dumping them all at the end of the file.+"""++from __future__ import annotations++import pytest++from headroom.transforms.code_compressor import (+ CodeAwareCompressor,+ CodeCompressorConfig,+ CodeLanguage,+ _check_tree_sitter_available,+)++pytestmark = pytest.mark.skipif(+ not _check_tree_sitter_available(),+ reason="tree-sitter not installed (pip install headroom-ai[code])",+)++TS_EXPORTED = """export interface User {+ id: string;+ name: string;+}++/**+ * Fetches a user by id.+ */+export function getUser(id: string): User {+ return { id, name: "test" };+}++/**+ * Greets a user by name.+ */+export function greet(user: User): string {+ return `Hello, ${user.name}!`;+}++export class UserStore {+ private users: User[] = [];++ add(user: User): void {+ this.users.push(user);+ }+}+"""++GO_DOC_COMMENTS = """package main++import "fmt"++// Add adds two integers together and returns the sum.+func Add(a, b int) int {+\treturn a + b+}++// Greet returns a friendly greeting for the given name.+func Greet(name string) string {+\treturn fmt.Sprintf("Hello, %s!", name)+}+"""+++def _compress_ast(code: str, language: CodeLanguage):+ compressor = CodeAwareCompressor(CodeCompressorConfig())+ compressed, _, _ = compressor._compress_with_ast(code, language, "", None)+ return compressor, compressed+++def test_ts_export_keyword_not_duplicated() -> None:+ """`export function`/`export class` must not become `export export ...`."""+ compressor, compressed = _compress_ast(TS_EXPORTED, CodeLanguage.TYPESCRIPT)++ assert "export export" not in compressed, compressed+ assert compressor._verify_syntax(compressed, CodeLanguage.TYPESCRIPT) is True+++def test_ts_doc_comments_stay_attached_to_declaration() -> None:+ """A `/** ... */` doc comment must stay immediately before the function it+ documents, not get dumped in a cluster at the end of the output."""+ _, compressed = _compress_ast(TS_EXPORTED, CodeLanguage.TYPESCRIPT)++ assert "/**\n * Fetches a user by id.\n */\nexport function getUser" in compressed+ assert "/**\n * Greets a user by name.\n */\nexport function greet" in compressed+++def test_go_doc_comments_stay_attached_to_function() -> None:+ """Same doc-comment-attachment bug, Go's `//` line-comment form."""+ compressor, compressed = _compress_ast(GO_DOC_COMMENTS, CodeLanguage.GO)++ assert "// Add adds two integers together and returns the sum.\nfunc Add" in compressed+ assert "// Greet returns a friendly greeting for the given name.\nfunc Greet" in compressed+ assert compressor._verify_syntax(compressed, CodeLanguage.GO) is True+++def test_actual_typescript_compression() -> None:+ """Parity with the existing JS/Python/Go 'actual compression' tests —+ real TS input must actually compress, not silently no-op."""+ config = CodeCompressorConfig(min_tokens_for_compression=10, enable_ccr=False)+ compressor = CodeAwareCompressor(config)+ code = TS_EXPORTED * 3 # large enough to trigger body elision++ result = compressor.compress(code, language="typescript")++ assert result.compression_ratio < 1.0+ assert result.syntax_valid is True+ assert result.language == CodeLanguage.TYPESCRIPTdiff --git a/tests/test_transforms/test_code_compressor.py b/tests/test_transforms/test_code_compressor.pyindex 774d9b6eb..059e35cac 100644--- a/tests/test_transforms/test_code_compressor.py+++ b/tests/test_transforms/test_code_compressor.py@@ -753,7 +753,7 @@ class TestTreeSitterIntegration:assert result.language == CodeLanguage.JAVASCRIPTdef test_actual_go_compression(self):- """Test Go code is processed (compression may fall back due to nested structures)."""+ """Test actual compression of Go code."""config = CodeCompressorConfig(min_tokens_for_compression=10,enable_ccr=False,@@ -763,12 +763,9 @@ class TestTreeSitterIntegration:result = compressor.compress(code, language="go")- # Go code is processed and returns valid output- # Note: compression_ratio may be 1.0 if compression produces invalid syntax- # and falls back to original (Go has complex nested brace handling)+ assert result.compression_ratio < 1.0assert result.syntax_valid is Trueassert result.language == CodeLanguage.GO- assert result.compressed # Some output is produced@pytest.mark.parametrize((