The world's first agentic reverse engineer.
fix disk issue
Amrutha0ef1aa54867
diff --git a/tools/xda/plan.md b/tools/xda/plan.mdnew file mode 100644index 0000000..fadc1d6--- /dev/null+++ b/tools/xda/plan.md@@ -0,0 +1,2748 @@+Random+docs/superpowers/plans/2026-03-20-xda-training-and-candle-inference.md++# XDA Per-Architecture Model Training & Candle Inference Implementation Plan++> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.++**Goal:** Build a complete pipeline to train per-architecture XDA function boundary detection models and load them for inference in Rust via candle.++**Architecture:** Two independent workstreams — (A) a Python training pipeline that produces `.safetensors` model artifacts per ISA, and (B) a Rust candle-based inference module that loads those artifacts and runs function boundary prediction. The training pipeline is a standalone tool in `tools/xda/`. The Rust inference lives in the `kong-rs` workspace as part of `kong-binary`.++**Tech Stack:**+- Training: Python, PyTorch, HuggingFace Transformers, pyelftools+- Inference: Rust, candle (candle-core, candle-nn, candle-transformers), serde+- Dataset: Debian arm64 packages, cross-compiled C/C++ corpus+- Model format: safetensors (shared between Python export and Rust loading)++---++## File Structure++### Python Training Pipeline (`tools/xda/`)++```+tools/xda/+├── pyproject.toml # uv project: torch, transformers, pyelftools, safetensors+├── README.md # How to train a new arch model+├── scripts/+│ ├── build_corpus.sh # Cross-compile C projects for a target arch+│ └── fetch_debian_packages.sh # Download Debian arm64 packages + dbgsym+├── src/+│ ├── __init__.py+│ ├── extract_labels.py # DWARF → function boundaries → per-byte labels+│ ├── dataset.py # ByteDataset: chunking, train/val split, DataLoader+│ ├── tokenizer.py # Byte tokenizer (0-255 + 5 special tokens)+│ ├── model.py # XDA model definition (BERT + 2-layer MLP head)+│ ├── pretrain.py # Masked byte modeling pre-training+│ ├── finetune.py # Function boundary fine-tuning+│ └── export.py # PyTorch checkpoint → safetensors + config.json+├── configs/+│ ├── pretrain.yaml # Pre-training hyperparameters+│ └── finetune.yaml # Fine-tuning hyperparameters+└── tests/+ ├── test_extract_labels.py+ ├── test_dataset.py+ ├── test_tokenizer.py+ └── test_model.py+```++### Rust Inference (`kong-rs/crates/kong-ml/`)++```+kong-rs/+├── Cargo.toml # Workspace root+├── crates/+│ ├── kong-types/+│ │ ├── Cargo.toml+│ │ └── src/+│ │ ├── lib.rs+│ │ ├── binary.rs # BinaryInfo, Arch enum+│ │ └── function.rs # FunctionBoundary, AmbiguousRegion+│ └── kong-ml/+│ ├── Cargo.toml # candle-core, candle-nn, candle-transformers, safetensors, serde, dirs+│ └── src/+│ ├── lib.rs # pub mod tokenizer, model, detector, registry+│ ├── tokenizer.rs # ByteTokenizer: bytes → token IDs+│ ├── config.rs # XdaConfig: loaded from config.json alongside weights+│ ├── model.rs # XdaModel: BERT encoder + classification head+│ ├── detector.rs # XdaDetector: sliding window, batching, thresholding+│ └── registry.rs # ModelRegistry: per-arch download, cache, load+├── tests/+│ └── xda_integration.rs # End-to-end: load model, run on test bytes, check predictions+└── models/+ └── .gitkeep # Model artifacts go here (gitignored, downloaded at runtime)+```++---++## Part A: Python Training Pipeline++### Task 1: Project Scaffolding & Byte Tokenizer++**Files:**+- Create: `tools/xda/pyproject.toml`+- Create: `tools/xda/src/__init__.py`+- Create: `tools/xda/src/tokenizer.py`+- Test: `tools/xda/tests/test_tokenizer.py`++- [ ] **Step 1: Create pyproject.toml**++```toml+[project]+name = "xda-training"+version = "0.1.0"+requires-python = ">=3.11"+dependencies = [+ "torch>=2.0",+ "transformers>=4.40",+ "pyelftools>=0.30",+ "safetensors>=0.4",+ "pyyaml>=6.0",+ "tqdm>=4.65",+]++[build-system]+requires = ["hatchling"]+build-backend = "hatchling.build"++[tool.hatch.build.targets.wheel]+packages = ["src"]+```++- [ ] **Step 2: Write the tokenizer test**++```python+# tools/xda/tests/test_tokenizer.py+from src.tokenizer import ByteTokenizer++def test_special_token_ids():+ tok = ByteTokenizer()+ assert tok.pad_id == 256+ assert tok.cls_id == 257+ assert tok.sep_id == 258+ assert tok.mask_id == 259+ assert tok.unk_id == 260+ assert tok.vocab_size == 261++def test_encode_bytes():+ tok = ByteTokenizer()+ raw = bytes([0x55, 0x48, 0x89, 0xe5])+ ids = tok.encode(raw)+ # [CLS] + byte IDs + [SEP]+ assert ids == [257, 0x55, 0x48, 0x89, 0xe5, 258]++def test_encode_with_padding():+ tok = ByteTokenizer()+ raw = bytes([0x55, 0x48])+ ids = tok.encode(raw, max_length=8)+ assert ids == [257, 0x55, 0x48, 258, 256, 256, 256, 256]+ assert len(ids) == 8++def test_attention_mask():+ tok = ByteTokenizer()+ raw = bytes([0x55, 0x48])+ ids = tok.encode(raw, max_length=8)+ mask = tok.attention_mask(ids)+ assert mask == [1, 1, 1, 1, 0, 0, 0, 0]+```++- [ ] **Step 3: Run test to verify it fails**++Run: `cd tools/xda && uv run pytest tests/test_tokenizer.py -v`+Expected: FAIL (module not found)++- [ ] **Step 4: Implement ByteTokenizer**++```python+# tools/xda/src/tokenizer.py+from dataclasses import dataclass+++@dataclass(frozen=True)+class ByteTokenizer:+ """Byte-level tokenizer for XDA. Maps raw bytes 0-255 to token IDs 0-255.+ Special tokens occupy IDs 256-260."""++ pad_id: int = 256+ cls_id: int = 257+ sep_id: int = 258+ mask_id: int = 259+ unk_id: int = 260+ vocab_size: int = 261++ def encode(self, raw_bytes: bytes, max_length: int | None = None) -> list[int]:+ ids = [self.cls_id] + list(raw_bytes) + [self.sep_id]+ if max_length is not None:+ ids = ids[:max_length]+ ids += [self.pad_id] * (max_length - len(ids))+ return ids++ def attention_mask(self, token_ids: list[int]) -> list[int]:+ return [0 if t == self.pad_id else 1 for t in token_ids]+```++- [ ] **Step 5: Run test to verify it passes**++Run: `cd tools/xda && uv run pytest tests/test_tokenizer.py -v`+Expected: PASS (all 4 tests)++- [ ] **Step 6: Commit**++```bash+git add tools/xda/pyproject.toml tools/xda/src/__init__.py tools/xda/src/tokenizer.py tools/xda/tests/test_tokenizer.py+git commit -m "feat(xda): scaffold training project, implement byte tokenizer"+```++---++### Task 2: Ground Truth Extraction from DWARF++**Files:**+- Create: `tools/xda/src/extract_labels.py`+- Test: `tools/xda/tests/test_extract_labels.py`++- [ ] **Step 1: Write the extraction test**++This test compiles a tiny C program, extracts labels, verifies correctness.++```python+# tools/xda/tests/test_extract_labels.py+import subprocess+import tempfile+from pathlib import Path++from src.extract_labels import extract_function_boundaries, generate_byte_labels+++def _compile_test_binary(src: str, out: Path, arch: str = "native") -> Path:+ """Compile a C string to a binary with debug info."""+ src_file = out / "test.c"+ bin_file = out / "test.elf"+ src_file.write_text(src)++ cc = "gcc" if arch == "native" else f"{arch}-linux-gnu-gcc"+ subprocess.run(+ [cc, "-g", "-O0", "-o", str(bin_file), str(src_file)],+ check=True,+ )+ return bin_file+++def test_extract_boundaries_from_simple_binary():+ src = """+ int add(int a, int b) { return a + b; }+ int mul(int a, int b) { return a * b; }+ int main() { return add(1, 2) + mul(3, 4); }+ """+ with tempfile.TemporaryDirectory() as tmp:+ binary = _compile_test_binary(src, Path(tmp))+ boundaries = extract_function_boundaries(str(binary))++ names = {b["name"] for b in boundaries}+ assert "add" in names+ assert "mul" in names+ assert "main" in names++ for b in boundaries:+ assert b["start"] < b["end"]+ assert b["end"] - b["start"] > 0+++def test_generate_byte_labels():+ src = "int foo() { return 42; } int main() { return foo(); }"+ with tempfile.TemporaryDirectory() as tmp:+ binary = _compile_test_binary(src, Path(tmp))+ boundaries = extract_function_boundaries(str(binary))+ text_bytes, labels = generate_byte_labels(str(binary), boundaries)++ assert len(text_bytes) == len(labels)+ assert len(text_bytes) > 0++ # At least some bytes should be labeled as function_start (1)+ assert 1 in labels+ # At least some bytes should be labeled as function_body (2)+ assert 2 in labels+ # At least some bytes should be non-function (0) unless .text is fully covered+```++- [ ] **Step 2: Run test to verify it fails**++Run: `cd tools/xda && uv run pytest tests/test_extract_labels.py -v`+Expected: FAIL (import error)++- [ ] **Step 3: Implement extract_labels.py**++```python+# tools/xda/src/extract_labels.py+"""Extract function boundaries from DWARF debug info and generate per-byte labels."""++from elftools.elf.elffile import ELFFile+++def extract_function_boundaries(elf_path: str) -> list[dict]:+ """Extract function start/end addresses from DWARF debug info.++ Returns list of {"name": str | None, "start": int, "end": int}.+ """+ functions: list[dict] = []+ with open(elf_path, "rb") as f:+ elf = ELFFile(f)+ if not elf.has_dwarf_info():+ return []++ dwarf = elf.get_dwarf_info()+ for cu in dwarf.iter_CUs():+ for die in cu.iter_DIEs():+ if die.tag != "DW_TAG_subprogram":+ continue+ if "DW_AT_low_pc" not in die.attributes:+ continue++ low_pc = die.attributes["DW_AT_low_pc"].value+ high_pc_attr = die.attributes.get("DW_AT_high_pc")+ if high_pc_attr is None:+ continue++ if high_pc_attr.form in ("DW_FORM_addr",):+ high_pc = high_pc_attr.value+ else:+ high_pc = low_pc + high_pc_attr.value++ if high_pc <= low_pc:+ continue++ name_attr = die.attributes.get("DW_AT_name")+ name = name_attr.value.decode() if name_attr else None+ functions.append({"name": name, "start": low_pc, "end": high_pc})++ return functions+++def generate_byte_labels(+ elf_path: str,+ boundaries: list[dict],+) -> tuple[bytes, list[int]]:+ """Generate per-byte labels for the .text section.++ Labels: 0 = non-function, 1 = function_start, 2 = function_body.+ Returns (text_section_bytes, labels).+ """+ with open(elf_path, "rb") as f:+ elf = ELFFile(f)+ text = elf.get_section_by_name(".text")+ if text is None:+ return b"", []++ text_start = text["sh_addr"]+ text_bytes = text.data()++ labels = [0] * len(text_bytes)++ for func in boundaries:+ start_off = func["start"] - text_start+ end_off = func["end"] - text_start++ if start_off < 0 or start_off >= len(text_bytes):+ continue+ end_off = min(end_off, len(text_bytes))++ labels[start_off] = 1 # function_start+ for i in range(start_off + 1, end_off):+ labels[i] = 2 # function_body++ return bytes(text_bytes), labels+```++- [ ] **Step 4: Run test to verify it passes**++Run: `cd tools/xda && uv run pytest tests/test_extract_labels.py -v`+Expected: PASS (both tests). Requires `gcc` available on system.++- [ ] **Step 5: Commit**++```bash+git add tools/xda/src/extract_labels.py tools/xda/tests/test_extract_labels.py+git commit -m "feat(xda): DWARF-based ground truth extraction for function boundaries"+```++---++### Task 3: Dataset Pipeline (Chunking + DataLoader)++**Files:**+- Create: `tools/xda/src/dataset.py`+- Test: `tools/xda/tests/test_dataset.py`++- [ ] **Step 1: Write the dataset test**++```python+# tools/xda/tests/test_dataset.py+import torch+from src.dataset import ByteDataset, chunk_labeled_bytes+from src.tokenizer import ByteTokenizer+++def test_chunk_labeled_bytes():+ text_bytes = bytes(range(256)) * 4 # 1024 bytes+ labels = [0] * 1024+ labels[0] = 1 # function start at byte 0+ labels[512] = 1 # function start at byte 512++ chunks = chunk_labeled_bytes(text_bytes, labels, window_size=512, stride=256)++ assert len(chunks) > 0+ for chunk in chunks:+ assert len(chunk["bytes"]) == 512+ assert len(chunk["labels"]) == 512+++def test_byte_dataset_shapes():+ text_bytes = bytes(range(256)) * 4+ labels = [0] * 1024+ labels[0] = 1++ chunks = chunk_labeled_bytes(text_bytes, labels, window_size=512, stride=256)+ ds = ByteDataset(chunks, ByteTokenizer())++ item = ds[0]+ assert item["input_ids"].shape == (514,) # 512 + [CLS] + [SEP]+ assert item["attention_mask"].shape == (514,)+ assert item["labels"].shape == (514,)+ assert item["input_ids"][0].item() == 257 # [CLS]+ assert item["input_ids"][-1].item() == 258 # [SEP]+ # Labels for [CLS] and [SEP] should be -100 (ignored in loss)+ assert item["labels"][0].item() == -100+ assert item["labels"][-1].item() == -100+++def test_byte_dataset_label_values():+ text_bytes = bytes([0x55, 0x48, 0x89, 0xe5, 0xc3, 0x00, 0x00, 0x00])+ labels = [1, 2, 2, 2, 2, 0, 0, 0]++ chunks = chunk_labeled_bytes(text_bytes, labels, window_size=8, stride=8)+ ds = ByteDataset(chunks, ByteTokenizer())++ item = ds[0]+ # Labels offset by 1 for [CLS] prefix+ assert item["labels"][1].item() == 1 # function_start+ assert item["labels"][2].item() == 2 # function_body+ assert item["labels"][6].item() == 0 # non-function+```++- [ ] **Step 2: Run test to verify it fails**++Run: `cd tools/xda && uv run pytest tests/test_dataset.py -v`+Expected: FAIL (import error)++- [ ] **Step 3: Implement dataset.py**++```python+# tools/xda/src/dataset.py+"""Chunking and DataLoader for XDA training."""++import torch+from torch.utils.data import Dataset++from .tokenizer import ByteTokenizer+++def chunk_labeled_bytes(+ text_bytes: bytes,+ labels: list[int],+ window_size: int = 512,+ stride: int = 256,+) -> list[dict]:+ """Chunk labeled bytes into fixed-size windows with overlap."""+ chunks = []+ for i in range(0, max(1, len(text_bytes) - window_size + 1), stride):+ end = i + window_size+ if end > len(text_bytes):+ # Pad the last chunk+ b = text_bytes[i:] + bytes(end - len(text_bytes))+ l = labels[i:] + [0] * (end - len(text_bytes))+ else:+ b = text_bytes[i:end]+ l = labels[i:end]+ chunks.append({"bytes": b, "labels": l})+ return chunks+++class ByteDataset(Dataset):+ """PyTorch dataset for XDA training. Each item is a fixed-size byte window."""++ def __init__(self, chunks: list[dict], tokenizer: ByteTokenizer) -> None:+ self.chunks = chunks+ self.tokenizer = tokenizer++ def __len__(self) -> int:+ return len(self.chunks)++ def __getitem__(self, idx: int) -> dict[str, torch.Tensor]:+ chunk = self.chunks[idx]+ raw_bytes = chunk["bytes"]+ raw_labels = chunk["labels"]++ # Tokenize: [CLS] + bytes + [SEP]+ input_ids = self.tokenizer.encode(raw_bytes)+ attention_mask = self.tokenizer.attention_mask(input_ids)++ # Labels: -100 for special tokens (ignored by CrossEntropyLoss)+ labels = [-100] + list(raw_labels) + [-100]++ return {+ "input_ids": torch.tensor(input_ids, dtype=torch.long),+ "attention_mask": torch.tensor(attention_mask, dtype=torch.long),+ "labels": torch.tensor(labels, dtype=torch.long),+ }+```++- [ ] **Step 4: Run test to verify it passes**++Run: `cd tools/xda && uv run pytest tests/test_dataset.py -v`+Expected: PASS (all 3 tests)++- [ ] **Step 5: Commit**++```bash+git add tools/xda/src/dataset.py tools/xda/tests/test_dataset.py+git commit -m "feat(xda): chunked byte dataset with sliding window for training"+```++---++### Task 4: XDA Model Definition++**Files:**+- Create: `tools/xda/src/model.py`+- Test: `tools/xda/tests/test_model.py`++- [ ] **Step 1: Write the model test**++```python+# tools/xda/tests/test_model.py+import torch+from src.model import XdaModel, XdaConfig+++def test_model_config_defaults():+ config = XdaConfig()+ assert config.vocab_size == 261+ assert config.hidden_size == 768+ assert config.num_hidden_layers == 12+ assert config.num_attention_heads == 12+ assert config.intermediate_size == 3072+ assert config.num_labels == 3+ assert config.max_position_embeddings == 514 # 512 + [CLS] + [SEP]+++def test_model_forward_shape():+ config = XdaConfig(+ num_hidden_layers=2, # smaller for test speed+ hidden_size=128,+ num_attention_heads=4,+ intermediate_size=256,+ )+ model = XdaModel(config)++ batch_size = 4+ seq_len = 514+ input_ids = torch.randint(0, 261, (batch_size, seq_len))+ attention_mask = torch.ones(batch_size, seq_len, dtype=torch.long)++ logits = model(input_ids, attention_mask)+ assert logits.shape == (batch_size, seq_len, 3)+++def test_model_parameter_count():+ """Sanity check that full-size model has ~87M params."""+ config = XdaConfig()+ model = XdaModel(config)+ total = sum(p.numel() for p in model.parameters())+ # Allow some tolerance — should be in the 80-95M range+ assert 80_000_000 < total < 100_000_000, f"Param count {total} outside expected range"+```++- [ ] **Step 2: Run test to verify it fails**++Run: `cd tools/xda && uv run pytest tests/test_model.py -v`+Expected: FAIL (import error)++- [ ] **Step 3: Implement XDA model**++```python+# tools/xda/src/model.py+"""XDA model: BERT encoder + 2-layer MLP classification head."""++from dataclasses import dataclass++import torch+from torch import nn+from transformers import BertConfig, BertModel+++@dataclass+class XdaConfig:+ vocab_size: int = 261 # 256 bytes + 5 special tokens+ hidden_size: int = 768+ num_hidden_layers: int = 12+ num_attention_heads: int = 12+ intermediate_size: int = 3072+ max_position_embeddings: int = 514 # 512 byte window + [CLS] + [SEP]+ num_labels: int = 3 # non-function, function_start, function_body+ classifier_hidden: int = 256 # MLP hidden dimension+ hidden_dropout_prob: float = 0.1+ attention_probs_dropout_prob: float = 0.1++ def to_bert_config(self) -> BertConfig:+ return BertConfig(+ vocab_size=self.vocab_size,+ hidden_size=self.hidden_size,+ num_hidden_layers=self.num_hidden_layers,+ num_attention_heads=self.num_attention_heads,+ intermediate_size=self.intermediate_size,+ max_position_embeddings=self.max_position_embeddings,+ hidden_dropout_prob=self.hidden_dropout_prob,+ attention_probs_dropout_prob=self.attention_probs_dropout_prob,+ )+++class XdaModel(nn.Module):+ """XDA: BERT encoder with a 2-layer MLP token classification head."""++ def __init__(self, config: XdaConfig) -> None:+ super().__init__()+ self.config = config+ self.encoder = BertModel(config.to_bert_config(), add_pooling_layer=False)+ self.classifier = nn.Sequential(+ nn.Linear(config.hidden_size, config.classifier_hidden),+ nn.GELU(),+ nn.Dropout(config.hidden_dropout_prob),+ nn.Linear(config.classifier_hidden, config.num_labels),+ )++ def forward(+ self,+ input_ids: torch.Tensor,+ attention_mask: torch.Tensor,+ ) -> torch.Tensor:+ """Returns logits of shape (batch, seq_len, num_labels)."""+ outputs = self.encoder(input_ids=input_ids, attention_mask=attention_mask)+ return self.classifier(outputs.last_hidden_state)+```++- [ ] **Step 4: Run tests to verify they pass**++Run: `cd tools/xda && uv run pytest tests/test_model.py -v`+Expected: PASS (all 3 tests). The param count test confirms architecture is correct.++- [ ] **Step 5: Commit**++```bash+git add tools/xda/src/model.py tools/xda/tests/test_model.py+git commit -m "feat(xda): BERT-based XDA model with 2-layer MLP classification head"+```++---++### Task 5: Corpus Build Scripts++**Files:**+- Create: `tools/xda/scripts/fetch_debian_packages.sh`+- Create: `tools/xda/scripts/build_corpus.sh`+- Create: `tools/xda/scripts/Dockerfile.corpus`++- [ ] **Step 1: Create Dockerfile for Debian arm64 corpus**++```dockerfile+# tools/xda/scripts/Dockerfile.corpus+# Build with: docker build --platform linux/arm64 -t xda-corpus -f scripts/Dockerfile.corpus .+# Run with: docker run --platform linux/arm64 -v $(pwd)/corpus:/out xda-corpus++FROM --platform=linux/arm64 debian:bookworm++RUN apt-get update && apt-get install -y --no-install-recommends \+ # Real-world C/C++ binaries+ coreutils binutils findutils diffutils grep gawk sed \+ tar gzip bzip2 xz-utils \+ curl wget \+ openssh-client openssh-server \+ openssl \+ sqlite3 \+ nginx-core \+ redis-server \+ lua5.4 \+ vim-tiny \+ tmux \+ git \+ # Debug symbols+ && echo "deb http://deb.debian.org/debian-debug bookworm-debug main" >> /etc/apt/sources.list \+ && apt-get update \+ && apt-get install -y --no-install-recommends \+ coreutils-dbgsym binutils-dbgsym findutils-dbgsym \+ tar-dbgsym grep-dbgsym gawk-dbgsym sed-dbgsym \+ curl-dbgsym openssh-client-dbgsym \+ libssl3-dbgsym libsqlite3-0-dbgsym \+ redis-server-dbgsym lua5.4-dbgsym \+ && rm -rf /var/lib/apt/lists/*++COPY scripts/fetch_debian_packages.sh /extract.sh+RUN chmod +x /extract.sh++CMD ["/extract.sh"]+```++- [ ] **Step 2: Create extraction script**++```bash+#!/usr/bin/env bash+# tools/xda/scripts/fetch_debian_packages.sh+# Pairs stripped binaries with their debug info from /usr/lib/debug+# Output: /out/pairs/<binary_name>/binary and /out/pairs/<binary_name>/debug++set -euo pipefail++OUT_DIR="/out/pairs"+mkdir -p "$OUT_DIR"++count=0++# Find all ELF binaries in standard paths+for bin in /usr/bin/* /usr/sbin/* /usr/lib/*/lib*.so*; do+ [ -f "$bin" ] || continue++ # Check if it's actually ELF+ file_type=$(file -b "$bin" 2>/dev/null) || continue+ echo "$file_type" | grep -q "ELF" || continue++ # Look for corresponding debug info+ # Debian stores debug files in /usr/lib/debug/<original-path>.debug+ debug_path="/usr/lib/debug${bin}.debug"+ if [ ! -f "$debug_path" ]; then+ # Also check build-id based paths+ build_id=$(readelf -n "$bin" 2>/dev/null | grep "Build ID" | awk '{print $3}') || continue+ if [ -n "$build_id" ]; then+ prefix="${build_id:0:2}"+ suffix="${build_id:2}"+ debug_path="/usr/lib/debug/.build-id/${prefix}/${suffix}.debug"+ fi+ fi++ [ -f "$debug_path" ] || continue++ name=$(basename "$bin")+ pair_dir="${OUT_DIR}/${name}"+ mkdir -p "$pair_dir"+ cp "$bin" "${pair_dir}/binary"+ cp "$debug_path" "${pair_dir}/debug"++ count=$((count + 1))+done++echo "Extracted $count binary/debug pairs to $OUT_DIR"+```++- [ ] **Step 3: Create cross-compilation corpus builder**++```bash+#!/usr/bin/env bash+# tools/xda/scripts/build_corpus.sh+# Cross-compiles C projects at multiple optimization levels for a target architecture.+# Usage: ./build_corpus.sh <arch> <output_dir>+# Example: ./build_corpus.sh aarch64 ./corpus/aarch64++set -euo pipefail++ARCH="${1:?Usage: build_corpus.sh <arch> <output_dir>}"+OUT="${2:?Usage: build_corpus.sh <arch> <output_dir>}"++case "$ARCH" in+ x86_64) CC="gcc"; STRIP="strip" ;;+ aarch64) CC="aarch64-linux-gnu-gcc"; STRIP="aarch64-linux-gnu-strip" ;;+ arm) CC="arm-linux-gnueabihf-gcc"; STRIP="arm-linux-gnueabihf-strip" ;;+ mips) CC="mips-linux-gnu-gcc"; STRIP="mips-linux-gnu-strip" ;;+ riscv64) CC="riscv64-linux-gnu-gcc"; STRIP="riscv64-linux-gnu-strip" ;;+ *) echo "Unknown arch: $ARCH"; exit 1 ;;+esac++# Verify compiler exists+command -v "$CC" >/dev/null 2>&1 || { echo "$CC not found. Install cross-compiler."; exit 1; }++mkdir -p "$OUT"++# Source files to compile — create a set of diverse test programs+SOURCES_DIR=$(mktemp -d)+trap 'rm -rf "$SOURCES_DIR"' EXIT++# Generate diverse source files+cat > "$SOURCES_DIR/algorithms.c" << 'CSRC'+#include <stdlib.h>+#include <string.h>++void bubble_sort(int *arr, int n) {+ for (int i = 0; i < n - 1; i++)+ for (int j = 0; j < n - i - 1; j++)+ if (arr[j] > arr[j + 1]) { int t = arr[j]; arr[j] = arr[j + 1]; arr[j + 1] = t; }+}+int binary_search(int *arr, int n, int target) {+ int lo = 0, hi = n - 1;+ while (lo <= hi) { int mid = (lo + hi) / 2; if (arr[mid] == target) return mid; if (arr[mid] < target) lo = mid + 1; else hi = mid - 1; }+ return -1;+}+void *mempool_alloc(size_t size) { static char pool[4096]; static size_t offset = 0; if (offset + size > 4096) return NULL; void *p = &pool[offset]; offset += size; return p; }+struct node { int val; struct node *next; };+struct node *list_insert(struct node *head, int val) { struct node *n = malloc(sizeof(*n)); n->val = val; n->next = head; return n; }+int list_sum(struct node *head) { int s = 0; while (head) { s += head->val; head = head->next; } return s; }+unsigned crc32_byte(unsigned crc, unsigned char b) { crc ^= b; for (int i = 0; i < 8; i++) crc = (crc >> 1) ^ (0xEDB88320 & -(crc & 1)); return crc; }+int main() { int arr[] = {5,3,1,4,2}; bubble_sort(arr, 5); return binary_search(arr, 5, 3); }+CSRC++cat > "$SOURCES_DIR/string_ops.c" << 'CSRC'+#include <string.h>+#include <ctype.h>+int str_count_char(const char *s, char c) { int n = 0; while (*s) { if (*s == c) n++; s++; } return n; }+void str_reverse(char *s) { int len = strlen(s); for (int i = 0; i < len / 2; i++) { char t = s[i]; s[i] = s[len - 1 - i]; s[len - 1 - i] = t; } }+void str_to_upper(char *s) { while (*s) { *s = toupper(*s); s++; } }+int str_is_palindrome(const char *s) { int l = 0, r = strlen(s) - 1; while (l < r) { if (s[l++] != s[r--]) return 0; } return 1; }+char *str_find_substr(const char *haystack, const char *needle) { int nlen = strlen(needle); if (!nlen) return (char*)haystack; for (; *haystack; haystack++) { if (!strncmp(haystack, needle, nlen)) return (char*)haystack; } return 0; }+int main() { char buf[] = "hello"; str_reverse(buf); return str_is_palindrome(buf); }+CSRC++cat > "$SOURCES_DIR/state_machine.c" << 'CSRC'+enum state { IDLE, RUNNING, PAUSED, ERROR, DONE };+enum event { START, PAUSE, RESUME, FAIL, FINISH };+enum state transition(enum state s, enum event e) {+ switch (s) {+ case IDLE: return e == START ? RUNNING : s;+ case RUNNING: switch (e) { case PAUSE: return PAUSED; case FAIL: return ERROR; case FINISH: return DONE; default: return s; }+ case PAUSED: return e == RESUME ? RUNNING : (e == FAIL ? ERROR : s);+ case ERROR: return s;+ case DONE: return s;+ }+ return s;+}+int run_machine(enum event *events, int n) { enum state s = IDLE; for (int i = 0; i < n; i++) s = transition(s, events[i]); return s; }+int main() { enum event seq[] = {START, PAUSE, RESUME, FINISH}; return run_machine(seq, 4); }+CSRC++# Compile each source at each optimization level+for src in "$SOURCES_DIR"/*.c; do+ name=$(basename "$src" .c)+ for opt in O0 O1 O2 O3 Os; do+ out_debug="${OUT}/${name}_${opt}_debug"+ out_stripped="${OUT}/${name}_${opt}_stripped"++ "$CC" -g "-${opt}" -o "$out_debug" "$src" 2>/dev/null || continue+ cp "$out_debug" "$out_stripped"+ "$STRIP" "$out_stripped"++ echo "Built: ${name}_${opt}"+ done+done++echo "Corpus built in $OUT"+```++- [ ] **Step 4: Test the corpus builder locally**++Run: `cd tools/xda && chmod +x scripts/build_corpus.sh && ./scripts/build_corpus.sh x86_64 ./corpus/x86_64`+Expected: `algorithms_O0`, `algorithms_O1`, ..., `state_machine_Os` (debug + stripped pairs) created in `corpus/x86_64/`.++- [ ] **Step 5: Commit**++```bash+git add tools/xda/scripts/+git commit -m "feat(xda): corpus build scripts for Debian packages and cross-compilation"+```++---++### Task 6: Full Dataset Builder (Corpus → Training Data)++**Files:**+- Create: `tools/xda/src/build_dataset.py`++This ties together extraction + chunking to produce a single training-ready dataset from a corpus directory.++- [ ] **Step 1: Implement the dataset builder**++```python+# tools/xda/src/build_dataset.py+"""Build a training dataset from a corpus of binary/debug pairs."""++import json+import random+from pathlib import Path++from tqdm import tqdm++from .extract_labels import extract_function_boundaries, generate_byte_labels+from .dataset import chunk_labeled_bytes+++def build_from_pairs_dir(+ pairs_dir: Path,+ output_path: Path,+ window_size: int = 512,+ stride: int = 256,+ val_split: float = 0.1,+) -> dict:+ """Process all binary/debug pairs into chunked training data.++ Expects pairs_dir to contain subdirectories, each with 'binary' and 'debug' files.+ Saves train.json and val.json to output_path.++ Returns stats dict.+ """+ all_chunks: list[dict] = []+ stats = {"binaries": 0, "functions": 0, "chunks": 0, "errors": []}++ for pair_dir in tqdm(sorted(pairs_dir.iterdir()), desc="Processing binaries"):+ if not pair_dir.is_dir():+ continue++ debug_file = pair_dir / "debug"+ binary_file = pair_dir / "binary"++ # Support both layouts: paired (binary + debug) and single (debug binary with DWARF)+ if debug_file.exists():+ label_source = str(debug_file)+ byte_source = str(binary_file) if binary_file.exists() else str(debug_file)+ else:+ # Single file with embedded DWARF (e.g., from build_corpus.sh debug binaries)+ candidates = [f for f in pair_dir.iterdir() if f.is_file() and "debug" in f.name]+ if not candidates:+ continue+ label_source = str(candidates[0])+ byte_source = label_source++ try:+ boundaries = extract_function_boundaries(label_source)+ if not boundaries:+ continue++ text_bytes, labels = generate_byte_labels(byte_source, boundaries)+ if not text_bytes:+ continue++ chunks = chunk_labeled_bytes(text_bytes, labels, window_size, stride)+ all_chunks.extend(chunks)++ stats["binaries"] += 1+ stats["functions"] += len(boundaries)+ stats["chunks"] += len(chunks)+ except Exception as e:+ stats["errors"].append({"file": str(pair_dir), "error": str(e)})++ # Shuffle and split+ random.shuffle(all_chunks)+ split_idx = int(len(all_chunks) * (1 - val_split))+ train_chunks = all_chunks[:split_idx]+ val_chunks = all_chunks[split_idx:]++ # Serialize — store bytes as list[int] for JSON compatibility+ output_path.mkdir(parents=True, exist_ok=True)++ def serialize_chunks(chunks: list[dict], path: Path) -> None:+ serialized = []+ for c in chunks:+ serialized.append({+ "bytes": list(c["bytes"]),+ "labels": c["labels"],+ })+ path.write_text(json.dumps(serialized))++ serialize_chunks(train_chunks, output_path / "train.json")+ serialize_chunks(val_chunks, output_path / "val.json")++ stats["train_chunks"] = len(train_chunks)+ stats["val_chunks"] = len(val_chunks)+ (output_path / "stats.json").write_text(json.dumps(stats, indent=2))++ return stats+++def build_from_corpus_dir(+ corpus_dir: Path,+ output_path: Path,+ window_size: int = 512,+ stride: int = 256,+ val_split: float = 0.1,+) -> dict:+ """Process debug binaries from build_corpus.sh output.++ Expects corpus_dir to contain *_debug and *_stripped files.+ Uses debug files for labels, stripped files for byte content.+ """+ all_chunks: list[dict] = []+ stats = {"binaries": 0, "functions": 0, "chunks": 0, "errors": []}++ debug_files = sorted(corpus_dir.glob("*_debug"))++ for debug_file in tqdm(debug_files, desc="Processing corpus"):+ try:+ boundaries = extract_function_boundaries(str(debug_file))+ if not boundaries:+ continue++ # Use the debug binary itself for .text bytes+ # (stripped version has same .text content but no DWARF)+ text_bytes, labels = generate_byte_labels(str(debug_file), boundaries)+ if not text_bytes:+ continue++ chunks = chunk_labeled_bytes(text_bytes, labels, window_size, stride)+ all_chunks.extend(chunks)++ stats["binaries"] += 1+ stats["functions"] += len(boundaries)+ stats["chunks"] += len(chunks)+ except Exception as e:+ stats["errors"].append({"file": str(debug_file), "error": str(e)})++ random.shuffle(all_chunks)+ split_idx = int(len(all_chunks) * (1 - val_split))+ train_chunks = all_chunks[:split_idx]+ val_chunks = all_chunks[split_idx:]++ output_path.mkdir(parents=True, exist_ok=True)++ def serialize_chunks(chunks: list[dict], path: Path) -> None:+ serialized = [{"bytes": list(c["bytes"]), "labels": c["labels"]} for c in chunks]+ path.write_text(json.dumps(serialized))++ serialize_chunks(train_chunks, output_path / "train.json")+ serialize_chunks(val_chunks, output_path / "val.json")++ stats["train_chunks"] = len(train_chunks)+ stats["val_chunks"] = len(val_chunks)+ (output_path / "stats.json").write_text(json.dumps(stats, indent=2))++ return stats+```++- [ ] **Step 2: Test end-to-end on x86_64 corpus**++Run: `cd tools/xda && uv run python -c "from src.build_dataset import build_from_corpus_dir; from pathlib import Path; stats = build_from_corpus_dir(Path('./corpus/x86_64'), Path('./data/x86_64')); print(stats)"`+Expected: `train.json` and `val.json` created in `data/x86_64/`, stats printed showing chunk/function counts.++- [ ] **Step 3: Commit**++```bash+git add tools/xda/src/build_dataset.py+git commit -m "feat(xda): end-to-end dataset builder from corpus to training data"+```++---++### Task 7: Pre-training Script (Masked Byte Modeling)++**Files:**+- Create: `tools/xda/src/pretrain.py`+- Create: `tools/xda/configs/pretrain.yaml`++This is the optional step for new architectures. For x86_64 you'd use the CUMLSec checkpoint. For AArch64/ARM/MIPS, you pre-train on unlabeled bytes first.++- [ ] **Step 1: Create pre-training config**++```yaml+# tools/xda/configs/pretrain.yaml+model:+ hidden_size: 768+ num_hidden_layers: 12+ num_attention_heads: 12+ intermediate_size: 3072++training:+ epochs: 10+ batch_size: 64+ learning_rate: 1.0e-4+ warmup_ratio: 0.1+ weight_decay: 0.01+ mask_probability: 0.20 # 20% of bytes masked+ mask_token_ratio: 0.50 # 50% of masked → [MASK], 50% → random byte+ max_length: 514 # 512 + [CLS] + [SEP]+ fp16: true++data:+ window_size: 512+ stride: 256++output:+ save_every_n_epochs: 2+ checkpoint_dir: checkpoints/pretrain+```++- [ ] **Step 2: Implement pre-training script**++```python+# tools/xda/src/pretrain.py+"""Masked byte modeling pre-training for XDA on a target architecture."""++import argparse+import json+import random+from pathlib import Path++import torch+import yaml+from torch import nn+from torch.utils.data import DataLoader, Dataset+from tqdm import tqdm++from .model import XdaConfig, XdaModel+from .tokenizer import ByteTokenizer+++class MaskedByteDataset(Dataset):+ """Dataset that applies masking on-the-fly for pre-training."""++ def __init__(+ self,+ chunks: list[dict],+ tokenizer: ByteTokenizer,+ mask_prob: float = 0.2,+ mask_token_ratio: float = 0.5,+ ) -> None:+ self.chunks = chunks+ self.tokenizer = tokenizer+ self.mask_prob = mask_prob+ self.mask_token_ratio = mask_token_ratio++ def __len__(self) -> int:+ return len(self.chunks)++ def __getitem__(self, idx: int) -> dict[str, torch.Tensor]:+ raw_bytes = bytes(self.chunks[idx]["bytes"])+ input_ids = self.tokenizer.encode(raw_bytes)+ labels = list(input_ids) # original tokens are labels++ # Mask byte positions (skip [CLS] at 0 and [SEP] at -1)+ for i in range(1, len(input_ids) - 1):+ if random.random() < self.mask_prob:+ if random.random() < self.mask_token_ratio:+ input_ids[i] = self.tokenizer.mask_id+ else:+ input_ids[i] = random.randint(0, 255)+ else:+ labels[i] = -100 # don't compute loss on unmasked++ labels[0] = -100 # [CLS]+ labels[-1] = -100 # [SEP]++ return {+ "input_ids": torch.tensor(input_ids, dtype=torch.long),+ "attention_mask": torch.ones(len(input_ids), dtype=torch.long),+ "labels": torch.tensor(labels, dtype=torch.long),+ }+++def pretrain(config_path: str, data_path: str) -> None:+ with open(config_path) as f:+ cfg = yaml.safe_load(f)++ device = torch.device("cuda" if torch.cuda.is_available() else "mps" if torch.backends.mps.is_available() else "cpu")+ print(f"Using device: {device}")++ # Load data+ train_chunks = json.loads(Path(data_path, "train.json").read_text())+ print(f"Loaded {len(train_chunks)} training chunks")++ tokenizer = ByteTokenizer()+ dataset = MaskedByteDataset(+ train_chunks,+ tokenizer,+ mask_prob=cfg["training"]["mask_probability"],+ mask_token_ratio=cfg["training"]["mask_token_ratio"],+ )+ loader = DataLoader(+ dataset,+ batch_size=cfg["training"]["batch_size"],+ shuffle=True,+ num_workers=4,+ pin_memory=True,+ )++ # Build model — for pre-training, the "classification" head predicts masked bytes+ # So num_labels = vocab_size (261), not 3+ model_cfg = XdaConfig(+ hidden_size=cfg["model"]["hidden_size"],+ num_hidden_layers=cfg["model"]["num_hidden_layers"],+ num_attention_heads=cfg["model"]["num_attention_heads"],+ intermediate_size=cfg["model"]["intermediate_size"],+ num_labels=tokenizer.vocab_size, # predict byte identity+ )+ model = XdaModel(model_cfg).to(device)++ optimizer = torch.optim.AdamW(+ model.parameters(),+ lr=cfg["training"]["learning_rate"],+ weight_decay=cfg["training"]["weight_decay"],+ )++ total_steps = len(loader) * cfg["training"]["epochs"]+ warmup_steps = int(total_steps * cfg["training"]["warmup_ratio"])+ scheduler = torch.optim.lr_scheduler.LinearLR(+ optimizer, start_factor=0.01, end_factor=1.0, total_iters=warmup_steps,+ )++ loss_fn = nn.CrossEntropyLoss(ignore_index=-100)+ scaler = torch.amp.GradScaler(enabled=cfg["training"]["fp16"] and device.type == "cuda")++ checkpoint_dir = Path(cfg["output"]["checkpoint_dir"])+ checkpoint_dir.mkdir(parents=True, exist_ok=True)++ for epoch in range(cfg["training"]["epochs"]):+ model.train()+ total_loss = 0.0+ steps = 0++ pbar = tqdm(loader, desc=f"Epoch {epoch + 1}/{cfg['training']['epochs']}")+ for batch in pbar:+ input_ids = batch["input_ids"].to(device)+ attention_mask = batch["attention_mask"].to(device)+ labels = batch["labels"].to(device)++ with torch.amp.autocast(device.type, enabled=cfg["training"]["fp16"]):+ logits = model(input_ids, attention_mask)+ loss = loss_fn(logits.view(-1, model_cfg.num_labels), labels.view(-1))++ scaler.scale(loss).backward()+ scaler.step(optimizer)+ scaler.update()+ optimizer.zero_grad()+ scheduler.step()++ total_loss += loss.item()+ steps += 1+ pbar.set_postfix(loss=f"{total_loss / steps:.4f}")++ avg_loss = total_loss / steps+ print(f"Epoch {epoch + 1}: avg_loss={avg_loss:.4f}")++ if (epoch + 1) % cfg["output"]["save_every_n_epochs"] == 0:+ torch.save(model.state_dict(), checkpoint_dir / f"pretrain_epoch{epoch + 1}.pt")++ torch.save(model.state_dict(), checkpoint_dir / "pretrain_final.pt")+ print(f"Pre-training complete. Final checkpoint: {checkpoint_dir / 'pretrain_final.pt'}")+++if __name__ == "__main__":+ parser = argparse.ArgumentParser()+ parser.add_argument("--config", required=True)+ parser.add_argument("--data", required=True)+ args = parser.parse_args()+ pretrain(args.config, args.data)+```++- [ ] **Step 3: Commit**++```bash+git add tools/xda/src/pretrain.py tools/xda/configs/pretrain.yaml+git commit -m "feat(xda): masked byte modeling pre-training script"+```++---++### Task 8: Fine-tuning Script++**Files:**+- Create: `tools/xda/src/finetune.py`+- Create: `tools/xda/configs/finetune.yaml`++- [ ] **Step 1: Create fine-tuning config**++```yaml+# tools/xda/configs/finetune.yaml+model:+ hidden_size: 768+ num_hidden_layers: 12+ num_attention_heads: 12+ intermediate_size: 3072++training:+ epochs: 5+ batch_size: 64+ learning_rate: 2.0e-5 # lower LR for fine-tuning+ warmup_ratio: 0.1+ weight_decay: 0.01+ fp16: true++data:+ window_size: 512+ stride: 256++output:+ checkpoint_dir: checkpoints/finetune+ save_best: true++# Optional: path to pre-trained checkpoint to initialize from+pretrained_checkpoint: null # e.g., "checkpoints/pretrain/pretrain_final.pt"+```++- [ ] **Step 2: Implement fine-tuning script**++```python+# tools/xda/src/finetune.py+"""Fine-tune XDA for function boundary detection (3-class token classification)."""++import argparse+import json+from pathlib import Path++import torch+import yaml+from torch import nn+from torch.utils.data import DataLoader+from tqdm import tqdm++from .dataset import ByteDataset, chunk_labeled_bytes+from .model import XdaConfig, XdaModel+from .tokenizer import ByteTokenizer+++def evaluate(model: XdaModel, loader: DataLoader, device: torch.device) -> dict:+ """Compute per-class precision, recall, F1 on validation set."""+ model.eval()+ # Per-class counts: 0=non-func, 1=func_start, 2=func_body+ tp = [0, 0, 0]+ fp = [0, 0, 0]+ fn = [0, 0, 0]++ with torch.no_grad():+ for batch in loader:+ input_ids = batch["input_ids"].to(device)+ attention_mask = batch["attention_mask"].to(device)+ labels = batch["labels"].to(device)++ logits = model(input_ids, attention_mask)+ preds = logits.argmax(dim=-1)++ mask = labels != -100+ preds = preds[mask]+ labels = labels[mask]++ for cls in range(3):+ tp[cls] += ((preds == cls) & (labels == cls)).sum().item()+ fp[cls] += ((preds == cls) & (labels != cls)).sum().item()+ fn[cls] += ((preds != cls) & (labels == cls)).sum().item()++ results = {}+ for cls, name in enumerate(["non_function", "function_start", "function_body"]):+ precision = tp[cls] / (tp[cls] + fp[cls]) if tp[cls] + fp[cls] > 0 else 0+ recall = tp[cls] / (tp[cls] + fn[cls]) if tp[cls] + fn[cls] > 0 else 0+ f1 = 2 * precision * recall / (precision + recall) if precision + recall > 0 else 0+ results[name] = {"precision": precision, "recall": recall, "f1": f1}++ return results+++def finetune(config_path: str, data_path: str) -> None:+ with open(config_path) as f:+ cfg = yaml.safe_load(f)++ device = torch.device("cuda" if torch.cuda.is_available() else "mps" if torch.backends.mps.is_available() else "cpu")+ print(f"Using device: {device}")++ # Load chunked data+ train_chunks = json.loads(Path(data_path, "train.json").read_text())+ val_chunks = json.loads(Path(data_path, "val.json").read_text())+ print(f"Train: {len(train_chunks)} chunks, Val: {len(val_chunks)} chunks")++ tokenizer = ByteTokenizer()+ train_ds = ByteDataset(train_chunks, tokenizer)+ val_ds = ByteDataset(val_chunks, tokenizer)++ train_loader = DataLoader(train_ds, batch_size=cfg["training"]["batch_size"], shuffle=True, num_workers=4, pin_memory=True)+ val_loader = DataLoader(val_ds, batch_size=cfg["training"]["batch_size"], num_workers=4)++ # Build model (3-class: non-function, function_start, function_body)+ model_cfg = XdaConfig(+ hidden_size=cfg["model"]["hidden_size"],+ num_hidden_layers=cfg["model"]["num_hidden_layers"],+ num_attention_heads=cfg["model"]["num_attention_heads"],+ intermediate_size=cfg["model"]["intermediate_size"],+ num_labels=3,+ )+ model = XdaModel(model_cfg)++ # Load pre-trained encoder weights if available+ pretrained_path = cfg.get("pretrained_checkpoint")+ if pretrained_path and Path(pretrained_path).exists():+ print(f"Loading pre-trained weights from {pretrained_path}")+ state = torch.load(pretrained_path, map_location="cpu", weights_only=True)+ # Load only encoder weights (skip classifier head — different num_labels)+ encoder_state = {k: v for k, v in state.items() if k.startswith("encoder.")}+ model.load_state_dict(encoder_state, strict=False)+ print(f"Loaded {len(encoder_state)} encoder weight tensors")++ model = model.to(device)++ optimizer = torch.optim.AdamW(+ model.parameters(),+ lr=cfg["training"]["learning_rate"],+ weight_decay=cfg["training"]["weight_decay"],+ )+ total_steps = len(train_loader) * cfg["training"]["epochs"]+ warmup_steps = int(total_steps * cfg["training"]["warmup_ratio"])+ scheduler = torch.optim.lr_scheduler.LinearLR(optimizer, start_factor=0.01, end_factor=1.0, total_iters=warmup_steps)++ loss_fn = nn.CrossEntropyLoss(ignore_index=-100)+ scaler = torch.amp.GradScaler(enabled=cfg["training"]["fp16"] and device.type == "cuda")++ checkpoint_dir = Path(cfg["output"]["checkpoint_dir"])+ checkpoint_dir.mkdir(parents=True, exist_ok=True)++ best_f1 = 0.0++ for epoch in range(cfg["training"]["epochs"]):+ # Train+ model.train()+ total_loss = 0.0+ steps = 0++ pbar = tqdm(train_loader, desc=f"Epoch {epoch + 1}/{cfg['training']['epochs']}")+ for batch in pbar:+ input_ids = batch["input_ids"].to(device)+ attention_mask = batch["attention_mask"].to(device)+ labels = batch["labels"].to(device)++ with torch.amp.autocast(device.type, enabled=cfg["training"]["fp16"]):+ logits = model(input_ids, attention_mask)+ loss = loss_fn(logits.view(-1, 3), labels.view(-1))++ scaler.scale(loss).backward()+ scaler.step(optimizer)+ scaler.update()+ optimizer.zero_grad()+ scheduler.step()++ total_loss += loss.item()+ steps += 1+ pbar.set_postfix(loss=f"{total_loss / steps:.4f}")++ # Evaluate+ results = evaluate(model, val_loader, device)+ fs_f1 = results["function_start"]["f1"]++ print(f"Epoch {epoch + 1}: loss={total_loss / steps:.4f}")+ for name, metrics in results.items():+ print(f" {name}: P={metrics['precision']:.4f} R={metrics['recall']:.4f} F1={metrics['f1']:.4f}")++ # Save best by function_start F1 (the metric that matters most)+ if cfg["output"]["save_best"] and fs_f1 > best_f1:+ best_f1 = fs_f1+ torch.save(model.state_dict(), checkpoint_dir / "best.pt")+ print(f" New best function_start F1: {fs_f1:.4f}")++ torch.save(model.state_dict(), checkpoint_dir / f"epoch{epoch + 1}.pt")++ # Save final config alongside weights (needed for Rust loading)+ config_out = {+ "vocab_size": model_cfg.vocab_size,+ "hidden_size": model_cfg.hidden_size,+ "num_hidden_layers": model_cfg.num_hidden_layers,+ "num_attention_heads": model_cfg.num_attention_heads,+ "intermediate_size": model_cfg.intermediate_size,+ "max_position_embeddings": model_cfg.max_position_embeddings,+ "num_labels": model_cfg.num_labels,+ "classifier_hidden": model_cfg.classifier_hidden,+ }+ (checkpoint_dir / "config.json").write_text(json.dumps(config_out, indent=2))++ print(f"Fine-tuning complete. Best F1: {best_f1:.4f}")+++if __name__ == "__main__":+ parser = argparse.ArgumentParser()+ parser.add_argument("--config", required=True)+ parser.add_argument("--data", required=True)+ args = parser.parse_args()+ finetune(args.config, args.data)+```++- [ ] **Step 3: Commit**++```bash+git add tools/xda/src/finetune.py tools/xda/configs/finetune.yaml+git commit -m "feat(xda): function boundary fine-tuning with per-class eval"+```++---++### Task 9: Export to Safetensors++**Files:**+- Create: `tools/xda/src/export.py`++This is the bridge between Python training and Rust inference. Produces the exact artifacts that candle loads.++- [ ] **Step 1: Implement export script**++```python+# tools/xda/src/export.py+"""Export trained XDA checkpoint to safetensors format for Rust/candle loading."""++import argparse+import json+import shutil+from pathlib import Path++import torch+from safetensors.torch import save_file++from .model import XdaConfig, XdaModel+++def export(checkpoint_path: str, config_path: str, output_dir: str, arch: str) -> None:+ """Export a PyTorch checkpoint + config to safetensors + config.json.++ Output structure:+ output_dir/+ xda_{arch}.safetensors # model weights+ xda_{arch}_config.json # model config (for candle)+ """+ out = Path(output_dir)+ out.mkdir(parents=True, exist_ok=True)++ # Load config+ with open(config_path) as f:+ config_dict = json.load(f)+ config = XdaConfig(**config_dict)++ # Load model and checkpoint+ model = XdaModel(config)+ state = torch.load(checkpoint_path, map_location="cpu", weights_only=True)+ model.load_state_dict(state)++ # Flatten state dict — candle needs simple key → tensor mapping+ # Rename keys to match what the Rust model expects+ flat_state = {}+ for key, tensor in model.state_dict().items():+ # Convert to float32 (candle safetensors loading expects f32)+ flat_state[key] = tensor.float()++ # Save as safetensors+ weights_path = out / f"xda_{arch}.safetensors"+ save_file(flat_state, str(weights_path))++ # Save config JSON (Rust side reads this to construct the model)+ config_out = out / f"xda_{arch}_config.json"+ config_out.write_text(json.dumps(config_dict, indent=2))++ # Print summary+ total_params = sum(t.numel() for t in flat_state.values())+ file_size_mb = weights_path.stat().st_size / (1024 * 1024)+ print(f"Exported {arch} model:")+ print(f" Weights: {weights_path} ({file_size_mb:.1f} MB)")+ print(f" Config: {config_out}")+ print(f" Parameters: {total_params:,}")+++if __name__ == "__main__":+ parser = argparse.ArgumentParser()+ parser.add_argument("--checkpoint", required=True, help="Path to .pt checkpoint")+ parser.add_argument("--config", required=True, help="Path to config.json from training")+ parser.add_argument("--output", required=True, help="Output directory")+ parser.add_argument("--arch", required=True, help="Architecture name (x86_64, aarch64, etc.)")+ args = parser.parse_args()+ export(args.checkpoint, args.config, args.output, args.arch)+```++- [ ] **Step 2: Commit**++```bash+git add tools/xda/src/export.py+git commit -m "feat(xda): export trained models to safetensors for Rust/candle loading"+```++---++## Part B: Rust Candle Inference++### Task 10: Cargo Workspace & kong-types Crate++**Files:**+- Create: `kong-rs/Cargo.toml` (workspace root)+- Create: `kong-rs/crates/kong-types/Cargo.toml`+- Create: `kong-rs/crates/kong-types/src/lib.rs`+- Create: `kong-rs/crates/kong-types/src/binary.rs`+- Create: `kong-rs/crates/kong-types/src/function.rs`++- [ ] **Step 1: Create workspace Cargo.toml**++```toml+# kong-rs/Cargo.toml+[workspace]+members = [+ "crates/kong-types",+ "crates/kong-ml",+]+resolver = "2"++[workspace.package]+version = "1.0.0"+edition = "2021"+license = "MIT"++[workspace.dependencies]+serde = { version = "1", features = ["derive"] }+serde_json = "1"+```++- [ ] **Step 2: Create kong-types crate**++```toml+# kong-rs/crates/kong-types/Cargo.toml+[package]+name = "kong-types"+version.workspace = true+edition.workspace = true++[dependencies]+serde.workspace = true+```++```rust+// kong-rs/crates/kong-types/src/lib.rs+pub mod binary;+pub mod function;+```++```rust+// kong-rs/crates/kong-types/src/binary.rs+use serde::{Deserialize, Serialize};++#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)]+#[serde(rename_all = "snake_case")]+pub enum Arch {+ X86_64,+ X86,+ Aarch64,+ Arm,+ Mips,+ Mips64,+ Riscv64,+ PowerPc,+}++impl Arch {+ /// Model artifact name for this architecture.+ pub fn model_name(&self) -> &'static str {+ match self {+ Arch::X86_64 => "x86_64",+ Arch::X86 => "x86",+ Arch::Aarch64 => "aarch64",+ Arch::Arm => "arm",+ Arch::Mips => "mips",+ Arch::Mips64 => "mips64",+ Arch::Riscv64 => "riscv64",+ Arch::PowerPc => "powerpc",+ }+ }+}++impl std::fmt::Display for Arch {+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {+ f.write_str(self.model_name())+ }+}+```++```rust+// kong-rs/crates/kong-types/src/function.rs+use serde::{Deserialize, Serialize};++/// A confirmed function boundary from either heuristic or ML detection.+#[derive(Debug, Clone, Serialize, Deserialize)]+pub struct FunctionBoundary {+ pub start: u64,+ pub end: u64,+ pub confidence: f32,+ pub source: DetectionSource,+}++/// How a function boundary was detected.+#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]+#[serde(rename_all = "snake_case")]+pub enum DetectionSource {+ Symbol, // from symbol table+ Heuristic, // from prologue/call-target analysis+ Ml, // from XDA model+ ExceptionInfo, // from .eh_frame / .pdata+}++/// A region of .text that heuristics couldn't confidently classify.+#[derive(Debug, Clone)]+pub struct AmbiguousRegion {+ pub start: u64,+ pub bytes: Vec<u8>,+ pub reason: AmbiguityReason,+}++#[derive(Debug, Clone, Copy, PartialEq, Eq)]+pub enum AmbiguityReason {+ NoPrologue, // reachable but no recognizable prologue+ IndirectTarget, // suspected indirect call/jump target+ GapRegion, // unreached .text gap between known functions+ OptimizedCode, // code region with non-standard patterns+}+```++- [ ] **Step 3: Verify it compiles**++Run: `cd kong-rs && cargo check`+Expected: Compiles with no errors.++- [ ] **Step 4: Commit**++```bash+git add kong-rs/+git commit -m "feat(kong-rs): workspace scaffolding with kong-types crate"+```++---++### Task 11: kong-ml Crate — Byte Tokenizer & Config++**Files:**+- Create: `kong-rs/crates/kong-ml/Cargo.toml`+- Create: `kong-rs/crates/kong-ml/src/lib.rs`+- Create: `kong-rs/crates/kong-ml/src/tokenizer.rs`+- Create: `kong-rs/crates/kong-ml/src/config.rs`++- [ ] **Step 1: Create kong-ml Cargo.toml**++```toml+# kong-rs/crates/kong-ml/Cargo.toml+[package]+name = "kong-ml"+version.workspace = true+edition.workspace = true++[dependencies]+kong-types = { path = "../kong-types" }+candle-core = "0.8"+candle-nn = "0.8"+candle-transformers = "0.8"+safetensors = "0.4"+serde.workspace = true+serde_json.workspace = true+dirs = "5"+```++- [ ] **Step 2: Implement byte tokenizer in Rust**++```rust+// kong-rs/crates/kong-ml/src/tokenizer.rs++/// Byte-level tokenizer for XDA. Maps raw bytes 0-255 to token IDs 0-255.+/// Special tokens occupy IDs 256-260.+pub struct ByteTokenizer {+ pub pad_id: u32,+ pub cls_id: u32,+ pub sep_id: u32,+ pub mask_id: u32,+ pub unk_id: u32,+ pub vocab_size: u32,+}++impl Default for ByteTokenizer {+ fn default() -> Self {+ Self {+ pad_id: 256,+ cls_id: 257,+ sep_id: 258,+ mask_id: 259,+ unk_id: 260,+ vocab_size: 261,+ }+ }+}++impl ByteTokenizer {+ pub fn new() -> Self {+ Self::default()+ }++ /// Encode raw bytes as token IDs: [CLS] + byte_values + [SEP], padded to max_length.+ pub fn encode(&self, bytes: &[u8], max_length: Option<usize>) -> Vec<u32> {+ let mut ids = Vec::with_capacity(bytes.len() + 2);+ ids.push(self.cls_id);+ ids.extend(bytes.iter().map(|&b| b as u32));+ ids.push(self.sep_id);++ if let Some(max_len) = max_length {+ ids.truncate(max_len);+ ids.resize(max_len, self.pad_id);+ }++ ids+ }++ /// Generate attention mask: 1 for real tokens, 0 for padding.+ pub fn attention_mask(&self, token_ids: &[u32]) -> Vec<u32> {+ token_ids+ .iter()+ .map(|&id| if id == self.pad_id { 0 } else { 1 })+ .collect()+ }+}++#[cfg(test)]+mod tests {+ use super::*;++ #[test]+ fn test_special_token_ids() {+ let tok = ByteTokenizer::new();+ assert_eq!(tok.pad_id, 256);+ assert_eq!(tok.cls_id, 257);+ assert_eq!(tok.sep_id, 258);+ assert_eq!(tok.vocab_size, 261);+ }++ #[test]+ fn test_encode_bytes() {+ let tok = ByteTokenizer::new();+ let raw = [0x55u8, 0x48, 0x89, 0xe5];+ let ids = tok.encode(&raw, None);+ assert_eq!(ids, vec![257, 0x55, 0x48, 0x89, 0xe5, 258]);+ }++ #[test]+ fn test_encode_with_padding() {+ let tok = ByteTokenizer::new();+ let raw = [0x55u8, 0x48];+ let ids = tok.encode(&raw, Some(8));+ assert_eq!(ids, vec![257, 0x55, 0x48, 258, 256, 256, 256, 256]);+ }++ #[test]+ fn test_attention_mask() {+ let tok = ByteTokenizer::new();+ let raw = [0x55u8, 0x48];+ let ids = tok.encode(&raw, Some(8));+ let mask = tok.attention_mask(&ids);+ assert_eq!(mask, vec![1, 1, 1, 1, 0, 0, 0, 0]);+ }+}+```++- [ ] **Step 3: Implement config loading**++```rust+// kong-rs/crates/kong-ml/src/config.rs+use serde::{Deserialize, Serialize};++/// XDA model configuration. Loaded from config.json alongside safetensors weights.+#[derive(Debug, Clone, Serialize, Deserialize)]+pub struct XdaConfig {+ pub vocab_size: usize,+ pub hidden_size: usize,+ pub num_hidden_layers: usize,+ pub num_attention_heads: usize,+ pub intermediate_size: usize,+ pub max_position_embeddings: usize,+ pub num_labels: usize,+ pub classifier_hidden: usize,+}++impl Default for XdaConfig {+ fn default() -> Self {+ Self {+ vocab_size: 261,+ hidden_size: 768,+ num_hidden_layers: 12,+ num_attention_heads: 12,+ intermediate_size: 3072,+ max_position_embeddings: 514,+ num_labels: 3,+ classifier_hidden: 256,+ }+ }+}++impl XdaConfig {+ pub fn from_json(path: &std::path::Path) -> Result<Self, Box<dyn std::error::Error>> {+ let content = std::fs::read_to_string(path)?;+ let config: Self = serde_json::from_str(&content)?;+ Ok(config)+ }++ /// Convert to candle-transformers BertConfig for the encoder.+ pub fn to_bert_config(&self) -> candle_transformers::models::bert::Config {+ candle_transformers::models::bert::Config {+ vocab_size: self.vocab_size,+ hidden_size: self.hidden_size,+ num_hidden_layers: self.num_hidden_layers,+ num_attention_heads: self.num_attention_heads,+ intermediate_size: self.intermediate_size,+ max_position_embeddings: self.max_position_embeddings,+ hidden_dropout_prob: 0.0, // no dropout during inference+ ..Default::default()+ }+ }+}+```++- [ ] **Step 4: Wire up lib.rs**++```rust+// kong-rs/crates/kong-ml/src/lib.rs+pub mod config;+pub mod tokenizer;+pub mod model;+pub mod detector;+pub mod registry;+```++- [ ] **Step 5: Verify it compiles**++Run: `cd kong-rs && cargo check`+Expected: Compiles (model.rs, detector.rs, registry.rs will be empty stubs initially — create them with just `// TODO` if needed to satisfy mod declarations, or gate them behind a cfg flag). For now, comment out the three unimplemented mods in lib.rs:++```rust+// kong-rs/crates/kong-ml/src/lib.rs+pub mod config;+pub mod tokenizer;+// pub mod model; // Task 12+// pub mod detector; // Task 13+// pub mod registry; // Task 14+```++Run: `cd kong-rs && cargo test -p kong-ml`+Expected: All 4 tokenizer tests pass.++- [ ] **Step 6: Commit**++```bash+git add kong-rs/crates/kong-ml/+git commit -m "feat(kong-ml): byte tokenizer and config loading for candle inference"+```++---++### Task 12: XDA Model in Candle++**Files:**+- Create: `kong-rs/crates/kong-ml/src/model.rs`++This is the core Rust implementation — BERT encoder (from candle-transformers) + 2-layer MLP classification head.++- [ ] **Step 1: Implement XdaModel**++```rust+// kong-rs/crates/kong-ml/src/model.rs+use candle_core::{DType, Device, Result, Tensor};+use candle_nn::{linear, Linear, Module, VarBuilder};+use candle_transformers::models::bert::{BertModel, Config as BertConfig};++use crate::config::XdaConfig;++/// 2-layer MLP classification head.+struct ClassifierHead {+ fc1: Linear,+ fc2: Linear,+}++impl ClassifierHead {+ fn load(vb: VarBuilder, hidden_size: usize, classifier_hidden: usize, num_labels: usize) -> Result<Self> {+ let fc1 = linear(hidden_size, classifier_hidden, vb.pp("classifier.0"))?;+ let fc2 = linear(classifier_hidden, num_labels, vb.pp("classifier.3"))?;+ Ok(Self { fc1, fc2 })+ }++ fn forward(&self, hidden_states: &Tensor) -> Result<Tensor> {+ let x = self.fc1.forward(hidden_states)?;+ let x = x.gelu_erf()?;+ self.fc2.forward(&x)+ }+}++/// XDA model: BERT encoder + classification head for function boundary detection.+pub struct XdaModel {+ encoder: BertModel,+ classifier: ClassifierHead,+}++impl XdaModel {+ /// Load model from safetensors weights.+ pub fn load(config: &XdaConfig, vb: VarBuilder) -> Result<Self> {+ let bert_config = config.to_bert_config();+ let encoder = BertModel::load(vb.pp("encoder"), &bert_config)?;+ let classifier = ClassifierHead::load(+ vb.clone(),+ config.hidden_size,+ config.classifier_hidden,+ config.num_labels,+ )?;+ Ok(Self { encoder, classifier })+ }++ /// Run inference. Returns logits of shape (batch, seq_len, num_labels).+ pub fn forward(&self, input_ids: &Tensor, attention_mask: &Tensor) -> Result<Tensor> {+ // BertModel::forward requires token_type_ids (mandatory) and attention_mask (optional).+ // For XDA we use single-segment input, so token_type_ids is all zeros.+ let token_type_ids = input_ids.zeros_like()?;+ let hidden = self.encoder.forward(input_ids, &token_type_ids, Some(attention_mask))?;+ self.classifier.forward(&hidden)+ }+}+```++**Important note on weight key mapping:** The Python model saves weights with keys like `encoder.embeddings.word_embeddings.weight`, `classifier.0.weight`, `classifier.3.weight`. The Rust side uses `VarBuilder::pp("encoder")` and `VarBuilder::pp("classifier.0")` etc. to match these keys. If the keys don't align, you'll get a "tensor not found" error at load time — debug by printing the keys in the safetensors file:++```rust+// Debug helper — use temporarily if weight loading fails+let tensors = safetensors::SafeTensors::deserialize(&std::fs::read(path)?)?;+for name in tensors.names() {+ println!("{name}");+}+```++- [ ] **Step 2: Uncomment model in lib.rs**++```rust+// kong-rs/crates/kong-ml/src/lib.rs+pub mod config;+pub mod tokenizer;+pub mod model;+// pub mod detector; // Task 13+// pub mod registry; // Task 14+```++- [ ] **Step 3: Verify it compiles**++Run: `cd kong-rs && cargo check -p kong-ml`+Expected: Compiles. No runtime test yet — we need actual weights for that.++- [ ] **Step 4: Commit**++```bash+git add kong-rs/crates/kong-ml/src/model.rs kong-rs/crates/kong-ml/src/lib.rs+git commit -m "feat(kong-ml): XDA model implementation in candle (BERT + MLP head)"+```++---++### Task 13: XDA Detector (Sliding Window + Batched Inference)++**Files:**+- Create: `kong-rs/crates/kong-ml/src/detector.rs`++- [ ] **Step 1: Implement XdaDetector**++```rust+// kong-rs/crates/kong-ml/src/detector.rs+use candle_core::{DType, Device, IndexOp, Result, Tensor};+use kong_types::function::{AmbiguousRegion, DetectionSource, FunctionBoundary};++use crate::config::XdaConfig;+use crate::model::XdaModel;+use crate::tokenizer::ByteTokenizer;++pub struct XdaDetector {+ model: XdaModel,+ tokenizer: ByteTokenizer,+ device: Device,+ confidence_threshold: f32,+ window_size: usize,+ stride: usize,+}++/// Per-byte prediction from a single window.+#[derive(Debug, Clone)]+struct BytePrediction {+ address: u64,+ label: u8, // 0=non-func, 1=func_start, 2=func_body+ confidence: f32,+}++impl XdaDetector {+ pub fn new(+ model: XdaModel,+ device: Device,+ confidence_threshold: f32,+ ) -> Self {+ Self {+ model,+ tokenizer: ByteTokenizer::new(),+ device,+ confidence_threshold,+ window_size: 512,+ stride: 256,+ }+ }++ /// Predict function boundaries in ambiguous regions.+ pub fn predict(&self, regions: &[AmbiguousRegion]) -> Result<Vec<FunctionBoundary>> {+ let mut boundaries = Vec::new();++ for region in regions {+ let region_predictions = self.predict_region(region)?;+ boundaries.extend(region_predictions);+ }++ Ok(boundaries)+ }++ fn predict_region(&self, region: &AmbiguousRegion) -> Result<Vec<FunctionBoundary>> {+ let bytes = ®ion.bytes;+ if bytes.is_empty() {+ return Ok(Vec::new());+ }++ // Slide windows over the region+ let windows = self.create_windows(bytes);+ if windows.is_empty() {+ return Ok(Vec::new());+ }++ // Batch inference+ let predictions = self.run_batch(&windows)?;++ // Merge overlapping window predictions — for overlapping bytes,+ // average the confidence scores+ let merged = self.merge_predictions(&predictions, bytes.len(), region.start);++ // Extract function boundaries from merged predictions+ Ok(self.extract_boundaries(&merged))+ }++ /// Returns (window_bytes, actual_byte_count) pairs. actual_byte_count tracks+ /// how many bytes are real vs padding, since binary code can legitimately contain 0x00.+ fn create_windows(&self, bytes: &[u8]) -> Vec<(Vec<u8>, usize)> {+ let mut windows = Vec::new();+ let mut offset = 0;++ while offset < bytes.len() {+ let end = (offset + self.window_size).min(bytes.len());+ let actual_len = end - offset;+ let mut window = bytes[offset..end].to_vec();++ // Pad if necessary+ if window.len() < self.window_size {+ window.resize(self.window_size, 0);+ }++ windows.push((window, actual_len));+ offset += self.stride;++ if end == bytes.len() {+ break;+ }+ }++ windows+ }++ fn run_batch(&self, windows: &[(Vec<u8>, usize)]) -> Result<Vec<Vec<(u8, f32)>>> {+ let seq_len = self.window_size + 2; // +2 for [CLS] and [SEP]+ let batch_size = windows.len();++ // Tokenize all windows+ let mut all_ids = Vec::with_capacity(batch_size * seq_len);+ let mut all_mask = Vec::with_capacity(batch_size * seq_len);++ for (window, _actual_len) in windows {+ let ids = self.tokenizer.encode(window, Some(seq_len));+ let mask = self.tokenizer.attention_mask(&ids);+ all_ids.extend(ids.iter().map(|&x| x as i64));+ all_mask.extend(mask.iter().map(|&x| x as i64));+ }++ let input_ids = Tensor::from_vec(all_ids, (batch_size, seq_len), &self.device)?;+ let attention_mask = Tensor::from_vec(all_mask, (batch_size, seq_len), &self.device)?;++ // Forward pass+ let logits = self.model.forward(&input_ids, &attention_mask)?;++ // Softmax over last dimension to get probabilities+ let probs = candle_nn::ops::softmax(&logits, candle_core::D::Minus1)?;++ // Extract per-byte predictions (skip [CLS] and [SEP])+ let mut batch_predictions = Vec::with_capacity(batch_size);++ for i in 0..batch_size {+ let window_probs = probs.i((i, 1..seq_len - 1))?; // skip [CLS] at 0, [SEP] at end+ let window_probs = window_probs.to_vec2::<f32>()?;++ // Use the tracked actual_len instead of inferring from byte content —+ // binary code can legitimately contain 0x00 bytes.+ let actual_len = windows[i].1;+ let mut preds = Vec::with_capacity(actual_len);++ for j in 0..actual_len {+ let p = &window_probs[j];+ let (label, confidence) = p+ .iter()+ .enumerate()+ .max_by(|a, b| a.1.partial_cmp(b.1).unwrap())+ .map(|(idx, &conf)| (idx as u8, conf))+ .unwrap_or((0, 0.0));+ preds.push((label, confidence));+ }++ batch_predictions.push(preds);+ }++ Ok(batch_predictions)+ }++ fn merge_predictions(+ &self,+ batch_predictions: &[Vec<(u8, f32)>],+ total_bytes: usize,+ base_address: u64,+ ) -> Vec<BytePrediction> {+ // For overlapping windows, accumulate (label_votes, confidence_sum, count)+ let mut votes: Vec<([f32; 3], u32)> = vec![([0.0; 3], 0); total_bytes];++ for (window_idx, preds) in batch_predictions.iter().enumerate() {+ let offset = window_idx * self.stride;+ for (byte_idx, &(label, confidence)) in preds.iter().enumerate() {+ let global_idx = offset + byte_idx;+ if global_idx < total_bytes {+ votes[global_idx].0[label as usize] += confidence;+ votes[global_idx].1 += 1;+ }+ }+ }++ votes+ .into_iter()+ .enumerate()+ .map(|(idx, (scores, count))| {+ let count = count.max(1) as f32;+ let avg_scores = [+ scores[0] / count,+ scores[1] / count,+ scores[2] / count,+ ];+ let (label, confidence) = avg_scores+ .iter()+ .enumerate()+ .max_by(|a, b| a.1.partial_cmp(b.1).unwrap())+ .map(|(l, &c)| (l as u8, c))+ .unwrap_or((0, 0.0));++ BytePrediction {+ address: base_address + idx as u64,+ label,+ confidence,+ }+ })+ .collect()+ }++ fn extract_boundaries(&self, predictions: &[BytePrediction]) -> Vec<FunctionBoundary> {+ let mut boundaries = Vec::new();+ let mut func_start: Option<&BytePrediction> = None;++ for pred in predictions {+ match pred.label {+ 1 if pred.confidence >= self.confidence_threshold => {+ // Close previous function if open+ if let Some(start) = func_start.take() {+ boundaries.push(FunctionBoundary {+ start: start.address,+ end: pred.address,+ confidence: start.confidence,+ source: DetectionSource::Ml,+ });+ }+ func_start = Some(pred);+ }+ 0 if func_start.is_some() => {+ // Non-function byte after function body — close the function+ let start = func_start.take().unwrap();+ boundaries.push(FunctionBoundary {+ start: start.address,+ end: pred.address,+ confidence: start.confidence,+ source: DetectionSource::Ml,+ });+ }+ _ => {}+ }+ }++ // Close trailing function+ if let Some(start) = func_start {+ if let Some(last) = predictions.last() {+ boundaries.push(FunctionBoundary {+ start: start.address,+ end: last.address + 1,+ confidence: start.confidence,+ source: DetectionSource::Ml,+ });+ }+ }++ boundaries+ }+}+```++- [ ] **Step 2: Uncomment detector in lib.rs**++- [ ] **Step 3: Verify it compiles**++Run: `cd kong-rs && cargo check -p kong-ml`+Expected: Compiles.++- [ ] **Step 4: Commit**++```bash+git add kong-rs/crates/kong-ml/src/detector.rs kong-rs/crates/kong-ml/src/lib.rs+git commit -m "feat(kong-ml): XDA detector with sliding window and batch inference"+```++---++### Task 14: Model Registry (Download, Cache, Load Per-Arch)++**Files:**+- Create: `kong-rs/crates/kong-ml/src/registry.rs`++- [ ] **Step 1: Implement ModelRegistry**++```rust+// kong-rs/crates/kong-ml/src/registry.rs+use std::path::{Path, PathBuf};++use candle_core::{DType, Device};+use candle_nn::VarBuilder;+use kong_types::binary::Arch;++use crate::config::XdaConfig;+use crate::detector::XdaDetector;+use crate::model::XdaModel;++pub struct ModelRegistry {+ cache_dir: PathBuf,+ confidence_threshold: f32,+}++impl ModelRegistry {+ pub fn new(confidence_threshold: f32) -> Self {+ let cache_dir = dirs::data_local_dir()+ .unwrap_or_else(|| PathBuf::from("."))+ .join("kong")+ .join("models");++ Self {+ cache_dir,+ confidence_threshold,+ }+ }++ pub fn with_cache_dir(cache_dir: PathBuf, confidence_threshold: f32) -> Self {+ Self {+ cache_dir,+ confidence_threshold,+ }+ }++ /// Check if a model is available locally for the given architecture.+ pub fn has_model(&self, arch: Arch) -> bool {+ self.weights_path(arch).exists() && self.config_path(arch).exists()+ }++ /// List all locally available architectures.+ pub fn available_architectures(&self) -> Vec<Arch> {+ let all = [+ Arch::X86_64,+ Arch::X86,+ Arch::Aarch64,+ Arch::Arm,+ Arch::Mips,+ Arch::Mips64,+ Arch::Riscv64,+ Arch::PowerPc,+ ];+ all.into_iter().filter(|a| self.has_model(*a)).collect()+ }++ /// Load a detector for the given architecture.+ /// Returns None if no model is available (graceful degradation).+ pub fn load_detector(&self, arch: Arch, device: &Device) -> Result<Option<XdaDetector>, Box<dyn std::error::Error>> {+ if !self.has_model(arch) {+ return Ok(None);+ }++ let config = XdaConfig::from_json(&self.config_path(arch))?;+ let vb = unsafe {+ VarBuilder::from_mmaped_safetensors(+ &[self.weights_path(arch)],+ DType::F32,+ device,+ )?+ };+ let model = XdaModel::load(&config, vb)?;+ let detector = XdaDetector::new(model, device.clone(), self.confidence_threshold);++ Ok(Some(detector))+ }++ /// Install a model from a local safetensors + config.json pair.+ pub fn install(&self, arch: Arch, weights_src: &Path, config_src: &Path) -> Result<(), Box<dyn std::error::Error>> {+ std::fs::create_dir_all(&self.cache_dir)?;+ std::fs::copy(weights_src, self.weights_path(arch))?;+ std::fs::copy(config_src, self.config_path(arch))?;+ Ok(())+ }++ fn weights_path(&self, arch: Arch) -> PathBuf {+ self.cache_dir.join(format!("xda_{}.safetensors", arch.model_name()))+ }++ fn config_path(&self, arch: Arch) -> PathBuf {+ self.cache_dir.join(format!("xda_{}_config.json", arch.model_name()))+ }+}+```++- [ ] **Step 2: Uncomment registry in lib.rs, wire up all modules**++```rust+// kong-rs/crates/kong-ml/src/lib.rs+pub mod config;+pub mod tokenizer;+pub mod model;+pub mod detector;+pub mod registry;+```++- [ ] **Step 3: Verify it compiles**++Run: `cd kong-rs && cargo check -p kong-ml`+Expected: Compiles.++- [ ] **Step 4: Commit**++```bash+git add kong-rs/crates/kong-ml/src/registry.rs kong-rs/crates/kong-ml/src/lib.rs+git commit -m "feat(kong-ml): model registry with per-arch cache and lazy loading"+```++---++### Task 15: End-to-End Integration Test++**Files:**+- Create: `kong-rs/tests/xda_integration.rs`++This test validates the full pipeline: tokenize → model → predict. It requires a trained model to exist, so it's gated behind an env var.++- [ ] **Step 1: Write the integration test**++```rust+// kong-rs/tests/xda_integration.rs++/// End-to-end integration test. Requires a trained model.+/// Run with: XDA_MODEL_DIR=/path/to/models cargo test --test xda_integration+#[test]+fn test_full_pipeline() {+ let model_dir = match std::env::var("XDA_MODEL_DIR") {+ Ok(dir) => std::path::PathBuf::from(dir),+ Err(_) => {+ eprintln!("Skipping integration test: XDA_MODEL_DIR not set");+ return;+ }+ };++ let device = candle_core::Device::Cpu;+ let registry = kong_ml::registry::ModelRegistry::with_cache_dir(model_dir, 0.85);++ // Check x86_64 model exists+ let arch = kong_types::binary::Arch::X86_64;+ assert!(registry.has_model(arch), "x86_64 model not found in model dir");++ // Load detector+ let detector = registry+ .load_detector(arch, &device)+ .expect("Failed to load model")+ .expect("Model should exist");++ // Create a fake ambiguous region with x86_64 function prologue bytes+ let prologue = vec![+ 0x55, // push rbp+ 0x48, 0x89, 0xe5, // mov rbp, rsp+ 0x48, 0x83, 0xec, 0x10, // sub rsp, 0x10+ 0x89, 0x7d, 0xfc, // mov [rbp-4], edi+ 0x8b, 0x45, 0xfc, // mov eax, [rbp-4]+ 0x83, 0xc0, 0x01, // add eax, 1+ 0xc9, // leave+ 0xc3, // ret+ // padding / gap+ 0xcc, 0xcc, 0xcc, 0xcc, 0xcc, 0xcc, 0xcc, 0xcc, 0xcc, 0xcc, 0xcc, 0xcc,+ // second function+ 0x55, // push rbp+ 0x48, 0x89, 0xe5, // mov rbp, rsp+ 0x89, 0x7d, 0xfc, // mov [rbp-4], edi+ 0x8b, 0x45, 0xfc, // mov eax, [rbp-4]+ 0x0f, 0xaf, 0xc0, // imul eax, eax+ 0x5d, // pop rbp+ 0xc3, // ret+ ];++ let region = kong_types::function::AmbiguousRegion {+ start: 0x401000,+ bytes: prologue,+ reason: kong_types::function::AmbiguityReason::GapRegion,+ };++ let boundaries = detector.predict(&[region]).expect("Inference failed");++ // With a well-trained model, we'd expect it to find function starts+ // at offsets corresponding to the two push rbp instructions.+ // For now, just verify inference runs without error and produces output.+ println!("Found {} function boundaries:", boundaries.len());+ for b in &boundaries {+ println!(+ " 0x{:x} - 0x{:x} (confidence: {:.2}, source: {:?})",+ b.start, b.end, b.confidence, b.source+ );+ }+}++/// Smoke test that runs without a model — verifies graceful degradation.+#[test]+fn test_no_model_graceful() {+ let tmp = std::env::temp_dir().join("kong_test_empty_models");+ let _ = std::fs::create_dir_all(&tmp);++ let registry = kong_ml::registry::ModelRegistry::with_cache_dir(tmp, 0.85);+ let device = candle_core::Device::Cpu;++ let detector = registry+ .load_detector(kong_types::binary::Arch::X86_64, &device)+ .expect("Should not error");++ assert!(detector.is_none(), "Should return None when no model exists");+}+```++- [ ] **Step 2: Verify the no-model test passes**++Run: `cd kong-rs && cargo test --test xda_integration test_no_model_graceful`+Expected: PASS.++- [ ] **Step 3: Commit**++```bash+git add kong-rs/tests/xda_integration.rs+git commit -m "feat(kong-ml): end-to-end integration test for XDA inference pipeline"+```++---++## Part C: Training Your First Model (x86_64)++### Task 16: Train x86_64 Model End-to-End++This task uses the scripts from Part A to actually train and export a model.++- [ ] **Step 1: Build x86_64 corpus**++```bash+cd tools/xda+chmod +x scripts/build_corpus.sh+./scripts/build_corpus.sh x86_64 ./corpus/x86_64+```++Expected: ~15 binaries (3 source files x 5 optimization levels) in `corpus/x86_64/`.++- [ ] **Step 2: Build training dataset**++```bash+cd tools/xda+uv run python -c "+from src.build_dataset import build_from_corpus_dir+from pathlib import Path+stats = build_from_corpus_dir(Path('./corpus/x86_64'), Path('./data/x86_64'))+print(f'Binaries: {stats[\"binaries\"]}')+print(f'Functions: {stats[\"functions\"]}')+print(f'Train chunks: {stats[\"train_chunks\"]}')+print(f'Val chunks: {stats[\"val_chunks\"]}')+"+```++Expected: `data/x86_64/train.json`, `data/x86_64/val.json`, `data/x86_64/stats.json` created. Small corpus — maybe 50-200 chunks. Good enough for a smoke test.++- [ ] **Step 3: Fine-tune (skip pre-training for x86_64 — use from scratch for now)**++```bash+cd tools/xda+uv run python -m src.finetune --config configs/finetune.yaml --data ./data/x86_64+```++Expected: Training runs for 5 epochs, prints per-class F1 scores. On this small corpus, results won't be great — that's fine, this validates the pipeline.++- [ ] **Step 4: Export to safetensors**++```bash+cd tools/xda+uv run python -m src.export \+ --checkpoint checkpoints/finetune/best.pt \+ --config checkpoints/finetune/config.json \+ --output ../../kong-rs/models/ \+ --arch x86_64+```++Expected: `kong-rs/models/xda_x86_64.safetensors` and `kong-rs/models/xda_x86_64_config.json` created.++- [ ] **Step 5: Run Rust integration test against trained model**++```bash+cd kong-rs+XDA_MODEL_DIR=./models cargo test --test xda_integration test_full_pipeline -- --nocapture+```++Expected: Test passes. Prints function boundaries found (may not be accurate with tiny training set — validates the pipeline works end-to-end).++- [ ] **Step 6: Commit**++```bash+git add kong-rs/models/.gitkeep+echo "*.safetensors" >> kong-rs/.gitignore+echo "*.pt" >> tools/xda/.gitignore+echo "corpus/" >> tools/xda/.gitignore+echo "data/" >> tools/xda/.gitignore+echo "checkpoints/" >> tools/xda/.gitignore+git add kong-rs/.gitignore tools/xda/.gitignore+git commit -m "feat: end-to-end XDA pipeline validated — train in Python, infer in Rust"+```++---++## Part D: Scale Up for Production Quality++### Task 17: Build Large Corpus with Debian Packages (AArch64)++This repeats the pattern for a new architecture.++- [ ] **Step 1: Build and run the Debian corpus Docker image**++```bash+cd tools/xda+docker build --platform linux/arm64 -t xda-corpus-aarch64 -f scripts/Dockerfile.corpus .+docker run --platform linux/arm64 -v $(pwd)/corpus/aarch64_debian:/out xda-corpus-aarch64+```++Expected: `corpus/aarch64_debian/pairs/` with 50-200+ binary/debug pairs.++- [ ] **Step 2: Also cross-compile custom corpus for AArch64**++```bash+# Requires: brew install aarch64-linux-gnu-gcc (or via apt on Linux)+cd tools/xda+./scripts/build_corpus.sh aarch64 ./corpus/aarch64_custom+```++- [ ] **Step 3: Build dataset from both sources**++```bash+cd tools/xda+uv run python -c "+from src.build_dataset import build_from_pairs_dir, build_from_corpus_dir+from pathlib import Path+import json++# Merge both corpus sources+all_chunks = []++# Debian packages+stats1 = build_from_pairs_dir(Path('./corpus/aarch64_debian/pairs'), Path('./data/aarch64_tmp1'))+t1 = json.loads(Path('./data/aarch64_tmp1/train.json').read_text())+v1 = json.loads(Path('./data/aarch64_tmp1/val.json').read_text())++# Cross-compiled+stats2 = build_from_corpus_dir(Path('./corpus/aarch64_custom'), Path('./data/aarch64_tmp2'))+t2 = json.loads(Path('./data/aarch64_tmp2/train.json').read_text())+v2 = json.loads(Path('./data/aarch64_tmp2/val.json').read_text())++# Merge+import random+train = t1 + t2; random.shuffle(train)+val = v1 + v2; random.shuffle(val)++out = Path('./data/aarch64'); out.mkdir(exist_ok=True)+(out / 'train.json').write_text(json.dumps(train))+(out / 'val.json').write_text(json.dumps(val))+print(f'Total train: {len(train)}, val: {len(val)}')+"+```++- [ ] **Step 4: Pre-train on AArch64 bytes (optional but recommended)**++```bash+cd tools/xda+uv run python -m src.pretrain --config configs/pretrain.yaml --data ./data/aarch64+```++Expected: Runs for 10 epochs. May take several hours depending on corpus size and hardware.++- [ ] **Step 5: Fine-tune on AArch64**++Edit `configs/finetune.yaml` to point `pretrained_checkpoint` at the pre-trained checkpoint:+```yaml+pretrained_checkpoint: "checkpoints/pretrain/pretrain_final.pt"+```++```bash+cd tools/xda+uv run python -m src.finetune --config configs/finetune.yaml --data ./data/aarch64+```++- [ ] **Step 6: Export AArch64 model**++```bash+cd tools/xda+uv run python -m src.export \+ --checkpoint checkpoints/finetune/best.pt \+ --config checkpoints/finetune/config.json \+ --output ../../kong-rs/models/ \+ --arch aarch64+```++- [ ] **Step 7: Verify in Rust**++```bash+cd kong-rs+XDA_MODEL_DIR=./models cargo test --test xda_integration -- --nocapture+```++- [ ] **Step 8: Commit**++```bash+git commit -m "feat(xda): trained and validated AArch64 function boundary model"+```++---++## Summary: What Each Task Produces++| Task | Deliverable |+|------|-------------|+| 1 | Python project + byte tokenizer |+| 2 | DWARF ground truth extractor |+| 3 | Chunked dataset pipeline |+| 4 | XDA model definition (Python) |+| 5 | Corpus build scripts (Debian + cross-compile) |+| 6 | End-to-end dataset builder |+| 7 | Pre-training script (masked byte modeling) |+| 8 | Fine-tuning script (3-class boundary detection) |+| 9 | Safetensors export for Rust |+| 10 | Cargo workspace + kong-types crate |+| 11 | Rust byte tokenizer + config loading |+| 12 | XDA model in candle (BERT + MLP) |+| 13 | Sliding window inference + batch prediction |+| 14 | Per-arch model registry with caching |+| 15 | End-to-end Rust integration test |+| 16 | First trained model (x86_64, small corpus) |+| 17 | Production AArch64 model (large corpus) |++**Tasks 1-9** (Python) and **Tasks 10-15** (Rust) are independent workstreams that can be parallelized. Task 16 bridges them. Task 17 is the repeatable pattern for new architectures.diff --git a/tools/xda/src/finetune.py b/tools/xda/src/finetune.pyindex fbffc53..3b7a539 100644--- a/tools/xda/src/finetune.py+++ b/tools/xda/src/finetune.py@@ -185,6 +185,9 @@ def finetune(config_path: str, data_path: str, arch: str = "unknown") -> None:torch.save(model.state_dict(), checkpoint_dir / "best.pt")print(f" New best function_start F1: {fs_f1:.4f}")+ prev = checkpoint_dir / f"epoch{epoch}.pt"+ if prev.exists():+ prev.unlink()torch.save(model.state_dict(), checkpoint_dir / f"epoch{epoch + 1}.pt")config_out = {