The world's first agentic reverse engineer.
rm plan
Amruth52878ed95f4e
diff --git a/tools/xda/plan.md b/tools/xda/plan.mddeleted file mode 100644index fadc1d6..0000000--- a/tools/xda/plan.md+++ /dev/null@@ -1,2748 +0,0 @@-Random-docs/superpowers/plans/2026-03-20-xda-training-and-candle-inference.md--# XDA Per-Architecture Model Training & Candle Inference Implementation Plan--> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.--**Goal:** Build a complete pipeline to train per-architecture XDA function boundary detection models and load them for inference in Rust via candle.--**Architecture:** Two independent workstreams — (A) a Python training pipeline that produces `.safetensors` model artifacts per ISA, and (B) a Rust candle-based inference module that loads those artifacts and runs function boundary prediction. The training pipeline is a standalone tool in `tools/xda/`. The Rust inference lives in the `kong-rs` workspace as part of `kong-binary`.--**Tech Stack:**-- Training: Python, PyTorch, HuggingFace Transformers, pyelftools-- Inference: Rust, candle (candle-core, candle-nn, candle-transformers), serde-- Dataset: Debian arm64 packages, cross-compiled C/C++ corpus-- Model format: safetensors (shared between Python export and Rust loading)-------## File Structure--### Python Training Pipeline (`tools/xda/`)--```-tools/xda/-├── pyproject.toml # uv project: torch, transformers, pyelftools, safetensors-├── README.md # How to train a new arch model-├── scripts/-│ ├── build_corpus.sh # Cross-compile C projects for a target arch-│ └── fetch_debian_packages.sh # Download Debian arm64 packages + dbgsym-├── src/-│ ├── __init__.py-│ ├── extract_labels.py # DWARF → function boundaries → per-byte labels-│ ├── dataset.py # ByteDataset: chunking, train/val split, DataLoader-│ ├── tokenizer.py # Byte tokenizer (0-255 + 5 special tokens)-│ ├── model.py # XDA model definition (BERT + 2-layer MLP head)-│ ├── pretrain.py # Masked byte modeling pre-training-│ ├── finetune.py # Function boundary fine-tuning-│ └── export.py # PyTorch checkpoint → safetensors + config.json-├── configs/-│ ├── pretrain.yaml # Pre-training hyperparameters-│ └── finetune.yaml # Fine-tuning hyperparameters-└── tests/- ├── test_extract_labels.py- ├── test_dataset.py- ├── test_tokenizer.py- └── test_model.py-```--### Rust Inference (`kong-rs/crates/kong-ml/`)--```-kong-rs/-├── Cargo.toml # Workspace root-├── crates/-│ ├── kong-types/-│ │ ├── Cargo.toml-│ │ └── src/-│ │ ├── lib.rs-│ │ ├── binary.rs # BinaryInfo, Arch enum-│ │ └── function.rs # FunctionBoundary, AmbiguousRegion-│ └── kong-ml/-│ ├── Cargo.toml # candle-core, candle-nn, candle-transformers, safetensors, serde, dirs-│ └── src/-│ ├── lib.rs # pub mod tokenizer, model, detector, registry-│ ├── tokenizer.rs # ByteTokenizer: bytes → token IDs-│ ├── config.rs # XdaConfig: loaded from config.json alongside weights-│ ├── model.rs # XdaModel: BERT encoder + classification head-│ ├── detector.rs # XdaDetector: sliding window, batching, thresholding-│ └── registry.rs # ModelRegistry: per-arch download, cache, load-├── tests/-│ └── xda_integration.rs # End-to-end: load model, run on test bytes, check predictions-└── models/- └── .gitkeep # Model artifacts go here (gitignored, downloaded at runtime)-```-------## Part A: Python Training Pipeline--### Task 1: Project Scaffolding & Byte Tokenizer--**Files:**-- Create: `tools/xda/pyproject.toml`-- Create: `tools/xda/src/__init__.py`-- Create: `tools/xda/src/tokenizer.py`-- Test: `tools/xda/tests/test_tokenizer.py`--- [ ] **Step 1: Create pyproject.toml**--```toml-[project]-name = "xda-training"-version = "0.1.0"-requires-python = ">=3.11"-dependencies = [- "torch>=2.0",- "transformers>=4.40",- "pyelftools>=0.30",- "safetensors>=0.4",- "pyyaml>=6.0",- "tqdm>=4.65",-]--[build-system]-requires = ["hatchling"]-build-backend = "hatchling.build"--[tool.hatch.build.targets.wheel]-packages = ["src"]-```--- [ ] **Step 2: Write the tokenizer test**--```python-# tools/xda/tests/test_tokenizer.py-from src.tokenizer import ByteTokenizer--def test_special_token_ids():- tok = ByteTokenizer()- assert tok.pad_id == 256- assert tok.cls_id == 257- assert tok.sep_id == 258- assert tok.mask_id == 259- assert tok.unk_id == 260- assert tok.vocab_size == 261--def test_encode_bytes():- tok = ByteTokenizer()- raw = bytes([0x55, 0x48, 0x89, 0xe5])- ids = tok.encode(raw)- # [CLS] + byte IDs + [SEP]- assert ids == [257, 0x55, 0x48, 0x89, 0xe5, 258]--def test_encode_with_padding():- tok = ByteTokenizer()- raw = bytes([0x55, 0x48])- ids = tok.encode(raw, max_length=8)- assert ids == [257, 0x55, 0x48, 258, 256, 256, 256, 256]- assert len(ids) == 8--def test_attention_mask():- tok = ByteTokenizer()- raw = bytes([0x55, 0x48])- ids = tok.encode(raw, max_length=8)- mask = tok.attention_mask(ids)- assert mask == [1, 1, 1, 1, 0, 0, 0, 0]-```--- [ ] **Step 3: Run test to verify it fails**--Run: `cd tools/xda && uv run pytest tests/test_tokenizer.py -v`-Expected: FAIL (module not found)--- [ ] **Step 4: Implement ByteTokenizer**--```python-# tools/xda/src/tokenizer.py-from dataclasses import dataclass---@dataclass(frozen=True)-class ByteTokenizer:- """Byte-level tokenizer for XDA. Maps raw bytes 0-255 to token IDs 0-255.- Special tokens occupy IDs 256-260."""-- pad_id: int = 256- cls_id: int = 257- sep_id: int = 258- mask_id: int = 259- unk_id: int = 260- vocab_size: int = 261-- def encode(self, raw_bytes: bytes, max_length: int | None = None) -> list[int]:- ids = [self.cls_id] + list(raw_bytes) + [self.sep_id]- if max_length is not None:- ids = ids[:max_length]- ids += [self.pad_id] * (max_length - len(ids))- return ids-- def attention_mask(self, token_ids: list[int]) -> list[int]:- return [0 if t == self.pad_id else 1 for t in token_ids]-```--- [ ] **Step 5: Run test to verify it passes**--Run: `cd tools/xda && uv run pytest tests/test_tokenizer.py -v`-Expected: PASS (all 4 tests)--- [ ] **Step 6: Commit**--```bash-git add tools/xda/pyproject.toml tools/xda/src/__init__.py tools/xda/src/tokenizer.py tools/xda/tests/test_tokenizer.py-git commit -m "feat(xda): scaffold training project, implement byte tokenizer"-```-------### Task 2: Ground Truth Extraction from DWARF--**Files:**-- Create: `tools/xda/src/extract_labels.py`-- Test: `tools/xda/tests/test_extract_labels.py`--- [ ] **Step 1: Write the extraction test**--This test compiles a tiny C program, extracts labels, verifies correctness.--```python-# tools/xda/tests/test_extract_labels.py-import subprocess-import tempfile-from pathlib import Path--from src.extract_labels import extract_function_boundaries, generate_byte_labels---def _compile_test_binary(src: str, out: Path, arch: str = "native") -> Path:- """Compile a C string to a binary with debug info."""- src_file = out / "test.c"- bin_file = out / "test.elf"- src_file.write_text(src)-- cc = "gcc" if arch == "native" else f"{arch}-linux-gnu-gcc"- subprocess.run(- [cc, "-g", "-O0", "-o", str(bin_file), str(src_file)],- check=True,- )- return bin_file---def test_extract_boundaries_from_simple_binary():- src = """- int add(int a, int b) { return a + b; }- int mul(int a, int b) { return a * b; }- int main() { return add(1, 2) + mul(3, 4); }- """- with tempfile.TemporaryDirectory() as tmp:- binary = _compile_test_binary(src, Path(tmp))- boundaries = extract_function_boundaries(str(binary))-- names = {b["name"] for b in boundaries}- assert "add" in names- assert "mul" in names- assert "main" in names-- for b in boundaries:- assert b["start"] < b["end"]- assert b["end"] - b["start"] > 0---def test_generate_byte_labels():- src = "int foo() { return 42; } int main() { return foo(); }"- with tempfile.TemporaryDirectory() as tmp:- binary = _compile_test_binary(src, Path(tmp))- boundaries = extract_function_boundaries(str(binary))- text_bytes, labels = generate_byte_labels(str(binary), boundaries)-- assert len(text_bytes) == len(labels)- assert len(text_bytes) > 0-- # At least some bytes should be labeled as function_start (1)- assert 1 in labels- # At least some bytes should be labeled as function_body (2)- assert 2 in labels- # At least some bytes should be non-function (0) unless .text is fully covered-```--- [ ] **Step 2: Run test to verify it fails**--Run: `cd tools/xda && uv run pytest tests/test_extract_labels.py -v`-Expected: FAIL (import error)--- [ ] **Step 3: Implement extract_labels.py**--```python-# tools/xda/src/extract_labels.py-"""Extract function boundaries from DWARF debug info and generate per-byte labels."""--from elftools.elf.elffile import ELFFile---def extract_function_boundaries(elf_path: str) -> list[dict]:- """Extract function start/end addresses from DWARF debug info.-- Returns list of {"name": str | None, "start": int, "end": int}.- """- functions: list[dict] = []- with open(elf_path, "rb") as f:- elf = ELFFile(f)- if not elf.has_dwarf_info():- return []-- dwarf = elf.get_dwarf_info()- for cu in dwarf.iter_CUs():- for die in cu.iter_DIEs():- if die.tag != "DW_TAG_subprogram":- continue- if "DW_AT_low_pc" not in die.attributes:- continue-- low_pc = die.attributes["DW_AT_low_pc"].value- high_pc_attr = die.attributes.get("DW_AT_high_pc")- if high_pc_attr is None:- continue-- if high_pc_attr.form in ("DW_FORM_addr",):- high_pc = high_pc_attr.value- else:- high_pc = low_pc + high_pc_attr.value-- if high_pc <= low_pc:- continue-- name_attr = die.attributes.get("DW_AT_name")- name = name_attr.value.decode() if name_attr else None- functions.append({"name": name, "start": low_pc, "end": high_pc})-- return functions---def generate_byte_labels(- elf_path: str,- boundaries: list[dict],-) -> tuple[bytes, list[int]]:- """Generate per-byte labels for the .text section.-- Labels: 0 = non-function, 1 = function_start, 2 = function_body.- Returns (text_section_bytes, labels).- """- with open(elf_path, "rb") as f:- elf = ELFFile(f)- text = elf.get_section_by_name(".text")- if text is None:- return b"", []-- text_start = text["sh_addr"]- text_bytes = text.data()-- labels = [0] * len(text_bytes)-- for func in boundaries:- start_off = func["start"] - text_start- end_off = func["end"] - text_start-- if start_off < 0 or start_off >= len(text_bytes):- continue- end_off = min(end_off, len(text_bytes))-- labels[start_off] = 1 # function_start- for i in range(start_off + 1, end_off):- labels[i] = 2 # function_body-- return bytes(text_bytes), labels-```--- [ ] **Step 4: Run test to verify it passes**--Run: `cd tools/xda && uv run pytest tests/test_extract_labels.py -v`-Expected: PASS (both tests). Requires `gcc` available on system.--- [ ] **Step 5: Commit**--```bash-git add tools/xda/src/extract_labels.py tools/xda/tests/test_extract_labels.py-git commit -m "feat(xda): DWARF-based ground truth extraction for function boundaries"-```-------### Task 3: Dataset Pipeline (Chunking + DataLoader)--**Files:**-- Create: `tools/xda/src/dataset.py`-- Test: `tools/xda/tests/test_dataset.py`--- [ ] **Step 1: Write the dataset test**--```python-# tools/xda/tests/test_dataset.py-import torch-from src.dataset import ByteDataset, chunk_labeled_bytes-from src.tokenizer import ByteTokenizer---def test_chunk_labeled_bytes():- text_bytes = bytes(range(256)) * 4 # 1024 bytes- labels = [0] * 1024- labels[0] = 1 # function start at byte 0- labels[512] = 1 # function start at byte 512-- chunks = chunk_labeled_bytes(text_bytes, labels, window_size=512, stride=256)-- assert len(chunks) > 0- for chunk in chunks:- assert len(chunk["bytes"]) == 512- assert len(chunk["labels"]) == 512---def test_byte_dataset_shapes():- text_bytes = bytes(range(256)) * 4- labels = [0] * 1024- labels[0] = 1-- chunks = chunk_labeled_bytes(text_bytes, labels, window_size=512, stride=256)- ds = ByteDataset(chunks, ByteTokenizer())-- item = ds[0]- assert item["input_ids"].shape == (514,) # 512 + [CLS] + [SEP]- assert item["attention_mask"].shape == (514,)- assert item["labels"].shape == (514,)- assert item["input_ids"][0].item() == 257 # [CLS]- assert item["input_ids"][-1].item() == 258 # [SEP]- # Labels for [CLS] and [SEP] should be -100 (ignored in loss)- assert item["labels"][0].item() == -100- assert item["labels"][-1].item() == -100---def test_byte_dataset_label_values():- text_bytes = bytes([0x55, 0x48, 0x89, 0xe5, 0xc3, 0x00, 0x00, 0x00])- labels = [1, 2, 2, 2, 2, 0, 0, 0]-- chunks = chunk_labeled_bytes(text_bytes, labels, window_size=8, stride=8)- ds = ByteDataset(chunks, ByteTokenizer())-- item = ds[0]- # Labels offset by 1 for [CLS] prefix- assert item["labels"][1].item() == 1 # function_start- assert item["labels"][2].item() == 2 # function_body- assert item["labels"][6].item() == 0 # non-function-```--- [ ] **Step 2: Run test to verify it fails**--Run: `cd tools/xda && uv run pytest tests/test_dataset.py -v`-Expected: FAIL (import error)--- [ ] **Step 3: Implement dataset.py**--```python-# tools/xda/src/dataset.py-"""Chunking and DataLoader for XDA training."""--import torch-from torch.utils.data import Dataset--from .tokenizer import ByteTokenizer---def chunk_labeled_bytes(- text_bytes: bytes,- labels: list[int],- window_size: int = 512,- stride: int = 256,-) -> list[dict]:- """Chunk labeled bytes into fixed-size windows with overlap."""- chunks = []- for i in range(0, max(1, len(text_bytes) - window_size + 1), stride):- end = i + window_size- if end > len(text_bytes):- # Pad the last chunk- b = text_bytes[i:] + bytes(end - len(text_bytes))- l = labels[i:] + [0] * (end - len(text_bytes))- else:- b = text_bytes[i:end]- l = labels[i:end]- chunks.append({"bytes": b, "labels": l})- return chunks---class ByteDataset(Dataset):- """PyTorch dataset for XDA training. Each item is a fixed-size byte window."""-- def __init__(self, chunks: list[dict], tokenizer: ByteTokenizer) -> None:- self.chunks = chunks- self.tokenizer = tokenizer-- def __len__(self) -> int:- return len(self.chunks)-- def __getitem__(self, idx: int) -> dict[str, torch.Tensor]:- chunk = self.chunks[idx]- raw_bytes = chunk["bytes"]- raw_labels = chunk["labels"]-- # Tokenize: [CLS] + bytes + [SEP]- input_ids = self.tokenizer.encode(raw_bytes)- attention_mask = self.tokenizer.attention_mask(input_ids)-- # Labels: -100 for special tokens (ignored by CrossEntropyLoss)- labels = [-100] + list(raw_labels) + [-100]-- return {- "input_ids": torch.tensor(input_ids, dtype=torch.long),- "attention_mask": torch.tensor(attention_mask, dtype=torch.long),- "labels": torch.tensor(labels, dtype=torch.long),- }-```--- [ ] **Step 4: Run test to verify it passes**--Run: `cd tools/xda && uv run pytest tests/test_dataset.py -v`-Expected: PASS (all 3 tests)--- [ ] **Step 5: Commit**--```bash-git add tools/xda/src/dataset.py tools/xda/tests/test_dataset.py-git commit -m "feat(xda): chunked byte dataset with sliding window for training"-```-------### Task 4: XDA Model Definition--**Files:**-- Create: `tools/xda/src/model.py`-- Test: `tools/xda/tests/test_model.py`--- [ ] **Step 1: Write the model test**--```python-# tools/xda/tests/test_model.py-import torch-from src.model import XdaModel, XdaConfig---def test_model_config_defaults():- config = XdaConfig()- assert config.vocab_size == 261- assert config.hidden_size == 768- assert config.num_hidden_layers == 12- assert config.num_attention_heads == 12- assert config.intermediate_size == 3072- assert config.num_labels == 3- assert config.max_position_embeddings == 514 # 512 + [CLS] + [SEP]---def test_model_forward_shape():- config = XdaConfig(- num_hidden_layers=2, # smaller for test speed- hidden_size=128,- num_attention_heads=4,- intermediate_size=256,- )- model = XdaModel(config)-- batch_size = 4- seq_len = 514- input_ids = torch.randint(0, 261, (batch_size, seq_len))- attention_mask = torch.ones(batch_size, seq_len, dtype=torch.long)-- logits = model(input_ids, attention_mask)- assert logits.shape == (batch_size, seq_len, 3)---def test_model_parameter_count():- """Sanity check that full-size model has ~87M params."""- config = XdaConfig()- model = XdaModel(config)- total = sum(p.numel() for p in model.parameters())- # Allow some tolerance — should be in the 80-95M range- assert 80_000_000 < total < 100_000_000, f"Param count {total} outside expected range"-```--- [ ] **Step 2: Run test to verify it fails**--Run: `cd tools/xda && uv run pytest tests/test_model.py -v`-Expected: FAIL (import error)--- [ ] **Step 3: Implement XDA model**--```python-# tools/xda/src/model.py-"""XDA model: BERT encoder + 2-layer MLP classification head."""--from dataclasses import dataclass--import torch-from torch import nn-from transformers import BertConfig, BertModel---@dataclass-class XdaConfig:- vocab_size: int = 261 # 256 bytes + 5 special tokens- hidden_size: int = 768- num_hidden_layers: int = 12- num_attention_heads: int = 12- intermediate_size: int = 3072- max_position_embeddings: int = 514 # 512 byte window + [CLS] + [SEP]- num_labels: int = 3 # non-function, function_start, function_body- classifier_hidden: int = 256 # MLP hidden dimension- hidden_dropout_prob: float = 0.1- attention_probs_dropout_prob: float = 0.1-- def to_bert_config(self) -> BertConfig:- return BertConfig(- vocab_size=self.vocab_size,- hidden_size=self.hidden_size,- num_hidden_layers=self.num_hidden_layers,- num_attention_heads=self.num_attention_heads,- intermediate_size=self.intermediate_size,- max_position_embeddings=self.max_position_embeddings,- hidden_dropout_prob=self.hidden_dropout_prob,- attention_probs_dropout_prob=self.attention_probs_dropout_prob,- )---class XdaModel(nn.Module):- """XDA: BERT encoder with a 2-layer MLP token classification head."""-- def __init__(self, config: XdaConfig) -> None:- super().__init__()- self.config = config- self.encoder = BertModel(config.to_bert_config(), add_pooling_layer=False)- self.classifier = nn.Sequential(- nn.Linear(config.hidden_size, config.classifier_hidden),- nn.GELU(),- nn.Dropout(config.hidden_dropout_prob),- nn.Linear(config.classifier_hidden, config.num_labels),- )-- def forward(- self,- input_ids: torch.Tensor,- attention_mask: torch.Tensor,- ) -> torch.Tensor:- """Returns logits of shape (batch, seq_len, num_labels)."""- outputs = self.encoder(input_ids=input_ids, attention_mask=attention_mask)- return self.classifier(outputs.last_hidden_state)-```--- [ ] **Step 4: Run tests to verify they pass**--Run: `cd tools/xda && uv run pytest tests/test_model.py -v`-Expected: PASS (all 3 tests). The param count test confirms architecture is correct.--- [ ] **Step 5: Commit**--```bash-git add tools/xda/src/model.py tools/xda/tests/test_model.py-git commit -m "feat(xda): BERT-based XDA model with 2-layer MLP classification head"-```-------### Task 5: Corpus Build Scripts--**Files:**-- Create: `tools/xda/scripts/fetch_debian_packages.sh`-- Create: `tools/xda/scripts/build_corpus.sh`-- Create: `tools/xda/scripts/Dockerfile.corpus`--- [ ] **Step 1: Create Dockerfile for Debian arm64 corpus**--```dockerfile-# tools/xda/scripts/Dockerfile.corpus-# Build with: docker build --platform linux/arm64 -t xda-corpus -f scripts/Dockerfile.corpus .-# Run with: docker run --platform linux/arm64 -v $(pwd)/corpus:/out xda-corpus--FROM --platform=linux/arm64 debian:bookworm--RUN apt-get update && apt-get install -y --no-install-recommends \- # Real-world C/C++ binaries- coreutils binutils findutils diffutils grep gawk sed \- tar gzip bzip2 xz-utils \- curl wget \- openssh-client openssh-server \- openssl \- sqlite3 \- nginx-core \- redis-server \- lua5.4 \- vim-tiny \- tmux \- git \- # Debug symbols- && echo "deb http://deb.debian.org/debian-debug bookworm-debug main" >> /etc/apt/sources.list \- && apt-get update \- && apt-get install -y --no-install-recommends \- coreutils-dbgsym binutils-dbgsym findutils-dbgsym \- tar-dbgsym grep-dbgsym gawk-dbgsym sed-dbgsym \- curl-dbgsym openssh-client-dbgsym \- libssl3-dbgsym libsqlite3-0-dbgsym \- redis-server-dbgsym lua5.4-dbgsym \- && rm -rf /var/lib/apt/lists/*--COPY scripts/fetch_debian_packages.sh /extract.sh-RUN chmod +x /extract.sh--CMD ["/extract.sh"]-```--- [ ] **Step 2: Create extraction script**--```bash-#!/usr/bin/env bash-# tools/xda/scripts/fetch_debian_packages.sh-# Pairs stripped binaries with their debug info from /usr/lib/debug-# Output: /out/pairs/<binary_name>/binary and /out/pairs/<binary_name>/debug--set -euo pipefail--OUT_DIR="/out/pairs"-mkdir -p "$OUT_DIR"--count=0--# Find all ELF binaries in standard paths-for bin in /usr/bin/* /usr/sbin/* /usr/lib/*/lib*.so*; do- [ -f "$bin" ] || continue-- # Check if it's actually ELF- file_type=$(file -b "$bin" 2>/dev/null) || continue- echo "$file_type" | grep -q "ELF" || continue-- # Look for corresponding debug info- # Debian stores debug files in /usr/lib/debug/<original-path>.debug- debug_path="/usr/lib/debug${bin}.debug"- if [ ! -f "$debug_path" ]; then- # Also check build-id based paths- build_id=$(readelf -n "$bin" 2>/dev/null | grep "Build ID" | awk '{print $3}') || continue- if [ -n "$build_id" ]; then- prefix="${build_id:0:2}"- suffix="${build_id:2}"- debug_path="/usr/lib/debug/.build-id/${prefix}/${suffix}.debug"- fi- fi-- [ -f "$debug_path" ] || continue-- name=$(basename "$bin")- pair_dir="${OUT_DIR}/${name}"- mkdir -p "$pair_dir"- cp "$bin" "${pair_dir}/binary"- cp "$debug_path" "${pair_dir}/debug"-- count=$((count + 1))-done--echo "Extracted $count binary/debug pairs to $OUT_DIR"-```--- [ ] **Step 3: Create cross-compilation corpus builder**--```bash-#!/usr/bin/env bash-# tools/xda/scripts/build_corpus.sh-# Cross-compiles C projects at multiple optimization levels for a target architecture.-# Usage: ./build_corpus.sh <arch> <output_dir>-# Example: ./build_corpus.sh aarch64 ./corpus/aarch64--set -euo pipefail--ARCH="${1:?Usage: build_corpus.sh <arch> <output_dir>}"-OUT="${2:?Usage: build_corpus.sh <arch> <output_dir>}"--case "$ARCH" in- x86_64) CC="gcc"; STRIP="strip" ;;- aarch64) CC="aarch64-linux-gnu-gcc"; STRIP="aarch64-linux-gnu-strip" ;;- arm) CC="arm-linux-gnueabihf-gcc"; STRIP="arm-linux-gnueabihf-strip" ;;- mips) CC="mips-linux-gnu-gcc"; STRIP="mips-linux-gnu-strip" ;;- riscv64) CC="riscv64-linux-gnu-gcc"; STRIP="riscv64-linux-gnu-strip" ;;- *) echo "Unknown arch: $ARCH"; exit 1 ;;-esac--# Verify compiler exists-command -v "$CC" >/dev/null 2>&1 || { echo "$CC not found. Install cross-compiler."; exit 1; }--mkdir -p "$OUT"--# Source files to compile — create a set of diverse test programs-SOURCES_DIR=$(mktemp -d)-trap 'rm -rf "$SOURCES_DIR"' EXIT--# Generate diverse source files-cat > "$SOURCES_DIR/algorithms.c" << 'CSRC'-#include <stdlib.h>-#include <string.h>--void bubble_sort(int *arr, int n) {- for (int i = 0; i < n - 1; i++)- for (int j = 0; j < n - i - 1; j++)- if (arr[j] > arr[j + 1]) { int t = arr[j]; arr[j] = arr[j + 1]; arr[j + 1] = t; }-}-int binary_search(int *arr, int n, int target) {- int lo = 0, hi = n - 1;- while (lo <= hi) { int mid = (lo + hi) / 2; if (arr[mid] == target) return mid; if (arr[mid] < target) lo = mid + 1; else hi = mid - 1; }- return -1;-}-void *mempool_alloc(size_t size) { static char pool[4096]; static size_t offset = 0; if (offset + size > 4096) return NULL; void *p = &pool[offset]; offset += size; return p; }-struct node { int val; struct node *next; };-struct node *list_insert(struct node *head, int val) { struct node *n = malloc(sizeof(*n)); n->val = val; n->next = head; return n; }-int list_sum(struct node *head) { int s = 0; while (head) { s += head->val; head = head->next; } return s; }-unsigned crc32_byte(unsigned crc, unsigned char b) { crc ^= b; for (int i = 0; i < 8; i++) crc = (crc >> 1) ^ (0xEDB88320 & -(crc & 1)); return crc; }-int main() { int arr[] = {5,3,1,4,2}; bubble_sort(arr, 5); return binary_search(arr, 5, 3); }-CSRC--cat > "$SOURCES_DIR/string_ops.c" << 'CSRC'-#include <string.h>-#include <ctype.h>-int str_count_char(const char *s, char c) { int n = 0; while (*s) { if (*s == c) n++; s++; } return n; }-void str_reverse(char *s) { int len = strlen(s); for (int i = 0; i < len / 2; i++) { char t = s[i]; s[i] = s[len - 1 - i]; s[len - 1 - i] = t; } }-void str_to_upper(char *s) { while (*s) { *s = toupper(*s); s++; } }-int str_is_palindrome(const char *s) { int l = 0, r = strlen(s) - 1; while (l < r) { if (s[l++] != s[r--]) return 0; } return 1; }-char *str_find_substr(const char *haystack, const char *needle) { int nlen = strlen(needle); if (!nlen) return (char*)haystack; for (; *haystack; haystack++) { if (!strncmp(haystack, needle, nlen)) return (char*)haystack; } return 0; }-int main() { char buf[] = "hello"; str_reverse(buf); return str_is_palindrome(buf); }-CSRC--cat > "$SOURCES_DIR/state_machine.c" << 'CSRC'-enum state { IDLE, RUNNING, PAUSED, ERROR, DONE };-enum event { START, PAUSE, RESUME, FAIL, FINISH };-enum state transition(enum state s, enum event e) {- switch (s) {- case IDLE: return e == START ? RUNNING : s;- case RUNNING: switch (e) { case PAUSE: return PAUSED; case FAIL: return ERROR; case FINISH: return DONE; default: return s; }- case PAUSED: return e == RESUME ? RUNNING : (e == FAIL ? ERROR : s);- case ERROR: return s;- case DONE: return s;- }- return s;-}-int run_machine(enum event *events, int n) { enum state s = IDLE; for (int i = 0; i < n; i++) s = transition(s, events[i]); return s; }-int main() { enum event seq[] = {START, PAUSE, RESUME, FINISH}; return run_machine(seq, 4); }-CSRC--# Compile each source at each optimization level-for src in "$SOURCES_DIR"/*.c; do- name=$(basename "$src" .c)- for opt in O0 O1 O2 O3 Os; do- out_debug="${OUT}/${name}_${opt}_debug"- out_stripped="${OUT}/${name}_${opt}_stripped"-- "$CC" -g "-${opt}" -o "$out_debug" "$src" 2>/dev/null || continue- cp "$out_debug" "$out_stripped"- "$STRIP" "$out_stripped"-- echo "Built: ${name}_${opt}"- done-done--echo "Corpus built in $OUT"-```--- [ ] **Step 4: Test the corpus builder locally**--Run: `cd tools/xda && chmod +x scripts/build_corpus.sh && ./scripts/build_corpus.sh x86_64 ./corpus/x86_64`-Expected: `algorithms_O0`, `algorithms_O1`, ..., `state_machine_Os` (debug + stripped pairs) created in `corpus/x86_64/`.--- [ ] **Step 5: Commit**--```bash-git add tools/xda/scripts/-git commit -m "feat(xda): corpus build scripts for Debian packages and cross-compilation"-```-------### Task 6: Full Dataset Builder (Corpus → Training Data)--**Files:**-- Create: `tools/xda/src/build_dataset.py`--This ties together extraction + chunking to produce a single training-ready dataset from a corpus directory.--- [ ] **Step 1: Implement the dataset builder**--```python-# tools/xda/src/build_dataset.py-"""Build a training dataset from a corpus of binary/debug pairs."""--import json-import random-from pathlib import Path--from tqdm import tqdm--from .extract_labels import extract_function_boundaries, generate_byte_labels-from .dataset import chunk_labeled_bytes---def build_from_pairs_dir(- pairs_dir: Path,- output_path: Path,- window_size: int = 512,- stride: int = 256,- val_split: float = 0.1,-) -> dict:- """Process all binary/debug pairs into chunked training data.-- Expects pairs_dir to contain subdirectories, each with 'binary' and 'debug' files.- Saves train.json and val.json to output_path.-- Returns stats dict.- """- all_chunks: list[dict] = []- stats = {"binaries": 0, "functions": 0, "chunks": 0, "errors": []}-- for pair_dir in tqdm(sorted(pairs_dir.iterdir()), desc="Processing binaries"):- if not pair_dir.is_dir():- continue-- debug_file = pair_dir / "debug"- binary_file = pair_dir / "binary"-- # Support both layouts: paired (binary + debug) and single (debug binary with DWARF)- if debug_file.exists():- label_source = str(debug_file)- byte_source = str(binary_file) if binary_file.exists() else str(debug_file)- else:- # Single file with embedded DWARF (e.g., from build_corpus.sh debug binaries)- candidates = [f for f in pair_dir.iterdir() if f.is_file() and "debug" in f.name]- if not candidates:- continue- label_source = str(candidates[0])- byte_source = label_source-- try:- boundaries = extract_function_boundaries(label_source)- if not boundaries:- continue-- text_bytes, labels = generate_byte_labels(byte_source, boundaries)- if not text_bytes:- continue-- chunks = chunk_labeled_bytes(text_bytes, labels, window_size, stride)- all_chunks.extend(chunks)-- stats["binaries"] += 1- stats["functions"] += len(boundaries)- stats["chunks"] += len(chunks)- except Exception as e:- stats["errors"].append({"file": str(pair_dir), "error": str(e)})-- # Shuffle and split- random.shuffle(all_chunks)- split_idx = int(len(all_chunks) * (1 - val_split))- train_chunks = all_chunks[:split_idx]- val_chunks = all_chunks[split_idx:]-- # Serialize — store bytes as list[int] for JSON compatibility- output_path.mkdir(parents=True, exist_ok=True)-- def serialize_chunks(chunks: list[dict], path: Path) -> None:- serialized = []- for c in chunks:- serialized.append({- "bytes": list(c["bytes"]),- "labels": c["labels"],- })- path.write_text(json.dumps(serialized))-- serialize_chunks(train_chunks, output_path / "train.json")- serialize_chunks(val_chunks, output_path / "val.json")-- stats["train_chunks"] = len(train_chunks)- stats["val_chunks"] = len(val_chunks)- (output_path / "stats.json").write_text(json.dumps(stats, indent=2))-- return stats---def build_from_corpus_dir(- corpus_dir: Path,- output_path: Path,- window_size: int = 512,- stride: int = 256,- val_split: float = 0.1,-) -> dict:- """Process debug binaries from build_corpus.sh output.-- Expects corpus_dir to contain *_debug and *_stripped files.- Uses debug files for labels, stripped files for byte content.- """- all_chunks: list[dict] = []- stats = {"binaries": 0, "functions": 0, "chunks": 0, "errors": []}-- debug_files = sorted(corpus_dir.glob("*_debug"))-- for debug_file in tqdm(debug_files, desc="Processing corpus"):- try:- boundaries = extract_function_boundaries(str(debug_file))- if not boundaries:- continue-- # Use the debug binary itself for .text bytes- # (stripped version has same .text content but no DWARF)- text_bytes, labels = generate_byte_labels(str(debug_file), boundaries)- if not text_bytes:- continue-- chunks = chunk_labeled_bytes(text_bytes, labels, window_size, stride)- all_chunks.extend(chunks)-- stats["binaries"] += 1- stats["functions"] += len(boundaries)- stats["chunks"] += len(chunks)- except Exception as e:- stats["errors"].append({"file": str(debug_file), "error": str(e)})-- random.shuffle(all_chunks)- split_idx = int(len(all_chunks) * (1 - val_split))- train_chunks = all_chunks[:split_idx]- val_chunks = all_chunks[split_idx:]-- output_path.mkdir(parents=True, exist_ok=True)-- def serialize_chunks(chunks: list[dict], path: Path) -> None:- serialized = [{"bytes": list(c["bytes"]), "labels": c["labels"]} for c in chunks]- path.write_text(json.dumps(serialized))-- serialize_chunks(train_chunks, output_path / "train.json")- serialize_chunks(val_chunks, output_path / "val.json")-- stats["train_chunks"] = len(train_chunks)- stats["val_chunks"] = len(val_chunks)- (output_path / "stats.json").write_text(json.dumps(stats, indent=2))-- return stats-```--- [ ] **Step 2: Test end-to-end on x86_64 corpus**--Run: `cd tools/xda && uv run python -c "from src.build_dataset import build_from_corpus_dir; from pathlib import Path; stats = build_from_corpus_dir(Path('./corpus/x86_64'), Path('./data/x86_64')); print(stats)"`-Expected: `train.json` and `val.json` created in `data/x86_64/`, stats printed showing chunk/function counts.--- [ ] **Step 3: Commit**--```bash-git add tools/xda/src/build_dataset.py-git commit -m "feat(xda): end-to-end dataset builder from corpus to training data"-```-------### Task 7: Pre-training Script (Masked Byte Modeling)--**Files:**-- Create: `tools/xda/src/pretrain.py`-- Create: `tools/xda/configs/pretrain.yaml`--This is the optional step for new architectures. For x86_64 you'd use the CUMLSec checkpoint. For AArch64/ARM/MIPS, you pre-train on unlabeled bytes first.--- [ ] **Step 1: Create pre-training config**--```yaml-# tools/xda/configs/pretrain.yaml-model:- hidden_size: 768- num_hidden_layers: 12- num_attention_heads: 12- intermediate_size: 3072--training:- epochs: 10- batch_size: 64- learning_rate: 1.0e-4- warmup_ratio: 0.1- weight_decay: 0.01- mask_probability: 0.20 # 20% of bytes masked- mask_token_ratio: 0.50 # 50% of masked → [MASK], 50% → random byte- max_length: 514 # 512 + [CLS] + [SEP]- fp16: true--data:- window_size: 512- stride: 256--output:- save_every_n_epochs: 2- checkpoint_dir: checkpoints/pretrain-```--- [ ] **Step 2: Implement pre-training script**--```python-# tools/xda/src/pretrain.py-"""Masked byte modeling pre-training for XDA on a target architecture."""--import argparse-import json-import random-from pathlib import Path--import torch-import yaml-from torch import nn-from torch.utils.data import DataLoader, Dataset-from tqdm import tqdm--from .model import XdaConfig, XdaModel-from .tokenizer import ByteTokenizer---class MaskedByteDataset(Dataset):- """Dataset that applies masking on-the-fly for pre-training."""-- def __init__(- self,- chunks: list[dict],- tokenizer: ByteTokenizer,- mask_prob: float = 0.2,- mask_token_ratio: float = 0.5,- ) -> None:- self.chunks = chunks- self.tokenizer = tokenizer- self.mask_prob = mask_prob- self.mask_token_ratio = mask_token_ratio-- def __len__(self) -> int:- return len(self.chunks)-- def __getitem__(self, idx: int) -> dict[str, torch.Tensor]:- raw_bytes = bytes(self.chunks[idx]["bytes"])- input_ids = self.tokenizer.encode(raw_bytes)- labels = list(input_ids) # original tokens are labels-- # Mask byte positions (skip [CLS] at 0 and [SEP] at -1)- for i in range(1, len(input_ids) - 1):- if random.random() < self.mask_prob:- if random.random() < self.mask_token_ratio:- input_ids[i] = self.tokenizer.mask_id- else:- input_ids[i] = random.randint(0, 255)- else:- labels[i] = -100 # don't compute loss on unmasked-- labels[0] = -100 # [CLS]- labels[-1] = -100 # [SEP]-- return {- "input_ids": torch.tensor(input_ids, dtype=torch.long),- "attention_mask": torch.ones(len(input_ids), dtype=torch.long),- "labels": torch.tensor(labels, dtype=torch.long),- }---def pretrain(config_path: str, data_path: str) -> None:- with open(config_path) as f:- cfg = yaml.safe_load(f)-- device = torch.device("cuda" if torch.cuda.is_available() else "mps" if torch.backends.mps.is_available() else "cpu")- print(f"Using device: {device}")-- # Load data- train_chunks = json.loads(Path(data_path, "train.json").read_text())- print(f"Loaded {len(train_chunks)} training chunks")-- tokenizer = ByteTokenizer()- dataset = MaskedByteDataset(- train_chunks,- tokenizer,- mask_prob=cfg["training"]["mask_probability"],- mask_token_ratio=cfg["training"]["mask_token_ratio"],- )- loader = DataLoader(- dataset,- batch_size=cfg["training"]["batch_size"],- shuffle=True,- num_workers=4,- pin_memory=True,- )-- # Build model — for pre-training, the "classification" head predicts masked bytes- # So num_labels = vocab_size (261), not 3- model_cfg = XdaConfig(- hidden_size=cfg["model"]["hidden_size"],- num_hidden_layers=cfg["model"]["num_hidden_layers"],- num_attention_heads=cfg["model"]["num_attention_heads"],- intermediate_size=cfg["model"]["intermediate_size"],- num_labels=tokenizer.vocab_size, # predict byte identity- )- model = XdaModel(model_cfg).to(device)-- optimizer = torch.optim.AdamW(- model.parameters(),- lr=cfg["training"]["learning_rate"],- weight_decay=cfg["training"]["weight_decay"],- )-- total_steps = len(loader) * cfg["training"]["epochs"]- warmup_steps = int(total_steps * cfg["training"]["warmup_ratio"])- scheduler = torch.optim.lr_scheduler.LinearLR(- optimizer, start_factor=0.01, end_factor=1.0, total_iters=warmup_steps,- )-- loss_fn = nn.CrossEntropyLoss(ignore_index=-100)- scaler = torch.amp.GradScaler(enabled=cfg["training"]["fp16"] and device.type == "cuda")-- checkpoint_dir = Path(cfg["output"]["checkpoint_dir"])- checkpoint_dir.mkdir(parents=True, exist_ok=True)-- for epoch in range(cfg["training"]["epochs"]):- model.train()- total_loss = 0.0- steps = 0-- pbar = tqdm(loader, desc=f"Epoch {epoch + 1}/{cfg['training']['epochs']}")- for batch in pbar:- input_ids = batch["input_ids"].to(device)- attention_mask = batch["attention_mask"].to(device)- labels = batch["labels"].to(device)-- with torch.amp.autocast(device.type, enabled=cfg["training"]["fp16"]):- logits = model(input_ids, attention_mask)- loss = loss_fn(logits.view(-1, model_cfg.num_labels), labels.view(-1))-- scaler.scale(loss).backward()- scaler.step(optimizer)- scaler.update()- optimizer.zero_grad()- scheduler.step()-- total_loss += loss.item()- steps += 1- pbar.set_postfix(loss=f"{total_loss / steps:.4f}")-- avg_loss = total_loss / steps- print(f"Epoch {epoch + 1}: avg_loss={avg_loss:.4f}")-- if (epoch + 1) % cfg["output"]["save_every_n_epochs"] == 0:- torch.save(model.state_dict(), checkpoint_dir / f"pretrain_epoch{epoch + 1}.pt")-- torch.save(model.state_dict(), checkpoint_dir / "pretrain_final.pt")- print(f"Pre-training complete. Final checkpoint: {checkpoint_dir / 'pretrain_final.pt'}")---if __name__ == "__main__":- parser = argparse.ArgumentParser()- parser.add_argument("--config", required=True)- parser.add_argument("--data", required=True)- args = parser.parse_args()- pretrain(args.config, args.data)-```--- [ ] **Step 3: Commit**--```bash-git add tools/xda/src/pretrain.py tools/xda/configs/pretrain.yaml-git commit -m "feat(xda): masked byte modeling pre-training script"-```-------### Task 8: Fine-tuning Script--**Files:**-- Create: `tools/xda/src/finetune.py`-- Create: `tools/xda/configs/finetune.yaml`--- [ ] **Step 1: Create fine-tuning config**--```yaml-# tools/xda/configs/finetune.yaml-model:- hidden_size: 768- num_hidden_layers: 12- num_attention_heads: 12- intermediate_size: 3072--training:- epochs: 5- batch_size: 64- learning_rate: 2.0e-5 # lower LR for fine-tuning- warmup_ratio: 0.1- weight_decay: 0.01- fp16: true--data:- window_size: 512- stride: 256--output:- checkpoint_dir: checkpoints/finetune- save_best: true--# Optional: path to pre-trained checkpoint to initialize from-pretrained_checkpoint: null # e.g., "checkpoints/pretrain/pretrain_final.pt"-```--- [ ] **Step 2: Implement fine-tuning script**--```python-# tools/xda/src/finetune.py-"""Fine-tune XDA for function boundary detection (3-class token classification)."""--import argparse-import json-from pathlib import Path--import torch-import yaml-from torch import nn-from torch.utils.data import DataLoader-from tqdm import tqdm--from .dataset import ByteDataset, chunk_labeled_bytes-from .model import XdaConfig, XdaModel-from .tokenizer import ByteTokenizer---def evaluate(model: XdaModel, loader: DataLoader, device: torch.device) -> dict:- """Compute per-class precision, recall, F1 on validation set."""- model.eval()- # Per-class counts: 0=non-func, 1=func_start, 2=func_body- tp = [0, 0, 0]- fp = [0, 0, 0]- fn = [0, 0, 0]-- with torch.no_grad():- for batch in loader:- input_ids = batch["input_ids"].to(device)- attention_mask = batch["attention_mask"].to(device)- labels = batch["labels"].to(device)-- logits = model(input_ids, attention_mask)- preds = logits.argmax(dim=-1)-- mask = labels != -100- preds = preds[mask]- labels = labels[mask]-- for cls in range(3):- tp[cls] += ((preds == cls) & (labels == cls)).sum().item()- fp[cls] += ((preds == cls) & (labels != cls)).sum().item()- fn[cls] += ((preds != cls) & (labels == cls)).sum().item()-- results = {}- for cls, name in enumerate(["non_function", "function_start", "function_body"]):- precision = tp[cls] / (tp[cls] + fp[cls]) if tp[cls] + fp[cls] > 0 else 0- recall = tp[cls] / (tp[cls] + fn[cls]) if tp[cls] + fn[cls] > 0 else 0- f1 = 2 * precision * recall / (precision + recall) if precision + recall > 0 else 0- results[name] = {"precision": precision, "recall": recall, "f1": f1}-- return results---def finetune(config_path: str, data_path: str) -> None:- with open(config_path) as f:- cfg = yaml.safe_load(f)-- device = torch.device("cuda" if torch.cuda.is_available() else "mps" if torch.backends.mps.is_available() else "cpu")- print(f"Using device: {device}")-- # Load chunked data- train_chunks = json.loads(Path(data_path, "train.json").read_text())- val_chunks = json.loads(Path(data_path, "val.json").read_text())- print(f"Train: {len(train_chunks)} chunks, Val: {len(val_chunks)} chunks")-- tokenizer = ByteTokenizer()- train_ds = ByteDataset(train_chunks, tokenizer)- val_ds = ByteDataset(val_chunks, tokenizer)-- train_loader = DataLoader(train_ds, batch_size=cfg["training"]["batch_size"], shuffle=True, num_workers=4, pin_memory=True)- val_loader = DataLoader(val_ds, batch_size=cfg["training"]["batch_size"], num_workers=4)-- # Build model (3-class: non-function, function_start, function_body)- model_cfg = XdaConfig(- hidden_size=cfg["model"]["hidden_size"],- num_hidden_layers=cfg["model"]["num_hidden_layers"],- num_attention_heads=cfg["model"]["num_attention_heads"],- intermediate_size=cfg["model"]["intermediate_size"],- num_labels=3,- )- model = XdaModel(model_cfg)-- # Load pre-trained encoder weights if available- pretrained_path = cfg.get("pretrained_checkpoint")- if pretrained_path and Path(pretrained_path).exists():- print(f"Loading pre-trained weights from {pretrained_path}")- state = torch.load(pretrained_path, map_location="cpu", weights_only=True)- # Load only encoder weights (skip classifier head — different num_labels)- encoder_state = {k: v for k, v in state.items() if k.startswith("encoder.")}- model.load_state_dict(encoder_state, strict=False)- print(f"Loaded {len(encoder_state)} encoder weight tensors")-- model = model.to(device)-- optimizer = torch.optim.AdamW(- model.parameters(),- lr=cfg["training"]["learning_rate"],- weight_decay=cfg["training"]["weight_decay"],- )- total_steps = len(train_loader) * cfg["training"]["epochs"]- warmup_steps = int(total_steps * cfg["training"]["warmup_ratio"])- scheduler = torch.optim.lr_scheduler.LinearLR(optimizer, start_factor=0.01, end_factor=1.0, total_iters=warmup_steps)-- loss_fn = nn.CrossEntropyLoss(ignore_index=-100)- scaler = torch.amp.GradScaler(enabled=cfg["training"]["fp16"] and device.type == "cuda")-- checkpoint_dir = Path(cfg["output"]["checkpoint_dir"])- checkpoint_dir.mkdir(parents=True, exist_ok=True)-- best_f1 = 0.0-- for epoch in range(cfg["training"]["epochs"]):- # Train- model.train()- total_loss = 0.0- steps = 0-- pbar = tqdm(train_loader, desc=f"Epoch {epoch + 1}/{cfg['training']['epochs']}")- for batch in pbar:- input_ids = batch["input_ids"].to(device)- attention_mask = batch["attention_mask"].to(device)- labels = batch["labels"].to(device)-- with torch.amp.autocast(device.type, enabled=cfg["training"]["fp16"]):- logits = model(input_ids, attention_mask)- loss = loss_fn(logits.view(-1, 3), labels.view(-1))-- scaler.scale(loss).backward()- scaler.step(optimizer)- scaler.update()- optimizer.zero_grad()- scheduler.step()-- total_loss += loss.item()- steps += 1- pbar.set_postfix(loss=f"{total_loss / steps:.4f}")-- # Evaluate- results = evaluate(model, val_loader, device)- fs_f1 = results["function_start"]["f1"]-- print(f"Epoch {epoch + 1}: loss={total_loss / steps:.4f}")- for name, metrics in results.items():- print(f" {name}: P={metrics['precision']:.4f} R={metrics['recall']:.4f} F1={metrics['f1']:.4f}")-- # Save best by function_start F1 (the metric that matters most)- if cfg["output"]["save_best"] and fs_f1 > best_f1:- best_f1 = fs_f1- torch.save(model.state_dict(), checkpoint_dir / "best.pt")- print(f" New best function_start F1: {fs_f1:.4f}")-- torch.save(model.state_dict(), checkpoint_dir / f"epoch{epoch + 1}.pt")-- # Save final config alongside weights (needed for Rust loading)- config_out = {- "vocab_size": model_cfg.vocab_size,- "hidden_size": model_cfg.hidden_size,- "num_hidden_layers": model_cfg.num_hidden_layers,- "num_attention_heads": model_cfg.num_attention_heads,- "intermediate_size": model_cfg.intermediate_size,- "max_position_embeddings": model_cfg.max_position_embeddings,- "num_labels": model_cfg.num_labels,- "classifier_hidden": model_cfg.classifier_hidden,- }- (checkpoint_dir / "config.json").write_text(json.dumps(config_out, indent=2))-- print(f"Fine-tuning complete. Best F1: {best_f1:.4f}")---if __name__ == "__main__":- parser = argparse.ArgumentParser()- parser.add_argument("--config", required=True)- parser.add_argument("--data", required=True)- args = parser.parse_args()- finetune(args.config, args.data)-```--- [ ] **Step 3: Commit**--```bash-git add tools/xda/src/finetune.py tools/xda/configs/finetune.yaml-git commit -m "feat(xda): function boundary fine-tuning with per-class eval"-```-------### Task 9: Export to Safetensors--**Files:**-- Create: `tools/xda/src/export.py`--This is the bridge between Python training and Rust inference. Produces the exact artifacts that candle loads.--- [ ] **Step 1: Implement export script**--```python-# tools/xda/src/export.py-"""Export trained XDA checkpoint to safetensors format for Rust/candle loading."""--import argparse-import json-import shutil-from pathlib import Path--import torch-from safetensors.torch import save_file--from .model import XdaConfig, XdaModel---def export(checkpoint_path: str, config_path: str, output_dir: str, arch: str) -> None:- """Export a PyTorch checkpoint + config to safetensors + config.json.-- Output structure:- output_dir/- xda_{arch}.safetensors # model weights- xda_{arch}_config.json # model config (for candle)- """- out = Path(output_dir)- out.mkdir(parents=True, exist_ok=True)-- # Load config- with open(config_path) as f:- config_dict = json.load(f)- config = XdaConfig(**config_dict)-- # Load model and checkpoint- model = XdaModel(config)- state = torch.load(checkpoint_path, map_location="cpu", weights_only=True)- model.load_state_dict(state)-- # Flatten state dict — candle needs simple key → tensor mapping- # Rename keys to match what the Rust model expects- flat_state = {}- for key, tensor in model.state_dict().items():- # Convert to float32 (candle safetensors loading expects f32)- flat_state[key] = tensor.float()-- # Save as safetensors- weights_path = out / f"xda_{arch}.safetensors"- save_file(flat_state, str(weights_path))-- # Save config JSON (Rust side reads this to construct the model)- config_out = out / f"xda_{arch}_config.json"- config_out.write_text(json.dumps(config_dict, indent=2))-- # Print summary- total_params = sum(t.numel() for t in flat_state.values())- file_size_mb = weights_path.stat().st_size / (1024 * 1024)- print(f"Exported {arch} model:")- print(f" Weights: {weights_path} ({file_size_mb:.1f} MB)")- print(f" Config: {config_out}")- print(f" Parameters: {total_params:,}")---if __name__ == "__main__":- parser = argparse.ArgumentParser()- parser.add_argument("--checkpoint", required=True, help="Path to .pt checkpoint")- parser.add_argument("--config", required=True, help="Path to config.json from training")- parser.add_argument("--output", required=True, help="Output directory")- parser.add_argument("--arch", required=True, help="Architecture name (x86_64, aarch64, etc.)")- args = parser.parse_args()- export(args.checkpoint, args.config, args.output, args.arch)-```--- [ ] **Step 2: Commit**--```bash-git add tools/xda/src/export.py-git commit -m "feat(xda): export trained models to safetensors for Rust/candle loading"-```-------## Part B: Rust Candle Inference--### Task 10: Cargo Workspace & kong-types Crate--**Files:**-- Create: `kong-rs/Cargo.toml` (workspace root)-- Create: `kong-rs/crates/kong-types/Cargo.toml`-- Create: `kong-rs/crates/kong-types/src/lib.rs`-- Create: `kong-rs/crates/kong-types/src/binary.rs`-- Create: `kong-rs/crates/kong-types/src/function.rs`--- [ ] **Step 1: Create workspace Cargo.toml**--```toml-# kong-rs/Cargo.toml-[workspace]-members = [- "crates/kong-types",- "crates/kong-ml",-]-resolver = "2"--[workspace.package]-version = "1.0.0"-edition = "2021"-license = "MIT"--[workspace.dependencies]-serde = { version = "1", features = ["derive"] }-serde_json = "1"-```--- [ ] **Step 2: Create kong-types crate**--```toml-# kong-rs/crates/kong-types/Cargo.toml-[package]-name = "kong-types"-version.workspace = true-edition.workspace = true--[dependencies]-serde.workspace = true-```--```rust-// kong-rs/crates/kong-types/src/lib.rs-pub mod binary;-pub mod function;-```--```rust-// kong-rs/crates/kong-types/src/binary.rs-use serde::{Deserialize, Serialize};--#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)]-#[serde(rename_all = "snake_case")]-pub enum Arch {- X86_64,- X86,- Aarch64,- Arm,- Mips,- Mips64,- Riscv64,- PowerPc,-}--impl Arch {- /// Model artifact name for this architecture.- pub fn model_name(&self) -> &'static str {- match self {- Arch::X86_64 => "x86_64",- Arch::X86 => "x86",- Arch::Aarch64 => "aarch64",- Arch::Arm => "arm",- Arch::Mips => "mips",- Arch::Mips64 => "mips64",- Arch::Riscv64 => "riscv64",- Arch::PowerPc => "powerpc",- }- }-}--impl std::fmt::Display for Arch {- fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {- f.write_str(self.model_name())- }-}-```--```rust-// kong-rs/crates/kong-types/src/function.rs-use serde::{Deserialize, Serialize};--/// A confirmed function boundary from either heuristic or ML detection.-#[derive(Debug, Clone, Serialize, Deserialize)]-pub struct FunctionBoundary {- pub start: u64,- pub end: u64,- pub confidence: f32,- pub source: DetectionSource,-}--/// How a function boundary was detected.-#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]-#[serde(rename_all = "snake_case")]-pub enum DetectionSource {- Symbol, // from symbol table- Heuristic, // from prologue/call-target analysis- Ml, // from XDA model- ExceptionInfo, // from .eh_frame / .pdata-}--/// A region of .text that heuristics couldn't confidently classify.-#[derive(Debug, Clone)]-pub struct AmbiguousRegion {- pub start: u64,- pub bytes: Vec<u8>,- pub reason: AmbiguityReason,-}--#[derive(Debug, Clone, Copy, PartialEq, Eq)]-pub enum AmbiguityReason {- NoPrologue, // reachable but no recognizable prologue- IndirectTarget, // suspected indirect call/jump target- GapRegion, // unreached .text gap between known functions- OptimizedCode, // code region with non-standard patterns-}-```--- [ ] **Step 3: Verify it compiles**--Run: `cd kong-rs && cargo check`-Expected: Compiles with no errors.--- [ ] **Step 4: Commit**--```bash-git add kong-rs/-git commit -m "feat(kong-rs): workspace scaffolding with kong-types crate"-```-------### Task 11: kong-ml Crate — Byte Tokenizer & Config--**Files:**-- Create: `kong-rs/crates/kong-ml/Cargo.toml`-- Create: `kong-rs/crates/kong-ml/src/lib.rs`-- Create: `kong-rs/crates/kong-ml/src/tokenizer.rs`-- Create: `kong-rs/crates/kong-ml/src/config.rs`--- [ ] **Step 1: Create kong-ml Cargo.toml**--```toml-# kong-rs/crates/kong-ml/Cargo.toml-[package]-name = "kong-ml"-version.workspace = true-edition.workspace = true--[dependencies]-kong-types = { path = "../kong-types" }-candle-core = "0.8"-candle-nn = "0.8"-candle-transformers = "0.8"-safetensors = "0.4"-serde.workspace = true-serde_json.workspace = true-dirs = "5"-```--- [ ] **Step 2: Implement byte tokenizer in Rust**--```rust-// kong-rs/crates/kong-ml/src/tokenizer.rs--/// Byte-level tokenizer for XDA. Maps raw bytes 0-255 to token IDs 0-255.-/// Special tokens occupy IDs 256-260.-pub struct ByteTokenizer {- pub pad_id: u32,- pub cls_id: u32,- pub sep_id: u32,- pub mask_id: u32,- pub unk_id: u32,- pub vocab_size: u32,-}--impl Default for ByteTokenizer {- fn default() -> Self {- Self {- pad_id: 256,- cls_id: 257,- sep_id: 258,- mask_id: 259,- unk_id: 260,- vocab_size: 261,- }- }-}--impl ByteTokenizer {- pub fn new() -> Self {- Self::default()- }-- /// Encode raw bytes as token IDs: [CLS] + byte_values + [SEP], padded to max_length.- pub fn encode(&self, bytes: &[u8], max_length: Option<usize>) -> Vec<u32> {- let mut ids = Vec::with_capacity(bytes.len() + 2);- ids.push(self.cls_id);- ids.extend(bytes.iter().map(|&b| b as u32));- ids.push(self.sep_id);-- if let Some(max_len) = max_length {- ids.truncate(max_len);- ids.resize(max_len, self.pad_id);- }-- ids- }-- /// Generate attention mask: 1 for real tokens, 0 for padding.- pub fn attention_mask(&self, token_ids: &[u32]) -> Vec<u32> {- token_ids- .iter()- .map(|&id| if id == self.pad_id { 0 } else { 1 })- .collect()- }-}--#[cfg(test)]-mod tests {- use super::*;-- #[test]- fn test_special_token_ids() {- let tok = ByteTokenizer::new();- assert_eq!(tok.pad_id, 256);- assert_eq!(tok.cls_id, 257);- assert_eq!(tok.sep_id, 258);- assert_eq!(tok.vocab_size, 261);- }-- #[test]- fn test_encode_bytes() {- let tok = ByteTokenizer::new();- let raw = [0x55u8, 0x48, 0x89, 0xe5];- let ids = tok.encode(&raw, None);- assert_eq!(ids, vec![257, 0x55, 0x48, 0x89, 0xe5, 258]);- }-- #[test]- fn test_encode_with_padding() {- let tok = ByteTokenizer::new();- let raw = [0x55u8, 0x48];- let ids = tok.encode(&raw, Some(8));- assert_eq!(ids, vec![257, 0x55, 0x48, 258, 256, 256, 256, 256]);- }-- #[test]- fn test_attention_mask() {- let tok = ByteTokenizer::new();- let raw = [0x55u8, 0x48];- let ids = tok.encode(&raw, Some(8));- let mask = tok.attention_mask(&ids);- assert_eq!(mask, vec![1, 1, 1, 1, 0, 0, 0, 0]);- }-}-```--- [ ] **Step 3: Implement config loading**--```rust-// kong-rs/crates/kong-ml/src/config.rs-use serde::{Deserialize, Serialize};--/// XDA model configuration. Loaded from config.json alongside safetensors weights.-#[derive(Debug, Clone, Serialize, Deserialize)]-pub struct XdaConfig {- pub vocab_size: usize,- pub hidden_size: usize,- pub num_hidden_layers: usize,- pub num_attention_heads: usize,- pub intermediate_size: usize,- pub max_position_embeddings: usize,- pub num_labels: usize,- pub classifier_hidden: usize,-}--impl Default for XdaConfig {- fn default() -> Self {- Self {- vocab_size: 261,- hidden_size: 768,- num_hidden_layers: 12,- num_attention_heads: 12,- intermediate_size: 3072,- max_position_embeddings: 514,- num_labels: 3,- classifier_hidden: 256,- }- }-}--impl XdaConfig {- pub fn from_json(path: &std::path::Path) -> Result<Self, Box<dyn std::error::Error>> {- let content = std::fs::read_to_string(path)?;- let config: Self = serde_json::from_str(&content)?;- Ok(config)- }-- /// Convert to candle-transformers BertConfig for the encoder.- pub fn to_bert_config(&self) -> candle_transformers::models::bert::Config {- candle_transformers::models::bert::Config {- vocab_size: self.vocab_size,- hidden_size: self.hidden_size,- num_hidden_layers: self.num_hidden_layers,- num_attention_heads: self.num_attention_heads,- intermediate_size: self.intermediate_size,- max_position_embeddings: self.max_position_embeddings,- hidden_dropout_prob: 0.0, // no dropout during inference- ..Default::default()- }- }-}-```--- [ ] **Step 4: Wire up lib.rs**--```rust-// kong-rs/crates/kong-ml/src/lib.rs-pub mod config;-pub mod tokenizer;-pub mod model;-pub mod detector;-pub mod registry;-```--- [ ] **Step 5: Verify it compiles**--Run: `cd kong-rs && cargo check`-Expected: Compiles (model.rs, detector.rs, registry.rs will be empty stubs initially — create them with just `// TODO` if needed to satisfy mod declarations, or gate them behind a cfg flag). For now, comment out the three unimplemented mods in lib.rs:--```rust-// kong-rs/crates/kong-ml/src/lib.rs-pub mod config;-pub mod tokenizer;-// pub mod model; // Task 12-// pub mod detector; // Task 13-// pub mod registry; // Task 14-```--Run: `cd kong-rs && cargo test -p kong-ml`-Expected: All 4 tokenizer tests pass.--- [ ] **Step 6: Commit**--```bash-git add kong-rs/crates/kong-ml/-git commit -m "feat(kong-ml): byte tokenizer and config loading for candle inference"-```-------### Task 12: XDA Model in Candle--**Files:**-- Create: `kong-rs/crates/kong-ml/src/model.rs`--This is the core Rust implementation — BERT encoder (from candle-transformers) + 2-layer MLP classification head.--- [ ] **Step 1: Implement XdaModel**--```rust-// kong-rs/crates/kong-ml/src/model.rs-use candle_core::{DType, Device, Result, Tensor};-use candle_nn::{linear, Linear, Module, VarBuilder};-use candle_transformers::models::bert::{BertModel, Config as BertConfig};--use crate::config::XdaConfig;--/// 2-layer MLP classification head.-struct ClassifierHead {- fc1: Linear,- fc2: Linear,-}--impl ClassifierHead {- fn load(vb: VarBuilder, hidden_size: usize, classifier_hidden: usize, num_labels: usize) -> Result<Self> {- let fc1 = linear(hidden_size, classifier_hidden, vb.pp("classifier.0"))?;- let fc2 = linear(classifier_hidden, num_labels, vb.pp("classifier.3"))?;- Ok(Self { fc1, fc2 })- }-- fn forward(&self, hidden_states: &Tensor) -> Result<Tensor> {- let x = self.fc1.forward(hidden_states)?;- let x = x.gelu_erf()?;- self.fc2.forward(&x)- }-}--/// XDA model: BERT encoder + classification head for function boundary detection.-pub struct XdaModel {- encoder: BertModel,- classifier: ClassifierHead,-}--impl XdaModel {- /// Load model from safetensors weights.- pub fn load(config: &XdaConfig, vb: VarBuilder) -> Result<Self> {- let bert_config = config.to_bert_config();- let encoder = BertModel::load(vb.pp("encoder"), &bert_config)?;- let classifier = ClassifierHead::load(- vb.clone(),- config.hidden_size,- config.classifier_hidden,- config.num_labels,- )?;- Ok(Self { encoder, classifier })- }-- /// Run inference. Returns logits of shape (batch, seq_len, num_labels).- pub fn forward(&self, input_ids: &Tensor, attention_mask: &Tensor) -> Result<Tensor> {- // BertModel::forward requires token_type_ids (mandatory) and attention_mask (optional).- // For XDA we use single-segment input, so token_type_ids is all zeros.- let token_type_ids = input_ids.zeros_like()?;- let hidden = self.encoder.forward(input_ids, &token_type_ids, Some(attention_mask))?;- self.classifier.forward(&hidden)- }-}-```--**Important note on weight key mapping:** The Python model saves weights with keys like `encoder.embeddings.word_embeddings.weight`, `classifier.0.weight`, `classifier.3.weight`. The Rust side uses `VarBuilder::pp("encoder")` and `VarBuilder::pp("classifier.0")` etc. to match these keys. If the keys don't align, you'll get a "tensor not found" error at load time — debug by printing the keys in the safetensors file:--```rust-// Debug helper — use temporarily if weight loading fails-let tensors = safetensors::SafeTensors::deserialize(&std::fs::read(path)?)?;-for name in tensors.names() {- println!("{name}");-}-```--- [ ] **Step 2: Uncomment model in lib.rs**--```rust-// kong-rs/crates/kong-ml/src/lib.rs-pub mod config;-pub mod tokenizer;-pub mod model;-// pub mod detector; // Task 13-// pub mod registry; // Task 14-```--- [ ] **Step 3: Verify it compiles**--Run: `cd kong-rs && cargo check -p kong-ml`-Expected: Compiles. No runtime test yet — we need actual weights for that.--- [ ] **Step 4: Commit**--```bash-git add kong-rs/crates/kong-ml/src/model.rs kong-rs/crates/kong-ml/src/lib.rs-git commit -m "feat(kong-ml): XDA model implementation in candle (BERT + MLP head)"-```-------### Task 13: XDA Detector (Sliding Window + Batched Inference)--**Files:**-- Create: `kong-rs/crates/kong-ml/src/detector.rs`--- [ ] **Step 1: Implement XdaDetector**--```rust-// kong-rs/crates/kong-ml/src/detector.rs-use candle_core::{DType, Device, IndexOp, Result, Tensor};-use kong_types::function::{AmbiguousRegion, DetectionSource, FunctionBoundary};--use crate::config::XdaConfig;-use crate::model::XdaModel;-use crate::tokenizer::ByteTokenizer;--pub struct XdaDetector {- model: XdaModel,- tokenizer: ByteTokenizer,- device: Device,- confidence_threshold: f32,- window_size: usize,- stride: usize,-}--/// Per-byte prediction from a single window.-#[derive(Debug, Clone)]-struct BytePrediction {- address: u64,- label: u8, // 0=non-func, 1=func_start, 2=func_body- confidence: f32,-}--impl XdaDetector {- pub fn new(- model: XdaModel,- device: Device,- confidence_threshold: f32,- ) -> Self {- Self {- model,- tokenizer: ByteTokenizer::new(),- device,- confidence_threshold,- window_size: 512,- stride: 256,- }- }-- /// Predict function boundaries in ambiguous regions.- pub fn predict(&self, regions: &[AmbiguousRegion]) -> Result<Vec<FunctionBoundary>> {- let mut boundaries = Vec::new();-- for region in regions {- let region_predictions = self.predict_region(region)?;- boundaries.extend(region_predictions);- }-- Ok(boundaries)- }-- fn predict_region(&self, region: &AmbiguousRegion) -> Result<Vec<FunctionBoundary>> {- let bytes = ®ion.bytes;- if bytes.is_empty() {- return Ok(Vec::new());- }-- // Slide windows over the region- let windows = self.create_windows(bytes);- if windows.is_empty() {- return Ok(Vec::new());- }-- // Batch inference- let predictions = self.run_batch(&windows)?;-- // Merge overlapping window predictions — for overlapping bytes,- // average the confidence scores- let merged = self.merge_predictions(&predictions, bytes.len(), region.start);-- // Extract function boundaries from merged predictions- Ok(self.extract_boundaries(&merged))- }-- /// Returns (window_bytes, actual_byte_count) pairs. actual_byte_count tracks- /// how many bytes are real vs padding, since binary code can legitimately contain 0x00.- fn create_windows(&self, bytes: &[u8]) -> Vec<(Vec<u8>, usize)> {- let mut windows = Vec::new();- let mut offset = 0;-- while offset < bytes.len() {- let end = (offset + self.window_size).min(bytes.len());- let actual_len = end - offset;- let mut window = bytes[offset..end].to_vec();-- // Pad if necessary- if window.len() < self.window_size {- window.resize(self.window_size, 0);- }-- windows.push((window, actual_len));- offset += self.stride;-- if end == bytes.len() {- break;- }- }-- windows- }-- fn run_batch(&self, windows: &[(Vec<u8>, usize)]) -> Result<Vec<Vec<(u8, f32)>>> {- let seq_len = self.window_size + 2; // +2 for [CLS] and [SEP]- let batch_size = windows.len();-- // Tokenize all windows- let mut all_ids = Vec::with_capacity(batch_size * seq_len);- let mut all_mask = Vec::with_capacity(batch_size * seq_len);-- for (window, _actual_len) in windows {- let ids = self.tokenizer.encode(window, Some(seq_len));- let mask = self.tokenizer.attention_mask(&ids);- all_ids.extend(ids.iter().map(|&x| x as i64));- all_mask.extend(mask.iter().map(|&x| x as i64));- }-- let input_ids = Tensor::from_vec(all_ids, (batch_size, seq_len), &self.device)?;- let attention_mask = Tensor::from_vec(all_mask, (batch_size, seq_len), &self.device)?;-- // Forward pass- let logits = self.model.forward(&input_ids, &attention_mask)?;-- // Softmax over last dimension to get probabilities- let probs = candle_nn::ops::softmax(&logits, candle_core::D::Minus1)?;-- // Extract per-byte predictions (skip [CLS] and [SEP])- let mut batch_predictions = Vec::with_capacity(batch_size);-- for i in 0..batch_size {- let window_probs = probs.i((i, 1..seq_len - 1))?; // skip [CLS] at 0, [SEP] at end- let window_probs = window_probs.to_vec2::<f32>()?;-- // Use the tracked actual_len instead of inferring from byte content —- // binary code can legitimately contain 0x00 bytes.- let actual_len = windows[i].1;- let mut preds = Vec::with_capacity(actual_len);-- for j in 0..actual_len {- let p = &window_probs[j];- let (label, confidence) = p- .iter()- .enumerate()- .max_by(|a, b| a.1.partial_cmp(b.1).unwrap())- .map(|(idx, &conf)| (idx as u8, conf))- .unwrap_or((0, 0.0));- preds.push((label, confidence));- }-- batch_predictions.push(preds);- }-- Ok(batch_predictions)- }-- fn merge_predictions(- &self,- batch_predictions: &[Vec<(u8, f32)>],- total_bytes: usize,- base_address: u64,- ) -> Vec<BytePrediction> {- // For overlapping windows, accumulate (label_votes, confidence_sum, count)- let mut votes: Vec<([f32; 3], u32)> = vec![([0.0; 3], 0); total_bytes];-- for (window_idx, preds) in batch_predictions.iter().enumerate() {- let offset = window_idx * self.stride;- for (byte_idx, &(label, confidence)) in preds.iter().enumerate() {- let global_idx = offset + byte_idx;- if global_idx < total_bytes {- votes[global_idx].0[label as usize] += confidence;- votes[global_idx].1 += 1;- }- }- }-- votes- .into_iter()- .enumerate()- .map(|(idx, (scores, count))| {- let count = count.max(1) as f32;- let avg_scores = [- scores[0] / count,- scores[1] / count,- scores[2] / count,- ];- let (label, confidence) = avg_scores- .iter()- .enumerate()- .max_by(|a, b| a.1.partial_cmp(b.1).unwrap())- .map(|(l, &c)| (l as u8, c))- .unwrap_or((0, 0.0));-- BytePrediction {- address: base_address + idx as u64,- label,- confidence,- }- })- .collect()- }-- fn extract_boundaries(&self, predictions: &[BytePrediction]) -> Vec<FunctionBoundary> {- let mut boundaries = Vec::new();- let mut func_start: Option<&BytePrediction> = None;-- for pred in predictions {- match pred.label {- 1 if pred.confidence >= self.confidence_threshold => {- // Close previous function if open- if let Some(start) = func_start.take() {- boundaries.push(FunctionBoundary {- start: start.address,- end: pred.address,- confidence: start.confidence,- source: DetectionSource::Ml,- });- }- func_start = Some(pred);- }- 0 if func_start.is_some() => {- // Non-function byte after function body — close the function- let start = func_start.take().unwrap();- boundaries.push(FunctionBoundary {- start: start.address,- end: pred.address,- confidence: start.confidence,- source: DetectionSource::Ml,- });- }- _ => {}- }- }-- // Close trailing function- if let Some(start) = func_start {- if let Some(last) = predictions.last() {- boundaries.push(FunctionBoundary {- start: start.address,- end: last.address + 1,- confidence: start.confidence,- source: DetectionSource::Ml,- });- }- }-- boundaries- }-}-```--- [ ] **Step 2: Uncomment detector in lib.rs**--- [ ] **Step 3: Verify it compiles**--Run: `cd kong-rs && cargo check -p kong-ml`-Expected: Compiles.--- [ ] **Step 4: Commit**--```bash-git add kong-rs/crates/kong-ml/src/detector.rs kong-rs/crates/kong-ml/src/lib.rs-git commit -m "feat(kong-ml): XDA detector with sliding window and batch inference"-```-------### Task 14: Model Registry (Download, Cache, Load Per-Arch)--**Files:**-- Create: `kong-rs/crates/kong-ml/src/registry.rs`--- [ ] **Step 1: Implement ModelRegistry**--```rust-// kong-rs/crates/kong-ml/src/registry.rs-use std::path::{Path, PathBuf};--use candle_core::{DType, Device};-use candle_nn::VarBuilder;-use kong_types::binary::Arch;--use crate::config::XdaConfig;-use crate::detector::XdaDetector;-use crate::model::XdaModel;--pub struct ModelRegistry {- cache_dir: PathBuf,- confidence_threshold: f32,-}--impl ModelRegistry {- pub fn new(confidence_threshold: f32) -> Self {- let cache_dir = dirs::data_local_dir()- .unwrap_or_else(|| PathBuf::from("."))- .join("kong")- .join("models");-- Self {- cache_dir,- confidence_threshold,- }- }-- pub fn with_cache_dir(cache_dir: PathBuf, confidence_threshold: f32) -> Self {- Self {- cache_dir,- confidence_threshold,- }- }-- /// Check if a model is available locally for the given architecture.- pub fn has_model(&self, arch: Arch) -> bool {- self.weights_path(arch).exists() && self.config_path(arch).exists()- }-- /// List all locally available architectures.- pub fn available_architectures(&self) -> Vec<Arch> {- let all = [- Arch::X86_64,- Arch::X86,- Arch::Aarch64,- Arch::Arm,- Arch::Mips,- Arch::Mips64,- Arch::Riscv64,- Arch::PowerPc,- ];- all.into_iter().filter(|a| self.has_model(*a)).collect()- }-- /// Load a detector for the given architecture.- /// Returns None if no model is available (graceful degradation).- pub fn load_detector(&self, arch: Arch, device: &Device) -> Result<Option<XdaDetector>, Box<dyn std::error::Error>> {- if !self.has_model(arch) {- return Ok(None);- }-- let config = XdaConfig::from_json(&self.config_path(arch))?;- let vb = unsafe {- VarBuilder::from_mmaped_safetensors(- &[self.weights_path(arch)],- DType::F32,- device,- )?- };- let model = XdaModel::load(&config, vb)?;- let detector = XdaDetector::new(model, device.clone(), self.confidence_threshold);-- Ok(Some(detector))- }-- /// Install a model from a local safetensors + config.json pair.- pub fn install(&self, arch: Arch, weights_src: &Path, config_src: &Path) -> Result<(), Box<dyn std::error::Error>> {- std::fs::create_dir_all(&self.cache_dir)?;- std::fs::copy(weights_src, self.weights_path(arch))?;- std::fs::copy(config_src, self.config_path(arch))?;- Ok(())- }-- fn weights_path(&self, arch: Arch) -> PathBuf {- self.cache_dir.join(format!("xda_{}.safetensors", arch.model_name()))- }-- fn config_path(&self, arch: Arch) -> PathBuf {- self.cache_dir.join(format!("xda_{}_config.json", arch.model_name()))- }-}-```--- [ ] **Step 2: Uncomment registry in lib.rs, wire up all modules**--```rust-// kong-rs/crates/kong-ml/src/lib.rs-pub mod config;-pub mod tokenizer;-pub mod model;-pub mod detector;-pub mod registry;-```--- [ ] **Step 3: Verify it compiles**--Run: `cd kong-rs && cargo check -p kong-ml`-Expected: Compiles.--- [ ] **Step 4: Commit**--```bash-git add kong-rs/crates/kong-ml/src/registry.rs kong-rs/crates/kong-ml/src/lib.rs-git commit -m "feat(kong-ml): model registry with per-arch cache and lazy loading"-```-------### Task 15: End-to-End Integration Test--**Files:**-- Create: `kong-rs/tests/xda_integration.rs`--This test validates the full pipeline: tokenize → model → predict. It requires a trained model to exist, so it's gated behind an env var.--- [ ] **Step 1: Write the integration test**--```rust-// kong-rs/tests/xda_integration.rs--/// End-to-end integration test. Requires a trained model.-/// Run with: XDA_MODEL_DIR=/path/to/models cargo test --test xda_integration-#[test]-fn test_full_pipeline() {- let model_dir = match std::env::var("XDA_MODEL_DIR") {- Ok(dir) => std::path::PathBuf::from(dir),- Err(_) => {- eprintln!("Skipping integration test: XDA_MODEL_DIR not set");- return;- }- };-- let device = candle_core::Device::Cpu;- let registry = kong_ml::registry::ModelRegistry::with_cache_dir(model_dir, 0.85);-- // Check x86_64 model exists- let arch = kong_types::binary::Arch::X86_64;- assert!(registry.has_model(arch), "x86_64 model not found in model dir");-- // Load detector- let detector = registry- .load_detector(arch, &device)- .expect("Failed to load model")- .expect("Model should exist");-- // Create a fake ambiguous region with x86_64 function prologue bytes- let prologue = vec![- 0x55, // push rbp- 0x48, 0x89, 0xe5, // mov rbp, rsp- 0x48, 0x83, 0xec, 0x10, // sub rsp, 0x10- 0x89, 0x7d, 0xfc, // mov [rbp-4], edi- 0x8b, 0x45, 0xfc, // mov eax, [rbp-4]- 0x83, 0xc0, 0x01, // add eax, 1- 0xc9, // leave- 0xc3, // ret- // padding / gap- 0xcc, 0xcc, 0xcc, 0xcc, 0xcc, 0xcc, 0xcc, 0xcc, 0xcc, 0xcc, 0xcc, 0xcc,- // second function- 0x55, // push rbp- 0x48, 0x89, 0xe5, // mov rbp, rsp- 0x89, 0x7d, 0xfc, // mov [rbp-4], edi- 0x8b, 0x45, 0xfc, // mov eax, [rbp-4]- 0x0f, 0xaf, 0xc0, // imul eax, eax- 0x5d, // pop rbp- 0xc3, // ret- ];-- let region = kong_types::function::AmbiguousRegion {- start: 0x401000,- bytes: prologue,- reason: kong_types::function::AmbiguityReason::GapRegion,- };-- let boundaries = detector.predict(&[region]).expect("Inference failed");-- // With a well-trained model, we'd expect it to find function starts- // at offsets corresponding to the two push rbp instructions.- // For now, just verify inference runs without error and produces output.- println!("Found {} function boundaries:", boundaries.len());- for b in &boundaries {- println!(- " 0x{:x} - 0x{:x} (confidence: {:.2}, source: {:?})",- b.start, b.end, b.confidence, b.source- );- }-}--/// Smoke test that runs without a model — verifies graceful degradation.-#[test]-fn test_no_model_graceful() {- let tmp = std::env::temp_dir().join("kong_test_empty_models");- let _ = std::fs::create_dir_all(&tmp);-- let registry = kong_ml::registry::ModelRegistry::with_cache_dir(tmp, 0.85);- let device = candle_core::Device::Cpu;-- let detector = registry- .load_detector(kong_types::binary::Arch::X86_64, &device)- .expect("Should not error");-- assert!(detector.is_none(), "Should return None when no model exists");-}-```--- [ ] **Step 2: Verify the no-model test passes**--Run: `cd kong-rs && cargo test --test xda_integration test_no_model_graceful`-Expected: PASS.--- [ ] **Step 3: Commit**--```bash-git add kong-rs/tests/xda_integration.rs-git commit -m "feat(kong-ml): end-to-end integration test for XDA inference pipeline"-```-------## Part C: Training Your First Model (x86_64)--### Task 16: Train x86_64 Model End-to-End--This task uses the scripts from Part A to actually train and export a model.--- [ ] **Step 1: Build x86_64 corpus**--```bash-cd tools/xda-chmod +x scripts/build_corpus.sh-./scripts/build_corpus.sh x86_64 ./corpus/x86_64-```--Expected: ~15 binaries (3 source files x 5 optimization levels) in `corpus/x86_64/`.--- [ ] **Step 2: Build training dataset**--```bash-cd tools/xda-uv run python -c "-from src.build_dataset import build_from_corpus_dir-from pathlib import Path-stats = build_from_corpus_dir(Path('./corpus/x86_64'), Path('./data/x86_64'))-print(f'Binaries: {stats[\"binaries\"]}')-print(f'Functions: {stats[\"functions\"]}')-print(f'Train chunks: {stats[\"train_chunks\"]}')-print(f'Val chunks: {stats[\"val_chunks\"]}')-"-```--Expected: `data/x86_64/train.json`, `data/x86_64/val.json`, `data/x86_64/stats.json` created. Small corpus — maybe 50-200 chunks. Good enough for a smoke test.--- [ ] **Step 3: Fine-tune (skip pre-training for x86_64 — use from scratch for now)**--```bash-cd tools/xda-uv run python -m src.finetune --config configs/finetune.yaml --data ./data/x86_64-```--Expected: Training runs for 5 epochs, prints per-class F1 scores. On this small corpus, results won't be great — that's fine, this validates the pipeline.--- [ ] **Step 4: Export to safetensors**--```bash-cd tools/xda-uv run python -m src.export \- --checkpoint checkpoints/finetune/best.pt \- --config checkpoints/finetune/config.json \- --output ../../kong-rs/models/ \- --arch x86_64-```--Expected: `kong-rs/models/xda_x86_64.safetensors` and `kong-rs/models/xda_x86_64_config.json` created.--- [ ] **Step 5: Run Rust integration test against trained model**--```bash-cd kong-rs-XDA_MODEL_DIR=./models cargo test --test xda_integration test_full_pipeline -- --nocapture-```--Expected: Test passes. Prints function boundaries found (may not be accurate with tiny training set — validates the pipeline works end-to-end).--- [ ] **Step 6: Commit**--```bash-git add kong-rs/models/.gitkeep-echo "*.safetensors" >> kong-rs/.gitignore-echo "*.pt" >> tools/xda/.gitignore-echo "corpus/" >> tools/xda/.gitignore-echo "data/" >> tools/xda/.gitignore-echo "checkpoints/" >> tools/xda/.gitignore-git add kong-rs/.gitignore tools/xda/.gitignore-git commit -m "feat: end-to-end XDA pipeline validated — train in Python, infer in Rust"-```-------## Part D: Scale Up for Production Quality--### Task 17: Build Large Corpus with Debian Packages (AArch64)--This repeats the pattern for a new architecture.--- [ ] **Step 1: Build and run the Debian corpus Docker image**--```bash-cd tools/xda-docker build --platform linux/arm64 -t xda-corpus-aarch64 -f scripts/Dockerfile.corpus .-docker run --platform linux/arm64 -v $(pwd)/corpus/aarch64_debian:/out xda-corpus-aarch64-```--Expected: `corpus/aarch64_debian/pairs/` with 50-200+ binary/debug pairs.--- [ ] **Step 2: Also cross-compile custom corpus for AArch64**--```bash-# Requires: brew install aarch64-linux-gnu-gcc (or via apt on Linux)-cd tools/xda-./scripts/build_corpus.sh aarch64 ./corpus/aarch64_custom-```--- [ ] **Step 3: Build dataset from both sources**--```bash-cd tools/xda-uv run python -c "-from src.build_dataset import build_from_pairs_dir, build_from_corpus_dir-from pathlib import Path-import json--# Merge both corpus sources-all_chunks = []--# Debian packages-stats1 = build_from_pairs_dir(Path('./corpus/aarch64_debian/pairs'), Path('./data/aarch64_tmp1'))-t1 = json.loads(Path('./data/aarch64_tmp1/train.json').read_text())-v1 = json.loads(Path('./data/aarch64_tmp1/val.json').read_text())--# Cross-compiled-stats2 = build_from_corpus_dir(Path('./corpus/aarch64_custom'), Path('./data/aarch64_tmp2'))-t2 = json.loads(Path('./data/aarch64_tmp2/train.json').read_text())-v2 = json.loads(Path('./data/aarch64_tmp2/val.json').read_text())--# Merge-import random-train = t1 + t2; random.shuffle(train)-val = v1 + v2; random.shuffle(val)--out = Path('./data/aarch64'); out.mkdir(exist_ok=True)-(out / 'train.json').write_text(json.dumps(train))-(out / 'val.json').write_text(json.dumps(val))-print(f'Total train: {len(train)}, val: {len(val)}')-"-```--- [ ] **Step 4: Pre-train on AArch64 bytes (optional but recommended)**--```bash-cd tools/xda-uv run python -m src.pretrain --config configs/pretrain.yaml --data ./data/aarch64-```--Expected: Runs for 10 epochs. May take several hours depending on corpus size and hardware.--- [ ] **Step 5: Fine-tune on AArch64**--Edit `configs/finetune.yaml` to point `pretrained_checkpoint` at the pre-trained checkpoint:-```yaml-pretrained_checkpoint: "checkpoints/pretrain/pretrain_final.pt"-```--```bash-cd tools/xda-uv run python -m src.finetune --config configs/finetune.yaml --data ./data/aarch64-```--- [ ] **Step 6: Export AArch64 model**--```bash-cd tools/xda-uv run python -m src.export \- --checkpoint checkpoints/finetune/best.pt \- --config checkpoints/finetune/config.json \- --output ../../kong-rs/models/ \- --arch aarch64-```--- [ ] **Step 7: Verify in Rust**--```bash-cd kong-rs-XDA_MODEL_DIR=./models cargo test --test xda_integration -- --nocapture-```--- [ ] **Step 8: Commit**--```bash-git commit -m "feat(xda): trained and validated AArch64 function boundary model"-```-------## Summary: What Each Task Produces--| Task | Deliverable |-|------|-------------|-| 1 | Python project + byte tokenizer |-| 2 | DWARF ground truth extractor |-| 3 | Chunked dataset pipeline |-| 4 | XDA model definition (Python) |-| 5 | Corpus build scripts (Debian + cross-compile) |-| 6 | End-to-end dataset builder |-| 7 | Pre-training script (masked byte modeling) |-| 8 | Fine-tuning script (3-class boundary detection) |-| 9 | Safetensors export for Rust |-| 10 | Cargo workspace + kong-types crate |-| 11 | Rust byte tokenizer + config loading |-| 12 | XDA model in candle (BERT + MLP) |-| 13 | Sliding window inference + batch prediction |-| 14 | Per-arch model registry with caching |-| 15 | End-to-end Rust integration test |-| 16 | First trained model (x86_64, small corpus) |-| 17 | Production AArch64 model (large corpus) |--**Tasks 1-9** (Python) and **Tasks 10-15** (Rust) are independent workstreams that can be parallelized. Task 16 bridges them. Task 17 is the repeatable pattern for new architectures.