Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

3.0 KB
Raw Blame
//! **Demo only — replace with a real embedding model.** Deterministic, dependency-free stand-in
//! embedder used only so this sample can call [`laya::LayaShortlist::predict`] without a network
//! call or an ONNX bi-encoder. Ports `tools/hashing_embedder.py` bit-for-bit (the same port also
//! lives at `tests/common/hashing_embedder.rs` for the test suite) — see that file's
//! docstring for the exact algorithm spec this follows step by step. It is a hashed
//! character-trigram bag-of-features vector, chosen only because it is a pure function of the
//! input text that is trivial to reproduce identically in every language and every SDK.
//!
//! Every step below is precision-sensitive:
//! - Lowercasing uses full Unicode case mapping (`str::to_lowercase`), matching Python's
//!   `str.lower()` — not a per-UTF-16-unit simple mapping.
//! - The 3-item sliding window is over Unicode scalar values (`char`, which in Rust already *is*
//!   a Unicode scalar value), matching Python's per-code-point iteration; an astral character
//!   (outside the BMP) is one window position, not two.
//! - The window is re-encoded to UTF-8 on its own each time (not sliced out of one pre-encoded
//!   byte buffer), because a multi-byte character's encoding does not change with position, but
//!   slicing a shared buffer at the wrong byte offset would corrupt it.
//! - The FNV-1a multiply wraps at 32 bits (`u32::wrapping_mul`), matching Python's explicit
//!   `& 0xFFFFFFFF`.

use laya::error::Result as LayaResult;

/// Output vector width.
pub const DIM: usize = 64;

const FNV_OFFSET_BASIS: u32 = 0x811c_9dc5;
const FNV_PRIME: u32 = 0x0100_0193;

/// 32-bit FNV-1a over raw bytes, wrapping like an unsigned 32-bit integer.
fn fnv1a_32(data: &[u8]) -> u32 {
    let mut hash = FNV_OFFSET_BASIS;
    for &byte in data {
        hash ^= u32::from(byte);
        hash = hash.wrapping_mul(FNV_PRIME);
    }
    hash
}

fn embed_one(text: &str) -> Vec<f64> {
    let mut vector = vec![0.0f64; DIM];
    let padded: Vec<char> = format!("  {}  ", text.to_lowercase()).chars().collect();
    let n = padded.len();
    // Padding alone is 4 characters, so `n >= 4` always and this subtraction never underflows;
    // the guard is defensive documentation, not a reachable branch.
    if n < 3 {
        return vector;
    }
    for start in 0..=(n - 3) {
        let window: String = padded[start..start + 3].iter().collect();
        let hash = fnv1a_32(window.as_bytes());
        let bucket = (hash % DIM as u32) as usize;
        let sign = if (hash >> 31) & 1 == 0 { 1.0 } else { -1.0 };
        vector[bucket] += sign;
    }
    vector
}

/// Embed a list of strings into one `(len(texts), 64)` matrix, in input order. See the module
/// docs for the exact algorithm.
///
/// Returns [`laya::error::Result`] so it can be used directly wherever [`laya::LayaShortlist`]
/// expects an `embed_fn` closure, even though this particular embedder never fails.
pub fn embed(texts: &[String]) -> LayaResult<Vec<Vec<f64>>> {
    Ok(texts.iter().map(|t| embed_one(t)).collect())
}