Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

5.1 KB
Raw Blame
"""Deterministic, dependency-free demo embedder for the shortlist golden vectors and samples.

This is **not** a real embedding model: it is a hashed character-trigram bag-of-features
vector, chosen only because it is a pure function of the input text and is trivial to
reimplement bit-for-bit in another language with nothing but a string, UTF-8 encoding and
32-bit integer arithmetic. It is deliberately named and documented as a demo: any real
deployment should pass `embed_fn_from_agent` (Python) or a real bi-encoder to the shortlist,
not this.

Every other-language port MUST reproduce this exact algorithm, because
`hashing_embedder_probe.json` and `case_shortlist_many_options.json` in the golden vectors
were both generated with it, and the samples in every language use it so that their
stdout can be diffed line for line.

Algorithm (this docstring is the spec -- read it precisely before porting):

1. Lowercase the input with a full Unicode case mapping (Python's `str.lower()`; other
   languages must use their own full Unicode case-folding function on the *string*, NOT a
   per-UTF-16-unit simple mapping. This matters for characters whose lowercase form is
   more than one code point, e.g. `"İ".lower() == "i̇"` (Turkish capital I with dot
   above lowercases to 2 code points: 'i' + combining dot above).
2. Pad the lowercased text with two ASCII space characters (U+0020) on each side:
   `"  " + text + "  "`. Padding is added even when `text` is empty, so the padded text is
   never shorter than 4 code points and there are always at least 2 windows (see step 3):
   this is what lets short inputs like `""` or `"a"` still produce a nonzero, well-defined
   vector instead of an empty one.
3. Slide a window of exactly 3 **Unicode code points** (not UTF-16 code units, not grapheme
   clusters, not bytes) across the padded text, one code point at a time. For padded text of
   length `n` code points there are `n - 2` windows, at start offsets `0, 1, ..., n - 3`
   inclusive. A code point outside the Basic Multilingual Plane (an "astral" character, e.g.
   an emoji) counts as exactly one code point in this count -- a port that iterates UTF-16
   code units instead of Unicode scalar values will split a surrogate pair into two units and
   get a different window count and different window contents; it must enumerate Unicode
   scalar values instead. Walking grapheme clusters is NOT correct either -- it must be
   rune/code-point enumeration (Rust's `char` already IS a Unicode scalar value, so iterating
   `str::chars()` is correct there).
4. Encode each 3-code-point window as UTF-8 bytes (the window's own bytes, not the whole
   padded text's bytes -- re-encode the 3-code-point substring on its own each time).
5. Hash those UTF-8 bytes with 32-bit FNV-1a:

       hash = 0x811c9dc5                      # FNV offset basis
       for byte in utf8_bytes:
           hash ^= byte
           hash = (hash * 0x01000193) & 0xFFFFFFFF   # FNV prime, wrapped to 32 bits

   `hash` is an unsigned 32-bit integer throughout; the multiply must wrap (mask to
   0xFFFFFFFF / use unchecked/wrapping arithmetic in the target language) exactly like
   Python's explicit `& 0xFFFFFFFF` above.
6. `bucket = hash % 64` (0..63, unsigned).
7. `sign = +1.0 if (hash >> 31) & 1 == 0 else -1.0` (the hash's top bit, as an unsigned
   32-bit value -- this is a sign-hashing trick to reduce feature collisions, not a
   statement about the hash being negative).
8. Accumulate `sign` into `vector[bucket]` for every window (i.e. `vector[bucket] += sign`;
   multiple windows can and do land in the same bucket and their signed contributions add).
   The accumulator is float64 throughout.
9. No normalisation is applied. The shortlist's cosine similarity handles scale on its own,
   and leaving the raw counts in the golden vectors makes them easier to hand-check.

`embed(texts)` returns one such vector per input string, as a `(len(texts), 64)` float64
NumPy array, in input order. It does not batch, cache, or special-case any input; every text
is processed independently and only via the steps above.
"""
from typing import Sequence

import numpy as np

DIM = 64
_FNV_OFFSET_BASIS = 0x811C9DC5
_FNV_PRIME = 0x01000193
_MASK32 = 0xFFFFFFFF


def _fnv1a_32(data: bytes) -> int:
    """32-bit FNV-1a over raw bytes, wrapping like an unsigned 32-bit integer."""
    h = _FNV_OFFSET_BASIS
    for b in data:
        h ^= b
        h = (h * _FNV_PRIME) & _MASK32
    return h


def _embed_one(text: str) -> np.ndarray:
    vec = np.zeros(DIM, dtype=np.float64)
    padded = "  " + text.lower() + "  "
    n = len(padded)
    for i in range(n - 2):
        window = padded[i : i + 3]
        h = _fnv1a_32(window.encode("utf-8"))
        bucket = h % DIM
        sign = 1.0 if (h >> 31) & 1 == 0 else -1.0
        vec[bucket] += sign
    return vec


def embed(texts: Sequence[str]) -> np.ndarray:
    """Embed a list of strings into a `(len(texts), 64)` float64 array. See the module docstring
    for the exact algorithm; every other-language port must match it bit-for-bit."""
    rows = [_embed_one("" if t is None else str(t)) for t in texts]
    if not rows:
        return np.zeros((0, DIM), dtype=np.float64)
    return np.stack(rows, axis=0)