💡 All-in-one AI framework for semantic search, LLM orchestration and language model workflows
Python agentsaiai-agentsembeddingsinformation-retrievallanguage-modellarge-language-modelsllmnlppythonragretrieval-augmented-generationsearchsearch-enginesemantic-searchsentence-embeddingstransformerstxtaivector-databasevector-search
Fix LEMUR collection centering
morgan-coded00ec81a10e7a
diff --git a/docs/embeddings/configuration/vectors.md b/docs/embeddings/configuration/vectors.mdindex a5b4c72e..c1c4e327 100644--- a/docs/embeddings/configuration/vectors.md+++ b/docs/embeddings/configuration/vectors.md@@ -150,9 +150,10 @@ center: boolean | dict```Centers normalized late-interaction token vectors and normalizes them again before MUVERA or LEMUR encoding. Centering is applied to-both query and document vectors. `true` uses batch scope and `false` disables centering. When omitted, centering defaults to batch-scope only for late-interaction models that load more than one `torch.nn.Linear` layer; models with zero or one loaded linear layer keep-their previous output.+both query and document vectors. `true` uses batch scope and `false` disables centering. An explicit setting always takes precedence.+When omitted, a LEMUR artifact with a stored collection mean uses collection scope. Other LEMUR artifacts and late-interaction models+default to batch scope only when they load more than one `torch.nn.Linear` layer; models with zero or one loaded linear layer keep their+previous output.Dictionary configuration supports the following scopes:@@ -162,8 +163,9 @@ center:````document` subtracts the mean of each document or query independently. `batch` subtracts the mean of all real token rows in the current-model batch. `collection` requires a caller-provided, one-dimensional mean vector through either `mean` (an array-like value) or `path`-(a safetensors file containing a `center.mean` tensor), but not both. Zero-padding rows are never centered and remain zero.+model batch. Explicit `collection` scope requires a caller-provided, one-dimensional mean vector through either `mean` (an array-like+value) or `path` (a safetensors file containing a `center.mean` tensor), but not both. LEMUR artifacts can provide this mean+automatically. Zero-padding rows are never centered and remain zero.### muvera```yaml@@ -184,8 +186,9 @@ lemur:Loads a trained [LEMUR](../../../pipeline/train/lemur) fixed dimensional encoder. LEMUR artifacts are corpus-specific and must becreated with `LemurTrainer` before configuring an embeddings index. Query vectors are summed learned features and document vectorsare ordinary least squares weights over the artifact's stored token sample. Each artifact contains `config.json` and-`model.safetensors`; the Safetensors file contains inference state only. See the trainer page for score filtering and Faiss exact/IVF-search behavior.+`model.safetensors`; newly trained artifacts also store the collection token mean as `lemur.center`. Loading that mean automatically+selects collection centering unless `center` is explicitly configured. Artifacts without `lemur.center` retain the previous default.+The Safetensors file contains inference state only. See the trainer page for score filtering and Faiss exact/IVF search behavior.### trust_remote_code```yamldiff --git a/docs/pipeline/train/lemur.md b/docs/pipeline/train/lemur.mdindex a14c582d..d9e222a3 100644--- a/docs/pipeline/train/lemur.md+++ b/docs/pipeline/train/lemur.md@@ -37,6 +37,10 @@ The learn distribution defaults to `learncategory="query"`. Target documents aredefault encodes selected corpus texts once as data and once as queries. Set `learncategory="data"` to reuse the data encodings, orpass a separate iterable of texts with `learn`.+When `vectors.center` is omitted, the trainer encodes the corpus without centering, computes one mean over all corpus token rows and+uses that mean to center both data and learn vectors before fitting. The collection mean is stored in the LEMUR artifact. An explicit+`vectors.center` setting keeps its configured behavior and does not store an automatic collection mean.+Set `corpussubsetsize` to a positive integer to sample that many raw corpus texts under `seed` before either encoding pass. Thedefault is `None`, which uses every corpus text. `trainsubsetsize` and `learnsubsetsize` apply later, after token vectors have alreadybeen created. `corpussubsetsize` applies to `data`; a separate `learn` iterable remains caller-sized.@@ -48,8 +52,9 @@ available as `selectedepoch`, `selectedloss` and `selectionmetric` on the fittedMLP training displays a `tqdm` progress bar on an interactive terminal, including percent complete and the current validation losswhen `validationsplit` is enabled (or training loss otherwise). Progress output is disabled for non-interactive runs.-The artifact contains `config.json` and `model.safetensors`. It stores the inference feature model, output-normalization statistics-and token sample needed to encode documents added after training. The training-only output readout is not saved.+The artifact contains `config.json` and `model.safetensors`. It stores the inference feature model, output-normalization statistics,+token sample needed to encode documents added after training and, by default, the collection token mean. The training-only output+readout is not saved.Load the artifact in an embeddings configuration.@@ -63,6 +68,11 @@ embeddings:## Search behavior+Loading a LEMUR artifact with a stored collection mean automatically centers both query and document token vectors with that mean.+This keeps training and search on the same representation and makes fixed vectors independent of indexing or query batch size. An+explicit `vectors.center` setting takes precedence. Older artifacts without a stored mean use the standard late-pooling fallback:+batch centering for models with multiple linear layers and no centering for models with zero or one linear layer.+LEMUR approximates standardized MaxSim targets, so useful ranking scores can be negative. txtai's dense vector path L2-normalizes thefixed vectors and removes results with scores less than or equal to zero. This can change ordering and return fewer than the requestednumber of candidates compared with raw maximum inner-product search.diff --git a/src/python/txtai/models/pooling/late.py b/src/python/txtai/models/pooling/late.pyindex d8f7b906..f0a75b61 100644--- a/src/python/txtai/models/pooling/late.py+++ b/src/python/txtai/models/pooling/late.py@@ -162,7 +162,10 @@ class LatePooling(Pooling):"""# Enable batch centering by default for multi-linear models- if not configured:+ artifactcenter = getattr(self.encoder, "center", None)+ if not configured and artifactcenter is not None:+ center = {"scope": "collection", "mean": artifactcenter.detach().cpu().numpy()}+ elif not configured:center = sum(isinstance(module, torch.nn.Linear) for module in linear.modules()) > 1if isinstance(center, bool):diff --git a/src/python/txtai/models/pooling/lemur.py b/src/python/txtai/models/pooling/lemur.pyindex 76f4b68e..15558866 100644--- a/src/python/txtai/models/pooling/lemur.py+++ b/src/python/txtai/models/pooling/lemur.py@@ -45,6 +45,7 @@ class Lemur:self.sample = Noneself.mean = Noneself.std = None+ self.center = Noneself.config = Noneself.pinv = Noneself.selectedepoch = None@@ -190,6 +191,8 @@ class Lemur:"lemur.sample": self.sample.detach().cpu().contiguous(),})+ if self.center is not None:+ tensors["lemur.center"] = self.center.detach().cpu().contiguous()safetensors.torch.save_file(tensors, os.path.join(path, "model.safetensors"))def load(self, path):@@ -217,6 +220,7 @@ class Lemur:self.mean = tensors.pop("lemur.mean")[0]self.std = tensors.pop("lemur.std")[0]self.sample = tensors.pop("lemur.sample")+ self.center = tensors.pop("lemur.center", None)self.model.load_state_dict(tensors)self.model.eval()self.pinv = Nonediff --git a/src/python/txtai/pipeline/train/lemur.py b/src/python/txtai/pipeline/train/lemur.pyindex 3444f05f..f85d99b0 100644--- a/src/python/txtai/pipeline/train/lemur.py+++ b/src/python/txtai/pipeline/train/lemur.py@@ -12,6 +12,7 @@ from ..base import Pipelinelibrary = Library()+np = library.numpy()torch = library.torch()tqdm = library.tqdm()@@ -80,6 +81,9 @@ class LemurTrainer(Pipeline):deviceid = Models.deviceid(gpu)modelargs = {**(vectors if vectors else {}), **{"muvera": None, "lemur": None}}+ centerconfigured = vectors is not None and "center" in vectors+ if not centerconfigured:+ modelargs["center"] = Falsepooling = PoolingFactory.create({"method": method,@@ -99,12 +103,21 @@ class LemurTrainer(Pipeline):learn = data if learn is None else learnlearndocuments = [pooling.encode([text], batch=1, category=learncategory)[0] for text in learn]+ centermean = None+ if not centerconfigured:+ centermean = np.concatenate(documents).mean(axis=0)+ pooling.center = {"scope": "collection", "mean": centermean}+ documents = pooling.centerdata(documents)+ if learndocuments is not None:+ learndocuments = pooling.centerdata(learndocuments)+return self.fit(documents,output=output,device=Models.device(deviceid),validationsplit=validationsplit,learn=learndocuments,+ centermean=centermean,**kwargs,)@@ -129,6 +142,7 @@ class LemurTrainer(Pipeline):validationsplit=0.0,learn=None,device=None,+ centermean=None,):"""Fits a LEMUR feature encoder and OLS sample.@@ -152,6 +166,7 @@ class LemurTrainer(Pipeline):validationsplit: fraction of sampled learn tokens held out for validationlearn: optional iterable of learn-token arrays, defaults to datadevice: tensor device+ centermean: optional collection token mean stored with the artifactReturns:fitted LEMUR encoder@@ -188,6 +203,12 @@ class LemurTrainer(Pipeline):generator,)+ if centermean is not None:+ centermean = torch.as_tensor(centermean, dtype=torch.float32, device=lemur.device)+ if centermean.ndim != 1 or centermean.shape[0] != training["dimension"] or not torch.isfinite(centermean).all():+ raise ValueError("centermean must be a finite one-dimensional array matching the token dimension")+ lemur.center = centermean.detach()+modeltype = "elm" if epochs == 0 else "mlp"config = {"modeltype": modeltype,diff --git a/test/python/testmodels/testpooling.py b/test/python/testmodels/testpooling.pyindex 3662f706..241f3293 100644--- a/test/python/testmodels/testpooling.py+++ b/test/python/testmodels/testpooling.py@@ -316,6 +316,106 @@ class TestPooling(unittest.TestCase):np.testing.assert_array_equal(pooling.encode(texts, batch=2, category="query"), queries)np.testing.assert_array_equal(pooling.encode(texts, batch=2, category="data"), documents)+ def testLemurCollectionCenter(self):+ """+ Test LEMUR artifact collection centering is batch-independent+ """++ corpus = [+ "Machine learning models retrieve relevant passages.",+ "Late interaction compares token embeddings.",+ "Dense indexes search fixed dimensional vectors.",+ "A query encoder produces contextual token vectors.",+ "Document encoders represent passages for retrieval.",+ "Maximum similarity aggregates token matches.",+ "LEMUR learns a corpus specific reduction.",+ "MUVERA uses randomized fixed dimensional encodings.",+ "The trainer stores reusable pooling artifacts.",+ "New documents can be encoded after training.",+ "Short text.",+ "A considerably longer synthetic document exercises padding behavior.",+ ]+ texts = corpus[-2:]++ def loadlinear(*_):+ layers = [torch.nn.Linear(128, 128, bias=False) for _ in range(2)]+ with torch.no_grad():+ for layer in layers:+ layer.weight.copy_(torch.eye(128))+ return torch.nn.Sequential(*layers)++ settings = {+ "gpu": False,+ "epochs": 0,+ "finalhiddendim": 32,+ "trainsubsetsize": 12,+ "learnsubsetsize": 128,+ "olssamplesize": 64,+ "seed": 42,+ }++ with (+ tempfile.TemporaryDirectory() as output,+ tempfile.TemporaryDirectory() as legacy,+ patch.object(LatePooling, "loadlinear", autospec=True, side_effect=loadlinear),+ ):+ raw = PoolingFactory.create(+ {+ "path": "neuml/colbert-bert-tiny",+ "device": Models.deviceid(False),+ "modelargs": {"muvera": None, "lemur": None, "center": False},+ }+ )+ documents = [raw.encode([text], batch=1, category="data")[0] for text in corpus]+ center = np.concatenate(documents).mean(axis=0)++ LemurTrainer()(+ "neuml/colbert-bert-tiny",+ corpus,+ output,+ **settings,+ )+ LemurTrainer()("neuml/colbert-bert-tiny", corpus, legacy, vectors={"center": False}, **settings)++ pooling = PoolingFactory.create(+ {+ "path": "neuml/colbert-bert-tiny",+ "device": Models.deviceid(False),+ "modelargs": {"lemur": {"path": output}},+ }+ )+ self.assertEqual(pooling.center["scope"], "collection")+ np.testing.assert_array_equal(pooling.center["mean"], center)+ np.testing.assert_array_equal(pooling.encoder.center.cpu().numpy(), center)++ separate = pooling.encode(texts, batch=1, category="data")+ together = pooling.encode(texts, batch=32, category="data")+ np.testing.assert_allclose(together, separate, rtol=1e-4, atol=1e-5)++ separate = pooling.encode(texts, batch=1, category="query")+ together = pooling.encode(texts, batch=32, category="query")+ np.testing.assert_allclose(together, separate, rtol=1e-4, atol=1e-5)++ disabled = PoolingFactory.create(+ {+ "path": "neuml/colbert-bert-tiny",+ "device": Models.deviceid(False),+ "modelargs": {"lemur": {"path": output}, "center": False},+ }+ )+ self.assertIsNone(disabled.center)+ self.assertIsNotNone(disabled.encoder.center)++ compatible = PoolingFactory.create(+ {+ "path": "neuml/colbert-bert-tiny",+ "device": Models.deviceid(False),+ "modelargs": {"lemur": {"path": legacy}},+ }+ )+ self.assertEqual(compatible.center, {"scope": "batch"})+ self.assertIsNone(compatible.encoder.center)+def testLemurDocuments(self):"""Test LEMUR document input conversions and validationdiff --git a/test/python/testpipeline/testtrain/testtrainer.py b/test/python/testpipeline/testtrain/testtrainer.pyindex d4be33da..b21c60d5 100644--- a/test/python/testpipeline/testtrain/testtrainer.py+++ b/test/python/testpipeline/testtrain/testtrainer.py@@ -292,6 +292,14 @@ class TestTrainer(unittest.TestCase):self.calls.append((texts[0], category, batch))return [torch.ones((1, 2))]+ @staticmethod+ def centerdata(data):+ """+ Returns synthetic token vectors after the centering seam.+ """++ return data+corpus = [f"document {index}" for index in range(12)]runs = []for _ in range(2):