Skip to content

armgit / LEANN

Public Imported from GitHub
0

[MLsys2026 Best Paper]: https://arxiv.org/abs/2506.08276. RAG on Everything with LEANN. Enjoy 97% storage savings while running a fast, accurate, and 100% private RAG application on your personal device.

fix: force fp32 embeddings when the resolved device is CPU (#405)

Christoph Stadlbauer9cbe7c2a3b2b
diff --git a/packages/leann-core/src/leann/embedding_compute.py b/packages/leann-core/src/leann/embedding_compute.py
index 6ff5d8b..c7125d7 100644
--- a/packages/leann-core/src/leann/embedding_compute.py
+++ b/packages/leann-core/src/leann/embedding_compute.py
@@ -546,6 +546,12 @@ def compute_embeddings_sentence_transformers(
else:
device = "cpu"
+ # CPU torch ships no fp16 kernels for several ops (LayerNorm raises
+ # "LayerNormKernelImpl" not implemented for 'Half') - force fp32 on CPU.
+ if device == "cpu" and use_fp16:
+ use_fp16 = False
+ logger.info("CPU device detected: disabling fp16 (no fp16 CPU kernels)")
+
# Apply optimizations based on benchmark results
if adaptive_optimization:
batch_size = _resolve_adaptive_batch_size(device, model_name)