DSPy: The framework for programming—not prompting—language models
Python
lancedb integration
PrashantDixit-devbb4a6e4f4782
diff --git a/README.md b/README.mdindex b840d508f..da5ddcb6a 100644--- a/README.md+++ b/README.md@@ -79,10 +79,10 @@ Or open our intro notebook in Google Colab: [<img align="center" src="https://coBy default, DSPy installs the latest `openai` from pip. However, if you install old version before OpenAI changed their API `openai~=0.28.1`, the library will use that just fine. Both are supported.-For the optional (alphabetically sorted) [Chromadb](https://github.com/chroma-core/chroma), [Groq](https://github.com/groq/groq-python), [Marqo](https://github.com/marqo-ai/marqo), [Milvus](https://github.com/milvus-io/milvus), [MongoDB](https://www.mongodb.com), [MyScaleDB](https://github.com/myscale/myscaledb), Pinecone, [Qdrant](https://github.com/qdrant/qdrant), [Snowflake](https://github.com/snowflakedb/snowpark-python), or [Weaviate](https://github.com/weaviate/weaviate) retrieval integration(s), include the extra(s) below:+For the optional (alphabetically sorted) [Chromadb](https://github.com/chroma-core/chroma), [LanceDB](https://github.com/lancedb/lancedb), [Groq](https://github.com/groq/groq-python), [Marqo](https://github.com/marqo-ai/marqo), [Milvus](https://github.com/milvus-io/milvus), [MongoDB](https://www.mongodb.com), [MyScaleDB](https://github.com/myscale/myscaledb), Pinecone, [Qdrant](https://github.com/qdrant/qdrant), [Snowflake](https://github.com/snowflakedb/snowpark-python), or [Weaviate](https://github.com/weaviate/weaviate) retrieval integration(s), include the extra(s) below:```-pip install dspy-ai[chromadb] # or [groq] or [marqo] or [milvus] or [mongodb] or [myscale] or [pinecone] or [qdrant] or [snowflake] or [weaviate]+pip install dspy-ai[chromadb] # or [lancedb] or [groq] or [marqo] or [milvus] or [mongodb] or [myscale] or [pinecone] or [qdrant] or [snowflake] or [weaviate]```## 2) Documentationdiff --git a/docs/docs/deep-dive/retrieval_models_clients/LancedbRM.mdx b/docs/docs/deep-dive/retrieval_models_clients/LancedbRM.mdxnew file mode 100644index 000000000..51b8b33c7--- /dev/null+++ b/docs/docs/deep-dive/retrieval_models_clients/LancedbRM.mdx@@ -0,0 +1,72 @@+---+sidebar_position: 1+---++# LancedbRM++[LanceDB](http://lancedb.com/) is a developer-friendly, open source database for AI. From hyper scalable vector search and advanced retrieval for RAG, to streaming training data and interactive exploration of large scale AI datasets, LanceDB is the best foundation for your AI application.++## Setting Up LancedbRM++`LancedbRM` can be instantiated with any custom vectorizer and configured to return any payload field.++- `table_name (str)`: The name of the table to query against.+- `persist_directory (str)`: directory where database is stored.+- `k (int, optional)`: The number of top passages to retrieve. Defaults to 3.++## Under the Hood++### `forward(self, query_or_queries: Union[str, list[str]], k: Optional[int] = None) -> dspy.Prediction`++**Parameters:**++- `query_or_queries (Union[str, List[str]])`: The query or queries to search for.+- `k (Optional[int])`: The number of top passages to retrieve. Defaults to `self.k`.++**Returns:**++- `dspy.Prediction`: Contains the retrieved passages, each represented as a `dotdict` with a `long_text` attribute.++## Example Usage++```python+import os+import pandas as pd++import dspy+from dspy.retrieve.lancedb import LancedbRM++from lancedb.pydantic import LanceModel, Vector+from lancedb.embeddings import get_registry++from datasets import load_dataset++# load_dataset+ds = load_dataset("fancyzhx/dbpedia_14")+df = pd.DataFrame(ds['train'])++os.environ["OPENAI_API_KEY"] = "<YOUR_OPENAI_API_KEY>"++uri = 'tmp/db'+table_name = 'passages'+db = lancedb.connect(uri)+model = get_registry().get("sentence-transformers").create(name="BAAI/bge-small-en-v1.5", device="cpu")++class Passages(LanceModel):+ text: str = model.SourceField()+ vector: Vector(model.ndims()) = model.VectorField()++table = db.create_table(table_name, schema=Passages)+table.add(df['content'])+++lancedb_retriever = LancedbRM(+ table_name=table_name,+ persist_directory=uri,+)++dspy.settings.configure(rm=lancedb_retriever)+retrieve = dspy.Retrieve()++retrieve("Integrated Circuits")+```diff --git a/docs/docs/deep-dive/retrieval_models_clients/custom-rm-client.mdx b/docs/docs/deep-dive/retrieval_models_clients/custom-rm-client.mdxindex 927288135..91fef7420 100644--- a/docs/docs/deep-dive/retrieval_models_clients/custom-rm-client.mdx+++ b/docs/docs/deep-dive/retrieval_models_clients/custom-rm-client.mdx@@ -6,7 +6,7 @@ import AuthorDetails from '@site/src/components/AuthorDetails';# Creating Custom RM Client-DSPy provides support for various retrieval modules out of the box like `ColBERTv2`, `AzureCognitiveSearch`, `Pinecone`, `Weaviate`, etc. Unlike Language Model (LM) modules, creating a custom RM module is much more simple and flexible.+DSPy provides support for various retrieval modules out of the box like `ColBERTv2`, `AzureCognitiveSearch`, `Lancedb`, `Pinecone`, `Weaviate`, etc. Unlike Language Model (LM) modules, creating a custom RM module is much more simple and flexible.As of now, DSPy offers 2 ways to create a custom RM: the Pythonic way and the DSPythonic way. We'll take a look at both, understand why both are performing the same behavior, and how you can implement each!@@ -152,7 +152,7 @@ class DSPythonicRMClient(dspy.Retrieve):)```-That's all!! This is the way to implement a custom RM model client within DSPy and how more recently-supported RM models like `QdrantRM`, `WeaviateRM`, etc. are implemented in DSPy.+That's all!! This is the way to implement a custom RM model client within DSPy and how more recently-supported RM models like `QdrantRM`, `WeaviateRM`, `LancedbRM`, etc. are implemented in DSPy.Let's take a look at how we use these retrievers.diff --git a/docs/docs/quick-start/installation.mdx b/docs/docs/quick-start/installation.mdxindex 8e2f81ef8..a1eca2d39 100644--- a/docs/docs/quick-start/installation.mdx+++ b/docs/docs/quick-start/installation.mdx@@ -15,7 +15,7 @@ Or open our intro notebook in Google Colab: [<img align="center" src="https://coBy default, DSPy depends on `openai==0.28`. However, if you install `openai>=1.0`, the library will use that just fine. Both are supported.-For the optional Pinecone, Qdrant, ChromaDB, Marqo, or Milvus retrieval integration(s), include the extra(s) below:+For the optional LanceDB, Pinecone, Qdrant, ChromaDB, Marqo, or Milvus retrieval integration(s), include the extra(s) below:import Tabs from '@theme/Tabs';import TabItem from '@theme/TabItem';@@ -28,6 +28,11 @@ import TabItem from '@theme/TabItem';pip install dspy-ai```</TabItem>+ <TabItem value="lancedb" label="LanceDB">+ ```text+ pip install "dspy-ai[lancedb]"+ ```+ </TabItem><TabItem value="pinecone" label="Pinecone">```textpip install "dspy-ai[pinecone]"diff --git a/dspy/retrieve/lancedb_rm.py b/dspy/retrieve/lancedb_rm.pynew file mode 100644index 000000000..aa0b9a1a0--- /dev/null+++ b/dspy/retrieve/lancedb_rm.py@@ -0,0 +1,142 @@+"""+Retriever model for LanceDB+Author: Prashant Dixit (@PrashantDixit0)+"""++from typing import List, Optional, Union++import backoff+import lancedb.table++from dspy import Retrieve, Prediction+from dsp.utils.settings import settings+from dsp.utils import dotdict++try:+ import lancedb+except ImportError:+ lancedb = None++if lancedb is None:+ raise ImportError(+ "The lancedb library is required to use LancedbRM. Install it with `pip install dspy-ai[lancedb]`",+ )++from lancedb.embeddings import EmbeddingFunctionRegistry, get_registry+from lancedb.pydantic import LanceModel, Vector++import openai+import os++try:+ OPENAI_LEGACY = int(openai.version.__version__[0]) == 0+except Exception:+ OPENAI_LEGACY = True++try:+ import openai.error+ ERRORS = (openai.error.RateLimitError, openai.error.ServiceUnavailableError, openai.error.APIError)+except Exception:+ ERRORS = (openai.RateLimitError, openai.APIError)+++class LancedbRM(Retrieve):+ """+ A retrieval module that uses LanceDB to return the top passages for a given query.++ Assumes that the LanceDB table has been created and populated with the following metadata:+ - text: The text of the passage++ Args:+ table_name (str): The name of the table to query against.+ persist_directory (str): directory where database is stored.+ k (int, optional): The number of top passages to retrieve. Defaults to 3.++ Returns:+ dspy.Prediction: An object containing the retrieved passages.++ Examples:+ Below is a code snippet that shows how to use this as the default retriever:+ ```python+ llm = dspy.OpenAI(model="gpt-3.5-turbo")+ retriever_model = LancedbRM()+ dspy.settings.configure(lm=llm, rm=retriever_model)+ ```++ Below is a code snippet that shows how to use this in the forward() function of a module+ ```python+ self.retrieve = LancedbRM(k=num_passages)+ ```+ """++ def __init__(+ self,+ table_name: str,+ persist_directory: str,+ k: int = 3,+ ):++ self._table = self._init_lancedb(+ table_name, persist_directory,+ )++ super().__init__(k=k)++ def _init_lancedb(+ self,+ table_name: str,+ persist_directory: str,+ ):+ """+ Initialize LanceDB database and return the table instance.++ Args:+ table_name (str): The name of the table to load. If the table does not exist, it will be Raise Value Error.++ Raises:+ ValueError: If table is already not created++ Returns:+ Lancedb.table: The loaded table.+ """++ self.db = lancedb.connect(persist_directory)+ table_names = self.db.table_names()++ if table_name not in table_names:+ raise ValueError(+ f"Table does not exist in lancedb database persist_directory. Create table with name {table_name}, then reintialize retriever instance",+ )+ else:+ table = self.db.open_table(table_name)+ return table+++ @backoff.on_exception(+ backoff.expo,+ ERRORS,+ max_time=settings.backoff_time,+ )++ def forward(self, query_or_queries: Union[str, List[str]]) -> Prediction:+ """Search with Lancedb for self.k top passages for query++ Args:+ query_or_queries (Union[str, List[str]]): The query or queries to search for.++ Returns:+ dspy.Prediction: An object containing the retrieved passages.+ """+ queries = (+ [query_or_queries]+ if isinstance(query_or_queries, str)+ else query_or_queries+ )+ queries = [q for q in queries if q] # Filter empty queries++ passages = []+ for q in queries:+ results = self._table.search(q).limit(self.k).to_list()+ for r in results:+ passages.append(r['text'])+ return Prediction(passages=[dotdict({"long_text": passage}) for passage in passages])diff --git a/examples/integrations/readme.md b/examples/integrations/readme.mdindex 19251ac67..d5abd5552 100644--- a/examples/integrations/readme.md+++ b/examples/integrations/readme.md@@ -3,6 +3,7 @@ This folder contains a cookbook of using DSPy with 3rd party integrations.DSPy currently supports integrations with: (sorted alphabetically):- ChromadbRM+- LancedbRM- ClarifaiRM- MarqoRM- MongoDBAtlasRMdiff --git a/poetry.lock b/poetry.lockindex 6efcf687f..a59d943ed 100644--- a/poetry.lock+++ b/poetry.lock@@ -1,4 +1,4 @@-# This file is automatically @generated by Poetry 1.8.3 and should not be changed by hand.+# This file is automatically @generated by Poetry 1.7.0 and should not be changed by hand.[[package]]name = "aiohttp"@@ -488,6 +488,17 @@ urllib3 = [[package.extras]crt = ["awscrt (==0.20.11)"]+[[package]]+name = "cachetools"+version = "5.5.0"+description = "Extensible memoizing collections and decorators"+optional = true+python-versions = ">=3.7"+files = [+ {file = "cachetools-5.5.0-py3-none-any.whl", hash = "sha256:02134e8439cdc2ffb62023ce1debca2944c3f289d66bb17ead3ab3dede74b292"},+ {file = "cachetools-5.5.0.tar.gz", hash = "sha256:2cc24fb4cbe39633fb7badd9db9ca6295d766d9c2995f245725a46715d050f2a"},+]+[[package]]name = "certifi"version = "2024.6.2"@@ -1000,6 +1011,20 @@ wrapt = ">=1.10,<2"[package.extras]dev = ["PyTest", "PyTest-Cov", "bump2version (<1)", "sphinx (<2)", "tox"]+[[package]]+name = "deprecation"+version = "2.1.0"+description = "A library to handle automated deprecations"+optional = true+python-versions = "*"+files = [+ {file = "deprecation-2.1.0-py2.py3-none-any.whl", hash = "sha256:a10811591210e1fb0e768a8c25517cabeabcba6f0bf96564f8ff45189f90b14a"},+ {file = "deprecation-2.1.0.tar.gz", hash = "sha256:72b3bde64e5d778694b0cf68178aed03d15e15477116add3fb773e581f9518ff"},+]++[package.dependencies]+packaging = "*"+[[package]]name = "dill"version = "0.3.8"@@ -2245,6 +2270,42 @@ traitlets = ">=5.3"docs = ["myst-parser", "pydata-sphinx-theme", "sphinx-autodoc-typehints", "sphinxcontrib-github-alt", "sphinxcontrib-spelling", "traitlets"]test = ["ipykernel", "pre-commit", "pytest (<8)", "pytest-cov", "pytest-timeout"]+[[package]]+name = "lancedb"+version = "0.11.0"+description = "lancedb"+optional = true+python-versions = ">=3.8"+files = [+ {file = "lancedb-0.11.0-cp38-abi3-macosx_10_15_x86_64.whl", hash = "sha256:11d56c26cd3c7db3757dd29e67f2bbe6d997f39cf4389cc56c7399ffec018fbb"},+ {file = "lancedb-0.11.0-cp38-abi3-macosx_11_0_arm64.whl", hash = "sha256:3e8da024de348dd08f2fdf72063349ca0eece2d66a58bba723adc36af957979a"},+ {file = "lancedb-0.11.0-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:c954dc44ede89d775159f42844f1862a18dde80c616b7bd7b5eb7c98217d0328"},+ {file = "lancedb-0.11.0-cp38-abi3-manylinux_2_24_aarch64.whl", hash = "sha256:fe76f0fcdc44203cfc5c0153fb4cf882a64678935ae9280a9751c64d07b04f4d"},+ {file = "lancedb-0.11.0-cp38-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:911277e45c3ee57c650fddeb4c85669fb1377103c3992ae73a6c681f4a1d68f0"},+ {file = "lancedb-0.11.0-cp38-abi3-win_amd64.whl", hash = "sha256:1a7b0c7fc3383f947a52e0692c36a6351dd0c841dd012071bfba66044c95b22a"},+]++[package.dependencies]+attrs = ">=21.3.0"+cachetools = "*"+deprecation = "*"+overrides = ">=0.7"+packaging = "*"+pydantic = ">=1.10"+pylance = "0.15.0"+ratelimiter = ">=1.0,<2.0"+requests = ">=2.31.0"+retry = ">=0.9.2"+tqdm = ">=4.27.0"++[package.extras]+azure = ["adlfs (>=2024.2.0)"]+clip = ["open-clip", "pillow", "torch"]+dev = ["pre-commit", "ruff"]+docs = ["mkdocs", "mkdocs-jupyter", "mkdocs-material", "mkdocstrings[python]"]+embeddings = ["awscli (>=1.29.57)", "boto3 (>=1.28.57)", "botocore (>=1.31.57)", "cohere", "google-generativeai", "huggingface-hub", "instructorembedding", "ollama", "open-clip-torch", "openai (>=1.6.1)", "pillow", "sentence-transformers", "torch"]+tests = ["aiohttp", "boto3", "duckdb", "pandas (>=1.4)", "polars (>=0.19)", "pytest", "pytest-asyncio", "pytest-mock", "pytz", "tantivy"]+[[package]]name = "llama-index"version = "0.10.45.post1"@@ -3443,56 +3504,47 @@ files = [[[package]]name = "numpy"-version = "2.0.0"+version = "1.26.4"description = "Fundamental package for array computing in Python"optional = falsepython-versions = ">=3.9"files = [- {file = "numpy-2.0.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:04494f6ec467ccb5369d1808570ae55f6ed9b5809d7f035059000a37b8d7e86f"},- {file = "numpy-2.0.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:2635dbd200c2d6faf2ef9a0d04f0ecc6b13b3cad54f7c67c61155138835515d2"},- {file = "numpy-2.0.0-cp310-cp310-macosx_14_0_arm64.whl", hash = "sha256:0a43f0974d501842866cc83471bdb0116ba0dffdbaac33ec05e6afed5b615238"},- {file = "numpy-2.0.0-cp310-cp310-macosx_14_0_x86_64.whl", hash = "sha256:8d83bb187fb647643bd56e1ae43f273c7f4dbcdf94550d7938cfc32566756514"},- {file = "numpy-2.0.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:79e843d186c8fb1b102bef3e2bc35ef81160ffef3194646a7fdd6a73c6b97196"},- {file = "numpy-2.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:6d7696c615765091cc5093f76fd1fa069870304beaccfd58b5dcc69e55ef49c1"},- {file = "numpy-2.0.0-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:b4c76e3d4c56f145d41b7b6751255feefae92edbc9a61e1758a98204200f30fc"},- {file = "numpy-2.0.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:acd3a644e4807e73b4e1867b769fbf1ce8c5d80e7caaef0d90dcdc640dfc9787"},- {file = "numpy-2.0.0-cp310-cp310-win32.whl", hash = "sha256:cee6cc0584f71adefe2c908856ccc98702baf95ff80092e4ca46061538a2ba98"},- {file = "numpy-2.0.0-cp310-cp310-win_amd64.whl", hash = "sha256:ed08d2703b5972ec736451b818c2eb9da80d66c3e84aed1deeb0c345fefe461b"},- {file = "numpy-2.0.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:ad0c86f3455fbd0de6c31a3056eb822fc939f81b1618f10ff3406971893b62a5"},- {file = "numpy-2.0.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:e7f387600d424f91576af20518334df3d97bc76a300a755f9a8d6e4f5cadd289"},- {file = "numpy-2.0.0-cp311-cp311-macosx_14_0_arm64.whl", hash = "sha256:34f003cb88b1ba38cb9a9a4a3161c1604973d7f9d5552c38bc2f04f829536609"},- {file = "numpy-2.0.0-cp311-cp311-macosx_14_0_x86_64.whl", hash = "sha256:b6f6a8f45d0313db07d6d1d37bd0b112f887e1369758a5419c0370ba915b3871"},- {file = "numpy-2.0.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:5f64641b42b2429f56ee08b4f427a4d2daf916ec59686061de751a55aafa22e4"},- {file = "numpy-2.0.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:a7039a136017eaa92c1848152827e1424701532ca8e8967fe480fe1569dae581"},- {file = "numpy-2.0.0-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:46e161722e0f619749d1cd892167039015b2c2817296104487cd03ed4a955995"},- {file = "numpy-2.0.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:0e50842b2295ba8414c8c1d9d957083d5dfe9e16828b37de883f51fc53c4016f"},- {file = "numpy-2.0.0-cp311-cp311-win32.whl", hash = "sha256:2ce46fd0b8a0c947ae047d222f7136fc4d55538741373107574271bc00e20e8f"},- {file = "numpy-2.0.0-cp311-cp311-win_amd64.whl", hash = "sha256:fbd6acc766814ea6443628f4e6751d0da6593dae29c08c0b2606164db026970c"},- {file = "numpy-2.0.0-cp312-cp312-macosx_10_9_x86_64.whl", hash = "sha256:354f373279768fa5a584bac997de6a6c9bc535c482592d7a813bb0c09be6c76f"},- {file = "numpy-2.0.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:4d2f62e55a4cd9c58c1d9a1c9edaedcd857a73cb6fda875bf79093f9d9086f85"},- {file = "numpy-2.0.0-cp312-cp312-macosx_14_0_arm64.whl", hash = "sha256:1e72728e7501a450288fc8e1f9ebc73d90cfd4671ebbd631f3e7857c39bd16f2"},- {file = "numpy-2.0.0-cp312-cp312-macosx_14_0_x86_64.whl", hash = "sha256:84554fc53daa8f6abf8e8a66e076aff6ece62de68523d9f665f32d2fc50fd66e"},- {file = "numpy-2.0.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:c73aafd1afca80afecb22718f8700b40ac7cab927b8abab3c3e337d70e10e5a2"},- {file = "numpy-2.0.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:49d9f7d256fbc804391a7f72d4a617302b1afac1112fac19b6c6cec63fe7fe8a"},- {file = "numpy-2.0.0-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:0ec84b9ba0654f3b962802edc91424331f423dcf5d5f926676e0150789cb3d95"},- {file = "numpy-2.0.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:feff59f27338135776f6d4e2ec7aeeac5d5f7a08a83e80869121ef8164b74af9"},- {file = "numpy-2.0.0-cp312-cp312-win32.whl", hash = "sha256:c5a59996dc61835133b56a32ebe4ef3740ea5bc19b3983ac60cc32be5a665d54"},- {file = "numpy-2.0.0-cp312-cp312-win_amd64.whl", hash = "sha256:a356364941fb0593bb899a1076b92dfa2029f6f5b8ba88a14fd0984aaf76d0df"},- {file = "numpy-2.0.0-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:e61155fae27570692ad1d327e81c6cf27d535a5d7ef97648a17d922224b216de"},- {file = "numpy-2.0.0-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:4554eb96f0fd263041baf16cf0881b3f5dafae7a59b1049acb9540c4d57bc8cb"},- {file = "numpy-2.0.0-cp39-cp39-macosx_14_0_arm64.whl", hash = "sha256:903703372d46bce88b6920a0cd86c3ad82dae2dbef157b5fc01b70ea1cfc430f"},- {file = "numpy-2.0.0-cp39-cp39-macosx_14_0_x86_64.whl", hash = "sha256:3e8e01233d57639b2e30966c63d36fcea099d17c53bf424d77f088b0f4babd86"},- {file = "numpy-2.0.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:1cde1753efe513705a0c6d28f5884e22bdc30438bf0085c5c486cdaff40cd67a"},- {file = "numpy-2.0.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:821eedb7165ead9eebdb569986968b541f9908979c2da8a4967ecac4439bae3d"},- {file = "numpy-2.0.0-cp39-cp39-musllinux_1_1_x86_64.whl", hash = "sha256:9a1712c015831da583b21c5bfe15e8684137097969c6d22e8316ba66b5baabe4"},- {file = "numpy-2.0.0-cp39-cp39-musllinux_1_2_aarch64.whl", hash = "sha256:9c27f0946a3536403efb0e1c28def1ae6730a72cd0d5878db38824855e3afc44"},- {file = "numpy-2.0.0-cp39-cp39-win32.whl", hash = "sha256:63b92c512d9dbcc37f9d81b123dec99fdb318ba38c8059afc78086fe73820275"},- {file = "numpy-2.0.0-cp39-cp39-win_amd64.whl", hash = "sha256:3f6bed7f840d44c08ebdb73b1825282b801799e325bcbdfa6bc5c370e5aecc65"},- {file = "numpy-2.0.0-pp39-pypy39_pp73-macosx_10_9_x86_64.whl", hash = "sha256:9416a5c2e92ace094e9f0082c5fd473502c91651fb896bc17690d6fc475128d6"},- {file = "numpy-2.0.0-pp39-pypy39_pp73-macosx_14_0_x86_64.whl", hash = "sha256:17067d097ed036636fa79f6a869ac26df7db1ba22039d962422506640314933a"},- {file = "numpy-2.0.0-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:38ecb5b0582cd125f67a629072fed6f83562d9dd04d7e03256c9829bdec027ad"},- {file = "numpy-2.0.0-pp39-pypy39_pp73-win_amd64.whl", hash = "sha256:cef04d068f5fb0518a77857953193b6bb94809a806bd0a14983a8f12ada060c9"},- {file = "numpy-2.0.0.tar.gz", hash = "sha256:cf5d1c9e6837f8af9f92b6bd3e86d513cdc11f60fd62185cc49ec7d1aba34864"},+ {file = "numpy-1.26.4-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:9ff0f4f29c51e2803569d7a51c2304de5554655a60c5d776e35b4a41413830d0"},+ {file = "numpy-1.26.4-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:2e4ee3380d6de9c9ec04745830fd9e2eccb3e6cf790d39d7b98ffd19b0dd754a"},+ {file = "numpy-1.26.4-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:d209d8969599b27ad20994c8e41936ee0964e6da07478d6c35016bc386b66ad4"},+ {file = "numpy-1.26.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ffa75af20b44f8dba823498024771d5ac50620e6915abac414251bd971b4529f"},+ {file = "numpy-1.26.4-cp310-cp310-musllinux_1_1_aarch64.whl", hash = "sha256:62b8e4b1e28009ef2846b4c7852046736bab361f7aeadeb6a5b89ebec3c7055a"},+ {file = "numpy-1.26.4-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:a4abb4f9001ad2858e7ac189089c42178fcce737e4169dc61321660f1a96c7d2"},+ {file = "numpy-1.26.4-cp310-cp310-win32.whl", hash = "sha256:bfe25acf8b437eb2a8b2d49d443800a5f18508cd811fea3181723922a8a82b07"},+ {file = "numpy-1.26.4-cp310-cp310-win_amd64.whl", hash = "sha256:b97fe8060236edf3662adfc2c633f56a08ae30560c56310562cb4f95500022d5"},+ {file = "numpy-1.26.4-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:4c66707fabe114439db9068ee468c26bbdf909cac0fb58686a42a24de1760c71"},+ {file = "numpy-1.26.4-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:edd8b5fe47dab091176d21bb6de568acdd906d1887a4584a15a9a96a1dca06ef"},+ {file = "numpy-1.26.4-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:7ab55401287bfec946ced39700c053796e7cc0e3acbef09993a9ad2adba6ca6e"},+ {file = "numpy-1.26.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:666dbfb6ec68962c033a450943ded891bed2d54e6755e35e5835d63f4f6931d5"},+ {file = "numpy-1.26.4-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:96ff0b2ad353d8f990b63294c8986f1ec3cb19d749234014f4e7eb0112ceba5a"},+ {file = "numpy-1.26.4-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:60dedbb91afcbfdc9bc0b1f3f402804070deed7392c23eb7a7f07fa857868e8a"},+ {file = "numpy-1.26.4-cp311-cp311-win32.whl", hash = "sha256:1af303d6b2210eb850fcf03064d364652b7120803a0b872f5211f5234b399f20"},+ {file = "numpy-1.26.4-cp311-cp311-win_amd64.whl", hash = "sha256:cd25bcecc4974d09257ffcd1f098ee778f7834c3ad767fe5db785be9a4aa9cb2"},+ {file = "numpy-1.26.4-cp312-cp312-macosx_10_9_x86_64.whl", hash = "sha256:b3ce300f3644fb06443ee2222c2201dd3a89ea6040541412b8fa189341847218"},+ {file = "numpy-1.26.4-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:03a8c78d01d9781b28a6989f6fa1bb2c4f2d51201cf99d3dd875df6fbd96b23b"},+ {file = "numpy-1.26.4-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:9fad7dcb1aac3c7f0584a5a8133e3a43eeb2fe127f47e3632d43d677c66c102b"},+ {file = "numpy-1.26.4-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:675d61ffbfa78604709862923189bad94014bef562cc35cf61d3a07bba02a7ed"},+ {file = "numpy-1.26.4-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:ab47dbe5cc8210f55aa58e4805fe224dac469cde56b9f731a4c098b91917159a"},+ {file = "numpy-1.26.4-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:1dda2e7b4ec9dd512f84935c5f126c8bd8b9f2fc001e9f54af255e8c5f16b0e0"},+ {file = "numpy-1.26.4-cp312-cp312-win32.whl", hash = "sha256:50193e430acfc1346175fcbdaa28ffec49947a06918b7b92130744e81e640110"},+ {file = "numpy-1.26.4-cp312-cp312-win_amd64.whl", hash = "sha256:08beddf13648eb95f8d867350f6a018a4be2e5ad54c8d8caed89ebca558b2818"},+ {file = "numpy-1.26.4-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:7349ab0fa0c429c82442a27a9673fc802ffdb7c7775fad780226cb234965e53c"},+ {file = "numpy-1.26.4-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:52b8b60467cd7dd1e9ed082188b4e6bb35aa5cdd01777621a1658910745b90be"},+ {file = "numpy-1.26.4-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:d5241e0a80d808d70546c697135da2c613f30e28251ff8307eb72ba696945764"},+ {file = "numpy-1.26.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f870204a840a60da0b12273ef34f7051e98c3b5961b61b0c2c1be6dfd64fbcd3"},+ {file = "numpy-1.26.4-cp39-cp39-musllinux_1_1_aarch64.whl", hash = "sha256:679b0076f67ecc0138fd2ede3a8fd196dddc2ad3254069bcb9faf9a79b1cebcd"},+ {file = "numpy-1.26.4-cp39-cp39-musllinux_1_1_x86_64.whl", hash = "sha256:47711010ad8555514b434df65f7d7b076bb8261df1ca9bb78f53d3b2db02e95c"},+ {file = "numpy-1.26.4-cp39-cp39-win32.whl", hash = "sha256:a354325ee03388678242a4d7ebcd08b5c727033fcff3b2f536aea978e15ee9e6"},+ {file = "numpy-1.26.4-cp39-cp39-win_amd64.whl", hash = "sha256:3373d5d70a5fe74a2c1bb6d2cfd9609ecf686d47a2d7b1d37a8f3b6bf6003aea"},+ {file = "numpy-1.26.4-pp39-pypy39_pp73-macosx_10_9_x86_64.whl", hash = "sha256:afedb719a9dcfc7eaf2287b839d8198e06dcd4cb5d276a3df279231138e83d30"},+ {file = "numpy-1.26.4-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:95a7476c59002f2f6c590b9b7b998306fba6a5aa646b1e22ddfeaf8f78c3a29c"},+ {file = "numpy-1.26.4-pp39-pypy39_pp73-win_amd64.whl", hash = "sha256:7e50d0a0cc3189f9cb0aeb3a6a6af18c16f59f004b866cd2be1c14b36134a4a0"},+ {file = "numpy-1.26.4.tar.gz", hash = "sha256:2a02aba9ed12e4ac4eb3ea9421c420301a0c6460d9830d74a9df87efa4912010"},][[package]]@@ -4622,6 +4674,32 @@ dev = ["coverage[toml] (==5.0.4)", "cryptography (>=3.4.0)", "pre-commit", "pytedocs = ["sphinx (>=4.5.0,<5.0.0)", "sphinx-rtd-theme", "zope.interface"]tests = ["coverage[toml] (==5.0.4)", "pytest (>=6.0.0,<7.0.0)"]+[[package]]+name = "pylance"+version = "0.15.0"+description = "python wrapper for Lance columnar format"+optional = true+python-versions = ">=3.9"+files = [+ {file = "pylance-0.15.0-cp39-abi3-macosx_10_15_x86_64.whl", hash = "sha256:c61b1e3532fa481505cf04197fc3e38f1f2169d724c9eaa889f3acbae183db41"},+ {file = "pylance-0.15.0-cp39-abi3-macosx_11_0_arm64.whl", hash = "sha256:12f1ce4c47cf78b720d62667297e956a593430fe4a7559988c0436d27e58f2b6"},+ {file = "pylance-0.15.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:8821c2cffeb6c273833edf7e3777de8e9f93a014a50a347a87e11c5b743f82fe"},+ {file = "pylance-0.15.0-cp39-abi3-manylinux_2_24_aarch64.whl", hash = "sha256:225df36e231f70b1c15df89c8c3ccc82f9dc7f443ebd24c2c808eac86ed99ed5"},+ {file = "pylance-0.15.0-cp39-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:fbce3e07c33e3d30dd07593cfda4e185e1a08fa75fdc3ae4dcd2941ce7137714"},+ {file = "pylance-0.15.0-cp39-abi3-win_amd64.whl", hash = "sha256:0260f4f93811c3888440412b6c7c071cc8fd22ac1a8b323472f919952926df2a"},+]++[package.dependencies]+numpy = ">=1.22,<2"+pyarrow = ">=12"++[package.extras]+benchmarks = ["pytest-benchmark"]+dev = ["ruff (==0.4.1)"]+ray = ["ray[data]"]+tests = ["boto3", "datasets", "duckdb", "h5py (<3.11)", "ml-dtypes", "pandas", "pillow", "polars[pandas,pyarrow]", "pytest", "tensorflow", "tqdm"]+torch = ["torch"]+[[package]]name = "pymdown-extensions"version = "10.8.1"@@ -5067,6 +5145,20 @@ urllib3 = ">=1.26.14,<3"fastembed = ["fastembed (==0.2.7)"]fastembed-gpu = ["fastembed-gpu (==0.2.7)"]+[[package]]+name = "ratelimiter"+version = "1.2.0.post0"+description = "Simple python rate limiting object"+optional = true+python-versions = "*"+files = [+ {file = "ratelimiter-1.2.0.post0-py3-none-any.whl", hash = "sha256:a52be07bc0bb0b3674b4b304550f10c769bbb00fead3072e035904474259809f"},+ {file = "ratelimiter-1.2.0.post0.tar.gz", hash = "sha256:5c395dcabdbbde2e5178ef3f89b568a3066454a6ddc223b76473dac22f89b4f7"},+]++[package.extras]+test = ["pytest (>=3.0)", "pytest-asyncio"]+[[package]]name = "referencing"version = "0.35.1"@@ -5205,6 +5297,21 @@ urllib3 = ">=1.21.1,<3"socks = ["PySocks (>=1.5.6,!=1.5.7)"]use-chardet-on-py3 = ["chardet (>=3.0.2,<6)"]+[[package]]+name = "retry"+version = "0.9.2"+description = "Easy to use retry decorator."+optional = true+python-versions = "*"+files = [+ {file = "retry-0.9.2-py2.py3-none-any.whl", hash = "sha256:ccddf89761fa2c726ab29391837d4327f819ea14d244c232a1d24c67a2f98606"},+ {file = "retry-0.9.2.tar.gz", hash = "sha256:f8bfa8b99b69c4506d6f5bd3b0aabf77f98cdb17f3c9fc3f5ca820033336fba4"},+]++[package.dependencies]+decorator = ">=3.4.2"+py = ">=1.4.26,<2.0.0"+[[package]]name = "rich"version = "13.7.1"@@ -7433,6 +7540,7 @@ chromadb = ["chromadb"]docs = ["autodoc_pydantic", "docutils", "furo", "m2r2", "myst-nb", "myst-parser", "sphinx", "sphinx-autobuild", "sphinx-automodapi", "sphinx-reredirects", "sphinx_rtd_theme"]epsilla = ["pyepsilla"]fastembed = ["fastembed"]+lancedb = ["lancedb"]marqo = ["marqo"]milvus = ["pymilvus"]pinecone = ["pinecone-client"]@@ -7443,4 +7551,4 @@ weaviate = ["weaviate-client"][metadata]lock-version = "2.0"python-versions = ">=3.9,<3.13"-content-hash = "3e0c99d3cc24175e5bd72ed0a3577a2766ecb7e4da688f78a9462a5dcfa5a457"+content-hash = "092a1600c1a634036d404be79b69fb38185e8743364061800a8938a5c432d932"diff --git a/pyproject.toml b/pyproject.tomlindex 80d3878ea..1c8844cd1 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -46,6 +46,7 @@ dependencies = [[project.optional-dependencies]anthropic = ["anthropic>=0.18.0,<1.0.0"]chromadb = ["chromadb~=0.4.14"]+lancedb = ["lancedb>=0.11.0"]qdrant = ["qdrant-client>=1.6.2", "fastembed>=0.2.0"]marqo = ["marqo"]epsilla = ["pyepsilla~=0.3.7"]@@ -101,6 +102,7 @@ requests = "^2.31.0"optuna = "^3.4.0"anthropic = { version = ">=0.18.0,<1.0.0", optional = true }chromadb = { version = "^0.4.14", optional = true }+lancedb = {version = "^0.11.0", optional = true}fastembed = { version = ">=0.2.0", optional = true }marqo = { version = "*", optional = true }pyepsilla = {version = "^0.3.7", optional = true}@@ -145,6 +147,7 @@ semver = "^3.0.2"[tool.poetry.extras]chromadb = ["chromadb"]+lancedb = ["lancedb"]qdrant = ["qdrant-client", "fastembed"]marqo = ["marqo"]epsilla = ["pyepsilla"]diff --git a/setup.py b/setup.pyindex 945a80f97..02fdf703c 100644--- a/setup.py+++ b/setup.py@@ -26,6 +26,7 @@ setup(extras_require={"chromadb": ["chromadb~=0.4.14"],+ "lancedb": ["lancedb~=0.11.0"],"qdrant": ["qdrant-client", "fastembed"],"marqo": ["marqo~=3.1.0"],"mongodb": ["pymongo~=3.12.0"],