DSPy: The framework for programming—not prompting—language models
Python
added ingestion in notebook
Mogith PN3c576712e0d1
diff --git a/clarifai_llm_retriever_example.ipynb b/clarifai_llm_retriever_example.ipynbindex e9c9a68b1..0eb899e89 100644--- a/clarifai_llm_retriever_example.ipynb+++ b/clarifai_llm_retriever_example.ipynb@@ -46,7 +46,7 @@"cell_type": "markdown","metadata": {},"source": [- "Initialize clarifai app id, user id and PAT.\n",+ "#### Initialize clarifai app id, user id and PAT.\n","\n","You can browse the portal to obtain [MODEL URL](https://clarifai.com/explore/models) for different models in clarifai community."]@@ -68,7 +68,43 @@"cell_type": "markdown","metadata": {},"source": [- "Initialize LLM class"+ "### Data ingestion into clarifai vectordatabase\n",+ "\n",+ "To use clarifai as retriever all you have to do is ingest the documents into clarifai app that serves as your vectordatabase to retrieve similar documents.\n",+ "To simplify the ingestion, we are utilising the clarifaivectordatabase integration for ingestion."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "#run this block to ingest the documents into clarifai app as chunks.\n",+ "# if you encounter any issue, make sure to run `pip install langchain`\n",+ "\n",+ "from langchain.text_splitter import CharacterTextSplitter\n",+ "from langchain.document_loaders import TextLoader\n",+ "from langchain.vectorstores import Clarifai \n",+ "\n",+ "loader = TextLoader(\"YOUR_TEXT_FILE_PATH\") #replace with your file path\n",+ "documents = loader.load()\n",+ "text_splitter = CharacterTextSplitter(chunk_size=1024, chunk_overlap=200)\n",+ "docs = text_splitter.split_documents(documents)\n",+ "\n",+ "clarifai_vector_db = Clarifai.from_documents(\n",+ " user_id=USER_ID,\n",+ " app_id=APP_ID,\n",+ " documents=docs,\n",+ " pat=PAT\n",+ ")"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "#### Initialize LLM class"]},{