Skip to content

armgit / dspy

Public Imported from GitHub
0

DSPy: The framework for programming—not prompting—language models

Python

added ingestion in notebook

Mogith PN3c576712e0d1
diff --git a/clarifai_llm_retriever_example.ipynb b/clarifai_llm_retriever_example.ipynb
index e9c9a68b1..0eb899e89 100644
--- a/clarifai_llm_retriever_example.ipynb
+++ b/clarifai_llm_retriever_example.ipynb
@@ -46,7 +46,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "Initialize clarifai app id, user id and PAT.\n",
+ "#### Initialize clarifai app id, user id and PAT.\n",
"\n",
"You can browse the portal to obtain [MODEL URL](https://clarifai.com/explore/models) for different models in clarifai community."
]
@@ -68,7 +68,43 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "Initialize LLM class"
+ "### Data ingestion into clarifai vectordatabase\n",
+ "\n",
+ "To use clarifai as retriever all you have to do is ingest the documents into clarifai app that serves as your vectordatabase to retrieve similar documents.\n",
+ "To simplify the ingestion, we are utilising the clarifaivectordatabase integration for ingestion."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "#run this block to ingest the documents into clarifai app as chunks.\n",
+ "# if you encounter any issue, make sure to run `pip install langchain`\n",
+ "\n",
+ "from langchain.text_splitter import CharacterTextSplitter\n",
+ "from langchain.document_loaders import TextLoader\n",
+ "from langchain.vectorstores import Clarifai \n",
+ "\n",
+ "loader = TextLoader(\"YOUR_TEXT_FILE_PATH\") #replace with your file path\n",
+ "documents = loader.load()\n",
+ "text_splitter = CharacterTextSplitter(chunk_size=1024, chunk_overlap=200)\n",
+ "docs = text_splitter.split_documents(documents)\n",
+ "\n",
+ "clarifai_vector_db = Clarifai.from_documents(\n",
+ " user_id=USER_ID,\n",
+ " app_id=APP_ID,\n",
+ " documents=docs,\n",
+ " pat=PAT\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "#### Initialize LLM class"
]
},
{