DSPy: The framework for programming—not prompting—language models
Python
modified example notebook
mogith-pn0da20c9cbe8f
diff --git a/examples/integrations/clarifai/clarifai_llm_retriever_example.ipynb b/examples/integrations/clarifai/clarifai_llm_retriever_example.ipynbindex 12b6bb8f4..131ba0ca4 100644--- a/examples/integrations/clarifai/clarifai_llm_retriever_example.ipynb+++ b/examples/integrations/clarifai/clarifai_llm_retriever_example.ipynb@@ -25,6 +25,15 @@"!pip install clarifai"]},+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "!pip install dspy-ai"+ ]+ },{"cell_type": "markdown","metadata": {},@@ -34,7 +43,7 @@},{"cell_type": "code",- "execution_count": 7,+ "execution_count": 4,"metadata": {},"outputs": [],"source": [@@ -47,21 +56,22 @@"metadata": {},"source": ["#### Initialize clarifai app id, user id and PAT.\n",+ "Create an AI app in clarifai portal in < 1 min by following this link [getting started](https://docs.clarifai.com/clarifai-basics/quick-start/your-first-predictions).\n","\n","You can browse the portal to obtain [MODEL URL](https://clarifai.com/explore/models) for different models in clarifai community."]},{"cell_type": "code",- "execution_count": 8,+ "execution_count": 6,"metadata": {},"outputs": [],"source": ["#for the demo we are going with llama2-70b-chat\n","MODEL_URL = \"https://clarifai.com/meta/Llama-2/models/llama2-70b-chat\" \n",- "PAT = CLARIFAI_PAT\n",- "USER_ID = \"YOUR_USER_ID\"\n",- "APP_ID = \"YOUR_APP_ID\""+ "PAT = \"CLARIFAI_PAT\"\n",+ "USER_ID = \"YOUR_ID\"\n",+ "APP_ID = \"YOUR_APP\""]},{@@ -87,7 +97,7 @@"from langchain.document_loaders import TextLoader\n","from langchain.vectorstores import Clarifai as clarifaivectorstore\n","\n",- "loader = TextLoader(\"YOUR_TEXT_FILE_PATH\") #replace with your file path\n",+ "loader = TextLoader(\"YOUR_TEXT_FILE\") #replace with your file path\n","documents = loader.load()\n","text_splitter = CharacterTextSplitter(chunk_size=1024, chunk_overlap=200)\n","docs = text_splitter.split_documents(documents)\n",@@ -116,7 +126,7 @@},{"cell_type": "code",- "execution_count": 10,+ "execution_count": 55,"metadata": {},"outputs": [],"source": [@@ -133,12 +143,12 @@},{"cell_type": "code",- "execution_count": 12,+ "execution_count": 56,"metadata": {},"outputs": [],"source": ["\n",- "retriever_model=ClarifaiRM(clarifai_user_id=USER_ID, clarfiai_app_id=APP_ID, clarifai_pat=PAT)"+ "retriever_model=ClarifaiRM(clarifai_user_id=USER_ID, clarfiai_app_id=APP_ID, clarifai_pat=PAT, k=2)"]},{@@ -150,7 +160,7 @@},{"cell_type": "code",- "execution_count": 13,+ "execution_count": 57,"metadata": {},"outputs": [],"source": [@@ -166,7 +176,7 @@},{"cell_type": "code",- "execution_count": 18,+ "execution_count": 11,"metadata": {},"outputs": [{@@ -199,24 +209,24 @@},{"cell_type": "code",- "execution_count": 21,+ "execution_count": 51,"metadata": {},"outputs": [],"source": [- "retrieve = dspy.Retrieve(k=1)\n",- "topK_passages = retrieve(\"what are the stages in planning, sanctioning and execution of public works\").passages"+ "retrieve = dspy.Retrieve()\n",+ "topK_passages = retrieve(\"can I test my vehicle engine in pit?\").passages"]},{"cell_type": "code",- "execution_count": 23,+ "execution_count": 52,"metadata": {},"outputs": [{"name": "stdout","output_type": "stream","text": [- "['1.11\\n\\nProcessing of Public Works\\n\\nFollowing are the stages in planning, sanctioning and execution of work.\\ni)\\n\\nPerspective Planning for works;\\n\\nii)\\n\\nPreparation of Preliminary Project Report (PPR) or Rough Cost Estimate;\\n\\niii) Acceptance of necessity and issue of in-Principle Approval;\\niv) Preparation of Detailed Project Report (DPR) or Preliminary Estimate (PE);\\n\\n15']\n"+ "['A 6.8.3\\n\\nCranking engines in the pits is allowed, when the following conditions are met:\\n• The vehicle has passed mechanical inspection.\\n• The driven axles are securely jacked up.\\n• Gearbox is in neutral.\\n• All driven wheels are removed.\\n• Connectors to all injectors and ignition coils are detached.\\n• A fire extinguisher must be placed next to the engine.\\n\\nA 6.9\\n\\nFueling and Oil\\n\\nA 6.9.1\\n\\nFueling may only take place at the fuel station and must be conducted by officials only.\\n\\nA 6.9.2\\n\\nOpen fuel containers are not permitted at the competition.\\n\\nA 6.9.3\\n\\nWaste oil must be taken to the fuel station for disposal.\\n\\nA 6.10\\n\\n[EV ONLY ] Working on the Vehicle\\n\\nA 6.10.1\\n\\nAll activities require the TSAL to be green.\\n\\nA 6.10.2\\n\\nA prominent manual sign indicating the “TSAL green” state must be present whenever the\\nLVS is switched off and the requirements for an only green TSAL according to EV 4.10 are\\nmet.\\n\\nA 6.10.3', 'A 6.8.3\\n\\nCranking engines in the pits is allowed, when the following conditions are met:\\n• The vehicle has passed mechanical inspection.\\n• The driven axles are securely jacked up.\\n• Gearbox is in neutral.\\n• All driven wheels are removed.\\n• Connectors to all injectors and ignition coils are detached.\\n• A fire extinguisher must be placed next to the engine.\\n\\nA 6.9\\n\\nFueling and Oil\\n\\nA 6.9.1\\n\\nFueling may only take place at the fuel station and must be conducted by officials only.\\n\\nA 6.9.2\\n\\nOpen fuel containers are not permitted at the competition.\\n\\nA 6.9.3\\n\\nWaste oil must be taken to the fuel station for disposal.\\n\\nA 6.10\\n\\n[EV ONLY ] Working on the Vehicle\\n\\nA 6.10.1\\n\\nAll activities require the TSAL to be green.\\n\\nA 6.10.2\\n\\nA prominent manual sign indicating the “TSAL green” state must be present whenever the\\nLVS is switched off and the requirements for an only green TSAL according to EV 4.10 are\\nmet.\\n\\nA 6.10.3']\n"]}],@@ -255,16 +265,16 @@},{"cell_type": "code",- "execution_count": 26,+ "execution_count": 53,"metadata": {},"outputs": [],"source": ["class GenerateAnswer(dspy.Signature):\n",- " \"\"\"Answer questions with short factoid answers.\"\"\"\n",+ " \"\"\"Think and Answer questions based on the context provided.\"\"\"\n","\n",- " context = dspy.InputField(desc=\"may contain relevant facts about \")\n",- " question = dspy.InputField()\n",- " answer = dspy.OutputField(desc=\"often between 1 and 5 words\")"+ " context = dspy.InputField(desc=\"may contain relevant facts about user query\")\n",+ " question = dspy.InputField(desc=\"User query\")\n",+ " answer = dspy.OutputField(desc=\"Answer in one or two lines\")"]},{@@ -276,15 +286,15 @@},{"cell_type": "code",- "execution_count": 27,+ "execution_count": 54,"metadata": {},"outputs": [],"source": ["class RAG(dspy.Module):\n",- " def __init__(self, num_passages=3):\n",+ " def __init__(self):\n"," super().__init__()\n","\n",- " self.retrieve = dspy.Retrieve(k=num_passages)\n",+ " self.retrieve = dspy.Retrieve()\n"," self.generate_answer = dspy.ChainOfThought(GenerateAnswer)\n"," \n"," def forward(self, question):\n",@@ -302,39 +312,133 @@},{"cell_type": "code",- "execution_count": 28,+ "execution_count": 59,"metadata": {},"outputs": [{"name": "stdout","output_type": "stream","text": [- "Question: Which bid will be termed as L1 ?\n",- "Predicted Answer: The bidder who quotes the lowest price among\n",- "Retrieved Contexts (truncated): ['If L1 bid is not a ‘Class-I local supplier’, 50 (fifty) percent of the order quantity\\nshall be awarded to L1. Thereafter, the lowest bidder among the ‘Class-I local\\nsupplier’ will be invited to match ...', 'If L1 bid is not a ‘Class-I local supplier’, 50 (fifty) percent of the order quantity\\nshall be awarded to L1. Thereafter, the lowest bidder among the ‘Class-I local\\nsupplier’ will be invited to match ...', 'If L1 bid is not a ‘Class-I local supplier’, 50 (fifty) percent of the order quantity\\nshall be awarded to L1. Thereafter, the lowest bidder among the ‘Class-I local\\nsupplier’ will be invited to match ...']\n"+ "Question: can I test my vehicle engine in pit before inspection?\n",+ "Predicted Answer: No, you cannot test your vehicle engine in the pit before inspection.\n",+ "Retrieved Contexts (truncated): ['A 6.8.3\\n\\nCranking engines in the pits is allowed, when the following conditions are met:\\n• The vehicle has passed mechanical inspection.\\n• The driven axles are securely jacked up.\\n• Gearbox is in neut...', 'A 6.8.3\\n\\nCranking engines in the pits is allowed, when the following conditions are met:\\n• The vehicle has passed mechanical inspection.\\n• The driven axles are securely jacked up.\\n• Gearbox is in neut...']\n"]}],"source": ["# Ask any question you like to this RAG program.\n",- "my_question = \"Which bid will be termed as L1?\"\n",+ "my_question = \"can I test my vehicle engine in pit before inspection?\"\n","\n","# Get the prediction. This contains `pred.context` and `pred.answer`.\n",- "obj= RAG()\n",- "pred=obj(my_question)\n",+ "Rag_obj= RAG()\n",+ "predict_response_llama70b=Rag_obj(my_question)\n","\n","# Print the contexts and the answer.\n","print(f\"Question: {my_question}\")\n",- "print(f\"Predicted Answer: {pred.answer}\")\n",- "print(f\"Retrieved Contexts (truncated): {[c[:200] + '...' for c in pred.context]}\")"+ "print(f\"Predicted Answer: {predict_response_llama70b.answer}\")\n",+ "print(f\"Retrieved Contexts (truncated): {[c[:200] + '...' for c in predict_response_llama70b.context]}\")"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "### Now we will compare our RAG DSPy module with different community models from clarifai and comapare responses"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "### Mistral-7b Instruct"]},{"cell_type": "code",- "execution_count": null,+ "execution_count": 64,+ "metadata": {},+ "outputs": [],+ "source": [+ "mistral_lm = dspy.Clarifai(model=\"https://clarifai.com/mistralai/completion/models/mistral-7B-Instruct\", api_key=PAT, n=2, inference_params={'temperature':0.6})\n",+ "dspy.settings.configure(lm=mistral_lm, rm=retriever_model)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 70,+ "metadata": {},+ "outputs": [+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "Question: can I test my vehicle engine in pit before inspection?\n",+ "Predicted Answer: Reasoning: According to the context, cranking engines in the pits is allowed only when the vehicle has passed mechanical inspection.\n",+ "\n",+ "Answer: No, you cannot test your vehicle engine in pit before inspection.\n",+ "Retrieved Contexts (truncated): ['A 6.8.3\\n\\nCranking engines in the pits is allowed, when the following conditions are met:\\n• The vehicle has passed mechanical inspection.\\n• The driven axles are securely jacked up.\\n• Gearbox is in neut...', 'A 6.8.3\\n\\nCranking engines in the pits is allowed, when the following conditions are met:\\n• The vehicle has passed mechanical inspection.\\n• The driven axles are securely jacked up.\\n• Gearbox is in neut...']\n"+ ]+ }+ ],+ "source": [+ "my_question = \"can I test my vehicle engine in pit before inspection?\"\n",+ "Rag_obj= RAG()\n",+ "predict_response_mistral=Rag_obj(my_question)\n",+ "\n",+ "print(f\"Question: {my_question}\")\n",+ "print(f\"Predicted Answer: {predict_response_mistral.answer}\")\n",+ "print(f\"Retrieved Contexts (truncated): {[c[:200] + '...' for c in predict_response_mistral.context]}\")"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "### Gemini Pro"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 66,"metadata": {},"outputs": [],- "source": []+ "source": [+ "\n",+ "gemini_lm = dspy.Clarifai(model=\"https://clarifai.com/gcp/generate/models/gemini-pro\", api_key=PAT, n=2)\n",+ "dspy.settings.configure(lm=gemini_lm, rm=retriever_model)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 67,+ "metadata": {},+ "outputs": [+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "Question: can I test my vehicle engine in pit before inspection?\n",+ "Predicted Answer: No, you can't test your vehicle engine in the pits before inspection.\n",+ "Retrieved Contexts (truncated): ['A 6.8.3\\n\\nCranking engines in the pits is allowed, when the following conditions are met:\\n• The vehicle has passed mechanical inspection.\\n• The driven axles are securely jacked up.\\n• Gearbox is in neut...', 'A 6.8.3\\n\\nCranking engines in the pits is allowed, when the following conditions are met:\\n• The vehicle has passed mechanical inspection.\\n• The driven axles are securely jacked up.\\n• Gearbox is in neut...']\n"+ ]+ }+ ],+ "source": [+ "my_question = \"can I test my vehicle engine in pit before inspection?\"\n",+ "Rag_obj= RAG()\n",+ "predict_response_gemini=Rag_obj(my_question)\n",+ "\n",+ "print(f\"Question: {my_question}\")\n",+ "print(f\"Predicted Answer: {predict_response_gemini.answer}\")\n",+ "print(f\"Retrieved Contexts (truncated): {[c[:200] + '...' for c in predict_response_gemini.context]}\")"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Clarifai enables you to test your dspy module with different llm models and compare the response as it is crucial part of prompt engineering to test and achieve the right combination of llm models with right prompt."+ ]}],"metadata": {