Skip to content

Convert any text to a graph of knowledge. This can be used for Graph Augmented Generation or Knowledge Graph based QnA

HTML
20.9 KB
Raw Blame
{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Extract relationshipts between concepts\n",
    "Currently not using this"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'name': 'mistral:instruct',\n",
       "  'modified_at': '2023-10-14T11:39:58.684144971+05:30',\n",
       "  'size': 4108916688,\n",
       "  'digest': '8aa307f73b2622af521e8f22d46e4b777123c4df91898dcb2e4079dc8fdf579e'},\n",
       " {'name': 'mistral-openorca:latest',\n",
       "  'modified_at': '2023-10-14T14:27:09.318837452+05:30',\n",
       "  'size': 4108928254,\n",
       "  'digest': '12dc6acc14d0c4d3309e035cac6ff83e64c023aa774b21c41e342982699f436c'}]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import ollama.client as client\n",
    "client.list()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'{{- if .System }}\\n<|im_start|>system {{ .System }}<|im_end|>\\n{{- end }}\\n<|im_start|>user\\n{{ .Prompt }}<|im_end|>\\n<|im_start|>assistant\\n'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mistral_instruct = 'mistral:instruct'\n",
    "mistral_openorca = 'mistral-openorca:latest'\n",
    "mcard = client.show(mistral_openorca)\n",
    "mcard['template']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " {\n",
      "    \"Entity 1\": \"Cryosurgery\",\n",
      "    \"Entity 2\": \"non-narcotic pain\",\n",
      "    \"Relationship\": \"Cryosurgery is a surgical technique that uses liquid nitrogen to freeze and kill cancer cells. It might be part of the Advanced Surgical Recovery Program (ASURE) which involves pain management with non-narcotic pain relievers.\",\n",
      "    \"Related\": 'Yes'\n",
      "}"
     ]
    }
   ],
   "source": [
    "system_prompt = \"\"\"\n",
    "Your task is to find find a relationships between the entities given as a python list.\\n\n",
    "You can use the provided context (delimited between ```) and no other prior knowledge. \\n\n",
    "Follow this chain of thought\n",
    "Thought 1: What are the two entities as per the context. \n",
    "Thought 2: What is the relationship between the given entities as per the given context?. \n",
    "Thought 3: Is this a direct relationship as per the context.\n",
    "Format your response as JSON with the following structure:\\n\n",
    "{ \n",
    "    \"Entity 1\": First Entity,\n",
    "    \"Entity 2\": Second Entity,\n",
    "    \"Relationship\": Explain the Relationship,\n",
    "    \"Related\": 'Yes' only if a direct relation can be infered, otherwise 'No'\n",
    "}\n",
    "\"\"\"\n",
    "\n",
    "prompt = \"\"\"\n",
    "Entites: ['Cryosurgery', 'non-narcotic pain'].\n",
    "\n",
    "Context:  ``` \n",
    "The two primary types of cancer surgery are open surgery and minimally invasive surgery.\n",
    "\n",
    "In open surgery, the surgical oncologist makes a large incision, usually to remove all or part of a tumor and some of the surrounding healthy tissue (margins).\n",
    "\n",
    "Minimally invasive surgical techniques may involve the techniques listed below.\n",
    "\n",
    "Laparoscopy: A surgical oncologist makes a few small incisions and inserts a laparoscope —a thin tube with a tiny camera attached to it—into one of the incisions to capture an interior image, while inserting surgical tools into the other incisions to excise malignancies and surrounding tissue.\n",
    "\n",
    "Laser surgery: The surgeon uses a narrow beam of high-intensity light to remove a tumor.\n",
    "\n",
    "Cryosurgery: The surgeon uses liquid nitrogen to freeze and kill cancer cells.\n",
    "\n",
    "Robotic surgery: This surgery is similar to a laparoscopic surgery. However, instead of manipulating surgical tools by hand, the surgeon uses a computer console to operate the robotic tools.\n",
    "\n",
    "Other procedures, such as endoscopies, embolization, Mohs micrographic surgery and pleuroscopies, may be performed by non-surgeons, including dermatologists, radiation oncologists and interventional pulmonologists, depending on the procedure.\n",
    "\n",
    "Non-surgical treatments may take place before surgery (neoadjuvant therapy) or after surgery (adjuvant therapy) to help prevent cancer growth, metastasis or recurrence. The treatments may include chemotherapy, radiation therapy or hormone therapy.\n",
    "\n",
    "The Advanced Surgical Recovery Program (ASURE) at City of Hope is designed to help patients recover from surgery more quickly and with fewer complications. ASURE is intended to improve surgical outcomes and enhance the patient experience before, during and after surgery, while also reducing the length of a patient’s hospital stay.\n",
    "\n",
    "The program is delivered by City of Hope multidisciplinary teams, which include surgeons, anesthesiologists, nutritionists, behavioral health therapists, nurses and other clinicians. They work collaboratively to implement evidenced-informed protocols, all under one roof. ASURE combines post-surgical rehabilitation with pre-habilitation, by preparing patients before surgery.\n",
    "\n",
    "Typically, ASURE protocols involve nutrition, pain management with non-narcotic pain relievers and physical activity within 24 hours after surgery. The use of enhanced recovery methods like those used in ASURE has been shown to reduce fatigue, post-surgery complications and the amount of narcotics used in post-surgical care, while helping patients resume their everyday activities more quickly.\n",
    "```\n",
    "\"\"\"\n",
    "\n",
    "combined_prompt = system_prompt + \"\\n\" + prompt\n",
    "response, context = client.generate(\n",
    "    model_name=mistral_openorca, prompt=prompt, system=system_prompt\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " [\n",
      "  {\n",
      "    \"node_1\": \"Health data interoperability\",\n",
      "    \"node_2\": \"health information technology systems\",\n",
      "    \"edge\": \"communicate and exchange data\"\n",
      "  },\n",
      "  {\n",
      "    \"node_1\": \"Patient's consent\",\n",
      "    \"node_2\": \"health data sharing\",\n",
      "    \"edge\": \"regardless of the technology platform\"\n",
      "  },\n",
      "  {\n",
      "    \"node_1\": \"Healthcare providers\",\n",
      "    \"node_2\": \"hospitals\",\n",
      "    \"edge\": \"sharing medical histories and other clinical information\"\n",
      "  },\n",
      "  {\n",
      "    \"node_1\": \"Hospital management system (HMS)\",\n",
      "    \"node_2\": \"Electronic Health Record (EHR)\",\n",
      "    \"edge\": \"manage healthcare services\"\n",
      "  },\n",
      "  {\n",
      "    \"node_1\": \"Patient Record (PHR) system\",\n",
      "    \"node_2\": \"Insurance coverage\",\n",
      "    \"edge\": \"authorised providers access data\"\n",
      "  },\n",
      "  {\n",
      "    \"node_1\": \"Diagnostic labs\",\n",
      "    \"node_2\": \"Insurance providers\",\n",
      "    \"edge\": \"health data interoperability\"\n",
      "  },\n",
      "  {\n",
      "    \"node_1\": \"Pharmacies\",\n",
      "    \"node_2\": \"Wellness Service providers (Gyms, Yoga centres)\",\n",
      "    \"edge\": \"purview of Health Data Interoperability\"\n",
      "  },\n",
      "  {\n",
      "    \"node_1\": \"Mobile Health and Wellness (HnW) apps\",\n",
      "    \"node_2\": \"Wearable devices (iPhone, Fitbit, Glucometer)\",\n",
      "    \"edge\": \"access patient data\"\n",
      "  },\n",
      "  {\n",
      "    \"node_1\": \"Government Healthcare Schemes like CGHS\",\n",
      "    \"node_2\": \"public health research\",\n",
      "    \"edge\": \"interoperability benefits\"\n",
      "  },\n",
      "  {\n",
      "    \"node_1\": \"Quench the Question\",\n",
      "    \"node_2\": \"Quest for a change from the Quiescence\",\n",
      "    \"edge\": \"dig deeper into that\"\n",
      "  }\n",
      "]"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import json\n",
    "# from helpers.prompts import graphPrompt\n",
    "\n",
    "def graphPrompt(input: str, metadata={}, model=\"mistral-openorca:latest\"):\n",
    "    SYS_PROMPT = (\n",
    "        \"You are a network graph maker. Your task is to extract a knowledge graph of ontologies from the given text. \"\n",
    "        \"The graph is made in such a way that, nodes represent the key concepts discussed in the text. \"\n",
    "        \"The edges of the graph represent the relation between two concepts.\\n\"\n",
    "        \"Thought 1: Think about the key concepts or entities mentioned in the text.\"\n",
    "        \"           The concepts may include object, entity, location, organisation, person, condition, acronym, documents, services, concept, etc.\\n\"\n",
    "        \"Thought 2: Find out the pairs of concepts that are strongly related to each other.\\n \"\n",
    "        \"Thought 3: Find out the relation between each pairs. \\n\"\n",
    "        \"Format your output as a list of json. Each element of the list contains two nodes \"\n",
    "        \"and a relation between them, like the follwing: \\n\"\n",
    "        \"[\\n\"\n",
    "        \"   {\\n\"\n",
    "        \"       \\\"node_1\\\": \\\"first concept or entity\\\",\\n\"\n",
    "        \"       \\\"node_2': \\\"second concept or entity\\\",\\n\"\n",
    "        \"       \\\"edge': \\\"relationship between the two concepts, node_1 and node_2\\\"\\n\"\n",
    "        \"   }, {...}\\n\"\n",
    "        \"]\"\n",
    "    )\n",
    "    response, _ = client.generate(model_name=model, system=SYS_PROMPT, prompt=input)\n",
    "    try:\n",
    "        result = json.loads(response)\n",
    "        result = [dict(item, **metadata) for item in result]\n",
    "    except BaseException as error:\n",
    "        print(\"\\n\\nERROR ### Here is the buggy response: \", response, \"\\n\\n\")\n",
    "        print(error)\n",
    "        result = None\n",
    "    return result\n",
    "\n",
    "input = \"\"\"\n",
    "Health data interoperability means the ability of different health information technology systems to communicate and exchange data with each other seamlessly, securely, and efficiently with the Patient’s consent. Interoperability enables health data to be shared among healthcare providers, hospitals, and other healthcare organisations, regardless of the technology platform they are using.\n",
    "Simply speaking, that means you should not be needed to lug your entire medical history in hard copy to every doctor, every time you sneeze. \n",
    "We have all experienced going to a doctor carrying that big fat folder, filed with withering pages of our medical history. And when the doctor asks questions about the missing information, so hard we try to recollect the name of the drug prescribed by the last doctor, or our allergies (Thou shan’t forget those!). And still coming out of the cabin only to remember a crucial detail we forgot to mention… Dam! \n",
    "Well… that was an oversimplified — and slightly exaggerated for dramatic effect — reconstruction of my medical experiences visiting a doctor. But it drives the point home. Remember the mobile charger hell… it is like that, only the stakes are higher. \n",
    "Every healthcare service provider these days uses a technology platform to manage its services. This includes a Hospital management system (HMS), Electronic Health Record (EHR), Patient Record (PHR) system etc. In practical terms, health data interoperability means that healthcare information can be accessed by authorised providers across different systems, regardless of where the information was originally generated or stored. This includes data such as medical histories, lab results, imaging studies, Insurance coverage, and other clinical information.\n",
    "Apart from healthcare providers like a hospital or doctor’s clinics, The purview of Health Data Interoperability includes Diagnostic labs, Insurance providers, Pharmacies, Wellness Service providers (Gyms, Yoga centres), Mobile Health and Wellness (HnW) apps, Wearable devices (iPhone, Fitbit, Glucometer), Government Healthcare Schemes like CGHS etc, and even public health research. \n",
    "The lack of interoperability has been a major barrier to the adoption of health information technology and the sharing of patient data in the past. \n",
    "But before we dig deeper into that, we must Quench the Question that is Quintessential to the Quest for a change from the Quiescence. (Wow.. what a Quip.. right?)\n",
    "\n",
    "\"\"\"\n",
    "\n",
    "res = graphPrompt(input)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'node_1': 'Health data interoperability',\n",
       "  'node_2': 'health information technology systems',\n",
       "  'edge': 'communicate and exchange data'},\n",
       " {'node_1': \"Patient's consent\",\n",
       "  'node_2': 'health data sharing',\n",
       "  'edge': 'regardless of the technology platform'},\n",
       " {'node_1': 'Healthcare providers',\n",
       "  'node_2': 'hospitals',\n",
       "  'edge': 'sharing medical histories and other clinical information'},\n",
       " {'node_1': 'Hospital management system (HMS)',\n",
       "  'node_2': 'Electronic Health Record (EHR)',\n",
       "  'edge': 'manage healthcare services'},\n",
       " {'node_1': 'Patient Record (PHR) system',\n",
       "  'node_2': 'Insurance coverage',\n",
       "  'edge': 'authorised providers access data'},\n",
       " {'node_1': 'Diagnostic labs',\n",
       "  'node_2': 'Insurance providers',\n",
       "  'edge': 'health data interoperability'},\n",
       " {'node_1': 'Pharmacies',\n",
       "  'node_2': 'Wellness Service providers (Gyms, Yoga centres)',\n",
       "  'edge': 'purview of Health Data Interoperability'},\n",
       " {'node_1': 'Mobile Health and Wellness (HnW) apps',\n",
       "  'node_2': 'Wearable devices (iPhone, Fitbit, Glucometer)',\n",
       "  'edge': 'access patient data'},\n",
       " {'node_1': 'Government Healthcare Schemes like CGHS',\n",
       "  'node_2': 'public health research',\n",
       "  'edge': 'interoperability benefits'},\n",
       " {'node_1': 'Quench the Question',\n",
       "  'node_2': 'Quest for a change from the Quiescence',\n",
       "  'edge': 'dig deeper into that'}]"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>node_1</th>\n",
       "      <th>node_2</th>\n",
       "      <th>edge</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Health data interoperability</td>\n",
       "      <td>health information technology systems</td>\n",
       "      <td>communicate and exchange data</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Patient's consent</td>\n",
       "      <td>health data sharing</td>\n",
       "      <td>regardless of the technology platform</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Healthcare providers</td>\n",
       "      <td>hospitals</td>\n",
       "      <td>sharing medical histories and other clinical i...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Hospital management system (HMS)</td>\n",
       "      <td>Electronic Health Record (EHR)</td>\n",
       "      <td>manage healthcare services</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Patient Record (PHR) system</td>\n",
       "      <td>Insurance coverage</td>\n",
       "      <td>authorised providers access data</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Diagnostic labs</td>\n",
       "      <td>Insurance providers</td>\n",
       "      <td>health data interoperability</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Pharmacies</td>\n",
       "      <td>Wellness Service providers (Gyms, Yoga centres)</td>\n",
       "      <td>purview of Health Data Interoperability</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Mobile Health and Wellness (HnW) apps</td>\n",
       "      <td>Wearable devices (iPhone, Fitbit, Glucometer)</td>\n",
       "      <td>access patient data</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Government Healthcare Schemes like CGHS</td>\n",
       "      <td>public health research</td>\n",
       "      <td>interoperability benefits</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Quench the Question</td>\n",
       "      <td>Quest for a change from the Quiescence</td>\n",
       "      <td>dig deeper into that</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                    node_1  \\\n",
       "0             Health data interoperability   \n",
       "1                        Patient's consent   \n",
       "2                     Healthcare providers   \n",
       "3         Hospital management system (HMS)   \n",
       "4              Patient Record (PHR) system   \n",
       "5                          Diagnostic labs   \n",
       "6                               Pharmacies   \n",
       "7    Mobile Health and Wellness (HnW) apps   \n",
       "8  Government Healthcare Schemes like CGHS   \n",
       "9                      Quench the Question   \n",
       "\n",
       "                                            node_2  \\\n",
       "0            health information technology systems   \n",
       "1                              health data sharing   \n",
       "2                                        hospitals   \n",
       "3                   Electronic Health Record (EHR)   \n",
       "4                               Insurance coverage   \n",
       "5                              Insurance providers   \n",
       "6  Wellness Service providers (Gyms, Yoga centres)   \n",
       "7    Wearable devices (iPhone, Fitbit, Glucometer)   \n",
       "8                           public health research   \n",
       "9           Quest for a change from the Quiescence   \n",
       "\n",
       "                                                edge  \n",
       "0                      communicate and exchange data  \n",
       "1              regardless of the technology platform  \n",
       "2  sharing medical histories and other clinical i...  \n",
       "3                         manage healthcare services  \n",
       "4                   authorised providers access data  \n",
       "5                       health data interoperability  \n",
       "6            purview of Health Data Interoperability  \n",
       "7                                access patient data  \n",
       "8                          interoperability benefits  \n",
       "9                               dig deeper into that  "
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame(res)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "OpenAI@3111",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.1"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}