DSPy: The framework for programming—not prompting—language models
Python
updates to notebook
klopsahlongd366d629b0ec
diff --git a/examples/qa/hotpot/hotpotqa_optimized.ipynb b/examples/qa/hotpot/hotpotqa_optimized.ipynbnew file mode 100644index 000000000..e97102d97--- /dev/null+++ b/examples/qa/hotpot/hotpotqa_optimized.ipynb@@ -0,0 +1,666 @@+{+ "cells": [+ {+ "cell_type": "markdown",+ "metadata": {+ "id": "li3F9kMOqZHz"+ },+ "source": [+ "<img src=\"../../../docs/images/DSPy8.png\" alt=\"DSPy7 Image\" height=\"150\"/>"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "id": "3wEDck3ZqZH0"+ },+ "source": [+ "# Using __<ins>M</ins>ulti-stage <ins>I</ins>nstruction <ins>P</ins>roposal & <ins>O</ins>ptimization (MIPRO)__ in DSPy"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "id": "7DzBCQ0UqZH0"+ },+ "source": [+ "### FAQ 🙋\n",+ "#### 1) How does MIPRO work?\n",+ "At a high level, the MIPRO program optimizer works by first __proposing__ candidate fewshot example sets and instructions for each prompt in your program, and then __optimizing__ over these fewshot example sets and instructions as hyperparameters for a specified number of trials. Each trial, the optimizer evaluates different combinations of prompts on a train set, which allows it to learn which combinations yield the best performance.\n",+ "\n",+ "#### 2) How much will MIPRO cost me to run?\n",+ "Note that __this notebook__ is free to run, because all LM calls have been cached. However, when using an optimizer on your own program, here is a breakdown of the upper bound of the number of calls to the task model and prompt model respectively:\n",+ "\n",+ "- **Task model calls**: MIPRO makes up to __O(TxPxM)__ task model calls, where T is the number of trials, P is the number of prompts in the program, and M is the size of the train set. This is because the model is evaluating the program on the train set each trial. In practice, this should be lower given that MIPRO tunes poor trials early (ie. it may stop a trial after running on the first 100 or so examples if performance is poor).\n",+ "\n",+ "- **Prompt model calls**: MIPRO makes up to N*P+10 prompt model calls, where N is the number of instruction / fewshot example set candidates to generate for each prompt, and P is the number of prompts in the program. The extra 10 calls comes from generating a summary of the data in the training set, which we use in the meta prompt to create better instructions.\n",+ "\n",+ "#### 3) How should I configure the hyperparameters?\n",+ "We have yet to run full hyperparameter sweeps with MIPRO, but based off of initial experimintation, we'd recommend the following:\n",+ "- __Trial num__: Gains can be seen after about 20-30 trials. However, 100-200 trials can help with adding on additional marginal gains.\n",+ "- __n__: This hyperparameter controls the number of candidate prompts and fewshot example sets that are generated to optimize over. With more trials and less prompts to optimize, we can set n to be higher, as we have more trials to explore different combinations of prompts. If your program has between 2-3 modules and is the `num_trials=30`, we'd recommend ~`n=10`. If n is higher (say `n=100`), then we can go higher to ~`n=15`. If you have a program with only 1 module and are keeping the program 0-shot (ie. no fewshot examples), then `num_trials` should be set to equal `n`, because each trial can explore a new instruction.\n",+ "- __Training set size__: Between 200 and 500 training examples are recommended. Increasing the training set size can help prevent overfitting, but adds to the expense to run.\n",+ "\n",+ "#### 4) What should I do if I want to reduce the cost?\n",+ "You can always update hyperparameters accordingly, such as using a smaller train set, using less trials, or using a program with less modules.\n",+ "Alternatively, one strategy would be to optimize using a cheaper task model (ie. locally hosted Llama-2), as initial experiments have shown that prompts optimized for a smaller model also transfer to working well on a larger model.\n"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "id": "SgTc-CutqZH1"+ },+ "source": [+ "### 0] Setup"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "id": "rbGIXWcqqZH1"+ },+ "source": [+ "First, we'll __load in the cached requests__ for this tasks, so that we don't actually need to call any LMs for this notebook. We'll also load in our pre optimized program from hugging face to inspect later."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 1,+ "metadata": {+ "colab": {+ "base_uri": "https://localhost:8080/"+ },+ "id": "l4Fsh7EhqZH1",+ "outputId": "bc43f9ad-e090-4e5d-bd54-1e998225b44d"+ },+ "outputs": [],+ "source": [+ "from huggingface_hub import hf_hub_download\n",+ "import zipfile\n",+ "import os\n",+ "\n",+ "repo_id = 'kopsahlong/DSPy_MIPRO_notebook_cache'\n",+ "cache_file_path = hf_hub_download(repo_id=repo_id, filename='MIPRO_notebook_cache.zip')\n",+ "compiled_program_file_path = hf_hub_download(repo_id=repo_id, filename='compiled_program.pickle')\n",+ "with zipfile.ZipFile(cache_file_path, 'r') as zip_ref:\n",+ " zip_ref.extractall(\".\")\n",+ "os.environ[\"DSP_NOTEBOOK_CACHEDIR\"] = f\"{os.getcwd()}/MIPRO_notebook_cache\""+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "id": "5Vo4Tb9srSow"+ },+ "source": [+ "Next, we will install __DSPy__ if it's not there already."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 2,+ "metadata": {+ "colab": {+ "base_uri": "https://localhost:8080/"+ },+ "id": "JpijP_d7qZH2",+ "outputId": "daf24b9e-7030-4bf1-a08f-ff8b4ad42e22"+ },+ "outputs": [],+ "source": [+ "%load_ext autoreload\n",+ "%autoreload 2\n",+ "\n",+ "import sys\n",+ "import os\n",+ "import regex as re\n",+ "\n",+ "try: # When on google Colab, let's clone the notebook so we download the cache.\n",+ " import google.colab\n",+ " repo_path = 'dspy'\n",+ "\n",+ " !git -C $repo_path pull origin || git clone https://github.com/stanfordnlp/dspy $repo_path\n",+ "except:\n",+ " repo_path = '.'\n",+ "\n",+ "if repo_path not in sys.path:\n",+ " sys.path.append(repo_path)\n",+ "\n",+ "\n",+ "import pkg_resources # Install the package if it's not installed\n",+ "if not \"dspy-ai\" in {pkg.key for pkg in pkg_resources.working_set}:\n",+ " !pip install -U pip\n",+ " !pip install dspy-ai\n",+ " !pip install openai~=0.28.1\n",+ " !pip install -e $repo_path\n",+ " !pip install --upgrade cloudpickle==3.0.0\n",+ "\n",+ "import dspy"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "id": "U-DaFCBvqZH2"+ },+ "source": [+ "We will also specify the __prompt LM model__ (in this case GPT 3.5), the __task LM model__ (Llama 13B) and the retrieval model we'll be using for our task (a HotPotQA multihop retrieval task)."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 3,+ "metadata": {+ "id": "UHWzGRVgqZH2"+ },+ "outputs": [],+ "source": [+ "### NOTE: if you'd like to run this code without a cache, you can remove these lines to configure your OPEN AI key ###\n",+ "# os.environ['OPENAI_API_KEY'] = \"TODO: ADD YOUR OPEN AI KEY HERE\"\n",+ "# openai.api_key = os.environ.get('OPENAI_API_KEY')\n",+ "# openai.api_base = \"https://api.openai.com/v1\"\n",+ "\n",+ "prompt_model_name = \"gpt-3.5-turbo-1106\"\n",+ "task_model_name = \"meta-llama/Llama-2-13b-chat-hf\"\n",+ "colbert_v2_endpoint = \"http://20.102.90.50:2017/wiki17_abstracts\"\n",+ "\n",+ "prompt_model = dspy.OpenAI(model=prompt_model_name, max_tokens=150)\n",+ "task_model = dspy.HFClientTGI(model=task_model_name, port=[7140, 7141, 7142, 7143], max_tokens=150)\n",+ "\n",+ "colbertv2 = dspy.ColBERTv2(url=colbert_v2_endpoint)\n",+ "\n",+ "dspy.settings.configure(rm=colbertv2, lm=task_model)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "id": "BFoPwDrUqZH2"+ },+ "source": [+ "### 1] Define Task"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "id": "s4Cyb1JtqZH2"+ },+ "source": [+ "Here, we'll define the program that we'd like to run, which is a multihop [...] (we can say that it was loosely inspired by a certain paper). We additionally load in the data, and define how we'd like to evaluate this task."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 4,+ "metadata": {+ "colab": {+ "base_uri": "https://localhost:8080/"+ },+ "id": "hiVgd3N7qZH3",+ "outputId": "09e1ea66-7c8d-438a-8c37-1ab96fe8cdf0"+ },+ "outputs": [+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ "/lfs/0/kristaoo/miniconda3/envs/dspy_test/lib/python3.10/site-packages/datasets/table.py:1421: FutureWarning: promote has been superseded by mode='default'.\n",+ " table = cls._concat_blocks(blocks, axis=0)\n"+ ]+ }+ ],+ "source": [+ "import re\n",+ "from dspy.evaluate import Evaluate\n",+ "from dspy.datasets import HotPotQA\n",+ "from dsp.utils import EM\n",+ "\n",+ "class ReturnRankedDocuments(dspy.Signature):\n",+ " \"\"\"Given a question we are trying to answer and a list of passages, return a comma separated list of the numbers associated with each passage. These numbers should be ordered by helpfulness in answering the question, with most helpful passage number first, and the least helpful last.\"\"\"\n",+ " question = dspy.InputField(desc=\"The question we're trying to answer.\")\n",+ " context = dspy.InputField(desc=\"List of potentially related passages.\")\n",+ " ranking = dspy.OutputField(desc=\"A comma separated list of numbers corresponding to passage indices, ranked in descending order by their helpfulness in answering our question.\")\n",+ "\n",+ "class RankingMultiHop(dspy.Module):\n",+ " def __init__(self, hops, num_passages_to_retrieve, max_passages_in_context):\n",+ " super().__init__()\n",+ " self.hops = hops\n",+ " self.num_passages_to_retrieve = num_passages_to_retrieve\n",+ " self.max_passages_in_context = max_passages_in_context\n",+ " self.retrieve = dspy.Retrieve(k = self.num_passages_to_retrieve)\n",+ " self.generate_query = dspy.ChainOfThought(\"context ,question->search_query\")\n",+ " self.generate_answer = dspy.ChainOfThought(\"context ,question->answer\")\n",+ " self.generate_ranking = dspy.ChainOfThought(ReturnRankedDocuments)\n",+ "\n",+ " def forward(self,question):\n",+ " context = []\n",+ " full_context = []\n",+ " top_context = []\n",+ " max_passage_num = self.max_passages_in_context\n",+ " for hop in range(self.hops):\n",+ " # Get a new query\n",+ " query = self.generate_query(context = context, question = question).search_query\n",+ " # Get new passages\n",+ " context = self.retrieve(query).passages\n",+ " # Add these new passages to the previous top context\n",+ " full_context = top_context + context\n",+ " # Get the most important indices, ranked\n",+ " most_important_indices = self.generate_ranking(question=question, context=full_context).ranking\n",+ " indices = [int(num) for num in re.findall(r'\\d+', most_important_indices)]\n",+ "\n",+ " if len(indices) < max_passage_num:\n",+ " indices = range(1,max_passage_num+1)\n",+ "\n",+ " valid_indices = [index-1 for index in indices if index-1 < len(context)]\n",+ " top_indices = sorted(valid_indices, key=lambda x: x)[:max_passage_num+1]\n",+ " most_important_context_list = [context[idx] for idx in top_indices]\n",+ " # Save the top context\n",+ " top_context = most_important_context_list\n",+ "\n",+ " return dspy.Prediction(context=context, answer=self.generate_answer(context = top_context , question = question).answer)\n",+ "\n",+ "program = RankingMultiHop(hops=4, num_passages_to_retrieve=5, max_passages_in_context=5)\n",+ "\n",+ "# Load and configure the datasets.\n",+ "TRAIN_SIZE = 500\n",+ "EVAL_SIZE = 500\n",+ "\n",+ "hotpot_dataset = HotPotQA(train_seed=1, eval_seed=2023, test_size=0)\n",+ "trainset = [x.with_inputs('question') for x in hotpot_dataset.train][:TRAIN_SIZE]\n",+ "devset = [x.with_inputs('question') for x in hotpot_dataset.dev][:EVAL_SIZE]\n",+ "\n",+ "# Set up metrics\n",+ "NUM_THREADS = 10\n",+ "\n",+ "metric = dspy.evaluate.answer_exact_match\n",+ "\n",+ "kwargs = dict(num_threads=NUM_THREADS, display_progress=True)\n",+ "evaluate = Evaluate(devset=devset, metric=metric, **kwargs)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "id": "hRGld1C1qZH3"+ },+ "source": [+ "### 2] Baseline Evaluation\n",+ "Now, we'll quickly evaluate our baseline program so that we can see how the performance using the Prompt Optimizer compares. We should see performance of about __21.6%__ on our trainset, and __22.6%__ on our devset."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 5,+ "metadata": {+ "colab": {+ "base_uri": "https://localhost:8080/"+ },+ "id": "MU2aHQBTqZH3",+ "outputId": "fd60fbb3-ca89-4ecb-911b-24751f220cc6"+ },+ "outputs": [+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ "Average Metric: 108 / 500 (21.6): 100%|██████████| 500/500 [00:33<00:00, 14.90it/s]\n",+ "/lfs/0/kristaoo/dspy/dspy/evaluate/evaluate.py:145: FutureWarning: DataFrame.applymap has been deprecated. Use DataFrame.map instead.\n",+ " df = df.applymap(truncate_cell)\n"+ ]+ },+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "Average Metric: 108 / 500 (21.6%)\n"+ ]+ },+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ "Average Metric: 113 / 500 (22.6): 100%|██████████| 500/500 [00:33<00:00, 15.00it/s]\n"+ ]+ },+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "Average Metric: 113 / 500 (22.6%)\n"+ ]+ }+ ],+ "source": [+ "baseline_train_score = evaluate(program,devset=trainset)\n",+ "baseline_eval_score = evaluate(program, devset=devset)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "id": "cjCoL27yqZH3"+ },+ "source": [+ "### 3] Optimizing with MIPRO"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "id": "eEGOKjXprc7z"+ },+ "source": [+ "Now let's get into the key method in this notebook - optimizing our program with MIPRO!"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "id": "G04L5j9iqZH3"+ },+ "source": [+ "#### 3a] Compile Program\n",+ "First, we'll get our optimized program. By default, we set `LOAD_PRECOMPILED_PROGRAM` to `True`, so that you can quickly access a program we've precompiled for you. However, if you wish to optimize yourself, `LOAD_PRECOMPILED_PROGRAM` can be set to `False` (though please note that this will require adding in your own LM API keys in the __Setup__ section above).\n",+ "\n",+ "MIPRO only needs a metric, DSPy module, and training set to see huge gains on your task! You can instantiate a MIPRO Optimizer and compile in just two lines:\n",+ "```python\n",+ "teleprompter = BayesianSignatureOptimizer(prompt_model=prompt_model, task_model=task_model, metric=metric, num_candidates=N, init_temperature=temperature)\n",+ "compiled_program = teleprompter.compile(program, trainset=trainset, num_trials=trials, max_bootstrapped_demos=1,max_labeled_demos=2, eval_kwargs=eval_kwargs)\n",+ "```"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 6,+ "metadata": {+ "colab": {+ "base_uri": "https://localhost:8080/"+ },+ "id": "TpFW7IaYqZH3",+ "outputId": "7125bbed-b79e-4178-c390-fc81cc698106"+ },+ "outputs": [+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "/lfs/0/kristaoo/dspy/examples/qa/hotpot/MIPRO_notebook_cache/compiler\n"+ ]+ }+ ],+ "source": [+ "import cloudpickle as pickle\n",+ "from dspy.teleprompt import BayesianSignatureOptimizer\n",+ "\n",+ "LOAD_PRECOMPILED_PROGRAM = True\n",+ "\n",+ "# By default, we will load the precompiled program\n",+ "if LOAD_PRECOMPILED_PROGRAM:\n",+ " # Load a our precompiled program\n",+ " with open(compiled_program_file_path, 'rb') as file:\n",+ " # Load the data from the file\n",+ " compiled_program = pickle.load(file)\n",+ "# Otherwise, if desired, the program can be compiled from scratch\n",+ "else:\n",+ " # Define hyperparameters:\n",+ " N = 10 # The number of instructions and fewshot examples that we will generate and optimize over\n",+ " trials = 30 # The number of optimization trials to be run (we will test out a new combination of instructions and fewshot examples in each trial) \n",+ " temperature = 1.0 # The temperature configured for generating new instructions\n",+ "\n",+ " # Compile\n",+ " eval_kwargs = dict(num_threads=16, display_progress=True, display_table=0)\n",+ " teleprompter = MIPRO(prompt_model=prompt_model, task_model=task_model, metric=metric, num_candidates=N, init_temperature=temperature, verbose=True)\n",+ " compiled_program = teleprompter.compile(program, trainset=trainset, trials_num=trials, max_bootstrapped_demos=1,max_labeled_demos=2, eval_kwargs=eval_kwargs)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "id": "uqVVnaEBqZH3"+ },+ "source": [+ "#### 3b] Evaluate optimized program\n",+ "Now, we evaluate our program that has been optimized with MIPRO. We see that performance on train and dev have improved by __+20pt__ and __+17.8pt__ respectively!"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 7,+ "metadata": {+ "colab": {+ "base_uri": "https://localhost:8080/"+ },+ "id": "VvnBp7huqZH3",+ "outputId": "fdba6cfe-51b4-4ea1-f5af-9c2a5fed2064"+ },+ "outputs": [+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ "Average Metric: 208 / 500 (41.6): 100%|██████████| 500/500 [00:32<00:00, 15.27it/s]\n"+ ]+ },+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "Average Metric: 208 / 500 (41.6%)\n"+ ]+ },+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ "Average Metric: 202 / 500 (40.4): 100%|██████████| 500/500 [00:33<00:00, 15.04it/s]"+ ]+ },+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "Average Metric: 202 / 500 (40.4%)\n"+ ]+ },+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ "\n"+ ]+ }+ ],+ "source": [+ "bayesian_train_score = evaluate(compiled_program, devset=trainset)\n",+ "bayesian_eval_score = evaluate(compiled_program, devset=devset)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "id": "j3UWn_UnqZH4"+ },+ "source": [+ "#### 3c] Visualizing scores & prompts over trials"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "id": "pBYTLTwWqZH4"+ },+ "source": [+ "Now, let's take a look at how this optimization looked over the course of each trial. We see that, in general, __performance increases overtime__, until it saturates after ~trial #13. Note that some of the 'pruned' trials have high scores, but were pruned early because they had comparitively lower scores on the easier slices of the data."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 8,+ "metadata": {+ "colab": {+ "base_uri": "https://localhost:8080/",+ "height": 333+ },+ "id": "rtMUNeicqZH4",+ "outputId": "453b554f-4654-4184-d125-758288ddd6ee"+ },+ "outputs": [+ {+ "data": {+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAckAAAE8CAYAAACrYErbAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/H5lhTAAAACXBIWXMAAA9hAAAPYQGoP6dpAABL40lEQVR4nO3deVyU1f4H8M+A7MOAKJuxqgio4IJaZCCaC26pYGZ6u+JWFpRmWnq7Jtri0q1cMuqWqTfDXC5atxQ1V3BLyTWJlNCxRDFKB0TZ5vz+8DXzcxwGmGGGGZjP+/XipfOs3zPPzHyf5zznnEcihBAgIiIiLTbmDoCIiMhSMUkSERHpwCRJRESkA5MkERGRDkySREREOjBJEhER6cAkSUREpAOTJBERkQ5MkkRERDowSVKt4uLiEBcXZ9C6EokEqampRo3HmunzfkokEqSkpJg2oEa2du1aSCQSXLp0ydyhkBVhkmzmJBJJvf72799vthhv3LiB6dOnIywsDE5OTvDy8kKvXr3w2muvobS01GxxWbrDhw8jNTUVN2/eNOp2L126pPHZsLW1RUBAAEaNGoVTp04ZdV9NSXZ2NgYPHoyHHnoIjo6OCAgIwPDhw5Genq5epqysDKmpqQ36PpnquJJhWpg7ADKtL774QuP1f/7zH+zevVtrenh4eI3r79q1y2SxAcCff/6JHj16QKFQYNKkSQgLC0NxcTHOnDmDtLQ0PP/885BKpSaNoam4c+cOWrT4/6/s4cOHsWDBAiQlJcHd3d3o+3v66acxZMgQVFdXIzc3F2lpadixYweOHj2Krl27Gn1/dXnmmWcwduxYODg4NPq+N2/ejKeeegpdu3bF9OnT0bJlSxQUFODgwYP49NNPMW7cOAD3kuSCBQsAwOAaGFMfV9IPk2Qz97e//U3j9dGjR7F7926t6Q8qKyuDs7Mz7O3tTRkeVq9eDblcjkOHDuHRRx/VmKdQKEy+//vdvn0bLi4ujbY/fTk6Ojbq/rp3767xOenduzeeeOIJpKWl4ZNPPqlxHVO+h7a2trC1tTXJtuuSmpqKjh074ujRo1qfyaKiIrPERI2D1a2EuLg4dO7cGTk5OYiNjYWzszP+8Y9/qOfdf0ZcUVGBN954A1FRUXBzc4OLiwtiYmKwb98+g/adn58PW1tbPPLII1rzZDKZVmI4duwYhgwZgpYtW8LFxQWRkZFYvny5xjJ79+5FTEwMXFxc4O7ujhEjRiA3N1djmdTUVEgkEpw/fx7jxo1Dy5Yt8dhjj6nnr1+/HlFRUXBycoKHhwfGjh2LK1euaGzjwoULSExMhI+PDxwdHeHn54exY8fi1q1bOsu7YsUK2NraalSlvffee5BIJJg5c6Z6WnV1NVxdXfHaa6+pp91/TzI1NRWzZ88GAAQHB6urRh+8X7dt2zZ07twZDg4O6NSpEzIzM3XGVpd+/foBAAoKCgD8/z3CAwcO4IUXXoCXlxf8/PwAAElJSQgKCtLahup9v5/q/mldsdZ0TzIoKAjDhg1DdnY2evXqBUdHR7Rt2xb/+c9/tPZ95swZ9OnTB05OTvDz88Nbb72FNWvW1Os+Z35+Pnr27FnjSZuXlxeAe9XUnp6eAIAFCxaoj4nqmJ05cwZJSUlo27YtHB0d4ePjg0mTJqG4uFjj/dF1XFXV4GvXrtWK4cH71SUlJZgxYwaCgoLg4OAALy8vDBgwAD/++GOt5SRtvJIkAEBxcTEGDx6MsWPH4m9/+xu8vb1rXE6hUOCzzz7D008/jalTp6KkpASrV6/GoEGD8MMPP+hdDRcYGIjq6mp88cUXmDBhQq3L7t69G8OGDYOvry+mT58OHx8f5Obm4ttvv8X06dMBAN9//z0GDx6Mtm3bIjU1FXfu3MHKlSvRu3dv/Pjjj1o/3E8++SRCQkLwzjvvQPXUuLfffhvz5s3DmDFjMGXKFNy4cQMrV65EbGwsTp48CXd3d1RUVGDQoEEoLy/Hiy++CB8fH/z+++/49ttvcfPmTbi5udVYhpiYGCiVSmRnZ2PYsGEAgKysLNjY2CArK0u93MmTJ1FaWorY2Ngat5OQkIBffvkFGzZswAcffIDWrVsDgPpHGrh3Dy0jIwMvvPACXF1dsWLFCiQmJkIul6NVq1a1vtc1yc/PBwCtdV944QV4enrijTfewO3bt/XebkNjvXjxIkaPHo3JkydjwoQJ+Pzzz5GUlISoqCh06tQJAPD777+jb9++kEgkmDt3LlxcXPDZZ5/Vu+o2MDAQe/bswW+//aY+EXiQp6en+hbBqFGjkJCQAACIjIwEcO/z++uvv2LixInw8fHBTz/9hH//+9/46aefcPToUUgkklqP640bN+oVKwBMmzYNW7ZsQUpKCjp27Iji4mJkZ2cjNzcX3bt3r/d2CIAgq5KcnCwePOx9+vQRAMTHH3+stXyfPn1Enz591K+rqqpEeXm5xjJ//fWX8Pb2FpMmTdKYDkDMnz+/1niuXbsmPD09BQARFhYmpk2bJtLT08XNmzc1lquqqhLBwcEiMDBQ/PXXXxrzlEql+v9du3YVXl5eori4WD3t9OnTwsbGRvz9739XT5s/f74AIJ5++mmNbV26dEnY2tqKt99+W2P62bNnRYsWLdTTT548KQCIzZs311q+B1VXVwuZTCZeffVVdeytWrUSTz75pLC1tRUlJSVCCCHef/99YWNjo1HWB9/Pd999VwAQBQUFWvsBIOzt7cXFixc13gcAYuXKlbXGWFBQIACIBQsWiBs3bohr166J/fv3i27dugkA4r///a8QQog1a9YIAOKxxx4TVVVVGtuYMGGCCAwM1Nq26n03JFbV/u4vb2BgoAAgDh48qJ5WVFQkHBwcxCuvvKKe9uKLLwqJRCJOnjypnlZcXCw8PDx0vof3W716tTrOvn37innz5omsrCxRXV2tsdyNGzd0fu7Lysq0pm3YsEErfl3HVXVc1qxZo7WdB/fp5uYmkpOTay0T1Q+rWwkA4ODggIkTJ9a5nK2trbrKSalU4s8//0RVVRV69OhhUFWOt7c3Tp8+jWnTpuGvv/7Cxx9/jHHjxsHLywtvvvmm+uru5MmTKCgowIwZM7QaM6iq7woLC3Hq1CkkJSXBw8NDPT8yMhIDBgzA9u3btfY/bdo0jdcZGRlQKpUYM2YM/vjjD/Wfj48PQkJC1NXKqivFnTt3oqysrN7ltbGxwaOPPoqDBw8CAHJzc1FcXIw5c+ZACIEjR44AuHd12blz5wY13Ojfvz/atWunfh0ZGQmZTIZff/21XuvPnz8fnp6e8PHxQVxcHPLz87FkyRL1FZLK1KlTG3yvsCGxduzYETExMerXnp6eCA0N1Vg3MzMT0dHRGjUdHh4eGD9+fL3imzRpEjIzMxEXF4fs7Gy8+eabiImJQUhICA4fPlyvbTg5Oan/f/fuXfzxxx/q2wzGrgZ1d3fHsWPHcPXqVaNu1xoxSRIA4KGHHqp3I5l169YhMjISjo6OaNWqFTw9PfHdd9/Vei+uNr6+vkhLS0NhYSHy8vKwYsUKdfXd6tWrAfx/VV/nzp11bufy5csAgNDQUK154eHh+OOPP7SqA4ODgzVeX7hwAUIIhISEwNPTU+MvNzdX3UgjODgYM2fOxGeffYbWrVtj0KBBWLVqVb3eg5iYGOTk5ODOnTvIysqCr68vunfvji5duqirXLOzszV++A0REBCgNa1ly5b466+/6rX+s88+i927d2PPnj3IyclBUVERXn31Va3lHnwPGzvW+qx7+fJltG/fXmu5mqbpMmjQIOzcuRM3b97EwYMHkZycjMuXL2PYsGH1arzz559/Yvr06fD29oaTkxM8PT3V752h3x1dli5dinPnzsHf3x+9evVCampqvU+OSBPvSRIAzbPc2qxfvx5JSUkYOXIkZs+eDS8vL9ja2mLRokXqRGYoiUSCDh06oEOHDhg6dChCQkLw5ZdfYsqUKQ3abm0eLLdSqYREIsGOHTtqvDq6vzvKe++9h6SkJHz99dfYtWsXXnrpJSxatAhHjx7Ved8KAB577DFUVlbiyJEjyMrKUifDmJgYZGVl4eeff8aNGzcanCR1Xd2prs7rEhISgv79+9e5XE2fnQcb56hUV1fXOL0hsTa0nPpydnZGTEwMYmJi0Lp1ayxYsAA7duyo8576mDFjcPjwYcyePRtdu3aFVCqFUqlEfHw8lEplnfvV5z0dM2YMYmJisHXrVuzatQvvvvsulixZgoyMDAwePLh+BSUATJKkpy1btqBt27bIyMjQ+NLOnz/fqPtp27YtWrZsicLCQgBQV8WdO3dO5w93YGAgACAvL09r3s8//4zWrVvX2T2hXbt2EEIgODgYHTp0qDPOiIgIRERE4J///CcOHz6M3r174+OPP8Zbb72lc51evXrB3t4eWVlZyMrKUrdmjI2Nxaeffoo9e/aoX9dG14+mJWjZsmWNneFVV/uNLTAwEBcvXtSaXtM0ffTo0QMA1J9TXcfkr7/+wp49e7BgwQK88cYb6ukXLlzQWlbXNlq2bAkAWu+rrvfU19cXL7zwAl544QUUFRWhe/fuePvtt5kk9cTqVtKL6qz9/rP0Y8eOqe+l6evYsWM1toj84YcfUFxcrK467d69O4KDg7Fs2TKtHwlVLL6+vujatSvWrVunscy5c+ewa9cuDBkypM54EhISYGtriwULFmhdiQgh1M31FQoFqqqqNOZHRETAxsYG5eXlte7D0dERPXv2xIYNGyCXyzWuJO/cuYMVK1agXbt28PX1rXU7qoRviSOztGvXDrdu3cKZM2fU0woLC7F161azxDNo0CAcOXJEY8SgP//8E19++WW91leduDxIdZ9b9Tl1dnYGoH1MavreAMCyZcu0tqnruMpkMrRu3Vp9P1vlo48+0nhdXV2tVX3r5eWFNm3a1PnZJG28kiS9DBs2DBkZGRg1ahSGDh2KgoICfPzxx+jYsaNBQ8h98cUX+PLLLzFq1ChERUXB3t4eubm5+Pzzz+Ho6Kjur2ljY4O0tDQMHz4cXbt2xcSJE+Hr64uff/4ZP/30E3bu3AkAePfddzF48GBER0dj8uTJ6i4gbm5u9Rr3tF27dnjrrbcwd+5cXLp0CSNHjoSrqysKCgqwdetWPPvss5g1axb27t2LlJQUPPnkk+jQoQOqqqrwxRdfwNbWFomJiXXuJyYmBosXL4abmxsiIiIA3PshCw0NRV5eHpKSkurcRlRUFADg9ddfx9ixY2FnZ4fhw4dbxIAIY8eOxWuvvYZRo0bhpZdeQllZGdLS0tChQwez9NV79dVXsX79egwYMAAvvviiugtIQEAA/vzzzzqvykeMGIHg4GAMHz4c7dq1w+3bt/H999/jf//7H3r27Inhw4cDuFf13LFjR2zcuBEdOnSAh4cHOnfujM6dOyM2NhZLly5FZWUlHnroIezatUvd5/R+tR3XKVOmYPHixZgyZQp69OiBgwcP4pdfftFYv6SkBH5+fhg9ejS6dOkCqVSK77//HsePH8d7771npHfUipinUS2Zi64uIJ06dapx+Qe7gCiVSvHOO++IwMBA4eDgILp16ya+/fbbGpv8ox5dQM6cOSNmz54tunfvLjw8PESLFi2Er6+vePLJJ8WPP/6otXx2drYYMGCAcHV1FS4uLiIyMlKrS8P3338vevfuLZycnIRMJhPDhw8X58+f11hG1RXhxo0bNcb13//+Vzz22GPCxcVFuLi4iLCwMJGcnCzy8vKEEEL8+uuvYtKkSaJdu3bC0dFReHh4iL59+4rvv/++1vKqfPfddwKAGDx4sMb0KVOmCABi9erVWuvU9H6++eab4qGHHhI2NjYa3QYA1NgFIDAwUEyYMKHW2FRdDd59991al1N1yTh+/HiN83ft2iU6d+4s7O3tRWhoqFi/fr3OLiD1iVVXF5ChQ4dqrfvg51aIe912YmJihIODg/Dz8xOLFi0SK1asEADEtWvXai3rhg0bxNixY0W7du2Ek5OTcHR0FB07dhSvv/66UCgUGssePnxYREVFCXt7e41j9ttvv4lRo0YJd3d34ebmJp588klx9epVvY5rWVmZmDx5snBzcxOurq5izJgxoqioSGMb5eXlYvbs2aJLly7q70mXLl3ERx99VGsZqWYSIUx0d5uIyMLNmDEDn3zyCUpLS8025B1ZNt6TJCKrcOfOHY3XxcXF+OKLL/DYY48xQZJOvCdJRFYhOjoacXFxCA8Px/Xr17F69WooFArMmzfP3KGRBWOSJCKrMGTIEGzZsgX//ve/IZFI0L17d6xevbrOrjZk3XhPkoiISAfekyQiItKBSZKIiEiHZn9PUqlU4urVq3B1dbXoYbyIiMh0hBAoKSlBmzZtYGNT/+vDZp8kr169Cn9/f3OHQUREFuDKlSu1PoDgQc0+Sbq6ugK498bIZDKDt1NZWYldu3Zh4MCBsLOzM1Z4FsvayguwzNZQZmsrL8Ayq8qsUCjg7++vzgn11eyTpKqKVSaTNThJOjs7QyaTWcUHzdrKC7DM1lBmaysvwDI/WGZ9b7ux4Q4REZEOTJJEREQ6NPvqViJqGqqV1ciSZ6GwpBC+rr6ICYiBrQ3HVCXzYpIkIrPLyM3A9Mzp+E3xm3qan8wPy+OXIyE8wYyRkbVjdSsRmVVGbgZGbxqtkSAB4HfF7xi9aTQycjPMFBkRkyQRmVG1shrTM6dDQHsIadW0GZkzUK2sbuzQiAAwSRKRGWXJs7SuIO8nIHBFcQVZ8qxGjIro/zFJEpHZFJYUGnU5ImNjkiQis/F19TXqckTGxiRJRGYTExADP5kfJKh5FBQJJPCX+SMmIKaRIyO6h0mSiMzG1sYWy+OXA4BWolS9Xha/jP0lyWyYJInIrBLCE7BlzBY8JHtIY7qfzA9bxmxhP0kyK4tJkosXL4ZEIsGMGTPU0+7evYvk5GS0atUKUqkUiYmJuH79uvmCJCKTSAhPwKXpl7Bvwj6kJ6Rj34R9KJhewARp4aqV1dh/aT82nN2A/Zf216urjr7rGLIPY7KIEXeOHz+OTz75BJGRkRrTX375ZXz33XfYvHkz3NzckJKSgoSEBBw6dMhMkRKRqdja2CIuKM7cYVA9GTJKkr7rWMJITGa/kiwtLcX48ePx6aefomXLlurpt27dwurVq/H++++jX79+iIqKwpo1a3D48GEcPXrUjBETEVk3Q0ZJ0ncdSxmJyexXksnJyRg6dCj69++Pt956Sz09JycHlZWV6N+/v3paWFgYAgICcOTIETzyyCM1bq+8vBzl5eXq1wqFAsC954tVVlYaHKdq3YZsoymxtvICLLM1sLbyAsYvc7WyGq/tfA2ONo41zpdAgjk752BI2yHqBlf6rmPIPu5XU5kNLb9ECKE9HlQj+eqrr/D222/j+PHjcHR0RFxcHLp27Yply5YhPT0dEydO1Eh4ANCrVy/07dsXS5YsqXGbqampWLBggdb09PR0ODs7m6QcRERk2crKyjBu3DjcunULMpms3uuZ7UryypUrmD59Onbv3g1Hx5rPFgwxd+5czJw5U/1aoVDA398fAwcO1OuNeVBlZSV2796NAQMGWMXTva2tvADLbA1ltrbyAsYv85bzWzD5m8l1Lrf6idUY3XG0QesYso/71VRmVa2ivsyWJHNyclBUVITu3burp1VXV+PgwYP48MMPsXPnTlRUVODmzZtwd3dXL3P9+nX4+Pjo3K6DgwMcHBy0ptvZ2RnlA2Ks7TQV1lZegGW2BtZWXsB4ZfZ188Ud5Z16Lafan77rGLKPmtxfZkPLbraGO48//jjOnj2LU6dOqf969OiB8ePHq/9vZ2eHPXv2qNfJy8uDXC5HdHS0ucImIrJqhoySpO86ljQSk9mSpKurKzp37qzx5+LiglatWqFz585wc3PD5MmTMXPmTOzbtw85OTmYOHEioqOjdTbaISIi0zJklCR917GkkZjM3gWkNh988AGGDRuGxMRExMbGwsfHBxkZfABrc2DuDsJEZDhDRknSdx1LGYnJ7F1A7rd//36N146Ojli1ahVWrVplnoDIJCyhg3Bjq1ZWI0uehcKSQvi6+iImIIbjkVKTlhCegBGhI/T6XOu7jiH7MDaLSpLU/Kk6CD/4JHpVB+HmOFanNZ4UkHUwZJQkfdcx90hMFl3dSs1LtbIa0zOnayVIAOppMzJnNKuqV0sZNYSIDMMkSY0mS56llSzuJyBwRXEFWfKsRozKdBr7pID3eYmMj9Wt1GgKSwqNupyl0+ekoKHVSazSNQ1T30vmvWrLxyRJjcbX1deoy1m6xjopsMb7vI3B1CcePLFpGljdSo3GkjoIN1R9qjYb46SgMat0rak619T3knmvuulgkqRGY0kdhBsiIzcDQcuD0HddX4zLGIe+6/oiaHmQ1g9bY5wUNNZ93vqWuTkw9YmHNTZga8qYJKlRWUoHYUPpcwXQGCcFjVGla21XPaY+8bC2BmxNHZMkNbqE8ARcmn4J+ybsQ3pCOvZN2IeC6QUWnyANuQIw9UmBqat0rfGqx9QnHtbWgK2pY8MdMgtzdxA2hKGtVU05aoiqSvd3xe81JjIJJPCT+RlcpduYLXQthalPPKytAVtTxyRJVE8NuQIw1UmBqkp39KbRkECikSiNUaVrjVc9pj7xMPX2ybhY3UpNgiW0rLTUKwBTVulaaplNydT3kptLAzZrwStJ0lKtrEa2PBsAkC3PRmxwrFm/sJbSn8ySrwBMVaVryWU2JdWJR02fu2Xxyxr8uTP19sl4mCRJgyohFZcWY0PkBgxNH4pW0lZm6+BsSR3lTV21aYz49KnSrc/JkCWX2dSj1Zj6CRSW8IQLqhurW+vhwR+Tuqr6LKFq0BCW1tTfEltWNvUuLCqqfo9D04cCAIamD9XZ79ESy9xY/TZVJx5PRzyNuKA4oycwU2+fGo5XknXQ98rKUqoG9VVXQpJAghmZMzAidESjfZEttWVlU78CuP/q3MnGST29tqtzSyqzJdUuNDdKpRJyuRwlJSVwdXVFQEAAbGys+1qKSbIW+v6YNOUvryUmJEtuWdkUu7AADTsZsoQyW+LJXHORm5uLzMxMKBQK9TSZTIb4+HiEh4ebMTLzsu5ThFroW9VniVWD+rDEhGSNLStNramP9tLU47dUubm52LRpk0aCBACFQoFNmzYhNzfXTJGZH5OkDvp+GZv6l9cSE1JjDoiu733npsoST4b00dTjt0RKpRKZmZm1LpOZmQmlUtlIEVkWJkkd9P0yNvUvryU+oaOx+pPp04ilsZiq8Zclngzpo6nHb4nkcrnWFeSDFAoF5HJ5I0VkWZgkddD3y9jUv7yW2sHZ1C0rLa1FryomU7XctMSTIX009fgtUUlJiVGXa26YJHXQ98vYHL68ltjUXxWXKQZEt8T7yKZO2pZ6MlRfTT1+S+Tq6mrU5ZobJkkd9P0yNpcvryohfTfuOwDAd+O+s4gndJiiP5ml3UdurKRtqSdD9dXU47c0AQEBkMlktS4jk8kQEBDQSBFZFibJWuj7ZTT0y2tpgw/Y2tjisYDHAACPBTxm8YndUJZ2H7kxk7alngzVV1N93JolsrGxQXx8fK3LxMfHW21/SfaTrIOqE/XBgoNQnFPgu3Hf1TqWqb6drpvq4APNgaXdR27spK06Gdp+bnuTPBmyhH6bzUV4eDjGjBnDfpI1YJKsB31/TOr75W3Kgw80B5Y2eLelJW2yLuHh4QgNDeWIOw+w7tKbkSU2GmlOlEolLl26hLNnz+LSpUs19vGytPvIzaHxF1kW1TBzwL2uHnX1dbSxsUFQUBAiIiIQFBRk9QkS4JWk2VjiMHDNhT7Da93/yKLi0mL1dHM8ssiSn7hBTY/qe1BaWorIyEikp6dDKpVaffWpvniaYCaW1mikuTBkeC1LasTClptkDBxmznh4JWkmvP9kfPUdXis0NFSrGsmSGrFY0hM3qOlpyPeAtDFJmomlNRppDvQZXisoKKhxgjIQW26SoZrT98AS8DTCTCyt0UhzwOG1iPg9MDYmSTPi/Sfj4vBaRPweGBurW82M95+MRzW8Vm1VTdY8vBZZB34PjItXkhbAFOOS3s/Shr0zFQ6vRcTvgbHxXWrmTPnYJUukGl7rwQGbZTIZxowZw/5hZBX4PTAeVrc2Y9Y67B2H1yL6/+9BQUEBzp07h3HjxiE4OJjfAz3x3WqmrH3YOw6vRXTve6C698gTRcPwHWumLO1ZiURETRGTZDPFYe+IiBqOSbKZ4rB3REQNxyTZTPGxS0REDcck2Uxx2Dui5qc+z0kl4zJrkkxLS0NkZCRkMhlkMhmio6OxY8cO9fy7d+8iOTkZrVq1glQqRWJiIq5fv27GiJsWDntH1Hzk5uZi+fLlWLduHTIyMrBu3TosX76cj70yMbP2k/Tz88PixYsREhICIQTWrVuHESNG4OTJk+jUqRNefvllfPfdd9i8eTPc3NyQkpKChIQEHDp0yJxhNykc9o6o6VM9H/JBqudDcoAA0zFrkhw+fLjG67fffhtpaWk4evQo/Pz8sHr1aqSnp6Nfv34AgDVr1iA8PBxHjx7FI488Yo6QmyQ+domo6eLzIc3LYkbcqa6uxubNm3H79m1ER0cjJycHlZWV6N+/v3qZsLAwBAQE4MiRIzqTZHl5OcrLy9WvVYP8VlZWorKy0uD4VOs2ZBtNibWVF2CZrUFTLK9cLkdpaWmtCbC0tBQFBQU1DlreFMvcUDWV2dDyS4QQ2kOyNKKzZ88iOjoad+/ehVQqRXp6OoYMGYL09HRMnDhRI+EBQK9evdC3b18sWbKkxu2lpqZiwYIFWtPT09Ph7OxskjIQEZFlKysrw7hx43Dr1i2tMW1rY/YrydDQUJw6dQq3bt3Cli1bMGHCBBw4cMDg7c2dOxczZ85Uv1YoFPD398fAgQP1emMeVFlZid27d2PAgAGws7MzeDtNhbWVF2CZjV1mpVKJ3377DaWlpZBKpfDz8zN7dWBTPMZyuRzp6el1Ljdu3DidV5JNrcwNVVOZa3t0WG3MniTt7e3Rvn17AEBUVBSOHz+O5cuX46mnnkJFRQVu3rwJd3d39fLXr1+Hj4+Pzu05ODjAwcFBa7qdnZ1RPiDG2k5TYW3lBVhmY8jNzUVmZqbGD5NMJkN8fLxFNDBpSsc4ODgYUqm0zudD1jV4eVMqs7HcX2ZDy25xd3mVSiXKy8sRFRUFOzs77NmzRz0vLy8Pcrkc0dHRZoyQiGqjaon54I+6qiUmuyzoh8+HNC+zXknOnTsXgwcPRkBAAEpKSpCeno79+/dj586dcHNzw+TJkzFz5kx4eHhAJpPhxRdfRHR0NFu2ElkotsQ0DdXzIS356ry5MmuSLCoqwt///ncUFhbCzc0NkZGR2LlzJwYMGAAA+OCDD2BjY4PExESUl5dj0KBB+Oijj8wZMhHVQi6X13nvR6FQQC6XIygoqHGCaib4nFTzMGuSXL16da3zHR0dsWrVKqxataqRIiKihigpKTHqcqRJ9ZxUajw8BSEio3F1dTXqckTmxiRJREYTEBBQZ1crmUxWY1cFIkvEJElERsOWmNTc8JNKREalaon54BWlTCbjQNzU5Jh9MAEian7YEpOaCyZJIjIJtsSk5oCndURERDowSRIREenAJElERKQDkyQREZEOTJJEREQ6MEkSERHpwCRJRESkA5MkERGRDkySREREOjBJEhER6cAkSUREpAOTJBERkQ4NSpIVFRXIy8tDVVWVseIhIiKyGAYlybKyMkyePBnOzs7o1KkT5HI5AODFF1/E4sWLjRogERGRuRiUJOfOnYvTp09j//79cHR0VE/v378/Nm7caLTgiIiIzMmg50lu27YNGzduxCOPPAKJRKKe3qlTJ+Tn5xstOGoalEolH65LRM2SQUnyxo0b8PLy0pp++/ZtjaRJzV9ubi4yMzOhUCjU02QyGeLj4xEeHm7GyIiIGs6g0/0ePXrgu+++U79WJcbPPvsM0dHRxomMLF5ubi42bdqkkSABQKFQYNOmTcjNzTVTZERExmHQleQ777yDwYMH4/z586iqqsLy5ctx/vx5HD58GAcOHDB2jGSBlEolMjMza10mMzMToaGhrHoloibLoF+vxx57DKdPn0ZVVRUiIiKwa9cueHl54ciRI4iKijJ2jGSB5HK51hXkgxQKhbrlMxFRU6T3lWRlZSWee+45zJs3D59++qkpYqI6VCurkSXPQmFJIXxdfRETEANbG9tGjaGkpMSoyxERWSK9ryTt7Ozw3//+1xSxUD1k5GYgaHkQ+q7ri3EZ49B3XV8ELQ9CRm5Go8bh6upq1OWIiCyRQdWtI0eOxLZt24wcCtUlIzcDozeNxm+K3zSm/674HaM3jW7URBkQEACZTFbrMjKZDAEBAY0UERGR8RnUcCckJAQLFy7EoUOHEBUVBRcXF435L730klGCo/9XrazG9MzpEBBa8wQEJJBgRuYMjAgd0ShVrzY2NoiPj8emTZt0LhMfH89GO0TUpBmUJFevXg13d3fk5OQgJydHY55EImGSNIEseZbWFeT9BASuKK4gS56FuKC4RokpPDwcY8aMYT9JImq2DEqSBQUFxo6D6lBYUmjU5YwlPDwcoaGhHHGHiJolg5Lk/YS4V/3HkXZMy9fV16jLGZONjQ2CgoIafb9ERKZm8On+f/7zH0RERMDJyQlOTk6IjIzEF198YczY6D4xATHwk/lBgppPRiSQwF/mj5iAmEaOjIio+TIoSb7//vt4/vnnMWTIEGzatAmbNm1CfHw8pk2bhg8++MDYMRIAWxtbLI9fDgBaiVL1eln8skbvL0lE1JwZVN26cuVKpKWl4e9//7t62hNPPIFOnTohNTUVL7/8stECpP+XEJ6ALWO2YHrmdI1GPH4yPyyLX4aE8AQzRkdE1PwYlCQLCwvx6KOPak1/9NFHUVjYuA1HrE1CeAJGhI4w+4g7RETWwKDq1vbt29fYP27jxo0ICQlpcFBUO1sbW8QFxeHpiKcRFxTHBElUD6rnngL3xh5WKpVmjoiaAoOuJBcsWICnnnoKBw8eRO/evQEAhw4dwp49e2rtXE5EZA6q556WlpYiMjIS6enpkEql7M9LdTLoSjIxMRHHjh1D69atsW3bNmzbtg2tW7fGDz/8gFGjRhk7RiIig/G5p9QQBveTjIqKwvr1640ZCxGRUfG5p9RQBn0qtm/fjp07d2pN37lzJ3bs2NHgoIiIjIHPPaWGMihJzpkzB9XV1VrThRCYM2dOg4MiIjIGPveUGsqgJHnhwgV07NhRa3pYWBguXrzY4KCIiIyBzz2lhjIoSbq5ueHXX3/Vmn7x4kWtx2bVZtGiRejZsydcXV3h5eWFkSNHIi8vT2OZu3fvIjk5Ga1atYJUKkViYiKuX79uSNhEZGX43FNqKIOS5IgRIzBjxgzk5+erp128eBGvvPIKnnjiiXpv58CBA0hOTsbRo0exe/duVFZWYuDAgbh9+7Z6mZdffhn/+9//sHnzZhw4cABXr15FQgJHliGiuqmee1obPveUamNQ69alS5ciPj4eYWFh8PPzAwBcuXIFsbGx+Ne//lXv7TzY6mzt2rXw8vJCTk4OYmNjcevWLaxevRrp6eno168fAGDNmjUIDw/H0aNH8cgjjxgSPhFZkfufe1paWqqezueeUn0YlCTd3Nxw+PBh7N69G6dPn4aTkxO6dOmCmJiGPYHi1q1bAAAPDw8AQE5ODiorK9G/f3/1MmFhYQgICMCRI0dqTJLl5eUoLy9Xv1a1bKusrERlZaXBsanWbcg2mhJrKy/AMjdn7du3xwsvvIDLly8jNzcXTz31FAIDA2FjY9Psy24tx/h+NZXZ0PJLhOqBkPVw5MgRFBcXY9iwYepp69atw/z581FWVoaRI0di5cqVcHBw0DsQpVKJJ554Ajdv3kR2djYAID09HRMnTtRIegDQq1cv9O3bF0uWLNHaTmpqKhYsWKA1PT09Hc7OznrHRURETV9ZWRnGjRuHW7du1Xmf+n56XUkuXLgQcXFx6iR59uxZTJ06FRMmTEB4eDjeffddtGnTBqmpqXoFDwDJyck4d+6cOkEaau7cuZg5c6b6tUKhgL+/PwYOHKjXG/OgyspK7N69GwMGDICdnV2DYmwKrK28AMtsDWW2tvICLLOqzHX1l9VFryR56tQpvPnmm+rXX331FXr16oVPP/0UAODv74/58+frnSRTUlLw7bff4uDBg+p7nADg4+ODiooK3Lx5E+7u7urp169fh4+PT43bcnBwqPFK1s7OzigfEGNtp6mwtvICLLM1sLbyAiyzoWXXq0nXX3/9BW9vb/XrAwcOYPDgwerXPXv2xJUrV+q9PSEEUlJSsHXrVuzduxfBwcEa86OiomBnZ4c9e/aop+Xl5UEulyM6Olqf0ImIiPSmV5L09vZGQUEBAKCiogI//vijRuOZkpISvbJ1cnIy1q9fj/T0dLi6uuLatWu4du0a7ty5A+BeA6HJkydj5syZ2LdvH3JycjBx4kRER0ezZSsREZmcXtWtQ4YMwZw5c7BkyRJs27YNzs7OGi1az5w5g3bt2tV7e2lpaQCAuLg4jelr1qxBUlISAOCDDz6AjY0NEhMTUV5ejkGDBuGjjz7SJ2wiIiKD6JUk33zzTSQkJKBPnz6QSqVYt24d7O3t1fM///xzDBw4sN7bq0/DWkdHR6xatQqrVq3SJ1QiIqIG0ytJtm7dGgcPHsStW7cglUpha2urMX/z5s2QSqVGDZCIiMhcDB5MoCaqQQCIiIiaAw5YSEREpAOTJBERkQ5MkkRERDowSRIREenAJElERKQDkyQREZEOTJJEREQ6MEkSERHpwCRJRESkA5MkERGRDkySREREOjBJEhER6cAkSUREpAOTJBERkQ5MkkRERDowSRIREenAJElERKQDkyQREZEOTJJEREQ6MEkSERHpwCRJRESkA5MkERGRDkySREREOjBJEhER6cAkSUREpAOTJBERkQ5MkkRERDowSRIREenAJElERKQDkyQREZEOTJJEREQ6MEkSERHpwCRJRESkA5MkERGRDkySREREOjBJEhER6cAkSUREpAOTJBERkQ5MkkRERDowSRIREenQwtwBWIrq6mpUVlbqnF9ZWYkWLVrg7t27qK6ubsTIzMPayguYtsx2dnawtbU16jaJyPTMmiQPHjyId999Fzk5OSgsLMTWrVsxcuRI9XwhBObPn49PP/0UN2/eRO/evZGWloaQkBCjxSCEwLVr13Dz5s06l/Px8cGVK1cgkUiMtn9LZW3lBUxfZnd3d/j4+FjN+0nUHJg1Sd6+fRtdunTBpEmTkJCQoDV/6dKlWLFiBdatW4fg4GDMmzcPgwYNwvnz5+Ho6GiUGFQJ0svLC87Ozjp/wJRKJUpLSyGVSmFj0/xrqa2tvIDpyiyEQFlZGYqKigAAvr6+Rts2EZmWWZPk4MGDMXjw4BrnCSGwbNky/POf/8SIESMAAP/5z3/g7e2Nbdu2YezYsQ3ef3V1tTpBtmrVqtZllUolKioq4OjoaBVJw9rKC5i2zE5OTgCAoqIieHl5seqVqImw2HuSBQUFuHbtGvr376+e5ubmhocffhhHjhzRmSTLy8tRXl6ufq1QKADcu9/04D3H8vJyCCHg6OgIpVJZazxCCPW/dS3bHFhbeQHTl9nR0RFCCNy5cwcODg5G374hVN+J2u7HNyfWVl6AZX5wmr4sNkleu3YNAODt7a0x3dvbWz2vJosWLcKCBQu0pu/atQvOzs4a01q0aAEfHx/cvn273m9gSUlJvZZrLqytvIDpylxRUYE7d+7gwIEDqKqqMsk+DLV7925zh9CorK28AMtcVlZm0DYsNkkaau7cuZg5c6b6tUKhgL+/PwYOHAiZTKax7N27d3HlyhVIpdI673EKIVBSUgJXV1eraHhhbeUFTF/mu3fvwsnJCbGxsUa7p95QlZWV2L17NwYMGAA7Oztzh2Ny1lZegGVWlVlVq6gvi02SPj4+AIDr169rNHS4fv06unbtqnM9BweHGquy7OzstD4g1dXVkEgksLGxqfMelKr6TbV8c2fO8h46dAjTpk3Dzz//jKFDh2Lbtm11rpOamopt27bh1KlTBu/3wTJLJBKtFtcNiUG1zZo+i+ZmiTGZkrWVF2CZDS27xf7aBwcHw8fHB3v27FFPUygUOHbsGKKjo80YmfnduHEDzz//PAICAuDg4AAfHx8MGjQIhw4dMndoRjFz5kx07doVBQUFWLt2bYO3l5qaColEUutfTQoLC3U2LCMi62DWK8nS0lJcvHhR/bqgoACnTp2Ch4cHAgICMGPGDLz11lsICQlRdwFp06ZNvc/sG0u1shpZ8iwUlhTC19UXMQExsLUxXevFxMREVFRUYN26dWjbti2uX7+OPXv2oLi42GT7bEz5+fmYNm0a/Pz8jLK9WbNmYdq0aerXPXv2xLPPPoupU6fWuLyqhauqNoOIrJdZryRPnDiBbt26oVu3bgDuXUF069YNb7zxBgDg1VdfxYsvvohnn30WPXv2RGlpKTIzMy3mfg4AZORmIGh5EPqu64txGePQd11fBC0PQkZuhkn2d/PmTWRlZWHJkiXo27cvAgMD0atXL8ydOxdPPPEEAODSpUuQSCQa1X43b96ERCLB/v371dN++uknDBs2DDKZDK6uroiJiUF+fr56/vr16xEREQEHBwf4+voiJSVFY3tTpkyBp6cnZDIZ+vXrh9OnT6vnnz59Gn379oWrqytkMhmioqJw4sQJAMDly5cxfPhwtGzZEi4uLujUqRO2b9+ujru4uBiTJk2CRCLB2rVrsXbtWri7u2u8D9u2bav3fUOpVAofHx/1n62tLVxdXdWvx44di5SUFLz88sto166d+upRIpFoVPW+9tpr6NChA5ydndG2bVvMmzfPqloMElkjs15JxsXFqZvd10QikWDhwoVYuHBhI0ZVfxm5GRi9aTQENMvwu+J3jN40GlvGbEFCuPYgCQ0hlUohlUqxbds2PPLIIwZ3Jfj9998RGxuLuLg47N27FzKZDIcOHVK3ukxLS8Ps2bOxaNEiDBkyBLdu3dKozn3yySfh5OSEHTt2wM3NDZ988gkef/xx/PLLL/Dw8MD48ePRrVs3pKWlwdbWFqdOnVLfE0hOTkZFRQUOHjwIFxcXnD9/HlKpFP7+/igsLERoaCgWLlyIp556Cm5ubti4cWPD37g6rFu3DtOmTUNmZiakUmmNy7i6umLt2rVo06YNzp49i6lTp8LV1RWvvvqqyeMjIvOw2IY7lq5aWY3pmdO1EiQACAhIIMGMzBkYETrCqFWvLVq0wNq1azF16lR8/PHH6N69O/r06YOxY8ciMjKy3ttZtWoV3Nzc8NVXX6mTV4cOHdTz33nnHSQnJ+Oll15SN9zp2bMnACA7Oxs//PADioqK1En6X//6F7Zt24YtW7bg2WefhVwux+zZsxEWFgYAGkMJyuVyJCYmIiIiAgDQtm1b9TzVsG1ubm6NWt0ZEhKCJUuWQKFQaLWCVvnnP/+p/n9QUBBmzZqFr776ikmSqBmz2IY7li5LnoXfFL/pnC8gcEVxBVnyLKPvOzExEVevXsU333yD+Ph47N+/H927d9erkcupU6cQExNTY4uvoqIiXL16FX369Klx3dOnT6O0tBStWrVSX9lKpVIUFBSoq2tnzpyJKVOmoH///li8eLFGNe5LL72Et956C71798b8+fNx5swZ/d4AE4iKiqpzmY0bN6J3797w8fGBVCrFP//5T8jl8kaIjojMhUnSQIUlhUZdTl+Ojo4YMGAA5s2bh8OHDyMpKQnz588HAPWV3/1V2Q/eO1MNk1aT2uYB9xpc+fr64tSpUxp/eXl5mD17NoB7LUp/+uknDB06FHv37kXHjh2xdetWAMCUKVPw66+/4plnnsHZs2fRo0cPrFy5Uuf+bGxstKrljX0v0MXFpdb5R44cwfjx4zFkyBB8++23OHnyJF5//XVUVFQYNQ4isixMkgbyda3fINX1Xa6hOnbsiNu3bwMAPD09AdzrwqDyYN+9yMhIZGVl1ZhsXF1dERQUhAMHDtS4r+7du+PatWto0aIF2rdvr/HXunVr9XIdOnTAyy+/jF27diEhIQFr1qxRz/P398e0adOQkZGBV155BZ9++qnOsnl6eqKkpERdvprKY2qHDx9GYGAgXn/9dfTo0QMhISG4fPlyo8ZARI2PSdJAMQEx8JP5QYKaW1hKIIG/zB8xATFG3W9xcTH69euH9evX48yZMygoKMDmzZuxdOlS9UDwTk5OeOSRR7B48WLk5ubiwIEDGvfTACAlJQUKhQJjx47FiRMncOHCBXzxxRfIy8sDALzxxhtYtWoVVq5ciQsXLuDHH39UX+31798f0dHRGDlyJHbt2oVLly7h8OHDeP3113HixAncuXMHKSkp2L9/Py5fvoxDhw7h+PHjCA8PBwDMmDEDO3fuREFBAX788Ufs27dPPa8mDz/8MJydnfGPf/wD+fn5SE9PN0r/SX2EhIRALpfjq6++Qn5+PlasWKG+Miai5otJ0kC2NrZYHr8cALQSper1svhlRu8vKZVK8fDDD+ODDz5AbGwsOnfujHnz5mHq1Kn48MMP1ct9/vnnqKqqQlRUlLq/6f1atWqFvXv3orS0FH369EFUVBQ+/fRT9T3KCRMm4J133kFaWho6deqEYcOG4cKFC/fKJ5Fg+/btiI2NxcSJE9GhQweMHTsWly9fhre3N2xtbVFcXIy///3v6NChA8aMGYPBgwerx9Strq5GcnIywsPDER8fjw4dOuCjjz7SWWYPDw+sX78e27dvR0REBDZs2IDU1FSjvq91eeKJJ/Dyyy8jJSUFXbt2xeHDhzFv3rxGjYGIGp9E1NYHoxlQKBRwc3PDrVu3ahy7taCgAMHBwXX2vVQqleqWj/cP05aRm4HpmdM1GvH4y/yxLH6Z0bt/NCZd5W3OTF1mfT5vjaWyshLbt2/HkCFDrGLIMmsrL8Ay3z92q65cUBt2AWmghPAEjAgd0agj7hARUeNgkjQCWxtbxAXFmTsMIiIyMuuoRyMiIjIAkyQREZEOTJJEREQ6MEkSERHpwCRJRESkA5MkERGRDkySREREOjBJUqMICgrCsmXLjLa9tWvXwt3d3awxEFHzxyTZBCUlJUEikUAikcDe3h7t27fHwoULUVVVZe7QDBIUFKQuT01/SUlJWus89dRT+OWXXxo/WCKyKhxxxwiUSiXkcjlKSkrg6uqKgIAAk493Gh8fjzVr1qC8vBzbt29HcnIy7OzsMHfuXK1lKyoqYG9vb9J4GuL48eOorq4GcO+RVImJicjLy1OPr/jg8y0rKyvh5ORU53MviYgaileSDZSbm4vly5dj3bp1yMjIwLp167B8+XLk5uaadL8ODg7w8fFBYGAgnn/+efTv3x/ffPMNgHtXmiNHjsTbb7+NNm3aIDQ0FMC9p3ds27ZNYzvu7u7qx05dunQJEokEGRkZePzxx9GmTRt069YNR44c0VgnOzsbMTExcHJygr+/P1566SWNZz0WFRVh+PDhcHJyQnBwML788stay+Lp6QkfHx/4+PjAw8MDAODl5QUfHx/cvXsX7u7u2LhxI/r06QNHR0d8+eWXWtWt+fn5GDFiBLy9vSGVStGzZ098//33hry1RERqTJINkJubi02bNkGhUGhMVygU2LRpk8kT5f2cnJxQUVGhfr1nzx7k5eVh9+7d+Pbbb/Xa1uuvv46ZM2fi4MGDCAkJwdNPP62uys3Pz0d8fDwSExNx5swZbNy4EdnZ2UhJSVGvn5SUhCtXrmDfvn3YsmULPvroIxQVFTWofHPmzMH06dORm5uLQYMGac0vLS3FkCFDsGfPHpw8eRLx8fEYPnw45HJ5g/ZLRNaN1a0GUiqVyMzMrHWZzMxMhIaGmrTqVQiBPXv2YOfOnXjxxRfV011cXPDZZ58ZVM06a9YsDB06FAqFAqmpqYiIiMDFixcRFhaGRYsWYfz48ZgxYwaAew8jXrFiBfr06YO0tDTI5XLs2LEDP/zwA3r27AkAWL16da0PVa6PGTNmICFB96PHunTpgi5duqhfv/nmm9i6dSu++eYbjQRORKQPJkkDyeVyrSvIBykUCsjlcgQFBRl9/99++y2kUikqKyuhVCoxbtw4jQcRR0REGHwfMjIyUv1/X19fAPeqUMPCwnD69GmcOXNGowpVCAGlUomCggL88ssvaNGiBaKiotTzw8LC9G6J+qAePXrUOr+0tBSpqan47rvvUFhYiKqqKty5c4dXkkTUIEySBiopKTHqcvrq27cv0tLSYG9vjzZt2qBFC81D6eLiorWORCLBg8/Yrqys1Fru/gezSiQSAPeunIF7yei5557DSy+9pLVeQECAyVqc1lSe+82aNQu7d+/Gv/71L7Rv3x5OTk4YPXq0RhU0EZG+mCQN5OrqatTl9OXi4oL27dvrtY6npycKCwvVry9cuICysjK9ttG9e3ecP39e577DwsJQVVWFnJwcdXVrXl4ebt68qdd+9HXo0CEkJSVh1KhRAO4l80uXLpl0n0TU/LHhjoECAgLUXRR0kclkCAgIaKSI6tavXz98+OGHOHnyJE6cOIFp06ZpXDXWx2uvvYbDhw8jJSUFp06dwoULF/D111+r7/uFhoYiPj4ezz33HI4dO4acnBxMmTLF5N01QkJCkJGRgVOnTuH06dMYN26c+uqXiMhQTJIGsrGxQXx8fK3LxMfHm7y/pD7ee+89+Pv7IyYmBuPGjcOsWbPg7Oys1zYiIyNx4MAB/PLLL4iJiUG3bt3wxhtvoE2bNupl1qxZgzZt2qBPnz5ISEjAs88+Cy8vL2MXR8P777+Pli1b4tFHH8Xw4cMxaNAgdO/e3aT7JHqQUqnEpUuXcPbsWVy6dIknas2ARDx4k6qZUSgUcHNzw61bt7Su/O7evYuCggIEBwfD0dGx1u0olUooFArIZDKNxJebm4vMzEyNRjwymQzx8fENbtFpTrrK25yZusz6fN4aS2VlJbZv344hQ4boXavQFJmyvJb6W2Btxxioucy15YLa8J5kA4WHhyM0NLTRR9whIsuh6jP9IFWf6TFjxjTpk2ZrxiRpBDY2Nibp5kFEls9S+kyTafCIERE1gD59pqnpYZIkImoAc/eZJtNikgS0OtgTmQI/Z82TuftMk2lZdZJUtXrSt0M9kSFUnzNraWFoLZpin2mqP6tuuGNrawt3d3f1EyqcnZ3Vw7A9SKlUoqKiAnfv3rWKm+/WVl7AdGUWQqCsrAxFRUVwd3eHra2t0bZN5qfqM11T61YVS+szTfVn1UkSAHx8fACgzkc5CSFw584dODk56UykzYm1lRcwfZnd3d3VnzdqXsLDwzFmzBiL7CdJDWP1SVIikcDX1xdeXl41DvatUllZiYMHDyI2NtYqqsusrbyAactsZ2fHK8hmjn2mmyerT5Iqtra2tf6I2draoqqqCo6OjlaRNKytvIB1lpmMi32mmx+e4hAREenAJElERKQDkyQREZEOzf6epKoDd13DRtWlsrISZWVlUCgUVnG/ytrKC7DM1lBmaysvwDLf/xQQQP9BPZp9klQNBeXv72/mSIiIyNxKSkrg5uZW7+Wb/fMklUolrl69CldX1wb1fVMoFPD398eVK1f0ehZZU2Vt5QVYZmsos7WVF2CZVWUWQqCkpARt2rTRq1tOs7+StLGxgZ+fn9G2J5PJrOaDBlhfeQGW2RpYW3kBlhmAXleQKmy4Q0REpAOTJBERkQ5MkvXk4OCA+fPnw8HBwdyhNAprKy/AMlsDaysvwDI3VLNvuENERGQoXkkSERHpwCRJRESkA5MkERGRDkySREREOjBJ1sOqVasQFBQER0dHPPzww/jhhx/MHZLJpKamQiKRaPyFhYWZOyyjOnjwIIYPH442bdpAIpFg27ZtGvOFEHjjjTfg6+sLJycn9O/fHxcuXDBPsEZQV3mTkpK0jnl8fLx5gjWSRYsWoWfPnnB1dYWXlxdGjhyJvLw8jWXu3r2L5ORktGrVClKpFImJibh+/bqZIm6Y+pQ3Li5O6zhPmzbNTBE3XFpaGiIjI9UDBkRHR2PHjh3q+cY6vkySddi4cSNmzpyJ+fPn48cff0SXLl0waNAgFBUVmTs0k+nUqRMKCwvVf9nZ2eYOyahu376NLl26YNWqVTXOX7p0KVasWIGPP/4Yx44dg4uLCwYNGoS7d+82cqTGUVd5ASA+Pl7jmG/YsKERIzS+AwcOIDk5GUePHsXu3btRWVmJgQMH4vbt2+plXn75Zfzvf//D5s2bceDAAVy9ehUJCQlmjNpw9SkvAEydOlXjOC9dutRMETecn58fFi9ejJycHJw4cQL9+vXDiBEj8NNPPwEw4vEVVKtevXqJ5ORk9evq6mrRpk0bsWjRIjNGZTrz588XXbp0MXcYjQaA2Lp1q/q1UqkUPj4+4t1331VPu3nzpnBwcBAbNmwwQ4TG9WB5hRBiwoQJYsSIEWaJp7EUFRUJAOLAgQNCiHvH1M7OTmzevFm9TG5urgAgjhw5Yq4wjebB8gohRJ8+fcT06dPNF1QjaNmypfjss8+Menx5JVmLiooK5OTkoH///uppNjY26N+/P44cOWLGyEzrwoULaNOmDdq2bYvx48dDLpebO6RGU1BQgGvXrmkcczc3Nzz88MPN+pjv378fXl5eCA0NxfPPP4/i4mJzh2RUt27dAgB4eHgAAHJyclBZWalxnMPCwhAQENAsjvOD5VX58ssv0bp1a3Tu3Blz585FWVmZOcIzuurqanz11Ve4ffs2oqOjjXp8m/0A5w3xxx9/oLq6Gt7e3hrTvb298fPPP5spKtN6+OGHsXbtWoSGhqKwsBALFixATEwMzp07B1dXV3OHZ3LXrl0DgBqPuWpecxMfH4+EhAQEBwcjPz8f//jHPzB48GAcOXIEtra25g6vwZRKJWbMmIHevXujc+fOAO4dZ3t7e7i7u2ss2xyOc03lBYBx48YhMDAQbdq0wZkzZ/Daa68hLy8PGRkZZoy2Yc6ePYvo6GjcvXsXUqkUW7duRceOHXHq1CmjHV8mSdIwePBg9f8jIyPx8MMPIzAwEJs2bcLkyZPNGBmZytixY9X/j4iIQGRkJNq1a4f9+/fj8ccfN2NkxpGcnIxz5841u3vruugq77PPPqv+f0REBHx9ffH4448jPz8f7dq1a+wwjSI0NBSnTp3CrVu3sGXLFkyYMAEHDhww6j5Y3VqL1q1bw9bWVqtF1PXr1+Hj42OmqBqXu7s7OnTogIsXL5o7lEahOq7WfMzbtm2L1q1bN4tjnpKSgm+//Rb79u3TeGSej48PKioqcPPmTY3lm/px1lXemjz88MMA0KSPs729Pdq3b4+oqCgsWrQIXbp0wfLly416fJkka2Fvb4+oqCjs2bNHPU2pVGLPnj2Ijo42Y2SNp7S0FPn5+fD19TV3KI0iODgYPj4+GsdcoVDg2LFjVnPMf/vtNxQXFzfpYy6EQEpKCrZu3Yq9e/ciODhYY35UVBTs7Ow0jnNeXh7kcnmTPM51lbcmp06dAoAmfZwfpFQqUV5ebtzja9y2Rc3PV199JRwcHMTatWvF+fPnxbPPPivc3d3FtWvXzB2aSbzyyiti//79oqCgQBw6dEj0799ftG7dWhQVFZk7NKMpKSkRJ0+eFCdPnhQAxPvvvy9OnjwpLl++LIQQYvHixcLd3V18/fXX4syZM2LEiBEiODhY3Llzx8yRG6a28paUlIhZs2aJI0eOiIKCAvH999+L7t27i5CQEHH37l1zh26w559/Xri5uYn9+/eLwsJC9V9ZWZl6mWnTpomAgACxd+9eceLECREdHS2io6PNGLXh6irvxYsXxcKFC8WJEydEQUGB+Prrr0Xbtm1FbGysmSM33Jw5c8SBAwdEQUGBOHPmjJgzZ46QSCRi165dQgjjHV8myXpYuXKlCAgIEPb29qJXr17i6NGj5g7JZJ566inh6+sr7O3txUMPPSSeeuopcfHiRXOHZVT79u0TALT+JkyYIIS41w1k3rx5wtvbWzg4OIjHH39c5OXlmTfoBqitvGVlZWLgwIHC09NT2NnZicDAQDF16tQmfxJYU3kBiDVr1qiXuXPnjnjhhRdEy5YthbOzsxg1apQoLCw0X9ANUFd55XK5iI2NFR4eHsLBwUG0b99ezJ49W9y6dcu8gTfApEmTRGBgoLC3txeenp7i8ccfVydIIYx3fPmoLCIiIh14T5KIiEgHJkkiIiIdmCSJiIh0YJIkIiLSgUmSiIhIByZJIiIiHZgkiYiIdGCSJCIi0oFJkqiRpKamomvXrnqtI5FIsG3bNpPEYwyXLl2CRCJRjwNK1NwwSRIZQCKR1PqXmpqqtc6sWbM0Blw2hqSkJEgkEixevFhj+rZt2yCRSIy6LyJrxOdJEhmgsLBQ/f+NGzfijTfeQF5ennqaVCpV/18IgerqakilUo3pxuLo6IglS5bgueeeQ8uWLY2+fXOoqKiAvb29ucMg4pUkkSF8fHzUf25ubpBIJOrXP//8M1xdXbFjxw5ERUXBwcEB2dnZWtWtx48fx4ABA9C6dWu4ubmhT58++PHHH/WOpX///vDx8cGiRYt0LlNTVe+yZcsQFBSkfp2UlISRI0finXfegbe3N9zd3bFw4UJUVVVh9uzZ8PDwgJ+fH9asWaO1/Z9//hmPPvooHB0d0blzZ60H3547dw6DBw+GVCqFt7c3nnnmGfzxxx/q+XFxcUhJScGMGTPQunVrDBo0SO/3gcgUmCSJTGTOnDlYvHgxcnNzERkZqTW/pKQEEyZMQHZ2No4ePYqQkBAMGTIEJSUleu3H1tYW77zzDlauXInffvutQTHv3bsXV69excGDB/H+++9j/vz5GDZsGFq2bIljx45h2rRpeO6557T2M3v2bLzyyis4efIkoqOjMXz4cBQXFwMAbt68iX79+qFbt244ceIEMjMzcf36dYwZM0ZjG+vWrYO9vT0OHTqEjz/+uEHlIDIWJkkiE1m4cCEGDBiAdu3awcPDQ2t+v3798Le//Q1hYWEIDw/Hv//9b5SVlWldhdXHqFGj0LVrV8yfP79BMXt4eGDFihUIDQ3FpEmTEBoairKyMvzjH/9ASEgI5s6dC3t7e2RnZ2usl5KSgsTERISHhyMtLQ1ubm5YvXo1AODDDz9Et27d8M477yAsLAzdunXD559/jn379uGXX35RbyMkJARLly5FaGgoQkNDG1QOImNhkiQykR49etQ6//r165g6dSpCQkLg5uYGmUyG0tJSyOVyg/a3ZMkSrFu3Drm5uQatDwCdOnWCjc3//yx4e3sjIiJC/drW1hatWrVCUVGRxnr3P+29RYsW6NGjhzqO06dPY9++fep7slKpFGFhYQCA/Px89XpRUVEGx01kKmy4Q2QiLi4utc6fMGECiouLsXz5cgQGBsLBwQHR0dGoqKgwaH+xsbEYNGgQ5s6di6SkJI15NjY2ePDRsZWVlVrbsLOz03gtkUhqnKZUKusdV2lpKYYPH44lS5ZozfP19VX/v673i8gcmCSJzOTQoUP46KOPMGTIEADAlStXNBqzGGLx4sXo2rWrVnWlp6cnrl27BiGEumuIMfs2Hj16FLGxsQCAqqoq5OTkICUlBQDQvXt3/Pe//0VQUBBatOBPDjUtrG4lMpOQkBB88cUXyM3NxbFjxzB+/Hg4OTk1aJsREREYP348VqxYoTE9Li4ON27cwNKlS5Gfn49Vq1Zhx44dDdrX/VatWoWtW7fi559/RnJyMv766y9MmjQJAJCcnIw///wTTz/9NI4fP478/Hzs3LkTEydORHV1tdFiIDIFJkkiM1m9ejX++usvdO/eHc888wxeeukleHl5NXi7Cxcu1KoODQ8Px0cffYRVq1ahS5cu+OGHHzBr1qwG70tl8eLFWLx4Mbp06YLs7Gx88803aN26NQCgTZs2OHToEKqrqzFw4EBERERgxowZcHd317j/SWSJJOLBGxVEREQEgFeSREREOjFJEhER6cAkSUREpAOTJBERkQ5MkkRERDowSRIREenAJElERKQDkyQREZEOTJJEREQ6MEkSERHpwCRJRESkw/8BttvSUsJRp3EAAAAASUVORK5CYII=",+ "text/plain": [+ "<Figure size 500x300 with 1 Axes>"+ ]+ },+ "metadata": {},+ "output_type": "display_data"+ }+ ],+ "source": [+ "import matplotlib.pyplot as plt\n",+ "\n",+ "trial_logs = compiled_program.trial_logs\n",+ "\n",+ "# Extracting trial numbers, scores, and pruning status\n",+ "trial_numbers = list(trial_logs.keys())\n",+ "scores = [trial_logs[trial]['score'] for trial in trial_numbers]\n",+ "pruning_status = [trial_logs[trial]['pruned'] for trial in trial_numbers]\n",+ "\n",+ "# Plot setup\n",+ "plt.figure(figsize=(5, 3))\n",+ "\n",+ "# Plotting each point\n",+ "for trial_number, score, pruned in zip(trial_numbers, scores, pruning_status):\n",+ " if pruned:\n",+ " plt.scatter(trial_number, score, color='grey', label='Pruned Trial' if 'Pruned Trial' not in plt.gca().get_legend_handles_labels()[1] else \"\")\n",+ " else:\n",+ " plt.scatter(trial_number, score, color='green', label='Successful Trial' if 'Successful Trial' not in plt.gca().get_legend_handles_labels()[1] else \"\")\n",+ "\n",+ "plt.xlabel('Trial Number')\n",+ "plt.ylabel('Score')\n",+ "plt.title('Trial Scores with Pruning Status')\n",+ "plt.grid(True)\n",+ "plt.legend()\n",+ "plt.show()"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {+ "id": "5_Rwxfa1qZH4"+ },+ "source": [+ "We can also __visualize the best prompts__ discovered by MIPRO as our trials progress... (though note that score increases are also due to the selected fewshot examples, which are not shown here for conciseness)."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 9,+ "metadata": {+ "colab": {+ "base_uri": "https://localhost:8080/"+ },+ "id": "NnARfPRHqZH4",+ "outputId": "9a9a9276-a56d-4f8d-c1d6-218d5d3d12f2"+ },+ "outputs": [+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "Basline program | Score: 0:\n",+ "Prompt 1 Instruction: Given the fields `context`, `question`, produce the fields `search_query`.\n",+ "Prompt 2 Instruction: Given the fields `context`, `question`, produce the fields `answer`.\n",+ "Prompt 3 Instruction: Given a question we are trying to answer and a list of passages, return a comma separated list of the numbers associated with each passage. These numbers should be ordered by helpfulness in answering the question, with most helpful passage number first, and the least helpful last.\n",+ "\n",+ "----------------\n",+ "Best program after 0 trials | Score: 35.2:\n",+ "Prompt 1 Instruction: Given the fields `context` and `question`, identify the specific information being asked for and produce a concise and accurate response.\n",+ "Prompt 2 Instruction: Given a fact-based question related to pop culture, history, or entertainment, identify specific works or individuals and provide a concise answer directly corresponding to the question posed.\n",+ "Prompt 3 Instruction: Given a fact-based question related to pop culture, history, or entertainment, and a list of passages, rank the passages by their relevance to the question. Return a comma-separated list of the numbers associated with each passage, ordered from the most relevant (helpful) to the least relevant.\n",+ "\n",+ "Best program after 5 trials | Score: 35.6:\n",+ "Prompt 1 Instruction: Given a fact-based question related to pop culture, history, or entertainment, with a focus on identifying specific works or individuals, generate a search query that includes the key elements of the question for precise information retrieval.\n",+ "Prompt 2 Instruction: Given the information in the `context`, answer the `question` with a concise and specific response.\n",+ "Prompt 3 Instruction: Given a question we are trying to answer and a list of passages, return a comma-separated list of the numbers associated with each passage, ordered by helpfulness in answering the question, with the most helpful passage number first and the least helpful last.\n",+ "\n",+ "Best program after 10 trials | Score: 38.2:\n",+ "Prompt 1 Instruction: Given the varied syntax and focus on identifying specific works or individuals in fact-based questions related to pop culture, history, and entertainment, generate a search query that retrieves precise information relevant to the specific and varied questions posed.\n",+ "Prompt 2 Instruction: Given a fact-based question related to pop culture, history, or entertainment and the corresponding context, provide a concise and accurate answer identifying specific works or individuals.\n",+ "Prompt 3 Instruction: Given a fact-based question related to pop culture, history, or entertainment, and a list of relevant passages, rank the passages by their relevance to the question. Return a comma-separated list of the numbers associated with each passage in order of relevance, with the most helpful passage number first, and the least helpful last.\n",+ "\n",+ "Best program after 15 trials | Score: 38.2:\n",+ "Prompt 1 Instruction: Given the varied syntax and focus on identifying specific works or individuals in fact-based questions related to pop culture, history, and entertainment, generate a search query that retrieves precise information relevant to the specific and varied questions posed.\n",+ "Prompt 2 Instruction: Given a fact-based question related to pop culture, history, or entertainment and the corresponding context, provide a concise and accurate answer identifying specific works or individuals.\n",+ "Prompt 3 Instruction: Given a fact-based question related to pop culture, history, or entertainment, and a list of relevant passages, rank the passages by their relevance to the question. Return a comma-separated list of the numbers associated with each passage in order of relevance, with the most helpful passage number first, and the least helpful last.\n",+ "\n",+ "Best program after 20 trials | Score: 41.6:\n",+ "Prompt 1 Instruction: Given the fields `context` and `question`, determine the specific information being asked and compile a concise search query to retrieve the requested information.\n",+ "Prompt 2 Instruction: Given a fact-based question related to pop culture, history, or entertainment, identify and provide the specific individual or work being referred to in the context.\n",+ "Prompt 3 Instruction: Given a fact-based question related to pop culture, history, or entertainment, and a list of passages, rank the passages in order of relevance to the question. Return a comma-separated list of the corresponding passage numbers, with the most relevant passage first and the least relevant last.\n",+ "\n",+ "Best program after 25 trials | Score: 41.6:\n",+ "Prompt 1 Instruction: Given the fields `context` and `question`, determine the specific information being asked and compile a concise search query to retrieve the requested information.\n",+ "Prompt 2 Instruction: Given a fact-based question related to pop culture, history, or entertainment, identify and provide the specific individual or work being referred to in the context.\n",+ "Prompt 3 Instruction: Given a fact-based question related to pop culture, history, or entertainment, and a list of passages, rank the passages in order of relevance to the question. Return a comma-separated list of the corresponding passage numbers, with the most relevant passage first and the least relevant last.\n",+ "\n"+ ]+ }+ ],+ "source": [+ "best_score = 0\n",+ "\n",+ "def get_signature(predictor):\n",+ " if (hasattr(predictor, 'extended_signature')):\n",+ " return predictor.extended_signature\n",+ " elif (hasattr(predictor, 'signature')):\n",+ " return predictor.signature\n",+ "\n",+ "print(f\"Basline program | Score: {best_score}:\")\n",+ "for i,predictor in enumerate(program.predictors()):\n",+ " print(f\"Prompt {i+1} Instruction: {get_signature(predictor).instructions}\")\n",+ "print()\n",+ "\n",+ "print(\"----------------\")\n",+ "\n",+ "for trial_num in compiled_program.trial_logs:\n",+ " program_score = compiled_program.trial_logs[trial_num][\"score\"]\n",+ " program_pruned = compiled_program.trial_logs[trial_num][\"pruned\"]\n",+ " if program_score > best_score and not program_pruned:\n",+ " best_score = program_score\n",+ " best_program_so_far = compiled_program.trial_logs[trial_num][\"program\"]\n",+ " if trial_num % 5 == 0:\n",+ " print(f\"Best program after {trial_num} trials | Score: {best_score}:\")\n",+ " for i,predictor in enumerate(best_program_so_far.predictors()):\n",+ " print(f\"Prompt {i+1} Instruction: {get_signature(predictor).instructions}\")\n",+ " print()"+ ]+ }+ ],+ "metadata": {+ "colab": {+ "provenance": []+ },+ "kernelspec": {+ "display_name": "dspy_test",+ "language": "python",+ "name": "python3"+ },+ "language_info": {+ "codemirror_mode": {+ "name": "ipython",+ "version": 3+ },+ "file_extension": ".py",+ "mimetype": "text/x-python",+ "name": "python",+ "nbconvert_exporter": "python",+ "pygments_lexer": "ipython3",+ "version": "3.10.10"+ }+ },+ "nbformat": 4,+ "nbformat_minor": 0+}