Skip to content

armgit / dspy

Public Imported from GitHub
0

DSPy: The framework for programming—not prompting—language models

Python

Updated notebook docs

Michael Ryanf04d03c0c76f
diff --git a/examples/nli/scone/scone_with_MIPRO.ipynb b/examples/nli/scone/scone_with_MIPRO.ipynb
index 453509e89..fc649c551 100644
--- a/examples/nli/scone/scone_with_MIPRO.ipynb
+++ b/examples/nli/scone/scone_with_MIPRO.ipynb
@@ -16,7 +16,7 @@
},
"source": [
"# Using __<ins>M</ins>ulti-stage <ins>I</ins>nstruction <ins>P</ins>roposal & <ins>O</ins>ptimization (MIPROv2)__ in DSPy\n",
- "[![colab-badge](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/stanfordnlp/dspy/blob/main/examples/qa/hotpot/hotpotqa_with_MIPRO.ipynb)"
+ "[![colab-badge](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/stanfordnlp/dspy/blob/main/examples/nli/scone/scone_with_MIPRO.ipynb)"
]
},
{
@@ -27,24 +27,24 @@
"source": [
"### FAQ 🙋\n",
"#### 1) How does MIPRO work?\n",
- "At a high level, the MIPRO program optimizer works by first __proposing__ candidate fewshot example sets and instructions for each prompt in your program, and then __optimizing__ over these fewshot example sets and instructions as hyperparameters for a specified number of trials. Each trial, the optimizer evaluates different combinations of prompts on a train set, which allows it to learn which combinations yield the best performance.\n",
+ "At a high level, the MIPRO program optimizer works by first __proposing__ candidate fewshot example sets and instructions for each prompt in your program, and then __optimizing__ over these fewshot example sets and instructions as hyperparameters for a specified number of batches. Each batch, the optimizer evaluates different combinations of prompts on a subset of training inputs, which allows it to learn which combinations yield the best performance.\n",
"\n",
"#### 2) How much will MIPRO cost me to run?\n",
"Note that __this notebook__ is free to run, because all LM calls have been cached. However, when using an optimizer on your own program, here is a breakdown of the upper bound of the number of calls to the task model and prompt model respectively:\n",
"\n",
- "- **Task model calls**: MIPRO makes up to __O(TxPxM)__ task model calls if you run without minibatching, where T is the number of trials, P is the number of prompts in the program, and M is the size of the train set. This is because the model is evaluating the program on the train set each trial. In practice, this should be lower given that MIPRO tunes poor trials early (ie. it may stop a trial after running on the first 100 or so examples if performance is poor). If you run **with minibatching** you can reduce calls even further to __O(TxPxB)__ where **B** is the minibatch size. Note that every few steps (a parameter you set) MIPRO will also run a full eval over all **M** examples.\n",
+ "- **Task model calls**: MIPRO makes up to __O(TxPxM)__ task model calls if you run without minibatching, where T is the number of batches, P is the number of prompts in the program, and M is the size of the train set. This is because the model is evaluating the program on the train set each batch. If you run **with minibatching** you can reduce calls even further to __O(TxPxB)__ where **B** is the minibatch size. Note that every few steps (a parameter you set) MIPRO will also run a full eval over all **M** examples.\n",
"\n",
"- **Prompt model calls**: MIPRO makes up to N*P+10+(P+1) prompt model calls, where N is the number of instruction / fewshot example set candidates to generate for each prompt, and P is the number of prompts in the program. The extra 10 calls comes from generating a summary of the data in the training set, which we use in the meta prompt to create better instructions. The extra (P+1) comes from program summarization where the proposer LLM will look at the program code and try to describe what each module does and what the whole program does.\n",
"\n",
"#### 3) How should I configure the hyperparameters?\n",
"We have yet to run full hyperparameter sweeps with MIPRO, but based off of initial experimintation, we'd recommend the following:\n",
- "- __Trial num__: Gains can be seen after about 20-30 trials. However, 100-200 trials can help with adding on additional marginal gains.\n",
- "- __num candidates__: This hyperparameter controls the number of candidate prompts and fewshot example sets that are generated to optimize over. With more trials and less prompts to optimize, we can set n to be higher, as we have more trials to explore different combinations of prompts. If your program has between 2-3 modules and is the `num_trials=30`, we'd recommend ~`n=10`. If n is higher (say `n=100`), then we can go higher to ~`n=15`. If you have a program with only 1 module and are keeping the program 0-shot (ie. no fewshot examples), then `num_trials` should be set to equal `n`, because each trial can explore a new instruction.\n",
- "- __Training set size__: Between 200 and 500 training examples are recommended. Increasing the training set size can help prevent overfitting, but adds to the expense to run.\n",
+ "- __Batch num__: Gains can be seen after about 20-30 batches. However, 100-200 batches can help with adding on additional marginal gains.\n",
+ "- __num candidates__: This hyperparameter controls the number of candidate prompts and fewshot example sets that are generated to optimize over. With more batches and less prompts to optimize, we can set n to be higher, as we have more batches to explore different combinations of prompts. If your program has between 2-3 modules and is the `num_batches=30`, we'd recommend ~`n=10`. If n is higher (say `n=100`), then we can go higher to ~`n=15`. If you have a program with only 1 module and are keeping the program 0-shot (ie. no fewshot examples), then `num_batches` should be set to equal `n`, because each batch can explore a new instruction.\n",
+ "- __Training set size__: Between 100 and 500 training examples are recommended, however MIPROv2 can still work well with fewer. Increasing the training set size can help prevent overfitting, and only slightly increases cost for the full evaluation runs.\n",
"\n",
"#### 4) What should I do if I want to reduce the cost?\n",
- "You can always update hyperparameters accordingly, such as using a smaller train set, using less trials, or using a program with less modules. You should take advantage of minibatching.\n",
- "Alternatively, one strategy would be to optimize using a cheaper task model (ie. locally hosted Llama-3), as initial experiments have shown that prompts optimized for a smaller model also transfer to working well on a larger model.\n"
+ "You can always update hyperparameters accordingly, such as using a smaller train set, using less batches, or using a program with less modules. You should take advantage of minibatching.\n",
+ "Alternatively, one strategy would be to optimize using a cheaper task model (ie. locally hosted Llama-3), as initial experiments have shown that prompts optimized for a smaller model also transfer to working well on a larger model."
]
},
{
@@ -81,8 +81,7 @@
"output_type": "stream",
"text": [
"/opt/miniconda3/envs/opt-prompt/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
- " from .autonotebook import tqdm as notebook_tqdm\n",
- "Downloading trial_logs.pickle: 100%|██████████| 124k/124k [00:00<00:00, 5.53MB/s]\n"
+ " from .autonotebook import tqdm as notebook_tqdm\n"
]
}
],
@@ -426,9 +425,9 @@
"name": "stderr",
"output_type": "stream",
"text": [
- "Average Metric: 116 / 200 (58.0): 100%|██████████| 200/200 [00:00<00:00, 937.35it/s]\n",
- "Average Metric: 99 / 200 (49.5): 100%|██████████| 200/200 [00:00<00:00, 915.78it/s]\n",
- "Average Metric: 110 / 200 (55.0): 100%|██████████| 200/200 [00:00<00:00, 883.65it/s]\n"
+ "Average Metric: 116 / 200 (58.0): 100%|██████████| 200/200 [00:00<00:00, 732.35it/s]\n",
+ "Average Metric: 99 / 200 (49.5): 100%|██████████| 200/200 [00:00<00:00, 899.25it/s]\n",
+ "Average Metric: 110 / 200 (55.0): 100%|██████████| 200/200 [00:00<00:00, 916.33it/s]\n"
]
}
],
@@ -467,8 +466,8 @@
"\n",
"MIPRO only needs a metric, DSPy module, and training set to see huge gains on your task! You can instantiate a MIPRO Optimizer and compile in just two lines:\n",
"```python\n",
- "teleprompter = MIPRO(prompt_model=prompt_model, task_model=task_model, metric=metric, num_candidates=N, init_temperature=temperature)\n",
- "compiled_program = teleprompter.compile(program, trainset=trainset, devset=devset, num_trials=trials, max_bootstrapped_demos=1,max_labeled_demos=2, eval_kwargs=eval_kwargs)\n",
+ "teleprompter = MIPROv2(prompt_model=prompt_model, task_model=task_model, metric=metric, num_candidates=N, init_temperature=temperature)\n",
+ "compiled_program = teleprompter.compile(program, trainset=trainset, valset=valset, num_batches=batches, max_bootstrapped_demos=1,max_labeled_demos=2, eval_kwargs=eval_kwargs)\n",
"```"
]
},
@@ -478,7 +477,4604 @@
"metadata": {
"id": "NVfMJ_FpBlSI"
},
- "outputs": [],
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\u001b[93m\u001b[1mWARNING: Projected Language Model (LM) Calls\u001b[0m\n",
+ "\n",
+ "Please be advised that based on the parameters you have set, the maximum number of LM calls is projected as follows:\n",
+ "\n",
+ "\n",
+ "\u001b[93m- Prompt Model: \u001b[94m\u001b[1m10\u001b[0m\u001b[93m data summarizer calls + \u001b[94m\u001b[1m10\u001b[0m\u001b[93m * \u001b[94m\u001b[1m1\u001b[0m\u001b[93m lm calls in program + (\u001b[94m\u001b[1m2\u001b[0m\u001b[93m) lm calls in program aware proposer = \u001b[94m\u001b[1m22\u001b[0m\u001b[93m prompt model calls\u001b[0m\n",
+ "\u001b[93m- Task Model: \u001b[94m\u001b[1m25\u001b[0m\u001b[93m examples in minibatch * \u001b[94m\u001b[1m30\u001b[0m\u001b[93m batches + \u001b[94m\u001b[1m200\u001b[0m\u001b[93m examples in train set * \u001b[94m\u001b[1m3\u001b[0m\u001b[93m full evals = \u001b[94m\u001b[1m1350\u001b[0m\u001b[93m task model calls\u001b[0m\n",
+ "\n",
+ "\u001b[93m\u001b[1mEstimated Cost Calculation:\u001b[0m\n",
+ "\n",
+ "\u001b[93mTotal Cost = (Number of calls to task model * (Avg Input Token Length per Call * Task Model Price per Input Token + Avg Output Token Length per Call * Task Model Price per Output Token) \n",
+ " + (Number of calls to prompt model * (Avg Input Token Length per Call * Task Prompt Price per Input Token + Avg Output Token Length per Call * Prompt Model Price per Output Token).\u001b[0m\n",
+ "\n",
+ "For a preliminary estimate of potential costs, we recommend you perform your own calculations based on the task\n",
+ "and prompt models you intend to use. If the projected costs exceed your budget or expectations, you may consider:\n",
+ "\n",
+ "\u001b[93m- Reducing the number of trials (`num_batches`), the size of the trainset, or the number of LM calls in your program.\u001b[0m\n",
+ "\u001b[93m- Using a cheaper task model to optimize the prompt.\u001b[0m\n",
+ "To proceed with the execution of this program, please confirm by typing \u001b[94m'y'\u001b[0m for yes or \u001b[94m'n'\u001b[0m for no.\n",
+ "\n",
+ "If you would like to bypass this confirmation step in future executions, set the \u001b[93m`requires_permission_to_run`\u001b[0m flag to \u001b[93m`False` when calling compile.\u001b[0m\n",
+ "\n",
+ "\u001b[93mAwaiting your input...\u001b[0m\n",
+ "\n",
+ "SOURCE CODE: ScoNeSignature(context, question -> answer\n",
+ " instructions='context, question -> answer'\n",
+ " context = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Context:', 'desc': '${context}'})\n",
+ " question = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Question:', 'desc': '${question}'})\n",
+ " answer = Field(annotation=str required=True json_schema_extra={'desc': 'Yes or No', '__dspy_field_type': 'output', 'prefix': 'Answer:'})\n",
+ ")\n",
+ "\n",
+ "class ScoNeCoT(dspy.Module):\n",
+ " def __init__(self):\n",
+ " super().__init__()\n",
+ " self.generate_answer = dspy.ChainOfThought(ScoNeSignature)\n",
+ "\n",
+ " def forward(self, context, question):\n",
+ " return self.generate_answer(context=context, question=question)\n",
+ "\n",
+ "b: 10\n",
+ "e 'NoneType' object has no attribute 'write'. using observations from past round for a summary.\n",
+ "summary: Prediction(\n",
+ " summary='The dataset consists of examples that involve logical tests and negation related to language understanding and logical reasoning. Each example includes a context, a question, an answer, and a category label, and encompasses both single-scoped and two-scoped logical statements. The dataset seems to be focused on training models for handling negation in logical reasoning and has a broad range of potential applications.'\n",
+ ")\n",
+ "DATA SUMMARY: The dataset consists of examples that involve logical tests and negation related to language understanding and logical reasoning. Each example includes a context, a question, an answer, and a category label, and encompasses both single-scoped and two-scoped logical statements. The dataset seems to be focused on training models for handling negation in logical reasoning and has a broad range of potential applications.\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ " 0%| | 1/200 [00:00<00:00, 474.84it/s]\n",
+ " 2%|▏ | 3/200 [00:00<00:00, 613.74it/s]\n",
+ " 0%| | 1/200 [00:00<00:00, 707.18it/s]\n",
+ " 0%| | 1/200 [00:00<00:00, 309.31it/s]\n",
+ " 0%| | 1/200 [00:00<00:00, 579.72it/s]\n",
+ " 1%| | 2/200 [00:00<00:00, 816.73it/s]\n",
+ " 0%| | 1/200 [00:00<00:00, 602.37it/s]\n",
+ " 0%| | 1/200 [00:00<00:00, 750.19it/s]\n",
+ "/opt/miniconda3/envs/opt-prompt/lib/python3.11/site-packages/optuna/samplers/_tpe/sampler.py:295: ExperimentalWarning: ``multivariate`` option is an experimental feature. The interface can change in the future.\n",
+ " warnings.warn(\n",
+ "[I 2024-06-20 22:06:46,767] A new study created in memory with name: no-name-1fe4456d-49ca-4355-9643-2e9660e5f63e\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Using a randomly generated configuration for our grounded proposer.\n",
+ "Selected tip: description\n",
+ "PROGRAM DESCRIPTION: The program appears to be used to generate answers (Yes or No) based on a given context and question.\n",
+ "task_demos \n",
+ "\n",
+ "\n",
+ "\n",
+ "Use the information below to learn about a task that we are trying to solve using calls to an LM, then generate a new instruction that will be used to prompt a Language Model to better solve the task.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "DATASET SUMMARY: A description of the dataset that we are using.\n",
+ "\n",
+ "PROGRAM CODE: Language model program designed to solve a particular task.\n",
+ "\n",
+ "PROGRAM DESCRIPTION: Summary of the task the program is designed to solve, and how it goes about solving it.\n",
+ "\n",
+ "MODULE: The module to create an instruction for.\n",
+ "\n",
+ "TASK DEMO(S): Example inputs/outputs of our module.\n",
+ "\n",
+ "BASIC INSTRUCTION: Basic instruction.\n",
+ "\n",
+ "TIP: A suggestion for how to go about generating the new instruction.\n",
+ "\n",
+ "PROPOSED INSTRUCTION: Propose an instruction that will be used to prompt a Language Model to perform this task.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "DATASET SUMMARY: The dataset consists of examples that involve logical tests and negation related to language understanding and logical reasoning. Each example includes a context, a question, an answer, and a category label, and encompasses both single-scoped and two-scoped logical statements. The dataset seems to be focused on training models for handling negation in logical reasoning and has a broad range of potential applications.\n",
+ "\n",
+ "PROGRAM CODE:\n",
+ "ScoNeSignature(context, question -> answer\n",
+ " instructions='context, question -> answer'\n",
+ " context = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Context:', 'desc': '${context}'})\n",
+ " question = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Question:', 'desc': '${question}'})\n",
+ " answer = Field(annotation=str required=True json_schema_extra={'desc': 'Yes or No', '__dspy_field_type': 'output', 'prefix': 'Answer:'})\n",
+ ")\n",
+ "\n",
+ "class ScoNeCoT(dspy.Module):\n",
+ " def __init__(self):\n",
+ " super().__init__()\n",
+ " self.generate_answer = dspy.ChainOfThought(ScoNeSignature)\n",
+ "\n",
+ " def forward(self, context, question):\n",
+ " return self.generate_answer(context=context, question=question)\n",
+ "\n",
+ "\n",
+ "PROGRAM DESCRIPTION: The program appears to be used to generate answers (Yes or No) based on a given context and question.\n",
+ "\n",
+ "MODULE: self.generate_answer = dspy.ChainOfThought(ScoNeSignature)\n",
+ "\n",
+ "TASK DEMO(S): \n",
+ "\n",
+ "BASIC INSTRUCTION: context, question -> answer\n",
+ "\n",
+ "TIP: Make sure your instruction is very informative and descriptive.\n",
+ "\n",
+ "PROPOSED INSTRUCTION:\u001b[32m Provide a yes or no answer based on the given context and question. The context and question are used as input to generate the answer.\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "PROPOSED INSTRUCTION: Provide a yes or no answer based on the given context and question. The context and question are used as input to generate the answer.\n",
+ "PROGRAM DESCRIPTION: The program appears to be used to generate answers (Yes or No) based on a given context and question.\n",
+ "task_demos \n",
+ "\n",
+ "\n",
+ "\n",
+ "Use the information below to learn about a task that we are trying to solve using calls to an LM, then generate a new instruction that will be used to prompt a Language Model to better solve the task.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "DATASET SUMMARY: A description of the dataset that we are using.\n",
+ "\n",
+ "PROGRAM CODE: Language model program designed to solve a particular task.\n",
+ "\n",
+ "PROGRAM DESCRIPTION: Summary of the task the program is designed to solve, and how it goes about solving it.\n",
+ "\n",
+ "MODULE: The module to create an instruction for.\n",
+ "\n",
+ "TASK DEMO(S): Example inputs/outputs of our module.\n",
+ "\n",
+ "BASIC INSTRUCTION: Basic instruction.\n",
+ "\n",
+ "TIP: A suggestion for how to go about generating the new instruction.\n",
+ "\n",
+ "PROPOSED INSTRUCTION: Propose an instruction that will be used to prompt a Language Model to perform this task.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "DATASET SUMMARY: The dataset consists of examples that involve logical tests and negation related to language understanding and logical reasoning. Each example includes a context, a question, an answer, and a category label, and encompasses both single-scoped and two-scoped logical statements. The dataset seems to be focused on training models for handling negation in logical reasoning and has a broad range of potential applications.\n",
+ "\n",
+ "PROGRAM CODE:\n",
+ "ScoNeSignature(context, question -> answer\n",
+ " instructions='context, question -> answer'\n",
+ " context = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Context:', 'desc': '${context}'})\n",
+ " question = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Question:', 'desc': '${question}'})\n",
+ " answer = Field(annotation=str required=True json_schema_extra={'desc': 'Yes or No', '__dspy_field_type': 'output', 'prefix': 'Answer:'})\n",
+ ")\n",
+ "\n",
+ "class ScoNeCoT(dspy.Module):\n",
+ " def __init__(self):\n",
+ " super().__init__()\n",
+ " self.generate_answer = dspy.ChainOfThought(ScoNeSignature)\n",
+ "\n",
+ " def forward(self, context, question):\n",
+ " return self.generate_answer(context=context, question=question)\n",
+ "\n",
+ "\n",
+ "PROGRAM DESCRIPTION: The program appears to be used to generate answers (Yes or No) based on a given context and question.\n",
+ "\n",
+ "MODULE: self.generate_answer = dspy.ChainOfThought(ScoNeSignature)\n",
+ "\n",
+ "TASK DEMO(S): \n",
+ "\n",
+ "BASIC INSTRUCTION: context, question -> answer\n",
+ "\n",
+ "TIP: Make sure your instruction is very informative and descriptive.\n",
+ "\n",
+ "PROPOSED INSTRUCTION:\u001b[32m Provide a yes or no answer based on the given context and question. The context and question are used as input to generate the answer.\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "PROPOSED INSTRUCTION: Provide a yes or no answer based on the given context and question. The context and question are used as input to generate the answer.\n",
+ "PROGRAM DESCRIPTION: This program is designed to solve logical reasoning tasks by using language models. It takes in a context and a question as input, and then generates an answer based on the logical reasoning of the given context and question. The program uses a Chain of Thought model to process the input and produce the answer. The example provided uses the program to determine the logical conclusion of a given statement about people walking in the city, demonstrating its ability to reason and generate responses based on the input context and question.\n",
+ "task_demos Context: It is not true that there is not a single person walking in the city.\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the statement \"there is not a single person walking in the city\" is false. However, we cannot logically conclude that the statement \"there is not a single slaver walking in the city\" is also false. There could be a single person walking in the city who is not a slaver.\n",
+ "Answer: No\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "Use the information below to learn about a task that we are trying to solve using calls to an LM, then generate a new instruction that will be used to prompt a Language Model to better solve the task.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "DATASET SUMMARY: A description of the dataset that we are using.\n",
+ "\n",
+ "PROGRAM CODE: Language model program designed to solve a particular task.\n",
+ "\n",
+ "PROGRAM DESCRIPTION: Summary of the task the program is designed to solve, and how it goes about solving it.\n",
+ "\n",
+ "MODULE: The module to create an instruction for.\n",
+ "\n",
+ "TASK DEMO(S): Example inputs/outputs of our module.\n",
+ "\n",
+ "BASIC INSTRUCTION: Basic instruction.\n",
+ "\n",
+ "TIP: A suggestion for how to go about generating the new instruction.\n",
+ "\n",
+ "PROPOSED INSTRUCTION: Propose an instruction that will be used to prompt a Language Model to perform this task.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "DATASET SUMMARY: The dataset consists of examples that involve logical tests and negation related to language understanding and logical reasoning. Each example includes a context, a question, an answer, and a category label, and encompasses both single-scoped and two-scoped logical statements. The dataset seems to be focused on training models for handling negation in logical reasoning and has a broad range of potential applications.\n",
+ "\n",
+ "PROGRAM CODE:\n",
+ "ScoNeSignature(context, question -> answer\n",
+ " instructions='context, question -> answer'\n",
+ " context = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Context:', 'desc': '${context}'})\n",
+ " question = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Question:', 'desc': '${question}'})\n",
+ " answer = Field(annotation=str required=True json_schema_extra={'desc': 'Yes or No', '__dspy_field_type': 'output', 'prefix': 'Answer:'})\n",
+ ")\n",
+ "\n",
+ "class ScoNeCoT(dspy.Module):\n",
+ " def __init__(self):\n",
+ " super().__init__()\n",
+ " self.generate_answer = dspy.ChainOfThought(ScoNeSignature)\n",
+ "\n",
+ " def forward(self, context, question):\n",
+ " return self.generate_answer(context=context, question=question)\n",
+ "\n",
+ "\n",
+ "PROGRAM DESCRIPTION: This program is designed to solve logical reasoning tasks by using language models. It takes in a context and a question as input, and then generates an answer based on the logical reasoning of the given context and question. The program uses a Chain of Thought model to process the input and produce the answer. The example provided uses the program to determine the logical conclusion of a given statement about people walking in the city, demonstrating its ability to reason and generate responses based on the input context and question.\n",
+ "\n",
+ "MODULE: self.generate_answer = dspy.ChainOfThought(ScoNeSignature)\n",
+ "\n",
+ "TASK DEMO(S):\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the statement \"there is not a single person walking in the city\" is false. However, we cannot logically conclude that the statement \"there is not a single slaver walking in the city\" is also false. There could be a single person walking in the city who is not a slaver.\n",
+ "Answer: No\n",
+ "\n",
+ "\n",
+ "BASIC INSTRUCTION: context, question -> answer\n",
+ "\n",
+ "TIP: Make sure your instruction is very informative and descriptive.\n",
+ "\n",
+ "PROPOSED INSTRUCTION:\u001b[32m LogicalNegation(context: str, question: str -> answer: str) \"Given a context and a question involving negation, determine the logical conclusion and provide a yes or no answer based on the reasoning.\"\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "PROPOSED INSTRUCTION: LogicalNegation(context: str, question: str -> answer: str) \"Given a context and a question involving negation, determine the logical conclusion and provide a yes or no answer based on the reasoning.\n",
+ "PROGRAM DESCRIPTION: This program appears to be designed to solve a task of logical reasoning and inference based on a given context and question. It uses a language model to generate an answer to the question based on the provided context. The program takes the context and question as input and returns a \"yes\" or \"no\" answer. In the example provided, the program uses the given context and question to produce a \"no\" answer, indicating that a logical conclusion cannot be made. The program seems to work by passing the context and question through a language model to generate the answer.\n",
+ "task_demos Context: A man is not standing on top of a ladder that is leaned against a not so tall tree.\n",
+ "Question: Can we logically conclude for sure that a man is not standing on top of a ladder that is leaned against a not so tall pine?\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the man is not standing on top of a ladder that is leaned against a not so tall tree. We can't conclude that the ladder is leaned against a pine tree, so we can't conclude that the man is not standing on top of a ladder that is leaned against a not so tall pine.\n",
+ "Answer: No\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "Use the information below to learn about a task that we are trying to solve using calls to an LM, then generate a new instruction that will be used to prompt a Language Model to better solve the task.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "DATASET SUMMARY: A description of the dataset that we are using.\n",
+ "\n",
+ "PROGRAM CODE: Language model program designed to solve a particular task.\n",
+ "\n",
+ "PROGRAM DESCRIPTION: Summary of the task the program is designed to solve, and how it goes about solving it.\n",
+ "\n",
+ "MODULE: The module to create an instruction for.\n",
+ "\n",
+ "TASK DEMO(S): Example inputs/outputs of our module.\n",
+ "\n",
+ "BASIC INSTRUCTION: Basic instruction.\n",
+ "\n",
+ "TIP: A suggestion for how to go about generating the new instruction.\n",
+ "\n",
+ "PROPOSED INSTRUCTION: Propose an instruction that will be used to prompt a Language Model to perform this task.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "DATASET SUMMARY: The dataset consists of examples that involve logical tests and negation related to language understanding and logical reasoning. Each example includes a context, a question, an answer, and a category label, and encompasses both single-scoped and two-scoped logical statements. The dataset seems to be focused on training models for handling negation in logical reasoning and has a broad range of potential applications.\n",
+ "\n",
+ "PROGRAM CODE:\n",
+ "ScoNeSignature(context, question -> answer\n",
+ " instructions='context, question -> answer'\n",
+ " context = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Context:', 'desc': '${context}'})\n",
+ " question = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Question:', 'desc': '${question}'})\n",
+ " answer = Field(annotation=str required=True json_schema_extra={'desc': 'Yes or No', '__dspy_field_type': 'output', 'prefix': 'Answer:'})\n",
+ ")\n",
+ "\n",
+ "class ScoNeCoT(dspy.Module):\n",
+ " def __init__(self):\n",
+ " super().__init__()\n",
+ " self.generate_answer = dspy.ChainOfThought(ScoNeSignature)\n",
+ "\n",
+ " def forward(self, context, question):\n",
+ " return self.generate_answer(context=context, question=question)\n",
+ "\n",
+ "\n",
+ "PROGRAM DESCRIPTION: This program appears to be designed to solve a task of logical reasoning and inference based on a given context and question. It uses a language model to generate an answer to the question based on the provided context. The program takes the context and question as input and returns a \"yes\" or \"no\" answer. In the example provided, the program uses the given context and question to produce a \"no\" answer, indicating that a logical conclusion cannot be made. The program seems to work by passing the context and question through a language model to generate the answer.\n",
+ "\n",
+ "MODULE: self.generate_answer = dspy.ChainOfThought(ScoNeSignature)\n",
+ "\n",
+ "TASK DEMO(S):\n",
+ "Context: A man is not standing on top of a ladder that is leaned against a not so tall tree.\n",
+ "Question: Can we logically conclude for sure that a man is not standing on top of a ladder that is leaned against a not so tall pine?\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the man is not standing on top of a ladder that is leaned against a not so tall tree. We can't conclude that the ladder is leaned against a pine tree, so we can't conclude that the man is not standing on top of a ladder that is leaned against a not so tall pine.\n",
+ "Answer: No\n",
+ "\n",
+ "\n",
+ "BASIC INSTRUCTION: context, question -> answer\n",
+ "\n",
+ "TIP: Make sure your instruction is very informative and descriptive.\n",
+ "\n",
+ "PROPOSED INSTRUCTION:\u001b[32m Given a context and a question, use logical reasoning to determine if a logical conclusion can be made and generate a \"yes\" or \"no\" answer based on the provided information.\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "PROPOSED INSTRUCTION: Given a context and a question, use logical reasoning to determine if a logical conclusion can be made and generate a \"yes\" or \"no\" answer based on the provided information.\n",
+ "PROGRAM DESCRIPTION: This program appears to be designed to solve tasks related to logical reasoning and inference based on a given context and question. The program takes a context and a question as input and uses a language model to generate an answer, which is either \"Yes\" or \"No\" based on logical inference. The program uses a Chain of Thought model to process the input and generate the answer. In the example provided, the program successfully processes the context and question to infer a logical conclusion and generate the answer \"Yes.\" Overall, the program is designed to use language models for logical reasoning tasks and inference.\n",
+ "task_demos Context: The cowboy did not tell other people that he fall off a bronco at the competition.\n",
+ "Question: Can we logically conclude for sure that the cowboy did not tell other people that he fall off a horse at the competition?\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the cowboy did not tell other people that he fall off a bronco at the competition. Bronco is a type of horse, so it is a type of horse that the cowboy fell off. Therefore, we can logically conclude that the cowboy did not tell other people that he fall off a horse at the competition.\n",
+ "Answer: Yes\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "Use the information below to learn about a task that we are trying to solve using calls to an LM, then generate a new instruction that will be used to prompt a Language Model to better solve the task.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "DATASET SUMMARY: A description of the dataset that we are using.\n",
+ "\n",
+ "PROGRAM CODE: Language model program designed to solve a particular task.\n",
+ "\n",
+ "PROGRAM DESCRIPTION: Summary of the task the program is designed to solve, and how it goes about solving it.\n",
+ "\n",
+ "MODULE: The module to create an instruction for.\n",
+ "\n",
+ "TASK DEMO(S): Example inputs/outputs of our module.\n",
+ "\n",
+ "BASIC INSTRUCTION: Basic instruction.\n",
+ "\n",
+ "TIP: A suggestion for how to go about generating the new instruction.\n",
+ "\n",
+ "PROPOSED INSTRUCTION: Propose an instruction that will be used to prompt a Language Model to perform this task.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "DATASET SUMMARY: The dataset consists of examples that involve logical tests and negation related to language understanding and logical reasoning. Each example includes a context, a question, an answer, and a category label, and encompasses both single-scoped and two-scoped logical statements. The dataset seems to be focused on training models for handling negation in logical reasoning and has a broad range of potential applications.\n",
+ "\n",
+ "PROGRAM CODE:\n",
+ "ScoNeSignature(context, question -> answer\n",
+ " instructions='context, question -> answer'\n",
+ " context = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Context:', 'desc': '${context}'})\n",
+ " question = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Question:', 'desc': '${question}'})\n",
+ " answer = Field(annotation=str required=True json_schema_extra={'desc': 'Yes or No', '__dspy_field_type': 'output', 'prefix': 'Answer:'})\n",
+ ")\n",
+ "\n",
+ "class ScoNeCoT(dspy.Module):\n",
+ " def __init__(self):\n",
+ " super().__init__()\n",
+ " self.generate_answer = dspy.ChainOfThought(ScoNeSignature)\n",
+ "\n",
+ " def forward(self, context, question):\n",
+ " return self.generate_answer(context=context, question=question)\n",
+ "\n",
+ "\n",
+ "PROGRAM DESCRIPTION: This program appears to be designed to solve tasks related to logical reasoning and inference based on a given context and question. The program takes a context and a question as input and uses a language model to generate an answer, which is either \"Yes\" or \"No\" based on logical inference. The program uses a Chain of Thought model to process the input and generate the answer. In the example provided, the program successfully processes the context and question to infer a logical conclusion and generate the answer \"Yes.\" Overall, the program is designed to use language models for logical reasoning tasks and inference.\n",
+ "\n",
+ "MODULE: self.generate_answer = dspy.ChainOfThought(ScoNeSignature)\n",
+ "\n",
+ "TASK DEMO(S):\n",
+ "Context: The cowboy did not tell other people that he fall off a bronco at the competition.\n",
+ "Question: Can we logically conclude for sure that the cowboy did not tell other people that he fall off a horse at the competition?\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the cowboy did not tell other people that he fall off a bronco at the competition. Bronco is a type of horse, so it is a type of horse that the cowboy fell off. Therefore, we can logically conclude that the cowboy did not tell other people that he fall off a horse at the competition.\n",
+ "Answer: Yes\n",
+ "\n",
+ "\n",
+ "BASIC INSTRUCTION: context, question -> answer\n",
+ "\n",
+ "TIP: Make sure your instruction is very informative and descriptive.\n",
+ "\n",
+ "PROPOSED INSTRUCTION:\u001b[32m Please use the provided context and question to generate a logical inference for the answer using the language model. The answer should be \"Yes\" or \"No\" based on the logical reasoning and inference from the given context and question.\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "PROPOSED INSTRUCTION: Please use the provided context and question to generate a logical inference for the answer using the language model. The answer should be \"Yes\" or \"No\" based on the logical reasoning and inference from the given context and question.\n",
+ "PROGRAM DESCRIPTION: The program is designed to solve a task of answering yes or no questions based on a given context. It appears to work by taking in a context and a question as inputs, and then using a language model to generate an answer to the question based on the provided context. In the example given, the program uses the provided context and question to generate a logical step-by-step reasoning and ultimately provide an answer of \"No\" to the question. This suggests that the program is designed to solve tasks that require logical reasoning and inference based on textual information.\n",
+ "task_demos Context: It is not true that there is not a single person walking in the city.\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single celebrity walking in the city?\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This does not necessarily mean that there is a celebrity walking in the city. There could be a non-celebrity person walking in the city.\n",
+ "Answer: No\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "Use the information below to learn about a task that we are trying to solve using calls to an LM, then generate a new instruction that will be used to prompt a Language Model to better solve the task.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "DATASET SUMMARY: A description of the dataset that we are using.\n",
+ "\n",
+ "PROGRAM CODE: Language model program designed to solve a particular task.\n",
+ "\n",
+ "PROGRAM DESCRIPTION: Summary of the task the program is designed to solve, and how it goes about solving it.\n",
+ "\n",
+ "MODULE: The module to create an instruction for.\n",
+ "\n",
+ "TASK DEMO(S): Example inputs/outputs of our module.\n",
+ "\n",
+ "BASIC INSTRUCTION: Basic instruction.\n",
+ "\n",
+ "TIP: A suggestion for how to go about generating the new instruction.\n",
+ "\n",
+ "PROPOSED INSTRUCTION: Propose an instruction that will be used to prompt a Language Model to perform this task.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "DATASET SUMMARY: The dataset consists of examples that involve logical tests and negation related to language understanding and logical reasoning. Each example includes a context, a question, an answer, and a category label, and encompasses both single-scoped and two-scoped logical statements. The dataset seems to be focused on training models for handling negation in logical reasoning and has a broad range of potential applications.\n",
+ "\n",
+ "PROGRAM CODE:\n",
+ "ScoNeSignature(context, question -> answer\n",
+ " instructions='context, question -> answer'\n",
+ " context = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Context:', 'desc': '${context}'})\n",
+ " question = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Question:', 'desc': '${question}'})\n",
+ " answer = Field(annotation=str required=True json_schema_extra={'desc': 'Yes or No', '__dspy_field_type': 'output', 'prefix': 'Answer:'})\n",
+ ")\n",
+ "\n",
+ "class ScoNeCoT(dspy.Module):\n",
+ " def __init__(self):\n",
+ " super().__init__()\n",
+ " self.generate_answer = dspy.ChainOfThought(ScoNeSignature)\n",
+ "\n",
+ " def forward(self, context, question):\n",
+ " return self.generate_answer(context=context, question=question)\n",
+ "\n",
+ "\n",
+ "PROGRAM DESCRIPTION: The program is designed to solve a task of answering yes or no questions based on a given context. It appears to work by taking in a context and a question as inputs, and then using a language model to generate an answer to the question based on the provided context. In the example given, the program uses the provided context and question to generate a logical step-by-step reasoning and ultimately provide an answer of \"No\" to the question. This suggests that the program is designed to solve tasks that require logical reasoning and inference based on textual information.\n",
+ "\n",
+ "MODULE: self.generate_answer = dspy.ChainOfThought(ScoNeSignature)\n",
+ "\n",
+ "TASK DEMO(S):\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single celebrity walking in the city?\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This does not necessarily mean that there is a celebrity walking in the city. There could be a non-celebrity person walking in the city.\n",
+ "Answer: No\n",
+ "\n",
+ "\n",
+ "BASIC INSTRUCTION: context, question -> answer\n",
+ "\n",
+ "TIP: Make sure your instruction is very informative and descriptive.\n",
+ "\n",
+ "PROPOSED INSTRUCTION:\u001b[32m Given the context and a question, use logical reasoning and inference to determine and provide a yes or no answer based on the information provided.\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "PROPOSED INSTRUCTION: Given the context and a question, use logical reasoning and inference to determine and provide a yes or no answer based on the information provided.\n",
+ "PROGRAM DESCRIPTION: This program appears to be designed to solve contextual reasoning tasks by using language models. It takes in a context and a question, and uses a language model to generate a yes or no answer based on the given context and question. In the provided example, the program is used to determine if it can be logically concluded that there is a kayak nearby based on the given context. The program uses a chain of thought to process the input and generate the answer, taking into account the context and question provided. It uses the language model to reason through the context and question in order to provide the appropriate yes or no answer.\n",
+ "task_demos Context: There is a boat nearby\n",
+ "Question: Can we logically conclude for sure that there is a kayak nearby?\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that there is a boat nearby, but we don't know what type of boat it is. It could be a kayak, but it could also be a sailboat, a speedboat, or even a rowboat. We can't logically conclude that it is a kayak for sure.\n",
+ "Answer: No\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "Use the information below to learn about a task that we are trying to solve using calls to an LM, then generate a new instruction that will be used to prompt a Language Model to better solve the task.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "DATASET SUMMARY: A description of the dataset that we are using.\n",
+ "\n",
+ "PROGRAM CODE: Language model program designed to solve a particular task.\n",
+ "\n",
+ "PROGRAM DESCRIPTION: Summary of the task the program is designed to solve, and how it goes about solving it.\n",
+ "\n",
+ "MODULE: The module to create an instruction for.\n",
+ "\n",
+ "TASK DEMO(S): Example inputs/outputs of our module.\n",
+ "\n",
+ "BASIC INSTRUCTION: Basic instruction.\n",
+ "\n",
+ "TIP: A suggestion for how to go about generating the new instruction.\n",
+ "\n",
+ "PROPOSED INSTRUCTION: Propose an instruction that will be used to prompt a Language Model to perform this task.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "DATASET SUMMARY: The dataset consists of examples that involve logical tests and negation related to language understanding and logical reasoning. Each example includes a context, a question, an answer, and a category label, and encompasses both single-scoped and two-scoped logical statements. The dataset seems to be focused on training models for handling negation in logical reasoning and has a broad range of potential applications.\n",
+ "\n",
+ "PROGRAM CODE:\n",
+ "ScoNeSignature(context, question -> answer\n",
+ " instructions='context, question -> answer'\n",
+ " context = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Context:', 'desc': '${context}'})\n",
+ " question = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Question:', 'desc': '${question}'})\n",
+ " answer = Field(annotation=str required=True json_schema_extra={'desc': 'Yes or No', '__dspy_field_type': 'output', 'prefix': 'Answer:'})\n",
+ ")\n",
+ "\n",
+ "class ScoNeCoT(dspy.Module):\n",
+ " def __init__(self):\n",
+ " super().__init__()\n",
+ " self.generate_answer = dspy.ChainOfThought(ScoNeSignature)\n",
+ "\n",
+ " def forward(self, context, question):\n",
+ " return self.generate_answer(context=context, question=question)\n",
+ "\n",
+ "\n",
+ "PROGRAM DESCRIPTION: This program appears to be designed to solve contextual reasoning tasks by using language models. It takes in a context and a question, and uses a language model to generate a yes or no answer based on the given context and question. In the provided example, the program is used to determine if it can be logically concluded that there is a kayak nearby based on the given context. The program uses a chain of thought to process the input and generate the answer, taking into account the context and question provided. It uses the language model to reason through the context and question in order to provide the appropriate yes or no answer.\n",
+ "\n",
+ "MODULE: self.generate_answer = dspy.ChainOfThought(ScoNeSignature)\n",
+ "\n",
+ "TASK DEMO(S):\n",
+ "Context: There is a boat nearby\n",
+ "Question: Can we logically conclude for sure that there is a kayak nearby?\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that there is a boat nearby, but we don't know what type of boat it is. It could be a kayak, but it could also be a sailboat, a speedboat, or even a rowboat. We can't logically conclude that it is a kayak for sure.\n",
+ "Answer: No\n",
+ "\n",
+ "\n",
+ "BASIC INSTRUCTION: context, question -> answer\n",
+ "\n",
+ "TIP: Make sure your instruction is very informative and descriptive.\n",
+ "\n",
+ "PROPOSED INSTRUCTION:\u001b[32m Determine if it can be logically concluded that a kayak is nearby based on the given context and question, and generate a yes or no answer using the language model.\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "PROPOSED INSTRUCTION: Determine if it can be logically concluded that a kayak is nearby based on the given context and question, and generate a yes or no answer using the language model.\n",
+ "PROGRAM DESCRIPTION: This program is designed to solve tasks related to logical reasoning and inference based on a given context and question. It appears to work by taking in a context and a question as input, and then using a language model to generate an answer based on logical inference from the context. The `ScoNeSignature` function takes in the context and question as input, and returns an answer (either \"Yes\" or \"No) based on the logical reasoning process. The `ScoNeCoT` class sets up a chain of thought using the `ScoNeSignature` function to generate the answer based on the input context and question. The example provided demonstrates how the program uses the given context and question to generate a logical answer, \"Yes\".\n",
+ "task_demos Context: The men were outside speaking loudly so as to wake the bridesmaid indoors\n",
+ "Question: Can we logically conclude for sure that the men were outside speaking loudly so as to wake the woman indoors?\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the men were outside speaking loudly, and we also know that the bridesmaid is a woman. Therefore, we can logically conclude that the men were outside speaking loudly so as to wake the woman indoors.\n",
+ "Answer: Yes\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "Use the information below to learn about a task that we are trying to solve using calls to an LM, then generate a new instruction that will be used to prompt a Language Model to better solve the task.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "DATASET SUMMARY: A description of the dataset that we are using.\n",
+ "\n",
+ "PROGRAM CODE: Language model program designed to solve a particular task.\n",
+ "\n",
+ "PROGRAM DESCRIPTION: Summary of the task the program is designed to solve, and how it goes about solving it.\n",
+ "\n",
+ "MODULE: The module to create an instruction for.\n",
+ "\n",
+ "TASK DEMO(S): Example inputs/outputs of our module.\n",
+ "\n",
+ "BASIC INSTRUCTION: Basic instruction.\n",
+ "\n",
+ "TIP: A suggestion for how to go about generating the new instruction.\n",
+ "\n",
+ "PROPOSED INSTRUCTION: Propose an instruction that will be used to prompt a Language Model to perform this task.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "DATASET SUMMARY: The dataset consists of examples that involve logical tests and negation related to language understanding and logical reasoning. Each example includes a context, a question, an answer, and a category label, and encompasses both single-scoped and two-scoped logical statements. The dataset seems to be focused on training models for handling negation in logical reasoning and has a broad range of potential applications.\n",
+ "\n",
+ "PROGRAM CODE:\n",
+ "ScoNeSignature(context, question -> answer\n",
+ " instructions='context, question -> answer'\n",
+ " context = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Context:', 'desc': '${context}'})\n",
+ " question = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Question:', 'desc': '${question}'})\n",
+ " answer = Field(annotation=str required=True json_schema_extra={'desc': 'Yes or No', '__dspy_field_type': 'output', 'prefix': 'Answer:'})\n",
+ ")\n",
+ "\n",
+ "class ScoNeCoT(dspy.Module):\n",
+ " def __init__(self):\n",
+ " super().__init__()\n",
+ " self.generate_answer = dspy.ChainOfThought(ScoNeSignature)\n",
+ "\n",
+ " def forward(self, context, question):\n",
+ " return self.generate_answer(context=context, question=question)\n",
+ "\n",
+ "\n",
+ "PROGRAM DESCRIPTION: This program is designed to solve tasks related to logical reasoning and inference based on a given context and question. It appears to work by taking in a context and a question as input, and then using a language model to generate an answer based on logical inference from the context. The `ScoNeSignature` function takes in the context and question as input, and returns an answer (either \"Yes\" or \"No) based on the logical reasoning process. The `ScoNeCoT` class sets up a chain of thought using the `ScoNeSignature` function to generate the answer based on the input context and question. The example provided demonstrates how the program uses the given context and question to generate a logical answer, \"Yes\".\n",
+ "\n",
+ "MODULE: self.generate_answer = dspy.ChainOfThought(ScoNeSignature)\n",
+ "\n",
+ "TASK DEMO(S):\n",
+ "Context: The men were outside speaking loudly so as to wake the bridesmaid indoors\n",
+ "Question: Can we logically conclude for sure that the men were outside speaking loudly so as to wake the woman indoors?\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the men were outside speaking loudly, and we also know that the bridesmaid is a woman. Therefore, we can logically conclude that the men were outside speaking loudly so as to wake the woman indoors.\n",
+ "Answer: Yes\n",
+ "\n",
+ "\n",
+ "BASIC INSTRUCTION: context, question -> answer\n",
+ "\n",
+ "TIP: Make sure your instruction is very informative and descriptive.\n",
+ "\n",
+ "PROPOSED INSTRUCTION:\u001b[32m Please generate a logical answer (either \"Yes\" or \"No\") based on the given context and question using the `ScoNeSignature` function to perform logical reasoning and inference. Take the context and question as input to generate the answer.\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "PROPOSED INSTRUCTION: Please generate a logical answer (either \"Yes\" or \"No\") based on the given context and question using the `ScoNeSignature` function to perform logical reasoning and inference. Take the context and question as input to generate the answer.\n",
+ "PROGRAM DESCRIPTION: This program appears to be designed to solve a task of answering yes or no questions based on a given context and question. The program takes in a context and a question as input and uses a language model to generate an answer. It seems to work by taking the input context and question, passing it through the ScoNeSignature function, and using the ChainOfThought method to generate the answer based on the input context and question. The example provided shows how the program can be used to answer a yes or no question based on a given context and question.\n",
+ "task_demos Context: There is a athlete walking in the city when it's not dark.\n",
+ "Question: Can we logically conclude for sure that there is a person walking in the city when it's not dark?\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the athlete is walking in the city, and we know that the athlete is a person. Therefore, we can logically conclude that there is a person walking in the city.\n",
+ "Answer: Yes\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "Use the information below to learn about a task that we are trying to solve using calls to an LM, then generate a new instruction that will be used to prompt a Language Model to better solve the task.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "DATASET SUMMARY: A description of the dataset that we are using.\n",
+ "\n",
+ "PROGRAM CODE: Language model program designed to solve a particular task.\n",
+ "\n",
+ "PROGRAM DESCRIPTION: Summary of the task the program is designed to solve, and how it goes about solving it.\n",
+ "\n",
+ "MODULE: The module to create an instruction for.\n",
+ "\n",
+ "TASK DEMO(S): Example inputs/outputs of our module.\n",
+ "\n",
+ "BASIC INSTRUCTION: Basic instruction.\n",
+ "\n",
+ "TIP: A suggestion for how to go about generating the new instruction.\n",
+ "\n",
+ "PROPOSED INSTRUCTION: Propose an instruction that will be used to prompt a Language Model to perform this task.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "DATASET SUMMARY: The dataset consists of examples that involve logical tests and negation related to language understanding and logical reasoning. Each example includes a context, a question, an answer, and a category label, and encompasses both single-scoped and two-scoped logical statements. The dataset seems to be focused on training models for handling negation in logical reasoning and has a broad range of potential applications.\n",
+ "\n",
+ "PROGRAM CODE:\n",
+ "ScoNeSignature(context, question -> answer\n",
+ " instructions='context, question -> answer'\n",
+ " context = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Context:', 'desc': '${context}'})\n",
+ " question = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Question:', 'desc': '${question}'})\n",
+ " answer = Field(annotation=str required=True json_schema_extra={'desc': 'Yes or No', '__dspy_field_type': 'output', 'prefix': 'Answer:'})\n",
+ ")\n",
+ "\n",
+ "class ScoNeCoT(dspy.Module):\n",
+ " def __init__(self):\n",
+ " super().__init__()\n",
+ " self.generate_answer = dspy.ChainOfThought(ScoNeSignature)\n",
+ "\n",
+ " def forward(self, context, question):\n",
+ " return self.generate_answer(context=context, question=question)\n",
+ "\n",
+ "\n",
+ "PROGRAM DESCRIPTION: This program appears to be designed to solve a task of answering yes or no questions based on a given context and question. The program takes in a context and a question as input and uses a language model to generate an answer. It seems to work by taking the input context and question, passing it through the ScoNeSignature function, and using the ChainOfThought method to generate the answer based on the input context and question. The example provided shows how the program can be used to answer a yes or no question based on a given context and question.\n",
+ "\n",
+ "MODULE: self.generate_answer = dspy.ChainOfThought(ScoNeSignature)\n",
+ "\n",
+ "TASK DEMO(S):\n",
+ "Context: There is a athlete walking in the city when it's not dark.\n",
+ "Question: Can we logically conclude for sure that there is a person walking in the city when it's not dark?\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the athlete is walking in the city, and we know that the athlete is a person. Therefore, we can logically conclude that there is a person walking in the city.\n",
+ "Answer: Yes\n",
+ "\n",
+ "\n",
+ "BASIC INSTRUCTION: context, question -> answer\n",
+ "\n",
+ "TIP: Make sure your instruction is very informative and descriptive.\n",
+ "\n",
+ "PROPOSED INSTRUCTION:\u001b[32m Given a context and a question, use the provided context and question to determine if the logical conclusion is \"Yes\" or \"No\" and generate the corresponding answer.\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "PROPOSED INSTRUCTION: Given a context and a question, use the provided context and question to determine if the logical conclusion is \"Yes\" or \"No\" and generate the corresponding answer.\n",
+ "PROGRAM DESCRIPTION: This program appears to be designed to solve tasks related to logical reasoning and inference based on given context and question. The program takes in a context and a question as input, and the goal is to produce a logical conclusion or answer based on the given context and question. It uses a language model to generate the answer by reasoning through the given context and question. The example provided demonstrates how the program works by taking in a context and a question and generating a logical answer based on the reasoning process.\n",
+ "task_demos Context: A man is not holding anything in his hands.\n",
+ "Question: Can we logically conclude for sure that a man is not holding tongues in his hands?\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that a man is not holding anything in his hands. This means that he is not holding any physical objects, including tongues. Tongues are not physical objects that can be held in one's hands. Therefore, we can logically conclude that a man is not holding tongues in his hands.\n",
+ "Answer: Yes\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "Use the information below to learn about a task that we are trying to solve using calls to an LM, then generate a new instruction that will be used to prompt a Language Model to better solve the task.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "DATASET SUMMARY: A description of the dataset that we are using.\n",
+ "\n",
+ "PROGRAM CODE: Language model program designed to solve a particular task.\n",
+ "\n",
+ "PROGRAM DESCRIPTION: Summary of the task the program is designed to solve, and how it goes about solving it.\n",
+ "\n",
+ "MODULE: The module to create an instruction for.\n",
+ "\n",
+ "TASK DEMO(S): Example inputs/outputs of our module.\n",
+ "\n",
+ "BASIC INSTRUCTION: Basic instruction.\n",
+ "\n",
+ "TIP: A suggestion for how to go about generating the new instruction.\n",
+ "\n",
+ "PROPOSED INSTRUCTION: Propose an instruction that will be used to prompt a Language Model to perform this task.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "DATASET SUMMARY: The dataset consists of examples that involve logical tests and negation related to language understanding and logical reasoning. Each example includes a context, a question, an answer, and a category label, and encompasses both single-scoped and two-scoped logical statements. The dataset seems to be focused on training models for handling negation in logical reasoning and has a broad range of potential applications.\n",
+ "\n",
+ "PROGRAM CODE:\n",
+ "ScoNeSignature(context, question -> answer\n",
+ " instructions='context, question -> answer'\n",
+ " context = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Context:', 'desc': '${context}'})\n",
+ " question = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Question:', 'desc': '${question}'})\n",
+ " answer = Field(annotation=str required=True json_schema_extra={'desc': 'Yes or No', '__dspy_field_type': 'output', 'prefix': 'Answer:'})\n",
+ ")\n",
+ "\n",
+ "class ScoNeCoT(dspy.Module):\n",
+ " def __init__(self):\n",
+ " super().__init__()\n",
+ " self.generate_answer = dspy.ChainOfThought(ScoNeSignature)\n",
+ "\n",
+ " def forward(self, context, question):\n",
+ " return self.generate_answer(context=context, question=question)\n",
+ "\n",
+ "\n",
+ "PROGRAM DESCRIPTION: This program appears to be designed to solve tasks related to logical reasoning and inference based on given context and question. The program takes in a context and a question as input, and the goal is to produce a logical conclusion or answer based on the given context and question. It uses a language model to generate the answer by reasoning through the given context and question. The example provided demonstrates how the program works by taking in a context and a question and generating a logical answer based on the reasoning process.\n",
+ "\n",
+ "MODULE: self.generate_answer = dspy.ChainOfThought(ScoNeSignature)\n",
+ "\n",
+ "TASK DEMO(S):\n",
+ "Context: A man is not holding anything in his hands.\n",
+ "Question: Can we logically conclude for sure that a man is not holding tongues in his hands?\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that a man is not holding anything in his hands. This means that he is not holding any physical objects, including tongues. Tongues are not physical objects that can be held in one's hands. Therefore, we can logically conclude that a man is not holding tongues in his hands.\n",
+ "Answer: Yes\n",
+ "\n",
+ "\n",
+ "BASIC INSTRUCTION: context, question -> answer\n",
+ "\n",
+ "TIP: Make sure your instruction is very informative and descriptive.\n",
+ "\n",
+ "PROPOSED INSTRUCTION:\u001b[32m Given the context and question, generate a logical answer based on the reasoning process.\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "PROPOSED INSTRUCTION: Given the context and question, generate a logical answer based on the reasoning process.\n",
+ "CANDIDATE PROGRAM:\n",
+ "Predictor 0\n",
+ "i: Provide a yes or no answer based on the given context and question. The context and question are used as input to generate the answer.\n",
+ "p: Answer:\n",
+ "\n",
+ "\n",
+ "...\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 13 / 25 (52.0): 100%|██████████| 25/25 [00:00<00:00, 1054.67it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "FULL TRACE\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 1 / 1 (100.0): 100%|██████████| 1/1 [00:00<00:00, 834.69it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "\n",
+ "\n",
+ "Provide a yes or no answer based on the given context and question. The context and question are used as input to generate the answer.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single celebrity walking in the city?\n",
+ "Answer: No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the statement \"there is not a single person walking in the city\" is false. However, we cannot logically conclude that the statement \"there is not a single slaver walking in the city\" is also false. There could be a single person walking in the city who is not a slaver.\n",
+ "\n",
+ "Answer: No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the statement \"there is not a single person walking in the city\" is false. However, we cannot logically conclude that the statement \"there is not a single slaver walking in the city\" is also false. There could be a single person walking in the city who is not a slaver.\n",
+ "\n",
+ "Answer:\u001b[32m No\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "Provide a yes or no answer based on the given context and question. The context and question are used as input to generate the answer.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single celebrity walking in the city?\n",
+ "Answer: No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the statement \"there is not a single person walking in the city\" is false. However, we cannot logically conclude that the statement \"there is not a single slaver walking in the city\" is also false. There could be a single person walking in the city who is not a slaver.\n",
+ "\n",
+ "Answer: No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the statement \"there is not a single person walking in the city\" is false. However, we cannot logically conclude that the statement \"there is not a single slaver walking in the city\" is also false. There could be a single person walking in the city who is not a slaver.\n",
+ "\n",
+ "Answer:\u001b[32m No\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "...\n",
+ "Score 52.0\n",
+ "Best Combination: 1,{0: [[], [Example({'context': 'the boy, not girl, will play an horn, but not for another week', 'question': 'Can we logically conclude for sure that the boy, not girl, will play an instrument, but not for another week?', 'answer': 'Yes', 'category': 'two_not_scoped'}) (input_keys={'context', 'question'}), Example({'context': 'The person did not like raspberries.', 'question': 'Can we logically conclude for sure that the person did not like fruit?', 'answer': 'No', 'category': 'one_scoped'}) (input_keys={'context', 'question'})], [Example({'augmented': True, 'context': 'It is not true that there is not a single person walking in the city.', 'question': 'Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?', 'rationale': 'produce the answer. We know that the statement \"there is not a single person walking in the city\" is false. However, we cannot logically conclude that the statement \"there is not a single slaver walking in the city\" is also false. There could be a single person walking in the city who is not a slaver.', 'answer': 'No'}) (input_keys=None), Example({'context': 'It is not true that there is not a single person walking in the city.', 'question': 'Can we logically conclude for sure that it is not true that there is not a single celebrity walking in the city?', 'answer': 'No', 'category': 'two_scoped'}) (input_keys={'context', 'question'})], [Example({'augmented': True, 'context': 'A man is not standing on top of a ladder that is leaned against a not so tall tree.', 'question': 'Can we logically conclude for sure that a man is not standing on top of a ladder that is leaned against a not so tall pine?', 'rationale': \"produce the answer. We know that the man is not standing on top of a ladder that is leaned against a not so tall tree. We can't conclude that the ladder is leaned against a pine tree, so we can't conclude that the man is not standing on top of a ladder that is leaned against a not so tall pine.\", 'answer': 'No'}) (input_keys=None), Example({'context': 'The people were outside not even trying to keep their voices down, and they woke up a mistress indoors.', 'question': 'Can we logically conclude for sure that the people were outside not even trying to keep their voices down, and they woke up a woman indoors?', 'answer': 'Yes', 'category': 'one_not_scoped'}) (input_keys={'context', 'question'})], [Example({'augmented': True, 'context': 'The cowboy did not tell other people that he fall off a bronco at the competition.', 'question': 'Can we logically conclude for sure that the cowboy did not tell other people that he fall off a horse at the competition?', 'rationale': 'produce the answer. We know that the cowboy did not tell other people that he fall off a bronco at the competition. Bronco is a type of horse, so it is a type of horse that the cowboy fell off. Therefore, we can logically conclude that the cowboy did not tell other people that he fall off a horse at the competition.', 'answer': 'Yes'}) (input_keys=None), Example({'context': 'The diver has not seen any fish on his dive that did not go too deep.', 'question': 'Can we logically conclude for sure that the diver has not seen any tuna on his dive that did not go too deep?', 'answer': 'Yes', 'category': 'one_scoped_one_not_scoped'}) (input_keys={'context', 'question'})], [Example({'augmented': True, 'context': 'It is not true that there is not a single person walking in the city.', 'question': 'Can we logically conclude for sure that it is not true that there is not a single celebrity walking in the city?', 'rationale': 'produce the answer. We know that it is not true that there is not a single person walking in the city. This does not necessarily mean that there is a celebrity walking in the city. There could be a non-celebrity person walking in the city.', 'answer': 'No'}) (input_keys=None), Example({'context': 'The cowboy did not tell other people that he fall off a bronco at the competition.', 'question': 'Can we logically conclude for sure that the cowboy did not tell other people that he fall off a horse at the competition?', 'answer': 'Yes', 'category': 'one_not_scoped'}) (input_keys={'context', 'question'})], [Example({'augmented': True, 'context': 'There is a boat nearby', 'question': 'Can we logically conclude for sure that there is a kayak nearby?', 'rationale': \"produce the answer. We know that there is a boat nearby, but we don't know what type of boat it is. It could be a kayak, but it could also be a sailboat, a speedboat, or even a rowboat. We can't logically conclude that it is a kayak for sure.\", 'answer': 'No'}) (input_keys=None), Example({'context': 'the boy plays an instrument', 'question': 'Can we logically conclude for sure that the boy plays an flute?', 'answer': 'No', 'category': 'no_negation'}) (input_keys={'context', 'question'})], [Example({'augmented': True, 'context': 'The men were outside speaking loudly so as to wake the bridesmaid indoors', 'question': 'Can we logically conclude for sure that the men were outside speaking loudly so as to wake the woman indoors?', 'rationale': 'produce the answer. We know that the men were outside speaking loudly, and we also know that the bridesmaid is a woman. Therefore, we can logically conclude that the men were outside speaking loudly so as to wake the woman indoors.', 'answer': 'Yes'}) (input_keys=None), Example({'context': 'There is not a single person not walking in the city.', 'question': 'Can we logically conclude for sure that there is not a single matriarch not walking in the city?', 'answer': 'Yes', 'category': 'one_scoped_one_not_scoped'}) (input_keys={'context', 'question'})], [Example({'augmented': True, 'context': \"There is a athlete walking in the city when it's not dark.\", 'question': \"Can we logically conclude for sure that there is a person walking in the city when it's not dark?\", 'rationale': 'produce the answer. We know that the athlete is walking in the city, and we know that the athlete is a person. Therefore, we can logically conclude that there is a person walking in the city.', 'answer': 'Yes'}) (input_keys=None), Example({'context': 'The man is not listening to techno.', 'question': 'Can we logically conclude for sure that the man is not listening to music?', 'answer': 'No', 'category': 'one_scoped'}) (input_keys={'context', 'question'})], [Example({'augmented': True, 'context': 'A man is not holding anything in his hands.', 'question': 'Can we logically conclude for sure that a man is not holding tongues in his hands?', 'rationale': \"produce the answer. We know that a man is not holding anything in his hands. This means that he is not holding any physical objects, including tongues. Tongues are not physical objects that can be held in one's hands. Therefore, we can logically conclude that a man is not holding tongues in his hands.\", 'answer': 'Yes'}) (input_keys=None), Example({'context': 'To believe that a man walks confidently not dropping produce is to believe a falsity.', 'question': 'Can we logically conclude for sure that to believe that a man walks confidently not dropping pears is to believe a falsity?', 'answer': 'No', 'category': 'two_scoped'}) (input_keys={'context', 'question'})]]} with Mean = 52.0\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 113 / 200 (56.5): 100%|██████████| 200/200 [00:00<00:00, 662.89it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "UPDATING BEST SCORE WITH 56.5\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 98 / 200 (49.0): 100%|██████████| 200/200 [00:00<00:00, 773.80it/s]\n",
+ "[I 2024-06-20 22:06:47,424] Trial 0 finished with value: 52.0 and parameters: {'0_predictor_instruction': 1, '0_predictor_demos': 2}. Best is trial 0 with value: 52.0.\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "CANDIDATE PROGRAM:\n",
+ "Predictor 0\n",
+ "i: Determine if it can be logically concluded that a kayak is nearby based on the given context and question, and generate a yes or no answer using the language model.\n",
+ "p: Answer:\n",
+ "\n",
+ "\n",
+ "...\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 11 / 25 (44.0): 100%|██████████| 25/25 [00:00<00:00, 1530.19it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "FULL TRACE\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 1 / 1 (100.0): 100%|██████████| 1/1 [00:00<00:00, 441.13it/s]\n",
+ "[I 2024-06-20 22:06:47,470] Trial 1 finished with value: 44.0 and parameters: {'0_predictor_instruction': 6, '0_predictor_demos': 2}. Best is trial 0 with value: 52.0.\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "\n",
+ "\n",
+ "Determine if it can be logically concluded that a kayak is nearby based on the given context and question, and generate a yes or no answer using the language model.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single celebrity walking in the city?\n",
+ "Answer: No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the statement \"there is not a single person walking in the city\" is false. However, we cannot logically conclude that the statement \"there is not a single slaver walking in the city\" is also false. There could be a single person walking in the city who is not a slaver.\n",
+ "\n",
+ "Answer: No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the statement \"there is not a single person walking in the city\" is false. However, we cannot logically conclude that the statement \"there is not a single slaver walking in the city\" is also false. There could be a single person walking in the city who is not a slaver.\n",
+ "\n",
+ "Answer:\u001b[32m No\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "Determine if it can be logically concluded that a kayak is nearby based on the given context and question, and generate a yes or no answer using the language model.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single celebrity walking in the city?\n",
+ "Answer: No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the statement \"there is not a single person walking in the city\" is false. However, we cannot logically conclude that the statement \"there is not a single slaver walking in the city\" is also false. There could be a single person walking in the city who is not a slaver.\n",
+ "\n",
+ "Answer: No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the statement \"there is not a single person walking in the city\" is false. However, we cannot logically conclude that the statement \"there is not a single slaver walking in the city\" is also false. There could be a single person walking in the city who is not a slaver.\n",
+ "\n",
+ "Answer:\u001b[32m No\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "...\n",
+ "Score 44.0\n",
+ "CANDIDATE PROGRAM:\n",
+ "Predictor 0\n",
+ "i: Given a context and a question, use the provided context and question to determine if the logical conclusion is \"Yes\" or \"No\" and generate the corresponding answer.\n",
+ "p: Answer:\n",
+ "\n",
+ "\n",
+ "...\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 14 / 25 (56.0): 100%|██████████| 25/25 [00:00<00:00, 2381.12it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "FULL TRACE\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 1 / 1 (100.0): 100%|██████████| 1/1 [00:00<00:00, 590.66it/s]\n",
+ "[I 2024-06-20 22:06:47,520] Trial 2 finished with value: 56.0 and parameters: {'0_predictor_instruction': 8, '0_predictor_demos': 6}. Best is trial 2 with value: 56.0.\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "\n",
+ "\n",
+ "Given a context and a question, use the provided context and question to determine if the logical conclusion is \"Yes\" or \"No\" and generate the corresponding answer.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: the boy plays an instrument\n",
+ "Question: Can we logically conclude for sure that the boy plays an flute?\n",
+ "Answer: No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: There is a boat nearby\n",
+ "\n",
+ "Question: Can we logically conclude for sure that there is a kayak nearby?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that there is a boat nearby, but we don't know what type of boat it is. It could be a kayak, but it could also be a sailboat, a speedboat, or even a rowboat. We can't logically conclude that it is a kayak for sure.\n",
+ "\n",
+ "Answer: No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city, but we don't know anything about the type of person. It could be a slaver, but it could also be a tourist, a local, or a homeless person. We can't logically conclude that it is a slaver for sure.\n",
+ "\n",
+ "Answer:\u001b[32m No\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "Given a context and a question, use the provided context and question to determine if the logical conclusion is \"Yes\" or \"No\" and generate the corresponding answer.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: the boy plays an instrument\n",
+ "Question: Can we logically conclude for sure that the boy plays an flute?\n",
+ "Answer: No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: There is a boat nearby\n",
+ "\n",
+ "Question: Can we logically conclude for sure that there is a kayak nearby?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that there is a boat nearby, but we don't know what type of boat it is. It could be a kayak, but it could also be a sailboat, a speedboat, or even a rowboat. We can't logically conclude that it is a kayak for sure.\n",
+ "\n",
+ "Answer: No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city, but we don't know anything about the type of person. It could be a slaver, but it could also be a tourist, a local, or a homeless person. We can't logically conclude that it is a slaver for sure.\n",
+ "\n",
+ "Answer:\u001b[32m No\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "...\n",
+ "Score 56.0\n",
+ "CANDIDATE PROGRAM:\n",
+ "Predictor 0\n",
+ "i: Please use the provided context and question to generate a logical inference for the answer using the language model. The answer should be \"Yes\" or \"No\" based on the logical reasoning and inference from the given context and question.\n",
+ "p: Answer:\n",
+ "\n",
+ "\n",
+ "...\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 17 / 25 (68.0): 100%|██████████| 25/25 [00:00<00:00, 382.65it/s] \n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "FULL TRACE\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 1 / 1 (100.0): 100%|██████████| 1/1 [00:00<00:00, 181.20it/s]\n",
+ "[I 2024-06-20 22:06:47,625] Trial 3 finished with value: 68.0 and parameters: {'0_predictor_instruction': 4, '0_predictor_demos': 5}. Best is trial 3 with value: 68.0.\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "\n",
+ "\n",
+ "Please use the provided context and question to generate a logical inference for the answer using the language model. The answer should be \"Yes\" or \"No\" based on the logical reasoning and inference from the given context and question.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: The cowboy did not tell other people that he fall off a bronco at the competition.\n",
+ "Question: Can we logically conclude for sure that the cowboy did not tell other people that he fall off a horse at the competition?\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single celebrity walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This does not necessarily mean that there is a celebrity walking in the city. There could be a non-celebrity person walking in the city.\n",
+ "\n",
+ "Answer: No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This does not necessarily mean that there is a slaver walking in the city. There could be a non-slaver person walking in the city. Additionally, slavery is illegal in most places, so it is highly unlikely that there would be a slaver walking in the city.\n",
+ "\n",
+ "Answer:\u001b[32m No\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "Please use the provided context and question to generate a logical inference for the answer using the language model. The answer should be \"Yes\" or \"No\" based on the logical reasoning and inference from the given context and question.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: The cowboy did not tell other people that he fall off a bronco at the competition.\n",
+ "Question: Can we logically conclude for sure that the cowboy did not tell other people that he fall off a horse at the competition?\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single celebrity walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This does not necessarily mean that there is a celebrity walking in the city. There could be a non-celebrity person walking in the city.\n",
+ "\n",
+ "Answer: No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This does not necessarily mean that there is a slaver walking in the city. There could be a non-slaver person walking in the city. Additionally, slavery is illegal in most places, so it is highly unlikely that there would be a slaver walking in the city.\n",
+ "\n",
+ "Answer:\u001b[32m No\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "...\n",
+ "Score 68.0\n",
+ "CANDIDATE PROGRAM:\n",
+ "Predictor 0\n",
+ "i: Given a context and a question, use logical reasoning to determine if a logical conclusion can be made and generate a \"yes\" or \"no\" answer based on the provided information.\n",
+ "p: Answer:\n",
+ "\n",
+ "\n",
+ "...\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 14 / 25 (56.0): 100%|██████████| 25/25 [00:00<00:00, 1328.35it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "FULL TRACE\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 1 / 1 (100.0): 100%|██████████| 1/1 [00:00<00:00, 568.87it/s]\n",
+ "[I 2024-06-20 22:06:47,675] Trial 4 finished with value: 56.0 and parameters: {'0_predictor_instruction': 3, '0_predictor_demos': 8}. Best is trial 3 with value: 68.0.\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "\n",
+ "\n",
+ "Given a context and a question, use logical reasoning to determine if a logical conclusion can be made and generate a \"yes\" or \"no\" answer based on the provided information.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: The man is not listening to techno.\n",
+ "Question: Can we logically conclude for sure that the man is not listening to music?\n",
+ "Answer: No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: There is a athlete walking in the city when it's not dark.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that there is a person walking in the city when it's not dark?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the athlete is walking in the city, and we know that the athlete is a person. Therefore, we can logically conclude that there is a person walking in the city.\n",
+ "\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. However, we cannot logically conclude that it is not true that there is not a single slaver walking in the city, because we have no information about the presence of slavers in the city.\n",
+ "\n",
+ "Answer:\u001b[32m No\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "Given a context and a question, use logical reasoning to determine if a logical conclusion can be made and generate a \"yes\" or \"no\" answer based on the provided information.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: The man is not listening to techno.\n",
+ "Question: Can we logically conclude for sure that the man is not listening to music?\n",
+ "Answer: No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: There is a athlete walking in the city when it's not dark.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that there is a person walking in the city when it's not dark?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the athlete is walking in the city, and we know that the athlete is a person. Therefore, we can logically conclude that there is a person walking in the city.\n",
+ "\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. However, we cannot logically conclude that it is not true that there is not a single slaver walking in the city, because we have no information about the presence of slavers in the city.\n",
+ "\n",
+ "Answer:\u001b[32m No\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "...\n",
+ "Score 56.0\n",
+ "CANDIDATE PROGRAM:\n",
+ "Predictor 0\n",
+ "i: LogicalNegation(context: str, question: str -> answer: str) \"Given a context and a question involving negation, determine the logical conclusion and provide a yes or no answer based on the reasoning.\n",
+ "p: Answer:\n",
+ "\n",
+ "\n",
+ "...\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 18 / 25 (72.0): 100%|██████████| 25/25 [00:00<00:00, 1372.19it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "FULL TRACE\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 1 / 1 (100.0): 100%|██████████| 1/1 [00:00<00:00, 293.27it/s]\n",
+ "[I 2024-06-20 22:06:47,724] Trial 5 finished with value: 72.0 and parameters: {'0_predictor_instruction': 2, '0_predictor_demos': 3}. Best is trial 5 with value: 72.0.\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "\n",
+ "\n",
+ "LogicalNegation(context: str, question: str -> answer: str) \"Given a context and a question involving negation, determine the logical conclusion and provide a yes or no answer based on the reasoning.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: The people were outside not even trying to keep their voices down, and they woke up a mistress indoors.\n",
+ "Question: Can we logically conclude for sure that the people were outside not even trying to keep their voices down, and they woke up a woman indoors?\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: A man is not standing on top of a ladder that is leaned against a not so tall tree.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that a man is not standing on top of a ladder that is leaned against a not so tall pine?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the man is not standing on top of a ladder that is leaned against a not so tall tree. We can't conclude that the ladder is leaned against a pine tree, so we can't conclude that the man is not standing on top of a ladder that is leaned against a not so tall pine.\n",
+ "\n",
+ "Answer: No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This means that there is at least one person walking in the city. We can't conclude that this person is a slaver, so we can't conclude that it is not true that there is not a single slaver walking in the city.\n",
+ "\n",
+ "Answer:\u001b[32m No\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "LogicalNegation(context: str, question: str -> answer: str) \"Given a context and a question involving negation, determine the logical conclusion and provide a yes or no answer based on the reasoning.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: The people were outside not even trying to keep their voices down, and they woke up a mistress indoors.\n",
+ "Question: Can we logically conclude for sure that the people were outside not even trying to keep their voices down, and they woke up a woman indoors?\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: A man is not standing on top of a ladder that is leaned against a not so tall tree.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that a man is not standing on top of a ladder that is leaned against a not so tall pine?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the man is not standing on top of a ladder that is leaned against a not so tall tree. We can't conclude that the ladder is leaned against a pine tree, so we can't conclude that the man is not standing on top of a ladder that is leaned against a not so tall pine.\n",
+ "\n",
+ "Answer: No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This means that there is at least one person walking in the city. We can't conclude that this person is a slaver, so we can't conclude that it is not true that there is not a single slaver walking in the city.\n",
+ "\n",
+ "Answer:\u001b[32m No\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "...\n",
+ "Score 72.0\n",
+ "CANDIDATE PROGRAM:\n",
+ "Predictor 0\n",
+ "i: Given the context and question, generate a logical answer based on the reasoning process.\n",
+ "p: Answer:\n",
+ "\n",
+ "\n",
+ "...\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 14 / 25 (56.0): 100%|██████████| 25/25 [00:00<00:00, 778.21it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "FULL TRACE\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 1 / 1 (100.0): 100%|██████████| 1/1 [00:00<00:00, 587.44it/s]\n",
+ "[I 2024-06-20 22:06:47,784] Trial 6 finished with value: 56.0 and parameters: {'0_predictor_instruction': 9, '0_predictor_demos': 5}. Best is trial 5 with value: 72.0.\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "\n",
+ "\n",
+ "Given the context and question, generate a logical answer based on the reasoning process.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: The cowboy did not tell other people that he fall off a bronco at the competition.\n",
+ "Question: Can we logically conclude for sure that the cowboy did not tell other people that he fall off a horse at the competition?\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single celebrity walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This does not necessarily mean that there is a celebrity walking in the city. There could be a non-celebrity person walking in the city.\n",
+ "\n",
+ "Answer: No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This does not necessarily mean that there is a slaver walking in the city. There could be a non-slaver person walking in the city. Additionally, slavery is illegal in most places, so it is highly unlikely that there is a slaver walking in the city.\n",
+ "\n",
+ "Answer:\u001b[32m No\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "Given the context and question, generate a logical answer based on the reasoning process.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: The cowboy did not tell other people that he fall off a bronco at the competition.\n",
+ "Question: Can we logically conclude for sure that the cowboy did not tell other people that he fall off a horse at the competition?\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single celebrity walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This does not necessarily mean that there is a celebrity walking in the city. There could be a non-celebrity person walking in the city.\n",
+ "\n",
+ "Answer: No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This does not necessarily mean that there is a slaver walking in the city. There could be a non-slaver person walking in the city. Additionally, slavery is illegal in most places, so it is highly unlikely that there is a slaver walking in the city.\n",
+ "\n",
+ "Answer:\u001b[32m No\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "...\n",
+ "Score 56.0\n",
+ "CANDIDATE PROGRAM:\n",
+ "Predictor 0\n",
+ "i: Please generate a logical answer (either \"Yes\" or \"No\") based on the given context and question using the `ScoNeSignature` function to perform logical reasoning and inference. Take the context and question as input to generate the answer.\n",
+ "p: Answer:\n",
+ "\n",
+ "\n",
+ "...\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 14 / 25 (56.0): 100%|██████████| 25/25 [00:00<00:00, 1888.78it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "FULL TRACE\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 0 / 1 (0.0): 100%|██████████| 1/1 [00:00<00:00, 319.69it/s]\n",
+ "[I 2024-06-20 22:06:47,831] Trial 7 finished with value: 56.0 and parameters: {'0_predictor_instruction': 7, '0_predictor_demos': 4}. Best is trial 5 with value: 72.0.\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "\n",
+ "\n",
+ "Please generate a logical answer (either \"Yes\" or \"No\") based on the given context and question using the `ScoNeSignature` function to perform logical reasoning and inference. Take the context and question as input to generate the answer.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: The diver has not seen any fish on his dive that did not go too deep.\n",
+ "Question: Can we logically conclude for sure that the diver has not seen any tuna on his dive that did not go too deep?\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: The cowboy did not tell other people that he fall off a bronco at the competition.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that the cowboy did not tell other people that he fall off a horse at the competition?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the cowboy did not tell other people that he fall off a bronco at the competition. Bronco is a type of horse, so it is a type of horse that the cowboy fell off. Therefore, we can logically conclude that the cowboy did not tell other people that he fall off a horse at the competition.\n",
+ "\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This means that there is at least one person walking in the city. Slaver is a type of person, so it is a type of person that is walking in the city. Therefore, we can logically conclude that it is not true that there is not a single slaver walking in the city.\n",
+ "\n",
+ "Answer:\u001b[32m Yes\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "Please generate a logical answer (either \"Yes\" or \"No\") based on the given context and question using the `ScoNeSignature` function to perform logical reasoning and inference. Take the context and question as input to generate the answer.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: The diver has not seen any fish on his dive that did not go too deep.\n",
+ "Question: Can we logically conclude for sure that the diver has not seen any tuna on his dive that did not go too deep?\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: The cowboy did not tell other people that he fall off a bronco at the competition.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that the cowboy did not tell other people that he fall off a horse at the competition?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the cowboy did not tell other people that he fall off a bronco at the competition. Bronco is a type of horse, so it is a type of horse that the cowboy fell off. Therefore, we can logically conclude that the cowboy did not tell other people that he fall off a horse at the competition.\n",
+ "\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This means that there is at least one person walking in the city. Slaver is a type of person, so it is a type of person that is walking in the city. Therefore, we can logically conclude that it is not true that there is not a single slaver walking in the city.\n",
+ "\n",
+ "Answer:\u001b[32m Yes\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "...\n",
+ "Score 56.0\n",
+ "CANDIDATE PROGRAM:\n",
+ "Predictor 0\n",
+ "i: context, question -> answer\n",
+ "p: Answer:\n",
+ "\n",
+ "\n",
+ "...\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 18 / 25 (72.0): 100%|██████████| 25/25 [00:00<00:00, 1093.36it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "FULL TRACE\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 0 / 1 (0.0): 100%|██████████| 1/1 [00:00<00:00, 405.17it/s]\n",
+ "[I 2024-06-20 22:06:47,882] Trial 8 finished with value: 72.0 and parameters: {'0_predictor_instruction': 0, '0_predictor_demos': 7}. Best is trial 5 with value: 72.0.\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "\n",
+ "\n",
+ "context, question -> answer\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: There is not a single person not walking in the city.\n",
+ "Question: Can we logically conclude for sure that there is not a single matriarch not walking in the city?\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: The men were outside speaking loudly so as to wake the bridesmaid indoors\n",
+ "\n",
+ "Question: Can we logically conclude for sure that the men were outside speaking loudly so as to wake the woman indoors?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the men were outside speaking loudly, and we also know that the bridesmaid is a woman. Therefore, we can logically conclude that the men were outside speaking loudly so as to wake the woman indoors.\n",
+ "\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. Therefore, we can logically conclude that it is not true that there is not a single slaver walking in the city.\n",
+ "\n",
+ "Answer:\u001b[32m Yes\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "context, question -> answer\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: There is not a single person not walking in the city.\n",
+ "Question: Can we logically conclude for sure that there is not a single matriarch not walking in the city?\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: The men were outside speaking loudly so as to wake the bridesmaid indoors\n",
+ "\n",
+ "Question: Can we logically conclude for sure that the men were outside speaking loudly so as to wake the woman indoors?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the men were outside speaking loudly, and we also know that the bridesmaid is a woman. Therefore, we can logically conclude that the men were outside speaking loudly so as to wake the woman indoors.\n",
+ "\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. Therefore, we can logically conclude that it is not true that there is not a single slaver walking in the city.\n",
+ "\n",
+ "Answer:\u001b[32m Yes\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "...\n",
+ "Score 72.0\n",
+ "CANDIDATE PROGRAM:\n",
+ "Predictor 0\n",
+ "i: Given the context and question, generate a logical answer based on the reasoning process.\n",
+ "p: Answer:\n",
+ "\n",
+ "\n",
+ "...\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 20 / 25 (80.0): 100%|██████████| 25/25 [00:00<00:00, 374.52it/s] \n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "FULL TRACE\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 0 / 1 (0.0): 100%|██████████| 1/1 [00:00<00:00, 257.40it/s]\n",
+ "[I 2024-06-20 22:06:47,997] Trial 9 finished with value: 80.0 and parameters: {'0_predictor_instruction': 9, '0_predictor_demos': 7}. Best is trial 9 with value: 80.0.\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "\n",
+ "\n",
+ "Given the context and question, generate a logical answer based on the reasoning process.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: There is not a single person not walking in the city.\n",
+ "Question: Can we logically conclude for sure that there is not a single matriarch not walking in the city?\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: The men were outside speaking loudly so as to wake the bridesmaid indoors\n",
+ "\n",
+ "Question: Can we logically conclude for sure that the men were outside speaking loudly so as to wake the woman indoors?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the men were outside speaking loudly, and we also know that the bridesmaid is a woman. Therefore, we can logically conclude that the men were outside speaking loudly so as to wake the woman indoors.\n",
+ "\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. Therefore, we can logically conclude that it is not true that there is not a single slaver walking in the city.\n",
+ "\n",
+ "Answer:\u001b[32m Yes\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "Given the context and question, generate a logical answer based on the reasoning process.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: There is not a single person not walking in the city.\n",
+ "Question: Can we logically conclude for sure that there is not a single matriarch not walking in the city?\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: The men were outside speaking loudly so as to wake the bridesmaid indoors\n",
+ "\n",
+ "Question: Can we logically conclude for sure that the men were outside speaking loudly so as to wake the woman indoors?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the men were outside speaking loudly, and we also know that the bridesmaid is a woman. Therefore, we can logically conclude that the men were outside speaking loudly so as to wake the woman indoors.\n",
+ "\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. Therefore, we can logically conclude that it is not true that there is not a single slaver walking in the city.\n",
+ "\n",
+ "Answer:\u001b[32m Yes\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "...\n",
+ "Score 80.0\n",
+ "CANDIDATE PROGRAM:\n",
+ "Predictor 0\n",
+ "i: Given the context and question, generate a logical answer based on the reasoning process.\n",
+ "p: Answer:\n",
+ "\n",
+ "\n",
+ "...\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 19 / 25 (76.0): 100%|██████████| 25/25 [00:00<00:00, 1396.18it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "FULL TRACE\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 0 / 1 (0.0): 100%|██████████| 1/1 [00:00<00:00, 459.95it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "\n",
+ "\n",
+ "Given the context and question, generate a logical answer based on the reasoning process.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: To believe that a man walks confidently not dropping produce is to believe a falsity.\n",
+ "Question: Can we logically conclude for sure that to believe that a man walks confidently not dropping pears is to believe a falsity?\n",
+ "Answer: No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: A man is not holding anything in his hands.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that a man is not holding tongues in his hands?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that a man is not holding anything in his hands. This means that he is not holding any physical objects, including tongues. Tongues are not physical objects that can be held in one's hands. Therefore, we can logically conclude that a man is not holding tongues in his hands.\n",
+ "\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This means that there must be at least one person walking in the city. Since the term \"person\" is a broader category that includes \"slaver\", we can logically conclude that it is not true that there is not a single slaver walking in the city.\n",
+ "\n",
+ "Answer:\u001b[32m Yes\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "Given the context and question, generate a logical answer based on the reasoning process.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: To believe that a man walks confidently not dropping produce is to believe a falsity.\n",
+ "Question: Can we logically conclude for sure that to believe that a man walks confidently not dropping pears is to believe a falsity?\n",
+ "Answer: No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: A man is not holding anything in his hands.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that a man is not holding tongues in his hands?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that a man is not holding anything in his hands. This means that he is not holding any physical objects, including tongues. Tongues are not physical objects that can be held in one's hands. Therefore, we can logically conclude that a man is not holding tongues in his hands.\n",
+ "\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This means that there must be at least one person walking in the city. Since the term \"person\" is a broader category that includes \"slaver\", we can logically conclude that it is not true that there is not a single slaver walking in the city.\n",
+ "\n",
+ "Answer:\u001b[32m Yes\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "...\n",
+ "Score 76.0\n",
+ "Best Combination: 2,{0: [[], [Example({'context': 'the boy, not girl, will play an horn, but not for another week', 'question': 'Can we logically conclude for sure that the boy, not girl, will play an instrument, but not for another week?', 'answer': 'Yes', 'category': 'two_not_scoped'}) (input_keys={'context', 'question'}), Example({'context': 'The person did not like raspberries.', 'question': 'Can we logically conclude for sure that the person did not like fruit?', 'answer': 'No', 'category': 'one_scoped'}) (input_keys={'context', 'question'})], [Example({'augmented': True, 'context': 'It is not true that there is not a single person walking in the city.', 'question': 'Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?', 'rationale': 'produce the answer. We know that the statement \"there is not a single person walking in the city\" is false. However, we cannot logically conclude that the statement \"there is not a single slaver walking in the city\" is also false. There could be a single person walking in the city who is not a slaver.', 'answer': 'No'}) (input_keys=None), Example({'context': 'It is not true that there is not a single person walking in the city.', 'question': 'Can we logically conclude for sure that it is not true that there is not a single celebrity walking in the city?', 'answer': 'No', 'category': 'two_scoped'}) (input_keys={'context', 'question'})], [Example({'augmented': True, 'context': 'A man is not standing on top of a ladder that is leaned against a not so tall tree.', 'question': 'Can we logically conclude for sure that a man is not standing on top of a ladder that is leaned against a not so tall pine?', 'rationale': \"produce the answer. We know that the man is not standing on top of a ladder that is leaned against a not so tall tree. We can't conclude that the ladder is leaned against a pine tree, so we can't conclude that the man is not standing on top of a ladder that is leaned against a not so tall pine.\", 'answer': 'No'}) (input_keys=None), Example({'context': 'The people were outside not even trying to keep their voices down, and they woke up a mistress indoors.', 'question': 'Can we logically conclude for sure that the people were outside not even trying to keep their voices down, and they woke up a woman indoors?', 'answer': 'Yes', 'category': 'one_not_scoped'}) (input_keys={'context', 'question'})], [Example({'augmented': True, 'context': 'The cowboy did not tell other people that he fall off a bronco at the competition.', 'question': 'Can we logically conclude for sure that the cowboy did not tell other people that he fall off a horse at the competition?', 'rationale': 'produce the answer. We know that the cowboy did not tell other people that he fall off a bronco at the competition. Bronco is a type of horse, so it is a type of horse that the cowboy fell off. Therefore, we can logically conclude that the cowboy did not tell other people that he fall off a horse at the competition.', 'answer': 'Yes'}) (input_keys=None), Example({'context': 'The diver has not seen any fish on his dive that did not go too deep.', 'question': 'Can we logically conclude for sure that the diver has not seen any tuna on his dive that did not go too deep?', 'answer': 'Yes', 'category': 'one_scoped_one_not_scoped'}) (input_keys={'context', 'question'})], [Example({'augmented': True, 'context': 'It is not true that there is not a single person walking in the city.', 'question': 'Can we logically conclude for sure that it is not true that there is not a single celebrity walking in the city?', 'rationale': 'produce the answer. We know that it is not true that there is not a single person walking in the city. This does not necessarily mean that there is a celebrity walking in the city. There could be a non-celebrity person walking in the city.', 'answer': 'No'}) (input_keys=None), Example({'context': 'The cowboy did not tell other people that he fall off a bronco at the competition.', 'question': 'Can we logically conclude for sure that the cowboy did not tell other people that he fall off a horse at the competition?', 'answer': 'Yes', 'category': 'one_not_scoped'}) (input_keys={'context', 'question'})], [Example({'augmented': True, 'context': 'There is a boat nearby', 'question': 'Can we logically conclude for sure that there is a kayak nearby?', 'rationale': \"produce the answer. We know that there is a boat nearby, but we don't know what type of boat it is. It could be a kayak, but it could also be a sailboat, a speedboat, or even a rowboat. We can't logically conclude that it is a kayak for sure.\", 'answer': 'No'}) (input_keys=None), Example({'context': 'the boy plays an instrument', 'question': 'Can we logically conclude for sure that the boy plays an flute?', 'answer': 'No', 'category': 'no_negation'}) (input_keys={'context', 'question'})], [Example({'augmented': True, 'context': 'The men were outside speaking loudly so as to wake the bridesmaid indoors', 'question': 'Can we logically conclude for sure that the men were outside speaking loudly so as to wake the woman indoors?', 'rationale': 'produce the answer. We know that the men were outside speaking loudly, and we also know that the bridesmaid is a woman. Therefore, we can logically conclude that the men were outside speaking loudly so as to wake the woman indoors.', 'answer': 'Yes'}) (input_keys=None), Example({'context': 'There is not a single person not walking in the city.', 'question': 'Can we logically conclude for sure that there is not a single matriarch not walking in the city?', 'answer': 'Yes', 'category': 'one_scoped_one_not_scoped'}) (input_keys={'context', 'question'})], [Example({'augmented': True, 'context': \"There is a athlete walking in the city when it's not dark.\", 'question': \"Can we logically conclude for sure that there is a person walking in the city when it's not dark?\", 'rationale': 'produce the answer. We know that the athlete is walking in the city, and we know that the athlete is a person. Therefore, we can logically conclude that there is a person walking in the city.', 'answer': 'Yes'}) (input_keys=None), Example({'context': 'The man is not listening to techno.', 'question': 'Can we logically conclude for sure that the man is not listening to music?', 'answer': 'No', 'category': 'one_scoped'}) (input_keys={'context', 'question'})], [Example({'augmented': True, 'context': 'A man is not holding anything in his hands.', 'question': 'Can we logically conclude for sure that a man is not holding tongues in his hands?', 'rationale': \"produce the answer. We know that a man is not holding anything in his hands. This means that he is not holding any physical objects, including tongues. Tongues are not physical objects that can be held in one's hands. Therefore, we can logically conclude that a man is not holding tongues in his hands.\", 'answer': 'Yes'}) (input_keys=None), Example({'context': 'To believe that a man walks confidently not dropping produce is to believe a falsity.', 'question': 'Can we logically conclude for sure that to believe that a man walks confidently not dropping pears is to believe a falsity?', 'answer': 'No', 'category': 'two_scoped'}) (input_keys={'context', 'question'})]]} with Mean = 72.0\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 139 / 200 (69.5): 100%|██████████| 200/200 [00:00<00:00, 853.68it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "UPDATING BEST SCORE WITH 69.5\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 128 / 200 (64.0): 100%|██████████| 200/200 [00:00<00:00, 738.31it/s]\n",
+ "[I 2024-06-20 22:06:48,603] Trial 10 finished with value: 76.0 and parameters: {'0_predictor_instruction': 9, '0_predictor_demos': 9}. Best is trial 9 with value: 80.0.\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "CANDIDATE PROGRAM:\n",
+ "Predictor 0\n",
+ "i: Given the context and question, generate a logical answer based on the reasoning process.\n",
+ "p: Answer:\n",
+ "\n",
+ "\n",
+ "...\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 18 / 25 (72.0): 100%|██████████| 25/25 [00:00<00:00, 1535.25it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "FULL TRACE\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 0 / 1 (0.0): 100%|██████████| 1/1 [00:00<00:00, 725.16it/s]\n",
+ "[I 2024-06-20 22:06:48,642] Trial 11 finished with value: 72.0 and parameters: {'0_predictor_instruction': 9, '0_predictor_demos': 9}. Best is trial 9 with value: 80.0.\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "\n",
+ "\n",
+ "Given the context and question, generate a logical answer based on the reasoning process.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: To believe that a man walks confidently not dropping produce is to believe a falsity.\n",
+ "Question: Can we logically conclude for sure that to believe that a man walks confidently not dropping pears is to believe a falsity?\n",
+ "Answer: No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: A man is not holding anything in his hands.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that a man is not holding tongues in his hands?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that a man is not holding anything in his hands. This means that he is not holding any physical objects, including tongues. Tongues are not physical objects that can be held in one's hands. Therefore, we can logically conclude that a man is not holding tongues in his hands.\n",
+ "\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This means that there must be at least one person walking in the city. Since the term \"person\" is a broader category that includes \"slaver\", we can logically conclude that it is not true that there is not a single slaver walking in the city.\n",
+ "\n",
+ "Answer:\u001b[32m Yes\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "Given the context and question, generate a logical answer based on the reasoning process.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: To believe that a man walks confidently not dropping produce is to believe a falsity.\n",
+ "Question: Can we logically conclude for sure that to believe that a man walks confidently not dropping pears is to believe a falsity?\n",
+ "Answer: No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: A man is not holding anything in his hands.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that a man is not holding tongues in his hands?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that a man is not holding anything in his hands. This means that he is not holding any physical objects, including tongues. Tongues are not physical objects that can be held in one's hands. Therefore, we can logically conclude that a man is not holding tongues in his hands.\n",
+ "\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This means that there must be at least one person walking in the city. Since the term \"person\" is a broader category that includes \"slaver\", we can logically conclude that it is not true that there is not a single slaver walking in the city.\n",
+ "\n",
+ "Answer:\u001b[32m Yes\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "...\n",
+ "Score 72.0\n",
+ "CANDIDATE PROGRAM:\n",
+ "Predictor 0\n",
+ "i: Given the context and question, generate a logical answer based on the reasoning process.\n",
+ "p: Answer:\n",
+ "\n",
+ "\n",
+ "...\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 20 / 25 (80.0): 100%|██████████| 25/25 [00:00<00:00, 1685.22it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "FULL TRACE\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 0 / 1 (0.0): 100%|██████████| 1/1 [00:00<00:00, 521.68it/s]\n",
+ "[I 2024-06-20 22:06:48,684] Trial 12 finished with value: 80.0 and parameters: {'0_predictor_instruction': 9, '0_predictor_demos': 7}. Best is trial 9 with value: 80.0.\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "\n",
+ "\n",
+ "Given the context and question, generate a logical answer based on the reasoning process.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: There is not a single person not walking in the city.\n",
+ "Question: Can we logically conclude for sure that there is not a single matriarch not walking in the city?\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: The men were outside speaking loudly so as to wake the bridesmaid indoors\n",
+ "\n",
+ "Question: Can we logically conclude for sure that the men were outside speaking loudly so as to wake the woman indoors?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the men were outside speaking loudly, and we also know that the bridesmaid is a woman. Therefore, we can logically conclude that the men were outside speaking loudly so as to wake the woman indoors.\n",
+ "\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. Therefore, we can logically conclude that it is not true that there is not a single slaver walking in the city.\n",
+ "\n",
+ "Answer:\u001b[32m Yes\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "Given the context and question, generate a logical answer based on the reasoning process.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: There is not a single person not walking in the city.\n",
+ "Question: Can we logically conclude for sure that there is not a single matriarch not walking in the city?\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: The men were outside speaking loudly so as to wake the bridesmaid indoors\n",
+ "\n",
+ "Question: Can we logically conclude for sure that the men were outside speaking loudly so as to wake the woman indoors?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the men were outside speaking loudly, and we also know that the bridesmaid is a woman. Therefore, we can logically conclude that the men were outside speaking loudly so as to wake the woman indoors.\n",
+ "\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. Therefore, we can logically conclude that it is not true that there is not a single slaver walking in the city.\n",
+ "\n",
+ "Answer:\u001b[32m Yes\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "...\n",
+ "Score 80.0\n",
+ "CANDIDATE PROGRAM:\n",
+ "Predictor 0\n",
+ "i: Given the context and question, generate a logical answer based on the reasoning process.\n",
+ "p: Answer:\n",
+ "\n",
+ "\n",
+ "...\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 15 / 25 (60.0): 100%|██████████| 25/25 [00:00<00:00, 3042.88it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "FULL TRACE\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 0 / 1 (0.0): 100%|██████████| 1/1 [00:00<00:00, 1492.63it/s]\n",
+ "[I 2024-06-20 22:06:48,780] Trial 13 finished with value: 60.0 and parameters: {'0_predictor_instruction': 9, '0_predictor_demos': 7}. Best is trial 9 with value: 80.0.\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "\n",
+ "\n",
+ "Given the context and question, generate a logical answer based on the reasoning process.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: There is not a single person not walking in the city.\n",
+ "Question: Can we logically conclude for sure that there is not a single matriarch not walking in the city?\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: The men were outside speaking loudly so as to wake the bridesmaid indoors\n",
+ "\n",
+ "Question: Can we logically conclude for sure that the men were outside speaking loudly so as to wake the woman indoors?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the men were outside speaking loudly, and we also know that the bridesmaid is a woman. Therefore, we can logically conclude that the men were outside speaking loudly so as to wake the woman indoors.\n",
+ "\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. Therefore, we can logically conclude that it is not true that there is not a single slaver walking in the city.\n",
+ "\n",
+ "Answer:\u001b[32m Yes\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "Given the context and question, generate a logical answer based on the reasoning process.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: There is not a single person not walking in the city.\n",
+ "Question: Can we logically conclude for sure that there is not a single matriarch not walking in the city?\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: The men were outside speaking loudly so as to wake the bridesmaid indoors\n",
+ "\n",
+ "Question: Can we logically conclude for sure that the men were outside speaking loudly so as to wake the woman indoors?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the men were outside speaking loudly, and we also know that the bridesmaid is a woman. Therefore, we can logically conclude that the men were outside speaking loudly so as to wake the woman indoors.\n",
+ "\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. Therefore, we can logically conclude that it is not true that there is not a single slaver walking in the city.\n",
+ "\n",
+ "Answer:\u001b[32m Yes\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "...\n",
+ "Score 60.0\n",
+ "CANDIDATE PROGRAM:\n",
+ "Predictor 0\n",
+ "i: Please use the provided context and question to generate a logical inference for the answer using the language model. The answer should be \"Yes\" or \"No\" based on the logical reasoning and inference from the given context and question.\n",
+ "p: Answer:\n",
+ "\n",
+ "\n",
+ "...\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 14 / 25 (56.0): 100%|██████████| 25/25 [00:00<00:00, 1288.13it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "FULL TRACE\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 0 / 1 (0.0): 100%|██████████| 1/1 [00:00<00:00, 739.08it/s]\n",
+ "[I 2024-06-20 22:06:48,824] Trial 14 finished with value: 56.0 and parameters: {'0_predictor_instruction': 4, '0_predictor_demos': 7}. Best is trial 9 with value: 80.0.\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "\n",
+ "\n",
+ "Please use the provided context and question to generate a logical inference for the answer using the language model. The answer should be \"Yes\" or \"No\" based on the logical reasoning and inference from the given context and question.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: There is not a single person not walking in the city.\n",
+ "Question: Can we logically conclude for sure that there is not a single matriarch not walking in the city?\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: The men were outside speaking loudly so as to wake the bridesmaid indoors\n",
+ "\n",
+ "Question: Can we logically conclude for sure that the men were outside speaking loudly so as to wake the woman indoors?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the men were outside speaking loudly, and we also know that the bridesmaid is a woman. Therefore, we can logically conclude that the men were outside speaking loudly so as to wake the woman indoors.\n",
+ "\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. Therefore, we can logically conclude that it is not true that there is not a single slaver walking in the city.\n",
+ "\n",
+ "Answer:\u001b[32m Yes\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "Please use the provided context and question to generate a logical inference for the answer using the language model. The answer should be \"Yes\" or \"No\" based on the logical reasoning and inference from the given context and question.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: There is not a single person not walking in the city.\n",
+ "Question: Can we logically conclude for sure that there is not a single matriarch not walking in the city?\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: The men were outside speaking loudly so as to wake the bridesmaid indoors\n",
+ "\n",
+ "Question: Can we logically conclude for sure that the men were outside speaking loudly so as to wake the woman indoors?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the men were outside speaking loudly, and we also know that the bridesmaid is a woman. Therefore, we can logically conclude that the men were outside speaking loudly so as to wake the woman indoors.\n",
+ "\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. Therefore, we can logically conclude that it is not true that there is not a single slaver walking in the city.\n",
+ "\n",
+ "Answer:\u001b[32m Yes\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "...\n",
+ "Score 56.0\n",
+ "CANDIDATE PROGRAM:\n",
+ "Predictor 0\n",
+ "i: Given the context and a question, use logical reasoning and inference to determine and provide a yes or no answer based on the information provided.\n",
+ "p: Answer:\n",
+ "\n",
+ "\n",
+ "...\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 14 / 25 (56.0): 100%|██████████| 25/25 [00:00<00:00, 1826.18it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "FULL TRACE\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 0 / 1 (0.0): 100%|██████████| 1/1 [00:00<00:00, 388.18it/s]\n",
+ "[I 2024-06-20 22:06:48,866] Trial 15 finished with value: 56.0 and parameters: {'0_predictor_instruction': 5, '0_predictor_demos': 0}. Best is trial 9 with value: 80.0.\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "\n",
+ "\n",
+ "Given the context and a question, use logical reasoning and inference to determine and provide a yes or no answer based on the information provided.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This means that there must be at least one person walking in the city. Now, we are asked if we can logically conclude for sure that it is not true that there is not a single slaver walking in the city. Since we know that there is at least one person walking in the city, we can conclude that it is not true that there is not a single slaver walking in the city. This is because a slaver is a type of person, and we already know that there is at least one person walking in the city.\n",
+ "\n",
+ "Answer:\u001b[32m Yes\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "Given the context and a question, use logical reasoning and inference to determine and provide a yes or no answer based on the information provided.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This means that there must be at least one person walking in the city. Now, we are asked if we can logically conclude for sure that it is not true that there is not a single slaver walking in the city. Since we know that there is at least one person walking in the city, we can conclude that it is not true that there is not a single slaver walking in the city. This is because a slaver is a type of person, and we already know that there is at least one person walking in the city.\n",
+ "\n",
+ "Answer:\u001b[32m Yes\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "...\n",
+ "Score 56.0\n",
+ "CANDIDATE PROGRAM:\n",
+ "Predictor 0\n",
+ "i: Please generate a logical answer (either \"Yes\" or \"No\") based on the given context and question using the `ScoNeSignature` function to perform logical reasoning and inference. Take the context and question as input to generate the answer.\n",
+ "p: Answer:\n",
+ "\n",
+ "\n",
+ "...\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 17 / 25 (68.0): 100%|██████████| 25/25 [00:00<00:00, 1738.99it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "FULL TRACE\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 0 / 1 (0.0): 100%|██████████| 1/1 [00:00<00:00, 1104.93it/s]\n",
+ "[I 2024-06-20 22:06:48,912] Trial 16 finished with value: 68.0 and parameters: {'0_predictor_instruction': 7, '0_predictor_demos': 7}. Best is trial 9 with value: 80.0.\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "\n",
+ "\n",
+ "Please generate a logical answer (either \"Yes\" or \"No\") based on the given context and question using the `ScoNeSignature` function to perform logical reasoning and inference. Take the context and question as input to generate the answer.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: There is not a single person not walking in the city.\n",
+ "Question: Can we logically conclude for sure that there is not a single matriarch not walking in the city?\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: The men were outside speaking loudly so as to wake the bridesmaid indoors\n",
+ "\n",
+ "Question: Can we logically conclude for sure that the men were outside speaking loudly so as to wake the woman indoors?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the men were outside speaking loudly, and we also know that the bridesmaid is a woman. Therefore, we can logically conclude that the men were outside speaking loudly so as to wake the woman indoors.\n",
+ "\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. Therefore, we can logically conclude that it is not true that there is not a single slaver walking in the city.\n",
+ "\n",
+ "Answer:\u001b[32m Yes\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "Please generate a logical answer (either \"Yes\" or \"No\") based on the given context and question using the `ScoNeSignature` function to perform logical reasoning and inference. Take the context and question as input to generate the answer.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: There is not a single person not walking in the city.\n",
+ "Question: Can we logically conclude for sure that there is not a single matriarch not walking in the city?\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: The men were outside speaking loudly so as to wake the bridesmaid indoors\n",
+ "\n",
+ "Question: Can we logically conclude for sure that the men were outside speaking loudly so as to wake the woman indoors?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the men were outside speaking loudly, and we also know that the bridesmaid is a woman. Therefore, we can logically conclude that the men were outside speaking loudly so as to wake the woman indoors.\n",
+ "\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. Therefore, we can logically conclude that it is not true that there is not a single slaver walking in the city.\n",
+ "\n",
+ "Answer:\u001b[32m Yes\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "...\n",
+ "Score 68.0\n",
+ "CANDIDATE PROGRAM:\n",
+ "Predictor 0\n",
+ "i: Given a context and a question, use the provided context and question to determine if the logical conclusion is \"Yes\" or \"No\" and generate the corresponding answer.\n",
+ "p: Answer:\n",
+ "\n",
+ "\n",
+ "...\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 17 / 25 (68.0): 100%|██████████| 25/25 [00:00<00:00, 1189.51it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "FULL TRACE\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 0 / 1 (0.0): 100%|██████████| 1/1 [00:00<00:00, 444.03it/s]\n",
+ "[I 2024-06-20 22:06:48,962] Trial 17 finished with value: 68.0 and parameters: {'0_predictor_instruction': 8, '0_predictor_demos': 7}. Best is trial 9 with value: 80.0.\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "\n",
+ "\n",
+ "Given a context and a question, use the provided context and question to determine if the logical conclusion is \"Yes\" or \"No\" and generate the corresponding answer.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: There is not a single person not walking in the city.\n",
+ "Question: Can we logically conclude for sure that there is not a single matriarch not walking in the city?\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: The men were outside speaking loudly so as to wake the bridesmaid indoors\n",
+ "\n",
+ "Question: Can we logically conclude for sure that the men were outside speaking loudly so as to wake the woman indoors?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the men were outside speaking loudly, and we also know that the bridesmaid is a woman. Therefore, we can logically conclude that the men were outside speaking loudly so as to wake the woman indoors.\n",
+ "\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. Therefore, we can logically conclude that it is not true that there is not a single slaver walking in the city.\n",
+ "\n",
+ "Answer:\u001b[32m Yes\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "Given a context and a question, use the provided context and question to determine if the logical conclusion is \"Yes\" or \"No\" and generate the corresponding answer.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: There is not a single person not walking in the city.\n",
+ "Question: Can we logically conclude for sure that there is not a single matriarch not walking in the city?\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: The men were outside speaking loudly so as to wake the bridesmaid indoors\n",
+ "\n",
+ "Question: Can we logically conclude for sure that the men were outside speaking loudly so as to wake the woman indoors?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the men were outside speaking loudly, and we also know that the bridesmaid is a woman. Therefore, we can logically conclude that the men were outside speaking loudly so as to wake the woman indoors.\n",
+ "\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. Therefore, we can logically conclude that it is not true that there is not a single slaver walking in the city.\n",
+ "\n",
+ "Answer:\u001b[32m Yes\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "...\n",
+ "Score 68.0\n",
+ "CANDIDATE PROGRAM:\n",
+ "Predictor 0\n",
+ "i: Given the context and question, generate a logical answer based on the reasoning process.\n",
+ "p: Answer:\n",
+ "\n",
+ "\n",
+ "...\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 18 / 25 (72.0): 100%|██████████| 25/25 [00:00<00:00, 385.39it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "FULL TRACE\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 1 / 1 (100.0): 100%|██████████| 1/1 [00:00<00:00, 205.17it/s]\n",
+ "[I 2024-06-20 22:06:49,060] Trial 18 finished with value: 72.0 and parameters: {'0_predictor_instruction': 9, '0_predictor_demos': 1}. Best is trial 9 with value: 80.0.\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "\n",
+ "\n",
+ "Given the context and question, generate a logical answer based on the reasoning process.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: the boy, not girl, will play an horn, but not for another week\n",
+ "Question: Can we logically conclude for sure that the boy, not girl, will play an instrument, but not for another week?\n",
+ "Answer: Yes\n",
+ "\n",
+ "Context: The person did not like raspberries.\n",
+ "Question: Can we logically conclude for sure that the person did not like fruit?\n",
+ "Answer: No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the statement \"there is not a single person walking in the city\" is false. This means that there is at least one person walking in the city. However, we cannot logically conclude that there is not a single slaver walking in the city. We only know that there is at least one person, but we don't have any information about the presence of slavers.\n",
+ "\n",
+ "Answer:\u001b[32m No\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "Given the context and question, generate a logical answer based on the reasoning process.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: the boy, not girl, will play an horn, but not for another week\n",
+ "Question: Can we logically conclude for sure that the boy, not girl, will play an instrument, but not for another week?\n",
+ "Answer: Yes\n",
+ "\n",
+ "Context: The person did not like raspberries.\n",
+ "Question: Can we logically conclude for sure that the person did not like fruit?\n",
+ "Answer: No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the statement \"there is not a single person walking in the city\" is false. This means that there is at least one person walking in the city. However, we cannot logically conclude that there is not a single slaver walking in the city. We only know that there is at least one person, but we don't have any information about the presence of slavers.\n",
+ "\n",
+ "Answer:\u001b[32m No\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "...\n",
+ "Score 72.0\n",
+ "CANDIDATE PROGRAM:\n",
+ "Predictor 0\n",
+ "i: Given the context and question, generate a logical answer based on the reasoning process.\n",
+ "p: Answer:\n",
+ "\n",
+ "\n",
+ "...\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 17 / 25 (68.0): 100%|██████████| 25/25 [00:00<00:00, 2591.19it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "FULL TRACE\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 1 / 1 (100.0): 100%|██████████| 1/1 [00:00<00:00, 648.57it/s]\n",
+ "[I 2024-06-20 22:06:49,106] Trial 19 finished with value: 68.0 and parameters: {'0_predictor_instruction': 9, '0_predictor_demos': 8}. Best is trial 9 with value: 80.0.\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "\n",
+ "\n",
+ "Given the context and question, generate a logical answer based on the reasoning process.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: The man is not listening to techno.\n",
+ "Question: Can we logically conclude for sure that the man is not listening to music?\n",
+ "Answer: No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: There is a athlete walking in the city when it's not dark.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that there is a person walking in the city when it's not dark?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the athlete is walking in the city, and we know that the athlete is a person. Therefore, we can logically conclude that there is a person walking in the city.\n",
+ "\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. However, we cannot logically conclude that it is not true that there is not a single slaver walking in the city, because we have no information about the presence of slavers in the city.\n",
+ "\n",
+ "Answer:\u001b[32m No\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "Given the context and question, generate a logical answer based on the reasoning process.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: The man is not listening to techno.\n",
+ "Question: Can we logically conclude for sure that the man is not listening to music?\n",
+ "Answer: No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: There is a athlete walking in the city when it's not dark.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that there is a person walking in the city when it's not dark?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the athlete is walking in the city, and we know that the athlete is a person. Therefore, we can logically conclude that there is a person walking in the city.\n",
+ "\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. However, we cannot logically conclude that it is not true that there is not a single slaver walking in the city, because we have no information about the presence of slavers in the city.\n",
+ "\n",
+ "Answer:\u001b[32m No\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "...\n",
+ "Score 68.0\n",
+ "CANDIDATE PROGRAM:\n",
+ "Predictor 0\n",
+ "i: Determine if it can be logically concluded that a kayak is nearby based on the given context and question, and generate a yes or no answer using the language model.\n",
+ "p: Answer:\n",
+ "\n",
+ "\n",
+ "...\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 18 / 25 (72.0): 100%|██████████| 25/25 [00:00<00:00, 2465.96it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "FULL TRACE\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 0 / 1 (0.0): 100%|██████████| 1/1 [00:00<00:00, 361.11it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "\n",
+ "\n",
+ "Determine if it can be logically concluded that a kayak is nearby based on the given context and question, and generate a yes or no answer using the language model.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: There is not a single person not walking in the city.\n",
+ "Question: Can we logically conclude for sure that there is not a single matriarch not walking in the city?\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: The men were outside speaking loudly so as to wake the bridesmaid indoors\n",
+ "\n",
+ "Question: Can we logically conclude for sure that the men were outside speaking loudly so as to wake the woman indoors?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the men were outside speaking loudly, and we also know that the bridesmaid is a woman. Therefore, we can logically conclude that the men were outside speaking loudly so as to wake the woman indoors.\n",
+ "\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. Therefore, we can logically conclude that it is not true that there is not a single slaver walking in the city.\n",
+ "\n",
+ "Answer:\u001b[32m Yes\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "Determine if it can be logically concluded that a kayak is nearby based on the given context and question, and generate a yes or no answer using the language model.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: There is not a single person not walking in the city.\n",
+ "Question: Can we logically conclude for sure that there is not a single matriarch not walking in the city?\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: The men were outside speaking loudly so as to wake the bridesmaid indoors\n",
+ "\n",
+ "Question: Can we logically conclude for sure that the men were outside speaking loudly so as to wake the woman indoors?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the men were outside speaking loudly, and we also know that the bridesmaid is a woman. Therefore, we can logically conclude that the men were outside speaking loudly so as to wake the woman indoors.\n",
+ "\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. Therefore, we can logically conclude that it is not true that there is not a single slaver walking in the city.\n",
+ "\n",
+ "Answer:\u001b[32m Yes\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "...\n",
+ "Score 72.0\n",
+ "Best Combination: 0,{0: [[], [Example({'context': 'the boy, not girl, will play an horn, but not for another week', 'question': 'Can we logically conclude for sure that the boy, not girl, will play an instrument, but not for another week?', 'answer': 'Yes', 'category': 'two_not_scoped'}) (input_keys={'context', 'question'}), Example({'context': 'The person did not like raspberries.', 'question': 'Can we logically conclude for sure that the person did not like fruit?', 'answer': 'No', 'category': 'one_scoped'}) (input_keys={'context', 'question'})], [Example({'augmented': True, 'context': 'It is not true that there is not a single person walking in the city.', 'question': 'Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?', 'rationale': 'produce the answer. We know that the statement \"there is not a single person walking in the city\" is false. However, we cannot logically conclude that the statement \"there is not a single slaver walking in the city\" is also false. There could be a single person walking in the city who is not a slaver.', 'answer': 'No'}) (input_keys=None), Example({'context': 'It is not true that there is not a single person walking in the city.', 'question': 'Can we logically conclude for sure that it is not true that there is not a single celebrity walking in the city?', 'answer': 'No', 'category': 'two_scoped'}) (input_keys={'context', 'question'})], [Example({'augmented': True, 'context': 'A man is not standing on top of a ladder that is leaned against a not so tall tree.', 'question': 'Can we logically conclude for sure that a man is not standing on top of a ladder that is leaned against a not so tall pine?', 'rationale': \"produce the answer. We know that the man is not standing on top of a ladder that is leaned against a not so tall tree. We can't conclude that the ladder is leaned against a pine tree, so we can't conclude that the man is not standing on top of a ladder that is leaned against a not so tall pine.\", 'answer': 'No'}) (input_keys=None), Example({'context': 'The people were outside not even trying to keep their voices down, and they woke up a mistress indoors.', 'question': 'Can we logically conclude for sure that the people were outside not even trying to keep their voices down, and they woke up a woman indoors?', 'answer': 'Yes', 'category': 'one_not_scoped'}) (input_keys={'context', 'question'})], [Example({'augmented': True, 'context': 'The cowboy did not tell other people that he fall off a bronco at the competition.', 'question': 'Can we logically conclude for sure that the cowboy did not tell other people that he fall off a horse at the competition?', 'rationale': 'produce the answer. We know that the cowboy did not tell other people that he fall off a bronco at the competition. Bronco is a type of horse, so it is a type of horse that the cowboy fell off. Therefore, we can logically conclude that the cowboy did not tell other people that he fall off a horse at the competition.', 'answer': 'Yes'}) (input_keys=None), Example({'context': 'The diver has not seen any fish on his dive that did not go too deep.', 'question': 'Can we logically conclude for sure that the diver has not seen any tuna on his dive that did not go too deep?', 'answer': 'Yes', 'category': 'one_scoped_one_not_scoped'}) (input_keys={'context', 'question'})], [Example({'augmented': True, 'context': 'It is not true that there is not a single person walking in the city.', 'question': 'Can we logically conclude for sure that it is not true that there is not a single celebrity walking in the city?', 'rationale': 'produce the answer. We know that it is not true that there is not a single person walking in the city. This does not necessarily mean that there is a celebrity walking in the city. There could be a non-celebrity person walking in the city.', 'answer': 'No'}) (input_keys=None), Example({'context': 'The cowboy did not tell other people that he fall off a bronco at the competition.', 'question': 'Can we logically conclude for sure that the cowboy did not tell other people that he fall off a horse at the competition?', 'answer': 'Yes', 'category': 'one_not_scoped'}) (input_keys={'context', 'question'})], [Example({'augmented': True, 'context': 'There is a boat nearby', 'question': 'Can we logically conclude for sure that there is a kayak nearby?', 'rationale': \"produce the answer. We know that there is a boat nearby, but we don't know what type of boat it is. It could be a kayak, but it could also be a sailboat, a speedboat, or even a rowboat. We can't logically conclude that it is a kayak for sure.\", 'answer': 'No'}) (input_keys=None), Example({'context': 'the boy plays an instrument', 'question': 'Can we logically conclude for sure that the boy plays an flute?', 'answer': 'No', 'category': 'no_negation'}) (input_keys={'context', 'question'})], [Example({'augmented': True, 'context': 'The men were outside speaking loudly so as to wake the bridesmaid indoors', 'question': 'Can we logically conclude for sure that the men were outside speaking loudly so as to wake the woman indoors?', 'rationale': 'produce the answer. We know that the men were outside speaking loudly, and we also know that the bridesmaid is a woman. Therefore, we can logically conclude that the men were outside speaking loudly so as to wake the woman indoors.', 'answer': 'Yes'}) (input_keys=None), Example({'context': 'There is not a single person not walking in the city.', 'question': 'Can we logically conclude for sure that there is not a single matriarch not walking in the city?', 'answer': 'Yes', 'category': 'one_scoped_one_not_scoped'}) (input_keys={'context', 'question'})], [Example({'augmented': True, 'context': \"There is a athlete walking in the city when it's not dark.\", 'question': \"Can we logically conclude for sure that there is a person walking in the city when it's not dark?\", 'rationale': 'produce the answer. We know that the athlete is walking in the city, and we know that the athlete is a person. Therefore, we can logically conclude that there is a person walking in the city.', 'answer': 'Yes'}) (input_keys=None), Example({'context': 'The man is not listening to techno.', 'question': 'Can we logically conclude for sure that the man is not listening to music?', 'answer': 'No', 'category': 'one_scoped'}) (input_keys={'context', 'question'})], [Example({'augmented': True, 'context': 'A man is not holding anything in his hands.', 'question': 'Can we logically conclude for sure that a man is not holding tongues in his hands?', 'rationale': \"produce the answer. We know that a man is not holding anything in his hands. This means that he is not holding any physical objects, including tongues. Tongues are not physical objects that can be held in one's hands. Therefore, we can logically conclude that a man is not holding tongues in his hands.\", 'answer': 'Yes'}) (input_keys=None), Example({'context': 'To believe that a man walks confidently not dropping produce is to believe a falsity.', 'question': 'Can we logically conclude for sure that to believe that a man walks confidently not dropping pears is to believe a falsity?', 'answer': 'No', 'category': 'two_scoped'}) (input_keys={'context', 'question'})]]} with Mean = 72.0\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 142 / 200 (71.0): 100%|██████████| 200/200 [00:00<00:00, 965.14it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "UPDATING BEST SCORE WITH 71.0\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 130 / 200 (65.0): 100%|██████████| 200/200 [00:00<00:00, 1004.12it/s]\n",
+ "[I 2024-06-20 22:06:49,669] Trial 20 finished with value: 72.0 and parameters: {'0_predictor_instruction': 6, '0_predictor_demos': 7}. Best is trial 9 with value: 80.0.\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "CANDIDATE PROGRAM:\n",
+ "Predictor 0\n",
+ "i: Given the context and a question, use logical reasoning and inference to determine and provide a yes or no answer based on the information provided.\n",
+ "p: Answer:\n",
+ "\n",
+ "\n",
+ "...\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 19 / 25 (76.0): 100%|██████████| 25/25 [00:00<00:00, 1511.22it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "FULL TRACE\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 0 / 1 (0.0): 100%|██████████| 1/1 [00:00<00:00, 500.69it/s]\n",
+ "[I 2024-06-20 22:06:49,733] Trial 21 finished with value: 76.0 and parameters: {'0_predictor_instruction': 5, '0_predictor_demos': 9}. Best is trial 9 with value: 80.0.\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "\n",
+ "\n",
+ "Given the context and a question, use logical reasoning and inference to determine and provide a yes or no answer based on the information provided.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: To believe that a man walks confidently not dropping produce is to believe a falsity.\n",
+ "Question: Can we logically conclude for sure that to believe that a man walks confidently not dropping pears is to believe a falsity?\n",
+ "Answer: No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: A man is not holding anything in his hands.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that a man is not holding tongues in his hands?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that a man is not holding anything in his hands. This means that he is not holding any physical objects, including tongues. Tongues are not physical objects that can be held in one's hands. Therefore, we can logically conclude that a man is not holding tongues in his hands.\n",
+ "\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This means that there must be at least one person walking in the city. Since the term \"person\" is a broader category that includes \"slaver\", we can logically conclude that it is not true that there is not a single slaver walking in the city.\n",
+ "\n",
+ "Answer:\u001b[32m Yes\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "Given the context and a question, use logical reasoning and inference to determine and provide a yes or no answer based on the information provided.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: To believe that a man walks confidently not dropping produce is to believe a falsity.\n",
+ "Question: Can we logically conclude for sure that to believe that a man walks confidently not dropping pears is to believe a falsity?\n",
+ "Answer: No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: A man is not holding anything in his hands.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that a man is not holding tongues in his hands?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that a man is not holding anything in his hands. This means that he is not holding any physical objects, including tongues. Tongues are not physical objects that can be held in one's hands. Therefore, we can logically conclude that a man is not holding tongues in his hands.\n",
+ "\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This means that there must be at least one person walking in the city. Since the term \"person\" is a broader category that includes \"slaver\", we can logically conclude that it is not true that there is not a single slaver walking in the city.\n",
+ "\n",
+ "Answer:\u001b[32m Yes\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "...\n",
+ "Score 76.0\n",
+ "CANDIDATE PROGRAM:\n",
+ "Predictor 0\n",
+ "i: Given the context and question, generate a logical answer based on the reasoning process.\n",
+ "p: Answer:\n",
+ "\n",
+ "\n",
+ "...\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 19 / 25 (76.0): 100%|██████████| 25/25 [00:00<00:00, 1637.73it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "FULL TRACE\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 0 / 1 (0.0): 100%|██████████| 1/1 [00:00<00:00, 673.03it/s]\n",
+ "[I 2024-06-20 22:06:49,824] Trial 22 finished with value: 76.0 and parameters: {'0_predictor_instruction': 9, '0_predictor_demos': 4}. Best is trial 9 with value: 80.0.\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "\n",
+ "\n",
+ "Given the context and question, generate a logical answer based on the reasoning process.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: The diver has not seen any fish on his dive that did not go too deep.\n",
+ "Question: Can we logically conclude for sure that the diver has not seen any tuna on his dive that did not go too deep?\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: The cowboy did not tell other people that he fall off a bronco at the competition.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that the cowboy did not tell other people that he fall off a horse at the competition?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the cowboy did not tell other people that he fall off a bronco at the competition. Bronco is a type of horse, so it is a type of horse that the cowboy fell off. Therefore, we can logically conclude that the cowboy did not tell other people that he fall off a horse at the competition.\n",
+ "\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This means that there is at least one person walking in the city. Therefore, we can logically conclude that it is not true that there is not a single slaver walking in the city, because there is at least one person walking in the city, and that person could be a slaver.\n",
+ "\n",
+ "Answer:\u001b[32m Yes\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "Given the context and question, generate a logical answer based on the reasoning process.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: The diver has not seen any fish on his dive that did not go too deep.\n",
+ "Question: Can we logically conclude for sure that the diver has not seen any tuna on his dive that did not go too deep?\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: The cowboy did not tell other people that he fall off a bronco at the competition.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that the cowboy did not tell other people that he fall off a horse at the competition?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the cowboy did not tell other people that he fall off a bronco at the competition. Bronco is a type of horse, so it is a type of horse that the cowboy fell off. Therefore, we can logically conclude that the cowboy did not tell other people that he fall off a horse at the competition.\n",
+ "\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This means that there is at least one person walking in the city. Therefore, we can logically conclude that it is not true that there is not a single slaver walking in the city, because there is at least one person walking in the city, and that person could be a slaver.\n",
+ "\n",
+ "Answer:\u001b[32m Yes\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "...\n",
+ "Score 76.0\n",
+ "CANDIDATE PROGRAM:\n",
+ "Predictor 0\n",
+ "i: context, question -> answer\n",
+ "p: Answer:\n",
+ "\n",
+ "\n",
+ "...\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 22 / 25 (88.0): 100%|██████████| 25/25 [00:00<00:00, 1825.07it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "FULL TRACE\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 1 / 1 (100.0): 100%|██████████| 1/1 [00:00<00:00, 344.95it/s]\n",
+ "[I 2024-06-20 22:06:49,870] Trial 23 finished with value: 88.0 and parameters: {'0_predictor_instruction': 0, '0_predictor_demos': 9}. Best is trial 23 with value: 88.0.\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "\n",
+ "\n",
+ "context, question -> answer\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: To believe that a man walks confidently not dropping produce is to believe a falsity.\n",
+ "Question: Can we logically conclude for sure that to believe that a man walks confidently not dropping pears is to believe a falsity?\n",
+ "Answer: No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: A man is not holding anything in his hands.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that a man is not holding tongues in his hands?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that a man is not holding anything in his hands. This means that he is not holding any physical objects, including tongues. Tongues are not physical objects that can be held in one's hands. Therefore, we can logically conclude that a man is not holding tongues in his hands.\n",
+ "\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This means that there is at least one person walking in the city. We cannot logically conclude that there is at least one slaver walking in the city, as the original statement does not mention anything about slavery. Therefore, we cannot logically conclude that it is not true that there is not a single slaver walking in the city.\n",
+ "\n",
+ "Answer:\u001b[32m No\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "context, question -> answer\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: To believe that a man walks confidently not dropping produce is to believe a falsity.\n",
+ "Question: Can we logically conclude for sure that to believe that a man walks confidently not dropping pears is to believe a falsity?\n",
+ "Answer: No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: A man is not holding anything in his hands.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that a man is not holding tongues in his hands?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that a man is not holding anything in his hands. This means that he is not holding any physical objects, including tongues. Tongues are not physical objects that can be held in one's hands. Therefore, we can logically conclude that a man is not holding tongues in his hands.\n",
+ "\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This means that there is at least one person walking in the city. We cannot logically conclude that there is at least one slaver walking in the city, as the original statement does not mention anything about slavery. Therefore, we cannot logically conclude that it is not true that there is not a single slaver walking in the city.\n",
+ "\n",
+ "Answer:\u001b[32m No\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "...\n",
+ "Score 88.0\n",
+ "CANDIDATE PROGRAM:\n",
+ "Predictor 0\n",
+ "i: context, question -> answer\n",
+ "p: Answer:\n",
+ "\n",
+ "\n",
+ "...\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 18 / 25 (72.0): 100%|██████████| 25/25 [00:00<00:00, 1318.08it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "FULL TRACE\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 1 / 1 (100.0): 100%|██████████| 1/1 [00:00<00:00, 705.64it/s]\n",
+ "[I 2024-06-20 22:06:49,915] Trial 24 finished with value: 72.0 and parameters: {'0_predictor_instruction': 0, '0_predictor_demos': 9}. Best is trial 23 with value: 88.0.\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "\n",
+ "\n",
+ "context, question -> answer\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: To believe that a man walks confidently not dropping produce is to believe a falsity.\n",
+ "Question: Can we logically conclude for sure that to believe that a man walks confidently not dropping pears is to believe a falsity?\n",
+ "Answer: No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: A man is not holding anything in his hands.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that a man is not holding tongues in his hands?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that a man is not holding anything in his hands. This means that he is not holding any physical objects, including tongues. Tongues are not physical objects that can be held in one's hands. Therefore, we can logically conclude that a man is not holding tongues in his hands.\n",
+ "\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This means that there is at least one person walking in the city. We cannot logically conclude that there is at least one slaver walking in the city, as the original statement does not mention anything about slavery. Therefore, we cannot logically conclude that it is not true that there is not a single slaver walking in the city.\n",
+ "\n",
+ "Answer:\u001b[32m No\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "context, question -> answer\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: To believe that a man walks confidently not dropping produce is to believe a falsity.\n",
+ "Question: Can we logically conclude for sure that to believe that a man walks confidently not dropping pears is to believe a falsity?\n",
+ "Answer: No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: A man is not holding anything in his hands.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that a man is not holding tongues in his hands?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that a man is not holding anything in his hands. This means that he is not holding any physical objects, including tongues. Tongues are not physical objects that can be held in one's hands. Therefore, we can logically conclude that a man is not holding tongues in his hands.\n",
+ "\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This means that there is at least one person walking in the city. We cannot logically conclude that there is at least one slaver walking in the city, as the original statement does not mention anything about slavery. Therefore, we cannot logically conclude that it is not true that there is not a single slaver walking in the city.\n",
+ "\n",
+ "Answer:\u001b[32m No\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "...\n",
+ "Score 72.0\n",
+ "CANDIDATE PROGRAM:\n",
+ "Predictor 0\n",
+ "i: Given a context and a question, use logical reasoning to determine if a logical conclusion can be made and generate a \"yes\" or \"no\" answer based on the provided information.\n",
+ "p: Answer:\n",
+ "\n",
+ "\n",
+ "...\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 18 / 25 (72.0): 100%|██████████| 25/25 [00:00<00:00, 1296.35it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "FULL TRACE\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 0 / 1 (0.0): 100%|██████████| 1/1 [00:00<00:00, 654.95it/s]\n",
+ "[I 2024-06-20 22:06:49,959] Trial 25 finished with value: 72.0 and parameters: {'0_predictor_instruction': 3, '0_predictor_demos': 7}. Best is trial 23 with value: 88.0.\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "\n",
+ "\n",
+ "Given a context and a question, use logical reasoning to determine if a logical conclusion can be made and generate a \"yes\" or \"no\" answer based on the provided information.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: There is not a single person not walking in the city.\n",
+ "Question: Can we logically conclude for sure that there is not a single matriarch not walking in the city?\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: The men were outside speaking loudly so as to wake the bridesmaid indoors\n",
+ "\n",
+ "Question: Can we logically conclude for sure that the men were outside speaking loudly so as to wake the woman indoors?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the men were outside speaking loudly, and we also know that the bridesmaid is a woman. Therefore, we can logically conclude that the men were outside speaking loudly so as to wake the woman indoors.\n",
+ "\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. Therefore, we can logically conclude that it is not true that there is not a single slaver walking in the city.\n",
+ "\n",
+ "Answer:\u001b[32m Yes\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "Given a context and a question, use logical reasoning to determine if a logical conclusion can be made and generate a \"yes\" or \"no\" answer based on the provided information.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: There is not a single person not walking in the city.\n",
+ "Question: Can we logically conclude for sure that there is not a single matriarch not walking in the city?\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: The men were outside speaking loudly so as to wake the bridesmaid indoors\n",
+ "\n",
+ "Question: Can we logically conclude for sure that the men were outside speaking loudly so as to wake the woman indoors?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the men were outside speaking loudly, and we also know that the bridesmaid is a woman. Therefore, we can logically conclude that the men were outside speaking loudly so as to wake the woman indoors.\n",
+ "\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. Therefore, we can logically conclude that it is not true that there is not a single slaver walking in the city.\n",
+ "\n",
+ "Answer:\u001b[32m Yes\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "...\n",
+ "Score 72.0\n",
+ "CANDIDATE PROGRAM:\n",
+ "Predictor 0\n",
+ "i: context, question -> answer\n",
+ "p: Answer:\n",
+ "\n",
+ "\n",
+ "...\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 17 / 25 (68.0): 100%|██████████| 25/25 [00:00<00:00, 1512.42it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "FULL TRACE\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 1 / 1 (100.0): 100%|██████████| 1/1 [00:00<00:00, 567.56it/s]\n",
+ "[I 2024-06-20 22:06:50,004] Trial 26 finished with value: 68.0 and parameters: {'0_predictor_instruction': 0, '0_predictor_demos': 3}. Best is trial 23 with value: 88.0.\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "\n",
+ "\n",
+ "context, question -> answer\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: The people were outside not even trying to keep their voices down, and they woke up a mistress indoors.\n",
+ "Question: Can we logically conclude for sure that the people were outside not even trying to keep their voices down, and they woke up a woman indoors?\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: A man is not standing on top of a ladder that is leaned against a not so tall tree.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that a man is not standing on top of a ladder that is leaned against a not so tall pine?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the man is not standing on top of a ladder that is leaned against a not so tall tree. We can't conclude that the ladder is leaned against a pine tree, so we can't conclude that the man is not standing on top of a ladder that is leaned against a not so tall pine.\n",
+ "\n",
+ "Answer: No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. We can't conclude that the person is a slaver, so we can't conclude that it is not true that there is not a single slaver walking in the city.\n",
+ "\n",
+ "Answer:\u001b[32m No\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "context, question -> answer\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: The people were outside not even trying to keep their voices down, and they woke up a mistress indoors.\n",
+ "Question: Can we logically conclude for sure that the people were outside not even trying to keep their voices down, and they woke up a woman indoors?\n",
+ "Answer: Yes\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: A man is not standing on top of a ladder that is leaned against a not so tall tree.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that a man is not standing on top of a ladder that is leaned against a not so tall pine?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the man is not standing on top of a ladder that is leaned against a not so tall tree. We can't conclude that the ladder is leaned against a pine tree, so we can't conclude that the man is not standing on top of a ladder that is leaned against a not so tall pine.\n",
+ "\n",
+ "Answer: No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. We can't conclude that the person is a slaver, so we can't conclude that it is not true that there is not a single slaver walking in the city.\n",
+ "\n",
+ "Answer:\u001b[32m No\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "...\n",
+ "Score 68.0\n",
+ "CANDIDATE PROGRAM:\n",
+ "Predictor 0\n",
+ "i: context, question -> answer\n",
+ "p: Answer:\n",
+ "\n",
+ "\n",
+ "...\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 9 / 25 (36.0): 100%|██████████| 25/25 [00:00<00:00, 2533.40it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "FULL TRACE\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 1 / 1 (100.0): 100%|██████████| 1/1 [00:00<00:00, 376.37it/s]\n",
+ "[I 2024-06-20 22:06:50,108] Trial 27 finished with value: 36.0 and parameters: {'0_predictor_instruction': 0, '0_predictor_demos': 6}. Best is trial 23 with value: 88.0.\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "\n",
+ "\n",
+ "context, question -> answer\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: the boy plays an instrument\n",
+ "Question: Can we logically conclude for sure that the boy plays an flute?\n",
+ "Answer: No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: There is a boat nearby\n",
+ "\n",
+ "Question: Can we logically conclude for sure that there is a kayak nearby?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that there is a boat nearby, but we don't know what type of boat it is. It could be a kayak, but it could also be a sailboat, a speedboat, or even a rowboat. We can't logically conclude that it is a kayak for sure.\n",
+ "\n",
+ "Answer: No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. However, we can't logically conclude that it is not true that there is not a single slaver walking in the city. We don't have any information about the presence of slavers in the city.\n",
+ "\n",
+ "Answer:\u001b[32m No\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "context, question -> answer\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: the boy plays an instrument\n",
+ "Question: Can we logically conclude for sure that the boy plays an flute?\n",
+ "Answer: No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: There is a boat nearby\n",
+ "\n",
+ "Question: Can we logically conclude for sure that there is a kayak nearby?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that there is a boat nearby, but we don't know what type of boat it is. It could be a kayak, but it could also be a sailboat, a speedboat, or even a rowboat. We can't logically conclude that it is a kayak for sure.\n",
+ "\n",
+ "Answer: No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. However, we can't logically conclude that it is not true that there is not a single slaver walking in the city. We don't have any information about the presence of slavers in the city.\n",
+ "\n",
+ "Answer:\u001b[32m No\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "...\n",
+ "Score 36.0\n",
+ "CANDIDATE PROGRAM:\n",
+ "Predictor 0\n",
+ "i: Provide a yes or no answer based on the given context and question. The context and question are used as input to generate the answer.\n",
+ "p: Answer:\n",
+ "\n",
+ "\n",
+ "...\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 16 / 25 (64.0): 100%|██████████| 25/25 [00:00<00:00, 1399.84it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "FULL TRACE\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 0 / 1 (0.0): 100%|██████████| 1/1 [00:00<00:00, 259.53it/s]\n",
+ "[I 2024-06-20 22:06:50,159] Trial 28 finished with value: 64.0 and parameters: {'0_predictor_instruction': 1, '0_predictor_demos': 0}. Best is trial 23 with value: 88.0.\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "\n",
+ "\n",
+ "Provide a yes or no answer based on the given context and question. The context and question are used as input to generate the answer.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This means that there must be at least one person walking in the city. Now, we are asked if we can logically conclude for sure that it is not true that there is not a single slaver walking in the city. Since we know that there is at least one person walking in the city, we can conclude that it is not true that there is not a single slaver walking in the city. This is because a slaver is a type of person, and we know that there is at least one person walking in the city.\n",
+ "\n",
+ "Answer:\u001b[32m Yes\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "Provide a yes or no answer based on the given context and question. The context and question are used as input to generate the answer.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This means that there must be at least one person walking in the city. Now, we are asked if we can logically conclude for sure that it is not true that there is not a single slaver walking in the city. Since we know that there is at least one person walking in the city, we can conclude that it is not true that there is not a single slaver walking in the city. This is because a slaver is a type of person, and we know that there is at least one person walking in the city.\n",
+ "\n",
+ "Answer:\u001b[32m Yes\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "...\n",
+ "Score 64.0\n",
+ "CANDIDATE PROGRAM:\n",
+ "Predictor 0\n",
+ "i: context, question -> answer\n",
+ "p: Answer:\n",
+ "\n",
+ "\n",
+ "...\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 11 / 25 (44.0): 100%|██████████| 25/25 [00:00<00:00, 1300.74it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "FULL TRACE\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 1 / 1 (100.0): 100%|██████████| 1/1 [00:00<00:00, 155.67it/s]\n",
+ "[I 2024-06-20 22:06:50,219] Trial 29 finished with value: 44.0 and parameters: {'0_predictor_instruction': 0, '0_predictor_demos': 2}. Best is trial 23 with value: 88.0.\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "\n",
+ "\n",
+ "context, question -> answer\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single celebrity walking in the city?\n",
+ "Answer: No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the statement \"there is not a single person walking in the city\" is false. However, we cannot logically conclude that the statement \"there is not a single slaver walking in the city\" is also false. There could be a single person walking in the city who is not a slaver.\n",
+ "\n",
+ "Answer: No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the statement \"there is not a single person walking in the city\" is false. However, we cannot logically conclude that the statement \"there is not a single slaver walking in the city\" is also false. There could be a single person walking in the city who is not a slaver.\n",
+ "\n",
+ "Answer:\u001b[32m No\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "context, question -> answer\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single celebrity walking in the city?\n",
+ "Answer: No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context: ${context}\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
+ "\n",
+ "Answer: Yes or No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the statement \"there is not a single person walking in the city\" is false. However, we cannot logically conclude that the statement \"there is not a single slaver walking in the city\" is also false. There could be a single person walking in the city who is not a slaver.\n",
+ "\n",
+ "Answer: No\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context: It is not true that there is not a single person walking in the city.\n",
+ "\n",
+ "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the statement \"there is not a single person walking in the city\" is false. However, we cannot logically conclude that the statement \"there is not a single slaver walking in the city\" is also false. There could be a single person walking in the city who is not a slaver.\n",
+ "\n",
+ "Answer:\u001b[32m No\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "...\n",
+ "Score 44.0\n"
+ ]
+ }
+ ],
"source": [
"import cloudpickle as pickle\n",
"from dspy.teleprompt import MIPROv2\n",
@@ -527,9 +5123,9 @@
"name": "stderr",
"output_type": "stream",
"text": [
- "Average Metric: 142 / 200 (71.0): 100%|██████████| 200/200 [00:00<00:00, 687.64it/s]\n",
- "Average Metric: 130 / 200 (65.0): 100%|██████████| 200/200 [00:00<00:00, 684.70it/s]\n",
- "Average Metric: 141 / 200 (70.5): 100%|██████████| 200/200 [00:00<00:00, 733.36it/s]\n"
+ "Average Metric: 142 / 200 (71.0): 100%|██████████| 200/200 [00:00<00:00, 839.20it/s]\n",
+ "Average Metric: 130 / 200 (65.0): 100%|██████████| 200/200 [00:00<00:00, 720.08it/s]\n",
+ "Average Metric: 141 / 200 (70.5): 100%|██████████| 200/200 [00:00<00:00, 900.15it/s]\n"
]
}
],
@@ -554,7 +5150,7 @@
"id": "pBYTLTwWqZH4"
},
"source": [
- "Now, let's take a look at how this optimization looked over the course of each trial. We see that, in general, __performance increases overtime__, until it saturates after ~trial #13."
+ "Now, let's take a look at how this optimization looked over the course of each trial."
]
},
{
@@ -566,7 +5162,7 @@
"outputs": [
{
"data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAckAAAE8CAYAAACrYErbAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/H5lhTAAAACXBIWXMAAA9hAAAPYQGoP6dpAABAmUlEQVR4nO3deViU9fo/8PcwssqmgDDIqoGQoiamooIbimsYmClWLqXZwRJJSyoVPebWOSaa6alMPRquB21xzx3FDfckjvlFMQVMEgYk2eb5/eGPOQ4wygwzPDPM+3Vdc10+y3zmvp+Hmdtn+3wkgiAIICIiohrMxA6AiIjIULFIEhERqcEiSUREpAaLJBERkRoskkRERGqwSBIREanBIklERKQGiyQREZEaLJJERERqsEgSNZDevXujd+/eWr1XIpEgMTFRp/EQ0bOxSBLVkUQiqdPryJEjosX4xx9/YOrUqQgICIC1tTVatGiBLl264MMPP0RxcbFocREZqyZiB0BkLDZs2KAy/e9//xsHDhyoMT8wMLDW9+/fv19vsQHAn3/+ic6dO0Mul2PChAkICAhAfn4+Ll++jFWrVuGdd96Bra2tXmMgamxYJInq6LXXXlOZPnXqFA4cOFBjfnUlJSWwsbGBhYWFPsPDmjVrkJ2djRMnTqB79+4qy+Ryud4//0kPHz5E06ZNG+zziPSFp1uJdKh3795o164d0tPTERYWBhsbG3z00UfKZU9ekywrK8Ps2bMRHBwMBwcHNG3aFKGhoTh8+LBWn33jxg1IpVJ069atxjJ7e3tYWVmpzDt9+jQGDx6MZs2aoWnTpmjfvj2SkpJU1jl06BBCQ0PRtGlTODo6IjIyEhkZGSrrJCYmQiKR4Nq1a4iJiUGzZs3Qs2dP5fKNGzciODgY1tbWaN68OUaNGoXbt2+rtHH9+nVER0fDzc0NVlZW8PDwwKhRo1BYWKjVtiDSFR5JEulYfn4+Bg0ahFGjRuG1116Dq6trrevJ5XJ88803GD16NCZOnIiioiKsWbMGEREROHPmDDp27KjR53p7e6OyshIbNmzA2LFjn7rugQMHMHToUMhkMkydOhVubm7IyMjATz/9hKlTpwIAfv75ZwwaNAitWrVCYmIi/vrrL6xYsQI9evTA+fPn4ePjo9LmK6+8Aj8/PyxYsABVI/B9+umnmDVrFkaOHIm33noLf/zxB1asWIGwsDBcuHABjo6OKCsrQ0REBEpLS/Huu+/Czc0Nd+7cwU8//YSCggI4ODhotB2IdEogIq3ExsYK1b9CvXr1EgAIq1evrrF+r169hF69eimnKyoqhNLSUpV1Hjx4ILi6ugoTJkxQmQ9AmDNnzlPjyc3NFVxcXAQAQkBAgDB58mQhOTlZKCgoUFmvoqJC8PX1Fby9vYUHDx6oLFMoFMp/d+zYUWjRooWQn5+vnHfp0iXBzMxMeOONN5Tz5syZIwAQRo8erdLWzZs3BalUKnz66acq869cuSI0adJEOf/ChQsCAGHbtm1PzY9IDDzdSqRjlpaWGD9+/DPXk0qlyuuECoUCf/75JyoqKtC5c2ecP39e4891dXXFpUuXMHnyZDx48ACrV69GTEwMWrRogb///e/Ko7sLFy4gKysLcXFxcHR0VGlDIpEAAHJycnDx4kWMGzcOzZs3Vy5v3749+vfvj927d9f4/MmTJ6tMp6SkQKFQYOTIkbh//77y5ebmBj8/P+Vp5aojxX379qGkpETjvIn0iUWSSMdatmxZ55tk1q9fj/bt28PKygpOTk5wcXHBrl27tL4WJ5PJsGrVKuTk5CAzMxPLly+Hi4sLZs+ejTVr1gB4fO0SANq1a6e2nVu3bgEA2rRpU2NZYGAg7t+/j4cPH6rM9/X1VZm+fv06BEGAn58fXFxcVF4ZGRm4d++e8n3x8fH45ptv4OzsjIiICKxcuZLXI8kg8JokkY5ZW1vXab2NGzdi3LhxGD58OGbMmIEWLVpAKpVi4cKFykKmLYlEAn9/f/j7+2PIkCHw8/PDd999h7feeqte7T5N9bwVCgUkEgn27NkDqVRaY/0nH0f55z//iXHjxuH777/H/v378d5772HhwoU4deoUPDw89BYz0bOwSBKJZPv27WjVqhVSUlKUpzkBYM6cOTr9nFatWqFZs2bIyckBALRu3RoAcPXqVYSHh9f6Hm9vbwBAZmZmjWW//vornJ2dn/mIR+vWrSEIAnx9feHv7//MOIOCghAUFIRPPvkEJ0+eRI8ePbB69WrMnz//me8l0heebiUSSdXRVdW1QuDxYxlpaWlatXf69Okap0AB4MyZM8jPz1eeOu3UqRN8fX2xbNkyFBQUqKxbFYtMJkPHjh2xfv16lXWuXr2K/fv3Y/Dgwc+MJyoqClKpFHPnzlXJsepz8vPzATy+y7eiokJleVBQEMzMzFBaWvrMzyHSJx5JEolk6NChSElJwcsvv4whQ4YgKysLq1evxvPPP69VF3IbNmzAd999h5dffhnBwcGwsLBARkYGvv32W1hZWSmf1zQzM8OqVaswbNgwdOzYEePHj4dMJsOvv/6KX375Bfv27QMAfPbZZxg0aBBCQkLw5ptvKh8BcXBwqFM/sq1bt8b8+fORkJCAmzdvYvjw4bCzs0NWVhZ27NiBSZMmYfr06Th06BCmTJmCV155Bf7+/qioqMCGDRsglUoRHR2t8XYg0iUWSSKRjBs3Drm5ufjXv/6Fffv24fnnn8fGjRuxbds2rfp/ffvtt2FjY4ODBw/i+++/h1wuh4uLCwYMGICEhAS88MILynUjIiJw+PBhzJ07F//85z+hUCjQunVrTJw4UblOeHg49u7dizlz5mD27NkwNzdHr169sHjx4ho36agzc+ZM+Pv74/PPP8fcuXMBAJ6enhgwYABeeuklAECHDh0QERGBH3/8EXfu3IGNjQ06dOiAPXv21NoxAlFDkgjVz4MQERERAF6TJCIiUotFkoiISA0WSSIiIjVYJImIiNRgkSQiIlJD1CJZVFSEuLg4eHt7w9raGt27d8fZs2eVywVBwOzZsyGTyWBtbY3w8HBcv35dxIiJiMiUiPqc5FtvvYWrV69iw4YNcHd3x8aNGxEeHo5r166hZcuWWLJkCZYvX47169fD19cXs2bNQkREBK5du1ZjAFl1FAoF7t69Czs7O5Wuv4iIyHQIgoCioiK4u7vDzEyD40ORhugSSkpKBKlUKvz0008q8zt16iR8/PHHgkKhENzc3ITPPvtMuaygoECwtLQUNm3aVOfPuX37tgCAL7744osvvoTbt29rVKtEO5KsqKhAZWVljSNCa2trpKamIisrC7m5uSodMDs4OKBr165IS0vDqFGjam23tLRUpb9H4f/3lZCVlQU7Ozut4y0vL8fhw4fRp08fmJuba92OsTC1fAHmbAo5m1q+AHOuyrmoqAi+vr4a1wFRe9zp3r07LCwskJycDFdXV2zatAljx47Fc889h7Vr16JHjx64e/cuZDKZ8j0jR46ERCLBli1bam0zMTFR2f3Vk5KTk2FjY6O3XIiIyHCVlJQgJiYGhYWFsLe3r/P7RL0muWHDBkyYMAEtW7aEVCpFp06dMHr0aKSnp2vdZkJCAuLj45XTcrlc2VekJhumuvLychw4cAD9+/c3if+NmVq+AHM2hZxNLV+AOVflLJfLtWpL1CLZunVrHD16FA8fPoRcLodMJsOrr76KVq1awc3NDQCQl5enciSZl5eHjh07qm3T0tISlpaWNeabm5vr5A9EV+0YC1PLF2DOpsDU8gWYs7a5G8Rzkk2bNoVMJsODBw+wb98+REZGwtfXF25ubjh48KByPblcjtOnTyMkJETEaImIdKNSUYkjN49g05VNOHLzCCoVlWKHRNWIeiS5b98+CIKANm3a4LfffsOMGTMQEBCA8ePHQyKRIC4uDvPnz4efn5/yERB3d3cMHz5cp3EIgqC8kUid8vJyNGnSBI8ePXrqeo2FqeUL6DdnqVSKJk2a8DEkUkrJSMHUvVPxu/x35TwPew8kDUxCVGCUiJHRk0QtkoWFhUhISMDvv/+O5s2bIzo6Gp9++qnysPiDDz7Aw4cPMWnSJBQUFKBnz57Yu3dvnZ+RrIuysjLk5OSgpKTkqesJggA3Nzfcvn3bJH7oTC1fQP8529jYQCaTwcLCQudtk3FJyUjBiK0jIED1vsk78jsYsXUEto/czkJpIEQtkiNHjsTIkSPVLpdIJJg3bx7mzZunl89XKBTIysqCVCqFu7s7LCws1P44KhQKFBcXw9bWVrMHUY2UqeUL6C9nQRBQVlaGP/74A1lZWfDz8zOZbUo1VSoqMXXv1BoFEgAECJBAgri9cYhsEwmpmVSECOlJohZJsZWVlUGhUMDT0/OZj4coFAqUlZXBysrKJH7gTC1fQL85W1tbw9zcHLdu3VJ+Bpmm49nHVU6xVidAwG35bRzPPo7ePr0bLjCqlWn8+j2DqRQBEhf/zggAcopydLoe6Re/tUREDUhmJ3v2ShqsR/rFIklE1IBCvULhYe8BCWq//0ECCTztPRHqFdrAkVFtWCTJ4Jw4cQJBQUEwNzev8+M+iYmJT+1kQhsSiQQ7d+6s8/r6iIEaH6mZFEkDkwCgRqGsml42cBlv2jEQLJJG6I8//sA777wDLy8vWFpaws3NDREREThx4oTYoelEfHw8OnbsiKysLKxbt67e7SUmJkIikTz1VZucnBwMGjSo3p9PVF1UYBS2j9yOlvYtVeZ72Hvw8Q8DY9J3t+pKpaISx7OPI6coBzI7GUK9QvX6v8Do6GiUlZVh/fr1aNWqFfLy8nDw4EHk5+fr7TMb0o0bNzB58mR4eHjopL3p06dj8uTJyukXX3wRkyZNwsSJE2tdv+ru06quEYn0ISowCpFtIhv0t4M0xyPJekrJSIFPkg/6rO+DmJQY9FnfBz5JPkjJSNHL5xUUFOD48eNYvHgx+vTpA29vb3Tp0gUJCQl46aWXAAA3b96ERCLBxYsXVd4nkUhw5MgR5bxffvkFQ4cOhb29Pezs7BAaGoobN24ol2/cuBFBQUGwtLSETCbDlClTVNp766234OLiAnt7e/Tt2xeXLl1SLr906RL69OkDOzs72NvbIzg4GOfOnQMA3Lp1C8OGDUOzZs3QtGlTtG3bFrt371bGnZ+fjwkTJkAikWDdunVYt24dHB0dVbbDzp076/zAv62tLdzc3JQvqVQKOzs75fSoUaMwZcoUTJs2Da1bt1YePVY/3frhhx/C398fNjY2aNWqFWbNmoXy8vI6xUBUG6mZFL19emN00Gj09unNAmmAWCTroarXjOrPPFX1mqGPQmlrawtbW1vs3LlTZdxMTd25cwdhYWGwtLTEoUOHkJ6ejgkTJqCiogIAsGrVKsyYMQMTJ07ElStX8MMPP+C5555Tvv+VV17BvXv3sGfPHqSnp6NTp07o168f/vzzTwDAmDFj4OHhgbNnzyI9PR0zZ85U9qQUGxuL0tJSHDt2DFeuXMHixYtha2sLT09P5OTkwN7eHsuWLUNOTg5effXVemytulu/fj0sLCywd+9efPnll7WuY2dnh3Xr1uHatWtISkrC119/jc8//7xB4iMicfB0q5bE6jWjSZMmWLduHSZOnIjVq1ejU6dO6NWrF0aNGoX27dvXuZ2VK1fCwcEBmzdvVhYvf39/5fIFCxYgNjYW7733nvL5vhdffBEAkJqaijNnzuDevXvKEVf+8Y9/YOfOndi+fTsmTZqE7OxsZV+8AODn56dsOzs7G9HR0QgKCgIAtGrVSrnMzc0NEokEDg4ODXq608/PD4sXL4ZcLlc7pNonn3yi/LePjw+mT5+OzZs344MPPmioMImogfFIUkua9Jqha9HR0bh79y5++OEHDBw4EEeOHEGnTp00usnl4sWLCA0NrXX4mHv37uHu3bvo1atXre+9dOkSiouL4eTkpDyytbW1RVZWlvJ0bXx8PN566y2Eh4dj0aJFKqdx33vvPcyfPx89evTAnDlzcPnyZc02gB4EBwc/c50tW7agR48ecHNzg62tLT755BNkZ2c3QHREJBYWSS2J3WuGlZUV+vfvj1mzZuHkyZMYN24c5syZA+B/PbsIwv+OcqtfO7O2tlbb9tOWAUBxcTFkMhkuXryo8srMzMSMGTMAPL6j9JdffsGQIUNw6NAhPP/889ixYwcA4K233sL//d//4fXXX8eVK1fQuXNnrFixQu3nmZmZqeRSWz711bRp06cuT0tLw5gxYzB48GD89NNPuHDhAj7++GOUlZXpNA4iMiwskloytF4znn/+eTx8+BAA4OLiAuDxIwxVnryJBwDat2+P48eP11ps7Ozs4OPjg6NHj9b6WZ06dUJubi6aNGmC5557TuXl7OysXM/f3x/Tpk3D/v37ERUVhbVr1yqXeXp6YvLkyUhJScH777+Pr7/+Wm1uLi4uKCoqUuZXWz76dvLkSXh7e+Pjjz9G586d4efnh1u3bjVoDETU8FgktSRWrxn5+fno27cvNm7ciMuXLyMrKwvbtm3DkiVLEBkZCeDxkWC3bt2waNEiZGRk4OjRoyrX0wBgypQpkMvlGDVqFM6dO4fr169jw4YNyMzMBADMnj0bK1euxIoVK3D9+nWcP39eebQXHh6OkJAQDB8+HPv378fNmzdx8uRJfPzxxzh37hz++usvTJkyBUeOHMGtW7dw4sQJnD17FoGBgQCAuLg47Nu3D1lZWTh//jwOHz6sXFabrl27wsbGBh999BFu3LiB5ORknTw/qQk/Pz9kZ2dj8+bNuHHjBpYvX648MiaixotFUkti9Zpha2uLrl274vPPP0dYWBjatWuHWbNmYeLEifjiiy+U63377beoqKhAcHCwcvDqJzk5OeHQoUMoLi5Gr169EBwcjK+//lp5jXLs2LFYsGABVq1ahbZt22Lo0KG4fv364/wkEuzevRthYWEYP348/P39MWrUKNy6dQuurq6QSqXIz8/HG2+8AX9/f4wcORKDBg3C3LlzAQCVlZWIjY1FYGAgBg4cCH9/f7V3lAJA8+bNsXHjRuzevRtBQUHYtGkTEhMTdbpdn+Wll17CtGnTMGXKFHTs2BEnT57ErFmzGjQGImp4EqH6xZ5GRi6Xw8HBAYWFhTXuWnz06BGysrLg6+v7zKGLFAqF8s7HJ0dzqG10cU97TywbuMyoe81Ql29jpu+cNfl7ayjl5eXYvXs3Bg8eXOtNXI2NqeULMOeqnJ9WC56Gj4DUE3vNICJqvFgkdaCq1wwiImpcTOM8GhERkRZYJImIiNRgkQRqPKhOpA/8OyMyPiZdJKvueiopKRE5EjIFVX9npnKHIVFjYNI37kilUjg6OuLevXsAABsbG7XDLykUCpSVleHRo0cm8UiEqeUL6C9nQRBQUlKCe/fuwdHREVIp73wmMhYmXSQBKEeaqCqU6giCgL/++gvW1tZ1HsfQmJlavoD+c3Z0dORAzkRGxuSLpEQigUwmQ4sWLZ7aaXZ5eTmOHTuGsLAwkzhdZmr5AvrN2dzcnEeQREbI5ItkFalU+tQfMalUioqKClhZWZlE0TC1fAHTzJmIns40LjYRERFpgUWSiIhIDRZJIiIiNXhNkhqlSkWlQXU6b2jxEFHdiHokWVlZiVmzZsHX1xfW1tZo3bo1/v73v6v0TCIIAmbPng2ZTAZra2uEh4crxzUkqk1KRgp8knzQZ30fxKTEoM/6PvBJ8kFKRgrjISKNiFokFy9ejFWrVuGLL75ARkYGFi9ejCVLlmDFihXKdZYsWYLly5dj9erVOH36NJo2bYqIiAg8evRIxMjJUKVkpGDE1hEq43sCwB35HYzYOqLBC5OhxUNEmhG1SJ48eRKRkZEYMmQIfHx8MGLECAwYMABnzpwB8PgoctmyZfjkk08QGRmJ9u3b49///jfu3r2LnTt3ihk6GaBKRSWm7p0KATX7SK2aF7c3DpWKSpOMh4g0J+o1ye7du+Orr77Cf//7X/j7++PSpUtITU3F0qVLAQBZWVnIzc1FeHi48j0ODg7o2rUr0tLSMGrUqBptlpaWorS0VDktl8sBPH5Q/GmdBTxL1Xvr04YxMcZ8U7NTkV+cD2sza7Xr3C++j2NZx9DTq2eNZbrOub7xNARj3M/1YWr5Asy5+jxNSQQRhyZQKBT46KOPsGTJEkilUlRWVuLTTz9FQkICgMdHmj169MDdu3chk8mU7xs5ciQkEgm2bNlSo83ExETMnTu3xvzk5GTY2NjoLxkiIjJYJSUliImJQWFhIezt7ev8PlGPJLdu3YrvvvsOycnJaNu2LS5evIi4uDi4u7tj7NixWrWZkJCA+Ph45bRcLoenpycGDBig0Yaprry8HAcOHED//v1NojcWY8w3NTsVQ5KHPHO9XTG71B5J6jLn+sbTEIxxP9eHqeULMOeqnKvOKmpK1CI5Y8YMzJw5U3naNCgoCLdu3cLChQsxduxYZWfQeXl5KkeSeXl56NixY61tWlpawtLSssZ8c3NznfyB6KodY2FM+Yb5hsHJ1gl35HdqvQ4ogQQe9h4I8w176uMXuspZV/E0BGPaz7pgavkCzFnb3EW9caekpKTGkERSqRQKhQIA4OvrCzc3Nxw8eFC5XC6X4/Tp0wgJCWnQWMnwSc2kSBqYBOBxAXpS1fSygcsarCAZWjxEpDlRi+SwYcPw6aefYteuXbh58yZ27NiBpUuX4uWXXwbweISOuLg4zJ8/Hz/88AOuXLmCN954A+7u7hg+fLiYoZOBigqMwvaR29HSvqXKfA97D2wfuR1RgVEmHQ8RaUbU060rVqzArFmz8Le//Q337t2Du7s73n77bcyePVu5zgcffICHDx9i0qRJKCgoQM+ePbF3715YWVmJGDkZsqjAKES2iTSYHm4MLR4iqjtRi6SdnR2WLVuGZcuWqV1HIpFg3rx5mDdvXsMFRkZPaiZFb5/eYoehZGjxEFHdsINzIiIiNVgkiYiI1GCRJCIiUoNFkoiISA2OJ0lEpIam44By3NDGh0WSiKgWKRkpmLp3qsowZx72HkgamFTr862ark/GgadbiYiq0XQcUI4b2nixSBIRPUHTcUA5bmjjxiJJRPSE49nHaxwRPkmAgNvy2ziefVyr9cm4sEgSET0hpyhHo/U0XZ+MC4skEdETZHayZ6/0xHqark/GhUWSiOgJoV6h8LD3qDG8WRUJJPC090SoV6hW65NxYZEkInqCpuOActzQxo1FkoioGk3HAeW4oY0XOxMgIqqFpuOActzQxolFkohIDU3HAeW4oY0PT7cSERGpwSJJRESkBoskERGRGiySREREavDGHaqhUlGJ1OxUAEBqdirCfMNEv0PP0MbpM7R4GgNu02fTZnxLQ/suGxsWSVJRNSZefnE+NrXfhCHJQ+Bk6yTqmHiGNk6focXTGHCbPpu241sa0nfZGPF0KykZ4ph4hhaTocXTGHCbPhvHtxQPiyQBMMwx8QwtJkOLpzHgNn02jm8pLhZJAmCYY+IZWkyGFk9j0JDbtPr1OWMpEhzfUlwskgTAMMfEM7SYDC2exqChtmlKRgp8knwwJHkIAGBI8hD4JPkYxWlHjm8pLhZJAmCYY+IZWkyGFk9j0BDb1Nivz3F8S3GxSBIAwxwTz9BiMrR4GgN9b9PGcH2O41uKi0WSABjmmHiGFpOhxdMY6HubNobrcxzfUlyiFkkfHx9IJJIar9jYWADAo0ePEBsbCycnJ9ja2iI6Ohp5eXlihtyoGeKYeIYWk6HF0xjoc5s2lutzHN9SPKJ2JnD27FlUVv7vNMfVq1fRv39/vPLKKwCAadOmYdeuXdi2bRscHBwwZcoUREVF4cSJE2KF3OhVjYl3LOsY5Ffl2BWzS/ReOhpinD5NeibhuIG6p69t2piuz2k7vqUhfZeNkahF0sXFRWV60aJFaN26NXr16oXCwkKsWbMGycnJ6Nu3LwBg7dq1CAwMxKlTp9CtWzcxQjYJUjMpenr1xO6ru9HTq6dBfKn0OU6fNj2TcNxA3dPHNq26PndHfqfW65ISSOBh72E01+e0Gd/S0L7LxsZguqUrKyvDxo0bER8fD4lEgvT0dJSXlyM8PFy5TkBAALy8vJCWlqa2SJaWlqK0tFQ5LZfLAQDl5eUoLy/XOr6q99anDWNiKvn+mPkjXt/xOgQIsDazBgBYm1njz+I/8fr214GXgWFthokcpf6Ywn5O6p+E13e8DgCwMrMC8HgfK6/P9V8GRaUCikqFaDHqkyns4+pqy1nb/CWCINT875UItm7dipiYGGRnZ8Pd3R3JyckYP368SsEDgC5duqBPnz5YvHhxre0kJiZi7ty5NeYnJyfDxsZGL7ETEZFhKykpQUxMDAoLC2Fvb1/n9xnMkeSaNWswaNAguLu716udhIQExMfHK6flcjk8PT0xYMAAjTZMdeXl5Thw4AD69+8Pc3PzesVoDEwh39TsVOXD5cDjo4tv232LCVcn4C/FX8r5u2J2oadXTzFC1DtT2M9VKhWVOHnrJIoyimAXaIfu3t1N4vSjKe3jKrXlXHVWUVMGUSRv3bqFn3/+GSkp/3uo183NDWVlZSgoKICjo6Nyfl5eHtzc3NS2ZWlpCUtLyxrzzc3NdfIHoqt2jEVjzje3JFelGFb5S/GXyvzcktxGuw2qNOb9XMUc5gj1DcXujN0I9Q1t9PlWZwr7uLonc9Y2d4MokmvXrkWLFi0wZMj//lcfHBwMc3NzHDx4ENHR0QCAzMxMZGdnIyQkRKxQDQLH3dONxnTnY0PQZixDfd+RzO+BbnGb1iR6kVQoFFi7di3Gjh2LJk3+F46DgwPefPNNxMfHo3nz5rC3t8e7776LkJAQk76zlePu6U5ju/NRn7Qdy1Bff6f8Huget2ntRO9x5+eff0Z2djYmTJhQY9nnn3+OoUOHIjo6GmFhYXBzc1M5JWtqjL0PSkPDnknqxtDGMuT3QPe4TdUTvUgOGDAAgiDA39+/xjIrKyusXLkSf/75Jx4+fIiUlJSnXo9szBpDH5SGiD2TPJ2hjWXI74HucZs+nehFkuqmMfRBaaiiAqNwc+pN7IrZBeDx3axZU7NMvkAChjeWIb8Husdt+nQskkaisfRBaaiqeiYBwJ5JnmBoYxnye6B73KZPxyJpJHgnJonB0MYy5PdA97hNn45F0khwjDgSg6GNZcjvge5xmz4di6SR4J2YJAZDG8uQ3wPd4zZ9OhZJI8I7MUkMhjaWIb8Husdtqp7onQmQZjiWIYlB27EM9fV3yu+B7nGb1o5F0ghxLEMSgzZjGerz75TfA93jNq2pXqdby8rKkJmZiYqKCl3FQ0REZDC0KpIlJSV48803YWNjg7Zt2yI7OxsA8O6772LRokU6DZCIiEgsWhXJhIQEXLp0CUeOHIGVlZVyfnh4OLZs2aKz4IiIiMSk1TXJnTt3YsuWLejWrRskkv/dMty2bVvcuHFDZ8ERERGJSasi+ccff6BFixY15j98+FClaBIRET3J2Mas1KpIdu7cGbt27cK7774LAMrC+M0335j8gMhERFQ7YxyzUqsiuWDBAgwaNAjXrl1DRUUFkpKScO3aNZw8eRJHjx7VdYxERGTkqsasrD4kV9WYlYbaaYFWN+707NkTly5dQkVFBYKCgrB//360aNECaWlpCA4O1nWMRERkxIx5zEqNjyTLy8vx9ttvY9asWfj666/1ERMRETUimoxZaWidGWh8JGlubo7//Oc/+oiFiIgaIWMes1Kr063Dhw/Hzp07dRwKERE1RsY8ZqVWN+74+flh3rx5OHHiBIKDg9G0aVOV5e+9955OgiMiIuNXNWblHfmdWq9LSiCBh72HQY5ZqVWRXLNmDRwdHZGeno709HSVZRKJhEWSiIiUqsasHLF1BCSQqBRKQx+zUqsimZWVpes4iIioEasas7K25ySXDVxmkI9/ADoYKksQHv+PgD3tEBHR0xjjmJVaD5X173//G0FBQbC2toa1tTXat2+PDRs26DI2IiJqZKrGrBwdNBq9fXobdIEEtDySXLp0KWbNmoUpU6agR48eAIDU1FRMnjwZ9+/fx7Rp03QaJBERkRi0KpIrVqzAqlWr8MYbbyjnvfTSS2jbti0SExNZJImIqFHQ6nRrTk4OunfvXmN+9+7dkZNjeA+DEhERaUOrIvncc89h69atNeZv2bIFfn5+9Q6KiIjIEGh1unXu3Ll49dVXcezYMeU1yRMnTuDgwYO1Fk8i0oyxjblH1FhpdSQZHR2N06dPw9nZGTt37sTOnTvh7OyMM2fO4OWXX9aorTt37uC1116Dk5MTrK2tERQUhHPnzimXC4KA2bNnQyaTwdraGuHh4bh+/bo2YRMZhZSMFPgk+aDP+j6ISYlBn/V94JPkg5SMFLFDIzI5Wj8nGRwcjI0bN9brwx88eIAePXqgT58+2LNnD1xcXHD9+nU0a9ZMuc6SJUuwfPlyrF+/Hr6+vpg1axYiIiJw7do1WFlZ1evziQyNsY65R9RYaVUkd+/eDalUioiICJX5+/btg0KhwKBBg+rUzuLFi+Hp6Ym1a9cq5/n6+ir/LQgCli1bhk8++QSRkZEAHj+f6erqip07d2LUqFHahE9kkJ415p4EEsTtjUNkm0ieeiVqIFoVyZkzZ2LRokU15guCgJkzZ9a5SP7www+IiIjAK6+8gqNHj6Jly5b429/+hokTJwJ43P1dbm4uwsPDle9xcHBA165dkZaWVmuRLC0tRWlpqXJaLpcDeDwOZnl5uUZ5PqnqvfVpw5iYWr6A+DmnZqcivzgf1mbWate5X3wfx7KOoadXT518ptg5NzRTyxdgztXnaUoiVPUrpwFra2tkZGTAx8dHZf7NmzfRtm1bPHz4sE7tVJ0ujY+PxyuvvIKzZ89i6tSpWL16NcaOHYuTJ0+iR48euHv3LmSy/w2hMnLkSEgkEmzZsqVGm4mJiZg7d26N+cnJybCxsdEgSyIiaixKSkoQExODwsJC2Nvb1/l9Wh1JOjg44P/+7/9qFMnffvutxrBZT6NQKNC5c2csWLAAAPDCCy/g6tWryiKpjYSEBMTHxyun5XI5PD09MWDAAI02THXl5eU4cOAA+vfvD3Nzc63bMRamli8gfs6p2akYkjzkmevtitml0yNJU9rPppYvwJyrcq46q6gprYpkZGQk4uLisGPHDrRu3RrA4wL5/vvv46WXXqpzOzKZDM8//7zKvMDAQPznP/8BALi5uQEA8vLyVI4k8/Ly0LFjx1rbtLS0hKWlZY355ubmOvkD0VU7xsLU8gXEyznMNwxOtk7PHHMvzDdM59ckTW0/m1q+AHPWNnetHgFZsmQJmjZtioCAAPj6+sLX1xcBAQFwcnLCP/7xjzq306NHD2RmZqrM++9//wtvb28Aj2/icXNzw8GDB5XL5XI5Tp8+jZCQEG1CJzJYVWPuAf8bY6+KoY+5R9RYaX269eTJkzhw4AAuXboEa2trdOjQAaGhmo0qPW3aNHTv3h0LFizAyJEjcebMGXz11Vf46quvADwefisuLg7z58+Hn5+f8hEQd3d3DB8+XJvQiQyasY65R9RYaVQk09LSkJ+fj6FDh0IikWDAgAHIycnBnDlzUFJSguHDh2PFihW1nu6szYsvvogdO3YgISEB8+bNg6+vL5YtW4YxY8Yo1/nggw/w8OFDTJo0CQUFBejZsyf27t3LZySp0TLGMfeIGiuNiuS8efPQu3dvDB06FABw5coVTJw4EWPHjkVgYCA+++wzuLu7IzExsc5tDh06VNlebSQSCebNm4d58+ZpEiqRUasac4+IxKXRNcmLFy+iX79+yunNmzejS5cu+PrrrxEfH4/ly5ez71YiImo0NCqSDx48gKurq3L66NGjKh0HvPjii7h9+7buoiMiIhKRRkXS1dUVWVlZAICysjKcP38e3bp1Uy4vKipqlLcYVyoqkZqdCuDxs2yVikqRIyIiooagUZEcPHgwZs6ciePHjyMhIQE2NjYqd7RevnxZ+dxkY1E1IkPVQ95DkodwRAYiIhOh0Y07f//73xEVFYVevXrB1tYW69evh4WFhXL5t99+iwEDBug8SLE8OSLDk/1p6npEBn2PHWiI7XO8RN3TdJtWP0Oij04KiIydRkXS2dkZx44dQ2FhIWxtbSGVqn6htm3bBltbW50GKJaGGpEhJSOl1mfikgYm6aQAG2L7+o7JFGm6TavWzy/Ox6b2mzAkeQicbJ24D4iq0arHHQcHhxoFEgCaN2+ucmRpzI5nH1f5walOgIDb8ts4nn1c68+oOlKt/jlVR6r1PaVriO3rOyZTpOk25T4gqjutiqQpyCnK0el61T3rSBUA4vbGaX2TkCG2r++YTJGm25T7gEgzLJJqyOxkz15Jg/Wq0/eRqiG23xBH56ZG023KfUCkGRZJNUK9QuFh71Gjo+kqEkjgae+JUC/N+qutou8jVUNsX98xmSJNtyn3AZFmWCTV0PeIDPo+UjXE9vUdkynSdJtyHxBphkXyKapGZGhp31Jlvoe9R70f/9D3kaohtq/vmEyRptuU+4BIMyySzxAVGIWbU29iV8wuAI9Hhc+amlXv2+T1faRqiO1zvETd03Sbch8QaYZFsg6kZlL09OoJAOjp1VNnPyD6PFI11Pb1HZMp0nSbch8Q1Z1Wgy6T7uh77EBDbJ/jJeqeptu0av1jWccgvyrHrphd7HGHqBYskgZA32MHGmL7HC9R9zTdplVnSHZf3a3TMyREjQlPtxIREanBIklERKQGiyQREZEaLJJERERqsEgSERGpwSJJRESkBoskERGRGiySREREarBIEhERqcEiSUREpAaLJBERkRoskkRERGqwSBIREanBIklERKSGqEUyMTEREolE5RUQEKBc/ujRI8TGxsLJyQm2traIjo5GXl6eiBETEZEpEf1Ism3btsjJyVG+UlNTlcumTZuGH3/8Edu2bcPRo0dx9+5dREVx1HQiImoYog+63KRJE7i5udWYX1hYiDVr1iA5ORl9+/YFAKxduxaBgYE4deoUunXr1tChEhGRiRG9SF6/fh3u7u6wsrJCSEgIFi5cCC8vL6Snp6O8vBzh4eHKdQMCAuDl5YW0tDS1RbK0tBSlpaXKablcDgAoLy9HeXm51nFWvbc+bRgTU8sXYM6mwNTyBZhz9XmakgiCIOgkKi3s2bMHxcXFaNOmDXJycjB37lzcuXMHV69exY8//ojx48erFDwA6NKlC/r06YPFixfX2mZiYiLmzp1bY35ycjJsbGz0kgcRERm2kpISxMTEoLCwEPb29nV+n6hFsrqCggJ4e3tj6dKlsLa21qpI1nYk6enpifv372u0YaorLy/HgQMH0L9/f5ibm2vdjrEwtXwB5mwKOZtavgBzrspZLpfD2dlZ4yIp+unWJzk6OsLf3x+//fYb+vfvj7KyMhQUFMDR0VG5Tl5eXq3XMKtYWlrC0tKyxnxzc3Od/IHoqh1jYWr5AszZFJhavgBz1jZ30e9ufVJxcTFu3LgBmUyG4OBgmJub4+DBg8rlmZmZyM7ORkhIiIhREhGRqRD1SHL69OkYNmwYvL29cffuXcyZMwdSqRSjR4+Gg4MD3nzzTcTHx6N58+awt7fHu+++i5CQEN7ZSkREDULUIvn7779j9OjRyM/Ph4uLC3r27IlTp07BxcUFAPD555/DzMwM0dHRKC0tRUREBL788ksxQyYiIhMiapHcvHnzU5dbWVlh5cqVWLlyZQNFRERE9D8GdU2SiIjIkLBIEhERqcEiSUREpAaLJBERkRoskkRERGqwSBIREanBIklERKQGiyQREZEaLJJERERqsEgSERGpwSJJRESkBoskERGRGgY16HJjUamoxPHs48gpyoHMToZQr1BIzaRih0VEZHTE/j1lkdSxlIwUTN07Fb/Lf1fO87D3QNLAJEQFRokYGRGRcTGE31OebtWhlIwUjNg6QmWHAsAd+R2M2DoCKRkpIkVGRGRcDOX3lEVSRyoVlZi6dyoECDWWVc2L2xuHSkVlQ4dGRGRUDOn3lEVSR45nH6/xP54nCRBwW34bx7OPN2BURETGx5B+T1kkdSSnKEen6xERmSpD+j1lkdQRmZ1Mp+sREZkqQ/o9ZZHUkVCvUHjYe0ACSa3LJZDA094ToV6hDRwZEZFxMaTfUxZJHZGaSZE0MAkAauzYqullA5fxeUkiomcwpN9TFkkdigqMwvaR29HSvqXKfA97D2wfuZ3PSRIR1ZGh/J6yMwEdiwqMQmSbSPa4Q0RUT4bwe8oiqQdSMyl6+/QWOwwiIqMn9u8pT7cSERGpwSJJRESkBoskERGRGiySREREarBIEhERqWEwRXLRokWQSCSIi4tTznv06BFiY2Ph5OQEW1tbREdHIy8vT7wgiYjIpBhEkTx79iz+9a9/oX379irzp02bhh9//BHbtm3D0aNHcffuXURF8YF8IiJqGKIXyeLiYowZMwZff/01mjVrppxfWFiINWvWYOnSpejbty+Cg4Oxdu1anDx5EqdOnRIxYiIiMhWidyYQGxuLIUOGIDw8HPPnz1fOT09PR3l5OcLDw5XzAgIC4OXlhbS0NHTr1q3W9kpLS1FaWqqclsvlAIDy8nKUl5drHWfVe+vThjExtXwB5mwKTC1fgDlXn6cpUYvk5s2bcf78eZw9e7bGstzcXFhYWMDR0VFlvqurK3Jzc9W2uXDhQsydO7fG/P3798PGxqbeMR84cKDebRgTU8sXYM6mwNTyBZhzSUmJVm2IViRv376NqVOn4sCBA7CystJZuwkJCYiPj1dOy+VyeHp6YsCAAbC3t9e63fLychw4cAD9+/eHubm5LkI1aKaWL8CcTSFnU8sXYM5VOVedVdSUaEUyPT0d9+7dQ6dOnZTzKisrcezYMXzxxRfYt28fysrKUFBQoHI0mZeXBzc3N7XtWlpawtLSssZ8c3NznfyB6KodY2Fq+QLM2RSYWr4Ac9Y2d9GKZL9+/XDlyhWVeePHj0dAQAA+/PBDeHp6wtzcHAcPHkR0dDQAIDMzE9nZ2QgJCREjZCIiMjGiFUk7Ozu0a9dOZV7Tpk3h5OSknP/mm28iPj4ezZs3h729Pd59912EhISovWmHiIhIl0S/u/VpPv/8c5iZmSE6OhqlpaWIiIjAl19+KXZYRERkIgyqSB45ckRl2srKCitXrsTKlSvFCYiIiEya6J0JEBERGSoWSSIiIjVYJImIiNRgkSQiIlLDoG7cISIi41GpqMTx7OPIKcqBzE6GUK9QSM2kYoelUyySRESksZSMFEzdOxW/y39XzvOw90DSwCREBTaeIQ15upWIiDSSkpGCEVtHqBRIALgjv4MRW0cgJSNFpMh0j0WSiIjqrFJRial7p0KAUGNZ1by4vXGoVFQ2dGh6wSJJRER1djz7eI0jyCcJEHBbfhvHs483YFT6wyJJRER1llOUo9P1DB2LJBER1ZnMTqbT9QwdiyQREdVZqFcoPOw9IIGk1uUSSOBp74lQr9AGjkw/WCSJiKjOpGZSJA1MAoAahbJqetnAZY3meUkWSSIi0khUYBS2j9yOlvYtVeZ72Htg+8jtjeo5SXYmQEREGosKjEJkm0j2uENERFQbqZkUvX16ix2GXvF0KxERkRoskkRERGqwSBIREanR6K9JCsLjvgTlcnm92ikvL0dJSQnkcjnMzc11EZpBM7V8AeZsCjmbWr4Ac67KuaoGVNWEumr0RbKoqAgA4OnpKXIkREQktqKiIjg4ONR5fYmgaVk1MgqFAnfv3oWdnR0kktp7iKgLuVwOT09P3L59G/b29jqM0DCZWr4AczaFnE0tX4A5V+UsCAKKiorg7u4OM7O6X2ls9EeSZmZm8PDw0Fl79vb2JvOHBphevgBzNgWmli/AnAFodARZhTfuEBERqcEiSUREpAaLZB1ZWlpizpw5sLS0FDuUBmFq+QLM2RSYWr4Ac66vRn/jDhERkbZ4JElERKQGiyQREZEaLJJERERqsEgSERGpwSJZBytXroSPjw+srKzQtWtXnDlzRuyQ9CYxMRESiUTlFRAQIHZYOnXs2DEMGzYM7u7ukEgk2Llzp8pyQRAwe/ZsyGQyWFtbIzw8HNevXxcnWB14Vr7jxo2rsc8HDhwoTrA6snDhQrz44ouws7NDixYtMHz4cGRmZqqs8+jRI8TGxsLJyQm2traIjo5GXl6eSBHXT13y7d27d439PHnyZJEirr9Vq1ahffv2yg4DQkJCsGfPHuVyXe1fFsln2LJlC+Lj4zFnzhycP38eHTp0QEREBO7duyd2aHrTtm1b5OTkKF+pqalih6RTDx8+RIcOHbBy5cpaly9ZsgTLly/H6tWrcfr0aTRt2hQRERF49OhRA0eqG8/KFwAGDhyoss83bdrUgBHq3tGjRxEbG4tTp07hwIEDKC8vx4ABA/Dw4UPlOtOmTcOPP/6Ibdu24ejRo7h79y6ioqJEjFp7dckXACZOnKiyn5csWSJSxPXn4eGBRYsWIT09HefOnUPfvn0RGRmJX375BYAO969AT9WlSxchNjZWOV1ZWSm4u7sLCxcuFDEq/ZkzZ47QoUMHscNoMACEHTt2KKcVCoXg5uYmfPbZZ8p5BQUFgqWlpbBp0yYRItSt6vkKgiCMHTtWiIyMFCWehnLv3j0BgHD06FFBEB7vU3Nzc2Hbtm3KdTIyMgQAQlpamlhh6kz1fAVBEHr16iVMnTpVvKAaQLNmzYRvvvlGp/uXR5JPUVZWhvT0dISHhyvnmZmZITw8HGlpaSJGpl/Xr1+Hu7s7WrVqhTFjxiA7O1vskBpMVlYWcnNzVfa5g4MDunbt2qj3+ZEjR9CiRQu0adMG77zzDvLz88UOSacKCwsBAM2bNwcApKeno7y8XGU/BwQEwMvLq1Hs5+r5Vvnuu+/g7OyMdu3aISEhASUlJWKEp3OVlZXYvHkzHj58iJCQEJ3u30bfwXl93L9/H5WVlXB1dVWZ7+rqil9//VWkqPSra9euWLduHdq0aYOcnBzMnTsXoaGhuHr1Kuzs7MQOT+9yc3MBoNZ9XrWssRk4cCCioqLg6+uLGzdu4KOPPsKgQYOQlpYGqVQqdnj1plAoEBcXhx49eqBdu3YAHu9nCwsLODo6qqzbGPZzbfkCQExMDLy9veHu7o7Lly/jww8/RGZmJlJSUkSMtn6uXLmCkJAQPHr0CLa2ttixYweef/55XLx4UWf7l0WSVAwaNEj57/bt26Nr167w9vbG1q1b8eabb4oYGenLqFGjlP8OCgpC+/bt0bp1axw5cgT9+vUTMTLdiI2NxdWrVxvdtXV11OU7adIk5b+DgoIgk8nQr18/3LhxA61bt27oMHWiTZs2uHjxIgoLC7F9+3aMHTsWR48e1eln8HTrUzg7O0Mqlda4IyovLw9ubm4iRdWwHB0d4e/vj99++03sUBpE1X415X3eqlUrODs7N4p9PmXKFPz00084fPiwypB5bm5uKCsrQ0FBgcr6xr6f1eVbm65duwKAUe9nCwsLPPfccwgODsbChQvRoUMHJCUl6XT/skg+hYWFBYKDg3Hw4EHlPIVCgYMHDyIkJETEyBpOcXExbty4AZlMJnYoDcLX1xdubm4q+1wul+P06dMms89///135OfnG/U+FwQBU6ZMwY4dO3Do0CH4+vqqLA8ODoa5ubnKfs7MzER2drZR7udn5VubixcvAoBR7+fqFAoFSktLdbt/dXtvUeOzefNmwdLSUli3bp1w7do1YdKkSYKjo6OQm5srdmh68f777wtHjhwRsrKyhBMnTgjh4eGCs7OzcO/ePbFD05mioiLhwoULwoULFwQAwtKlS4ULFy4It27dEgRBEBYtWiQ4OjoK33//vXD58mUhMjJS8PX1Ff766y+RI9fO0/ItKioSpk+fLqSlpQlZWVnCzz//LHTq1Enw8/MTHj16JHboWnvnnXcEBwcH4ciRI0JOTo7yVVJSolxn8uTJgpeXl3Do0CHh3LlzQkhIiBASEiJi1Np7Vr6//fabMG/ePOHcuXNCVlaW8P333wutWrUSwsLCRI5cezNnzhSOHj0qZGVlCZcvXxZmzpwpSCQSYf/+/YIg6G7/skjWwYoVKwQvLy/BwsJC6NKli3Dq1CmxQ9KbV199VZDJZIKFhYXQsmVL4dVXXxV+++03scPSqcOHDwsAarzGjh0rCMLjx0BmzZoluLq6CpaWlkK/fv2EzMxMcYOuh6flW1JSIgwYMEBwcXERzM3NBW9vb2HixIlG/5/A2vIFIKxdu1a5zl9//SX87W9/E5o1aybY2NgIL7/8spCTkyNe0PXwrHyzs7OFsLAwoXnz5oKlpaXw3HPPCTNmzBAKCwvFDbweJkyYIHh7ewsWFhaCi4uL0K9fP2WBFATd7V8OlUVERKQGr0kSERGpwSJJRESkBoskERGRGiySREREarBIEhERqcEiSUREpAaLJBERkRoskkRERGqwSBI1kMTERHTs2FGj90gkEuzcuVMv8ejCzZs3IZFIlP2AEjU2LJJEWpBIJE99JSYm1njP9OnTVTpc1oVx48ZBIpFg0aJFKvN37twJiUSi088iMkUcT5JICzk5Ocp/b9myBbNnz0ZmZqZynq2trfLfgiCgsrIStra2KvN1xcrKCosXL8bbb7+NZs2a6bx9MZSVlcHCwkLsMIh4JEmkDTc3N+XLwcEBEolEOf3rr7/Czs4Oe/bsQXBwMCwtLZGamlrjdOvZs2fRv39/ODs7w8HBAb169cL58+c1jiU8PBxubm5YuHCh2nVqO9W7bNky+Pj4KKfHjRuH4cOHY8GCBXB1dYWjoyPmzZuHiooKzJgxA82bN4eHhwfWrl1bo/1ff/0V3bt3h5WVFdq1a1dj4NurV69i0KBBsLW1haurK15//XXcv39fubx3796YMmUK4uLi4OzsjIiICI23A5E+sEgS6cnMmTOxaNEiZGRkoH379jWWFxUVYezYsUhNTcWpU6fg5+eHwYMHo6ioSKPPkUqlWLBgAVasWIHff/+9XjEfOnQId+/exbFjx7B06VLMmTMHQ4cORbNmzXD69GlMnjwZb7/9do3PmTFjBt5//31cuHABISEhGDZsGPLz8wEABQUF6Nu3L1544QWcO3cOe/fuRV5eHkaOHKnSxvr162FhYYETJ05g9erV9cqDSFdYJIn0ZN68eejfvz9at26N5s2b11jet29fvPbaawgICEBgYCC++uorlJSU1DgKq4uXX34ZHTt2xJw5c+oVc/PmzbF8+XK0adMGEyZMQJs2bVBSUoKPPvoIfn5+SEhIgIWFBVJTU1XeN2XKFERHRyMwMBCrVq2Cg4MD1qxZAwD44osv8MILL2DBggUICAjACy+8gG+//RaHDx/Gf//7X2Ubfn5+WLJkCdq0aYM2bdrUKw8iXWGRJNKTzp07P3V5Xl4eJk6cCD8/Pzg4OMDe3h7FxcXIzs7W6vMWL16M9evXIyMjQ6v3A0Dbtm1hZva/nwVXV1cEBQUpp6VSKZycnHDv3j2V9z052nuTJk3QuXNnZRyXLl3C4cOHlddkbW1tERAQAAC4ceOG8n3BwcFax02kL7xxh0hPmjZt+tTlY8eORX5+PpKSkuDt7Q1LS0uEhISgrKxMq88LCwtDREQEEhISMG7cOJVlZmZmqD50bHl5eY02zM3NVaYlEkmt8xQKRZ3jKi4uxrBhw7B48eIay2QymfLfz9peRGJgkSQSyYkTJ/Dll19i8ODBAIDbt2+r3MyijUWLFqFjx441Tle6uLggNzcXgiAoHw3R5bONp06dQlhYGACgoqIC6enpmDJlCgCgU6dO+M9//gMfHx80acKfHDIuPN1KJBI/Pz9s2LABGRkZOH36NMaMGQNra+t6tRkUFIQxY8Zg+fLlKvN79+6NP/74A0uWLMGNGzewcuVK7Nmzp16f9aSVK1dix44d+PXXXxEbG4sHDx5gwoQJAIDY2Fj8+eefGD16NM6ePYsbN25g3759GD9+PCorK3UWA5E+sEgSiWTNmjV48OABOnXqhNdffx3vvfceWrRoUe92582bV+N0aGBgIL788kusXLkSHTp0wJkzZzB9+vR6f1aVRYsWYdGiRejQoQNSU1Pxww8/wNnZGQDg7u6OEydOoLKyEgMGDEBQUBDi4uLg6Oiocv2TyBBJhOoXKoiIiAgAjySJiIjUYpEkIiJSg0WSiIhIDRZJIiIiNVgkiYiI1GCRJCIiUoNFkoiISA0WSSIiIjVYJImIiNRgkSQiIlKDRZKIiEiN/wfyPpxKndSj6gAAAABJRU5ErkJggg==",
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAckAAAE8CAYAAACrYErbAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/H5lhTAAAACXBIWXMAAA9hAAAPYQGoP6dpAABFeklEQVR4nO3deViU5f4/8PeAMKwDiiggq+JC5pK4hAqKouCWCmYuJeVWqSXSJnUMNculMsnMTstBTx5cQ07mFlEquKUYLicz8ovigmgWDEvAyNy/P/wxOcIgMww8A/N+XRfX5dzPM/d8Ps/j8OHZ7lsmhBAgIiKiaiykDoCIiMhUsUgSERHpwCJJRESkA4skERGRDiySREREOrBIEhER6cAiSUREpAOLJBERkQ4skkRERDqwSBI1ERs2bIBMJsPJkyelDoXIbLBIEtWgqiDd+9OmTRuEhoZi7969Bvf7zjvvICUlxXiB6ikjIwMjRoxAu3btYGNjA29vb4wZMwZJSUmSxURkylgkiWqxdOlSfPnll/j3v/+NV199Fbdu3cLIkSPxzTffGNSflEVy+/btCAkJQX5+PubPn4+1a9fiySefxJ9//onPPvtMkpiITF0LqQMgMmUjRoxA7969Na9nzJiBtm3bYvPmzRg9erSEkelv8eLFeOihh3Ds2DFYW1trLbt582ajxSGEQFlZGWxtbRvtM4kMxSNJIj04OzvD1tYWLVpo/3353nvvoX///nBxcYGtrS0CAwOxY8cOrXVkMhlKSkqwceNGzSncp59+WrP82rVrmDFjBjw8PCCXy+Hn54fnn38eFRUVWv2Ul5cjNjYWrq6usLe3x/jx43Hr1q0Hxn7x4kX06dOnWoEEgDZt2mi9VqvVSEhIQLdu3WBjYwNXV1dERERoXQ+9c+cO3nrrLXTo0AFyuRy+vr54/fXXUV5ertWXr68vRo8ejf3796N3796wtbXFP//5TwBAQUEBYmJi4OXlBblcDn9/f6xcuRJqtVqrjy1btiAwMBCOjo5QKBTo1q0bEhISHpgzUX3xSJKoFoWFhfj9998hhMDNmzexdu1aFBcX48knn9RaLyEhAY899himTp2KiooKbNmyBY8//ji++eYbjBo1CgDw5ZdfYubMmejbty9mz54NAOjQoQMA4Pr16+jbty8KCgowe/ZsdOnSBdeuXcOOHTtQWlqqVdheeOEFtGzZEvHx8bh06RLWrFmDefPmYevWrbXm4uPjg7S0NFy9ehWenp61rjtjxgxs2LABI0aMwMyZM3Hnzh2kp6fj2LFjmiPrmTNnYuPGjZgwYQJeeuklHD9+HMuXL8f58+exc+dOrf4uXLiAyZMn49lnn8WsWbPQuXNnlJaWYtCgQbh27RqeffZZeHt748iRI4iLi0NeXh7WrFkDAEhNTcXkyZMxdOhQrFy5EgBw/vx5HD58GPPnz681D6J6E0RUTWJiogBQ7Ucul4sNGzZUW7+0tFTrdUVFhXj44YfFkCFDtNrt7e1FdHR0tfdPmzZNWFhYiBMnTlRbplartWIKCwvTtAkhxIIFC4SlpaUoKCioNacvvvhCABDW1tYiNDRULFq0SKSnp4vKykqt9b7//nsBQLz44os6Y8nKyhIAxMyZM7WWv/zyywKA+P777zVtPj4+AoDYt2+f1rpvvfWWsLe3F7/++qtW+8KFC4WlpaXIzc0VQggxf/58oVAoxJ07d2rNj6gh8HQrUS3WrVuH1NRUpKamYtOmTQgNDcXMmTORnJystd6919f+/PNPFBYWIjg4GKdOnXrgZ6jVaqSkpGDMmDFa1z+ryGQyrdezZ8/WagsODkZlZSUuX75c6+dMnz4d+/btw+DBg5GRkYG33noLwcHB6NixI44cOaJZ76uvvoJMJkN8fLzOWPbs2QMAiI2N1Vr+0ksvAQB2796t1e7n54fw8HCttu3btyM4OBgtW7bE77//rvkJCwtDZWUlDh06BODuKe6SkhKkpqbWmh9RQ+DpVqJa9O3bV6twTZ48GY888gjmzZuH0aNHa06DfvPNN1i2bBmysrK0rsndX+BqcuvWLSiVSjz88MN1isnb21vrdcuWLQHcLc4PEh4ejvDwcJSWliIzMxNbt27FJ598gtGjR+OXX35BmzZtcPHiRXh4eKBVq1Y6+7l8+TIsLCzg7++v1e7m5gZnZ+dqBdvPz69aH9nZ2Thz5gxcXV1r/Iyqm4nmzJmDbdu2aR5dGT58OCZOnIiIiIgH5ktUXyySRHqwsLBAaGgoEhISkJ2dja5duyI9PR2PPfYYQkJC8PHHH8Pd3R1WVlZITExskOcPLS0ta2wXQtS5Dzs7OwQHByM4OBitW7fGkiVLsHfvXkRHR+sVS13+CABQ452sarUaw4YNw6uvvlrjezp16gTg7k1FWVlZ2L9/P/bu3Yu9e/ciMTER06ZNw8aNG/WKl0hfLJJEerpz5w4AoLi4GMDd05M2NjbYv38/5HK5Zr3ExMRq762pqLi6ukKhUODcuXMNFHHtqo6U8/LyANy9mWj//v34448/dB5N+vj4QK1WIzs7GwEBAZr2/Px8FBQUwMfH54Gf26FDBxQXFyMsLOyB61pbW2PMmDEYM2YM1Go15syZg3/+859YtGhRtaNZImPiNUkiPahUKnz77bewtrbWFAdLS0vIZDJUVlZq1rt06VKNgwbY29ujoKBAq83CwgLjxo3Drl27ahxyTp8jxNqkpaXV2F51fbFz584AgKioKAghsGTJEp2xjBw5EgA0d6BWWb16NQBo7uitzcSJE3H06FHs37+/2rKCggLNHyO3b9/WWmZhYYHu3bsDQLXHTYiMjUeSRLXYu3cvfvnlFwB3r5ElJSUhOzsbCxcuhEKhAHC3IKxevRoRERGYMmUKbt68iXXr1sHf3x9nzpzR6i8wMBDfffcdVq9eDQ8PD/j5+aFfv35455138O2332LQoEGYPXs2AgICkJeXh+3btyMjIwPOzs71zmXs2LHw8/PDmDFj0KFDB5SUlOC7777Drl270KdPH4wZMwYAEBoaiqeeegoffvghsrOzERERAbVajfT0dISGhmLevHno0aMHoqOj8emnn6KgoACDBg3Cjz/+iI0bN2LcuHEIDQ19YDyvvPIKvv76a4wePRpPP/00AgMDUVJSgrNnz2LHjh24dOkSWrdujZkzZ+KPP/7AkCFD4OnpicuXL2Pt2rXo2bOn1lEsUYOQ9uZaItNU0yMgNjY2omfPnmL9+vVaj2AIcffxio4dOwq5XC66dOkiEhMTRXx8vLj/K/bLL7+IkJAQYWtrKwBoPQ5y+fJlMW3aNOHq6irkcrlo3769mDt3rigvL9eK6f7HRH744QcBQPzwww+15rR582YxadIk0aFDB2FraytsbGzEQw89JN544w2hVCq11r1z54549913RZcuXYS1tbVwdXUVI0aMEJmZmZp1VCqVWLJkifDz8xNWVlbCy8tLxMXFibKyMq2+fHx8xKhRo2qMqaioSMTFxQl/f39hbW0tWrduLfr37y/ee+89UVFRIYQQYseOHWL48OGiTZs2wtraWnh7e4tnn31W5OXl1ZovkTHIhDDSuRwiIqJmhtckiYiIdGCRJCIi0oFFkoiISAcWSSIiIh1YJImIiHSQtEgWFRUhJiYGPj4+sLW1Rf/+/XHixAnNciEE3nzzTbi7u8PW1hZhYWHIzs6WMGIiIjInkg4mMHPmTJw7dw5ffvklPDw8sGnTJoSFheHnn39Gu3btsGrVKnz44YfYuHEj/Pz8sGjRIoSHh+Pnn3+GjY1NnT5DrVbj+vXrcHR0rPM4k0RE1LwIIVBUVAQPDw9YWOhxfCjVA5qlpaXC0tJSfPPNN1rtvXr1Em+88YZQq9XCzc1NvPvuu5plBQUFQi6Xi82bN9f5c65cuVLjvID84Q9/+MMf8/u5cuWKXrVKsiPJO3fuoLKystoRoa2tLTIyMpCTk4MbN25oDX7s5OSEfv364ejRo5g0aVKN/ZaXl2uN5yj+/1gJOTk5cHR0NDhelUqFH374AaGhobCysjK4n6bC3PIFmLM55Gxu+QLMuSrnoqIi+Pn56V0HJB1xp3///rC2tkZSUhLatm2LzZs3Izo6Gv7+/khMTMSAAQNw/fp1uLu7a94zceJEyGQybN26tcY+Fy9eXOPAzElJSbCzs2uwXIiIyHSVlpZiypQpKCws1Iy7XBeSXpP88ssvMX36dLRr1w6Wlpbo1asXJk+ejMzMTIP7jIuL05otXalUwsvLC8OHD9drw9xPpVIhNTUVw4YNM4u/xswtX4A5m0PO5pYvwJyrclYqlQb1JWmR7NChAw4ePIiSkhIolUq4u7vjiSeeQPv27eHm5gbg7vx09x5J5ufno2fPnjr7lMvlWnP6VbGysjLKfxBj9dNUmFu+AHM2B+aWL8CcDc3dJJ6TtLe3h7u7O/7880/s379fM6WPm5ub1hx4SqUSx48fR1BQkITREhEZR6W6EgcuHcDms5tx4NIBVKorH/wmalSSHknu378fQgh07twZv/32G1555RV06dIFzzzzDGQyGWJiYrBs2TJ07NhR8wiIh4cHxo0bZ9Q4hBCaG4l0UalUaNGiBcrKympdr7kwt3wB08rZysoKlpaWksZADSv5fDLm75uPq8qrmjZPhScSIhIQGRApYWR0L0mLZGFhIeLi4nD16lW0atUKUVFRePvttzWHxa+++ipKSkowe/ZsFBQUYODAgdi3b1+dn5Gsi4qKCuTl5aG0tLTW9YQQcHNzw5UrV8zieUtzyxcwrZxlMhk8PT3h4OAgaRzUMJLPJ2PCtgkQ0L5v8pryGiZsm4AdE3ewUJoISYvkxIkTMXHiRJ3LZTIZli5diqVLlzbI56vVauTk5MDS0hIeHh6wtrbW+ctRrVajuLgYDg4O+j2I2kSZW76A6eQshMCtW7dw9epVdOzYkUeUzUyluhLz982vViABQEBABhli9sVgbOexsLTgvpeapEVSahUVFVCr1fDy8nrg4yFqtRoVFRWwsbExi6JhbvkCppWzq6srLl26BJVKxSLZzKTnpmudYr2fgMAV5RWk56ZjsO/gxguMamQev/0eQOpfiET3k/p0LzWcvKI8o65HDYvVgYioEbk7uj94JT3Wo4bFIklE1IiCvYPhqfCEDDWfLZBBBi+FF4K9gxs5MqoJiySZnMOHD6Nbt26wsrKq8+M+ixcvrnWQCan5+vpizZo1UodBJsDSwhIJEQkAUK1QVr1eE7GGN+2YCBbJJujWrVt4/vnn4e3tDblcDjc3N4SHh+Pw4cNSh2YUsbGx6NmzJ3JycrBhwwaj9Hnp0iXIZDLNj7W1Nfz9/bFs2TLoO3yxTCZDSkqKUeIi8xQZEIkdE3egnaKdVrunwpOPf5gYs7671Vgq1ZVIz01HXlEe3B3dEewd3KB/BUZFRaGiogIbN25E+/btkZ+fj7S0NNy+fbvBPrMxXbx4Ec899xw8PT2N3vd3332Hrl27ory8HBkZGZg5cybc3d0xY8YMo38WUW0iAyIxtvPYRv3dQfrjkWQ9JZ9Phm+CL0I3hmJK8hSEbgyFb4Ivks8nN8jnFRQUID09HStXrkRoaCh8fHzQt29fxMXF4bHHHgPw91FTVlaW1vtkMhkOHDigafvf//6H0aNHQ6FQwNHREcHBwbh48aJm+aZNm9CtWzfI5XK4u7tj3rx5Wv3NnDkTrq6uUCgUGDJkCE6fPq1Zfvr0aYSGhsLR0REKhQKBgYE4efIkAODy5csYM2YMWrZsCXt7e3Tt2hV79uzRxH379m1Mnz4dMpkMGzZswIYNG+Ds7Ky1HVJSUgy6A9TFxQVubm7w8fHB1KlTMWDAAJw6dUqz/NSpUxg+fDhat24NJycnDBo0SGu5r68vAGD8+PGQyWSa1wCwa9cu9OnTBzY2NmjdujXGjx+v9dmlpaWYPn06HB0d4e3tjU8//VTv+Kl5sbSwxGDfwZjcbTIG+w5mgTRBLJL1UDVqxv3PPFWNmtEQhdLBwQEODg5ISUnRmjdTX9euXUNISAjkcjm+//57ZGZmYvr06bhz5w4AYP369XjllVcwa9YsnD17Fl9//TX8/f0173/88cdx8+ZN7N27F5mZmejVqxeGDh2KP/74AwAwdepUeHp64sSJE8jMzMTChQs1IynNnTsX5eXlOHToEM6ePYuVK1fCwcEBXl5eyMvLg0KhwJo1a5CXl4cnnniiHlurdidPnkRmZib69eunaSsuLsa0adOQkZGBY8eOoWPHjhg5ciSKiooAACdOnAAAJCYmIi8vT/N69+7dGD9+PEaOHImffvoJaWlp6Nu3r9bnvf/+++jduzd++uknzJkzB88//zwuXLjQYPkRUf3xdKuBpBo1o0WLFtiwYQNmzZqFTz75BL169cKgQYMwadIkdO/evc79rFu3Dk5OTtiyZYumeHXq1Emz/J133sHcuXPx4osvap4j7dOnDwAgIyMDP/74I27evKmZceW9995DSkoKduzYgdmzZyM3N1czFi8AdOzYUdN3bm4uoqKi0K1bNwBA+/btNcvc3Nwgk8ng5OSkmQnGmPr37w8LCwtUVFRApVJh9uzZmDZtmmZ5SEgIFAqFJudPP/0Uzs7OOHjwIEaPHg1XV1cAgLOzs1Z8b7/9NiZNmqQ1l2mPHj20PnvkyJGYM2cOAOC1117DBx98gB9++AGdO3c2ep5EZBw8kjSQPqNmGFtUVBSuX7+Or7/+GhEREThw4AB69eql100uWVlZCA4OrnH6mJs3b+L69esYNGhQje89ffo0iouL4eLiojmydXBwQE5OjuZ0bWxsLGbOnImwsDCsWLFC6zTuiy++iGXLlmHAgAGIj4/HmTNn9NsA9bB161ZkZWXh9OnT2LZtG/773/9i4cKFmuU3b97E7Nmz0bFjRzg5OUGhUKC4uBi5ubm19puVlYWhQ4fWus69f8TIZDK4ubnh5s2b9UuIiBoUi6SBpB41w8bGBsOGDcOiRYtw5MgRPP3004iPjwfw9whC9961qVKptN5va2urs+/algF3T0m6u7sjKytL6+fChQt45ZVXANx9JON///sfRo0ahe+//x4PPfQQdu7cCQCYOXMm/u///g9PPfUUzp49i969e2Pt2rU6P8/CwqLaHaj351NXXl5e8Pf3R0BAAB5//HHExMTg/fffR1lZGQBgzpw5OH36NBISEnDkyBFkZWXBxcUFFRUVtfb7oG0GVJ/PTiaTQa1WG5QHETUOFkkDmdqoGQ899BBKSkoAQHNKMC/v7wJ97008wN2jmvT09BqLjaOjI3x9fXHw4MEaP6tXr164ceMGWrRoAX9/f62f1q1ba9br1KkTFixYgG+//RaRkZFITEzULPPy8sJzzz2H5ORkvPTSS/jss8905ubq6oqioiJNfjXlYyhLS0vcuXNHUwSPHz+OefPmYeTIkejatSvkcjl+//13rfdYWVlVm0qre/fuWnOfElHzwCJpIKlGzbh9+zaGDBmCTZs24cyZM8jJycH27duxatUqjB07FsDdo5pHH30UK1aswPnz53Hw4EH84x//0Opn3rx5UCqVmDRpEk6ePIns7Gx8+eWXmhtJ3nzzTaxbtw5r165FdnY2Tp06pTnaCwsLQ1BQEMaNG4dvv/0Wly5dwpEjR/DGG2/g5MmT+OuvvzBv3jwcOHAAly9fxuHDh3HixAkEBAQAAGJiYrB//37k5OTg1KlT+OGHHzTLatKvXz/Y2dnh9ddfx8WLF5GUlGTw85O3b9/GjRs3cPXqVezduxcJCQkIDQ2FQqEAcPf66KZNm3D+/HkcP34cU6dOrXaU6Ovri7S0NNy4cQN//vknACA+Ph6bN29GfHw8zp8/r7khiYiaNhZJA0k1aoaDgwP69euHDz74ACEhIXj44YexaNEizJo1Cx999JFmvX/961+4c+cOAgMDNZNX38vFxQXff/89iouLMWjQIAQGBuKzzz7TnBKMjo7GO++8g/Xr16Nr164YPXo0srOz7+Ynk2HPnj0ICQnBM888g06dOmHSpEm4fPky2rZtC0tLS9y+fRvTpk1Dp06dMHHiRIwYMUJzU0tlZSXmzp2LgIAAREREoFOnTvj444915tyqVSts2rQJe/bsQbdu3bB582YsXrzYoO0XFhYGd3d3+Pr6Yvbs2Rg5ciS2bt2qWb527VoUFBSgV69eeOqpp/Diiy+iTZs2Wn28//77SE1NhZeXFx555BEAwODBg7F9+3Z8/fXX6NmzJ4YMGYIff/zRoBiJyHTIhL7DjTQxSqUSTk5OKCws1BwtVCkrK0NOTg78/PweOJGzWq2GUqnUuvMRqHl2cS+FF9ZErGnSo2boyrc5M6Wc9fm/WR8qlQp79uzByJEja7yJq7kxt3wB5lyVc221oDZ8BKSeOGoGEVHzxSJpBFWjZhARUfNiHufRiIiIDMAiSUREpAOLJKD3VElEDY3/J4lMg1kXyaq7nkpLSyWOhEhb1eAGlpa8AYxISmZ9446lpSWcnZ0142fa2dnpnH5JrVajoqICZWVlkj8e0BjMLV/AdHJWq9W4desW7Ozs0KKFWX9FiSRn9t/AqpkcHjTQtBACf/31F2xtbQ2ax7CpMbd8AdPK2cLCAt7e3pLHQWTuzL5IymQyuLu7o02bNrUOmq1SqXDo0CGEhISYxQO55pYvYFo5W1tbm80RPJEpM/siWcXS0rLW6z9VA2Hb2NhI/gu0MZhbvoB55kxEteOfqkRERDqwSBIREenAIklERKQDr0lSs1SprjSpQedNLR4iqhtJjyQrKyuxaNEi+Pn5wdbWFh06dMBbb72lNdqIEAJvvvkm3N3dYWtri7CwMM28hkQ1ST6fDN8EX4RuDMWU5CkI3RgK3wRfJJ9PZjxEpBdJi+TKlSuxfv16fPTRRzh//jxWrlyJVatWYe3atZp1Vq1ahQ8//BCffPIJjh8/Dnt7e4SHh6OsrEzCyMlUJZ9PxoRtE7Tm9wSAa8prmLBtQqMXJlOLh4j0I2mRPHLkCMaOHYtRo0bB19cXEyZMwPDhwzUzugshsGbNGvzjH//A2LFj0b17d/z73//G9evXkZKSImXoZIIq1ZWYv28+BKqPe1rVFrMvBpXqSrOMh4j0J+k1yf79++PTTz/Fr7/+ik6dOuH06dPIyMjA6tWrAQA5OTm4ceMGwsLCNO9xcnJCv379cPToUUyaNKlan+Xl5SgvL9e8ViqVAO4+KF7bYAEPUvXe+vTRlDTFfDNyM3C7+DZsLWx1rvN78e84lHMIA70HVltm7JzrG09jaIr7uT7MLV+AOd/fpi+ZkHC6AbVajddffx2rVq2CpaUlKisr8fbbbyMuLg7A3SPNAQMG4Pr163B3d9e8b+LEiZDJZNi6dWu1PhcvXowlS5ZUa09KSoKdnV3DJUNERCartLQUU6ZMQWFhIRQKRZ3fJ+mR5LZt2/Cf//wHSUlJ6Nq1K7KyshATEwMPDw9ER0cb1GdcXBxiY2M1r5VKJby8vDB8+HC9Nsz9VCoVUlNTMWzYMLMYjaUp5puRm4FRSaMeuN7uKbt1HkkaM+f6xtMYmuJ+rg9zyxdgzlU5V51V1JekRfKVV17BwoULNadNu3XrhsuXL2P58uWIjo7WDD6en5+vdSSZn5+Pnj171tinXC6HXC6v1m5lZWWU/yDG6qepaEr5hviFwMXBBdeU12q8DiiDDJ4KT4T4hdT6+IWxcjZWPI2hKe1nYzC3fAHmbGjukt64U1paWm0QZ0tLS6jVagCAn58f3NzckJaWplmuVCpx/PhxBAUFNWqsZPosLSyREJEA4G4BulfV6zURaxqtIJlaPESkP0mL5JgxY/D2229j9+7duHTpEnbu3InVq1dj/PjxAO7O0BETE4Nly5bh66+/xtmzZzFt2jR4eHhg3LhxUoZOJioyIBI7Ju5AO0U7rXZPhSd2TNyByIBIs46HiPQj6enWtWvXYtGiRZgzZw5u3rwJDw8PPPvss3jzzTc167z66qsoKSnB7NmzUVBQgIEDB2Lfvn2wsbGRMHIyZZEBkRjbeazJjHBjavEQUd1JWiQdHR2xZs0arFmzRuc6MpkMS5cuxdKlSxsvMGryLC0sMdh3sNRhaJhaPERUNxzgnIiISAcWSSIiIh1YJImIiHRgkSQiItKB80kSEemg7zygnDe0+WGRJCKqQfL5ZMzfN19rmjNPhScSIhJqfL5V3/WpaeDpViKi++g7DyjnDW2+WCSJiO6h7zygnDe0eWORJCK6R3puerUjwnsJCFxRXkF6brpB61PTwiJJRHSPvKI8vdbTd31qWlgkiYju4e7o/uCV7llP3/WpaWGRJCK6R7B3MDwVntWmN6sigwxeCi8EewcbtD41LSySRET30HceUM4b2ryxSBIR3UffeUA5b2jzxcEEiIhqoO88oJw3tHlikSQi0kHfeUA5b2jzw9OtREREOrBIEhER6cAiSUREpAOLJBERkQ68cYeqqVRXIiM3AwCQkZuBEL8Qye/QM7V5+kwtnuaA2/TBDJnf0tS+y00NiyRpqZoT73bxbWzuvhmjkkbBxcFF0jnxTG2ePlOLpzngNn0wQ+e3NKXvclPE062kYYpz4plaTKYWT3PAbfpgnN9SOiySBMA058QztZhMLZ7mgNv0wTi/pbRYJAmAac6JZ2oxmVo8zUFjbtP7r881lSLB+S2lxSJJAExzTjxTi8nU4mkOGmubJp9Phm+CL0YljQIAjEoaBd8E3yZx2pHzW0qLRZIAmOaceKYWk6nF0xw0xjZt6tfnOL+ltFgkCYBpzolnajGZWjzNQUNv0+ZwfY7zW0qLRZIAmOaceKYWk6nF0xw09DZtDtfnOL+ltCQtkr6+vpDJZNV+5s6dCwAoKyvD3Llz4eLiAgcHB0RFRSE/P1/KkJs1U5wTz9RiMrV4moOG3KbN5foc57eUjqSDCZw4cQKVlX+f5jh37hyGDRuGxx9/HACwYMEC7N69G9u3b4eTkxPmzZuHyMhIHD58WKqQm72qOfEO5RyC8pwSu6fslnyUjsaYp0+fkUk4b6DxNdQ2bU7X5wyd39KUvstNkaRF0tXVVev1ihUr0KFDBwwaNAiFhYX44osvkJSUhCFDhgAAEhMTERAQgGPHjuHRRx+VImSzYGlhiYHeA7Hn3B4M9B5oEl+qhpynz5CRSThvoPE1xDatuj53TXmtxuuSMsjgqfBsMtfnDJnf0tS+y02NyQxLV1FRgU2bNiE2NhYymQyZmZlQqVQICwvTrNOlSxd4e3vj6NGjOotkeXk5ysvLNa+VSiUAQKVSQaVSGRxf1Xvr00dTYi757rqwC0/tfAoCArYWtgAAWwtb/FH8B57a8RQwHhjTeYzEUTYcc9jPCcMS8NTOpwAANhY2AO7uY831uWFroK5UQ12plizGhmQO+/h+NeVsaP4yIUT1P68ksG3bNkyZMgW5ubnw8PBAUlISnnnmGa2CBwB9+/ZFaGgoVq5cWWM/ixcvxpIlS6q1JyUlwc7OrkFiJyIi01ZaWoopU6agsLAQCoWizu8zmSPJL774AiNGjICHh0e9+omLi0NsbKzmtVKphJeXF4YPH67XhrmfSqVCamoqhg0bBisrq3rF2BSYQ74ZuRmah8uBu0cX/3r4X5h+bjr+Uv+lad89ZTcGeg+UIsQGZw77uUqluhJHLh9B0fkiOAY4or9Pf7M4/WhO+7hKTTlXnVXUl0kUycuXL+O7775DcvLfD/W6ubmhoqICBQUFcHZ21rTn5+fDzc1NZ19yuRxyubxau5WVlVH+gxirn6aiOed7o/SGVjGs8pf6L632G6U3mu02qNKc93MVK1gh2C8Ye87vQbBfcLPP937msI/vd2/OhuZuEkUyMTERbdq0wahRf/9VHxgYCCsrK6SlpSEqKgoAcOHCBeTm5iIoKEiqUE0C590zjuZ052NjMGQuw4a+I5nfA+PiNq1O8iKpVquRmJiI6OhotGjxdzhOTk6YMWMGYmNj0apVKygUCrzwwgsICgoy6ztbOe+e8TS3Ox8bkqFzGTbU/1N+D4yP27Rmko+489133yE3NxfTp0+vtuyDDz7A6NGjERUVhZCQELi5uWmdkjU3TX0MSlPDkUnqxtTmMuT3wPi4TXWTvEgOHz4cQgh06tSp2jIbGxusW7cOf/zxB0pKSpCcnFzr9cjmrDmMQWmKODJJ7UxtLkN+D4yP27R2khdJqpvmMAalqYoMiMSl+Zewe8puAHfvZs2Zn2P2BRIwvbkM+T0wPm7T2rFINhHNZQxKU1U1MgkAjkxyD1Oby5DfA+PjNq0di2QTwTsxSQqmNpchvwfGx21aOxbJJoJzxJEUTG0uQ34PjI/btHYskk0E78QkKZjaXIb8Hhgft2ntWCSbEN6JSVIwtbkM+T0wPm5T3SQfTID0w7kMSQqGzmXYUP9P+T0wPm7TmrFINkGcy5CkYMhchg35/5TfA+PjNq2uXqdbKyoqcOHCBdy5c8dY8RAREZkMg4pkaWkpZsyYATs7O3Tt2hW5ubkAgBdeeAErVqwwaoBERERSMahIxsXF4fTp0zhw4ABsbGw07WFhYdi6davRgiMiIpKSQdckU1JSsHXrVjz66KOQyf6+Zbhr1664ePGi0YIjIiKSkkFF8tatW2jTpk219pKSEq2iSUREdK+mNmelQUWyd+/e2L17N1544QUA0BTGzz//3OwnRCYiopo1xTkrDSqS77zzDkaMGIGff/4Zd+7cQUJCAn7++WccOXIEBw8eNHaMRETUxFXNWXn/lFxVc1aa6qAFBt24M3DgQJw+fRp37txBt27d8O2336JNmzY4evQoAgMDjR0jERE1YU15zkq9jyRVKhWeffZZLFq0CJ999llDxERERM2IPnNWmtpgBnofSVpZWeGrr75qiFiIiKgZaspzVhp0unXcuHFISUkxcihERNQcNeU5Kw26cadjx45YunQpDh8+jMDAQNjb22stf/HFF40SHBERNX1Vc1ZeU16r8bqkDDJ4KjxNcs5Kg4rkF198AWdnZ2RmZiIzM1NrmUwmY5EkIiKNqjkrJ2ybABlkWoXS1OesNKhI5uTkGDsOIiJqxqrmrKzpOck1EWtM8vEPwAhTZQlx9y8CjrRDRES1aYpzVho8Vda///1vdOvWDba2trC1tUX37t3x5ZdfGjM2IiJqZqrmrJzcbTIG+w426QIJGHgkuXr1aixatAjz5s3DgAEDAAAZGRl47rnn8Pvvv2PBggVGDZKIiEgKBhXJtWvXYv369Zg2bZqm7bHHHkPXrl2xePFiFkkiImoWDDrdmpeXh/79+1dr79+/P/LyTO9hUCIiIkMYVCT9/f2xbdu2au1bt25Fx44d6x0UERGRKTDodOuSJUvwxBNP4NChQ5prkocPH0ZaWlqNxZOI9NPU5twjaq4MOpKMiorC8ePH0bp1a6SkpCAlJQWtW7fGjz/+iPHjx+vV17Vr1/Dkk0/CxcUFtra26NatG06ePKlZLoTAm2++CXd3d9ja2iIsLAzZ2dmGhE3UJCSfT4Zvgi9CN4ZiSvIUhG4MhW+CL5LPJ0sdGpHZMfg5ycDAQGzatKleH/7nn39iwIABCA0Nxd69e+Hq6ors7Gy0bNlSs86qVavw4YcfYuPGjfDz88OiRYsQHh6On3/+GTY2NvX6fCJT01Tn3CNqrgwqknv27IGlpSXCw8O12vfv3w+1Wo0RI0bUqZ+VK1fCy8sLiYmJmjY/Pz/Nv4UQWLNmDf7xj39g7NixAO4+n9m2bVukpKRg0qRJhoRPZJIeNOeeDDLE7IvB2M5jeeqVqJEYVCQXLlyIFStWVGsXQmDhwoV1LpJff/01wsPD8fjjj+PgwYNo164d5syZg1mzZgG4O/zdjRs3EBYWpnmPk5MT+vXrh6NHj9ZYJMvLy1FeXq55rVQqAdydB1OlUumV572q3lufPpoSc8sXkD7njNwM3C6+DVsLW53r/F78Ow7lHMJA74FG+Uypc25s5pYvwJzvb9OXTFSNK6cHW1tbnD9/Hr6+vlrtly5dQteuXVFSUlKnfqpOl8bGxuLxxx/HiRMnMH/+fHzyySeIjo7GkSNHMGDAAFy/fh3u7n9PoTJx4kTIZDJs3bq1Wp+LFy/GkiVLqrUnJSXBzs5OjyyJiKi5KC0txZQpU1BYWAiFQlHn9xl0JOnk5IT/+7//q1Ykf/vtt2rTZtVGrVajd+/eeOeddwAAjzzyCM6dO6cpkoaIi4tDbGys5rVSqYSXlxeGDx+u14a5n0qlQmpqKoYNGwYrKyuD+2kqzC1fQPqcM3IzMCpp1APX2z1lt1GPJM1pP5tbvgBzrsq56qyivgwqkmPHjkVMTAx27tyJDh06ALhbIF966SU89thjde7H3d0dDz30kFZbQEAAvvrqKwCAm5sbACA/P1/rSDI/Px89e/assU+5XA65XF6t3crKyij/QYzVT1NhbvkC0uUc4hcCFweXB865F+IXYvRrkua2n80tX4A5G5q7QY+ArFq1Cvb29ujSpQv8/Pzg5+eHLl26wMXFBe+9916d+xkwYAAuXLig1fbrr7/Cx8cHwN2beNzc3JCWlqZZrlQqcfz4cQQFBRkSOpHJqppzD/h7jr0qpj7nHlFzZfDp1iNHjiA1NRWnT5+Gra0tevTogeBg/WaVXrBgAfr374933nkHEydOxI8//ohPP/0Un376KYC702/FxMRg2bJl6Nixo+YREA8PD4wbN86Q0IlMWlOdc4+oudKrSB49ehS3b9/G6NGjIZPJMHz4cOTl5SE+Ph6lpaUYN24c1q5dW+Ppzpr06dMHO3fuRFxcHJYuXQo/Pz+sWbMGU6dO1azz6quvoqSkBLNnz0ZBQQEGDhyIffv28RlJaraa4px7RM2VXkVy6dKlGDx4MEaPHg0AOHv2LGbNmoXo6GgEBATg3XffhYeHBxYvXlznPkePHq3pryYymQxLly7F0qVL9QmVqEmrmnOPiKSl1zXJrKwsDB06VPN6y5Yt6Nu3Lz777DPExsbiww8/5NitRETUbOhVJP/880+0bdtW8/rgwYNaAwf06dMHV65cMV50REREEtKrSLZt2xY5OTkAgIqKCpw6dQqPPvqoZnlRUVGzvMW4Ul2JjNwMAHefZatUV0ocERERNQa9iuTIkSOxcOFCpKenIy4uDnZ2dlp3tJ45c0bz3GRzUTUjQ9VD3qOSRnFGBiIiM6HXjTtvvfUWIiMjMWjQIDg4OGDjxo2wtrbWLP/Xv/6F4cOHGz1Iqdw7I8O942kae0aGhp470BT753yJxqfvNr3/DElDDFJA1NTpVSRbt26NQ4cOobCwEA4ODrC01P5Cbd++HQ4ODkYNUCqNNSND8vnkGp+JS4hIMEoBNsX+Gzomc6TvNq1a/3bxbWzuvhmjkkbBxcGF+4DoPgaNuOPk5FStQAJAq1attI4sm7L03HStXzj3ExC4oryC9Nx0gz+j6kj1/s+pOlKt7yldU+y/oWMyR/puU+4DorozqEiag7yiPKOud78HHakCQMy+GINvEjLF/hs6JnOk7zblPiDSD4ukDu6O7g9eSY/17tfQR6qm2H9jHJ2bG323KfcBkX5YJHUI9g6Gp8Kz2kDTVWSQwUvhhWBv/carrdLQR6qm2H9Dx2SO9N2m3AdE+mGR1KGhZ2Ro6CNVU+y/oWMyR/puU+4DIv2wSNaiakaGdop2Wu2eCs96P/7R0Eeqpth/Q8dkjvTdptwHRPphkXyAyIBIXJp/Cbun7AZwd1b4nPk59b5NvqGPVE2xf86XaHz6blPuAyL9sEjWgaWFJQZ6DwQADPQeaLRfIA15pGqq/Td0TOZI323KfUBUdwZNukzG09BzB5pi/5wv0fj03aZV6x/KOQTlOSV2T9nNEXeIasAiaQIaeu5AU+yf8yUan77btOoMyZ5ze4x6hoSoOeHpViIiIh1YJImIiHRgkSQiItKBRZKIiEgHFkkiIiIdWCSJiIh0YJEkIiLSgUWSiIhIBxZJIiIiHVgkiYiIdGCRJCIi0oFFkoiISAcWSSIiIh1YJImIiHSQtEguXrwYMplM66dLly6a5WVlZZg7dy5cXFzg4OCAqKgo5OfnSxgxERGZE8mPJLt27Yq8vDzNT0ZGhmbZggULsGvXLmzfvh0HDx7E9evXERnJWdOJiKhxSD7pcosWLeDm5latvbCwEF988QWSkpIwZMgQAEBiYiICAgJw7NgxPProo40dKhERmRnJi2R2djY8PDxgY2ODoKAgLF++HN7e3sjMzIRKpUJYWJhm3S5dusDb2xtHjx7VWSTLy8tRXl6uea1UKgEAKpUKKpXK4Dir3lufPpoSc8sXYM7mwNzyBZjz/W36kgkhhFGiMsDevXtRXFyMzp07Iy8vD0uWLMG1a9dw7tw57Nq1C88884xWwQOAvn37IjQ0FCtXrqyxz8WLF2PJkiXV2pOSkmBnZ9cgeRARkWkrLS3FlClTUFhYCIVCUef3SVok71dQUAAfHx+sXr0atra2BhXJmo4kvby88Pvvv+u1Ye6nUqmQmpqKYcOGwcrKyuB+mgpzyxdgzuaQs7nlCzDnqpyVSiVat26td5GU/HTrvZydndGpUyf89ttvGDZsGCoqKlBQUABnZ2fNOvn5+TVew6wil8shl8urtVtZWRnlP4ix+mkqzC1fgDmbA3PLF2DOhuYu+d2t9youLsbFixfh7u6OwMBAWFlZIS0tTbP8woULyM3NRVBQkIRREhGRuZD0SPLll1/GmDFj4OPjg+vXryM+Ph6WlpaYPHkynJycMGPGDMTGxqJVq1ZQKBR44YUXEBQUxDtbiYioUUhaJK9evYrJkyfj9u3bcHV1xcCBA3Hs2DG4uroCAD744ANYWFggKioK5eXlCA8Px8cffyxlyEREZEYkLZJbtmypdbmNjQ3WrVuHdevWNVJEREREfzOpa5JERESmhEWSiIhIBxZJIiIiHVgkiYiIdGCRJCIi0oFFkoiISAcWSSIiIh1YJImIiHRgkSQiItKBRZKIiEgHFkkiIiIdWCSJiIh0MKlJl5uLSnUl0nPTkVeUB3dHdwR7B8PSwlLqsIiImhypf5+ySBpZ8vlkzN83H1eVVzVtngpPJEQkIDIgUsLIiIiaFlP4fcrTrUaUfD4ZE7ZN0NqhAHBNeQ0Ttk1A8vlkiSIjImpaTOX3KYukkVSqKzF/33wIiGrLqtpi9sWgUl3Z2KERETUppvT7lEXSSNJz06v9xXMvAYEryitIz01vxKiIiJoeU/p9yiJpJHlFeUZdj4jIXJnS71MWSSNxd3Q36npERObKlH6fskgaSbB3MDwVnpBBVuNyGWTwUngh2Du4kSMjImpaTOn3KYukkVhaWCIhIgEAqu3YqtdrItbweUkiogcwpd+nLJJGFBkQiR0Td6Cdop1Wu6fCEzsm7uBzkkREdWQqv085mICRRQZEYmznsRxxh4ionkzh9ymLZAOwtLDEYN/BUodBRNTkSf37lKdbiYiIdGCRJCIi0oFFkoiISAcWSSIiIh1YJImIiHQwmSK5YsUKyGQyxMTEaNrKysowd+5cuLi4wMHBAVFRUcjPz5cuSCIiMismUSRPnDiBf/7zn+jevbtW+4IFC7Br1y5s374dBw8exPXr1xEZyQfyiYiocUheJIuLizF16lR89tlnaNmypaa9sLAQX3zxBVavXo0hQ4YgMDAQiYmJOHLkCI4dOyZhxEREZC4kH0xg7ty5GDVqFMLCwrBs2TJNe2ZmJlQqFcLCwjRtXbp0gbe3N44ePYpHH320xv7Ky8tRXl6uea1UKgEAKpUKKpXK4Dir3lufPpoSc8sXYM7mwNzyBZjz/W36krRIbtmyBadOncKJEyeqLbtx4wasra3h7Oys1d62bVvcuHFDZ5/Lly/HkiVLqrV/++23sLOzq3fMqamp9e6jKTG3fAHmbA7MLV+AOZeWlhrUh2RF8sqVK5g/fz5SU1NhY2NjtH7j4uIQGxurea1UKuHl5YXhw4dDoVAY3K9KpUJqaiqGDRsGKysrY4Rq0swtX4A5m0PO5pYvwJyrcq46q6gvyYpkZmYmbt68iV69emnaKisrcejQIXz00UfYv38/KioqUFBQoHU0mZ+fDzc3N539yuVyyOXyau1WVlZG+Q9irH6aCnPLF2DO5sDc8gWYs6G5S1Ykhw4dirNnz2q1PfPMM+jSpQtee+01eHl5wcrKCmlpaYiKigIAXLhwAbm5uQgKCpIiZCIiMjOSFUlHR0c8/PDDWm329vZwcXHRtM+YMQOxsbFo1aoVFAoFXnjhBQQFBem8aYeIiMiYJL+7tTYffPABLCwsEBUVhfLycoSHh+Pjjz+WOiwiIjITJlUkDxw4oPXaxsYG69atw7p166QJiIiIzJrkgwkQERGZKhZJIiIiHVgkiYiIdGCRJCIi0sGkbtwhIqKmo1JdifTcdOQV5cHd0R3B3sGwtLCUOiyjYpEkIiK9JZ9Pxvx983FVeVXT5qnwREJEAiIDms+UhjzdSkREekk+n4wJ2yZoFUgAuKa8hgnbJiD5fLJEkRkfiyQREdVZpboS8/fNh4CotqyqLWZfDCrVlY0dWoNgkSQiojpLz02vdgR5LwGBK8orSM9Nb8SoGg6LJBER1VleUZ5R1zN1LJJERFRn7o7uRl3P1LFIEhFRnQV7B8NT4QkZZDUul0EGL4UXgr2DGzmyhsEiSUREdWZpYYmEiAQAqFYoq16viVjTbJ6XZJEkIiK9RAZEYsfEHWinaKfV7qnwxI6JO5rVc5IcTICIiPQWGRCJsZ3HcsQdIiKimlhaWGKw72Cpw2hQPN1KRESkA4skERGRDiySREREOjT7a5JC3B1LUKlU1qsflUqF0tJSKJVKWFlZGSM0k2Zu+QLM2RxyNrd8AeZclXNVDaiqCXXV7ItkUVERAMDLy0viSIiISGpFRUVwcnKq8/oyoW9ZbWLUajWuX78OR0dHyGQ1jxBRF0qlEl5eXrhy5QoUCoURIzRN5pYvwJzNIWdzyxdgzlU5CyFQVFQEDw8PWFjU/Upjsz+StLCwgKenp9H6UygUZvMfDTC/fAHmbA7MLV+AOQPQ6wiyCm/cISIi0oFFkoiISAcWyTqSy+WIj4+HXC6XOpRGYW75AszZHJhbvgBzrq9mf+MOERGRoXgkSUREpAOLJBERkQ4skkRERDqwSBIREenAIlkH69atg6+vL2xsbNCvXz/8+OOPUofUYBYvXgyZTKb106VLF6nDMqpDhw5hzJgx8PDwgEwmQ0pKitZyIQTefPNNuLu7w9bWFmFhYcjOzpYmWCN4UL5PP/10tX0eEREhTbBGsnz5cvTp0weOjo5o06YNxo0bhwsXLmitU1ZWhrlz58LFxQUODg6IiopCfn6+RBHXT13yHTx4cLX9/Nxzz0kUcf2tX78e3bt31wwYEBQUhL1792qWG2v/skg+wNatWxEbG4v4+HicOnUKPXr0QHh4OG7evCl1aA2ma9euyMvL0/xkZGRIHZJRlZSUoEePHli3bl2Ny1etWoUPP/wQn3zyCY4fPw57e3uEh4ejrKyskSM1jgflCwARERFa+3zz5s2NGKHxHTx4EHPnzsWxY8eQmpoKlUqF4cOHo6SkRLPOggULsGvXLmzfvh0HDx7E9evXERkZKWHUhqtLvgAwa9Ysrf28atUqiSKuP09PT6xYsQKZmZk4efIkhgwZgrFjx+J///sfACPuX0G16tu3r5g7d67mdWVlpfDw8BDLly+XMKqGEx8fL3r06CF1GI0GgNi5c6fmtVqtFm5ubuLdd9/VtBUUFAi5XC42b94sQYTGdX++QggRHR0txo4dK0k8jeXmzZsCgDh48KAQ4u4+tbKyEtu3b9esc/78eQFAHD16VKowjeb+fIUQYtCgQWL+/PnSBdUIWrZsKT7//HOj7l8eSdaioqICmZmZCAsL07RZWFggLCwMR48elTCyhpWdnQ0PDw+0b98eU6dORW5urtQhNZqcnBzcuHFDa587OTmhX79+zXqfHzhwAG3atEHnzp3x/PPP4/bt21KHZFSFhYUAgFatWgEAMjMzoVKptPZzly5d4O3t3Sz28/35VvnPf/6D1q1b4+GHH0ZcXBxKS0ulCM/oKisrsWXLFpSUlCAoKMio+7fZD3BeH7///jsqKyvRtm1brfa2bdvil19+kSiqhtWvXz9s2LABnTt3Rl5eHpYsWYLg4GCcO3cOjo6OUofX4G7cuAEANe7zqmXNTUREBCIjI+Hn54eLFy/i9ddfx4gRI3D06FFYWlpKHV69qdVqxMTEYMCAAXj44YcB3N3P1tbWcHZ21lq3OeznmvIFgClTpsDHxwceHh44c+YMXnvtNVy4cAHJyckSRls/Z8+eRVBQEMrKyuDg4ICdO3fioYceQlZWltH2L4skaRkxYoTm3927d0e/fv3g4+ODbdu2YcaMGRJGRg1l0qRJmn9369YN3bt3R4cOHXDgwAEMHTpUwsiMY+7cuTh37lyzu7aui658Z8+erfl3t27d4O7ujqFDh+LixYvo0KFDY4dpFJ07d0ZWVhYKCwuxY8cOREdH4+DBg0b9DJ5urUXr1q1haWlZ7Y6o/Px8uLm5SRRV43J2dkanTp3w22+/SR1Ko6jar+a8z9u3b4/WrVs3i30+b948fPPNN/jhhx+0psxzc3NDRUUFCgoKtNZv6vtZV7416devHwA06f1sbW0Nf39/BAYGYvny5ejRowcSEhKMun9ZJGthbW2NwMBApKWladrUajXS0tIQFBQkYWSNp7i4GBcvXoS7u7vUoTQKPz8/uLm5ae1zpVKJ48ePm80+v3r1Km7fvt2k97kQAvPmzcPOnTvx/fffw8/PT2t5YGAgrKystPbzhQsXkJub2yT384PyrUlWVhYANOn9fD+1Wo3y8nLj7l/j3lvU/GzZskXI5XKxYcMG8fPPP4vZs2cLZ2dncePGDalDaxAvvfSSOHDggMjJyRGHDx8WYWFhonXr1uLmzZtSh2Y0RUVF4qeffhI//fSTACBWr14tfvrpJ3H58mUhhBArVqwQzs7O4r///a84c+aMGDt2rPDz8xN//fWXxJEbprZ8i4qKxMsvvyyOHj0qcnJyxHfffSd69eolOnbsKMrKyqQO3WDPP/+8cHJyEgcOHBB5eXman9LSUs06zz33nPD29hbff/+9OHnypAgKChJBQUESRm24B+X722+/iaVLl4qTJ0+KnJwc8d///le0b99ehISESBy54RYuXCgOHjwocnJyxJkzZ8TChQuFTCYT3377rRDCePuXRbIO1q5dK7y9vYW1tbXo27evOHbsmNQhNZgnnnhCuLu7C2tra9GuXTvxxBNPiN9++03qsIzqhx9+EACq/URHRwsh7j4GsmjRItG2bVshl8vF0KFDxYULF6QNuh5qy7e0tFQMHz5cuLq6CisrK+Hj4yNmzZrV5P8IrClfACIxMVGzzl9//SXmzJkjWrZsKezs7MT48eNFXl6edEHXw4Pyzc3NFSEhIaJVq1ZCLpcLf39/8corr4jCwkJpA6+H6dOnCx8fH2FtbS1cXV3F0KFDNQVSCOPtX06VRUREpAOvSRIREenAIklERKQDiyQREZEOLJJEREQ6sEgSERHpwCJJRESkA4skERGRDiySREREOrBIEpm4DRs2VJvypyl4+umnMW7cOKnDIKoXFkmiOnj66achk8k0Py4uLoiIiMCZM2f06mfx4sXo2bNnwwR5j0uXLkEmk6FNmzYoKirSWtazZ08sXry4wWMgag5YJInqKCIiAnl5ecjLy0NaWhpatGiB0aNHSx1WrYqKivDee+9JHYbRCCFw584dqcMgM8IiSVRHcrkcbm5ucHNzQ8+ePbFw4UJcuXIFt27d0qzz2muvoVOnTrCzs0P79u2xaNEiqFQqAHdPmy5ZsgSnT5/WHJFu2LABAFBQUIBnn30Wbdu2hY2NDR5++GF88803Wp+/f/9+BAQEwMHBQVOwH+SFF17A6tWrcfPmTZ3ryGQypKSkaLU5OztrYqs6Kt22bRuCg4Nha2uLPn364Ndff8WJEyfQu3dvODg4YMSIEVrbosqSJUvg6uoKhUKB5557DhUVFZplarUay5cvh5+fH2xtbdGjRw/s2LFDs/zAgQOQyWTYu3cvAgMDIZfLzWbyZDINLaQOgKgpKi4uxqZNm+Dv7w8XFxdNu6OjIzZs2AAPDw+cPXsWs2bNgqOjI1599VU88cQTOHfuHPbt24fvvvsOAODk5AS1Wo0RI0agqKgImzZtQocOHfDzzz/D0tJS029paSnee+89fPnll7CwsMCTTz6Jl19+Gf/5z39qjXPy5MlITU3F0qVL8dFHH9Ur5/j4eKxZswbe3t6YPn06pkyZAkdHRyQkJMDOzg4TJ07Em2++ifXr12vek5aWBhsbGxw4cACXLl3CM888AxcXF7z99tsAgOXLl2PTpk345JNP0LFjRxw6dAhPPvkkXF1dMWjQIE0/CxcuxHvvvYf27dujZcuW9cqDSC9Gm7eEqBmLjo4WlpaWwt7eXtjb2wsAwt3dXWRmZtb6vnfffVcEBgZqXsfHx4sePXporbN//35hYWGhczquxMREAUBryrJ169aJtm3b6vzcnJwcAUD89NNPYt++fcLKykrz/h49eoj4+HjNugDEzp07td7v5OSkmWapqq/PP/9cs3zz5s0CgEhLS9O0LV++XHTu3FnzOjo6WrRq1UqUlJRo2tavXy8cHBxEZWWlKCsrE3Z2duLIkSNanz1jxgwxefJkIcTf03ylpKTozJWoIfFIkqiOQkNDNUdJf/75Jz7++GOMGDECP/74I3x8fAAAW7duxYcffoiLFy+iuLgYd+7cgUKhqLXfrKwseHp6olOnTjrXsbOzQ4cOHTSv3d3daz2Feq/w8HAMHDgQixYtQlJSUp3eU5Pu3btr/t22bVsAQLdu3bTa7o+pR48esLOz07wOCgpCcXExrly5guLiYpSWlmLYsGFa76moqMAjjzyi1da7d2+D4yaqDxZJojqyt7eHv7+/5vXnn38OJycnfPbZZ1i2bBmOHj2KqVOnYsmSJQgPD4eTkxO2bNmC999/v9Z+bW1tH/jZVlZWWq9lMhmEHlPBrlixAkFBQXjllVeqLaupr6rrqLpikMlkNbap1eo6x1RcXAwA2L17N9q1a6e1TC6Xa722t7evc79ExsQiSWQgmUwGCwsL/PXXXwCAI0eOwMfHB2+88YZmncuXL2u9x9raGpWVlVpt3bt3x9WrV/Hrr7/WejRZH3379kVkZCQWLlxYbZmrq6vWTUDZ2dkoLS01yueePn0af/31l+YPgWPHjsHBwQFeXl5o1aoV5HI5cnNzta4/EpkSFkmiOiovL8eNGzcA3D3d+tFHH6G4uBhjxowBAHTs2BG5ubnYsmUL+vTpg927d2Pnzp1affj6+iInJ0dzitXR0RGDBg1CSEgIoqKisHr1avj7++OXX36BTCZDRESE0eJ/++230bVrV7Roof21HzJkCD766CMEBQWhsrISr732WrUjV0NVVFRgxowZ+Mc//oFLly4hPj4e8+bNg4WFBRwdHfHyyy9jwYIFUKvVGDhwIAoLC3H48GEoFApER0cbJQai+uAjIER1tG/fPri7u8Pd3R39+vXDiRMnsH37dgwePBgA8Nhjj2HBggWYN28eevbsiSNHjmDRokVafURFRSEiIgKhoaFwdXXF5s2bAQBfffUV+vTpg8mTJ+Ohhx7Cq6++Wu2Is746deqE6dOno6ysTKv9/fffh5eXF4KDgzFlyhS8/PLLWtcR62Po0KHo2LEjQkJC8MQTT+Cxxx7TGsjgrbfewqJFi7B8+XIEBAQgIiICu3fvhp+fn1E+n6i+ZEKfCxtERERmhEeSREREOrBIEhER6cAiSUREpAOLJBERkQ4skkRERDqwSBIREenAIklERKQDiyQREZEOLJJEREQ6sEgSERHpwCJJRESkw/8Dr/YMdZSGXVcAAAAASUVORK5CYII=",
"text/plain": [
"<Figure size 500x300 with 1 Axes>"
]
@@ -591,13 +5187,13 @@
"# Plotting each point\n",
"for trial_number, score, pruned in zip(trial_numbers, scores, pruning_status):\n",
" if pruned:\n",
- " plt.scatter(trial_number, score, color='grey', label='Pruned Trial' if 'Pruned Trial' not in plt.gca().get_legend_handles_labels()[1] else \"\")\n",
+ " plt.scatter(trial_number, score, color='grey', label='Pruned Batch' if 'Pruned Batch' not in plt.gca().get_legend_handles_labels()[1] else \"\")\n",
" else:\n",
- " plt.scatter(trial_number, score, color='green', label='Successful Trial' if 'Successful Trial' not in plt.gca().get_legend_handles_labels()[1] else \"\")\n",
+ " plt.scatter(trial_number, score, color='green', label='Successful Batch' if 'Successful Batch' not in plt.gca().get_legend_handles_labels()[1] else \"\")\n",
"\n",
- "plt.xlabel('Trial Number')\n",
+ "plt.xlabel('Batch Number')\n",
"plt.ylabel('Score')\n",
- "plt.title('Trial Scores')\n",
+ "plt.title('Batch Scores')\n",
"plt.grid(True)\n",
"plt.legend()\n",
"plt.show()"
@@ -627,22 +5223,22 @@
"Prompt 1 Instruction: context, question -> answer\n",
"\n",
"----------------\n",
- "Best program after 0 trials | Score: 56.5:\n",
+ "Best program after 0 batches | Score: 56.5:\n",
"Prompt 1 Instruction: Provide a yes or no answer based on the given context and question. The context and question are used as input to generate the answer.\n",
"\n",
- "Best program after 5 trials | Score: 56.5:\n",
+ "Best program after 5 batches | Score: 56.5:\n",
"Prompt 1 Instruction: Provide a yes or no answer based on the given context and question. The context and question are used as input to generate the answer.\n",
"\n",
- "Best program after 10 trials | Score: 69.5:\n",
+ "Best program after 10 batches | Score: 69.5:\n",
"Prompt 1 Instruction: Given the context and question, generate a logical answer based on the reasoning process.\n",
"\n",
- "Best program after 15 trials | Score: 69.5:\n",
+ "Best program after 15 batches | Score: 69.5:\n",
"Prompt 1 Instruction: Given the context and question, generate a logical answer based on the reasoning process.\n",
"\n",
- "Best program after 20 trials | Score: 71.0:\n",
+ "Best program after 20 batches | Score: 71.0:\n",
"Prompt 1 Instruction: Determine if it can be logically concluded that a kayak is nearby based on the given context and question, and generate a yes or no answer using the language model.\n",
"\n",
- "Best program after 25 trials | Score: 71.0:\n",
+ "Best program after 25 batches | Score: 71.0:\n",
"Prompt 1 Instruction: Determine if it can be logically concluded that a kayak is nearby based on the given context and question, and generate a yes or no answer using the language model.\n",
"\n"
]
@@ -671,7 +5267,7 @@
" best_score = program_score\n",
" best_program_so_far = compiled_program.trial_logs[trial_num][\"program\"]\n",
" if trial_num % 5 == 0:\n",
- " print(f\"Best program after {trial_num} trials | Score: {best_score}:\")\n",
+ " print(f\"Best program after {trial_num} batches | Score: {best_score}:\")\n",
" for i,predictor in enumerate(best_program_so_far.predictors()):\n",
" print(f\"Prompt {i+1} Instruction: {get_signature(predictor).instructions}\")\n",
" print()"
@@ -690,7 +5286,7 @@
},
{
"cell_type": "code",
- "execution_count": 14,
+ "execution_count": 13,
"metadata": {},
"outputs": [],
"source": [
diff --git a/examples/qa/hotpot/hotpotqa_with_MIPRO.ipynb b/examples/qa/hotpot/hotpotqa_with_MIPRO.ipynb
index 0cb44bb1e..fa6c32a15 100644
--- a/examples/qa/hotpot/hotpotqa_with_MIPRO.ipynb
+++ b/examples/qa/hotpot/hotpotqa_with_MIPRO.ipynb
@@ -27,24 +27,24 @@
"source": [
"### FAQ 🙋\n",
"#### 1) How does MIPRO work?\n",
- "At a high level, the MIPRO program optimizer works by first __proposing__ candidate fewshot example sets and instructions for each prompt in your program, and then __optimizing__ over these fewshot example sets and instructions as hyperparameters for a specified number of trials. Each trial, the optimizer evaluates different combinations of prompts on a train set, which allows it to learn which combinations yield the best performance.\n",
+ "At a high level, the MIPRO program optimizer works by first __proposing__ candidate fewshot example sets and instructions for each prompt in your program, and then __optimizing__ over these fewshot example sets and instructions as hyperparameters for a specified number of batches. Each batch, the optimizer evaluates different combinations of prompts on a subset of training inputs, which allows it to learn which combinations yield the best performance.\n",
"\n",
"#### 2) How much will MIPRO cost me to run?\n",
"Note that __this notebook__ is free to run, because all LM calls have been cached. However, when using an optimizer on your own program, here is a breakdown of the upper bound of the number of calls to the task model and prompt model respectively:\n",
"\n",
- "- **Task model calls**: MIPRO makes up to __O(TxPxM)__ task model calls if you run without minibatching, where T is the number of trials, P is the number of prompts in the program, and M is the size of the train set. This is because the model is evaluating the program on the train set each trial. In practice, this should be lower given that MIPRO tunes poor trials early (ie. it may stop a trial after running on the first 100 or so examples if performance is poor). If you run **with minibatching** you can reduce calls even further to __O(TxPxB)__ where **B** is the minibatch size. Note that every few steps (a parameter you set) MIPRO will also run a full eval over all **M** examples.\n",
+ "- **Task model calls**: MIPRO makes up to __O(TxPxM)__ task model calls if you run without minibatching, where T is the number of batches, P is the number of prompts in the program, and M is the size of the train set. This is because the model is evaluating the program on the train set each batch. If you run **with minibatching** you can reduce calls even further to __O(TxPxB)__ where **B** is the minibatch size. Note that every few steps (a parameter you set) MIPRO will also run a full eval over all **M** examples.\n",
"\n",
"- **Prompt model calls**: MIPRO makes up to N*P+10+(P+1) prompt model calls, where N is the number of instruction / fewshot example set candidates to generate for each prompt, and P is the number of prompts in the program. The extra 10 calls comes from generating a summary of the data in the training set, which we use in the meta prompt to create better instructions. The extra (P+1) comes from program summarization where the proposer LLM will look at the program code and try to describe what each module does and what the whole program does.\n",
"\n",
"#### 3) How should I configure the hyperparameters?\n",
"We have yet to run full hyperparameter sweeps with MIPRO, but based off of initial experimintation, we'd recommend the following:\n",
- "- __Trial num__: Gains can be seen after about 20-30 trials. However, 100-200 trials can help with adding on additional marginal gains.\n",
- "- __num candidates__: This hyperparameter controls the number of candidate prompts and fewshot example sets that are generated to optimize over. With more trials and less prompts to optimize, we can set n to be higher, as we have more trials to explore different combinations of prompts. If your program has between 2-3 modules and is the `num_trials=30`, we'd recommend ~`n=10`. If n is higher (say `n=100`), then we can go higher to ~`n=15`. If you have a program with only 1 module and are keeping the program 0-shot (ie. no fewshot examples), then `num_trials` should be set to equal `n`, because each trial can explore a new instruction.\n",
- "- __Training set size__: Between 200 and 500 training examples are recommended. Increasing the training set size can help prevent overfitting, but adds to the expense to run.\n",
+ "- __Batch num__: Gains can be seen after about 20-30 batches. However, 100-200 batches can help with adding on additional marginal gains.\n",
+ "- __num candidates__: This hyperparameter controls the number of candidate prompts and fewshot example sets that are generated to optimize over. With more batches and less prompts to optimize, we can set n to be higher, as we have more batches to explore different combinations of prompts. If your program has between 2-3 modules and is the `num_batches=30`, we'd recommend ~`n=10`. If n is higher (say `n=100`), then we can go higher to ~`n=15`. If you have a program with only 1 module and are keeping the program 0-shot (ie. no fewshot examples), then `num_batches` should be set to equal `n`, because each batch can explore a new instruction.\n",
+ "- __Training set size__: Between 100 and 500 training examples are recommended, however MIPROv2 can still work well with fewer. Increasing the training set size can help prevent overfitting, and only slightly increases cost for the full evaluation runs.\n",
"\n",
"#### 4) What should I do if I want to reduce the cost?\n",
- "You can always update hyperparameters accordingly, such as using a smaller train set, using less trials, or using a program with less modules. You should take advantage of minibatching.\n",
- "Alternatively, one strategy would be to optimize using a cheaper task model (ie. locally hosted Llama-3), as initial experiments have shown that prompts optimized for a smaller model also transfer to working well on a larger model.\n"
+ "You can always update hyperparameters accordingly, such as using a smaller train set, using less batches, or using a program with less modules. You should take advantage of minibatching.\n",
+ "Alternatively, one strategy would be to optimize using a cheaper task model (ie. locally hosted Llama-3), as initial experiments have shown that prompts optimized for a smaller model also transfer to working well on a larger model."
]
},
{
@@ -81,8 +81,7 @@
"output_type": "stream",
"text": [
"/opt/miniconda3/envs/opt-prompt/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
- " from .autonotebook import tqdm as notebook_tqdm\n",
- "Downloading trial_logs.pickle: 100%|██████████| 805k/805k [00:00<00:00, 29.3MB/s]\n"
+ " from .autonotebook import tqdm as notebook_tqdm\n"
]
}
],
@@ -404,8 +403,8 @@
"name": "stderr",
"output_type": "stream",
"text": [
- "Average Metric: 177 / 500 (35.4): 100%|██████████| 500/500 [00:05<00:00, 98.61it/s] \n",
- "Average Metric: 191 / 500 (38.2): 100%|██████████| 500/500 [00:05<00:00, 96.78it/s] \n"
+ "Average Metric: 177 / 500 (35.4): 100%|██████████| 500/500 [00:04<00:00, 108.21it/s]\n",
+ "Average Metric: 191 / 500 (38.2): 100%|██████████| 500/500 [00:04<00:00, 105.17it/s]\n"
]
}
],
@@ -443,8 +442,8 @@
"\n",
"MIPRO only needs a metric, DSPy module, and training set to see huge gains on your task! You can instantiate a MIPRO Optimizer and compile in just two lines:\n",
"```python\n",
- "teleprompter = MIPRO(prompt_model=prompt_model, task_model=task_model, metric=metric, num_candidates=N, init_temperature=temperature)\n",
- "compiled_program = teleprompter.compile(program, trainset=trainset, devset=devset, num_trials=trials, max_bootstrapped_demos=1,max_labeled_demos=2, eval_kwargs=eval_kwargs)\n",
+ "teleprompter = MIPROv2(prompt_model=prompt_model, task_model=task_model, metric=metric, num_candidates=N, init_temperature=temperature)\n",
+ "compiled_program = teleprompter.compile(program, trainset=trainset, valset=valset, num_batches=batches, max_bootstrapped_demos=1,max_labeled_demos=2, eval_kwargs=eval_kwargs)\n",
"```"
]
},
@@ -511,8 +510,8 @@
"name": "stderr",
"output_type": "stream",
"text": [
- "Average Metric: 232 / 500 (46.4): 100%|██████████| 500/500 [00:05<00:00, 87.66it/s] \n",
- "Average Metric: 225 / 500 (45.0): 100%|██████████| 500/500 [00:05<00:00, 97.87it/s] \n"
+ "Average Metric: 232 / 500 (46.4): 100%|██████████| 500/500 [00:05<00:00, 97.66it/s] \n",
+ "Average Metric: 225 / 500 (45.0): 100%|██████████| 500/500 [00:05<00:00, 99.59it/s] \n"
]
}
],
@@ -541,14 +540,14 @@
},
{
"cell_type": "code",
- "execution_count": 7,
+ "execution_count": 10,
"metadata": {
"id": "rtMUNeicqZH4"
},
"outputs": [
{
"data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAckAAAE8CAYAAACrYErbAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/H5lhTAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA/7ElEQVR4nO3de1xUdf4/8NcwchUGFBRQQFFRMEET00hBUhTUCgMz0928pWuLhZruapuBVt52N0VD3TUzV5dMCS0zb+sFRcEL3pNYZTFMEZOEAZHbzPn94Y/5OsIgDMOcM8zr+XjMI+Zzzpx5v8/nNG/P7XNkgiAIICIiolosxA6AiIhIqlgkiYiIdGCRJCIi0oFFkoiISAcWSSIiIh1YJImIiHRgkSQiItKBRZKIiEgHFkkiIiIdWCSJjCQ0NBShoaF6fVYmkyE+Pt6g8RDR07FIEjWQTCZr0Ovo0aOixfjrr78iNjYWvr6+sLW1Rfv27dG/f3/8+c9/RmlpqWhxEZmqVmIHQGQqtmzZovX+X//6Fw4ePFir3c/Pr87PHzhwoNliA4DffvsN/fr1g1KpxJQpU+Dr64vCwkJcunQJ69atw9tvvw17e/tmjYGopWGRJGqg3/3ud1rvMzIycPDgwVrtTyorK4OdnR2srKyaMzxs3LgReXl5OHHiBF544QWtaUqlstm//3EPHjxA69atjfZ9RM2Fh1uJDCg0NBS9evVCZmYmQkJCYGdnh/fff18z7fFzkpWVlfjwww8RGBgIR0dHtG7dGsHBwThy5Ihe352TkwO5XI7nn3++1jSFQgEbGxuttlOnTmHkyJFo06YNWrdujYCAACQkJGjNc/jwYQQHB6N169ZwcnJCZGQksrKytOaJj4+HTCbD1atXMX78eLRp0waDBg3STN+6dSsCAwNha2uLtm3bYty4cbh586bWMq5du4bo6Gi4ubnBxsYGHh4eGDduHIqLi/VaF0SGwj1JIgMrLCzEiBEjMG7cOPzud7+Dq6trnfMplUp8/vnneOONNzBt2jSUlJRg48aNCA8Px+nTp9GnT59GfW+nTp2gUqmwZcsWTJw4sd55Dx48iJdeegnu7u6IjY2Fm5sbsrKy8P333yM2NhYA8J///AcjRoxAly5dEB8fj4cPH2LNmjUYOHAgzp07h86dO2st87XXXoOPjw+WLFmCmifwffLJJ1i4cCHGjh2Lt956C7/++ivWrFmDkJAQnD9/Hk5OTqisrER4eDgqKirwzjvvwM3NDbdu3cL333+PoqIiODo6Nmo9EBmUQER6iYmJEZ78X2jw4MECAGH9+vW15h88eLAwePBgzfvq6mqhoqJCa5779+8Lrq6uwpQpU7TaAQhxcXH1xnPnzh2hXbt2AgDB19dXmDFjhpCUlCQUFRVpzVddXS14e3sLnTp1Eu7fv681Ta1Wa/7u06eP0L59e6GwsFDTdvHiRcHCwkJ48803NW1xcXECAOGNN97QWtaNGzcEuVwufPLJJ1rtly9fFlq1aqVpP3/+vABA2LFjR735EYmBh1uJDMza2hqTJ09+6nxyuVxznlCtVuO3335DdXU1+vXrh3PnzjX6e11dXXHx4kXMmDED9+/fx/r16zF+/Hi0b98eH330kWbv7vz588jNzcWsWbPg5OSktQyZTAYAyM/Px4ULFzBp0iS0bdtWMz0gIADDhg3DDz/8UOv7Z8yYofU+JSUFarUaY8eOxb179zQvNzc3+Pj4aA4r1+wp7t+/H2VlZY3Om6g5sUgSGVjHjh0bfJHM5s2bERAQABsbGzg7O6Ndu3bYs2eP3ufi3N3dsW7dOuTn5yM7OxurV69Gu3bt8OGHH2Ljxo0AHp27BIBevXrpXM7PP/8MAOjRo0etaX5+frh37x4ePHig1e7t7a31/tq1axAEAT4+PmjXrp3WKysrC3fv3tV8bs6cOfj888/h4uKC8PBwJCYm8nwkSQLPSRIZmK2tbYPm27p1KyZNmoTRo0dj3rx5aN++PeRyOZYuXaopZPqSyWTo3r07unfvjlGjRsHHxwf//ve/8dZbbzVpufV5Mm+1Wg2ZTIa9e/dCLpfXmv/x21H+/ve/Y9KkSfj2229x4MABvPvuu1i6dCkyMjLg4eHRbDETPQ2LJJFIkpOT0aVLF6SkpGgOcwJAXFycQb+nS5cuaNOmDfLz8wEAXbt2BQBcuXIFYWFhdX6mU6dOAIDs7Oxa03766Se4uLg89RaPrl27QhAEeHt7o3v37k+N09/fH/7+/vjggw9w8uRJDBw4EOvXr8fHH3/81M8SNRcebiUSSc3eVc25QuDRbRnp6el6Le/UqVO1DoECwOnTp1FYWKg5dNq3b194e3tj1apVKCoq0pq3JhZ3d3f06dMHmzdv1prnypUrOHDgAEaOHPnUeKKioiCXy7Fo0SKtHGu+p7CwEMCjq3yrq6u1pvv7+8PCwgIVFRVP/R6i5sQ9SSKRvPTSS0hJScGrr76KUaNGITc3F+vXr0fPnj31GkJuy5Yt+Pe//41XX30VgYGBsLKyQlZWFr744gvY2Nho7te0sLDAunXr8PLLL6NPnz6YPHky3N3d8dNPP+HHH3/E/v37AQB//etfMWLECAQFBWHq1KmaW0AcHR0bNI5s165d8fHHH2PBggW4ceMGRo8eDQcHB+Tm5mLnzp2YPn065s6di8OHD2PmzJl47bXX0L17d1RXV2PLli2Qy+WIjo5u9HogMiQWSSKRTJo0CXfu3ME//vEP7N+/Hz179sTWrVuxY8cOvcZ//cMf/gA7OzscOnQI3377LZRKJdq1a4fhw4djwYIFePbZZzXzhoeH48iRI1i0aBH+/ve/Q61Wo2vXrpg2bZpmnrCwMOzbtw9xcXH48MMPYWlpicGDB2P58uW1LtLRZf78+ejevTtWrlyJRYsWAQA8PT0xfPhwvPLKKwCA3r17Izw8HLt378atW7dgZ2eH3r17Y+/evXUOjEBkTDLhyeMgREREBIDnJImIiHRikSQiItKBRZKIiEgHFkkiIiIdWCSJiIh0YJEkIiLSocXfJ6lWq3H79m04ODhoDf1FRETmQxAElJSUoEOHDrCwaPj+YYsvkrdv34anp6fYYRARkQTcvHmzUYPmt/gi6eDgAODRilEoFHovp6qqCgcOHMDw4cNhaWlpqPAky9zyBZizOeRsbvkCzLkmZ6VSCU9PT01NaKgWXyRrDrEqFIomF0k7OzsoFAqz2NDMLV+AOZtDzuaWL8Ccn8y5safdeOEOERGRDiySREREOrBIEhEZiEqtwtEbR/HV5a9w9MZRqNQq0eNJy0sDAKTlpYkejykS/ZzkrVu38Oc//xl79+5FWVkZunXrhk2bNqFfv34AHl22GxcXhw0bNqCoqAgDBw7EunXr4OPjY7AYBEFAdXU1VCrdG1BVVRVatWqF8vLyeudrKcwtX6B5c5bL5WjVqhVvQ2rBUrJSELsvFr8of9G0eSg8kBCRgCi/KNHiKSwtxFcBX2FU0ig42zuLFo+pErVI3r9/HwMHDsSLL76IvXv3ol27drh27RratGmjmWfFihVYvXo1Nm/eDG9vbyxcuBDh4eG4evUqbGxsmhxDZWUl8vPzUVZWVu98giDAzc0NN2/eNIsfOnPLF2j+nO3s7ODu7g4rKyuDL5vElZKVgjHbx0CA9pMHbylvYcz2MUgem2zUwvR4PLYWtqLHY8pELZLLly+Hp6cnNm3apGl7/GGugiBg1apV+OCDDxAZGQkA+Ne//gVXV1fs2rUL48aNa9L3q9Vq5ObmQi6Xo0OHDrCystL546hWq1FaWgp7e/tG3YhqqswtX6D5chYEAZWVlfj111+Rm5sLHx8fs1mn5kClViF2X2ytAgkAAgTIIMOsfbMQ2SMScgu52cVj6kQtkt999x3Cw8Px2muvITU1FR07dsQf//hHzdPRc3NzcefOHYSFhWk+4+joiAEDBiA9Pb3OIllRUYGKigrNe6VSCeDRobSqqqpa86pUKnTs2BF2dnb1xlrzQ2dtbW0We1bmli/QvDlbW1tDLpcjLy8PZWVlsLa2Nujy9VXz/8ST/2+0VM2Rb1peGgpLC7X22J50r/QejuUewyCvQQb73obG8+R/jR2PGOrqZ337XCYIQu1/bhhJzeHSOXPm4LXXXsOZM2cQGxuL9evXY+LEiTh58iQGDhyI27dvw93dXfO5sWPHQiaT4euvv661zPj4eCxatKhWe1JSUq1C2KpVK7i5ucHT05OHwKjZVVZW4ubNm7hz5w6qq6vFDofIrJSVlWH8+PEoLi5u1D3zohZJKysr9OvXDydPntS0vfvuuzhz5gzS09P1KpJ17Ul6enri3r17tVZMeXk5bt68ic6dOz/1/GbNuH/mMgasueULNH/O5eXluHHjBjw9PQ1yPt0QqqqqcPDgQQwbNswsbjRvjnzT8tIwKmnUU+fbM36P0fYkH4/H1sIWX/T6AlOuTMFD9UOjxyOGuvpZqVTCxcWl0UVS1MOt7u7u6Nmzp1abn58fvvnmGwCAm5sbAKCgoECrSBYUFKBPnz51LtPa2rrOQ1mWlpa1/qdQqVSQyWSwsLB46jkitVoNAJr5Wzpzyxdo/pwtLCwgk8nq3BbFJsWYmpMh8w3xDoGzvTNuKW/VeR5QBhk8FB4I8Q4xyjlAXfE8VD/EQ/VDo8cjpsf7Wd/+FvXXb+DAgcjOztZq++9//4tOnToBeHQRj5ubGw4dOqSZrlQqcerUKQQFBRk1VjKeEydOwN/fH5aWlhg9enSDPhMfH6/zH076kslk2LVrV4Pnb44YSPrkFnIkRCQAeFQQH1fzflXEKqMVJKnFY+pELZKzZ89GRkYGlixZguvXryMpKQn//Oc/ERMTA+DRj9SsWbPw8ccf47vvvsPly5fx5ptvokOHDg3+8WyJfv31V7z99tvw8vKCtbU13NzcEB4ejhMnTogdmkHMmTMHffr0QW5uLr788ssmLy8+Ph4ymazeV13y8/MxYsSIJn8/tXxRflFIHpuMjoqOWu0eCg9RbreQWjymTNTDrc899xx27tyJBQsWYPHixfD29saqVaswYcIEzTx/+tOf8ODBA0yfPh1FRUUYNGgQ9u3bJ5lzOsCjS66P5x1Hfkk+3B3cEewV3Kz/SouOjkZlZSU2b96MLl26oKCgAIcOHUJhYWGzfacx5eTkYMaMGY16nE195s6dixkzZmjeP/fcc5g+fbrmKuonVVZWwsbGRnO4n6ghovyiENkj0qi/BQ2J51juMSivKLFn/B6zOMRqcEILV1xcLAAQiouLa017+PChcPXqVeHhw4dPXY5KpRLu378vqFQqrfZvrn4jeHzqISAempfHpx7CN1e/MVgOj7t//74AQDh69KjOeXJzcwUAwvnz52t97siRI5q2K1euCKNGjRIcHBwEe3t7YdCgQcL169cFQXiU75o1a4SePXsKVlZWgpubmxATE6O1vKlTpwouLi6Cg4OD8OKLLwoXLlzQTL9w4YIQGhoq2NvbCw4ODkLfvn2FM2fOCIIgCDdu3BBeeuklwcnJSbCzsxN69uwp7NmzRxP3469NmzYJmzZtEhwdHbVy3Llzp/D45hsXFyf07t27QeuwU6dOwsqVKzXvBw8eLMTExAjvvvuu0LZtWyE0NFQQBEEAIOzcuVMz35/+9CfBx8dHsLW1Fby9vYUPPvhAqKysbHAMjdnejKWyslLYtWuXVh4tmbnlKwjMuUZ9taA+5nFFRjOpGdXi8WGogP8b1SIlK8Xg32lvbw97e3vs2rVL6yrexrp16xZCQkJgbW2Nw4cPIzMzE1OmTNHcmrBu3TrMmzcP06ZNw+XLl/Hdd9+hW7dums+/9tpruHv3Lvbu3YvMzEz07dsXQ4cOxW+//QYAmDBhAjw8PHDmzBlkZmZi/vz5mhPnMTExqKiowLFjx3D58mUsX74c9vb28PT0RH5+PhQKBVatWoX8/Hy8/vrrTVhbDbd582ZYWVlh3759WLt2bZ3zODg44Msvv8TVq1eRkJCADRs2YOXKlUaJj4jEIfrYraZKrFEtWrVqhS+//BLTpk3D+vXr0bdvXwwePBjjxo1DQEBAg5eTmJgIR0dHbNu2TVO8unfvrpm+ZMkSxMTE4N1339Vc6fncc88BANLS0nD69GncvXtXcyXx3/72N+zatQvJycmYPn068vLyMG/ePPj6+gKA1li7eXl5iI6Ohr+/PwCgS5cummlubm6QyWRwdHQ06uFOHx8fLF++HEqlUufl4R988IHm786dO2Pu3LnYtm0b/vSnPxkrTCIyMu5J6ul43vFae5CPEyDgpvImjucdN/h3R0dH4/bt2/juu+8QERGBo0ePom/fvo26yOXChQsIDg6u87Lou3fv4vbt2xg8eHCdn7148SJKS0vh7Oys2bO1t7dHbm4ucnJyADy6+Oatt95CWFgYli1bpmkHHt0L+/HHH2PgwIGIi4vDpUuXGrcCmkFgYOBT5/n6668xcOBAuLm5wd7eHh988AHy8vKMEB0RiYVFUk/5JfkGna+xbGxsMGzYMCxcuBAnT57EpEmTEBcXBwCaPT/hsXEinhySydZW9xBa9U0DgNLSUri7u+PChQtar+zsbMybNw/AoytKf/zxR4waNQqHDx9Gz549sXPnTgDAW2+9hf/973/4/e9/j8uXL6Nfv35Ys2aNzu+zsLDQyqWufJqqdevW9U5PT0/HhAkTMHLkSHz//fc4f/48/vKXv6CystKgcRCRtLBI6sndwf3pMzVivqbq2bMnHjx4AABo164dgEe3MNS4cOGC1vwBAQE4fvx4ncXGwcEBnTt3Rmpqap3f1bdvX9y5cwetWrVCt27dtF4uLi6a+bp3747Zs2fjwIEDiIqK0hrI3tPTEzNmzEBKSgree+89bNiwQWdu7dq1Q0lJiSa/uvJpbidPnkSnTp3wl7/8Bf369YOPjw9+/vlno8ZARMbHIqmnYK9geCg8at2sW0MGGTwVngj2Cjbo9xYWFmLIkCHYunUrLl26hNzcXOzYsQMrVqzQPCnF1tYWzz//PJYtW4asrCykpqZqnU8DgJkzZ0KpVGLcuHE4e/Ysrl27hi1btmgGd/jwww+RmJiINWvW4Nq1azh37pxmby8sLAxBQUEYPXo0Dhw4gBs3buDkyZP4y1/+grNnz+Lhw4eYOXMmjh49ip9//hknTpzAmTNn4OfnBwCYNWsW9u/fj9zcXJw7dw5HjhzRTKvLgAEDYGdnh/fffx85OTlISkoyyP2TjeHj44O8vDxs27YNOTk5WL16tWbPmIhaLhZJPYk1qoW9vT0GDBiAlStXIiQkBL169cLChQsxbdo0fPbZZ5r5vvjiC1RXVyMwMFAzIMPjnJ2dcfjwYZSWlmLw4MEIDAzEhg0bNOcoJ06ciCVLlmDdunV45pln8NJLL+HatWuP8pPJ8MMPPyAkJASTJ09G9+7dMW7cOPz8889wdXWFXC5HYWEh3nzzTXTv3h1jx47FiBEjNAPPq1QqxMTEwM/PDxEREejevbvOK0oBoG3btti6dSt++OEH+Pv746uvvkJ8fLxB1+vTvPLKK5g9ezZmzpyJPn364OTJk1i4cKFRYyAi4xN1gHNjUCqVcHR0rHNQ2/LycuTm5sLb2/upgxOo1WrNlY+Pj+tZ19PIPRWeWBWxyqRHtdCVb0vW3Dk3ZnszlqqqKvzwww8YOXKkWYzdam75Asz58QHOddWC+vAWkCaS2igbRERkOCySBiC3kCO0c6jYYRARNYmxh9g0BSySRERU56kjD4UHEiISTPrUUVOZx8kmIiLSSYwhNk0FiyRQ60Z1oubA7Yyk6GlDbALArH2zoFKrjB2aJJh1kay56qmsrEzkSMgc1Gxn5nKFIZkGMYfYNAVmfU5SLpfDyckJd+/eBQDY2dnpfACvWq1GZWUlysvLzeKWCHPLF2i+nAVBQFlZGe7evQsnJyfI5eZ9IQRJi9hDbEqdWRdJAJonTdQUSl0EQcDDhw9ha2urs5C2JOaWL9D8OTs5OfFBziQ5UhtiU2rMvkjKZDK4u7ujffv29Q6aXVVVhWPHjiEkJMQsDpeZW75A8+ZsaWnJPUiSpJohNm8pb9V5XlIGGTwUHgYfYtNUmH2RrCGXy+v9EZPL5aiuroaNjY1ZFA1zyxcwz5yJaobYHLN9DGSQaRXK5hxi01SYx8kmIiLSKcovCsljk9FR0VGr3UPhgeSxyWZ9nyT3JImIiENs6sAiSUREADjEZl14uJWIiEgHFkkiIiIdWCSJiIh0YJEkIiLSgUWSiIhIBxZJIiIiHVgkiYiIdGCRJCIi0oFFkoiISAeOuEOiUKlVjRr+qrHzmyOprSOpxSNFXEfSxyJJRpeSlYLYfbFaT0P3UHggISKhzoGUGzu/OZLaOpJaPFLEdWQaeLiVjColKwVjto/R+mEAgFvKWxizfQxSslKaNL85kto6klo8UsR1ZDpYJMloVGoVYvfF1vlg15q2WftmQaVW6TW/OZLaOpJaPFLEdWRaWCTJaI7nHa/1L+fHCRBwU3kTx/OO6zW/OZLaOpJaPFLEdWRaWCTJaPJL8hs1X2PnN0dSW0dSi0eKuI5MC4skGY27g3uj5mvs/OZIautIavFIEdeRaWGRJKMJ9gqGh8IDMsjqnC6DDJ4KTwR7Bes1vzmS2jqSWjxSxHVkWlgkyWjkFnIkRCQAQK0fiJr3qyJWae4Ta+z85khq60hq8UgR15FpYZEko4ryi0Ly2GR0VHTUavdQeCB5bHKt+8MaO785kto6klo8UsR1ZDo4mAAZXZRfFCJ7RDZ4pJHGzm+OpLaOpBaPFHEdmQYWSRKF3EKO0M6hzTa/OZLaOpJaPFLEdSR9oh5ujY+Ph0wm03r5+vpqppeXlyMmJgbOzs6wt7dHdHQ0CgoKRIyYiIjMiejnJJ955hnk5+drXmlpaZpps2fPxu7du7Fjxw6kpqbi9u3biIrisXoiIjIO0Q+3tmrVCm5ubrXai4uLsXHjRiQlJWHIkCEAgE2bNsHPzw8ZGRl4/vnnjR0qERGZGdGL5LVr19ChQwfY2NggKCgIS5cuhZeXFzIzM1FVVYWwsDDNvL6+vvDy8kJ6errOIllRUYGKigrNe6VSCQCoqqpCVVWV3nHWfLYpyzAl5pYvwJzNgbnlCzDnJ9saSyYIQu1Rdo1k7969KC0tRY8ePZCfn49Fixbh1q1buHLlCnbv3o3JkydrFTwA6N+/P1588UUsX768zmXGx8dj0aJFtdqTkpJgZ2fXLHkQEZG0lZWVYfz48SguLoZCoWjw50Qtkk8qKipCp06d8Omnn8LW1lavIlnXnqSnpyfu3bvXqBXzpKqqKhw8eBDDhg2DpaWl3ssxFeaWL8CczSFnc8sXYM41OSuVSri4uDS6SIp+uPVxTk5O6N69O65fv45hw4ahsrISRUVFcHJy0sxTUFBQ5znMGtbW1rC2tq7VbmlpaZANxFDLMRXmli/AnM2BueULMGd9cxf96tbHlZaWIicnB+7u7ggMDISlpSUOHTqkmZ6dnY28vDwEBQWJGCUREZkLUfck586di5dffhmdOnXC7du3ERcXB7lcjjfeeAOOjo6YOnUq5syZg7Zt20KhUOCdd95BUFAQr2wlIiKjELVI/vLLL3jjjTdQWFiIdu3aYdCgQcjIyEC7du0AACtXroSFhQWio6NRUVGB8PBwrF27VsyQJUGlVnEoK2oylVqFtLxH9yWn5aUhxDuE2xHRE0Qtktu2bat3uo2NDRITE5GYmGikiKQvJSsFsftitZ5s7qHwQEJEAgdFpgar2Y4KSwvxVcBXGJU0Cs72ztyOiJ4gqXOSVL+UrBSM2T5Gq0ACwC3lLYzZPgYpWSkiRUamhNsRUcOxSJoIlVqF2H2xEFD7jp2atln7ZkGlVhk7NDIh3I6IGodF0kQczzte61/+jxMg4KbyJo7nHTdiVGRquB0RNQ6LpInIL8k36HxknrgdETUOi6SJcHdwN+h8ZJ64HRE1DoukiQj2CoaHwgMyyOqcLoMMngpPBHsFGzkyMiXcjogah0XSRMgt5EiISACAWj9wNe9XRazifW5UL25HRI3DImlCovyikDw2GR0VHbXaPRQeSB6bzPvbqEHMdTt6cvAEXsFLDSGpAc7p6aL8ohDZI5Ij7lCT1GxHx3KPQXlFiT3j97ToEXc4eALpi3uSJkhuIUdo51C84f8GQjuHttgfNmpecgs5BnkNAgAM8hrUYrcjDp5ATcEiSUQtFgdPoKZikSSiFouDJ1BTsUgSUYvFwROoqVgkiajF4uAJ1FQskkTUYnHwBGoqFkkiarE4eAI1FYskEbVo5jp4AhkGBxMgohbP3AZPIMPhniQRmQVzGTyBDIt7khKgUqs4zBxp4TZBJA0skg3w5MDIhjxMUzOm5OM3PHsoPDimpBnjNkEkHTzc+hQpWSnonNAZo5JGAQBGJY1C54TOBhnvkWNK0pO4TRBJC4tkPZrzB4tjStKTuE0QSQ+LpA7N/YPFMSXpSdwmiKSHRVKH5v7B4piS9CRuE0TSwyKpQ3P/YHFMSXoStwki6WGR1KG5f7A4piQ9idsEkfSwSOrQ3D9YHFNSWp68zUeMi2O4TRBJT5OKZGVlJbKzs1FdXW2oeCTDGD9YHFNSGprzNp/G4jZBJC16DSZQVlaGd955B5s3bwYA/Pe//0WXLl3wzjvvoGPHjpg/f75BgxRLzQ9W7L5YFJYWato9FB5YFbHKID9YNWNKcnQVcdTc5iNAgK2Fraa95jYfMQoTtwki6dCrSC5YsAAXL17E0aNHERERoWkPCwtDfHx8iymSgHEGRpZbyBHaOdRgy6OGedptPjLIMGvfLET2iDR6geI2QSQNeh1u3bVrFz777DMMGjQIMtn/HYp85plnkJOTY7DgpIIDI7dMvC+RiJ5GryL566+/on379rXaHzx4oFU0iaSM9yUS0dPoVST79euHPXv2aN7XFMbPP/8cQUFBhomMqJnxvkQiehq9zkkuWbIEI0aMwNWrV1FdXY2EhARcvXoVJ0+eRGpqqqFjJGoWNbf53FLeqvO8pAwyeCg8eF8ikRnTa09y0KBBuHjxIqqrq+Hv748DBw6gffv2SE9PR2BgoKFjJGoWvC+RiJ6m0UWyqqoKU6ZMgUwmw4YNG3D69GlcvXoVW7duhb+/f3PESNRseF8iEdWn0UXS0tIS33zzTXPEQiSKKL8o3Ii9gT3jH51n3zN+D3Jjc1kgiUi/w62jR4/Grl27DBwKkXh4mw8R1UWvC3d8fHywePFinDhxAoGBgWjdurXW9HfffdcgwREREYlJryK5ceNGODk5ITMzE5mZmVrTZDKZ2RdJlVplVkOKSTFfKcZE9ZNin0kxJjIuvYpkbm6uoePAsmXLsGDBAsTGxmLVqlUAgPLycrz33nvYtm0bKioqEB4ejrVr18LV1dXg328oKVkpiN0XqzWSi4fCAwkRCS3yHJcU85ViTFQ/KfaZFGMi42vyo7IEQYAg1L7HrDHOnDmDf/zjHwgICNBqnz17Nnbv3o0dO3YgNTUVt2/fRlSUdDfOmsGynxzqrGawbDGeKtGcpJivFGOi+kmxz6QYE4lD7yL5r3/9C/7+/rC1tYWtrS0CAgKwZcuWRi+ntLQUEyZMwIYNG9CmTRtNe3FxMTZu3IhPP/0UQ4YMQWBgIDZt2oSTJ08iIyND37CbzdMGywaAWftmifKcwuYgxXylGBPVT4p9JsWYSDx6HW799NNPsXDhQsycORMDBw4EAKSlpWHGjBm4d+8eZs+e3eBlxcTEYNSoUQgLC8PHH3+sac/MzERVVRXCwsI0bb6+vvDy8kJ6ejqef/75OpdXUVGBiooKzXulUgng0f2dVVVVjcrzcTWf1bWMtLw0FJYWaj1u6Un3Su/hWO4xzVWUUmaK+TY1pqfl3BKJnbOxt6OG5CvFbbspxO5jMdSVs775ywQ9jpV6e3tj0aJFePPNN7XaN2/ejPj4+Aafs9y2bRs++eQTnDlzBjY2NggNDUWfPn2watUqJCUlYfLkyVoFDwD69++PF198EcuXL69zmfHx8Vi0aFGt9qSkJNjZ2TUwQyIiaknKysowfvx4FBcXQ6FQNPhzeu1J5ufn44UXXqjV/sILLyA/v2FPTLh58yZiY2Nx8OBB2NjY6BNGnRYsWIA5c+Zo3iuVSnh6emL48OGNWjFPqqqqwsGDBzFs2DBYWlrWmp6Wl6Z5sn199ozfYzL/+jS1fJsa09NybonEztnY21FD8pXitt0UYvexGOrKueaoYmPpVSS7deuG7du34/3339dq//rrr+Hj49OgZWRmZuLu3bvo27evpk2lUuHYsWP47LPPsH//flRWVqKoqAhOTk6aeQoKCuDm5qZzudbW1rC2tq7VbmlpaZANRNdyQrxD4Gzv/NTBsg39wObmZkr5GiomQ20rpkSsnMXajurLV4rbtiGY+3atb+56XbizaNEifPjhh4iIiMBHH32Ejz76CBEREVi0aBEWL17coGUMHToUly9fxoULFzSvfv36YcKECZq/LS0tcejQIc1nsrOzkZeXJ8nHcZnbYNlSzFeKMVH9pNhnUoyJxKNXkYyOjsapU6fg4uKCXbt2YdeuXXBxccHp06fx6quvNmgZDg4O6NWrl9ardevWcHZ2Rq9eveDo6IipU6dizpw5OHLkCDIzMzF58mQEBQXpvGhHbOY2WLYU85ViTFQ/KfaZFGMiceh1uBUAAgMDsXXrVkPGUsvKlSthYWGB6OhorcEEpCzKLwqRPSLNZpQOKeYrxZioflLsMynGRManV5H84YcfIJfLER4ertW+f/9+qNVqjBgxQq9gjh49qvXexsYGiYmJSExM1Gt5YpFbyBHaOVTsMIxGivlKMSaqnxT7TIoxkXHpdbh1/vz5UKlq30grCALmz5/f5KCIiIikQK8iee3aNfTs2bNWu6+vL65fv97koIiIiKRAryLp6OiI//3vf7Xar1+/XuuxWURERKZKryIZGRmJWbNmIScnR9N2/fp1vPfee3jllVcMFhwREZGY9CqSK1asQOvWreHr6wtvb294e3vD19cXzs7O+Nvf/mboGImIiESh19Wtjo6OOHnyJA4ePIiLFy/C1tYWvXv3RnBwsKHjIyIiEk2j9iTT09Px/fffAwBkMhmGDx+O9u3b429/+xuio6Mxffr0WgOSExERmapGFcnFixfjxx9/1Ly/fPkypk2bhmHDhmH+/PnYvXs3li5davAgiYiIxNCoInnhwgUMHTpU837btm3o378/NmzYgDlz5mD16tXYvn27wYMkIiISQ6POSd6/fx+urq6a96mpqVqj6zz33HO4efOm4aIjUajUKqTlpQF49NggU3vagRSp1CqTH95MajlILR5qmMb2m9j93Kgi6erqitzcXHh6eqKyshLnzp3TesBxSUmJ2T2KpaVJyUpB7L5YFJYW4quArzAqaRSc7Z2REJHAQZ31VLNOf1H+omnzUHiY1DqVWg5Si4caprH9JoV+btTh1pEjR2L+/Pk4fvw4FixYADs7O60rWi9duoSuXbsaPEgyjpSsFIzZPkZrgwSAW8pbGLN9DFKyUkSKzHS1hHUqtRykFg81TGP7TSr93Kgi+dFHH6FVq1YYPHgwNmzYgA0bNsDKykoz/YsvvsDw4cMNHiQ1P5Vahdh9sXU+ZLambda+WVCpa4/ZS3VrCetUajlILR5qmMb2m5T6uVFF0sXFBceOHcP9+/dx//79Ws+O3LFjB+Li4gwaIBnH8bzjtf7F9jgBAm4qb+J43nEjRmXaWsI6lVoOUouHGqax/SalftZ7MIG6tG3btknBkHjyS/INOh+1jHUqtRykFg81TGP7TUr9rNewdNTyuDu4G3Q+ahnrVGo5SC0eapjG9puU+plFkgAAwV7B8FB4QAZZndNlkMFT4YlgLw492FAtYZ1KLQepxUMN09h+k1I/s0gSgEdPYE+ISACAWhtmzftVEat4H1ojtIR1KrUcpBYPNUxj+01K/cwiSRpRflFIHpuMjoqOWu0eCg8kj03m/Wd6aAnrVGo5SC0eapjG9ptU+lmvC3eo5Yryi0Jkj0gcyz0G5RUl9ozfwxF3mqhmnZry6DBSy0Fq8VDDNLbfpNDPLJJUi9xCjkFeg/DDlR8wyGsQf3gMQG4hR2jnULHDaBKp5SC1eKhhGttvYvczD7cSERHpwCJJRESkA4skERGRDiySREREOrBIEhER6cAiSUREpAOLJBERkQ4skkRERDqwSBIREenAIklERKQDiyQREZEOLJJEREQ6sEgSERHpwCJJRESkA4skERGRDiySREREOrBIEhER6cAiSUREpAOLJBERkQ4skkRERDqwSBIREekgapFct24dAgICoFAooFAoEBQUhL1792qml5eXIyYmBs7OzrC3t0d0dDQKCgpEjJiIiMyJqEXSw8MDy5YtQ2ZmJs6ePYshQ4YgMjISP/74IwBg9uzZ2L17N3bs2IHU1FTcvn0bUVFRYoZMRERmpJWYX/7yyy9rvf/kk0+wbt06ZGRkwMPDAxs3bkRSUhKGDBkCANi0aRP8/PyQkZGB559/XoyQiYjIjIhaJB+nUqmwY8cOPHjwAEFBQcjMzERVVRXCwsI08/j6+sLLywvp6ek6i2RFRQUqKio075VKJQCgqqoKVVVVesdX89mmLMOUmFu+AHM2B+aWL8Ccn2xrLNGL5OXLlxEUFITy8nLY29tj586d6NmzJy5cuAArKys4OTlpze/q6oo7d+7oXN7SpUuxaNGiWu0HDhyAnZ1dk+M9ePBgk5dhSswtX4A5mwNzyxdgzmVlZXotQ/Qi2aNHD1y4cAHFxcVITk7GxIkTkZqaqvfyFixYgDlz5mjeK5VKeHp6Yvjw4VAoFHovt6qqCgcPHsSwYcNgaWmp93JMhbnlCzBnc8jZ3PIFmHNNzjVHFRtL9CJpZWWFbt26AQACAwNx5swZJCQk4PXXX0dlZSWKioq09iYLCgrg5uamc3nW1tawtrau1W5paWmQDcRQyzEV5pYvwJzNgbnlCzBnfXOX3H2SarUaFRUVCAwMhKWlJQ4dOqSZlp2djby8PAQFBYkYIRERmQtR9yQXLFiAESNGwMvLCyUlJUhKSsLRo0exf/9+ODo6YurUqZgzZw7atm0LhUKBd955B0FBQbyylYiIjELUInn37l28+eabyM/Ph6OjIwICArB//34MGzYMALBy5UpYWFggOjoaFRUVCA8Px9q1a8UMmYiI/j+VWoXjeceRX5IPdwd3BHsFQ24hFzssgxK1SG7cuLHe6TY2NkhMTERiYqKRIiIiooZIyUpB7L5Y/KL8RdPmofBAQkQCovxazqAvkjsnSURE0paSlYIx28doFUgAuKW8hTHbxyAlK0WkyAyPRZKIiBpMpVYhdl8sBAi1ptW0zdo3Cyq1ytihNQsWSSIiarDjecdr7UE+ToCAm8qbOJ533IhRNR8WSSIiarD8knyDzid1LJJERNRg7g7uBp1P6lgkiYiowYK9guGh8IAMsjqnyyCDp8ITwV7BRo6sebBIEhFRg8kt5EiISACAWoWy5v2qiFUt5n5JFkkiImqUKL8oJI9NRkdFR612D4UHkscmt6j7JEUf4JyIiExPlF8UIntEcsQdIiKiusgt5AjtHCp2GM2Kh1uJiIh0YJEkIiLSgUWSiIhIBxZJIiIiHVgkiYiIdGCRJCIi0oFFkoiISAcWSSIiIh1YJImIiHTgiDtmQKVWtfiho4iImgOLZAuXkpWC2H2xWk8S91B4ICEioUUNQkxE1Bx4uLUFS8lKwZjtY7QKJADcUt7CmO1jkJKVIlJkRESmgUWyhVKpVYjdFwsBQq1pNW2z9s2CSq0ydmhERCaDRbKFOp53vNYe5OMECLipvInjeceNGBURkWlhkWyh8kvyDTofEZE5YpFsodwd3A06HxGROWKRbKGCvYLhofCADLI6p8sgg6fCE8FewUaOjIjIdLBItlByCzkSIhIAoFahrHm/KmIV75ckIqoHi2QLFuUXheSxyeio6KjV7qHwQPLYZN4nSUT0FBxMoIWL8otCZI9IjrhDRKQHFkkzILeQI7RzqNhhEBGZHB5uJSIi0oFFkoiISAcWSSIiIh1YJImIiHRgkSQiItKBRZKIiEgHFkkiIiIdWCSJiIh0YJEkIiLSgSPuEOlBpVZxqD8yOm53xifqnuTSpUvx3HPPwcHBAe3bt8fo0aORnZ2tNU95eTliYmLg7OwMe3t7REdHo6CgQKSIiYCUrBR0TuiMFze/iPEp4/Hi5hfROaEzUrJSxA6NWjBud+IQtUimpqYiJiYGGRkZOHjwIKqqqjB8+HA8ePBAM8/s2bOxe/du7NixA6mpqbh9+zaiovj0ChJHSlYKxmwfg1+Uv2i131LewpjtY/iDRc2C2514RD3cum/fPq33X375Jdq3b4/MzEyEhISguLgYGzduRFJSEoYMGQIA2LRpE/z8/JCRkYHnn39ejLDJTKnUKsTui4UAodY0AQJkkGHWvlmI7BHJQ2BkMNzuxCWpc5LFxcUAgLZt2wIAMjMzUVVVhbCwMM08vr6+8PLyQnp6ep1FsqKiAhUVFZr3SqUSAFBVVYWqqiq9Y6v5bFOWYUrMLV/g6Tmn5aWhsLQQtha2Opdxr/QejuUewyCvQc0So6GZWz+bYr5N3e5MMeemqitnffOXCYJQ+58nIlCr1XjllVdQVFSEtLQ0AEBSUhImT56sVfQAoH///njxxRexfPnyWsuJj4/HokWLarUnJSXBzs6ueYInIiJJKysrw/jx41FcXAyFQtHgz0lmTzImJgZXrlzRFEh9LViwAHPmzNG8VyqV8PT0xPDhwxu1Yp5UVVWFgwcPYtiwYbC0tGxSjKbA3PIFnp5zWl4aRiWNeupy9ozfY1J7kubUz6aYb1O3O1PMuanqyrnmqGJjSaJIzpw5E99//z2OHTsGDw8PTbubmxsqKytRVFQEJycnTXtBQQHc3NzqXJa1tTWsra1rtVtaWhpkAzHUckyFueUL6M45xDsEzvbOuKW8Vef5IRlk8FB4IMQ7xOTODZlbP5tSvoba7kwpZ0N5PGd9cxf16lZBEDBz5kzs3LkThw8fhre3t9b0wMBAWFpa4tChQ5q27Oxs5OXlISgoyNjhkpmTW8iREJEA4NEP0+Nq3q+KWGVyBZKkjduduEQtkjExMdi6dSuSkpLg4OCAO3fu4M6dO3j48CEAwNHREVOnTsWcOXNw5MgRZGZmYvLkyQgKCuKVrSSKKL8oJI9NRkdFR612D4UHkscmI8qPtyeR4XG7E4+oh1vXrVsHAAgNDdVq37RpEyZNmgQAWLlyJSwsLBAdHY2KigqEh4dj7dq1Ro6U6P9E+UUhskckRz4ho+J2Jw5Ri2RDLqy1sbFBYmIiEhMTjRARUcPILeQI7RwqdhhkZrjdGR8HOCciItKBRZKIiEgHFkkiIiIdJHGfZHOqOe+p742kNaqqqlBWVgalUmkW9xqZW74AczaHnM0tX4A5PzmYQGMHmWvxRbKkpAQA4OnpKXIkREQktpKSEjg6OjZ4fsmM3dpc1Go1bt++DQcHB8hksqd/QIea4e1u3rzZpOHtTIW55QswZ3PI2dzyBZhzTc6CIKCkpAQdOnSAhUXDzzS2+D1JCwsLraHumkqhUJjNhgaYX74AczYH5pYvwJwBNGoPsgYv3CEiItKBRZKIiEgHFskGsra2RlxcXJ1PGGmJzC1fgDmbA3PLF2DOTdXiL9whIiLSF/ckiYiIdGCRJCIi0oFFkoiISAcWSSIiIh1YJBsgMTERnTt3ho2NDQYMGIDTp0+LHVKziY+Ph0wm03r5+vqKHZZBHTt2DC+//DI6dOgAmUyGXbt2aU0XBAEffvgh3N3dYWtri7CwMFy7dk2cYA3gaflOmjSpVp9HRESIE6yBLF26FM899xwcHBzQvn17jB49GtnZ2VrzlJeXIyYmBs7OzrC3t0d0dDQKCgpEirhpGpJvaGhorX6eMWOGSBE33bp16xAQEKAZMCAoKAh79+7VTDdU/7JIPsXXX3+NOXPmIC4uDufOnUPv3r0RHh6Ou3fvih1as3nmmWeQn5+veaWlpYkdkkE9ePAAvXv31vkg7xUrVmD16tVYv349Tp06hdatWyM8PBzl5eVGjtQwnpYvAERERGj1+VdffWXECA0vNTUVMTExyMjIwMGDB1FVVYXhw4fjwYMHmnlmz56N3bt3Y8eOHUhNTcXt27cRFRUlYtT6a0i+ADBt2jStfl6xYoVIETedh4cHli1bhszMTJw9exZDhgxBZGQkfvzxRwAG7F+B6tW/f38hJiZG816lUgkdOnQQli5dKmJUzScuLk7o3bu32GEYDQBh586dmvdqtVpwc3MT/vrXv2raioqKBGtra+Grr74SIULDejJfQRCEiRMnCpGRkaLEYyx3794VAAipqamCIDzqU0tLS2HHjh2aebKysgQAQnp6ulhhGsyT+QqCIAwePFiIjY0VLygjaNOmjfD5558btH+5J1mPyspKZGZmIiwsTNNmYWGBsLAwpKenixhZ87p27Ro6dOiALl26YMKECcjLyxM7JKPJzc3FnTt3tPrc0dERAwYMaNF9fvToUbRv3x49evTA22+/jcLCQrFDMqji4mIAQNu2bQEAmZmZqKqq0upnX19feHl5tYh+fjLfGv/+97/h4uKCXr16YcGCBSgrKxMjPINTqVTYtm0bHjx4gKCgIIP2b4sf4Lwp7t27B5VKBVdXV612V1dX/PTTTyJF1bwGDBiAL7/8Ej169EB+fj4WLVqE4OBgXLlyBQ4ODmKH1+zu3LkDAHX2ec20liYiIgJRUVHw9vZGTk4O3n//fYwYMQLp6emQy+Vih9dkarUas2bNwsCBA9GrVy8Aj/rZysoKTk5OWvO2hH6uK18AGD9+PDp16oQOHTrg0qVL+POf/4zs7GykpKSIGG3TXL58GUFBQSgvL4e9vT127tyJnj174sKFCwbrXxZJ0jJixAjN3wEBARgwYAA6deqE7du3Y+rUqSJGRs1l3Lhxmr/9/f0REBCArl274ujRoxg6dKiIkRlGTEwMrly50uLOreuiK9/p06dr/vb394e7uzuGDh2KnJwcdO3a1dhhGkSPHj1w4cIFFBcXIzk5GRMnTkRqaqpBv4OHW+vh4uICuVxe64qogoICuLm5iRSVcTk5OaF79+64fv262KEYRU2/mnOfd+nSBS4uLi2iz2fOnInvv/8eR44c0XpknpubGyorK1FUVKQ1v6n3s6586zJgwAAAMOl+trKyQrdu3RAYGIilS5eid+/eSEhIMGj/skjWw8rKCoGBgTh06JCmTa1W49ChQwgKChIxMuMpLS1FTk4O3N3dxQ7FKLy9veHm5qbV50qlEqdOnTKbPv/ll19QWFho0n0uCAJmzpyJnTt34vDhw/D29taaHhgYCEtLS61+zs7ORl5enkn289PyrcuFCxcAwKT7+UlqtRoVFRWG7V/DXlvU8mzbtk2wtrYWvvzyS+Hq1avC9OnTBScnJ+HOnTtih9Ys3nvvPeHo0aNCbm6ucOLECSEsLExwcXER7t69K3ZoBlNSUiKcP39eOH/+vABA+PTTT4Xz588LP//8syAIgrBs2TLByclJ+Pbbb4VLly4JkZGRgre3t/Dw4UORI9dPffmWlJQIc+fOFdLT04Xc3FzhP//5j9C3b1/Bx8dHKC8vFzt0vb399tuCo6OjcPToUSE/P1/zKisr08wzY8YMwcvLSzh8+LBw9uxZISgoSAgKChIxav09Ld/r168LixcvFs6ePSvk5uYK3377rdClSxchJCRE5Mj1N3/+fCE1NVXIzc0VLl26JMyfP1+QyWTCgQMHBEEwXP+ySDbAmjVrBC8vL8HKykro37+/kJGRIXZIzeb1118X3N3dBSsrK6Fjx47C66+/Lly/fl3ssAzqyJEjAoBar4kTJwqC8Og2kIULFwqurq6CtbW1MHToUCE7O1vcoJugvnzLysqE4cOHC+3atRMsLS2FTp06CdOmTTP5fwTWlS8AYdOmTZp5Hj58KPzxj38U2rRpI9jZ2QmvvvqqkJ+fL17QTfC0fPPy8oSQkBChbdu2grW1tdCtWzdh3rx5QnFxsbiBN8GUKVOETp06CVZWVkK7du2EoUOHagqkIBiuf/moLCIiIh14TpKIiEgHFkkiIiIdWCSJiIh0YJEkIiLSgUWSiIhIBxZJIiIiHVgkiYiIdGCRJCIi0oFFkshI4uPj0adPn0Z9RiaTYdeuXc0SjyHcuHEDMplMMw4oUUvDIkmkB5lMVu8rPj6+1mfmzp2rNeCyIUyaNAkymQzLli3Tat+1axdkMplBv4vIHPF5kkR6yM/P1/z99ddf48MPP0R2dramzd7eXvO3IAhQqVSwt7fXajcUGxsbLF++HH/4wx/Qpk0bgy9fDJWVlbCyshI7DCLuSRLpw83NTfNydHSETCbTvP/pp5/g4OCAvXv3IjAwENbW1khLS6t1uPXMmTMYNmwYXFxc4OjoiMGDB+PcuXONjiUsLAxubm5YunSpznnqOtS7atUqdO7cWfN+0qRJGD16NJYsWQJXV1c4OTlh8eLFqK6uxrx589C2bVt4eHhg06ZNtZb/008/4YUXXoCNjQ169epV68G3V65cwYgRI2Bvbw9XV1f8/ve/x7179zTTQ0NDMXPmTMyaNQsuLi4IDw9v9Hogag4skkTNZP78+Vi2bBmysrIQEBBQa3pJSQkmTpyItLQ0ZGRkwMfHByNHjkRJSUmjvkcul2PJkiVYs2YNfvnllybFfPjwYdy+fRvHjh3Dp59+iri4OLz00kto06YNTp06hRkzZuAPf/hDre+ZN28e3nvvPZw/fx5BQUF4+eWXUVhYCAAoKirCkCFD8Oyzz+Ls2bPYt28fCgoKMHbsWK1lbN68GVZWVjhx4gTWr1/fpDyIDIVFkqiZLF68GMOGDUPXrl3Rtm3bWtOHDBmC3/3ud/D19YWfnx/++c9/oqysrNZeWEO8+uqr6NOnD+Li4poUc9u2bbF69Wr06NEDU6ZMQY8ePVBWVob3338fPj4+WLBgAaysrJCWlqb1uZkzZyI6Ohp+fn5Yt24dHB0dsXHjRgDAZ599hmeffRZLliyBr68vnn32WXzxxRc4cuQI/vvf/2qW4ePjgxUrVqBHjx7o0aNHk/IgMhQWSaJm0q9fv3qnFxQUYNq0afDx8YGjoyMUCgVKS0uRl5en1/ctX74cmzdvRlZWll6fB4BnnnkGFhb/97Pg6uoKf39/zXu5XA5nZ2fcvXtX63OPP+29VatW6NevnyaOixcv4siRI5pzsvb29vD19QUA5OTkaD4XGBiod9xEzYUX7hA1k9atW9c7feLEiSgsLERCQgI6deoEa2trBAUFobKyUq/vCwkJQXh4OBYsWIBJkyZpTbOwsMCTj46tqqqqtQxLS0ut9zKZrM42tVrd4LhKS0vx8ssvY/ny5bWmubu7a/5+2voiEgOLJJFITpw4gbVr12LkyJEAgJs3b2pdzKKPZcuWoU+fPrUOV7Zr1w537tyBIAiaW0MMeW9jRkYGQkJCAADV1dXIzMzEzJkzAQB9+/bFN998g86dO6NVK/7kkGnh4VYikfj4+GDLli3IysrCqVOnMGHCBNja2jZpmf7+/pgwYQJWr16t1R4aGopff/0VK1asQE5ODhITE7F3794mfdfjEhMTsXPnTvz000+IiYnB/fv3MWXKFABATEwMfvvtN7zxxhs4c+YMcnJysH//fkyePBkqlcpgMRA1BxZJIpFs3LgR9+/fR9++ffH73/8e7777Ltq3b9/k5S5evLjW4VA/Pz+sXbsWiYmJ6N27N06fPo25c+c2+btqLFu2DMuWLUPv3r2RlpaG7777Di4uLgCADh064MSJE1CpVBg+fDj8/f0xa9YsODk5aZ3/JJIimfDkiQoiIiICwD1JIiIinVgkiYiIdGCRJCIi0oFFkoiISAcWSSIiIh1YJImIiHRgkSQiItKBRZKIiEgHFkkiIiIdWCSJiIh0YJEkIiLS4f8BXpniEyji2cUAAAAASUVORK5CYII=",
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAckAAAE8CAYAAACrYErbAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/H5lhTAAAACXBIWXMAAA9hAAAPYQGoP6dpAABEzUlEQVR4nO3deVxU9f4/8NcMwgDCgCIKOmyKC+5XXKIERVFwSwOz1K5mLpVYqG3avV6kzeqaQWbWtS5686IpoTczMUJFSC2lcEkz9aK44HplE4Fx5vP7wx/zdYRBloFzhnk9H4951HzOmTPv9/kc583ZPkchhBAgIiKiKpRSB0BERCRXLJJEREQmsEgSERGZwCJJRERkAoskERGRCSySREREJrBIEhERmcAiSUREZAKLJBERkQkskkQWYu3atVAoFDh06JDUoRBZDRZJompUFqR7X23btkVoaCh27NhR7+W+88472Lp1q/kCraOsrCyMGjUKHTp0gL29Pby9vTFu3DgkJSVJFhORnLFIEtXgjTfewJdffol//etfePXVV3Ht2jWMHj0a3377bb2WJ2WR3Lx5M0JCQnDlyhXExMRg5cqVeOqpp3Dz5k2sWbNGkpiI5K6F1AEQydmoUaPQv39/w/uZM2eiXbt22LBhA8aOHSthZHW3dOlSdO/eHQcOHICdnZ3RtKtXrzZZHEIIlJWVwcHBocm+k6i+uCdJVAeurq5wcHBAixbGf18uX74cDz/8MNzc3ODg4IDAwEAkJycbzaNQKHDr1i2sW7fOcAj36aefNky/ePEiZs6cifbt20OlUsHPzw/PP/88KioqjJZTXl6OhQsXwt3dHS1btsRjjz2Ga9euPTD2M2fOYMCAAVUKJAC0bdvW6L1er0dCQgJ69eoFe3t7uLu7IyIiwuh86J07d/Dmm2+iU6dOUKlU8PX1xeuvv47y8nKjZfn6+mLs2LHYuXMn+vfvDwcHB3z22WcAgIKCAsyfPx9eXl5QqVTw9/fHe++9B71eb7SMjRs3IjAwEM7OzlCr1ejVqxcSEhIemDNRQ3FPkqgGhYWFuH79OoQQuHr1KlauXImSkhI89dRTRvMlJCTg0UcfxdSpU1FRUYGNGzfi8ccfx7fffosxY8YAAL788kvMmjULAwcOxJw5cwAAnTp1AgBcunQJAwcOREFBAebMmYNu3brh4sWLSE5ORmlpqVFhe+GFF9CqVSvExsbi7NmziI+Px7x58/DVV1/VmIuPjw/S09Nx4cIFaDSaGuedOXMm1q5di1GjRmHWrFm4c+cOMjMzceDAAcOe9axZs7Bu3TpMnDgRL730En766ScsW7YMJ06cwJYtW4yWd/LkSUyePBnPPvssZs+eja5du6K0tBRDhgzBxYsX8eyzz8Lb2xv79u3D4sWLkZ+fj/j4eABAWloaJk+ejOHDh+O9994DAJw4cQI//vgjYmJiasyDqMEEEVWRmJgoAFR5qVQqsXbt2irzl5aWGr2vqKgQPXv2FMOGDTNqb9mypZg+fXqVz0+bNk0olUpx8ODBKtP0er1RTGFhYYY2IYRYsGCBsLGxEQUFBTXm9MUXXwgAws7OToSGhoolS5aIzMxModPpjObbtWuXACBefPFFk7Hk5OQIAGLWrFlG019++WUBQOzatcvQ5uPjIwCI1NRUo3nffPNN0bJlS/HHH38YtS9atEjY2NiIvLw8IYQQMTExQq1Wizt37tSYH1Fj4OFWohqsWrUKaWlpSEtLw/r16xEaGopZs2YhJSXFaL57z6/dvHkThYWFCA4Oxi+//PLA79Dr9di6dSvGjRtndP6zkkKhMHo/Z84co7bg4GDodDqcO3euxu955plnkJqaiqFDhyIrKwtvvvkmgoOD0blzZ+zbt88w39dffw2FQoHY2FiTsXz33XcAgIULFxpNf+mllwAA27dvN2r38/NDeHi4UdvmzZsRHByMVq1a4fr164ZXWFgYdDod9u7dC+DuIe5bt24hLS2txvyIGgMPtxLVYODAgUaFa/LkyfjTn/6EefPmYezYsYbDoN9++y3eeust5OTkGJ2Tu7/AVefatWsoKipCz549axWTt7e30ftWrVoBuFucHyQ8PBzh4eEoLS1FdnY2vvrqK3z66acYO3Ysfv/9d7Rt2xZnzpxB+/bt0bp1a5PLOXfuHJRKJfz9/Y3aPTw84OrqWqVg+/n5VVnGqVOncOTIEbi7u1f7HZUXE82dOxebNm0y3LoycuRITJo0CREREQ/Ml6ihWCSJ6kCpVCI0NBQJCQk4deoUevTogczMTDz66KMICQnBJ598Ak9PT9ja2iIxMbFR7j+0sbGptl0IUetlODo6Ijg4GMHBwWjTpg3i4uKwY8cOTJ8+vU6x1OaPAADVXsmq1+sxYsQIvPrqq9V+pkuXLgDuXlSUk5ODnTt3YseOHdixYwcSExMxbdo0rFu3rk7xEtUViyRRHd25cwcAUFJSAuDu4Ul7e3vs3LkTKpXKMF9iYmKVz1ZXVNzd3aFWq3Hs2LFGirhmlXvK+fn5AO5eTLRz507873//M7k36ePjA71ej1OnTiEgIMDQfuXKFRQUFMDHx+eB39upUyeUlJQgLCzsgfPa2dlh3LhxGDduHPR6PebOnYvPPvsMS5YsqbI3S2ROPCdJVAdarRbff/897OzsDMXBxsYGCoUCOp3OMN/Zs2erHTSgZcuWKCgoMGpTKpWYMGECtm3bVu2Qc3XZQ6xJenp6te2V5xe7du0KAIiKioIQAnFxcSZjGT16NAAYrkCttGLFCgAwXNFbk0mTJmH//v3YuXNnlWkFBQWGP0Zu3LhhNE2pVKJ3794AUOV2EyJz454kUQ127NiB33//HcDdc2RJSUk4deoUFi1aBLVaDeBuQVixYgUiIiIwZcoUXL16FatWrYK/vz+OHDlitLzAwED88MMPWLFiBdq3bw8/Pz8MGjQI77zzDr7//nsMGTIEc+bMQUBAAPLz87F582ZkZWXB1dW1wbmMHz8efn5+GDduHDp16oRbt27hhx9+wLZt2zBgwACMGzcOABAaGoo///nP+Oijj3Dq1ClERERAr9cjMzMToaGhmDdvHvr06YPp06fjH//4BwoKCjBkyBD8/PPPWLduHSZMmIDQ0NAHxvPKK6/gm2++wdixY/H0008jMDAQt27dwtGjR5GcnIyzZ8+iTZs2mDVrFv73v/9h2LBh0Gg0OHfuHFauXIm+ffsa7cUSNQppL64lkqfqbgGxt7cXffv2FatXrza6BUOIu7dXdO7cWahUKtGtWzeRmJgoYmNjxf3/xH7//XcREhIiHBwcBACj20HOnTsnpk2bJtzd3YVKpRIdO3YU0dHRory83Cim+28T2b17twAgdu/eXWNOGzZsEE8++aTo1KmTcHBwEPb29qJ79+7iL3/5iygqKjKa986dO+Lvf/+76Natm7CzsxPu7u5i1KhRIjs72zCPVqsVcXFxws/PT9ja2govLy+xePFiUVZWZrQsHx8fMWbMmGpjKi4uFosXLxb+/v7Czs5OtGnTRjz88MNi+fLloqKiQgghRHJyshg5cqRo27atsLOzE97e3uLZZ58V+fn5NeZLZA4KIcx0LIeIiKiZ4TlJIiIiE1gkiYiITGCRJCIiMoFFkoiIyAQWSSIiIhNYJImIiExo9oMJ6PV6XLp0Cc7OzrUeZ5KIiJoXIQSKi4vRvn17KJW13z9s9kXy0qVL8PLykjoMIiKSgfPnzz/woeP3avZF0tnZGcDdFVM5jFh9VI7ZOXLkSNja2porPNmytnwB5mwNOVtbvgBzrsy5qKgIXl5ehppQW82+SFYeYlWr1Q0uko6OjlCr1VaxoVlbvgBztoacrS1fgDnfn3NdT7vxwh0iIiITWCSJiIhMYJEkIjITnV6HPWf3YMPRDdhzdg90et2DP9TI8WTlZQEAsvKyJI/HEkl+TvLixYt47bXXsGPHDpSWlsLf3x+JiYmGp6ULIRAbG4s1a9agoKAAjzzyCFavXo3OnTubLQYhBO7cuWP00Nz7abVatGjRAmVlZTXO11xYW76AvHK2tbWFjY2NpDFQ3aScSEFMagwuFF0wtGnUGiREJCAyIFKyeG6U3MCG3hswJmkM3JzcJIvHUklaJG/evIlHHnkEoaGh2LFjB9zd3XHq1Cm0atXKMM/777+Pjz76COvWrYOfnx+WLFmC8PBwHD9+HPb29g2OoaKiAvn5+SgtLa1xPiEEPDw8cP78eau439La8gXklbNCoYBGo4GTk5OkcVDtpJxIwcRNEyFg/OTBi0UXMXHTRCRPSm7SwnRvPA5KB8njsWSSFsn33nsPXl5eSExMNLT5+fkZ/l8Igfj4ePz1r3/F+PHjAQD/+te/0K5dO2zduhVPPvlkg75fr9cjNzcXNjY2aN++Pezs7Ez+OOr1epSUlMDJyalON6JaKmvLF5BPzkIIXLt2DRcuXEDnzp25RylzOr0OMakxVQokAAgIKKDA/NT5GN91PGyUjd+XcovH0klaJL/55huEh4fj8ccfR0ZGBjp06IC5c+di9uzZAIDc3FxcvnwZYWFhhs+4uLhg0KBB2L9/f7VFsry8HOXl5Yb3RUVFAO4eStNqtVXm1el06NChAxwdHWuMVQiBiooKqFQqyfcymoK15QvIK2c3NzeUlJTg9u3bUKlUjfY9lf8m7v+30Vw1Rr5ZeVm4UXLDaI/tftdLrmNv7l4M9h5stu+tbTz3/7ep45FCdf1c3z5XCCGq/rnRRCoPly5cuBCPP/44Dh48iJiYGHz66aeYPn069u3bh0ceeQSXLl2Cp6en4XOTJk2CQqHAV199VWWZS5cuRVxcXJX2pKSkKoWwRYsW8PDwgJeXF+zs7MycHVH9VVRU4Pz587h8+TLu3LkjdThEFq+0tBRTpkxBYWFhne6Zl7RI2tnZoX///ti3b5+h7cUXX8TBgwexf//+ehXJ6vYkvby8cP369SorpqysDOfPn4evr+8Dz29WjvtnLWPAWlu+gLxyLisrw9mzZ+Hl5WWWc++maLVapKWlYcSIEVZxo3lj5JuVl4UxSWMeON/2KdubbE/y3ngclA74Z89/4pljz+C2/naTxyOF6vq5qKgIbdq0qXORlPRwq6enJ7p3727UFhAQgK+//hoA4OHhAQC4cuWKUZG8cuUK+vbtW+0yVSpVtYenbG1tq/yj0Ol0UCgUUCqVDzwHpdfrAcAwf3NnbfkC8spZqVRCoVBUu902hqb6HrkwZ74hfiFwc3LDxaKL1Z4HVEABjVqDEL+QJjkHaCqe2/rbuK2/3eTxSOnefq5vf0v6S/DII4/g5MmTRm1//PEHfHx8ANy9iMfDwwPp6emG6UVFRfjpp58QFBTUpLFS0/nxxx/Rq1cv2NraYsKECbX6zNKlS03+4SQHvr6+iI+PlzoMagQ2ShskRCQAuFsQ71X5Pj4ivskKktzisXSSFskFCxbgwIEDeOedd3D69GkkJSXhH//4B6KjowHc/Yt+/vz5eOutt/DNN9/g6NGjmDZtGtq3b1/rH8/m6Nq1a3j++efh7e0NlUoFDw8PhIeH48cff5Q6NLNYuHAh+vbti9zcXKxdu9Ysyzx79iwUCoXhZWdnB39/f7z11luo6xkHhUKBrVu3miUuah4iAyKRPCkZHdQdjNo1ao0kt1vILR5LJunh1gEDBmDLli1YvHgx3njjDfj5+SE+Ph5Tp041zPPqq6/i1q1bmDNnDgoKCjB48GCkpqY26nmautLpdcjMy0R+cT48nT0R7B3cqH+lRUVFoaKiAuvWrUPHjh1x5coVpKen48aNG432nU3pzJkzeO655+r0OJva+uGHH9CjRw+Ul5cjKysLs2bNgqenJ2bOnGn27yLrEhkQifFdxzfpb0Ft4tmbuxdFx4qwfcp2qzjEanaimSssLBQARGFhYZVpt2/fFsePHxe3b99+4HJ0Op24efOm0Ol0Ru1fH/9aaFZoBJbC8NKs0Iivj39tthzudfPmTQFA7Nmzx+Q8ubm5AoD49ddfq3xu9+7dhrZjx46JMWPGCGdnZ+Hk5CQGDx4sTp8+LYS4m+/KlStF9+7dhZ2dnfDw8BDR0dFGy5s5c6Zo06aNcHZ2FqGhoSInJ8cwPScnRwwdOlQ4OTkJZ2dn0a9fP3Hw4EEhhBBnz54VY8eOFa6ursLR0VF0795dbN++3RD3va/ExESRmJgoXFxcjHLcsmWLuHfzjY2NFX369KnTOhFCiOHDh4u5c+cack5PTxfDhw8Xbm5uQq1Wi5CQEJGdnW2Y38fHxyg+Hx8fw7RvvvlG9O/fX6hUKuHm5iYmTJhg9Lm3335bzJgxQzg5OQkvLy/x2WefmYy3LttmQ1RUVIitW7eKioqKRv0eubC2fIVgzpVqqgU1sY4rMhpJ5agW9w5DBfzfqBYpJ1LM/p1OTk5wcnLC1q1bja7irauLFy8iJCQEKpUKu3btQnZ2Np555hnD7QarV6/GK6+8gtmzZ+Po0aP45ptv4O/vb/j8448/jqtXr2LHjh3Izs5Gv379MHz4cPzvf/8DAEydOhUajQYHDx5EdnY2Fi1aZDhxHh0djfLycuzduxdHjx7Fe++9BycnJ3h5eSE/Px9qtRrx8fHIz8/HE0880YC1VbNDhw4hOzsbgwYNMrSVlJRg2rRpyMrKwoEDB9C5c2eMHj0axcXFAICDBw8CABITE5Gfn294v337djz22GMYPXo0fv31V6Snp2PgwIFG3/fBBx+gf//++PXXXzF37lw8//zzVc7JE5G8SD52q6WSalSLFi1aYO3atZg9ezY+/fRT9OvXD0OGDMGTTz6J3r1713o5q1atgouLCzZu3GgoXl26dDFMf+eddxAdHY0XX3zRcKXngAEDAABZWVn4+eefcfXqVcOVxMuXL8fWrVuRnJyMOXPmIC8vD6+88gq6desGAEZj7ebl5SEqKgq9evUCAHTs2NEwzcPDAwqFAi4uLoarm83p4YcfhlKpREVFBbRaLebMmYNp06YZpoeEhECtVhty/sc//gFXV1dkZGRg7NixcHd3BwC4uroaxff222/jySefNLpHt0+fPkbfPXr0aMydOxcA8Nprr+HDDz/E7t270bVrV7PnSUTmwT3JesrMy6yyB3kvAYHzReeRmZdp9u+OiorCpUuX8M033yAiIgJ79uxBv3796nSRS05ODoKDg6u9LPrq1au4dOkShgwZUu1nDx8+jJKSEri5uRn2bJ2cnJCbm4szZ84AuHvxzaxZsxAWFoZ3333X0A7cvRf2rbfewiOPPILY2FgcOXKkbiugAb766ivk5OTg8OHD2LRpE/7zn/9g0aJFhulXr17FnDlz0LlzZ7i4uECtVqOkpAR5eXk1LjcnJwfDhw+vcZ57/4hRKBTw8PDA1atXG5YQETUqFsl6yi/ON+t8dWVvb48RI0ZgyZIl2LdvH55++mnExsYCgGEvSNxz1eb9QzI5OJgeQqumacDdQ5Kenp7Iyckxep08eRKvvPIKgLu3ZPz2228YM2YMdu3ahe7du2PLli0AgFmzZuG///0v/vznP+Po0aPo378/Vq5cafL7lEpllStQ6zvElJeXF/z9/REQEIDHH38c8+fPxwcffICysjIAwNy5c3H48GEkJCRg3759yMnJgZubGyoqKmpc7oPWGVD1Pi2FQmG4N5OI5IlFsp48nT0fPFMd5muo7t2749atWwBgOCSYn/9/BTonJ8do/t69eyMzM7PaYuPs7AxfX19kZGRU+139+vXD5cuX0aJFC/j7+xu92rRpY5ivS5cuWLBgAb7//ntERkYaDWTv5eWF5557DikpKXjppZewZs0ak7m5u7ujuLjYkF91+dSXjY0N7ty5YyiCP/30E+bNm4fRo0ejR48eUKlUuH79utFnbG1tqzxKq3fv3kb38xJR88AiWU/B3sHQqDVVbtatpIACXmovBHsHm/V7b9y4gWHDhmH9+vU4cuQIcnNzsXnzZrz//vuGJ6U4ODjgoYcewrvvvosTJ04gIyMDf/3rX42WM2/ePBQVFeHJJ5/EoUOHcOrUKXz55ZeGC0n+9re/YdWqVVi5ciVOnTqFX375xbC3FxYWhqCgIEyYMAHff/89zp49i3379uEvf/kLDh06hNu3b2PevHnYs2cPzp07hx9//BEHDx5EQEAAAGD+/PnYuXMncnNz8csvv2D37t2GadUZNGgQHB0d8frrr+PMmTNISkqq9/2TN27cwOXLl3HhwgXs2LEDCQkJCA0NNQxT1bFjR6xfvx4nTpzATz/9hKlTp1bZS/T19UV6ejouX76MmzdvAgBiY2OxYcMGxMbG4sSJE4YLkojIsrFI1pNUo1o4OTlh0KBB+PDDDxESEoKePXtiyZIlmD17Nj7++GPDfP/85z9x584dBAYGGgZkuJebmxt27dqFkpISDBkyBIGBgVizZo3hkOD06dPxzjvvYPXq1ejRowfGjh2LU6dO3c1PocB3332HkJAQzJgxA126dMGTTz6Jc+fOoV27drCxscGNGzcwbdo0dOnSBZMmTcKoUaMMF7XodDpER0cjICAAERER6NKlCz755BOTObdu3Rrr16/Hd999h169emHDhg1YunRpvdZfWFgYPD094evrizlz5mD06NFGYwCvXLkSBQUF6NevH/785z/jxRdfRNu2bY2W8cEHHyAtLQ1eXl7405/+BAAYOnQoNm/ejG+++QZ9+/bFsGHD8PPPP9crRiKSD0kHOG8KRUVFcHFxqXZQ27KyMuTm5sLPz++BgxPo9XoUFRUZXfkIVP80ci+1F+Ij4i16VAtT+TZncsq5LttmQ2i1Wnz33XcYPXq0VYzdam35Asz53gHOTdWCmvAWkAaS2ygbRERkPiySZmCjtMFQ36FSh0FE1CBNPcSmJWCRJCKiak8dadQaJEQkWPSpo4ayjpNNRERkkhRDbFoKFkmgzo9KImps3CapqTxoiE0AmJ86Hzq9rsp0a2DVRbLyqqfS0lKJIyEyVjm4gY2NdZ8PosYn5RCblsCqz0na2NjA1dXVMH6mo6MjFIrqBwfQ6/WoqKhAWVmZ5LcHNAVryxeQT856vR7Xrl2Do6MjWrSw6n+i1ASkHmJT7qz+X2DlkxweNNC0EAK3b9+Gg4ODyULanFhbvoC8clYqlfD29pY8Dmr+5DbEptxYfZFUKBTw9PRE27Ztaxw0W6vVYu/evQgJCbGKG3KtLV9AXjnb2dlZzR48SatyiM2LRRerPS+pgAIatcbsQ2xaCqsvkpVsbGxqPP9TORC2vb295D+gTcHa8gWsM2eiyiE2J26aCAUURoWyMYfYtBT8U5WIyMpFBkQieVIyOqg7GLVr1BokT0q26vskuSdJREQcYtMEFkkiIgLAITarw8OtREREJrBIEhERmcAiSUREZAKLJBERkQkskkRERCawSBIREZnAIklERGQCiyQREZEJLJJEREQmcMQdkoROr6vT8Fd1nd8ayW0dyS0eOeI6kj8WSWpyKSdSEJMaY/Q0dI1ag4SIhGoHUq7r/NZIbutIbvHIEdeRZeDhVmpSKSdSMHHTRKMfBgC4WHQREzdNRMqJlAbNb43kto7kFo8ccR1ZDhZJajI6vQ4xqTHVPti1sm1+6nzo9Lp6zW+N5LaO5BaPHHEdWRYWSWoymXmZVf5yvpeAwPmi88jMy6zX/NZIbutIbvHIEdeRZWGRpCaTX5xfp/nqOr81kts6kls8csR1ZFlYJKnJeDp71mm+us5vjeS2juQWjxxxHVkWFklqMsHewdCoNVBAUe10BRTwUnsh2Du4XvNbI7mtI7nFI0dcR5aFRZKajI3SBgkRCQBQ5Qei8n18RLzhPrG6zm+N5LaO5BaPHHEdWRYWSWpSkQGRSJ6UjA7qDkbtGrUGyZOSq9wfVtf5rZHc1pHc4pEjriPLwcEEqMlFBkRifNfxtR5ppK7zWyO5rSO5xSNHXEeWgUWSJGGjtMFQ36GNNr81kts6kls8csR1JH+SHm5dunQpFAqF0atbt26G6WVlZYiOjoabmxucnJwQFRWFK1euSBgxERFZE8nPSfbo0QP5+fmGV1ZWlmHaggULsG3bNmzevBkZGRm4dOkSIiN5rJ6IiJqG5IdbW7RoAQ8PjyrthYWF+OKLL5CUlIRhw4YBABITExEQEIADBw7goYceaupQiYjIykheJE+dOoX27dvD3t4eQUFBWLZsGby9vZGdnQ2tVouwsDDDvN26dYO3tzf2799vskiWl5ejvLzc8L6oqAgAoNVqodVq6x1n5WcbsgxLYm35AszZGlhbvgBzvr+trhRCiKqj7DaRHTt2oKSkBF27dkV+fj7i4uJw8eJFHDt2DNu2bcOMGTOMCh4ADBw4EKGhoXjvvfeqXebSpUsRFxdXpT0pKQmOjo6NkgcREclbaWkppkyZgsLCQqjV6lp/TtIieb+CggL4+PhgxYoVcHBwqFeRrG5P0svLC9evX6/TirmfVqtFWloaRowYAVtb23ovx1JYW74Ac7aGnK0tX4A5V+ZcVFSENm3a1LlISn649V6urq7o0qULTp8+jREjRqCiogIFBQVwdXU1zHPlypVqz2FWUqlUUKlUVdptbW3NsoGYazmWwtryBZizNbC2fAHmXN/cJb+69V4lJSU4c+YMPD09ERgYCFtbW6Snpxumnzx5Enl5eQgKCpIwSiIishaS7km+/PLLGDduHHx8fHDp0iXExsbCxsYGkydPhouLC2bOnImFCxeidevWUKvVeOGFFxAUFMQrW4mIqElIWiQvXLiAyZMn48aNG3B3d8fgwYNx4MABuLu7AwA+/PBDKJVKREVFoby8HOHh4fjkk0+kDFkWdHodh7KiBtPpdcjKu3tfclZeFkL8QrgdEd1H0iK5cePGGqfb29tj1apVWLVqVRNFJH8pJ1IQkxpj9GRzjVqDhIgEDopMtVa5Hd0ouYENvTdgTNIYuDm5cTsiuo+szklSzVJOpGDipolGBRIALhZdxMRNE5FyIkWiyMiScDsiqj0WSQuh0+sQkxoDgap37FS2zU+dD51e19ShkQXhdkRUNyySFiIzL7PKX/73EhA4X3QemXmZTRgVWRpuR0R1wyJpIfKL8806H1knbkdEdcMiaSE8nT3NOh9ZJ25HRHXDImkhgr2DoVFroICi2ukKKOCl9kKwd3ATR0aWhNsRUd2wSFoIG6UNEiISAKDKD1zl+/iIeN7nRjXidkRUNyySFiQyIBLJk5LRQd3BqF2j1iB5UjLvb6Nasdbt6P7BE3gFL9WGrAY4pweLDIjE+K7jOeIONUjldrQ3dy+KjhVh+5TtzXrEHQ6eQPXFPUkLZKO0wVDfoZjcazKG+g5ttj9s1LhslDYY7D0YADDYe3Cz3Y44eAI1BIskETVbHDyBGopFkoiaLQ6eQA3FIklEzRYHT6CGYpEkomaLgydQQ7FIElGzxcETqKFYJImo2eLgCdRQLJJE1KxZ6+AJZB4cTICImj1rGzyBzId7kkRkFaxl8AQyL+5JyoBOr+Mwc2SE2wSRPLBI1sL9AyOb8zBN5ZiS997wrFFrOKakFeM2QSQfPNz6ACknUuCb4IsxSWMAAGOSxsA3wdcs4z1yTEm6H7cJInlhkaxBY/5gcUxJuh+3CSL5YZE0obF/sDimJN2P2wSR/LBImtDYP1gcU5Lux22CSH5YJE1o7B8sjilJ9+M2QSQ/LJImNPYPFseUpPtxmyCSHxZJExr7B4tjSsrL/bf5SHFxDLcJIvlpUJGsqKjAyZMncefOHXPFIxtN8YPFMSXloTFv86krbhNE8lKvwQRKS0vxwgsvYN26dQCAP/74Ax07dsQLL7yADh06YNGiRWYNUiqVP1gxqTG4UXLD0K5RaxAfEW+WH6zKMSU5uoo0Km/zERBwUDoY2itv85GiMHGbIJKPehXJxYsX4/Dhw9izZw8iIiIM7WFhYVi6dGmzKZJA0wyMbKO0wVDfoWZbHtXOg27zUUCB+anzMb7r+CYvUNwmiOShXodbt27dio8//hiDBw+GQvF/hyJ79OiBM2fOmC04ueDAyM0T70skogepV5G8du0a2rZtW6X91q1bRkWTSM54XyIRPUi9imT//v2xfft2w/vKwvj5558jKCjIPJERNTLel0hED1Kvc5LvvPMORo0ahePHj+POnTtISEjA8ePHsW/fPmRkZJg7RqJGUXmbz8Wii9Wel1RAAY1aw/sSiaxYvfYkBw8ejMOHD+POnTvo1asXvv/+e7Rt2xb79+9HYGCguWMkahS8L5GIHqTORVKr1eKZZ56BQqHAmjVr8PPPP+P48eNYv349evXq1RgxEjUa3pdIRDWpc5G0tbXF119/3RixEEkiMiASZ2POYvuUu+fZt0/ZjtyYXBZIIqrf4dYJEyZg69atZg6FSDq8zYeIqlOvC3c6d+6MN954Az/++CMCAwPRsmVLo+kvvviiWYIjIiKSUr2K5BdffAFXV1dkZ2cjOzvbaJpCobD6IqnT66xqSDE55ivHmKhmcuwzOcZETateRTI3N9fcceDdd9/F4sWLERMTg/j4eABAWVkZXnrpJWzcuBHl5eUIDw/HJ598gnbt2pn9+80l5UQKYlJjjEZy0ag1SIhIaJbnuOSYrxxjoprJsc/kGBM1vQY/KksIASGq3mNWFwcPHsRnn32G3r17G7UvWLAA27Ztw+bNm5GRkYFLly4hMlK+G2flYNn3D3VWOVi2FE+VaExyzFeOMVHN5NhncoyJpFHvIvmvf/0LvXr1goODAxwcHNC7d298+eWXdV5OSUkJpk6dijVr1qBVq1aG9sLCQnzxxRdYsWIFhg0bhsDAQCQmJmLfvn04cOBAfcNuNA8aLBsA5qfOl+Q5hY1BjvnKMSaqmRz7TI4xkXTqdbh1xYoVWLJkCebNm4dHHnkEAJCVlYXnnnsO169fx4IFC2q9rOjoaIwZMwZhYWF46623DO3Z2dnQarUICwsztHXr1g3e3t7Yv38/HnrooWqXV15ejvLycsP7oqIiAHfv79RqtXXK816VnzW1jKy8LNwouWH0uKX7XS+5jr25ew1XUcqZJebb0JgelHNzJHXOTb0d1SZfOW7bDSF1H0uhupzrm79C1ONYqZ+fH+Li4jBt2jSj9nXr1mHp0qW1Pme5ceNGvP322zh48CDs7e0xdOhQ9O3bF/Hx8UhKSsKMGTOMCh4ADBw4EKGhoXjvvfeqXebSpUsRFxdXpT0pKQmOjo61zJCIiJqT0tJSTJkyBYWFhVCr1bX+XL32JPPz8/Hwww9XaX/44YeRn1+7JyacP38eMTExSEtLg729fX3CqNbixYuxcOFCw/uioiJ4eXlh5MiRdVox99NqtUhLS8OIESNga2tbZXpWXpbhyfY12T5lu8X89Wlp+TY0pgfl3BxJnXNTb0e1yVeO23ZDSN3HUqgu58qjinVVryLp7++PTZs24fXXXzdq/+qrr9C5c+daLSM7OxtXr15Fv379DG06nQ579+7Fxx9/jJ07d6KiogIFBQVwdXU1zHPlyhV4eHiYXK5KpYJKparSbmtra5YNxNRyQvxC4Obk9sDBss39wObGZkn5mismc20rlkSqnKXajmrKV47btjlY+3Zd39zrdeFOXFwc/va3vyEiIgJvvvkm3nzzTURERCAuLg5vvPFGrZYxfPhwHD16FDk5OYZX//79MXXqVMP/29raIj093fCZkydPIi8vT5aP47K2wbLlmK8cY6KaybHP5BgTSadeRTIqKgo//fQT2rRpg61bt2Lr1q1o06YNfv75Zzz22GO1WoazszN69uxp9GrZsiXc3NzQs2dPuLi4YObMmVi4cCF2796N7OxszJgxA0FBQSYv2pGatQ2WLcd85RgT1UyOfSbHmEga9TrcCgCBgYFYv369OWOp4sMPP4RSqURUVJTRYAJyFhkQifFdx1vNKB1yzFeOMVHN5NhncoyJml69iuR3330HGxsbhIeHG7Xv3LkTer0eo0aNqlcwe/bsMXpvb2+PVatWYdWqVfVanlRslDYY6jtU6jCajBzzlWNMVDM59pkcY6KmVa/DrYsWLYJOV/VGWiEEFi1a1OCgiIiI5KBeRfLUqVPo3r17lfZu3brh9OnTDQ6KiIhIDupVJF1cXPDf//63Svvp06erPDaLiIjIUtWrSI4fPx7z58/HmTNnDG2nT5/GSy+9hEcffdRswREREUmpXkXy/fffR8uWLdGtWzf4+fnBz88P3bp1g5ubG5YvX27uGImIiCRRr6tbXVxcsG/fPqSlpeHw4cNwcHBAnz59EBwcbO74iIiIJFOnPcn9+/fj22+/BQAoFAqMHDkSbdu2xfLlyxEVFYU5c+ZUGZCciIjIUtWpSL7xxhv47bffDO+PHj2K2bNnY8SIEVi0aBG2bduGZcuWmT1IIiIiKdSpSObk5GD48OGG9xs3bsTAgQOxZs0aLFy4EB999BE2bdpk9iCJiIikUKdzkjdv3kS7du0M7zMyMoxG1xkwYADOnz9vvuhIEjq9Dll5WQDuPjbI0p52IEc6vc7ihzeTWw5yi4dqp679JnU/16lItmvXDrm5ufDy8kJFRQV++eUXowccFxcXW92jWJqblBMpiEmNwY2SG9jQewPGJI2Bm5MbEiISOKhzPVWu0wtFFwxtGrXGotap3HKQWzxUO3XtNzn0c50Ot44ePRqLFi1CZmYmFi9eDEdHR6MrWo8cOYJOnTqZPUhqGiknUjBx00SjDRIALhZdxMRNE5FyIkWiyCxXc1incstBbvFQ7dS13+TSz3Uqkm+++SZatGiBIUOGYM2aNVizZg3s7OwM0//5z39i5MiRZg+SGp9Or0NMaky1D5mtbJufOh86fdUxe6l6zWGdyi0HucVDtVPXfpNTP9epSLZp0wZ79+7FzZs3cfPmzSrPjty8eTNiY2PNGiA1jcy8zCp/sd1LQOB80Xlk5mU2YVSWrTmsU7nlILd4qHbq2m9y6ud6DyZQndatWzcoGJJOfnG+Weej5rFO5ZaD3OKh2qlrv8mpn+s1LB01P57Onmadj5rHOpVbDnKLh2qnrv0mp35mkSQAQLB3MDRqDRRQVDtdAQW81F4I9ubQg7XVHNap3HKQWzxUO3XtNzn1M4skAbj7BPaEiAQAqLJhVr6Pj4jnfWh10BzWqdxykFs8VDt17Tc59TOLJBlEBkQieVIyOqg7GLVr1BokT0rm/Wf10BzWqdxykFs8VDt17Te59HO9Ltyh5isyIBLju47H3ty9KDpWhO1TtnPEnQaqXKeWPDqM3HKQWzxUO3XtNzn0M4skVWGjtMFg78H47th3GOw9mD88ZmCjtMFQ36FSh9EgcstBbvFQ7dS136TuZx5uJSIiMoFFkoiIyAQWSSIiIhNYJImIiExgkSQiIjKBRZKIiMgEFkkiIiITWCSJiIhMYJEkIiIygUWSiIjIBBZJIiIiE1gkiYiITGCRJCIiMoFFkoiIyAQWSSIiIhNYJImIiExgkSQiIjKBRZKIiMgEFkkiIiITWCSJiIhMYJEkIiIyQdIiuXr1avTu3RtqtRpqtRpBQUHYsWOHYXpZWRmio6Ph5uYGJycnREVF4cqVKxJGTERE1kTSIqnRaPDuu+8iOzsbhw4dwrBhwzB+/Hj89ttvAIAFCxZg27Zt2Lx5MzIyMnDp0iVERkZKGTIREVmRFlJ++bhx44zev/3221i9ejUOHDgAjUaDL774AklJSRg2bBgAIDExEQEBAThw4AAeeughKUImIiIrImmRvJdOp8PmzZtx69YtBAUFITs7G1qtFmFhYYZ5unXrBm9vb+zfv99kkSwvL0d5ebnhfVFREQBAq9VCq9XWO77KzzZkGZbE2vIFmLM1sLZ8AeZ8f1tdSV4kjx49iqCgIJSVlcHJyQlbtmxB9+7dkZOTAzs7O7i6uhrN365dO1y+fNnk8pYtW4a4uLgq7d9//z0cHR0bHG9aWlqDl2FJrC1fgDlbA2vLF2DOpaWl9VqG5EWya9euyMnJQWFhIZKTkzF9+nRkZGTUe3mLFy/GwoULDe+Liorg5eWFkSNHQq1W13u5Wq0WaWlpGDFiBGxtbeu9HEthbfkCzNkacra2fAHmXJlz5VHFupK8SNrZ2cHf3x8AEBgYiIMHDyIhIQFPPPEEKioqUFBQYLQ3eeXKFXh4eJhcnkqlgkqlqtJua2trlg3EXMuxFNaWL8CcrYG15Qsw5/rmLrv7JPV6PcrLyxEYGAhbW1ukp6cbpp08eRJ5eXkICgqSMEIiIrIWku5JLl68GKNGjYK3tzeKi4uRlJSEPXv2YOfOnXBxccHMmTOxcOFCtG7dGmq1Gi+88AKCgoJ4ZSsRETUJSYvk1atXMW3aNOTn58PFxQW9e/fGzp07MWLECADAhx9+CKVSiaioKJSXlyM8PByffPKJlCETEdH/p9PrkJmXifzifHg6eyLYOxg2ShupwzIrSYvkF198UeN0e3t7rFq1CqtWrWqiiIiIqDZSTqQgJjUGF4ouGNo0ag0SIhIQGdB8Bn2R3TlJIiKSt5QTKZi4aaJRgQSAi0UXMXHTRKScSJEoMvNjkSQiolrT6XWISY2BgKgyrbJtfup86PS6pg6tUbBIEhFRrWXmZVbZg7yXgMD5ovPIzMtswqgaD4skERHVWn5xvlnnkzsWSSIiqjVPZ0+zzid3LJJERFRrwd7B0Kg1UEBR7XQFFPBSeyHYO7iJI2scLJJERFRrNkobJEQkAECVQln5Pj4ivtncL8kiSUREdRIZEInkScnooO5g1K5Ra5A8KblZ3Scp+QDnRERkeSIDIjG+63iOuENERFQdG6UNhvoOlTqMRsXDrURERCawSBIREZnAIklERGQCiyQREZEJLJJEREQmsEgSERGZwCJJRERkAoskERGRCSySREREJnDEHSug0+ua/dBRRESNgUWymUs5kYKY1BijJ4lr1BokRCQ0q0GIiYgaAw+3NmMpJ1IwcdNEowIJABeLLmLipolIOZEiUWRERJaBRbKZ0ul1iEmNgYCoMq2ybX7qfOj0uqYOjYjIYrBINlOZeZlV9iDvJSBwvug8MvMymzAqIiLLwiLZTOUX55t1PiIia8Qi2Ux5OnuadT4iImvEItlMBXsHQ6PWQAFFtdMVUMBL7YVg7+AmjoyIyHKwSDZTNkobJEQkAECVQln5Pj4invdLEhHVgEWyGYsMiETypGR0UHcwateoNUielMz7JImIHoCDCTRzkQGRGN91PEfcISKqBxZJK2CjtMFQ36FSh0FEZHF4uJWIiMgEFkkiIiITWCSJiIhMYJEkIiIygUWSiIjIBBZJIiIiE1gkiYiITGCRJCIiMoFFkoiIyASOuENUDzq9jkP9UZPjdtf0JN2TXLZsGQYMGABnZ2e0bdsWEyZMwMmTJ43mKSsrQ3R0NNzc3ODk5ISoqChcuXJFooiJgJQTKfBN8EXoulBMSZmC0HWh8E3wRcqJFKlDo2aM2500JC2SGRkZiI6OxoEDB5CWlgatVouRI0fi1q1bhnkWLFiAbdu2YfPmzcjIyMClS5cQGcmnV5A0Uk6kYOKmibhQdMGo/WLRRUzcNJE/WNQouN1JR9LDrampqUbv165di7Zt2yI7OxshISEoLCzEF198gaSkJAwbNgwAkJiYiICAABw4cAAPPfSQFGGTldLpdYhJjYGAqDJNQEABBeanzsf4ruN5CIzMhtudtGR1TrKwsBAA0Lp1awBAdnY2tFotwsLCDPN069YN3t7e2L9/f7VFsry8HOXl5Yb3RUVFAACtVgutVlvv2Co/25BlWBJryxd4cM5ZeVm4UXIDDkoHk8u4XnIde3P3YrD34EaJ0dysrZ8tMd+GbneWmHNDVZdzffNXCCGq/nkiAb1ej0cffRQFBQXIysoCACQlJWHGjBlGRQ8ABg4ciNDQULz33ntVlrN06VLExcVVaU9KSoKjo2PjBE9ERLJWWlqKKVOmoLCwEGq1utafk82eZHR0NI4dO2YokPW1ePFiLFy40PC+qKgIXl5eGDlyZJ1WzP20Wi3S0tIwYsQI2NraNihGS2Bt+QIPzjkrLwtjksY8cDnbp2y3qD1Ja+pnS8y3odudJebcUNXlXHlUsa5kUSTnzZuHb7/9Fnv37oVGozG0e3h4oKKiAgUFBXB1dTW0X7lyBR4eHtUuS6VSQaVSVWm3tbU1ywZiruVYCmvLFzCdc4hfCNyc3HCx6GK154cUUECj1iDEL8Tizg1ZWz9bUr7m2u4sKWdzuTfn+uYu6dWtQgjMmzcPW7Zswa5du+Dn52c0PTAwELa2tkhPTze0nTx5Enl5eQgKCmrqcMnK2ShtkBCRAODuD9O9Kt/HR8RbXIEkeeN2Jy1Ji2R0dDTWr1+PpKQkODs74/Lly7h8+TJu374NAHBxccHMmTOxcOFC7N69G9nZ2ZgxYwaCgoJ4ZStJIjIgEsmTktFB3cGoXaPWIHlSMiIDeHsSmR+3O+lIerh19erVAIChQ4catScmJuLpp58GAHz44YdQKpWIiopCeXk5wsPD8cknnzRxpET/JzIgEuO7jufIJ9SkuN1JQ9IiWZsLa+3t7bFq1SqsWrWqCSIiqh0bpQ2G+g6VOgyyMtzumh4HOCciIjKBRZKIiMgEFkkiIiITZHGfZGOqPO9Z3xtJK2m1WpSWlqKoqMgq7jWytnwB5mwNOVtbvgBzvn8wgboOMtfsi2RxcTEAwMvLS+JIiIhIasXFxXBxcan1/LIZu7Wx6PV6XLp0Cc7OzlAoFA/+gAmVw9udP3++QcPbWQpryxdgztaQs7XlCzDnypyFECguLkb79u2hVNb+TGOz35NUKpVGQ901lFqttpoNDbC+fAHmbA2sLV+AOQOo0x5kJV64Q0REZAKLJBERkQkskrWkUqkQGxtb7RNGmiNryxdgztbA2vIFmHNDNfsLd4iIiOqLe5JEREQmsEgSERGZwCJJRERkAoskERGRCSyStbBq1Sr4+vrC3t4egwYNws8//yx1SI1m6dKlUCgURq9u3bpJHZZZ7d27F+PGjUP79u2hUCiwdetWo+lCCPztb3+Dp6cnHBwcEBYWhlOnTkkTrBk8KN+nn366Sp9HRERIE6yZLFu2DAMGDICzszPatm2LCRMm4OTJk0bzlJWVITo6Gm5ubnByckJUVBSuXLkiUcQNU5t8hw4dWqWfn3vuOYkibrjVq1ejd+/ehgEDgoKCsGPHDsN0c/Uvi+QDfPXVV1i4cCFiY2Pxyy+/oE+fPggPD8fVq1elDq3R9OjRA/n5+YZXVlaW1CGZ1a1bt9CnTx+TD/J+//338dFHH+HTTz/FTz/9hJYtWyI8PBxlZWVNHKl5PChfAIiIiDDq8w0bNjRhhOaXkZGB6OhoHDhwAGlpadBqtRg5ciRu3bplmGfBggXYtm0bNm/ejIyMDFy6dAmRkZESRl1/tckXAGbPnm3Uz++//75EETecRqPBu+++i+zsbBw6dAjDhg3D+PHj8dtvvwEwY/8KqtHAgQNFdHS04b1OpxPt27cXy5YtkzCqxhMbGyv69OkjdRhNBoDYsmWL4b1erxceHh7i73//u6GtoKBAqFQqsWHDBgkiNK/78xVCiOnTp4vx48dLEk9TuXr1qgAgMjIyhBB3+9TW1lZs3rzZMM+JEycEALF//36pwjSb+/MVQoghQ4aImJgY6YJqAq1atRKff/65WfuXe5I1qKioQHZ2NsLCwgxtSqUSYWFh2L9/v4SRNa5Tp06hffv26NixI6ZOnYq8vDypQ2oyubm5uHz5slGfu7i4YNCgQc26z/fs2YO2bduia9eueP7553Hjxg2pQzKrwsJCAEDr1q0BANnZ2dBqtUb93K1bN3h7ezeLfr4/30r//ve/0aZNG/Ts2ROLFy9GaWmpFOGZnU6nw8aNG3Hr1i0EBQWZtX+b/QDnDXH9+nXodDq0a9fOqL1du3b4/fffJYqqcQ0aNAhr165F165dkZ+fj7i4OAQHB+PYsWNwdnaWOrxGd/nyZQCots8rpzU3ERERiIyMhJ+fH86cOYPXX38do0aNwv79+2FjYyN1eA2m1+sxf/58PPLII+jZsyeAu/1sZ2cHV1dXo3mbQz9Xly8ATJkyBT4+Pmjfvj2OHDmC1157DSdPnkRKSoqE0TbM0aNHERQUhLKyMjg5OWHLli3o3r07cnJyzNa/LJJkZNSoUYb/7927NwYNGgQfHx9s2rQJM2fOlDAyaixPPvmk4f979eqF3r17o1OnTtizZw+GDx8uYWTmER0djWPHjjW7c+ummMp3zpw5hv/v1asXPD09MXz4cJw5cwadOnVq6jDNomvXrsjJyUFhYSGSk5Mxffp0ZGRkmPU7eLi1Bm3atIGNjU2VK6KuXLkCDw8PiaJqWq6urujSpQtOnz4tdShNorJfrbnPO3bsiDZt2jSLPp83bx6+/fZb7N692+iReR4eHqioqEBBQYHR/Jbez6byrc6gQYMAwKL72c7ODv7+/ggMDMSyZcvQp08fJCQkmLV/WSRrYGdnh8DAQKSnpxva9Ho90tPTERQUJGFkTaekpARnzpyBp6en1KE0CT8/P3h4eBj1eVFREX766Ser6fMLFy7gxo0bFt3nQgjMmzcPW7Zswa5du+Dn52c0PTAwELa2tkb9fPLkSeTl5VlkPz8o3+rk5OQAgEX38/30ej3Ky8vN27/mvbao+dm4caNQqVRi7dq14vjx42LOnDnC1dVVXL58WerQGsVLL70k9uzZI3Jzc8WPP/4owsLCRJs2bcTVq1elDs1siouLxa+//ip+/fVXAUCsWLFC/Prrr+LcuXNCCCHeffdd4erqKv7zn/+II0eOiPHjxws/Pz9x+/ZtiSOvn5ryLS4uFi+//LLYv3+/yM3NFT/88IPo16+f6Ny5sygrK5M69Hp7/vnnhYuLi9izZ4/Iz883vEpLSw3zPPfcc8Lb21vs2rVLHDp0SAQFBYmgoCAJo66/B+V7+vRp8cYbb4hDhw6J3Nxc8Z///Ed07NhRhISESBx5/S1atEhkZGSI3NxcceTIEbFo0SKhUCjE999/L4QwX/+ySNbCypUrhbe3t7CzsxMDBw4UBw4ckDqkRvPEE08IT09PYWdnJzp06CCeeOIJcfr0aanDMqvdu3cLAFVe06dPF0LcvQ1kyZIlol27dkKlUonhw4eLkydPSht0A9SUb2lpqRg5cqRwd3cXtra2wsfHR8yePdvi/wisLl8AIjEx0TDP7du3xdy5c0WrVq2Eo6OjeOyxx0R+fr50QTfAg/LNy8sTISEhonXr1kKlUgl/f3/xyiuviMLCQmkDb4BnnnlG+Pj4CDs7O+Hu7i6GDx9uKJBCmK9/+agsIiIiE3hOkoiIyAQWSSIiIhNYJImIiExgkSQiIjKBRZKIiMgEFkkiIiITWCSJiIhMYJEkIiIygUWSSObWrl1b5ZE/luDpp5/GhAkTpA6DqEFYJIlq4emnn4ZCoTC83NzcEBERgSNHjtRpOUuXLkXfvn0bJ8h7nD17FgqFAm3btkVxcbHRtL59+2Lp0qWNHgNRc8AiSVRLERERyM/PR35+PtLT09GiRQuMHTtW6rBqVFxcjOXLl0sdhtkIIXDnzh2pwyArwiJJVEsqlQoeHh7w8PBA3759sWjRIpw/fx7Xrl0zzPPaa6+hS5cucHR0RMeOHbFkyRJotVoAdw+bxsXF4fDhw4Y90rVr1wIACgoK8Oyzz6Jdu3awt7dHz5498e233xp9/86dOxEQEAAnJydDwX6QF154AStWrMDVq1dNzqNQKLB161ajNldXV0NslXulmzZtQnBwMBwcHDBgwAD88ccfOHjwIPr37w8nJyeMGjXKaF1UiouLg7u7O9RqNZ577jlUVFQYpun1eixbtgx+fn5wcHBAnz59kJycbJi+Z88eKBQK7NixA4GBgVCpVFbz8GSShxZSB0BkiUpKSrB+/Xr4+/vDzc3N0O7s7Iy1a9eiffv2OHr0KGbPng1nZ2e8+uqreOKJJ3Ds2DGkpqbihx9+AAC4uLhAr9dj1KhRKC4uxvr169GpUyccP34cNjY2huWWlpZi+fLl+PLLL6FUKvHUU0/h5Zdfxr///e8a45w8eTLS0tLwxhtv4OOPP25QzrGxsYiPj4e3tzeeeeYZTJkyBc7OzkhISICjoyMmTZqEv/3tb1i9erXhM+np6bC3t8eePXtw9uxZzJgxA25ubnj77bcBAMuWLcP69evx6aefonPnzti7dy+eeuopuLu7Y8iQIYblLFq0CMuXL0fHjh3RqlWrBuVBVCdme24JUTM2ffp0YWNjI1q2bClatmwpAAhPT0+RnZ1d4+f+/ve/i8DAQMP72NhY0adPH6N5du7cKZRKpcnHcSUmJgoARo8sW7VqlWjXrp3J783NzRUAxK+//ipSU1OFra2t4fN9+vQRsbGxhnkBiC1bthh93sXFxfCYpcplff7554bpGzZsEABEenq6oW3ZsmWia9euhvfTp08XrVu3Frdu3TK0rV69Wjg5OQmdTifKysqEo6Oj2Ldvn9F3z5w5U0yePFkI8X+P+dq6davJXIkaE/ckiWopNDTUsJd08+ZNfPLJJxg1ahR+/vln+Pj4AAC++uorfPTRRzhz5gxKSkpw584dqNXqGpebk5MDjUaDLl26mJzH0dERnTp1Mrz39PSs8RDqvcLDwzF48GAsWbIESUlJtfpMdXr37m34/3bt2gEAevXqZdR2f0x9+vSBo6Oj4X1QUBBKSkpw/vx5lJSUoLS0FCNGjDD6TEVFBf70pz8ZtfXv37/ecRM1BIskUS21bNkS/v7+hveff/45XFxcsGbNGrz11lvYv38/pk6diri4OISHh8PFxQUbN27EBx98UONyHRwcHvjdtra2Ru8VCgVEHR4F++677yIoKAivvPJKlWnVLavyPKqpGBQKRbVter2+1jGVlJQAALZv344OHToYTVOpVEbvW7ZsWevlEpkTiyRRPSkUCiiVSty+fRsAsG/fPvj4+OAvf/mLYZ5z584ZfcbOzg46nc6orXfv3rhw4QL++OOPGvcmG2LgwIGIjIzEokWLqkxzd3c3ugjo1KlTKC0tNcv3Hj58GLdv3zb8IXDgwAE4OTnBy8sLrVu3hkqlQl5entH5RyI5YZEkqqXy8nJcvnwZwN3DrR9//DFKSkowbtw4AEDnzp2Rl5eHjRs3YsCAAdi+fTu2bNlitAxfX1/k5uYaDrE6OztjyJAhCAkJQVRUFFasWAF/f3/8/vvvUCgUiIiIMFv8b7/9Nnr06IEWLYz/2Q8bNgwff/wxgoKCoNPp8Nprr1XZc62viooKzJw5E3/9619x9uxZxMbGYt68eVAqlXB2dsbLL7+MBQsWQK/XY/DgwSgsLMSPP/4ItVqN6dOnmyUGoobgLSBEtZSamgpPT094enpi0KBBOHjwIDZv3oyhQ4cCAB599FEsWLAA8+bNQ9++fbFv3z4sWbLEaBlRUVGIiIhAaGgo3N3dsWHDBgDA119/jQEDBmDy5Mno3r07Xn311Sp7nA3VpUsXPPPMMygrKzNq/+CDD+Dl5YXg4GBMmTIFL7/8stF5xIYYPnw4OnfujJCQEDzxxBN49NFHjQYyePPNN7FkyRIsW7YMAQEBiIiIwPbt2+Hn52eW7ydqKIWoy4kNIiIiK8I9SSIiIhNYJImIiExgkSQiIjKBRZKIiMgEFkkiIiITWCSJiIhMYJEkIiIygUWSiIjIBBZJIiIiE1gkiYiITGCRJCIiMuH/AdSyMr4sbpALAAAAAElFTkSuQmCC",
"text/plain": [
"<Figure size 500x300 with 1 Axes>"
]
@@ -573,13 +572,13 @@
"# Plotting each point\n",
"for trial_number, score, pruned in zip(trial_numbers, scores, pruning_status):\n",
" if pruned:\n",
- " plt.scatter(trial_number, score, color='grey', label='Pruned Trial' if 'Pruned Trial' not in plt.gca().get_legend_handles_labels()[1] else \"\")\n",
+ " plt.scatter(trial_number, score, color='grey', label='Pruned Batch' if 'Pruned Batch' not in plt.gca().get_legend_handles_labels()[1] else \"\")\n",
" else:\n",
- " plt.scatter(trial_number, score, color='green', label='Successful Trial' if 'Successful Trial' not in plt.gca().get_legend_handles_labels()[1] else \"\")\n",
+ " plt.scatter(trial_number, score, color='green', label='Successful Batch' if 'Successful Batch' not in plt.gca().get_legend_handles_labels()[1] else \"\")\n",
"\n",
- "plt.xlabel('Trial Number')\n",
+ "plt.xlabel('Batch Number')\n",
"plt.ylabel('Score')\n",
- "plt.title('Trial Scores')\n",
+ "plt.title('Batch Scores')\n",
"plt.grid(True)\n",
"plt.legend()\n",
"plt.show()"
@@ -611,32 +610,32 @@
"Prompt 3 Instruction: Given a question we are trying to answer and a list of passages, return a comma separated list of the numbers associated with each passage. These numbers should be ordered by helpfulness in answering the question, with most helpful passage number first, and the least helpful last.\n",
"\n",
"----------------\n",
- "Best program after 0 trials | Score: 43.8:\n",
+ "Best program after 0 batches | Score: 43.8:\n",
"Prompt 1 Instruction: Given a set of context and question pairs, prompt the model to generate a search query that accurately captures the key information needed to answer the question within the given context.\n",
"Prompt 2 Instruction: Given the provided context about the album \"Plastic Beach\" and the specific question, \"Who released the album that 'Stylo' was a single on in the United States?\", generate the accurate and precise answer to the question.\n",
"Prompt 3 Instruction: Rank the passages in order of helpfulness to answer the question \"Which genus is part of the mallow family; Ferraria or Abelmoschus?\" based on the information provided in the passages. Start by identifying the mentions of the mallow family in each passage and determine if the genus is part of the mallow family. Then, assign a number to each passage in the order of helpfulness, with the most helpful passage having the lowest number and the least helpful passage having the highest number. Separate the numbers with commas.\n",
"\n",
- "Best program after 5 trials | Score: 43.8:\n",
+ "Best program after 5 batches | Score: 43.8:\n",
"Prompt 1 Instruction: Given a set of context and question pairs, prompt the model to generate a search query that accurately captures the key information needed to answer the question within the given context.\n",
"Prompt 2 Instruction: Given the provided context about the album \"Plastic Beach\" and the specific question, \"Who released the album that 'Stylo' was a single on in the United States?\", generate the accurate and precise answer to the question.\n",
"Prompt 3 Instruction: Rank the passages in order of helpfulness to answer the question \"Which genus is part of the mallow family; Ferraria or Abelmoschus?\" based on the information provided in the passages. Start by identifying the mentions of the mallow family in each passage and determine if the genus is part of the mallow family. Then, assign a number to each passage in the order of helpfulness, with the most helpful passage having the lowest number and the least helpful passage having the highest number. Separate the numbers with commas.\n",
"\n",
- "Best program after 10 trials | Score: 43.8:\n",
+ "Best program after 10 batches | Score: 43.8:\n",
"Prompt 1 Instruction: Given a set of context and question pairs, prompt the model to generate a search query that accurately captures the key information needed to answer the question within the given context.\n",
"Prompt 2 Instruction: Given the provided context about the album \"Plastic Beach\" and the specific question, \"Who released the album that 'Stylo' was a single on in the United States?\", generate the accurate and precise answer to the question.\n",
"Prompt 3 Instruction: Rank the passages in order of helpfulness to answer the question \"Which genus is part of the mallow family; Ferraria or Abelmoschus?\" based on the information provided in the passages. Start by identifying the mentions of the mallow family in each passage and determine if the genus is part of the mallow family. Then, assign a number to each passage in the order of helpfulness, with the most helpful passage having the lowest number and the least helpful passage having the highest number. Separate the numbers with commas.\n",
"\n",
- "Best program after 15 trials | Score: 43.8:\n",
+ "Best program after 15 batches | Score: 43.8:\n",
"Prompt 1 Instruction: Given a set of context and question pairs, prompt the model to generate a search query that accurately captures the key information needed to answer the question within the given context.\n",
"Prompt 2 Instruction: Given the provided context about the album \"Plastic Beach\" and the specific question, \"Who released the album that 'Stylo' was a single on in the United States?\", generate the accurate and precise answer to the question.\n",
"Prompt 3 Instruction: Rank the passages in order of helpfulness to answer the question \"Which genus is part of the mallow family; Ferraria or Abelmoschus?\" based on the information provided in the passages. Start by identifying the mentions of the mallow family in each passage and determine if the genus is part of the mallow family. Then, assign a number to each passage in the order of helpfulness, with the most helpful passage having the lowest number and the least helpful passage having the highest number. Separate the numbers with commas.\n",
"\n",
- "Best program after 20 trials | Score: 46.4:\n",
+ "Best program after 20 batches | Score: 46.4:\n",
"Prompt 1 Instruction: Given the provided context and question, generate a search query that will yield the specific information needed to answer the question accurately and directly.\n",
"Prompt 2 Instruction: Given the provided context about various film directors and their nominations for the Academy Award Oscar, and the specific question asking which director received more nominations, please generate the correct answer based on the information provided.\n",
"Prompt 3 Instruction: Rank the passages in order of helpfulness to answer the question \"Which genus is part of the mallow family; Ferraria or Abelmoschus?\" based on the information provided in the passages. Start by identifying the mentions of the mallow family in each passage and determine if the genus is part of the mallow family. Then, assign a number to each passage in the order of helpfulness, with the most helpful passage having the lowest number and the least helpful passage having the highest number. Separate the numbers with commas.\n",
"\n",
- "Best program after 25 trials | Score: 46.4:\n",
+ "Best program after 25 batches | Score: 46.4:\n",
"Prompt 1 Instruction: Given the provided context and question, generate a search query that will yield the specific information needed to answer the question accurately and directly.\n",
"Prompt 2 Instruction: Given the provided context about various film directors and their nominations for the Academy Award Oscar, and the specific question asking which director received more nominations, please generate the correct answer based on the information provided.\n",
"Prompt 3 Instruction: Rank the passages in order of helpfulness to answer the question \"Which genus is part of the mallow family; Ferraria or Abelmoschus?\" based on the information provided in the passages. Start by identifying the mentions of the mallow family in each passage and determine if the genus is part of the mallow family. Then, assign a number to each passage in the order of helpfulness, with the most helpful passage having the lowest number and the least helpful passage having the highest number. Separate the numbers with commas.\n",
@@ -667,7 +666,7 @@
" best_score = program_score\n",
" best_program_so_far = compiled_program.trial_logs[trial_num][\"program\"]\n",
" if trial_num % 5 == 0:\n",
- " print(f\"Best program after {trial_num} trials | Score: {best_score}:\")\n",
+ " print(f\"Best program after {trial_num} batches | Score: {best_score}:\")\n",
" for i,predictor in enumerate(best_program_so_far.predictors()):\n",
" print(f\"Prompt {i+1} Instruction: {get_signature(predictor).instructions}\")\n",
" print()"