Skip to content

armgit / dspy

Public Imported from GitHub
0

DSPy: The framework for programming—not prompting—language models

Python

Readying for release. Working notebook without cache yet

Omar Khattabdc6e1843e096
diff --git a/.gitignore b/.gitignore
index 08d3e2f50..40e8231be 100644
--- a/.gitignore
+++ b/.gitignore
@@ -33,4 +33,5 @@ build/
*.json
!/data/*.json
/dist/
-**/*.pkl
\ No newline at end of file
+**/*.pkl
+checklist.md
\ No newline at end of file
diff --git a/README.md b/README.md
index b8b286c77..d1de675a9 100644
--- a/README.md
+++ b/README.md
@@ -1,69 +1,40 @@
-### Note:
-
-The DSP framework will rebrand to **DSPy** soon as we release a completely redesigned **v2** of our framework.
-
-The **DSPy** (i.e., **v2**) release will start today Aug 14th. DSPv1 will remain supported while we transition completely.
-
<p align="center">
- <img align="center" src="docs/images/DSPy7.png" width="260px" />
+ <img align="center" src="docs/images/DSPy_whitebg.png" width="460px" />
</p>
<p align="left">
+# DSPy: The Framework for Programming with Foundation Models
-# 🎓𝗗𝗦𝗣: The Demonstrate–Search–Predict Framework
-
-The **DSP** framework provides a programming abstraction for _rapidly building sophisticated AI systems_. It's primarily (but not exclusively) designed for tasks that are knowledge intensive (e.g., answering user questions or researching complex topics).
-
-You write a **DSP program** in a few lines of code, describing at high level how the problem you'd like to solve should be _decomposed_ into smaller _transformations_. Transformations generate text (by invoking a language model; LM) and/or search for information (by invoking a retrieval model; RM) in high-level steps like `generate a search query to find missing information` or `answer this question using the supplied context`. Our [research paper](https://arxiv.org/abs/2212.14024) shows that building NLP systems with **DSP** can easily outperform GPT-3.5 by up to 120%.
-
-**DSP** programs invoke LMs in a declarative way: you focus on the _what_ (i.e., the algorithmic design of decomposing the problem) and delegate _how_ the transformations are mapped to LM (or RM) calls to the **DSP** runtime. In particular, **DSP** discourages "prompt engineering", which we view much the same way as hyperparameter tuning in traditional ML: a final and minor step that's best done _after_ building up an effective architecture (and which could be delegated to automatic tuning).
-
-To this end, **DSP** offers a number of powerful _primitives_ for building architectures that compose transformations and offers corresponding implementations that map these transformations to effective LM and RM calls. For instance, **DSP** *annotates* few-shot demonstrations for the LM calls within your arbitrary pipeline automatically, and uses them to improve the quality of your transformations. Once you're happy with things, **DSP** can *compile* your program into a much cheaper version in which LM calls are transparently replaced with calls to a tiny LM created by the **DSP** runtime.
+This section will introduce **DSPy** briefly.
-
-<p align="center">
- <img align="center" src="docs/images/DSP-tasks.png" width="460px" />
-</p>
-<p align="left">
- <b>Figure 1:</b> A comparison between three GPT3.5-based systems. The LM often makes false assertions, while the popular retrieve-then-read pipeline fails when simple search can’t find an answer. In contrast, a task-aware DSP program systematically decomposes the problem and produces a correct response. Texts edited for presentation.
-</p>
+If you're looking for DSP (v1), it's on a branch.
## Installation
-```pip install dsp-ml```
-
-## 🏃 Getting Started
-
-Our [intro notebook](intro.ipynb) provides examples of five "multi-hop" question answering programs of increasing complexity written in DSP.
-
-You can **[open the intro notebook in Google Colab](https://colab.research.google.com/github/stanfordnlp/dsp/blob/main/intro.ipynb)**. You don't even need an API key to get started with it.
+```
+pip install dspy-ai
+```
-Once you go through the notebook, you'll be ready to create your own DSP pipelines!
-### Temporary Note:
+## Getting Started
-The ColBERTv2 Wikipedia index has been moved to the following URL. **http://index.contextual.ai:8893/api/search?query=who%20is%20the%20first%20person%20on%20the%20moon**.
+Our [intro notebook](intro.ipynb).
-To use in the notebooks, please set the colbert server URL to be `http://index.contextual.ai:8893/api/search`.
+## Key Concepts
-<p align="center">
- <img align="center" src="docs/images/DSP-example.png" width="850px" />
-</p>
-<p align="left">
- <b>Figure 2:</b> A DSP program for multi-hop question answering, given an input question and a 2-shot training set. The Demonstrate stage programmatically annotates intermediate transformations on the training examples. Learning from the resulting demonstration, the Search stage decomposes the complex input question and retrieves supporting information over two hops. The Predict stage uses the retrieved passages to answer the question.
-</p>
+in terms of things DSPy can do for you (or allows you to do easily)
+and code examples for a flavor of the abstraction and ergonomics.
-## ⚡️ DSP Compiler
-Our [compiler notebook](compiler.ipynb) introduces the new experimental compiler, which can optimize DSP programs automatically for (much) cheaper execution.
+## 📜 Reading More
-You can **[open the compiler notebook in Google Colab](https://colab.research.google.com/github/stanfordnlp/dsp/blob/main/compiler.ipynb)**. You don't even need an API key to get started with it.
+Follow @lateinteraction on Twitter.
-## 📜 Reading More
+OLD stuff:
You can get an overview via our Twitter threads:
* [**Introducing DSP**](https://twitter.com/lateinteraction/status/1617953413576425472) (Jan 24, 2023)
@@ -72,9 +43,15 @@ You can get an overview via our Twitter threads:
And read more in the academic paper:
* [**Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP**](https://arxiv.org/abs/2212.14024.pdf)
+
+## Contributors
+
+
+
+
## ✍️ Reference
-If you use DSP in a research paper, please cite our work as follows:
+If you use DSPy (or DSPv1) in a research paper, please cite our work as follows:
```
@article{khattab2022demonstrate,
diff --git a/data/.gitignore b/cache/data/.gitignore
similarity index 100%
rename from data/.gitignore
rename to cache/data/.gitignore
diff --git a/data/multihop_800_unlabeled_questions.json b/data/multihop_800_unlabeled_questions.json
deleted file mode 100644
index f3ff435c1..000000000
--- a/data/multihop_800_unlabeled_questions.json
+++ /dev/null
@@ -1 +0,0 @@
-[{"question":"Which year was the 49th NHL Entry Draft where Emil Molin selected by the Dallas Stars in the 4th round?"},{"question":"When did the man overthrown in the Libyan Crisis become the leader of Libya?"},{"question":"Are Cynara and Piptanthus both flowering plants?"},{"question":"what chairman of the Senate Foreign Relations Committee inspired songs on the studio album Social Studies? "},{"question":"At 2006 Dubai Tennis Championships \u2013 Men's Singles, Rafael Nadal defeated the defending champion who has won how many Grand Slam singles titles?"},{"question":" What city was the birthplace of the German physicist for which Hund's coupling cases in the spectroscopy of diatomic molecules are name for?"},{"question":"Which party princesses fictional character is voiced by Eva Bella?"},{"question":"Who is the mother of Princess Carolina of Orange-Nassau and the daughter of King George II of Great Britain?"},{"question":"Ignacio E. Lozano Jr. was appointed to his position by what 38th President"},{"question":"What midfielder is captain of the US men's soccer team and joined Toronto FC for the 2014 season?"},{"question":"Chow Chow and German Wirehaired Pointer are both what?"},{"question":"What is the full name of the Welsh singer who was also a guest on the Logie Awards of 1993?"},{"question":"Which game has more pieces, King of Tokyo or Tsuro?"},{"question":"Are Condorman and The African Lion both comedy films?"},{"question":"Who currently owns the New York fashion house in which Lee Williams was the face of?"},{"question":"When was the former Provisional Irish Republican Army (IRA) Terrorist born with whom Joanna Cynthia Berry publicly met in November 2000 for Good Friday Agreement?"},{"question":"Which American television comedy writer wrote The Winter of His Content?"},{"question":"Who produced the album that included a re-recording of \"Lithium\"?"},{"question":"Gnaeus Pompeius of Pompeia became a member of the First Triumvirate along with this general and politician who played a key role in the transformation of the Roman Republic into the Roman Empire and was the second wealthiest person next to which person?"},{"question":"first movie exhibited in the Ottoman Empire was translated from French into English as what?"},{"question":"Which party won the 1995 San Francisco mayoral election?"},{"question":"What is name of the company that resulted from the merging of Cheung Kong Limited and a Fortune Global 500 investment holding company based in Hong Kong?"},{"question":"Who has more scope of profession, E. L. Doctorow or Julia Peterkin?"},{"question":"What documentary about the Gilgo Beach Killer debuted on November, 2016 on A&E?"},{"question":"What is the setting for the show in which Joel Tobeck plays the demon Baal?"},{"question":"The song Tim Nichols won a Grammy Award for Best Country Song in 2004 was on which Tim McGraw album?"},{"question":"Time Out's 100 best British films was topped by a film adapted from a short story by which author ?"},{"question":"Besides Mederer GmbH, what German companies compete with Haribo?"},{"question":"Who directed the 2010 film that starred the actor that played the scientist Jake on the CW's \"The 100\"?"},{"question":"Was Rien que les heures or The 11th Hour released first?"},{"question":"Are Truckin' Magazine and Girlfriends both publications that offer relationship advice?"},{"question":"A Simple Noodle Story was inspired by the debut of the American filmmakers who created what 1996 film?"},{"question":"When did Morrissey release a cover version of a song from the singer whose debut album was Horses?"},{"question":"Which documentary was filmed in the United States, A Story of Healing or The End of America?"},{"question":"\"Not Giving Up\" is a song by a girl group formed through what record label?"},{"question":"At It Again contains lyrics co-written by the singer and actor from what city?"},{"question":"Which author is English, Studs Terkel or John Braine?"},{"question":"What type of gesture does Minority languages of Austria and Austrian State Treaty have in common?"},{"question":"Who directs the upcoming monster film that a man known for his role as Richard Hendricks in the HBO series \"Silicon Valley\" star in?"},{"question":" John O'Sullivan is currently the editor of what Australian monthly magazine that reviews literature as well as features essays on ideas such as politics, history, universities and the arts?"},{"question":"The Haverhill Historical Society Historic District is located in what county?"},{"question":"Which American musician and front man for the alternative band Dinosaur Jr. also appears on two songs by the band Inherit?"},{"question":"Where is the sports nutrition business established by Oliver Cookson based ?"},{"question":"In what year was the star of Searching for the Elephant star Jo "},{"question":"What American dancer, actor, singer, director, and choreographer was in the 1943 musical comedy Du Barry Was a Lady?"},{"question":"How large is the memorial that forms the boundary of Constitution Gardens on the west ?"},{"question":"What type of dramatic work does Tom Bosley and Fiorello! have in common?"},{"question":"An analog board was used in which third model of the Macintosh line?"},{"question":"What is the scientific name of the fungus studied byby Ann Gravatt?"},{"question":"In what city's borough would one find the neighborhood in which the \"electric park\" which inspired many others is located in?"},{"question":"EXIT : E is the first extended play by a South Korean boy group that was originally what?"},{"question":"Who was the director of the 1988 American comedy-fantasy which stars Jeffrey Duncan Jones?"},{"question":"Who directed the remake of the movie that Richard C. Franklin was nominated for at the 63rd Academy Awards?"},{"question":"What director directed both The Ferryman and Road to Perdition?"},{"question":"Are both Jackson Hole Airport and Coastal Carolina Regional Airport located in North Carolina?"},{"question":"What type of competition was the 1963-64 Yugoslav Cup, also named honoring a Slavic communist statesman?"},{"question":"Dazzling Dunks and Basketball Bloopers was hosted by which American sportscaster?"},{"question":"In what year did the voice actor of \"Doc Hudson\" in the animated film \"Cars\" die?"},{"question":"What guided missile destroyer class was comprised of one ship named after a city in Yorkshire and two ships serving the Argentine Navy?"},{"question":"Fritz Goro died in Chappaqua, a hamlet of what town in Westchester County, New York?"},{"question":"Former Dominican-American Major League Baseball, Ruddy Lugo, attended a high school with notable alumni such as which 24th round pick of the Texas Rangers in the 1992 draft?"},{"question":"Did Clyde Geronimi and Maurice Tourneur live during any of the same years?"},{"question":"What was the name of the tour launched in support of Taylor Swift's album focused on the color of her emotions and beginning on March 13, 2013?"},{"question":"Which readers was the series The Lost Journals of Ven Polypheme published for?"},{"question":"Inter IIT Sports Meet is the annual sports tournament of what autonomous public insititutes of higher education in India?"},{"question":"What American animated series is composed by James L. Venable, garnering an Emmy nomination for Main Title Theme Music?"},{"question":"In what decade did the band with an album called The Crystal Method form?"},{"question":"What are the two branches of Robert D. Braun's career field? "},{"question":"The movie Apollo 13 featured an astronaut that was born in what year?"},{"question":"Are Wells Fargo and Southwest Airlines both air carrier companies?"},{"question":"At which University did the sociologist who popularised the term \"Unintended consequences\" spend most of his career and attain the rank of University Professor?"},{"question":"The sea whose port is The Port of Reggio is sometimes considered as a part of which ocean?"},{"question":"Which movie has a performance by the daughter of Christina Harrison?"},{"question":"Are Joe Dante and Ivan Passer both American film directors?"},{"question":"The Urswick School was named after a close friend of Henry VII and which grandmother of Henry VIII?"},{"question":"What medal did this Eritrean-born American long distance runner who was coached by Bob Larsen win during the 2004 Olympics?"},{"question":"Mermaids of Alcatraz Tour had an appearance by the country music songwriter who released what album in 2009?"},{"question":"Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?"},{"question":"Who was the drummer of the band formed by a founding member of Nirvana after Kurt Cobain's death?"},{"question":"George Ogilvy, 3rd Lord Banff died in which Scottish parish? "},{"question":"What is the birthyear of the American composer that borrowed from \"Second Connecticut\" on four occasions and combined American popular and church-music traditions with European art music?"},{"question":"Sebastian DeVicente starred in The City of Lost Souls with a Hong Kong actress who was the winner of two pageants in what year?"},{"question":"Who did the 1957 winner of the Nobel Prize in Physics work with on the problem of magnetic flux?"},{"question":"At which University does the father of the lead singer of The Joggers work ?"},{"question":"The heir to the Du Pont family fortune sponsored what wrestling team?"},{"question":"What former NBA star with the Philadelphia 76ers is now the captian and coach of a team in the BIG3?"},{"question":"Who did the box cover art for Keith Martin to make Night Below?"},{"question":"Which American actor born in 1915 starred in How to Steal a Million?"},{"question":"In what year was the star of To Hell and Back born?"},{"question":"2016 included the released of the farming simulation RPG published by which company?"},{"question":"For whom did Baker & Myers compose a song for who also starred in the PBS music program \"Austin County Limits\"?"},{"question":"Who co-invented the first electromagnetic telegraph along with one of the G\u00f6ttingen Seven?"},{"question":"Main Rahoon Ya Na Rahoon was a song by a finalist on what program?"},{"question":"Are Laurence Olivier and Aleksandr Ivanovsky both directors?"},{"question":"Who wrote the movie in which the song \"Nirvana\" was used?"},{"question":" Shane Lee Dawson was known for making comedy videos that included celebrity impersonations of an American singer dubbed the king of what?"},{"question":"Is Morgan State University or Boise State University further west?"},{"question":"Heart on My Sleeve is a song first released by which Australian singer?"},{"question":"\"War Song\" is a 1971 song by Graham Nash and what former member of Buffalo Springfield?"},{"question":"Just Wright starred an American rapper who was born in Newark, and signed with what record label in 1989?"},{"question":"In what year was the host of the 2016 Kids' Choice Sports Awards born?"},{"question":"Who released their debut album first, The Operation M.D. or Framing Hanley ?"},{"question":"The Olympus E-450 is a DSLR camera made by Olympus Corporation, which is headquartered in which city?"},{"question":"Who starred in the last film by Jed Johnson's lover?"},{"question":"Caesars Palace is located between The Mirage and a resort casino owned by what company?"},{"question":"Jet by Day is a band from what Georgia county?"},{"question":"Amigos para la aventura was Directed by this Argentine singer and Actor who was born on what date?"},{"question":"Which Hollywood figure is older, Angelina Jolie or Michael Anderson? "},{"question":" Sophie Gr\u00e9goire is married to what Canadian politician?"},{"question":"Gene Hackman won an Academy Award for best actor in a movie directed by who ?"},{"question":"What State has a Disney Resort & Spa that is a beachside hotel, resort and vacation destination offering complimentary children's activities and programs and that Djuan Rivers was a General Manager at?"},{"question":"Where was the subject of Godric born?"},{"question":"Narinder Kumar Gupta is Professor of Mechanics at a public institution that was declared to be Institute of National Importance by Government of India under what?"},{"question":"What American vocalist is famous for co-founding the Bangles and collaborating with Matthew Sweet?"},{"question":"Who established the British musical theatre project that revived a lesser-known work of award-winning lyricist Alan Jay Lerner?"},{"question":"At what university was Georg Nees appointed honorary professor of the scientific and practical approach to computation and its applications?"},{"question":"The Equal Access to COBRA Act was introduced on March 10, 2011 by the American former congressman who represented what district?"},{"question":"A British television dance contest has featured what Brazilian dancing style?"},{"question":"Which French professional football player was awarded the Ballon d'Or in 1991?"},{"question":"In what year did Christela Jacques compete for Miss Universe?"},{"question":"What was the estimated population, in 2016, of the city for which Simon Scott designed artwork for some of the most visually engaging restaurants, clubs and bars?"},{"question":"The play on which the 1953 film directed by Joseph L. Mankiewicz was based on was believed to have been written in what year?"},{"question":"Which actress in Galveston also played a character named \"Betty Cooper\" in another production?"},{"question":"Do Sydney Freeland and Genndy Tartakovsky work in the same industry?"},{"question":"Which disease outbreak of 2002\/3 caused 774 reported deaths, the majority being in Hong Kong, as reported by Andrew Brown for CNN?"},{"question":"What is the occupation of Robi Reed and Theodore Witcher"},{"question":"which university has the highest number of graduated students University of Texas at Austin or Minnesota State University, Mankato "},{"question":"Which surname of Mongolian origin do the male actors of Bollywood Sha, Aamir, and Salman share?"},{"question":"When was the President of India born who inaugurated Maharashtra Airport Development Company (MADC) on 1 October 2017?"},{"question":"James Ellison founded an extremist terrorist group that had its compound in the northern part of what state?"},{"question":"The mother of Katherine Eunice Schwarzenegger received what award for her coverage of the 1988 Summer Olympics?"},{"question":"What is the name of this musical, writtedn by Jeanie Linders, debuting in March 2001, starring Jeon Soo-kyung, in its Korean production?"},{"question":"How many American causalities were there in the battle that earned Clinton LaVor Romesha the Medal of Honor?"},{"question":"Kimber Aegis is a series of which single-action pistols?"},{"question":"FreeMIDI was a program made for the operating system developed by what company?"},{"question":"The group known as the Continuity IRA, along with the Real IRA, targeted what successor of the Royal Ulster Constabulary?"},{"question":"What person of british nationality also has a Ghanian passport?"},{"question":"In which city is the college Phillip Henry Bridenbaugh attended located?"},{"question":"What was the middle name of the aviator featured in \"The Smithsonian Institution\"?"},{"question":"\"Sal de Mi Piel\" is a song by a woman originally of what nationality?"},{"question":"Both Nalini by Day, Nancy by Night and Capturing the Friedmans are what type of films?"},{"question":"Which red wine variety made in Nistreana is increasingly celebrated as an Argentine varietal wine?"},{"question":"What country is Kristaps Porzi\u0146\u0123is, the the current champion of the New York Knicks, from?"},{"question":"What group did Marketa Irglova's \"Once\" co-star perform in?"},{"question":"What film stars Will Smith as police officer, and Happy Anderson as a human who works for the FBI's magic division?"},{"question":"In what country is the artist who released Ice on Fire born?"},{"question":"Which cast member from the original 1984 movie made a cameo appearance in the 2016 comedy remake?"},{"question":"What Superdome was SuperBowl XLVI played at in New Orleans, LA?"},{"question":"What type of ore was mined at both Thompson-Lundmark Mine and Leckie Mine?"},{"question":"Was Adam Anderson in the the Hurts or The Virgins?"},{"question":"Do the genera Bloomeria and Weigela describe species that live in the same region?"},{"question":"Radiohead and Oceansize are bands that play which type of music?"},{"question":"Who produced the 1985 film starring Bruce M. Fischer?"},{"question":"What was the name of the gang that the authors of \"Our Story\" were involved with?"},{"question":"Where did the older brother of Michael Dean Perry play college football?"},{"question":"What caused members of Alfred Shaker Historic District move to Sabbathday Lake Shaker Village?"},{"question":"Are Frida, en trotj\u00e4narinna and Stations of the Elevated both documentaries aired in Denmark?"},{"question":"Jimmy Eriksson will race in the last 2016 GP2 Series before it gets renamed"},{"question":"When was the Austrian band formed that made the album Solo?"},{"question":"The list of players who have played for the Princeton Rays includes what player nicknamed \"The Perfect Storm\"?"},{"question":"What occupation do Aleksandr Lyapunov and Yuri Matiyasevich have in common?"},{"question":"Who controlled the commando that kidnapped Dani Yako and detained him for three days in the Libertador Building?"},{"question":"A global hospitality and entertainment company operates destination resorts including Bellagio, and a luxyry resort on the Las Vegas Strip, that is how many stories?"},{"question":"What is the name of the first collection of poems or the American poet who appeared in numerous literary magazines, including one that added a book division with Elizabeth Ellen as editor?"},{"question":"When was the American stand-up comedian who was a supporting cast in \"Who's the Caboose?\" born? "},{"question":"Which film that Steve Yedlin worked on stars Bruce Willis? "},{"question":"Attack of the 5 Ft. 2 In. Women parodied the attack committed by the former figure skater who won the US Figure Skating Championships in what year?"},{"question":"who sings By the way and Universally speaking?"},{"question":"Which medal did the writer of \"Smile\" earn?"},{"question":"Which documentary series included an episode discussing the album including songs composed by Gedy Lee and Alex Lifeson with lyrics by Neil Peart telling a story set in a dystopian future?"},{"question":"Who currently holds the same title that Ian Gundry-White held in 1961?"},{"question":"What show that Tom Warburton worked on aired from September 3, 2010 to April 4, 2014?"},{"question":"What is the name of the Japanese manga series written by Eiichrio Oda and voice-acted by Travis Willingham?"},{"question":"Do the Canadian band Marianas Trench and the American singer Poe have different music styles?"},{"question":"In between Jason Kartalian and Fernando Arrabal who describes himself as \"desterrado\"?"},{"question":"Which game is mathematical, Acronymble or Rithmomachy?"},{"question":"Are Tony Hadley and Ian Ball of the same nationality?"},{"question":"What was the contribution made by de Graaff to the technology John Trump and Robert Van de Graaff developed together?"},{"question":"A Brilliant Genocide was written about a revolution started by which resistance leader?"},{"question":"What genre of music has American musicians Jonathan Davis and Philip Labonte both have in common in the bands they have played in?"},{"question":"Which team defeated the 49ers by 34-31 in the 2013 Super Bowl?"},{"question":"Which musician is older, M. Shadows or Justine Frischmann?"},{"question":"When did the criminal gang formed by someone serving six life sentences at the ADX Florence supermax prison in Colorado?"},{"question":"What Amherst, Masschusetts band was John Agnello involved with?"},{"question":"What type of authors are Alfred Bester and Iain Banks?"},{"question":"Who had a solo career, Paul Carrack of Charlotte Wessels?"},{"question":"What ethnic group does the incumbent who was challenged by R.J. Harris in the 2010 for the Republican Party nomination in the primary election for Oklahoma's 4th congressional district?"},{"question":"Which magazine was published first, Air Enthusiast or Air International?"},{"question":"Producers that won in the 24th Hong Knog Film Awards?"},{"question":"Johnny English and the killer cleaner were in what installment of the Johnny English film series?"},{"question":"The actor tha plays Jules Maigret in the ITV serial \"Maigret\" plays a supporting role in a 2003 film starring Dylan Moran that was directed by who?"},{"question":"What is the title of the \"SCTV\" sketch and the subsequent Canadian comedy album featuring the actors Rick Moranis and Dave Thomas?"},{"question":"What is the name of the country whose capital is in the north-east and birthplace of footballer, Khama Billiat?"},{"question":"Beside the one at Magic Kingdom Walt Disney World, what other theme park would one find the \"Happily Ever After\" firework\/projection mapping across Cinderella Castle?"},{"question":"Compass was produced by a record producer that is known primarily for working with who?"},{"question":"What actress played supporting roles in the movie \"Shout\" and the movie \"Swingers\" featuring Vince Vaughn?"},{"question":"Are either Frank Herbert and Paul Auster Australian?"},{"question":"Which Mallrats actress also stared in the comedy Police Academy: Mission to Moscow?"},{"question":"Which ReDRagon member was born on March 1, 1987 ?"},{"question":"Who starred as Brian Tanner in the 1990 American sitcom ALF?"},{"question":"Are both Yasunari Kawabata and Antonin Artaud from the same country?"},{"question":"\"Faith\" by Stevie Wonder is the lead single from a sountrack for a 2016 film directed by who?"},{"question":"What is the official name and code of the team Ernst Willimowski played for besides Poland when he played for them?"},{"question":"Where was the Northern Pacific Depot built after the previous depot was destroyed in the 200000 acre fire on September 1, 1894?"},{"question":"What is the journal that has an editor that is from a university that is part of PASSHE?"},{"question":"Keri Collins and James Kerwin are both what?"},{"question":"What was the larger conflict surrounding the revolt led by Ralph de Gael?"},{"question":"Which team won the 1914 championship of the Big Ten Conference? "},{"question":"Reunion is a sculpture by a man born in what year?"},{"question":"Who won the championship the same year that Sevilla F\u00fatbol Club failed to qualify for European competition?"},{"question":"Which university in the United States contains the only law school in the state and also had American basketball player Larry Moffett play for them?"},{"question":"Who was the county named for in which the unincorporated community call Cockrell is located?"},{"question":"Were both Nine Black Alps and Brand New formed in 2000 or later?"},{"question":"WHat German-born French composer created the operetta La Grande-Duchesse de G\u00e9rolstein?"},{"question":"The Wolf of Wall Street starred the actor who starred in what American sitcom?"},{"question":"Which ran longer Arizona Canal and Allegheny Portage Railroad"},{"question":"Which of the following won the 2001 GMA award for \"Male Vocalist of the Year\": ICS Vortex or Mac Powell?"},{"question":"Which is from farther south, Incubus or Crash Test Dummies?"},{"question":"Edmund Veesenmayer served under a senior the highest ranking officer in which part of the Nazi German army?"},{"question":"Which magazine is published in more countried, Womankind or Rabotnitsa?"},{"question":"As of June 27, 2014, the United States Route 15 going through New York State was redesignated as what motorway, running from central Pennsylvania through southern New York?"},{"question":"What role does Ghanashyam Nayak play in India's longest running sitcom serial?"},{"question":"Are the Nicandra genus and the Calpurnia genus both in the Fabaceae family?"},{"question":"One of the most prestigious watch manufacturers founded by a French horologist is located where? "},{"question":"What type of media does When the Levees Broke and Mala Mala have in common?"},{"question":"Batman: Knight Gallery was a comic book that featured art from the artist best known for work on what two other comics?"},{"question":"Who is considered one of the central founders of the science of thermodynamics and is one of the people that the Clausius\u2013Mossotti relation was named after?"},{"question":"Where is the band who released the live DVD Funeral Marches and Warsongs from?"},{"question":"Are The Makropulos Affair and From the House of the Dead both written by Leo\u0161 Jan\u00e1\u010dek?"},{"question":"Cloud Nine featured the Irish band originally known by what name?"},{"question":"Jean Martinelli was the star of an Alfred Hitchcock film set in what resort location?"},{"question":"Somos t\u00fa y yo is a Venezuelan telenovela filmed in Caracas, Venezuela and developed by which cable and satellite television channel owned by Turner Broadcasting System, a unit of Time Warner and its main flagship channel of Cartoon Network Asia?"},{"question":"When was the school that John O. Schwenn worked at established?"},{"question":"which English actor, comedian, screenwriter, has Rebecca Ferrand worked alongsid with "},{"question":"Who does \u00d3scar P\u00e9rez Rojas currently play for?"},{"question":"Are the magazines Sunset and Chesapeake Bay Magazine published with the same frequency?"},{"question":"The two children of Cassius Marcellus Clay Sr. and Odessa Grady Clay was a three-time World Heavyweight Champion and who?"},{"question":"Who was Empress of Japan prior to Empress Michiko, the current Empress?"},{"question":"Which genus has more species Tetradium or Schlumbergera?"},{"question":"The studios of WBLI are located in what New York county?"},{"question":"What spin-off of \"All That\" aired on a two-hour programming block on the American cable television network Nickelodeon, geared toward older (preteen to teen) audiences?"},{"question":"Ian MacKaye and Martin Johnson were both what in the rock bands?"},{"question":"Bailey Hall was built by the same architect as that who created the building that goes by which other name?"},{"question":"What Former American football player who was a highly touted and sought-after college football recruit and played for Oklahoma also played for the 1990 Los Angeles Rams? "},{"question":"Afshin Zinoori dubbed over the voice of which actor who made his debut on David Copperfield?"},{"question":"What Jerusalem location was part of the naming of the city where the National Naval Medical Center stands?"},{"question":"Preacher was an adaptation of the comic book written by the writer of what nationality?"},{"question":"Which television station licensed to Kansas City, Missouri is adjacent to television station KCPT?"},{"question":"Which atmosphere might be enjoyed in a place that serves Jaz Beer?"},{"question":"Albert Anthony Iafrate and Zdeno Ch\u00e1ra both set a record for? "},{"question":"What was the name of the last woman that Eddie Fisher of \"The Eddie Fisher Show\" was married to and had two daughter with?"},{"question":"In what city is the college preparatory private school where the author of the novel featuring symbologist Robert Langdon and cryptologist Sophie Neveu located?"},{"question":"What is special about the African-American cultural and educational center the descendants of enslaved Africans from South Carolina attended?"},{"question":"At what university have both Tamar Gendler and Vincent Scully worked?"},{"question":"What city is 560 m above sea level in Singapore and the birthplace of Parag Dahiwal?"},{"question":"What album was released by Top Dawg Entertainment and contains lyrical themes delivered by rasism and medication tolerence."},{"question":"What 'instrument' do both Sigurd Wongraven and Tina Turner play\/use to perform?"},{"question":"What do Freedomland U.S.A. and Baychester both have in common?"},{"question":"What was the nationality of the costume designer of Homage to the Queen"},{"question":"Where is the institute formerly directed by food technologist Vishweshwaraiah Prakash located?"},{"question":"What was the name of the German team that featured Robert Garrett and the 7ft power forward from the Dallas Mavericks?"},{"question":"Who is older, Caroline Wozniacki or Jared Palmer?"},{"question":"The Wright 3 is a novel about the characters trying to save a landmark that was designed by what architect?"},{"question":"Of the two documentaries Crazy Love and The Life of Mammals, which was filmed after 2004?"},{"question":"What area is known as the \"Isle of 8 Flags\" and is home to the Fernandina Beach Historic District?"},{"question":"What gender does Steve Sarkisian and Mike Connors have in common?"},{"question":"How far away from Marsciano is Perugia?"},{"question":"Denise Richards Played Ross Geller's cousin on a tv sitcom created by who?"},{"question":"Concertos for Four Violins was composed by this man who went to University to study law?"},{"question":"Where is broadcaster of World Have Your Say headquartered?"},{"question":"Are Josh Franceschi and Lars Frederiksen in the same band?"},{"question":"What sport was the multi-purpose stadium in which the 1992 Peach Bowl took place originally built to attract?"},{"question":"John Campanius was a Lutheran priest assigned to a colony established during what devastating war?"},{"question":"Where was the actress who played the Priest's niece born?"},{"question":"Shari McEwen admires a civil rights activist born in what year?"},{"question":"Are both A Midsummer Night's Dream and La sonnambula operas?"},{"question":"The actor who played Chino Ho Kelly on the T.V. series \"Hawaii Five-O\" Starred along side his son that played which character?"},{"question":"The famous character brought to America on the ship Lord Ligonier passed away in which year ?"},{"question":"Which person Jim Cassell has been responsible for the development of plays for Premier League club Liverpool and the England national team?"},{"question":"Beyond Our Ken and High Paddock were connected though what?"},{"question":"The Bravados starred what English actress who was born in Paddington, west London?"},{"question":"What work is the keynote speaker at the Future of Freedom Conference in 1969 known for?"},{"question":"Besides a flower delivery network, what other organization is best known by its telegraphic address?"},{"question":"Which of these Asian directors was behind the science fiction adventure film \"Snowpiercer\", Bong Joon-ho or Vincent Kok?"},{"question":"Renaldo James Woolridge placed for what team in the NVA Development League?"},{"question":"A Life Too Short was a biography about the goalkeeper of what nationality?"},{"question":"On what album was Lonnie Melvin Tillis' song written by Bob Corbin?"},{"question":"Best-Lock Construction Toys are compatible with a line of construction toys manufactured by a privately held company based out of where?"},{"question":"What size do Cape Peninsula University of Technology and Alexandria University have in common?"},{"question":"What member of Garin i Giperboloidy was born on June 24, 1963?"},{"question":"The album Wildflower, which includes the song Always on Your Side, is one of ten albums released by which American singer?"},{"question":"\"Hurt\" is a 2009 gothic horror movie starring William Mapother, but what TV series is he most associated with?"},{"question":"Back to the Future: The Ride was previously located at Universal studios where it has since been replaced by that simulator ride based on the animated television series \"The Simpsons\""},{"question":"What Spitsbergen based seed bank serves to protects agricultural diversity in the far north, 1300 km from the North Pole?"},{"question":"When was the university, which the American paleontologist and geologist known for the discovery of Dreadnoughtus, founded?"},{"question":"Nigel Bruce in Coming out Party was best known for his role in what other series?"},{"question":"Frederick Hibbard graduated from which public land-grant university?"},{"question":"The Lake Koocanusa Scenic Byway begins in what county?"},{"question":"Are both Mariqueen Maandig and Pelle Almqvist from the same country?"},{"question":"Were Subhash Ghai and Nora Ephron both in the film industry?"},{"question":"The Mother first appears in the episode that is part of which season of \"How I Met Your Mother\"?"},{"question":"Who has the highest scope of profession in Dirk Frimout or Andr\u00e9 Kuipers"},{"question":"What is the sport that Henrik Sedin and Daniel Sedin played together throughout their careers ?"},{"question":"Kewanee is how many miles west of the only east-west Highway connecting Mexico and Canada?"},{"question":"Who is this American politician and academic who got elected as President of the United States after becoming nominee at the Democratic National Convention in Baltimore, Maryland?"},{"question":"Xestaspis paulina is endemic to a country that is located northeast of where?"},{"question":"Are Thurston Moore and Pharrell Williams both rappers?"},{"question":"The crown name after Edward the Confessor is one of the oldest Crown Jewels of England, or now known as what?"},{"question":"What type of person does Deirdre Connelly and Andrew Witty have in common?"},{"question":"Which former Governor of Arkansas was elected president the same year as the United States Senate elections in 1992?"},{"question":"What is the river 3 mi north-west of Gainsborough said to take its name from?"},{"question":"Who is Claudio Ranieri in reference 1996\u201397 A.C. Fiorentina season?"},{"question":"What company owns brand of goal tending equipment endorsed by former head coach of the Colorado Avalanche? "},{"question":"Jonathan Smith's song is one of three covers on an album released by what group?"},{"question":"What football team represented the University of Oklahoma and was featured as #11 (BCS ranked) on the The 80th annual Sugar Bowl?"},{"question":"Which alternative rock band was founded first, Galaxie 500 or Sponge?"},{"question":"Which of Adelaide Metro's stations is located at the base of the Mount Lofty Ranges?"},{"question":"The top brand of beer in North Korea is brewed by which company ?"},{"question":"Which actress was nominated for three Tony Awards and played the character of Mollie in Take Her, She's Mine?"},{"question":"Nico Constantine is an American music director that is known for his work with this rock band from Atlanta, Georgia?"},{"question":"Change is Now: Renewing America's Promise includes previously released tracks from an American hip hop recording artist who debuted in 1992 with what album?"},{"question":"Where was the band which went along with Ludichrist formed?"},{"question":"What award, included in the term EGOT, did A Chorus Line win?"},{"question":"When is the tennis tournament held annually in which Bernard Tomic reached the fourth round in 2012, 2015 and 2016?"},{"question":"What is another name for a Garryowen, the kick John Peard is famous for?"},{"question":"What group released a song inspired by Debbie Harry from the band Blondie as well the song \"Love Shack\"?"},{"question":"Powder Room is a film that stars an actress of what nationality?"},{"question":"What artist did Kanye West produce singles for that was supposed to use album beats produced by DJ Premier?"},{"question":"What is the nickname of the airplane model flown in Malaysian Airlines Flight 370?"},{"question":"Which magazine is published more foten, Darling or Femmes d\u2019Aujourd\u2019hui?"},{"question":"Between The All-American Rejects and Chvrches, which band has Lauren Mayberry as its lead vocalist?"},{"question":"What type of facility was Roger Batzel best known for as director? "},{"question":"Was Atomic Ed and the Black Hole or Giuliani Time released first?"},{"question":"What year did the band who had a top 40 hit in the UK and a promotional video directed by Spike Jonze have a reunion tour?"},{"question":"What country does Rand Show and Johannesburg have in common?"},{"question":" Montrose Regional Airport a public airport as opposed to Luis Mu\u00f1oz Mar\u00edn International Airport which is what?"},{"question":"Who wrote the screenplay for a movie featuring ChristIan bale?"},{"question":"Bad Kostritz and the Kostritzer brewery are located in what country?"},{"question":"In which country was the author of \"A Method for Prayer\" born ?"},{"question":"What was succeeded by the Aston Martin DB6 and was released in 1963?"},{"question":"Are Prevention and Drum! both American magazines?"},{"question":"Aaron S. Rosenberg, an American novelist and game designer, wrote what fantasy novel? "},{"question":"What country leads the ranking system that was used to calculate the seeding for the 2006 FIFA World Cup?"},{"question":"Which school in the Kansas Unified School District 233 has the Owl for a mascot?"},{"question":"Which airport is closer to the city, Gerald R. Ford International Airport or Grand Canyon West Airport?"},{"question":"In what year was the album released that had That Ol' Wind as the sixth single?"},{"question":"Both Brian Warren and Jake Shields are considered what?"},{"question":"Which of the town in which Borough of Broxbourne council is based is a town in the Broxbourne borough of the English county of Hertfordshire, situated in the Lea Valley?"},{"question":"The Cheakamus Valley basalts are part of what southwestern British Columbia formation?"},{"question":"Which is the largest airport which serves Sauce Viejo Airport?"},{"question":"Are both Diamond v. Chakrabarty and Gates v. Collier United States Supreme Court cases?"},{"question":"Who is noted for their work on the history of Egypt's largest Christian church?"},{"question":"What was the score of the National Football League (NFL)'s champion Green Bay Packers first overall Super Bowl victory since the Orange Bowl?"},{"question":"Who is from farther west, Sunye or Alex Chilton?"},{"question":"Which system is most likely to have impacted current computer technology, Atari or Comx-35?"},{"question":"What main thoroughfare in Warsaw's borough of Ursyn\u00f3w connects to a 6,500 km international road which coincides with the Trans-Siberian Highway?"},{"question":"Were Songyuan and Shulan in the same province in China?"},{"question":"Who starred in the film that opened the 48th annual Berlin International Film Festival?"},{"question":"The eight part British television series \"Rhodes\" is based on the life of a man who was Prime Minister of what then-British colony?"},{"question":"The 1998 American horror film produced and directed by Gus Van Sant was a remake of a film that had an actress win what Golden Globe Award?"},{"question":" Both Roulette and Game of the Goose are what?"},{"question":"In 1017 Gervonta Davis was ranked as the world's sixth best boxer in the division by what magazine owned by Golden Boy Enterprises?"},{"question":"What did the first book of Gary Zukav explore ?"},{"question":"How long is the monorail that has a platform 2000 feet away from Caesars Palace?"},{"question":"Which character is married to the Vision but also the daughter of Magneto?"},{"question":"What countr is Raquel Welch from when she acted on the show Bandolero!?"},{"question":"In regards to the Mexican American country music artist who signed with Columbia Nashville in 1993, what was his only number one single on the Billboard Hot Country Singles & Tracks chart?"},{"question":"Is Appleton International Airport or Lihue Airport further north?"},{"question":"The city where Intesa Sanpaolo is based is the capital city of what region in Italy?"},{"question":"Which college football team that is currently a member of the Big Ten Conference, once had Canadian football player, Jason Nugent, play for them?"},{"question":"Which drink would be more likely to be served at a wedding, Mimosa or Blow Job?"},{"question":"When was the last film in the series \"American Pie\" released?"},{"question":"Does Paramount Plaza or Times Square Tower have more stories?"},{"question":"The Chinese men's national under-20 ice hockey team plays in the events traditionally held in what month?"},{"question":"What city in New York, previously known as Little Cow Harbor, has the historic John Harned House?"},{"question":"Matt Schultz is the lead singer of a band from which state?"},{"question":" Winston Churchill and Joseph Wambaugh were both what?"},{"question":"Who was born first, Reginald Denham or Jack Conway?"},{"question":"What year was the murderer of the Princes in the Tower born?"},{"question":"When did the cover athlete for the video game NHL 12 win the Maurice Richard Trophy?"},{"question":"Who is the actor that appeared in both Small Apartments (2012) and the SBS comedy series Pizza?"},{"question":"Are Li Yang and Mark Robson both Chinese directors?"},{"question":"The Hills Have Thighs was based on the movie The HIlls Have Eyes that was released on what date? "},{"question":"Who is Israeli illusionist, magician, television personality, and self-proclaimed psychic that appeared in the NBC TV show Phenomenon along side Criss Angel and hosted by Tim Vincent ?"},{"question":"Passu Sar and Trivor are mountain peaks in what country?"},{"question":"Which Portuguese winger played beside Zlatan Ibrahimovi\u0107 in the 2008-09 Inter Milan team ?"},{"question":"Which was founded first, Foreign Policy or Domino?"},{"question":"The Dutch guilder is the currency of the country that has what type of governmental system?"},{"question":" Fabrizio Ciano was the son of a foreign minister who left behind a diary that was used in an HBO documentry-drama entitled what?"},{"question":"Bill Nye hosted a self-titled show, which was predecessor to \"The Eyes of Nye\"; what was the final air date of that predecessor show?"},{"question":"What actor played the role of Michael Stivic on \"All in the Family\" and also starred in The Wolf of Wall Street?"},{"question":"The Minneapolis-St. Paul Airport Trams and the airport also known as World Chamberlain Field were both constructed by whom?"},{"question":"When did the person who ruled over Per St\u00e5lhammar have rule?"},{"question":"What full service law firm has the Congressman who won re-election in Florida's 21st district in 2012 been affiliated with?"},{"question":"Which group that employs nearly 6,800 employees is the largest share holder of Nordea?"},{"question":"Nine-card Brag is just one of the varying ancestors to which card card game?"},{"question":"Which Oscar- and Emmy-nominated English actor starred in the 1982 British romantic adventure film \"The Scarlet Pimpernel\"?"},{"question":"Who is younger, Paul Meany or Nigel Clark?"},{"question":"The Treaty of Fort Harmar was an agreement between the United States government and numerous Native American tribes with claims to what post-AMerican revolutionary War territory northwest of the River Ohio, and that legally existed from July 13 1787 until March 1, 1803?"},{"question":"Which Centre did the author of \"The Strange Death of Europe\" found?"},{"question":"Who has won more awards for their work, Darren Benjamin Shepherd or Lilibet Foster?"},{"question":"Which character did Bill Melendez voice who was a pet beagle?"},{"question":"The cheap gold jewelry in which a Cockney Wanker wears is manufactured by a catalog retailer that is a subsidiary of which company?"},{"question":"Both Canadian and American Chinese Cuisine were influenced by what particular immigrants?"},{"question":"How many minutes did the current goalkeeper for Borussia Dortmund remain unbeaten at his previous club?"},{"question":"Are Charles Olson and Gay Talese both poets?"},{"question":"Was the commissioner of the building Brenda A. Levin worked on? "},{"question":"In addition to the individual born in October 1959, who else helped produce the studio album Featuring Songs from Their Television Show?"},{"question":"The actress of the ex-wife of St. John in \"No Such Thing as Vampires\" has appeared in what other films?"},{"question":"Does Remy Zero have more members than Skunk Anansie?"},{"question":"This star of Ayal Njannalla was an actor and what else?"},{"question":"The 2015 film starring an actor whose debut was in the television series White Christmas was directed by whom?"},{"question":"When was the British statesman under which Peter Kirk was a minister in his government born? "},{"question":"Solely based on the given paragraphs above which band has released more albums, Rammstein or Chiodos?"},{"question":"Which civil unrest (protests) led to this armed conflict in 2011 in Libya during which Arrai TV was used by overseas Libyans to defend the Gaddafi government?"},{"question":"Clic is a 1974 album whose music has stylistic similarities with an American composer widely regarded as one of the most influential musicians of what?"},{"question":"Who produced the television series to which Brett Gelman for his roles?"},{"question":"Of Herbert Achternbusch an David Seltzer, which lived closer to Canada?"},{"question":"Sir Francis Nethersole was secretary to an Electress that is often referred to by what nickname?"},{"question":"Who starred in Where'd You Go, Bernadette and is the daughter of the producer Donald P. Bellisario?"},{"question":"The Argentine city Crespo holds an annual beer festival based on a German festival that began in what year?"},{"question":"Tay Garnett and Alexander Kluge both have what job?"},{"question":"Who was an enemy of the fictional character worked on by Esad T. Ribic that was the adoptive brother of Thor?"},{"question":"In what year did William \"Gipsy\" Daniels knock out the future heavyweight champion of the world?"},{"question":"Are Beth Gibbons and Ian Curtis the same nationality?"},{"question":"Matthew Trupiano was the boss of which St. Louis group?"},{"question":"In which province did percussionist Masaki Tanazawa perform in on 7 November 1985?"},{"question":"Which of those that anchored \"The Collection at RiverPark\" is an American specialty retail chain company that operates The Container Store, which offers storage and organization products?"},{"question":"The books in which Caras Caladhon is a fictional city have sold how many copies?"},{"question":"Wiremu \"Bill\" Whareaitu represented New Zealand in the commonwealth games held on which dates ?"},{"question":"The Schuylerville Stakes is held annually at a horse racing track in which Saratoga County town?"},{"question":"The album that 808s & Sax Breaks is a jazz interpretation of was released in what year?"},{"question":"What is the birth date of the composer who created the opera the Consul? "},{"question":"In what year was Bradley Orr's football playing nephew born?"},{"question":"Otto Dickel became a proponent of laissez-faire capitalism which translates to what?"},{"question":"This song, which got heavy rotation on MTV was written by Sebastian Bach, Rachel Bolan and a rock guitarist in what kind of band?"},{"question":"Who co-directed the eight-episode Danish television mini-series that the American paranormal hospital drama, All Souls, was based on?"},{"question":"What is the co-founder of The Alliance for Lupus Research currently serving as?"},{"question":"What is Anthony Stark trying to do with Amahl Farouk?"},{"question":"In which island the RCC Broadcasting Company broadcasts?"},{"question":"Which band has more members, Saint Motel or Curve?"},{"question":"What tour grossed 75 million dollars and was also the name of Michael Jackson's sixth studio album?"},{"question":"Where is the UK band from who's single is \"Unfinished Business\"?"},{"question":"What was the first flagship sedan of the class of Mercedes-Benz which preceded the Mercedes-Benz CL-Class?"},{"question":"Bazigit Atajev and Tsuyoshi Kohsaka are both what type of athlete?"},{"question":"Football Under Cover and 10 MPH are both what type of films?"},{"question":"Who is older, Tsuyoshi Kohsaka or Ant\u00f4nio Silva ?"},{"question":"Sorghum \u00d7 drummondii is a type of grass harvested for what type of edible fruit?"},{"question":"Which chain was founded first, Blaze Pizza or Marion's Piazza?"},{"question":"Who was born first Mike Judge or Albert S. Rogell?"},{"question":"How many people are employed by the company operating the Varsseveld railway station?"},{"question":"Se ilden lyse (in English: Fire In Your Heart) is a single from which Norwegian soprano?"},{"question":"what is the largest university press in the world that published Oxford Companions?"},{"question":"How many episodes were there of the show where Raven Goodwin played Tangie Cunningham?"},{"question":"Which university has more colleges within the school, University of Alabama in Huntsville or University of North Texas?"},{"question":"One of the founders of lastminute.com is known as a philanthropist, a public servant, as well as a...?"},{"question":"Who is older, John Fitzgerald or Andy Murray?"},{"question":"Bianca's Smut Shack became a popular theme camp at annual gathering held at which city ?"},{"question":"What is the another name for the singer of Standing Hampton?"},{"question":"F\u00fcnf Minuten Amerika and Felix Salten both were creators of what media?"},{"question":"Which magazine prvides information on nurtrion, Prevention or Bloomberg Businessweek?"},{"question":"Which five uniformed services of the United States are eligible for the National Defense Service Medal?"},{"question":"The Taxpayer Choice Act was supported by a U.S. Representative from what state? "},{"question":"What is the film that Shuna Harwood was nominated for an Academy Award for adapted from?"},{"question":"Chiquitita had lead vocals from the singer who was part of the group which has sold how many albums and singles?"},{"question":"Which successful American immigrant was born most recently, Varaz Samuelian or Samuel Der-Yeghiayan?"},{"question":"To what country did Iannis Xenakis flee in the same year that Mark Kidel was born?"},{"question":"Where does the team Rick Bennett coaches play?"},{"question":"Are both Lydia Zimmermann and Miranda July Canadian film directors?"},{"question":"Morning Glory starred the American actor who is a regular on what Showtime\/BBC series?"},{"question":"Who ordered Jewish people to be sent to Nazi concentration camps as described in the book The Lost Childhood, in March 1933?"},{"question":"Which Soviet mathematician was born first Nikolay Bogolyubov or Israel Gelfand?"},{"question":"How populated is this minor civil division in Maine, of which The Forks, Maine is a good example?"},{"question":"Ren\u00e9e Montagne succeeded which American broadcast journalist, a Peabody Award-winning member of the National Radio Hall of Fame?"},{"question":"Who is a ficitional superhero that appeared in DC Comics and had Rama as a potential love interest?"},{"question":"Was Nick Heyward a better guitarist than Yeo Hoon-min?"},{"question":"In which issue did this fictional superhero in the DC Comics Universe, who was killed by Captain Cold, make his first full appearance?"},{"question":"Rome was named the 4th most desirable city to visit in the world in a 2007 edition of a magazine based in what city?"},{"question":"What is the nationality of the person who Space Art opened for in 1981?"},{"question":"What historic site in Macon County was supplied by Lucius B. Northrop?"},{"question":"David Lane of The Order encourages white nationalists to move to which 5 states?"},{"question":"Solider's Home starring actress started her career when?"},{"question":"Killing Mr. Griffin starred an actress born October 6 with what nationality?"},{"question":"Who wrote Elephant Stone with the song writer John Squire?"},{"question":"Alfred Sherman became an adviser to a Prime Minister of the United Kingdom that died in this year?"},{"question":"The Airdrome Dream Classic uses what visual arts style?"},{"question":"Uwajimaya had a store from 1968-91 in the shopping mall in what Washington city?"},{"question":"Michael Sandberg, Baron Sandberg was the executive chairman of what largest bank of Hong Kong?"},{"question":"Who was President of Argentina when C\u00e9sar Augusto Guzzetti became the first foreign minister of the military government?"},{"question":"When did the politician who represented the electoral district in Sydney's West resign from his political party?"},{"question":"Are Veronica and Ulex both in the flowering plant family?"},{"question":"Who was born in Barry, Wales and was involved in the E-borders programme?"},{"question":"What city was Joseph Druces victim working in?"},{"question":"What company headquartered in Kowloon, Hong Kong has Yvan Rodic contributed to?"},{"question":"What gang were Buck Barrow and Blanche Barrow apart of?"},{"question":"How many nations participated in the Winter Olympics where Jesse V\u00e4\u00e4n\u00e4nen placed 22nd in the individual sprint event?"},{"question":"Which rocket propelled weapon, sometimes referred to as the \"Stovepipe,\" did Colonel Leslie Alfred Skinner help develop?"},{"question":"Who was this Mughal Emperor that reigned nearly half a century whom Saadat Ali Khan I fought alongside with against the Maratha in the Deccan?"},{"question":"What was Eannatum's name known as for the ancient semitic-speaking people from Syria?"},{"question":"How does Hilton Singapore connected to an adjacent Hotel which belongs to a Canadian international luxury hospitality company headquartered in Toronto, Ontario?"},{"question":"The bassist for Marianas Trench and Alice in Chains share what common first name?"},{"question":"Where is the suburb of the racetrack which supplanted the Camden Motordome?"},{"question":"What is the elevation range for the area that the eastern sector of the Colorado orogeny extends into?"},{"question":"Which continent were the Australian Shepherd and the West Highland White Terrier breeds' ancestors located in?"},{"question":"Which largest city of the Indian state of Rajasthan in Western India is the capital of Rajasthan"},{"question":"What Lakota Tribe Reservation was Dolly Akers affiliated with?"},{"question":"Waltz #2 is a song by what Nebraska singer-songwriter with a distinctive vocal style?"},{"question":"How many residents are there in the city that the band \"The Shadows of Knight\" are from?"},{"question":"What name did Patrick Keeler's The Racounteurs use in Australia?"},{"question":"What ingredient of Yumil-gwa, in a botanical context, implies that the shell is indehiscent?"},{"question":"Which of Time magazines 100 most influential people in the world made a guest appearance in Billu?"},{"question":"The 9th AVN awards ceremony took place in a hotel formly known as?"},{"question":"What county is Cleavland School district based in?"},{"question":"What fictional antihero from comic books published by DC Comics, who sometimes acts as a cab driver, was created by Alan Moore, Steve Bissette, and John Totleben?"},{"question":"What city was a founder of the The National Basketball Retired Players Association the mayor of?"},{"question":"What media commentator and businessman won the XII Race of Champions?"},{"question":"Considering the Porsche 928 and Porsche Cayenne. Which of the two vehicles was exclusively manufactured before the 21st century?"},{"question":"Who did the actor who played J.D. Salinger in Coming Through the Rye play in Lonesome Dove?"},{"question":"The third largest inland port in Australia during the great river boat era of the mid-19th century is west and north-west to an area in a county in New South Wales that has what River as it's north-west boundry?"},{"question":"Which two units of government jointly share a ministerial position formerly held by the conservative MP from Hemel Hempstead?"},{"question":"Favre Lake can be found in what corner of the state of Nevada?"},{"question":"What structure did the Annie C. Maguire hit in the Gulf of Maine?"},{"question":"The Mexican Kickapoo also have a tribe in this state, home to one of three Federally recognized tribes of Kickapoo people?"},{"question":"What position is currently held by the husband of Brigitte Macron?"},{"question":"John Sandeman Allen was part of the South Norfolk constituency in the UK general election held on what date?"},{"question":"In what city did Lester Everett Lyles play for the New York Jets?"},{"question":"The empire that Ayurbarwada Buyantu Khan was the eighth Great Khan originated where?"},{"question":"What was the last album U2 released before the song Breathe?"},{"question":"This famous baseball player played the same position as his older brother, Vince, and was best know. For a hitting streak that still stands today "},{"question":"What is the occupation of the first wife of Robert Kardashian?"},{"question":"when was the Australian rules football club in which Greg Taube played for extablished "},{"question":"Are Ms. and Fucsia targeted at the same gender for their audience?"},{"question":"Name the songwriter from San Francisco who operates the Art of Ears Studio and who worked with the lead vocalist for the hip hop group Digital Underground?"},{"question":"Are Hinduism Today and Star Wars Insider both magazines?"},{"question":"George Herbert Jose was a canon of what cathedral in South Australia?"},{"question":"Who developed the game that features Somebody New on its soundtrack?"},{"question":"What type of media does Lion and Rooney Mara have in common?"},{"question":"The Mechanicville Bridgecarries New York State Route 67 across the Hudson River in New York connecting Mechanicville with Hemstreet Park in which town in Rensselaer County, New York, United States?"},{"question":"What was the name given to a series of pre-Christmas toyshop incidents in the United States in 1983? "},{"question":"What weekly newsmagazine is published in the capital and largest city of Peru?"},{"question":"Who was the director of the 2012 American musical comedy film starring the actress who owns the fashion line \"Hanamahn\"?"},{"question":"Diego Schwartzmann won the annual finals tournament in 2014, but failed to qualify to defend it in 2015, at which sport?"},{"question":"Both the Pyrenean Mastiff and the Braque d'Auvergne are what?"},{"question":"Who created the record company that released the Strokes' debut studio album?"},{"question":"What 2012 award was won by the writer of the \"Smile\" episode of the \"Doctor Who\" television series?"},{"question":"Who directed the 1987 romantic drama dance film that is directly parodied by the television film \"Totally Awesome\"?"},{"question":"What award did Scarlett Johansson win from starring in the 2003 film \"Girl with a Pearl Earring\"?"},{"question":"In which country are Wellingrove, New South Wales and Glen Innes, New South wales located?"},{"question":"What year did the recorder of Young Man Mose die?"},{"question":"Who was the goalkeeper who blocked the most decorated English footballer player of all time?"},{"question":"What did Susan Gottesman aid in the main research purpose at NCI?"},{"question":"Where is the maker of Citrus Hill headquartered?"},{"question":"Bullseye is a character who has a vendetta against the superhero appearing in comics by which company?"},{"question":"Which airport is closer in proximity to Richmond County, Georgia, Augusta Regional Airport, or Wichita Dwight D. Eisenhower National Airport?"},{"question":"When did the children's Indian television series in which Anushka Sen was popularly known started to broadcast? "},{"question":"What was the army rank of Mildred Gale's grandson?"},{"question":"What was the book about which Paul R. Gross written with Barbara Forrest in 2004?"},{"question":"Which Sustainment Brigade descended from an infantry regiment of the New York Army National Guard?"},{"question":"Who's army was Sanite Belair a member of during the Haitian Revolution?"},{"question":"Did Andrei Platonov or Alice Walker win more Pulitzer prizes?"},{"question":"Where in England does Maxime Teremoana \"Max\" Crocombe play football? "},{"question":"What was the contribution of the \"quiet Beatle\" to a song Ian McDonald described as having \"rich and unusual harmonic motion\"?"},{"question":"Jason Hooten is the head coach of the Men's Basketball team at a public institution of higher learning founded in what year?"},{"question":"How many feature films has the Irish-American actor who appeared in \"Frankenstein\" (1994) and \"Practical Magic\" 1998 starred in"},{"question":"Who was born second, Anita Shreve or Guillaume Apollinaire?"},{"question":"In which period the son of John Hunyadi was a King of Hungary and Croatia?"},{"question":"What characteristic is shared by Deadpool and Taskmaster? "},{"question":"In what year did a Soviet, Russian, and American theoretical physicist, whose main contributions are in the field of condensed matter physics, come up with his famous theory?"},{"question":"The Filipino theme park known as EK features the Space Shuttle which is what kind of ride?"},{"question":"The Logie Awards of 1998 featured guests including one of the stars of what popular NBC sitcom?"},{"question":"The album by Silver Jews featuring an American musician best known for being the lead singer in Pavement was recorded where in Brooklyn?"},{"question":"In which iteration of the Miss Universe pageant will Elizabeta Burg represent Croatia?"},{"question":"Are Hilton Head Airport and Mammoth Yosemite Airport both public airports? "},{"question":"Who was the author of the novel which was the basis for the 2011 film of the same nae which starred the actress whose screen debut was in the television film \"Replacing Dad\"?"},{"question":"What type of gesture does Donets-Krivoy Rog Soviet Republic and have in common?"},{"question":"Which drummer was Roger Daltrey's solo album, \"Under a Raging Moon\" dedicated to, who was also posthumously inducted into the \"Modern Drummer\" Hall of Fame in 1982?"},{"question":"Which ride, at a San Diego amusement park, mimics whitewater rafting?"},{"question":"Which magazine is more informative National Geographic or Men's Fitness?"},{"question":"Who's musicall style includes more than one style, Sammy Hagar or Kevin Rowland?"},{"question":"Which group released the album Good Morning Vietnam 2: The Golden Triangle which features samples from the film \"American Gangster\"?"},{"question":"What was the birth name of the actress who co-starred with Fred Astaire in \"Royal Wedding\"?"},{"question":"What city has the greatest population, Tongxiang or Yulin, Shaanxi?"},{"question":"Colin Campbell was replaced as NHL Senior Vice President by which president of the Toronto Maple Leafs?"},{"question":"Saint Vitus and Pentagram had what in common when it came to the music industry?"},{"question":"Royal Crescent is a crescent-shaped terrace of houses has been placed on what?"},{"question":"Jonas Rivera's film, Inside Out, was released in 2015 by which studio?"},{"question":"Which village within the town of York, Maine was the filming location for the final film role of Colleen Dewhurst?"},{"question":"Baleine, Nova Scotia is best known as the landing site for a pilot born in which year ?"},{"question":"David Boris Pall founded a corporation located where?"},{"question":"What city in France has a hall named after the commander of the American Expeditionary Force on the Western Front in World War I?"},{"question":"We Need a Little Christmas was first performed by what British-American-Irish actress whose career has spanned seven decades?"},{"question":"Which of the following is a home computer designed by Radofin and released in 1983: Mattel Aquarius or Tandy 1000?"},{"question":"Which cricketer is the younger brother of Jack Lindwall and played top-flight rugby league football with St. George?"},{"question":"Retired pro wrestler John \"Bradshaw\" Layfield created The Cabinet as a pro wrestling stable as part of what WWE brand?"},{"question":" Tim McGauley is currently playing for the longest continuously operating member club of the league still playing in what?"},{"question":"When I've Been Drinkin' was cowritten by what American Country singer born in 1985"},{"question":"Geary County, Kansas is named after what Civil War union general? "},{"question":"who died first Arthur Wahl or Glenn T. Seaborg ?"},{"question":"Among two literary figures, Robert Stone and Paul Heyse, who was born earlier?"},{"question":"The show \"O\" at the Bellagio is located in what city?"},{"question":"Are a Fuzzy navel and Pastis both alcoholic beverages?"},{"question":"Who is the director of this 1977 American musical drama film starring Adrienne King and John Travolta?"},{"question":"Who did Charles Everett Lively assassinate during a series of armed labor conflicts in the United States, roughly between 1890 and 1930?"},{"question":"Jeanette MacDonald and Maurice Chevalier famously made many films in the 1930s. In which one did she play a princess?"},{"question":"What developer of the Bragg\u2013Gray cavity theory was knighted in 1920?"},{"question":"Which movie is based on a book, Darby O'Gill and the Little People and College Road Trip?"},{"question":"What cartridge was developed in 1910 and is currently used in the M2 machine gun? "},{"question":"What animal first described by the Sri Lankan zoologist Deraniyagala is found at Yala National Park?"},{"question":"What countries have land borders with the country which Johannes (Juho) Lepist\u00f6 served as member of Parliament? "},{"question":"The Wrestlers is a grade II listed public house on on a read linking between which cities ?"},{"question":"Of the film directors Kenji Mizoguchi and Andrzej \u017bu\u0142awski, which one often went against mainstream commercialism in his films?"},{"question":"According to a British tabloid founded in 1903, how many signatures were collected for a petition from the Snowdrop Campaign to ban ownership and use of handguns?"},{"question":"What city is home to both the Istanbul Aviation Museum and the Crimea Memorial Church?"},{"question":"What is the birth date of the man who John Frusciante replaced as the Red Hot Chili Peppers guitarist?"},{"question":"Which filmmaker is not German, David Gordon Green or Detlev Buck?"},{"question":"Which cocktail is named for a popular hotel waiter, John Collins or Porchcrawler?"},{"question":"Which award did a actress of No Entry crowned in 1997?"},{"question":"Which Italian restaurant is based in Tennessee, Gino's Pizza and Spaghetti or Pizzeria Venti?"},{"question":"Dovyalis is a genus of shrubs and small trees belonging to the family Salicaceae, whereas Townsendia is a genus of NOrth American plants within what flower family?"},{"question":"What region of Tennessee is Embreeville located in?"},{"question":"Which 1968 Beatles' song, which is written by Paul McCartney but credited to the song partnership Lennon-McCarney, opens the double-disc album \"The Beatles\" and then seques into \"Dear Prudence\"?"},{"question":"Who acts as an analyst for a TV network for Kansas Jayhawks women's basketball team?"},{"question":"Ahmad Kola is in a country with how many inhabitants?"},{"question":"Who is the main character in the animated television series that orginally aired on Nickolodeon and featured the underwater adventures of the this title character and his friends from Bikini Bottom?"},{"question":"What WNBA team formerly played at the location of the Pacers\u2013Pistons brawl?"},{"question":"What show did Chuck Woolery host 1 year before hosting the game show \"Think like a Cat\"?"},{"question":"The father of Bobby Bare Jr made famous the song \"Detroit City\" along with what two other artists?"},{"question":"What is the mountain called that was named after William Barton Rogers and has a summit elevation of 5729 feet above mean sea level?"},{"question":"Edwin van der Sar spoke to the crowd after the fireworks incident on April 20, 2014 during the KNVB Cup final match. Which team won that game?"},{"question":"the star of the film Love from a Stranger is from which country who acted along with Binnie Hale?"},{"question":" \"Big Fish\" is a song from an album released by which record labels ?"},{"question":"Which plant has more species, Ruta or Tobacco?"},{"question":"Sightseek was a horse trained by the trainer inducted into the National Museum of Racing and Hall of Fame in what year?"},{"question":"Christopher Sclater Millard's work was helpful in the estate of which prolific Irish writer?"},{"question":"Which French- and Arabic-speaking country was the first in Africa to air its own version of the TV show \"Deal or No Deal\"?"},{"question":"What is the state in which one of the Quill Foundation's supporters is located?"},{"question":"What banking group, founded in 1818, sponsors Club Sportive Sedan Ardennes?"},{"question":"What NFL team did the host of \"The Challenge\" play for when they won the Super Bowl?"},{"question":"The Old Post Office Block is situated between the stage production venue at 76-96 Hanover Street and which other building?"},{"question":"Perry the Platypus stop the destrustion of all of the lawn gnomes in the \" Lawn Gnome Beach Party of Terror\" episode of which animated television series"},{"question":"The actress best known for her roles in the West End and Oliver frequented a Music Hall registered for what?"},{"question":"Turbidimetric inhibition immunoassay may be used to test your blood if you have what condition?"},{"question":"Which plant has more species, Microcachrys or Dichroa?"},{"question":"When did the Japenese series Moribito: Guardian of the Spirit first premier of Caroon Network's Adult Swim prgramming block?"},{"question":"\u015eahan Arzruni was an ethnic Armenian classical pianist whereas Shavo Odadjian was known for what?"},{"question":"When was the company CareFusion is a subsidiary of founded?"},{"question":"Who is older, Langston Hughes or James Agee?"},{"question":"The 2013 California Golden Bears football team had a head coach who played what sport in college?"},{"question":"John of Arsuf was the regent of Jerusalem under which memberof the Hohenstaufen dynasty?"},{"question":"What type of collection does Etarcomol and Medb have in common?"},{"question":"Anne Jacqueline Hathaway stared in a romantic comedy in 2010 which was written by Katherine Fugate, and directed by who?"},{"question":"Which company markets Melarsomine as Immiticide in order to treat a small thread-like worm that causes filariasis in dogs, mainly, but also cats, ferrets, bears, sea lions, and coyotes?"},{"question":"Where Jack Thompson is from and where the Victorian Football League's Collingwood Football Club?"},{"question":"Call a spade a spade has been used in literary works by what British playwright, novelist, and short story writer who was reportedly the highest-paid author during the 1930s?"},{"question":"Bob Scucci formerly worked at a casino located where?"},{"question":" Who was born first, Ren\u00e1ta Tomanov\u00e1 or Travis Parrott?"},{"question":"The artist that released the debut album \"Who Came First\" was the lead guitarist and main songwriter for what band?"},{"question":"Google's rival santa tracker has existed since what year?"},{"question":"What aviator born in 1925 was featured in Rocket Science?"},{"question":"Are both Martin Heidegger and Jos\u00e9 Echegaray engineers?"},{"question":"What important political role did the female star of the film Talk About a Stranger hold ?"},{"question":"Which anime television series aired 9 October 2016 was adapted by a Japanese animation studio founded by Mikihiro Iwata? "},{"question":"Who logged more hours in space, James P. Bagian or Edward Avedisian"},{"question":"George Newhouse was the Labor candidate for the seat that was first proclaimed in what year?"},{"question":"What local government area oversees the city where the Lake Oval is located?"},{"question":"What type of film was the 2006 comedy film that the host of \"The Andy Dick Show\" wrote\/directed\/starred in? "},{"question":"What is another name for the Mediterranean Theater of Operations, United States Army invasion of French North Africa during World War II?"},{"question":"When was the Australian screenwriter, producer, director and actor born who screenplay the Saw III?"},{"question":"Which happened first, the Battle of the Ch'ongch'on River or the Battle of Guam?"},{"question":"Who wrote the 2011 television drama that Gong Yoo played a role in?"},{"question":"Pudon is the location of a theme park featuring a show called?"},{"question":"Which board game was created in Finland, King Oil or Inkan aarre?"},{"question":"New York State Route 436 passes through a state park 35 mi southwest of what city?"},{"question":"The Lockenhaus Chamber Music Festival has included performances by the solo pianist who has held what country's citizenship since 1972?"},{"question":"When was the governor who appointed Lana Gordon first elected governor?"},{"question":"Which one of these universities founded first - Technical University of Crete and Mongolian University of Science and Technology?"},{"question":"How many bombs were used in the attack that killed Tim and Jonathan Ball?"},{"question":"Were the films Meet the Robinsons and Blackbeard's Ghost both based on novels?"},{"question":"Lutefisk often served at Svensk Hyllningsfest is native to what region?"},{"question":"Which park covers more land area, Sierra Nevada National Park or Atlantic Islands of Galicia National Park?"},{"question":"Which group contains more species, Willow or Stipa?"},{"question":"Which writer and journalist contributing to \"The New Yorker\" writes about subjects such as the Art Deco-style skyscraper, the Chrysler Building?"},{"question":"Renowned MC Rakim appeared on \"Guilty All The Same\" by Linkin Park, and is usually associated with which hip-hop partner?"},{"question":"Anderson Silva has been lauded by the current president of what organization?"},{"question":"What was the name of the fighter jet flown by the subject of John Barron's 1980 biography?"},{"question":"Which profession does Boris Barnet have in common with Baltasar Korm\u00e1kur?"},{"question":"the club that won the 29th edition of Serie A was founded in which year ?"},{"question":"What is the name of the award named in honor of the English actor who studied at the Royal Academy and had an eight decade career?"},{"question":"Alfred Goldthwaite was an attorney for the party that traces its roots to whom?"},{"question":"Who was the director of the 2012 American horror film, which was the sequel to the 2009 home invasion horror film The Collector?"},{"question":"Nora Roberts High noon is one of how many novels written by the bestselling author?"},{"question":"The 14th Golden Melody Awards was pushed back to August 2, 2003 from the original May date due to what viral respiratory disease of zoonotic origin caused by coronavirus?"},{"question":"What year saw the release of the first album by the boy band featured in a remix of Todo Cambio?"},{"question":"What relationship does Stevie Mann have to an Irish singer-songwriter born in 1973?"},{"question":"How many shops does the larger of the Miami International Mall and Dolphin Mall have?"},{"question":"The Way to Rainy Mountain was a book by the Pulitzer Prize-winning novelist of what heritage?"},{"question":"Which aircraft did ace pilot Gordon Arthur Stanley fly from aircraft carriers during World War II?"},{"question":"The first movie exhibited in the Ottoman Empire was which 1895 French short black-and-white silent documentary film directed and produced by Auguste and Louis Lumi\u00e8re?"},{"question":"Olivia Newton-John has how many more Grammy wins than the number of performers in the Main Event Tour?"},{"question":"Who was the co-star in the movie Mood of the Day of the common star of Mood of the Day and \"Re-encounter\" (2011)?"},{"question":"Which is the smaller dog, Dalmatian or Norfolk Terrier?"},{"question":"What type of human communication does Papeeha and Hindi have in common?"},{"question":"What did Drake do prior to working with Ricci Riera?"},{"question":"What type of university was the college that Arena Football League player Johnny Sims attended?"},{"question":"What number Grand Master was the person for whom the DemOlay INternational organization is named?"},{"question":"This American artist made the Billboard top 25 and was a well known YouTuber and also for her participation in which NBC singing competition? "},{"question":"Which organization did both Billy McMillen and the Irish National Liberation army have affiliations with until 'the Troubles' 1974?"},{"question":"In 2007 Paddy Considine starred in Hot Fuzz, who was the director?"},{"question":"Claude Nicollier and Andr\u00e9 Kuipers were both astronauts of which space agency?"},{"question":"Are both Around the World in 80 Days and The Nutcracker and the Four Realms films?"},{"question":"Maurice Hines and his brother were famous for what?"},{"question":"What song was inspired by a breakup and a hit single in 1985?"},{"question":"Who is younger, Bernhard Wicki or Lloyd Kaufman?"},{"question":"What helmet typically associated with the Prussian Army was replaced by a steel helmet during World War I?"},{"question":"Are Gustavus Airport and Augusta Regional Airport located in the same country?"},{"question":"Who were the three leading actors in a film that shows Traitor's Ford?"},{"question":"When was the Associate Justice of the Supreme Court of the United States born under whom Carrie Severino served as a law clerk?"},{"question":"Which was a game for more than two people, Medina or Hijara?"},{"question":"What year did the anchor store of the GVK One mall open?"},{"question":"Does Darren Bravo's half brother bat left-handed or right-handed?"},{"question":"The shopping center that incorporates multiple factory outlets in one center is located on which side of the Adelaide Airport?"},{"question":"The coach of the 2017 Houston Texans coached which team from 2012 to 2013?"},{"question":"Steven \"Steve\" Smith is a politician representing a disctrict that includes a town with how many lakes and ponds?"},{"question":"Which airport is located closer to the equator, Greater Rochester International Airport or Flagstaff Pulliam Airport?"},{"question":"What team, managed by a man who is also and advisor to Central Coast Mariners, won the 2008 FA Cup?"},{"question":"Which film was released later, Shocker or Very Bad Things?"},{"question":"Which municipality established in 2013 is where Cezar Mateus works?"},{"question":"Juan Pablo Di Pace appeared in a musical with music from what successful pop group?"},{"question":"Which group was formed first Hurricane No. 1 or DC Talk?"},{"question":"Which of the following is known for directing \"Die Another Day\": Lee Tamahori or Terry Zwigoff?"},{"question":"What class of submarines is based upon a class of diesel-electric attack submarines jointly developed by French Direction des Construtiones Navales and the Spanish company Navantia?"},{"question":"What was the category of the hurricane that damaged the Hanapepe Town Lot No. 18?"},{"question":"Which British French Crime drama series did William Ash star in?"},{"question":"Which city in California was the sole candidate for the 2028 games, officially known as the Games of the XXXIV Olympiad?"},{"question":"What departed \"Gossip Girl\" cast member was born in 1993?"},{"question":"NanoScale Corporation was founded to develop properties of a university with a main campus in what Kansas city?"},{"question":"Hristo Atanasov Bonev-Zumais a former Bulgarian footballer, the second all-time leading scorer for the Bulgarian national team behind which Bulgarian professional footballer who plays for Indian Super League club Kerala Blasters?"},{"question":"Type 07 military uniforms were first seen during the transfer of sovereignty from the United Kingdom to which country?"},{"question":"Slane Beg is northwest of a town in what Irish county?"},{"question":"Michael Kozlowski handled the career of an Israeli professional boxer who held the WBA super welterweight title in what year?"},{"question":"Which French video game publisher, published the war-theme real-time strategy game \"Conflict Zone\"?"},{"question":"Which documentary was released first, Walking with Cavemen or Life in the Freezer?"},{"question":"Buncle is a name derived from which medieval fortress?"},{"question":"What height was the singer who in 1952, along with Buddy Morrow recorded the song \"Got You on My Mind ? "},{"question":"What awards did the novel which precedes Count Zero win?"},{"question":"When was the first year reports of Aaron Burr's ghost appearing at One If By Land, Two If By Sea could have started?"},{"question":"You Got the Silver is a song from the album released in what month?"},{"question":"What was the first stage name of the Hollywood Undead frontman that produced the solo studio album 'Nine Lives'?"},{"question":"What is the birth name of the singer who inspired Julie Mintz and is more commonly known as Lana Del Rey?"},{"question":"When was the brother of Chris James born?"},{"question":"The 19th-century transportation route that connected Idependence, Missouri and Santa Fe, New Mexico runs the same path as what east to west highway?"},{"question":"Which is a propaganda film, Triumph of the Will or Food, Inc.?"},{"question":"In 2017 it was ranked as the world's 26th most reputable university and it awarded Kenneth Turner what honorary degree?"},{"question":"Mary and Max and Fantastic Mr. Fox are what kind of movie?"},{"question":"What 1993 movie was Emilie Schindler's husband the subject of?"},{"question":"Which American football coach, who is now the defensive coordinator at Arizona State Sun Devils, succeeded Mike Cavan as the head football coach at Southern Methodist University from 2002 to 2007?"},{"question":"Which band formed farther west, Unwritten Law or Mono Puff?"},{"question":"Which comes later: the day that Aden was born, or the day that IMG re-launched its men's division after its dismantling?"},{"question":"The Black Windmill is a film produced by a man best known for coproducing what? "},{"question":"Are Crash! The bankrupt game and P\u00dcNCT both types of games?"},{"question":"The Rucellai Sepulchre was built by an Italian humanist author who epitomised what?"},{"question":"What position did the Croation goalscorer in the 2009\u201310 DFB-Pokal season final primarily play?"},{"question":"Which film includes more game organizations, Tar Creek and Monster Camp?"},{"question":"Eric Kissinger currently plays for an outdoor socker team that plays in which region?"},{"question":"The Snake River is a tributary of which river on which Richland stands?"},{"question":"Which Swedish author and teacher wrote the 1912 novel Thy Soul Shall Bear Witness! ?"},{"question":"In which year was the first of the vampire-themed fantasy romance novels for which The Twilight Saga: The Official Illustrated Guide serves as a spin-off encyclopedic reference book first published?"},{"question":"Tripel beer was first introduced in 1956, by which Belgian abbey brewery?"},{"question":"Which opera has more acts Destiny or Rienzi?"},{"question":"What was Peter Browngradt's best known animated television series based on?"},{"question":"Which genus is in the verbena family, Citharexylum or Laelia?"},{"question":"Q1: Which actress played roles in both The Perfect Family and Bones?"},{"question":"When did the reality singing competition where a Dutch rapper of Moroccan descent appeared as a judge air?"},{"question":"What type of schools are both Carre's Grammar School and Kesteven and Sleaford High School?"},{"question":"What McMOnagle, Perri & McHugh attorney is notable for representing a hip hop artist with a stage name from a street in South Philadelphia"},{"question":"Justin Warfield released a hip hop album that featured production from the producer of what dual nationality?"},{"question":"What city did Edouard get his MBA in?"},{"question":"The Marathon is a mixtape by a rapper whose recodings inlcude one called what in which which rapper Jay-Z, bought 100 copies for $100 each?"},{"question":"What was the country of origin of one of the major users of Butterfly style in 1980's hockey?"},{"question":"When was the Polish professional boxer over which Joe Smith Jr. rose to prominence in 2016 by scoring an upset knockout victory born? "},{"question":"Of the two, 56 Leonard Street and Latting Observatory, which was known as a wooden tower in New York City?"},{"question":"In what nation is it the job of Mousa Esmaeilpour to support learns or clients in achieving a specific personal or professional goal specifically in bodybuilding?"},{"question":"Where did Richard Gillow found the furniture company, Gillows of Lancaster and London?"},{"question":"What one-time American Red Cross affiliate and African American medical researcher aided Dr. John Scudder with his Plasma for Britain program?"},{"question":"Which midsize car of 1973 to 2005 used a shared platform with Pontiac 6000?"},{"question":"Are Portsmouth International Airport at Pease and Fort Smith Regional Airport both located in the northeast US?"},{"question":"The American conservative Christian nonprofit organization that Alan Sears served as the president, CEO, and general counsel of until January 2017 is headquartered in what Arizona city?"},{"question":"Which instrument is Clara Rockmore a virtuoso of in the novel Us Conductors?"},{"question":"Nell Hodgson Woodruff School of Nursing awards whatcertification for a doctoral program, other than a PhD?"},{"question":"Which architect that designed 8 House was born on 2 October 1974? "},{"question":"Oumar Tchomogo who was born in Cotonou, where is this city located?"},{"question":"SASTRA Ramanujan Prize was founded in the hometown of the mathematician who compiled nearly how many results?"},{"question":"What is the group's title that covered the song that is the reason that is the main reason the album Raincoats is so well known?"},{"question":"What year did the war during which Pompiliu \u0218tefu was executed end?"},{"question":" Stony Awards presented a posthumous award to what American cannabis rights activist and the author of \"The Emperor Wears No Clothes\","},{"question":"Which major mid-20th century film director lived longer, Philip Ford or Julien Duvivier?"},{"question":"Do Rika Hiraki and Mara Santangelo come from the same country?"},{"question":"Were The Adventures of Huck Finn and Piglet's Big Movie distributed by the same studio?"}]
diff --git a/docs/images/DSPy_whitebg.png b/docs/images/DSPy_whitebg.png
new file mode 100644
index 000000000..5f89976ca
Binary files /dev/null and b/docs/images/DSPy_whitebg.png differ
diff --git a/dsp/__init__.py b/dsp/__init__.py
index cf5378343..fc8a74241 100644
--- a/dsp/__init__.py
+++ b/dsp/__init__.py
@@ -1,4 +1,43 @@
from .modules import *
from .primitives import *
from .templates import *
-from .utils import settings
\ No newline at end of file
+from .utils import settings
+
+
+"""
+TODO:
+
+The DspModule class serves as a proxy to our original 'dsp' module. It provides direct access to settings
+stored in `dsp_settings` as if they were top-level attributes of the 'dsp' module, while also ensuring that
+all other regular attributes (like functions, classes, or submodules) of the 'dsp' module remain accessible.
+
+By replacing the module's symbols with an instance of DspModule, we allow users to access settings
+with the syntax `dsp.<setting_name>` instead of the longer `dsp.dsp_settings.<setting_name>`. This makes
+for more concise and intuitive code. However, due to its unconventional nature, developers should be
+careful when modifying this module to ensure they maintain the expected behavior and access patterns.
+"""
+
+
+"""
+
+class DspModule:
+
+ def __init__(self):
+ # Import and store the original module object
+ self._original_module = sys.modules[__name__]
+
+ def __getattr__(self, name):
+ # First, try getting the attribute from the original module
+ if hasattr(self._original_module, name):
+ return getattr(self._original_module, name)
+
+ # Next, check dsp_settings
+ if hasattr(dsp_settings, name):
+ return getattr(dsp_settings, name)
+
+ raise AttributeError(f"'{type(self).__name__}' object and the original module have no attribute '{name}'")
+
+import sys
+sys.modules[__name__] = DspModule()
+
+"""
\ No newline at end of file
diff --git a/dsp/evaluation/utils.py b/dsp/evaluation/utils.py
index 8c2fbb1cc..5ce9a4b1d 100644
--- a/dsp/evaluation/utils.py
+++ b/dsp/evaluation/utils.py
@@ -1,3 +1,6 @@
+from openai import InvalidRequestError
+from openai.error import APIError
+
import dsp
import tqdm
import pandas as pd
@@ -80,3 +83,5 @@ def evaluate(fn, dev, metric=EM):
display(df.style.set_table_styles([{'selector': 'th', 'props': [('text-align', 'left')]}, {'selector': 'td', 'props': [('text-align', 'left')]}]))
return percentage
+
+
diff --git a/dsp/modules/__init__.py b/dsp/modules/__init__.py
index 1d8890b91..d0d9e7583 100644
--- a/dsp/modules/__init__.py
+++ b/dsp/modules/__init__.py
@@ -4,4 +4,6 @@ from .hf import HFModel
from .colbertv2 import ColBERTv2
from .sentence_vectorizer import *
from .cohere import *
-from .sbert import *
\ No newline at end of file
+from .sbert import *
+
+from .hf_client import HFClientTGI
\ No newline at end of file
diff --git a/dsp/modules/cohere.py b/dsp/modules/cohere.py
index bce5ab324..789704cac 100644
--- a/dsp/modules/cohere.py
+++ b/dsp/modules/cohere.py
@@ -9,7 +9,7 @@ try:
cohere_api_error = cohere.CohereAPIError
except ImportError:
cohere_api_error = Exception
- print("Not loading Cohere because it is not installed.")
+ # print("Not loading Cohere because it is not installed.")
def backoff_hdlr(details):
diff --git a/dsp/modules/gpt3.py b/dsp/modules/gpt3.py
index b2b1d297f..ab7ddf87c 100644
--- a/dsp/modules/gpt3.py
+++ b/dsp/modules/gpt3.py
@@ -36,12 +36,14 @@ class GPT3(LM):
model: str = "text-davinci-002",
api_key: Optional[str] = None,
api_provider: Literal["openai", "azure"] = "openai",
- model_type: Literal["chat", "text"] = "text",
+ model_type: Literal["chat", "text"] = None,
**kwargs,
):
super().__init__(model)
self.provider = "openai"
- self.model_type = model_type
+
+ default_model_type = "chat" if 'gpt-3.5' in model or 'gpt-4' in model else "text"
+ self.model_type = model_type if model_type else default_model_type
if api_provider == "azure":
assert (
@@ -68,7 +70,8 @@ class GPT3(LM):
"n": 1,
**kwargs,
} # TODO: add kwargs above for </s>
- if api_provider == "openai":
+
+ if api_provider != "azure":
self.kwargs["model"] = model
self.history: list[dict[str, Any]] = []
diff --git a/dsp/modules/hf.py b/dsp/modules/hf.py
index 3d7fa2462..13ef0499f 100644
--- a/dsp/modules/hf.py
+++ b/dsp/modules/hf.py
@@ -1,6 +1,13 @@
from typing import Optional, Literal
-from dsp.modules.lm import LM
+import os
+import json
+# from peft import PeftConfig, PeftModel
+# from transformers import AutoModelForSeq2SeqLM, AutoModelForCausalLM, AutoTokenizer, AutoConfig
+from dsp.modules.lm import LM
+# from dsp.modules.finetuning.finetune_hf import preprocess_prompt
+from dsp.modules.cache_utils import CacheMemory, NotebookCacheMemory, cache_turn_on
+import functools
def openai_to_hf(**kwargs):
hf_kwargs = {}
@@ -34,7 +41,7 @@ class HFModel(LM):
Recommeded to use "auto", which will help loading large models using accelerate. Defaults to "auto".
"""
try:
- from transformers import AutoModelForSeq2SeqLM, AutoModelForCausalLM, AutoTokenizer
+ from transformers import AutoModelForSeq2SeqLM, AutoModelForCausalLM, AutoTokenizer, AutoConfig
import torch
except ImportError as exc:
raise ModuleNotFoundError(
@@ -47,10 +54,26 @@ class HFModel(LM):
self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
if not self.is_client:
try:
- self.model = AutoModelForSeq2SeqLM.from_pretrained(
- model if checkpoint is None else checkpoint,
- device_map=hf_device_map
- )
+ architecture = AutoConfig.from_pretrained(model).__dict__["architectures"][0]
+ self.encoder_decoder_model = ("ConditionalGeneration" in architecture) or ("T5WithLMHeadModel" in architecture)
+ self.decoder_only_model = ("CausalLM" in architecture) or ("GPT2LMHeadModel" in architecture)
+ assert self.encoder_decoder_model or self.decoder_only_model, f"Unknown HuggingFace model class: {model}"
+ self.tokenizer = AutoTokenizer.from_pretrained(model if checkpoint is None else checkpoint)
+
+ self.rationale = True
+ AutoModelClass = AutoModelForSeq2SeqLM if self.encoder_decoder_model else AutoModelForCausalLM
+ if checkpoint:
+ with open(os.path.join(checkpoint, '..', 'compiler_config.json'), 'r') as f:
+ config = json.load(f)
+ self.rationale = config['rationale']
+ # if config['peft']:
+ # peft_config = PeftConfig.from_pretrained(checkpoint)
+ # self.model = AutoModelClass.from_pretrained(peft_config.base_model_name_or_path, return_dict=True, load_in_8bit=True, device_map=hf_device_map)
+ # self.model = PeftModel.from_pretrained(self.model, checkpoint)
+ # else:
+ self.model = AutoModelClass.from_pretrained(checkpoint).to("cuda")
+ else:
+ self.model = AutoModelClass.from_pretrained(model).to("cuda")
self.drop_prompt_from_output = False
except ValueError:
self.model = AutoModelForCausalLM.from_pretrained(
@@ -58,7 +81,8 @@ class HFModel(LM):
device_map=hf_device_map
)
self.drop_prompt_from_output = True
- self.tokenizer = AutoTokenizer.from_pretrained(model)
+ self.tokenizer = AutoTokenizer.from_pretrained(model)
+ self.drop_prompt_from_output = True
self.history = []
def basic_request(self, prompt, **kwargs):
@@ -80,6 +104,12 @@ class HFModel(LM):
assert not self.is_client
# TODO: Add caching
kwargs = {**openai_to_hf(**self.kwargs), **openai_to_hf(**kwargs)}
+ print(prompt)
+ if isinstance(prompt, dict):
+ try:
+ prompt = prompt['messages'][0]['content']
+ except (KeyError, IndexError, TypeError):
+ print("Failed to extract 'content' from the prompt.")
inputs = self.tokenizer(prompt, return_tensors="pt").to(self.device)
outputs = self.model.generate(**inputs, **kwargs)
if self.drop_prompt_from_output:
@@ -100,13 +130,13 @@ class HFModel(LM):
assert return_sorted is False, "for now"
if kwargs.get("n", 1) > 1:
- kwargs["num_beams"] = max(5, kwargs["n"])
+ kwargs["do_sample"] = True
response = self.request(prompt, **kwargs)
return [c["text"] for c in response["choices"]]
-if __name__ == "__main__":
- model = HFModel(model="google/flan-t5-base")
- response = model("Who was the first man to walk on the moon?\nFinal answer: ")
- print(response)
+# @functools.lru_cache(maxsize=None if cache_turn_on else 0)
+# @NotebookCacheMemory.cache
+# def cached_generate(self, prompt, **kwargs):
+# return self._generate(prompt, **kwargs)
diff --git a/dsp/modules/hf_client.py b/dsp/modules/hf_client.py
index 589676c7e..af4cb495f 100644
--- a/dsp/modules/hf_client.py
+++ b/dsp/modules/hf_client.py
@@ -1,19 +1,94 @@
+# import requests
+
+# from dsp.modules.hf import HFModel
+
+
+# class HFModelClient(HFModel):
+# def __init__(self, port, model, url="http://0.0.0.0"):
+# super().__init__(model=model, is_client=True)
+# self.url = f"{url}:{port}"
+# self.headers = {"Content-Type": "application/json; charset=utf-8"}
+
+# def _generate(self, prompt, **kwargs):
+# payload = {"prompt": prompt, **kwargs}
+# response = requests.post(self.url, json=payload, headers=self.headers)
+# try:
+# return response.json()
+# except:
+# print("Failed to parse JSON response:", response.text)
+# raise Exception("Received invalid JSON response from server")
+
+
+import dsp
+
import requests
+from dsp.modules.hf import HFModel, openai_to_hf
+from dsp.modules.cache_utils import CacheMemory, NotebookCacheMemory, cache_turn_on
-from dsp.modules.hf import HFModel
+# from dsp.modules.adapter import TurboAdapter, DavinciAdapter, LlamaAdapter
-class HFModelClient(HFModel):
- def __init__(self, port, model, url="http://0.0.0.0"):
+class HFClientTGI(HFModel):
+ def __init__(self, model, port, url="http://future-hgx-1", **kwargs):
super().__init__(model=model, is_client=True)
self.url = f"{url}:{port}"
- self.headers = {"Content-Type": "application/json; charset=utf-8"}
+ self.headers = {"Content-Type": "application/json"}
+
+ self.kwargs = {
+ "temperature": 0.1,
+ "max_tokens": 75,
+ "top_p": 0.97,
+ "n": 1,
+ "stop": ["\n", "\n\n"],
+ **kwargs,
+ }
def _generate(self, prompt, **kwargs):
- payload = {"prompt": prompt, **kwargs}
- response = requests.post(self.url, json=payload, headers=self.headers)
+ kwargs = {**self.kwargs, **kwargs}
+
+ payload = {
+ "inputs": prompt,
+ "parameters": {
+ "do_sample": kwargs["n"] > 1,
+ "best_of": kwargs["n"],
+ "details": kwargs["n"] > 1,
+ # "max_new_tokens": kwargs.get('max_tokens', kwargs.get('max_new_tokens', 75)),
+ "stop": ["\n", "\n\n"],
+ **kwargs,
+ }
+ }
+
+ payload['parameters'] = openai_to_hf(**payload['parameters'])
+
+ payload['parameters']['temperature'] = max(0.1, payload['parameters']['temperature'])
+
+ # print(payload['parameters'])
+
+ # response = requests.post(self.url + "/generate", json=payload, headers=self.headers)
+ response = send_hftgi_request_v00(self.url + "/generate", json=payload, headers=self.headers)
+
try:
- return response.json()
- except:
- print("Failed to parse JSON response:", response.text)
- raise Exception("Received invalid JSON response from server")
+ json_response = response.json()
+ # completions = json_response["generated_text"]
+
+ completions = [json_response['generated_text']]
+
+ if 'details' in json_response and 'best_of_sequences' in json_response['details']:
+ completions += [x['generated_text'] for x in json_response['details']['best_of_sequences']]
+
+ response = {
+ "prompt": prompt,
+ "choices": [
+ {"text": c} for c in completions
+ ]
+ }
+ return response
+ except Exception as e:
+ print("Failed to parse JSON response:", response.text)
+ raise Exception("Received invalid JSON response from server")
+
+
+
+def send_hftgi_request_v00(arg, **kwargs):
+ return requests.post(arg, **kwargs)
diff --git a/dsp/modules/hf_server.py b/dsp/modules/hf_server.py
index 60ae2cad5..a5c95c5e4 100644
--- a/dsp/modules/hf_server.py
+++ b/dsp/modules/hf_server.py
@@ -1,74 +1,74 @@
-# To Run:
-# python -m dsp.modules.hf_server --port 4242 --model "google/flan-t5-base"
-
-# To Query:
-# curl -d '{"prompt":".."}' -X POST "http://0.0.0.0:4242" -H 'Content-Type: application/json'
-# Or use the HF client. TODO: Add support for kwargs to the server.
-
-
-from functools import lru_cache
-import argparse
-import time
-import random
-import os
-import sys
-import uvicorn
-import warnings
-
-from fastapi import FastAPI
-from pydantic import BaseModel
-from argparse import ArgumentParser
-from starlette.middleware.cors import CORSMiddleware
-
-from dsp.modules.hf import HFModel
-
-
-class Query(BaseModel):
- prompt: str
- kwargs: dict = {}
-
-
-warnings.filterwarnings("ignore")
-
-app = FastAPI()
-app.add_middleware(
- CORSMiddleware, allow_origins=["*"], allow_methods=["*"], allow_headers=["*"]
-)
-
-parser = argparse.ArgumentParser("Server for Hugging Face models")
-parser.add_argument("--port", type=int, required=True, help="Server port")
-parser.add_argument("--model", type=str, required=True, help="Hugging Face model")
-args = parser.parse_args()
-# TODO: Convert this to a log message
-print(f"#> Loading the language model {args.model}")
-lm = HFModel(args.model)
-
-
-@lru_cache(maxsize=None)
-def generate(prompt, **kwargs):
- global lm
- generateStart = time.time()
- # TODO: Convert this to a log message
- print(f'#> kwargs: "{kwargs}" (type={type(kwargs)})')
- response = lm._generate(prompt, **kwargs)
- # TODO: Convert this to a log message
- print(f'#> Response: "{response}"')
- latency = (time.time() - generateStart) * 1000.0
- response["latency"] = latency
- print(f'#> Latency:', '{:.3f}'.format(latency / 1000.0), 'seconds')
- return response
-
-
-async def generate_post(query: Query):
- return generate(query.prompt, **query.kwargs)
-
-
-if __name__ == "__main__":
- uvicorn.run(
- app,
- host="0.0.0.0",
- port=args.port,
- reload=False,
- log_level="info",
- ) # can make reload=True later
+# # To Run:
+# # python -m dsp.modules.hf_server --port 4242 --model "google/flan-t5-base"
+
+# # To Query:
+# # curl -d '{"prompt":".."}' -X POST "http://0.0.0.0:4242" -H 'Content-Type: application/json'
+# # Or use the HF client. TODO: Add support for kwargs to the server.
+
+
+# from functools import lru_cache
+# import argparse
+# import time
+# import random
+# import os
+# import sys
+# import uvicorn
+# import warnings
+
+# from fastapi import FastAPI
+# from pydantic import BaseModel
+# from argparse import ArgumentParser
+# from starlette.middleware.cors import CORSMiddleware
+
+# from dsp.modules.hf import HFModel
+
+
+# class Query(BaseModel):
+# prompt: str
+# kwargs: dict = {}
+
+
+# warnings.filterwarnings("ignore")
+
+# app = FastAPI()
+# app.add_middleware(
+# CORSMiddleware, allow_origins=["*"], allow_methods=["*"], allow_headers=["*"]
+# )
+
+# parser = argparse.ArgumentParser("Server for Hugging Face models")
+# parser.add_argument("--port", type=int, required=True, help="Server port")
+# parser.add_argument("--model", type=str, required=True, help="Hugging Face model")
+# args = parser.parse_args()
+# # TODO: Convert this to a log message
+# print(f"#> Loading the language model {args.model}")
+# lm = HFModel(args.model)
+
+
+# @lru_cache(maxsize=None)
+# def generate(prompt, **kwargs):
+# global lm
+# generateStart = time.time()
+# # TODO: Convert this to a log message
+# print(f'#> kwargs: "{kwargs}" (type={type(kwargs)})')
+# response = lm._generate(prompt, **kwargs)
+# # TODO: Convert this to a log message
+# print(f'#> Response: "{response}"')
+# latency = (time.time() - generateStart) * 1000.0
+# response["latency"] = latency
+# print(f'#> Latency:', '{:.3f}'.format(latency / 1000.0), 'seconds')
+# return response
+
+
+# @app.post("/")
+# async def generate_post(query: Query):
+# return generate(query.prompt, **query.kwargs)
+
+
+# if __name__ == "__main__":
+# uvicorn.run(
+# app,
+# host="0.0.0.0",
+# port=args.port,
+# reload=False,
+# log_level="info",
+# ) # can make reload=True later
diff --git a/dsp/modules/lm.py b/dsp/modules/lm.py
index 9524694f3..592d0585e 100644
--- a/dsp/modules/lm.py
+++ b/dsp/modules/lm.py
@@ -65,7 +65,7 @@ class LM(ABC):
if provider == "cohere":
text = choices[0].text
elif provider == "openai":
- text = self._get_choice_text(choices[0])
+ text = ' ' + self._get_choice_text(choices[0]).strip()
else:
text = choices[0]["text"]
self.print_green(text, end="")
@@ -77,3 +77,7 @@ class LM(ABC):
@abstractmethod
def __call__(self, prompt, only_completed=True, return_sorted=False, **kwargs):
pass
+
+ def copy(self, **kwargs):
+ """Returns a copy of the language model with the same parameters."""
+ return self.__class__(self.kwargs['model'], **{**self.kwargs, **kwargs})
diff --git a/dsp/primitives/demonstrate.py b/dsp/primitives/demonstrate.py
index 5265b1867..525314105 100644
--- a/dsp/primitives/demonstrate.py
+++ b/dsp/primitives/demonstrate.py
@@ -55,7 +55,7 @@ def annotate(*transformations):
for example in train: # tqdm.tqdm
raw_example = dsp.Example(example)
- if k and len(ademos) >= k:
+ if (k is not None) and len(ademos) >= k:
example = None
for f in transformations:
@@ -82,7 +82,7 @@ def annotate(*transformations):
def sample(train: list[Example], k: int):
"""Sample k examples from train."""
rng = random.Random(dsp.settings.branch_idx)
- shuffled_train = [example.copy() for example in train]
+ shuffled_train = [dsp.Example(example) for example in train]
rng.shuffle(shuffled_train)
return shuffled_train[:k]
diff --git a/dsp/primitives/predict.py b/dsp/primitives/predict.py
index 70b9193fc..6f442f643 100644
--- a/dsp/primitives/predict.py
+++ b/dsp/primitives/predict.py
@@ -1,5 +1,6 @@
from collections import Counter
from typing import Callable, Any, Optional
+
import dsp
from dsp.utils import zipstar, normalize_text
from dsp.primitives.inspect import FuncInspector
@@ -177,6 +178,7 @@ def extract_final_answer(example, completions, extract=None):
p.strip().split("\n")[-1].split(":", 1)[-1].strip() for p in completions
]
+ # TODO: make thread-safe?
dsp.settings.lm.history.append(
{**dsp.settings.lm.history[-1], "completions": completions}
)
diff --git a/dsp/primitives/search.py b/dsp/primitives/search.py
index 3a8c6c8ab..833c91494 100644
--- a/dsp/primitives/search.py
+++ b/dsp/primitives/search.py
@@ -43,8 +43,12 @@ def retrieveEnsemble(queries: list[str], k: int, by_prob: bool = True) -> list[s
raise AssertionError("No RM is loaded.")
if dsp.settings.reranker:
return retrieveRerankEnsemble(queries, k)
+
queries = [q for q in queries if q]
+ if len(queries) == 1:
+ return retrieve(queries[0], k)
+
passages = {}
for q in queries:
for psg in dsp.settings.rm(q, k=k * 3):
diff --git a/dsp/templates/__init__.py b/dsp/templates/__init__.py
index 240ce9424..b4aed7643 100644
--- a/dsp/templates/__init__.py
+++ b/dsp/templates/__init__.py
@@ -1,7 +1,4 @@
from .utils import *
from .template_v2 import *
from .template_v3 import *
-# from .template import *
-# from .qa import *
-# from .first_hop import *
-# from .nth_hop import *
+
diff --git a/dsp/templates/template_v2.py b/dsp/templates/template_v2.py
index 0b0dcaf1a..0a5e73fec 100644
--- a/dsp/templates/template_v2.py
+++ b/dsp/templates/template_v2.py
@@ -94,16 +94,17 @@ class TemplateV2:
def format_handler(x):
return " ".join(x.split())
+
+ formatted_value = format_handler(example[field.input_variable])
+ separator = '\n' if field.separator == ' ' and '\n' in formatted_value else field.separator
result.append(
- f"{field.name}{field.separator}{format_handler(example[field.input_variable])}"
+ f"{field.name}{separator}{formatted_value}"
)
- if self._has_augmented_guidelines() and (
- "augmented" in example and example.augmented
- ):
- return "\n\n".join(result)
- return "\n".join(result)
+ if self._has_augmented_guidelines() and ("augmented" in example and example.augmented):
+ return "\n\n".join([r for r in result if r])
+ return "\n".join([r for r in result if r])
def guidelines(self, show_guidelines=True) -> str:
"""Returns the task guidelines as described in the lm prompt"""
@@ -125,7 +126,7 @@ class TemplateV2:
def _has_augmented_guidelines(self):
return len(self.fields) > 3 or any(
- field.separator == "\n" for field in self.fields
+ ("\n" in field.separator) or ('\n' in field.description) for field in self.fields
)
def extract(
@@ -203,10 +204,36 @@ class TemplateV2:
if "augmented" in demo and demo.augmented
]
+ # Move the rdemos to ademos if rdemo has all the fields filled in
+ rdemos_ = []
+ for rdemo in rdemos:
+ if all(
+ field.name in rdemo
+ for field in self.fields
+ if field.input_variable in example
+ ):
+ ademos.append(rdemo)
+ else:
+ rdemos_.append(rdemo)
+
+ rdemos = rdemos_
+
+
long_query = self._has_augmented_guidelines()
+
if long_query:
example["augmented"] = True
+
query = self.query(example)
+
+ # if it has more lines than fields
+ if len(query.split('\n')) > len(self.fields):
+ long_query = True
+
+ if "augmented" not in example or not example.augmented:
+ example["augmented"] = True
+ query = self.query(example)
+
rdemos = "\n\n".join(rdemos)
if len(rdemos) >= 1 and len(ademos) == 0 and not long_query:
rdemos_and_query = "\n\n".join([rdemos, query])
diff --git a/dsp/templates/template_v3.py b/dsp/templates/template_v3.py
index d0e67e250..9fec3c036 100644
--- a/dsp/templates/template_v3.py
+++ b/dsp/templates/template_v3.py
@@ -5,15 +5,10 @@ from dsp.templates import TemplateV2, passages2text, format_answers, Field
class Type:
"""A primitive datatype that defines and represents a prompt label."""
- def __init__(self, prefix: str, desc: str, format=None, aliases=[]) -> None:
+ def __init__(self, prefix: str, desc: str, format=None) -> None:
self.prefix = prefix
self.desc = desc
self.format = format
- self.aliases = aliases
-
- assert (
- aliases == []
- ), "TODO: If aliases is used, handling of input_variable in V2 needs to be richer."
def __call__(self, **kwargs):
kwargs = {**self.__dict__, **kwargs}
@@ -25,6 +20,7 @@ class Template(TemplateV2):
def __init__(self, instructions: str, **kwargs):
self.instructions = instructions
+ self.kwargs = kwargs
self.fields: list[Field] = []
self.format_handlers: dict[str, Callable] = {
@@ -35,7 +31,7 @@ class Template(TemplateV2):
for key, value in kwargs.items():
prefix: str = value.prefix
separator: str = (
- " " if prefix.rstrip() == prefix else prefix[len(prefix.rstrip()) :]
+ " " if prefix.rstrip() == prefix and len(prefix) > 0 else prefix[len(prefix.rstrip()) :]
)
field = Field(
name=prefix.strip(),
@@ -47,5 +43,4 @@ class Template(TemplateV2):
self.fields.append(field)
if value.format:
- for name in [key, *value.aliases]:
- self.format_handlers[name] = value.format
+ self.format_handlers[key] = value.format
diff --git a/dsp/templates/utils.py b/dsp/templates/utils.py
index e21423bef..0b0d4891f 100644
--- a/dsp/templates/utils.py
+++ b/dsp/templates/utils.py
@@ -8,12 +8,40 @@ def passages2text(passages: Union[str, list, tuple]) -> str:
assert type(passages) in [list, tuple]
+ if len(passages) == 0:
+ return "N/A"
+
if len(passages) == 1:
return f"«{passages[0]}»"
return "\n".join([f"[{idx+1}] «{txt}»" for idx, txt in enumerate(passages)])
+def passages2textV2(passages: Union[str, list, tuple]) -> str:
+ """Formats the given one or more passages into a single structured string."""
+ if isinstance(passages, str):
+ return passages
+
+ assert type(passages) in [list, tuple]
+
+ def psg2text(psg):
+ try:
+ title, snippet = psg.split("|", 1)
+ return f"Title: {title.strip()} | Snippet: «{snippet.strip()}»"
+ except Exception:
+ pass
+
+ return f"«{psg}»"
+
+ if len(passages) == 0:
+ return "N/A"
+
+ if len(passages) == 1:
+ return psg2text(passages[0])
+
+ return "\n".join([f"[{idx+1}] {psg2text(txt)}" for idx, txt in enumerate(passages)])
+
+
def format_answers(answers: Union[str, list]) -> Optional[str]:
"""Parses the given answers and returns the appropriate answer string.
diff --git a/dsp/utils/dpr.py b/dsp/utils/dpr.py
index f268d85be..9ee24dd5a 100644
--- a/dsp/utils/dpr.py
+++ b/dsp/utils/dpr.py
@@ -1,6 +1,7 @@
"""
Source: DPR Implementation from Facebook Research
https://github.com/facebookresearch/DPR/tree/master/dpr
+ Original license: https://github.com/facebookresearch/DPR/blob/main/LICENSE
"""
import string
diff --git a/dsp/utils/settings.py b/dsp/utils/settings.py
index 5457cf0cb..3d27eb384 100644
--- a/dsp/utils/settings.py
+++ b/dsp/utils/settings.py
@@ -1,12 +1,11 @@
from contextlib import contextmanager
from dsp.utils.utils import dotdict
-
+import threading
class Settings(object):
"""DSP configuration settings."""
_instance = None
- branch_idx: int = 0
def __new__(cls):
"""
@@ -15,7 +14,10 @@ class Settings(object):
if cls._instance is None:
cls._instance = super().__new__(cls)
- cls._instance.stack = []
+ cls._instance.main_tid = threading.get_ident()
+ cls._instance.main_stack = []
+ cls._instance.stack_by_thread = {}
+ cls._instance.stack_by_thread[threading.get_ident()] = cls._instance.main_stack
# TODO: remove first-class support for re-ranker and potentially combine with RM to form a pipeline of sorts
# eg: RetrieveThenRerankPipeline(RetrievalModel, Reranker)
@@ -23,10 +25,13 @@ class Settings(object):
config = dotdict(
lm=None,
rm=None,
+ branch_idx=0,
reranker=None,
compiled_lm=None,
force_reuse_cached_compilation=False,
compiling=False,
+ skip_logprobs=False,
+ trace=None
)
cls._instance.__append(config)
@@ -34,7 +39,7 @@ class Settings(object):
@property
def config(self):
- return self.stack[-1]
+ return self.stack_by_thread[threading.get_ident()][-1]
def __getattr__(self, name):
if hasattr(self.config, name):
@@ -46,10 +51,10 @@ class Settings(object):
super().__getattr__(name)
def __append(self, config):
- self.stack.append(config)
+ self.stack_by_thread[threading.get_ident()].append(config)
def __pop(self):
- self.stack.pop()
+ self.stack_by_thread[threading.get_ident()].pop()
def configure(self, inherit_config: bool = True, **kwargs):
"""Set configuration settings.
diff --git a/dsp/utils/settings_v2.py b/dsp/utils/settings_v2.py
new file mode 100644
index 000000000..2998ca7cd
--- /dev/null
+++ b/dsp/utils/settings_v2.py
@@ -0,0 +1,81 @@
+import threading
+from contextlib import contextmanager
+from concurrent.futures import ThreadPoolExecutor, as_completed
+import copy
+
+class Settings:
+ def __init__(self):
+ # A lock for ensuring thread-safety when accessing _parent_configs
+ self._lock = threading.Lock()
+
+ # Dictionary to hold parent thread configurations
+ self._parent_configs = {}
+
+ # Using thread-local storage to ensure that each thread has its own configuration stack
+ self._local = threading.local()
+
+ def _get_current_config(self):
+ return self._local.config_stack[-1] if hasattr(self._local, 'config_stack') and self._local.config_stack else {}
+
+ def initialize_for_thread(self, parent_tid):
+ """Initialize thread-local data for a new thread using its parent's config."""
+ with self._lock:
+ parent_config = self._parent_configs.get(parent_tid)
+ if parent_config:
+ self._local.config_stack = [copy.deepcopy(parent_config)]
+ else:
+ self._local.config_stack = [{}]
+
+ @contextmanager
+ def context(self, **kwargs):
+ current_config = copy.deepcopy(self._get_current_config()) # Deep copy the current configuration
+ current_config.update(kwargs)
+
+ if not hasattr(self._local, 'config_stack'):
+ self._local.config_stack = []
+
+ self._local.config_stack.append(current_config)
+
+ # Register the modified config as the potential parent config
+ with self._lock:
+ self._parent_configs[threading.get_ident()] = copy.deepcopy(current_config) # Deep copy to ensure immutability
+
+ try:
+ yield
+ finally:
+ self._local.config_stack.pop()
+
+ # Cleanup after exiting the context
+ with self._lock:
+ self._parent_configs.pop(threading.get_ident(), None)
+
+# Singleton instance
+dsp_settings = Settings()
+
+
+# Wrapper for ThreadPoolExecutor usage
+def thread_wrapper(program, parent_tid, *args, **kwargs):
+ dsp_settings.initialize_for_thread(parent_tid)
+ return program(*args, **kwargs)
+
+
+# Example test
+def sample_program(arg):
+ print(f"Thread {threading.get_ident()} with arg={arg} has config: {dsp_settings._get_current_config()}")
+
+
+def main():
+ parent_tid = threading.get_ident()
+
+ with dsp_settings.context(a=10, b=20): # Setting main thread's context
+ with ThreadPoolExecutor(max_workers=2) as executor:
+ futures = {executor.submit(thread_wrapper, sample_program, parent_tid, arg) for arg in range(3)}
+
+ for future in as_completed(futures):
+ res = future.result()
+
+ print(f"Main thread {parent_tid} config after threads: {dsp_settings._get_current_config()}")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/dsp/utils/utils.py b/dsp/utils/utils.py
index f4a738ce5..7e93933e4 100644
--- a/dsp/utils/utils.py
+++ b/dsp/utils/utils.py
@@ -64,15 +64,43 @@ def batch(group, bsize, provide_offset=False):
return
+# class dotdict(dict):
+# """
+# dot.notation access to dictionary attributes
+# Credit: derek73 @ https://stackoverflow.com/questions/2352181
+# """
+
+# __getattr__ = dict.__getitem__
+# __setattr__ = dict.__setitem__
+# __delattr__ = dict.__delitem__
+
+
+import copy
+
class dotdict(dict):
- """
- dot.notation access to dictionary attributes
- Credit: derek73 @ https://stackoverflow.com/questions/2352181
- """
+ def __getattr__(self, key):
+ if key.startswith('__') and key.endswith('__'):
+ return super().__getattr__(key)
+ try:
+ return self[key]
+ except KeyError:
+ raise AttributeError(f"'{type(self).__name__}' object has no attribute '{key}'")
+
+ def __setattr__(self, key, value):
+ if key.startswith('__') and key.endswith('__'):
+ super().__setattr__(key, value)
+ else:
+ self[key] = value
- __getattr__ = dict.__getitem__
- __setattr__ = dict.__setitem__
- __delattr__ = dict.__delitem__
+ def __delattr__(self, key):
+ if key.startswith('__') and key.endswith('__'):
+ super().__delattr__(key)
+ else:
+ del self[key]
+
+ def __deepcopy__(self, memo):
+ # Use the default dict copying method to avoid infinite recursion.
+ return dotdict(copy.deepcopy(dict(self), memo))
class dotdict_lax(dict):
diff --git a/demos/Multi-hop QA Program 1 - Direct Query.ipynb b/dsp_v1_demos/Multi-hop QA Program 1 - Direct Query.ipynb
similarity index 100%
rename from demos/Multi-hop QA Program 1 - Direct Query.ipynb
rename to dsp_v1_demos/Multi-hop QA Program 1 - Direct Query.ipynb
diff --git a/demos/Multi-hop QA Program 2 - Retrieve then Read.ipynb b/dsp_v1_demos/Multi-hop QA Program 2 - Retrieve then Read.ipynb
similarity index 100%
rename from demos/Multi-hop QA Program 2 - Retrieve then Read.ipynb
rename to dsp_v1_demos/Multi-hop QA Program 2 - Retrieve then Read.ipynb
diff --git a/demos/Multi-hop QA Program 3 - Retrieve then Read with Self-Consistency.ipynb b/dsp_v1_demos/Multi-hop QA Program 3 - Retrieve then Read with Self-Consistency.ipynb
similarity index 100%
rename from demos/Multi-hop QA Program 3 - Retrieve then Read with Self-Consistency.ipynb
rename to dsp_v1_demos/Multi-hop QA Program 3 - Retrieve then Read with Self-Consistency.ipynb
diff --git a/demos/Multi-hop QA Program 4 - Multi-hop Retrieval.ipynb b/dsp_v1_demos/Multi-hop QA Program 4 - Multi-hop Retrieval.ipynb
similarity index 100%
rename from demos/Multi-hop QA Program 4 - Multi-hop Retrieval.ipynb
rename to dsp_v1_demos/Multi-hop QA Program 4 - Multi-hop Retrieval.ipynb
diff --git a/demos/Multi-hop QA Program 5 - Multi-hop Condensed Retrieval with Automatic Demos and Query Fusion.ipynb b/dsp_v1_demos/Multi-hop QA Program 5 - Multi-hop Condensed Retrieval with Automatic Demos and Query Fusion.ipynb
similarity index 100%
rename from demos/Multi-hop QA Program 5 - Multi-hop Condensed Retrieval with Automatic Demos and Query Fusion.ipynb
rename to dsp_v1_demos/Multi-hop QA Program 5 - Multi-hop Condensed Retrieval with Automatic Demos and Query Fusion.ipynb
diff --git a/compiler.ipynb b/dsp_v1_demos/compiler.ipynb
similarity index 100%
rename from compiler.ipynb
rename to dsp_v1_demos/compiler.ipynb
diff --git a/dsp_v1_demos/intro.ipynb b/dsp_v1_demos/intro.ipynb
new file mode 100644
index 000000000..8fd0704d1
--- /dev/null
+++ b/dsp_v1_demos/intro.ipynb
@@ -0,0 +1,2289 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 🎓 𝗗𝗦𝗣: 𝗗emonstrate–𝗦earch–𝗣redict\n",
+ "\n",
+ "This notebook illustrates the Demonstrate–Search–Predict (𝗗𝗦𝗣) framework on the multi-hop question answering (QA) task.\n",
+ "\n",
+ "Below, we'll build **five** small 𝗗𝗦𝗣 programs that implement strategies for this task. The programs range from extremely simple ones (programs 1 and 2) to moderate ones (programs 3 and 4). Program 5 considers a fairly advanced pipeline."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Installation\n",
+ "\n",
+ "If you haven't installed **DSP** already, let's do that."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "try: # When on google Colab, let's clone the notebook so we download the cache.\n",
+ " import google.colab \n",
+ " !git -C dsp/ pull || git clone https://github.com/stanfordnlp/dsp\n",
+ "except: pass\n",
+ "\n",
+ "!pip install -U pip dsp-ml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Setting Up\n",
+ "\n",
+ "We'll start by setting up the language model (LM) and retrieval model (RM).\n",
+ "\n",
+ "We will work with the **GPT-3.5** LM (`text-davinci-002`) and the **ColBERTv2** RM.\n",
+ "\n",
+ "To use GPT-3, you'll need an OpenAI key. For ColBERTv2, we've set up a server hosting a Wikipedia (Dec 2018) search index, so you don't need to worry about setting one up!\n",
+ "\n",
+ "To make things easy, we've set up a cache in this repository. _If you want to run this notebook without changing the code or examples, you don't need an API key. All examples are cached._"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "%load_ext autoreload\n",
+ "%autoreload 2\n",
+ "\n",
+ "try: import google.colab; root_path = 'dsp'\n",
+ "except: root_path = '.'\n",
+ "\n",
+ "import os\n",
+ "os.environ[\"DSP_NOTEBOOK_CACHEDIR\"] = os.path.join(root_path, 'cache')\n",
+ "\n",
+ "import dsp\n",
+ "\n",
+ "openai_key = os.getenv('OPENAI_API_KEY') # or replace with your API key (optional)\n",
+ "colbert_server = 'http://ec2-44-228-128-229.us-west-2.compute.amazonaws.com:8893/api/search'\n",
+ "\n",
+ "lm = dsp.GPT3(model='text-davinci-002', api_key=openai_key)\n",
+ "rm = dsp.ColBERTv2(url=colbert_server)\n",
+ "\n",
+ "dsp.settings.configure(lm=lm, rm=rm)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Task Examples\n",
+ "\n",
+ "Next, let's look at a few examples of the task. Each example consists of a question and one or more gold answers.\n",
+ "\n",
+ "We have six training examples (`train`), which we'll feed into the programs. These will help define the task.\n",
+ "\n",
+ "Notice that our examples only have input (`question`) and output (`answer`) fields. When our advanced programs build sophisticated pipelines, training \"demonstrations\" for other fields will be constructed automatically."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "train = [('Who produced the album that included a re-recording of \"Lithium\"?', ['Butch Vig']),\n",
+ " ('Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?', ['Kevin Greutert']),\n",
+ " ('The heir to the Du Pont family fortune sponsored what wrestling team?', ['Foxcatcher', 'Team Foxcatcher', 'Foxcatcher Team']),\n",
+ " ('In what year was the star of To Hell and Back born?', ['1925']),\n",
+ " ('Which award did the first book of Gary Zukav receive?', ['U.S. National Book Award', 'National Book Award']),\n",
+ " ('What city was the victim of Joseph Druces working in?', ['Boston, Massachusetts', 'Boston']),]\n",
+ "\n",
+ "train = [dsp.Example(question=question, answer=answer) for question, answer in train]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "The development examples (`dev`) will be used to assess the behavior of each program we build. Of course, this tiny set is not meant to be a reliable benchmark, but it'll be instructive to use it for illustration."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "dev = [('Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?', ['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']),\n",
+ " ('What documentary about the Gilgo Beach Killer debuted on A&E?', ['The Killing Season']),\n",
+ " ('Right Back At It Again contains lyrics co-written by the singer born in what city?', ['Gainesville, Florida', 'Gainesville']),\n",
+ " ('What year was the party of the winner of the 1971 San Francisco mayoral election founded?', ['1828']),\n",
+ " ('Which author is English: John Braine or Studs Terkel?', ['John Braine']),\n",
+ " ('Anthony Dirrell is the brother of which super middleweight title holder?', ['Andre Dirrell']),\n",
+ " ('In which city is the sports nutrition business established by Oliver Cookson based ?', ['Cheshire', 'Cheshire, UK']),\n",
+ " ('Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.', ['February 13, 1980']),\n",
+ " ('Kyle Moran was born in the town on what river?', ['Castletown', 'Castletown River']),\n",
+ " (\"What is the name of one branch of Robert D. Braun's speciality?\", ['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']),\n",
+ " (\"Where was the actress who played the niece in the Priest film born?\", ['Surrey', 'Guildford, Surrey']),\n",
+ " ('Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.', ['Portrait of a Marriage']),\n",
+ " ('What year was the father of the Princes in the Tower born?', ['1442'])]\n",
+ "\n",
+ "dev = [dsp.Example(question=question, answer=answer) for question, answer in dev]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Program 1: Vanilla GPT-3.5\n",
+ "\n",
+ "Let's begin by implementing the simplest program for this task. We'll prompt **GPT-3.5** to answer each question based on its internal parameteric knowledge.\n",
+ "\n",
+ "We'll start by defining the `Template` that defines how we will communicate with the LM.\n",
+ "\n",
+ "Specifically, the question–answer template (`qa_template`) will include a question and a short answer for each example."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "Question = dsp.Type(prefix=\"Question:\", desc=\"${the question to be answered}\")\n",
+ "Answer = dsp.Type(prefix=\"Answer:\", desc=\"${a short factoid answer, often between 1 and 5 words}\", format=dsp.format_answers)\n",
+ "\n",
+ "qa_template = dsp.Template(instructions=\"Answer questions with short factoid answers.\", question=Question(), answer=Answer())"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Then, let's define the actual program, `vanilla_LM_QA`. It'll accept a string (`question`) and returns another string (its short `answer`)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def vanilla_LM_QA(question: str) -> str:\n",
+ " demos = dsp.sample(train, k=7)\n",
+ " example = dsp.Example(question=question, demos=demos)\n",
+ "\n",
+ " example, completions = dsp.generate(qa_template)(example, stage='qa')\n",
+ " return completions.answer"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's invoke the program on a sample question."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?\n",
+ "E. L. Doctorow\n"
+ ]
+ }
+ ],
+ "source": [
+ "print(dev[0].question)\n",
+ "print(vanilla_LM_QA(dev[0].question))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's inspect the last call to the LM to learn more about the behavior of the program."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "Answer questions with short factoid answers.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Question: ${the question to be answered}\n",
+ "Answer: ${a short factoid answer, often between 1 and 5 words}\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Question: Which award did the first book of Gary Zukav receive?\n",
+ "Answer: U.S. National Book Award\n",
+ "\n",
+ "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
+ "Answer: Foxcatcher\n",
+ "\n",
+ "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
+ "Answer: Kevin Greutert\n",
+ "\n",
+ "Question: Who produced the album that included a re-recording of \"Lithium\"?\n",
+ "Answer: Butch Vig\n",
+ "\n",
+ "Question: What city was the victim of Joseph Druces working in?\n",
+ "Answer: Boston, Massachusetts\n",
+ "\n",
+ "Question: In what year was the star of To Hell and Back born?\n",
+ "Answer: 1925\n",
+ "\n",
+ "Question: Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?\n",
+ "Answer:\u001b[32m E. L. Doctorow\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n"
+ ]
+ }
+ ],
+ "source": [
+ "lm.inspect_history(n=1)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We can see that the program has organized the examples in a simple prompt and generated an answer for the `dev[0]` question we asked.\n",
+ "\n",
+ "Let's try on the remaining development examples."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "100%|██████████| 13/13 [00:00<00:00, 456.27it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Answered 3 / 13 (23.1%) correctly.\n"
+ ]
+ },
+ {
+ "data": {
+ "text/html": [
+ "<style type=\"text/css\" >\n",
+ " #T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2 th {\n",
+ " text-align: left;\n",
+ " } #T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2 td {\n",
+ " text-align: left;\n",
+ " }</style><table id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2\" ><thead> <tr> <th class=\"blank level0\" ></th> <th class=\"col_heading level0 col0\" >question</th> <th class=\"col_heading level0 col1\" >answer</th> <th class=\"col_heading level0 col2\" >prediction</th> <th class=\"col_heading level0 col3\" >correct</th> </tr></thead><tbody>\n",
+ " <tr>\n",
+ " <th id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
+ " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",
+ " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",
+ " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row0_col2\" class=\"data row0 col2\" >E. L. Doctorow</td>\n",
+ " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row0_col3\" class=\"data row0 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
+ " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",
+ " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",
+ " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row1_col2\" class=\"data row1 col2\" >The Long Island Serial Killer</td>\n",
+ " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row1_col3\" class=\"data row1 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
+ " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",
+ " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",
+ " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row2_col2\" class=\"data row2 col2\" >Melbourne, Australia</td>\n",
+ " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row2_col3\" class=\"data row2 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
+ " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",
+ " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",
+ " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row3_col2\" class=\"data row3 col2\" >1966</td>\n",
+ " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row3_col3\" class=\"data row3 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
+ " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",
+ " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",
+ " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row4_col2\" class=\"data row4 col2\" >John Braine</td>\n",
+ " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row4_col3\" class=\"data row4 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
+ " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",
+ " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",
+ " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",
+ " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row5_col3\" class=\"data row5 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
+ " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",
+ " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",
+ " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row6_col2\" class=\"data row6 col2\" >Manchester, England</td>\n",
+ " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row6_col3\" class=\"data row6 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
+ " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",
+ " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",
+ " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row7_col2\" class=\"data row7 col2\" >July 30, 1953</td>\n",
+ " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row7_col3\" class=\"data row7 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
+ " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",
+ " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",
+ " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row8_col2\" class=\"data row8 col2\" >Hudson River</td>\n",
+ " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row8_col3\" class=\"data row8 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
+ " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",
+ " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",
+ " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row9_col2\" class=\"data row9 col2\" >Aerospace engineering</td>\n",
+ " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row9_col3\" class=\"data row9 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
+ " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",
+ " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",
+ " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row10_col2\" class=\"data row10 col2\" >Hong Kong</td>\n",
+ " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row10_col3\" class=\"data row10 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
+ " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",
+ " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",
+ " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row11_col2\" class=\"data row11 col2\" >Venus</td>\n",
+ " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row11_col3\" class=\"data row11 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
+ " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",
+ " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",
+ " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row12_col2\" class=\"data row12 col2\" >1457</td>\n",
+ " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row12_col3\" class=\"data row12 col3\" >❌</td>\n",
+ " </tr>\n",
+ " </tbody></table>"
+ ],
+ "text/plain": [
+ "<pandas.io.formats.style.Styler at 0x7f9bd91ce1c0>"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/plain": [
+ "23.1"
+ ]
+ },
+ "execution_count": 9,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "from dsp.evaluation.utils import evaluate\n",
+ "\n",
+ "evaluate(vanilla_LM_QA, dev)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Program 2: Retrieve-then-Read w/ GPT-3.5\n",
+ "\n",
+ "The program above relies entirely on the factual knowledge memorized by GPT-3. As we have seen, it's sometimes correct but it's far from reliable.\n",
+ "\n",
+ "Let's try to address this. Our first improvement to the vanilla **GPT-3.5** strategy is to include search results in the LM prompt, often called a *retrieve-then-read* approach.\n",
+ "\n",
+ "In particular, we'll retrieve the most relevant passage to the question (according to the retriever) from the corpus and concatenate it into the prompt. This may help GPT-3.5 answer more factually.\n",
+ "\n",
+ "To do this, we'll need to modify the `Template`. In particular, we'll define a `qa_template_with_passages` template, which will include a `context` field in addition to a question and an answer."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "Context = dsp.Type(\n",
+ " prefix=\"Context:\\n\",\n",
+ " desc=\"${sources that may contain relevant content}\",\n",
+ " format=dsp.passages2text\n",
+ ")\n",
+ "\n",
+ "qa_template_with_passages = dsp.Template(\n",
+ " instructions=qa_template.instructions,\n",
+ " context=Context(), question=Question(), answer=Answer()\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Now let's modify the program. We'll call `dsp.retrieve`, which we hooked earlier in this notebook to a **ColBERTv2** retriever serving a Wikipedia (Dec 2018) index."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 11,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def retrieve_then_read_QA(question: str) -> str:\n",
+ " demos = dsp.sample(train, k=7)\n",
+ " passages = dsp.retrieve(question, k=1)\n",
+ " \n",
+ " example = dsp.Example(question=question, context=passages, demos=demos)\n",
+ " example, completions = dsp.generate(qa_template_with_passages)(example, stage='qa')\n",
+ "\n",
+ " return completions.answer"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's invoke this on a sample question and let's inspect the last call to the LM to learn more about the behavior of the program."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 12,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "Answer questions with short factoid answers.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Question: Which award did the first book of Gary Zukav receive?\n",
+ "Answer: U.S. National Book Award\n",
+ "\n",
+ "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
+ "Answer: Foxcatcher\n",
+ "\n",
+ "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
+ "Answer: Kevin Greutert\n",
+ "\n",
+ "Question: Who produced the album that included a re-recording of \"Lithium\"?\n",
+ "Answer: Butch Vig\n",
+ "\n",
+ "Question: What city was the victim of Joseph Druces working in?\n",
+ "Answer: Boston, Massachusetts\n",
+ "\n",
+ "Question: In what year was the star of To Hell and Back born?\n",
+ "Answer: 1925\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context:\n",
+ "${sources that may contain relevant content}\n",
+ "\n",
+ "Question: ${the question to be answered}\n",
+ "\n",
+ "Answer: ${a short factoid answer, often between 1 and 5 words}\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context:\n",
+ "«Julia Peterkin | Julia Peterkin Julia Peterkin (October 31, 1880 – August 10, 1961) was an American author from South Carolina. In 1929 she won the Pulitzer Prize for Novel/Literature, for her novel \"Scarlet Sister Mary.\" She wrote several novels about the plantation South, especially the Gullah people of the Low Country. She was one of the few white authors who wrote about the African-American experience. Julia Mood was born in Laurens County, South Carolina. Her father was a physician, and she was the third of his four children. Her mother died soon after her birth, and her father later married Janie Brogdon.»\n",
+ "\n",
+ "Question: Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?\n",
+ "\n",
+ "Answer:\u001b[32m E. L. Doctorow\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "('E. L. Doctorow', None)"
+ ]
+ },
+ "execution_count": 12,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "retrieve_then_read_QA(dev[0].question), lm.inspect_history(n=1)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Now, let's evaluate this on the dev examples."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 13,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "100%|██████████| 13/13 [00:00<00:00, 376.92it/s]"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Answered 5 / 13 (38.5%) correctly.\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "\n"
+ ]
+ },
+ {
+ "data": {
+ "text/html": [
+ "<style type=\"text/css\" >\n",
+ " #T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2 th {\n",
+ " text-align: left;\n",
+ " } #T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2 td {\n",
+ " text-align: left;\n",
+ " }</style><table id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2\" ><thead> <tr> <th class=\"blank level0\" ></th> <th class=\"col_heading level0 col0\" >question</th> <th class=\"col_heading level0 col1\" >answer</th> <th class=\"col_heading level0 col2\" >prediction</th> <th class=\"col_heading level0 col3\" >correct</th> </tr></thead><tbody>\n",
+ " <tr>\n",
+ " <th id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
+ " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",
+ " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",
+ " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row0_col2\" class=\"data row0 col2\" >E. L. Doctorow</td>\n",
+ " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row0_col3\" class=\"data row0 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
+ " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",
+ " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",
+ " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row1_col2\" class=\"data row1 col2\" >The Killing Season</td>\n",
+ " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row1_col3\" class=\"data row1 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
+ " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",
+ " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",
+ " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row2_col2\" class=\"data row2 col2\" >Daytona Beach, Florida.</td>\n",
+ " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row2_col3\" class=\"data row2 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
+ " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",
+ " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",
+ " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row3_col2\" class=\"data row3 col2\" >1854</td>\n",
+ " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row3_col3\" class=\"data row3 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
+ " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",
+ " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",
+ " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row4_col2\" class=\"data row4 col2\" >John Braine is English.</td>\n",
+ " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row4_col3\" class=\"data row4 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
+ " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",
+ " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",
+ " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",
+ " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row5_col3\" class=\"data row5 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
+ " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",
+ " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",
+ " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row6_col2\" class=\"data row6 col2\" >Manchester, England</td>\n",
+ " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row6_col3\" class=\"data row6 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
+ " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",
+ " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",
+ " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row7_col2\" class=\"data row7 col2\" >August 5, 1962</td>\n",
+ " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row7_col3\" class=\"data row7 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
+ " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",
+ " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",
+ " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row8_col2\" class=\"data row8 col2\" >Dundalk, Ireland</td>\n",
+ " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row8_col3\" class=\"data row8 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
+ " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",
+ " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",
+ " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row9_col2\" class=\"data row9 col2\" >Jet Propulsion</td>\n",
+ " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row9_col3\" class=\"data row9 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
+ " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",
+ " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",
+ " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row10_col2\" class=\"data row10 col2\" >Enid, Oklahoma</td>\n",
+ " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row10_col3\" class=\"data row10 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
+ " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",
+ " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",
+ " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row11_col2\" class=\"data row11 col2\" >Portrait of a Marriage</td>\n",
+ " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row11_col3\" class=\"data row11 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
+ " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",
+ " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",
+ " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row12_col2\" class=\"data row12 col2\" >1442</td>\n",
+ " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row12_col3\" class=\"data row12 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " </tbody></table>"
+ ],
+ "text/plain": [
+ "<pandas.io.formats.style.Styler at 0x7f9bb40bea30>"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/plain": [
+ "38.5"
+ ]
+ },
+ "execution_count": 13,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "evaluate(retrieve_then_read_QA, dev)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Program 3: Retrieve-then-Read w/ Self-Consistency\n",
+ "\n",
+ "It's clear that retrieval has the capacity to help **GPT-3.5** answer a larger number of questions more factually. However, these questions are too complicated for a single retrieved passage to suffice.\n",
+ "\n",
+ "In this program, we try to make some improvements to the pipeline from **Program 2**.\n",
+ "\n",
+ "In particular:\n",
+ "\n",
+ "- We will include 5 passages (instead of a single passage) into the prompt.\n",
+ "- We will ask GPT-3.5 to generate a Chain-of-Thought (CoT) rationale to more effectively extract the answer from the passages.\n",
+ "- We will use Self-Consistency (SC) to marginalize this prediction across many chains of thought."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 14,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "Rationale = dsp.Type(\n",
+ " prefix=\"Rationale: Let's think step by step.\",\n",
+ " desc=\"${a step-by-step deduction that identifies the correct response, which will be provided below}\"\n",
+ ")\n",
+ "\n",
+ "qa_template_with_CoT = dsp.Template(\n",
+ " instructions=qa_template.instructions,\n",
+ " context=Context(), question=Question(), rationale=Rationale(), answer=Answer()\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "The snippet above updates our template to include a `rationale` field.\n",
+ "\n",
+ "Below, we define the new program `retrieve_then_read_QA_v2`. In it, we define a new DSP _transformation_ `QA_predict`, which we will re-use in multiple programs.\n",
+ "\n",
+ "The transformation `QA_predict` takes an `Example` (which has a question, context, and demonstrations) and generates an answer. It can be configured to use self-consistency (`sc=True`) or not."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 15,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "@dsp.transformation\n",
+ "def QA_predict(example: dsp.Example, sc=True):\n",
+ " if sc:\n",
+ " example, completions = dsp.generate(qa_template_with_CoT, n=20, temperature=0.7)(example, stage='qa')\n",
+ " completions = dsp.majority(completions)\n",
+ " else:\n",
+ " example, completions = dsp.generate(qa_template_with_CoT)(example, stage='qa')\n",
+ " \n",
+ " return example.copy(answer=completions.answer)\n",
+ "\n",
+ "def retrieve_then_read_QA_v2(question: str) -> str:\n",
+ " demos = dsp.sample(train, k=7)\n",
+ " passages = dsp.retrieve(question, k=5)\n",
+ " example = dsp.Example(question=question, context=passages, demos=demos)\n",
+ " \n",
+ " return QA_predict(example).answer"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's inspect an example."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 16,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "Answer questions with short factoid answers.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Question: Which award did the first book of Gary Zukav receive?\n",
+ "Answer: U.S. National Book Award\n",
+ "\n",
+ "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
+ "Answer: Foxcatcher\n",
+ "\n",
+ "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
+ "Answer: Kevin Greutert\n",
+ "\n",
+ "Question: Who produced the album that included a re-recording of \"Lithium\"?\n",
+ "Answer: Butch Vig\n",
+ "\n",
+ "Question: What city was the victim of Joseph Druces working in?\n",
+ "Answer: Boston, Massachusetts\n",
+ "\n",
+ "Question: In what year was the star of To Hell and Back born?\n",
+ "Answer: 1925\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context:\n",
+ "${sources that may contain relevant content}\n",
+ "\n",
+ "Question: ${the question to be answered}\n",
+ "\n",
+ "Rationale: Let's think step by step. ${a step-by-step deduction that identifies the correct response, which will be provided below}\n",
+ "\n",
+ "Answer: ${a short factoid answer, often between 1 and 5 words}\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context:\n",
+ "[1] «Right Back at It Again | Right Back at It Again \"Right Back at It Again\" is the second track and the first single from A Day to Remember's fifth album, \"Common Courtesy\" (2013). In October 20, 2015, the song was featured in Activision rhythm-music game, \"\". Vocalist, Jeremy McKinnon wrote the lyrics, while the music was written by McKinnon, former guitarist Tom Denney, guitarist Neil Westfall and producer Andrew Wade. \"Right Back at It Again\" almost wasn't included on the album as it was one of the excess songs the band had recorded, \"we realised that it sounded great, so on it went.\" \"Right Back»\n",
+ "[2] «Right Back at It Again | at the Kerrang! Awards. Personnel per digital booklet. Right Back at It Again \"Right Back at It Again\" is the second track and the first single from A Day to Remember's fifth album, \"Common Courtesy\" (2013). In October 20, 2015, the song was featured in Activision rhythm-music game, \"\". Vocalist, Jeremy McKinnon wrote the lyrics, while the music was written by McKinnon, former guitarist Tom Denney, guitarist Neil Westfall and producer Andrew Wade. \"Right Back at It Again\" almost wasn't included on the album as it was one of the excess songs the band had recorded, \"we realised that it»\n",
+ "[3] «Right Back at It Again | band played \"Right Back at It Again\" on both occasions. AbsolutePunk writer Thomas Nassiff said the \"mosh call\" lyric \"\"Brace for impact\"\" had \"pretty much ruin[ed] that track\". Tamsyn Wilce for Alter the Press! noted that the way \"Right Back at it Again\" follows on from opening track \"City of Ocala\" \"continues the catchy hardcore beats\" that is present in all of the band's albums. \"Rock Sound\"'s Andy Ritchie called the song \"unmistakably the 'All I Want' of 'Common Courtesy'\". A music video for the song was released on 19 December 2013. The music video was nominated for Best Video»\n",
+ "[4] «Born Again in Dixieland | Born Again in Dixieland \"Born Again in Dixieland\" is a song co-written and recorded by Canadian country music artist Jason McCoy. Released in June 1997, it was the lead single from his album \"Playin' for Keeps\". The song reached #3 on the \"RPM\" Country Tracks chart in September 1997. The song won a SOCAN Song of the Year award. McCoy said that he started writing the song with Sheridan and he was playing songs for MCA Canada label and they asked if he had any other songs. He played a verse and the chorus from the song thinking it wasn't»\n",
+ "[5] «Right Back Where We Started From | 1974. In a 3 May 2008 interview with Michael Shelley of WFMU, Edwards recalled that after hearing Maxine Nightingale sing on the session for Al Matthews' \"Fool\" that track's producer Pierre Tubbs had come up with \"Right Back Where We Started From\" as a good title for a song for Nightingale herself to record and had invited Edwards to co-write the song. Utilizing a tune which Edwards had written \"a couple of years before\", Tubbs and Edwards wrote \"Right Back Where We Started From\" in about seven minutes while driving to Charing Cross Hospital where Tubbs' wife Gabrielle (née Zimmerman)»\n",
+ "\n",
+ "Question: Right Back At It Again contains lyrics co-written by the singer born in what city?\n",
+ "\n",
+ "Rationale: Let's think step by step.\u001b[32m The song is by A Day to Remember, and the lyrics were co-written by the singer, Jeremy McKinnon. We just need to find out where McKinnon was born.\n",
+ "\n",
+ "Answer: Orlando, Florida\u001b[0m\u001b[31m \t (and 19 other completions)\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "('Ocala, Florida', None)"
+ ]
+ },
+ "execution_count": 16,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "retrieve_then_read_QA_v2(dev[2].question), lm.inspect_history(n=1)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "And let's run this on the other dev examples."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 17,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "100%|██████████| 13/13 [00:01<00:00, 11.23it/s]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Answered 7 / 13 (53.8%) correctly.\n"
+ ]
+ },
+ {
+ "data": {
+ "text/html": [
+ "<style type=\"text/css\" >\n",
+ " #T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2 th {\n",
+ " text-align: left;\n",
+ " } #T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2 td {\n",
+ " text-align: left;\n",
+ " }</style><table id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2\" ><thead> <tr> <th class=\"blank level0\" ></th> <th class=\"col_heading level0 col0\" >question</th> <th class=\"col_heading level0 col1\" >answer</th> <th class=\"col_heading level0 col2\" >prediction</th> <th class=\"col_heading level0 col3\" >correct</th> </tr></thead><tbody>\n",
+ " <tr>\n",
+ " <th id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
+ " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",
+ " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",
+ " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row0_col2\" class=\"data row0 col2\" >E. L. Doctorow</td>\n",
+ " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row0_col3\" class=\"data row0 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
+ " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",
+ " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",
+ " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row1_col2\" class=\"data row1 col2\" >The Killing Season</td>\n",
+ " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row1_col3\" class=\"data row1 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
+ " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",
+ " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",
+ " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row2_col2\" class=\"data row2 col2\" >Ocala, Florida</td>\n",
+ " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row2_col3\" class=\"data row2 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
+ " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",
+ " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",
+ " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row3_col2\" class=\"data row3 col2\" >1824</td>\n",
+ " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row3_col3\" class=\"data row3 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
+ " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",
+ " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",
+ " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row4_col2\" class=\"data row4 col2\" >John Braine</td>\n",
+ " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row4_col3\" class=\"data row4 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
+ " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",
+ " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",
+ " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",
+ " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row5_col3\" class=\"data row5 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
+ " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",
+ " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",
+ " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row6_col2\" class=\"data row6 col2\" >Cheshire, UK</td>\n",
+ " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row6_col3\" class=\"data row6 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
+ " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",
+ " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",
+ " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row7_col2\" class=\"data row7 col2\" >September 27, 1974</td>\n",
+ " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row7_col3\" class=\"data row7 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
+ " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",
+ " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",
+ " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row8_col2\" class=\"data row8 col2\" >Dundalk</td>\n",
+ " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row8_col3\" class=\"data row8 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
+ " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",
+ " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",
+ " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row9_col2\" class=\"data row9 col2\" >Aeronautics and Astronautics</td>\n",
+ " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row9_col3\" class=\"data row9 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
+ " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",
+ " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",
+ " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row10_col2\" class=\"data row10 col2\" >Italy</td>\n",
+ " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row10_col3\" class=\"data row10 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
+ " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",
+ " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",
+ " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row11_col2\" class=\"data row11 col2\" >Portrait of a Marriage</td>\n",
+ " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row11_col3\" class=\"data row11 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
+ " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",
+ " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",
+ " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row12_col2\" class=\"data row12 col2\" >1442</td>\n",
+ " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row12_col3\" class=\"data row12 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " </tbody></table>"
+ ],
+ "text/plain": [
+ "<pandas.io.formats.style.Styler at 0x7f9baab4a850>"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/plain": [
+ "53.8"
+ ]
+ },
+ "execution_count": 17,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "evaluate(retrieve_then_read_QA_v2, dev)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Program 4: Multi-Hop Retrieval\n",
+ "\n",
+ "From these simple programs, it becomes clear that a single search query is not enough for this task. This can be seen when trying to find the birth city of the writer of \"Right Back At It Again\". A search query identifies the author correctly as \"Jeremy McKinnon\", but it doesn't mention when he was born.\n",
+ "\n",
+ "The standard approach for this challenge in the retrieval-augmented NLP literature is to build multi-hop search systems, like GoldEn (Qi et al., 2019), IRRR (Qi et al., 2021) and Baleen (Khattab et al., 2021). These systems read the retrieved results and then generate additional queries to gather additional information if necessary."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "With **DSP**, we can easily simulate such systems in a few lines of code. Let's begin by defining two new templates.\n",
+ "\n",
+ "First, in `rewrite_template`, we will re-write the input question into a simple search query. This will serve as our \"first hop\" query."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 18,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "SearchRationale = dsp.Type(\n",
+ " prefix=\"Rationale: Let's think step by step. To answer this question, we first need to find out\",\n",
+ " desc=\"${the missing information}\"\n",
+ ")\n",
+ "\n",
+ "SearchQuery = dsp.Type(\n",
+ " prefix=\"Search Query:\",\n",
+ " desc=\"${a simple question for seeking the missing information}\"\n",
+ ")\n",
+ "\n",
+ "rewrite_template = dsp.Template(\n",
+ " instructions=\"Write a search query that will help answer a complex question.\",\n",
+ " question=Question(), rationale=SearchRationale(), query=SearchQuery()\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Next, in `hop_template`, we will use the retrieved information from earlier hops to generate additional search queries for missing information."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 19,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "CondenseRationale = dsp.Type(\n",
+ " prefix=\"Rationale: Let's think step by step. Based on the context, we have learned the following.\",\n",
+ " desc=\"${information from the context that provides useful clues}\"\n",
+ ")\n",
+ "\n",
+ "hop_template = dsp.Template(\n",
+ " instructions=rewrite_template.instructions,\n",
+ " context=Context(), question=Question(), rationale=CondenseRationale(), query=SearchQuery()\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's build our `multihop_QA_v1` program. The hallmark of this program is the search transformation called `multihop_search_v1`.\n",
+ "\n",
+ "This will largely simulate the IRRR system. In each hop, we will retrieve `k` (e.g., `k=2`) passages and concatenate them to the context retrieved from earlier hops to form a `context` chain.\n",
+ "\n",
+ "For simplicity, we will repeat this for a total of `max_hops` hops (e.g., `max_hops=2`). It's easy to add a stopping criteria in which the LM is asked whether more hops are required."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 20,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from dsp.utils import deduplicate\n",
+ "\n",
+ "@dsp.transformation\n",
+ "def multihop_search_v1(example: dsp.Example, max_hops=2, k=2) -> dsp.Example:\n",
+ " example.context = []\n",
+ " \n",
+ " for hop in range(max_hops):\n",
+ " # Generate a query based\n",
+ " template = rewrite_template if hop == 0 else hop_template\n",
+ " example, completions = dsp.generate(template)(example, stage=f'h{hop}')\n",
+ "\n",
+ " # Retrieve k results based on the query generated\n",
+ " passages = dsp.retrieve(completions.query, k=k)\n",
+ "\n",
+ " # Update the context by concatenating old and new passages\n",
+ " example.context = deduplicate(example.context + passages)\n",
+ "\n",
+ " return example\n",
+ "\n",
+ "\n",
+ "def multihop_QA_v1(question: str) -> str:\n",
+ " demos = dsp.sample(train, k=7)\n",
+ " x = dsp.Example(question=question, demos=demos)\n",
+ " \n",
+ " x = multihop_search_v1(x)\n",
+ " x = QA_predict(x, sc=False)\n",
+ "\n",
+ " return x.answer"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's run this on one example and inspect the last 3 calls to the LM (i.e., search query 1, search query 2, and QA)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 21,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "Write a search query that will help answer a complex question.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Question: ${the question to be answered}\n",
+ "Rationale: Let's think step by step. To answer this question, we first need to find out ${the missing information}\n",
+ "Search Query: ${a simple question for seeking the missing information}\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Question: Right Back At It Again contains lyrics co-written by the singer born in what city?\n",
+ "Rationale: Let's think step by step. To answer this question, we first need to find out\u001b[32m the name of the singer\n",
+ "Search Query: Who sings Right Back At It Again\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "Write a search query that will help answer a complex question.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context:\n",
+ "${sources that may contain relevant content}\n",
+ "\n",
+ "Question: ${the question to be answered}\n",
+ "\n",
+ "Rationale: Let's think step by step. Based on the context, we have learned the following. ${information from the context that provides useful clues}\n",
+ "\n",
+ "Search Query: ${a simple question for seeking the missing information}\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context:\n",
+ "[1] «Right Back at It Again | at the Kerrang! Awards. Personnel per digital booklet. Right Back at It Again \"Right Back at It Again\" is the second track and the first single from A Day to Remember's fifth album, \"Common Courtesy\" (2013). In October 20, 2015, the song was featured in Activision rhythm-music game, \"\". Vocalist, Jeremy McKinnon wrote the lyrics, while the music was written by McKinnon, former guitarist Tom Denney, guitarist Neil Westfall and producer Andrew Wade. \"Right Back at It Again\" almost wasn't included on the album as it was one of the excess songs the band had recorded, \"we realised that it»\n",
+ "[2] «Right Back at It Again | Right Back at It Again \"Right Back at It Again\" is the second track and the first single from A Day to Remember's fifth album, \"Common Courtesy\" (2013). In October 20, 2015, the song was featured in Activision rhythm-music game, \"\". Vocalist, Jeremy McKinnon wrote the lyrics, while the music was written by McKinnon, former guitarist Tom Denney, guitarist Neil Westfall and producer Andrew Wade. \"Right Back at It Again\" almost wasn't included on the album as it was one of the excess songs the band had recorded, \"we realised that it sounded great, so on it went.\" \"Right Back»\n",
+ "\n",
+ "Question: Right Back At It Again contains lyrics co-written by the singer born in what city?\n",
+ "\n",
+ "Rationale: Let's think step by step. Based on the context, we have learned the following.\u001b[32m The song is called \"Right Back at It Again\" and it is by the band A Day to Remember. The singer, Jeremy McKinnon, wrote the lyrics.\n",
+ "\n",
+ "Search Query: Where was Jeremy McKinnon born?\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "Answer questions with short factoid answers.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Question: Which award did the first book of Gary Zukav receive?\n",
+ "Answer: U.S. National Book Award\n",
+ "\n",
+ "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
+ "Answer: Foxcatcher\n",
+ "\n",
+ "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
+ "Answer: Kevin Greutert\n",
+ "\n",
+ "Question: Who produced the album that included a re-recording of \"Lithium\"?\n",
+ "Answer: Butch Vig\n",
+ "\n",
+ "Question: What city was the victim of Joseph Druces working in?\n",
+ "Answer: Boston, Massachusetts\n",
+ "\n",
+ "Question: In what year was the star of To Hell and Back born?\n",
+ "Answer: 1925\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context:\n",
+ "${sources that may contain relevant content}\n",
+ "\n",
+ "Question: ${the question to be answered}\n",
+ "\n",
+ "Rationale: Let's think step by step. ${a step-by-step deduction that identifies the correct response, which will be provided below}\n",
+ "\n",
+ "Answer: ${a short factoid answer, often between 1 and 5 words}\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context:\n",
+ "[1] «Right Back at It Again | at the Kerrang! Awards. Personnel per digital booklet. Right Back at It Again \"Right Back at It Again\" is the second track and the first single from A Day to Remember's fifth album, \"Common Courtesy\" (2013). In October 20, 2015, the song was featured in Activision rhythm-music game, \"\". Vocalist, Jeremy McKinnon wrote the lyrics, while the music was written by McKinnon, former guitarist Tom Denney, guitarist Neil Westfall and producer Andrew Wade. \"Right Back at It Again\" almost wasn't included on the album as it was one of the excess songs the band had recorded, \"we realised that it»\n",
+ "[2] «Right Back at It Again | Right Back at It Again \"Right Back at It Again\" is the second track and the first single from A Day to Remember's fifth album, \"Common Courtesy\" (2013). In October 20, 2015, the song was featured in Activision rhythm-music game, \"\". Vocalist, Jeremy McKinnon wrote the lyrics, while the music was written by McKinnon, former guitarist Tom Denney, guitarist Neil Westfall and producer Andrew Wade. \"Right Back at It Again\" almost wasn't included on the album as it was one of the excess songs the band had recorded, \"we realised that it sounded great, so on it went.\" \"Right Back»\n",
+ "[3] «Jeremy McKinnon | Jeremy McKinnon Jeremy Wade McKinnon (born December 17, 1985) is an American musician and record producer, best known as a founding member and the lead vocalist of A Day to Remember. He has produced full-length albums for The Devil Wears Prada, The Ghost Inside, Neck Deep, and Wage War. Jeremy McKinnon was born in Gainesville, Florida and grew up with two sisters. His parents are originally from Brooklyn, New York and is of Irish and Italian descent. McKinnon first got a job at Boston Market and later went on to do construction work. McKinnon's interest in music came as a»\n",
+ "[4] «Martin McKinnon | Martin McKinnon Martin \"Marty\" McKinnon (born 5 July 1975 in Adelaide) is a former Australian rules footballer who played with Adelaide, Geelong and the Brisbane Lions in the Australian Football League (AFL). McKinnon was recruited by Adelaide in the 1992 AFL Draft, with their first ever national draft pick. He was the youngest player on Adelaide's list at the time and played for Central District in the SANFL when not appearing with Adelaide. He made his AFL debut late in the 1994 AFL season and was a regular in the team the following year, playing mostly as a wingman. He»\n",
+ "\n",
+ "Question: Right Back At It Again contains lyrics co-written by the singer born in what city?\n",
+ "\n",
+ "Rationale: Let's think step by step.\u001b[32m The song \"Right Back at It Again\" is co-written by the singer Jeremy McKinnon. A quick search of \"Jeremy McKinnon\" reveals that he was born in Gainesville, Florida.\n",
+ "\n",
+ "Answer: Gainesville, Florida\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "('Gainesville, Florida', None)"
+ ]
+ },
+ "execution_count": 21,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "multihop_QA_v1(dev[2].question), lm.inspect_history(n=3)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Now, let's run this on the remaining dev examples."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 22,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "100%|██████████| 13/13 [00:00<00:00, 167.43it/s]"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Answered 7 / 13 (53.8%) correctly.\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "\n"
+ ]
+ },
+ {
+ "data": {
+ "text/html": [
+ "<style type=\"text/css\" >\n",
+ " #T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2 th {\n",
+ " text-align: left;\n",
+ " } #T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2 td {\n",
+ " text-align: left;\n",
+ " }</style><table id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2\" ><thead> <tr> <th class=\"blank level0\" ></th> <th class=\"col_heading level0 col0\" >question</th> <th class=\"col_heading level0 col1\" >answer</th> <th class=\"col_heading level0 col2\" >prediction</th> <th class=\"col_heading level0 col3\" >correct</th> </tr></thead><tbody>\n",
+ " <tr>\n",
+ " <th id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
+ " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",
+ " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",
+ " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row0_col2\" class=\"data row0 col2\" >E. L. Doctorow</td>\n",
+ " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row0_col3\" class=\"data row0 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
+ " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",
+ " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",
+ " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row1_col2\" class=\"data row1 col2\" >The Long Island Serial Killer: A Mother's Hunt for Justice</td>\n",
+ " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row1_col3\" class=\"data row1 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
+ " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",
+ " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",
+ " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row2_col2\" class=\"data row2 col2\" >Gainesville, Florida</td>\n",
+ " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row2_col3\" class=\"data row2 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
+ " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",
+ " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",
+ " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row3_col2\" class=\"data row3 col2\" >1947</td>\n",
+ " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row3_col3\" class=\"data row3 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
+ " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",
+ " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",
+ " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row4_col2\" class=\"data row4 col2\" >John Braine</td>\n",
+ " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row4_col3\" class=\"data row4 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
+ " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",
+ " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",
+ " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",
+ " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row5_col3\" class=\"data row5 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
+ " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",
+ " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",
+ " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row6_col2\" class=\"data row6 col2\" >UK</td>\n",
+ " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row6_col3\" class=\"data row6 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
+ " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",
+ " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",
+ " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row7_col2\" class=\"data row7 col2\" >September 28, 1969</td>\n",
+ " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row7_col3\" class=\"data row7 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
+ " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",
+ " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",
+ " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row8_col2\" class=\"data row8 col2\" >Dundalk</td>\n",
+ " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row8_col3\" class=\"data row8 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
+ " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",
+ " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",
+ " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row9_col2\" class=\"data row9 col2\" >space technology</td>\n",
+ " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row9_col3\" class=\"data row9 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
+ " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",
+ " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",
+ " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row10_col2\" class=\"data row10 col2\" >Guildford, Surrey</td>\n",
+ " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row10_col3\" class=\"data row10 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
+ " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",
+ " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",
+ " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row11_col2\" class=\"data row11 col2\" >Portrait of a Marriage</td>\n",
+ " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row11_col3\" class=\"data row11 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
+ " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",
+ " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",
+ " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row12_col2\" class=\"data row12 col2\" >1442</td>\n",
+ " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row12_col3\" class=\"data row12 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " </tbody></table>"
+ ],
+ "text/plain": [
+ "<pandas.io.formats.style.Styler at 0x7f9bb041dc10>"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/plain": [
+ "53.8"
+ ]
+ },
+ "execution_count": 22,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "evaluate(multihop_QA_v1, dev)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Program 5: Multi-Hop Condensed Retrieval w/ Automatic Demos and Query Fusion\n",
+ "\n",
+ "Through Program 4, we've begun to explore some of the power of the DSP abstraction. However, if you look closely, you will see a few downsides of the previous approach:\n",
+ "\n",
+ "1. The search transformations invoke the LM without any demonstrations in the prompt. That is because we only have training data for the question–answer pairs and not for the intermediate labels (e.g., search queries).\n",
+ "2. The QA prompt uses passages (`context`) and a Chain-of-Thought (`rationale`) for the question to be answered. However, the training demonstrations include neither context nor CoT because they aren't available in our labels.\n",
+ "3. The search transformations commit to a single query per hop, which may single out an unproductive chain of passages and hence fail to uncover relevant information."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We address these problems automatically in the program below.\n",
+ "\n",
+ "In it, we begin by defining `multihop_demonstrate` (which uses `multihop_attempt`) to automatically **annotate** demonstrations for the complex multi-hop pipeline. These demonstrations will be provided to the LM when it's invoked for each transformation."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 23,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "@dsp.transformation\n",
+ "def multihop_attempt(d: dsp.Example) -> dsp.Example:\n",
+ " # Prepare unaugmented demonstrations for the example.\n",
+ " x = dsp.Example(question=d.question, demos=dsp.all_but(train, d))\n",
+ " \n",
+ " # Search. And skip examples where search fails.\n",
+ " # Annotate demonstrations for multihop_search_v2 with the simpler multihop_search_v1 pipeline.\n",
+ " x = multihop_search_v1(x)\n",
+ " if not dsp.passage_match(x.context, d.answer): return None\n",
+ " \n",
+ " # Predict. And skip examples where predict fails.\n",
+ " x = QA_predict(x, sc=False)\n",
+ " if not dsp.answer_match(x.answer, d.answer): return None\n",
+ " \n",
+ " return d.copy(**x)\n",
+ "\n",
+ "@dsp.transformation\n",
+ "def multihop_demonstrate(x: dsp.Example) -> dsp.Example:\n",
+ " demos = dsp.sample(train, k=7)\n",
+ " x.demos = dsp.annotate(multihop_attempt)(demos, k=3, return_all=True)\n",
+ " return x"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We now implement `multihop_search_v2` as part of `multihop_QA_v2`.\n",
+ "\n",
+ "In addition to the changes mentioned earlier, this program simulates the Baleen system (Khattab et al., 2021) in a few lines of code.\n",
+ "\n",
+ "In each retrieval hop (after the very first hop), the summary of the previous hop(s) is included in the prompt. This allows us to efficiently read a larger number of passages from the current hop."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 24,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "@dsp.transformation\n",
+ "def multihop_search_v2(example: dsp.Example, max_hops=2, k=5) -> dsp.Example:\n",
+ " example.context = []\n",
+ "\n",
+ " for hop in range(max_hops):\n",
+ " # Generate queries\n",
+ " template = rewrite_template if hop == 0 else hop_template\n",
+ " example, completions = dsp.generate(template, n=10, temperature=0.7)(example, stage=f'h{hop}')\n",
+ " \n",
+ " # Collect the queries and search with result fusion\n",
+ " queries = [c.query for c in completions] + [example.question]\n",
+ " example.context = dsp.retrieveEnsemble(queries, k=k)\n",
+ "\n",
+ " # Arrange the passages for the next hop\n",
+ " if hop > 0:\n",
+ " example.context = [completions[0].rationale] + example.context\n",
+ " \n",
+ " return example\n",
+ "\n",
+ "def multihop_QA_v2(question: str) -> str:\n",
+ " x = dsp.Example(question=question)\n",
+ " x = multihop_demonstrate(x)\n",
+ " x = multihop_search_v2(x)\n",
+ " x = QA_predict(x)\n",
+ " return x.answer"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 25,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "Write a search query that will help answer a complex question.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Question: ${the question to be answered}\n",
+ "Rationale: Let's think step by step. To answer this question, we first need to find out ${the missing information}\n",
+ "Search Query: ${a simple question for seeking the missing information}\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Question: Which award did the first book of Gary Zukav receive?\n",
+ "Rationale: Let's think step by step. To answer this question, we first need to find out the name of the first book of Gary Zukav.\n",
+ "Search Query: \"What is the name of Gary Zukav's first book?\"\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
+ "Rationale: Let's think step by step. To answer this question, we first need to find out who the heir to the Du Pont family fortune is.\n",
+ "Search Query: \"Heir to the Du Pont family fortune\"\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
+ "Rationale: Let's think step by step. To answer this question, we first need to find out the name of the 2009 movie featuring Peter Outerbridge as William Easton.\n",
+ "Search Query: \"2009 movie featuring Peter Outerbridge as William Easton\"\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Question: What year was the party of the winner of the 1971 San Francisco mayoral election founded?\n",
+ "Rationale: Let's think step by step. To answer this question, we first need to find out\u001b[32m the name of the party of the winner of the 1971 San Francisco mayoral election.\n",
+ "Search Query: \"Party of the winner of the 1971 San Francisco mayoral election\"\u001b[0m\u001b[31m \t (and 9 other completions)\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "Write a search query that will help answer a complex question.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context:\n",
+ "${sources that may contain relevant content}\n",
+ "\n",
+ "Question: ${the question to be answered}\n",
+ "\n",
+ "Rationale: Let's think step by step. Based on the context, we have learned the following. ${information from the context that provides useful clues}\n",
+ "\n",
+ "Search Query: ${a simple question for seeking the missing information}\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context:\n",
+ "[1] «Gary Zukav | Gary Zukav Gary Zukav (born October 17, 1942) is an American spiritual teacher and the author of four consecutive New York Times Best Sellers. Beginning in 1998, he appeared more than 30 times on \"The Oprah Winfrey Show\" to discuss transformation in human consciousness concepts presented in his book \"The Seat of the Soul\". His first book, \"The Dancing Wu Li Masters\" (1979), won a U.S. National Book Award. Gary Zukav was born in Port Arthur, Texas, and spent his early childhood in San Antonio and Houston. His family moved to Pittsburg, Kansas, while he was in fourth grade. In»\n",
+ "[2] «Gary Zukav | experience that led to his first book, \"Dancing Wu Li Masters\", written with extensive help from Jack Sarfatti and other physicists he met through Sarfatti as described in David Kaiser's book \"How the Hippies Saved Physics\". He later described this book as his \"first gift to Life\". In 1987 he moved to Mount Shasta, California, where he lived in a cabin as a self-described \"secular monk\" and spent extensive time in the surrounding wilderness. In 1993 he met Linda Francis. They co-founded the Seat of the Soul Institute in 1998 and moved to Ashland, Oregon, in 2000. Christopher Lehmann-Haupt reviewed»\n",
+ "\n",
+ "Question: Which award did the first book of Gary Zukav receive?\n",
+ "\n",
+ "Rationale: Let's think step by step. Based on the context, we have learned the following. Gary Zukav is an American spiritual teacher and the author of four consecutive New York Times Best Sellers. His first book, \"The Dancing Wu Li Masters\" was written with extensive help from Jack Sarfatti and other physicists.\n",
+ "\n",
+ "Search Query: Which award did \"The Dancing Wu Li Masters\" receive?\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context:\n",
+ "[1] «John du Pont | John du Pont John Eleuthère du Pont (November 22, 1938 – December 9, 2010) was an American philanthropist, heir to the Du Pont family fortune, and a convicted murderer. He had been a published ornithologist, philatelist, conchologist, sports enthusiast, and self-styled wrestling coach. He died in prison while serving a sentence of 30 years for the murder of Dave Schultz. In 1972, du Pont founded and directed the Delaware Museum of Natural History and contributed to Villanova University and other institutions. In the 1980s, he established a wrestling facility at his Foxcatcher Farm after becoming interested in the sport and»\n",
+ "[2] «Du Pont family | with political and business ventures, as well as philanthropic efforts on behalf of various organizations such as The Jockey Club, the Philadelphia Orchestra, the Delaware Art Museum, and the Grand Opera House. Two family members were the subjects of well-publicized criminal cases. Heir John Eleuthère du Pont was convicted of murdering wrestling coach Dave Schultz in 1996 and heir Robert H. Richards IV was convicted of raping his 3-year-old daughter in 2009. The family is depicted in the 2014 biographical film \"Foxcatcher\" with Steve Carell playing John Eleuthère du Pont and Vanessa Redgrave playing Jean du Pont, the wife of»\n",
+ "\n",
+ "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
+ "\n",
+ "Rationale: Let's think step by step. Based on the context, we have learned the following. The Du Pont family is a wealthy American family with a long history of business and philanthropic ventures. In the 1980s, one of the heirs, John Eleuthère du Pont, became interested in wrestling and established a wrestling facility at his farm.\n",
+ "\n",
+ "Search Query: Which wrestling team did John Eleuthère du Pont sponsor?\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context:\n",
+ "[1] «Peter Outerbridge | Peter Outerbridge Peter Outerbridge (born June 30, 1966) is a Canadian actor, best known for his role as Ari Tasarov in the CW action series \"Nikita\", Dr. David Sandström in the TMN series \"ReGenesis\", Henrik \"Hank\" Johanssen in \"Orphan Black\", Bob Corbett in \"Bomb Girls\", William Easton in \"Saw VI\" and George Brown in the television film \"\". He also played the lead role of Detective William Murdoch in a three-episode mini-series, \"The Murdoch Mysteries\", in its initial run on Canadian television, with two episodes shown in 2004 and a third in 2005. Outerbridge was born and raised in Toronto,»\n",
+ "[2] «Except the Dying | 2008. On March 3, 2015, Acorn Media announced a re-release for all three movies, set for May 26, 2015. Except the Dying Except the Dying is a 2004 made-for-TV film starring Peter Outerbridge, Colm Meaney, Keeley Hawes, William B. Davis and Flora Montgomery. It was adapted by Janet MacLean from the novel of the same name by Maureen Jennings. In this film, William Murdoch is introduced, as a man of strong principles, who uses his unique abilities to solve crimes, sometimes using advanced science for his time. On the street of Toronto, in the 1890s, the naked body of a»\n",
+ "\n",
+ "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
+ "\n",
+ "Rationale: Let's think step by step. Based on the context, we have learned the following. Outerbridge was born and raised in Toronto, he is a Canadian actor, best known for his role as Ari Tasarov in the CW action series \"Nikita\", Dr. David Sandström in the TMN series \"ReGenesis\", Henrik \"Hank\" Johanssen in \"Orphan Black\", Bob Corbett in \"Bomb Girls\", William Easton in \"Saw VI\" and George Brown in the television film.\n",
+ "\n",
+ "Search Query: Who was the director of the 2009 movie \"Saw VI\"?\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context:\n",
+ "[1] «1971 San Francisco mayoral election | 1971 San Francisco mayoral election The San Francisco mayoral election of 1971 was held on November 2, 1971, with incumbent Joseph Alioto being re-elected with 38.6 percent of the vote, from among 11 candidates, there being no provision for a runoff. The three major contenders—Joe Alioto; Harold Dobbs, former member of the Board of Supervisors; and Dianne Feinstein, president of the Board of Supervisors—tried to outdo one another as advocates for law and order. Alioto challenged Dobbs to commit to keeping Police Chief Donald Scott and accused Feinstein of wanting to create a civilian review board to investigate police conduct.»\n",
+ "[2] «1971 San Francisco mayoral election | the November 2 election was the re-election of Mayor Joseph Alioto by a 3–2 margin over his nearest challenger. Alioto quoted Jackie Gleason—\"How sweet it is!\"—at a victory celebration and referred to his federal indictment as an attempt by the \"national administration\" to \"knock out a mayor\". Among other races on the ballot was Richard Hongisto's surprise victory for sheriff over incumbent Matthew Carberry. On the ballot was a referendum limiting new-building heights to six stories, which was defeated. A referendum to make the school board an elective one, it being the only appointive board in the state, passed. The»\n",
+ "[3] «1971 San Francisco mayoral election | school-board measure was regarded as a reaction to busing initiated by the school board under court order. Another measure approved was to keep cable car service at no less than current levels. 1971 San Francisco mayoral election The San Francisco mayoral election of 1971 was held on November 2, 1971, with incumbent Joseph Alioto being re-elected with 38.6 percent of the vote, from among 11 candidates, there being no provision for a runoff. The three major contenders—Joe Alioto; Harold Dobbs, former member of the Board of Supervisors; and Dianne Feinstein, president of the Board of Supervisors—tried to outdo one another»\n",
+ "[4] «1971 San Francisco mayoral election | achieve ethnic balance in elementary schools. Former newspaper editor Scott Newhall proposed staging a world ecology fair and replacing California's water plan by importing icebergs, and imputed little difference between the top three contenders, whom he lumped together as \"Joe Dobbstein\". Alioto was a slight favorite to win in spite of being scheduled to go on trial the following January on charges of conspiracy, mail fraud, and using the mail to promote bribery of a public official. As of November, Dobbs, the only Republican in the race, had passed Feinstein in polls and was gaining on Alioto. The result of»\n",
+ "[5] «2003 San Francisco mayoral election | against leftist/progressive Gonzalez, a member of the Green Party. It was the first mayoral election in San Francisco that a Green Party candidate took a noticeable amount of the vote. The election was close, with Gonzalez leading in the polls and winning the popular vote among ballots cast on election day, while Newsom had a larger lead on absentee ballots. 2003 San Francisco mayoral election The 2003 San Francisco mayoral election occurred on November 4, 2003. The incumbent, Willie Brown, was termed out of office and could not seek a third term. The general election included three top candidates including»\n",
+ "\n",
+ "Question: What year was the party of the winner of the 1971 San Francisco mayoral election founded?\n",
+ "\n",
+ "Rationale: Let's think step by step. Based on the context, we have learned the following.\u001b[32m The San Francisco mayoral election of 1971 was held on November 2, 1971, with incumbent Joseph Alioto being re-elected with 38.6 percent of the vote. Alioto was a slight favorite to win in spite of being scheduled to go on trial the following January on charges of conspiracy, mail fraud, and using the mail to promote bribery of a public official.\n",
+ "\n",
+ "Search Query: What year was the Democratic Party founded?\u001b[0m\u001b[31m \t (and 9 other completions)\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "Answer questions with short factoid answers.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Question: Who produced the album that included a re-recording of \"Lithium\"?\n",
+ "Answer: Butch Vig\n",
+ "\n",
+ "Question: What city was the victim of Joseph Druces working in?\n",
+ "Answer: Boston, Massachusetts\n",
+ "\n",
+ "Question: In what year was the star of To Hell and Back born?\n",
+ "Answer: 1925\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context:\n",
+ "${sources that may contain relevant content}\n",
+ "\n",
+ "Question: ${the question to be answered}\n",
+ "\n",
+ "Rationale: Let's think step by step. ${a step-by-step deduction that identifies the correct response, which will be provided below}\n",
+ "\n",
+ "Answer: ${a short factoid answer, often between 1 and 5 words}\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context:\n",
+ "[1] «Gary Zukav | Gary Zukav Gary Zukav (born October 17, 1942) is an American spiritual teacher and the author of four consecutive New York Times Best Sellers. Beginning in 1998, he appeared more than 30 times on \"The Oprah Winfrey Show\" to discuss transformation in human consciousness concepts presented in his book \"The Seat of the Soul\". His first book, \"The Dancing Wu Li Masters\" (1979), won a U.S. National Book Award. Gary Zukav was born in Port Arthur, Texas, and spent his early childhood in San Antonio and Houston. His family moved to Pittsburg, Kansas, while he was in fourth grade. In»\n",
+ "[2] «Gary Zukav | experience that led to his first book, \"Dancing Wu Li Masters\", written with extensive help from Jack Sarfatti and other physicists he met through Sarfatti as described in David Kaiser's book \"How the Hippies Saved Physics\". He later described this book as his \"first gift to Life\". In 1987 he moved to Mount Shasta, California, where he lived in a cabin as a self-described \"secular monk\" and spent extensive time in the surrounding wilderness. In 1993 he met Linda Francis. They co-founded the Seat of the Soul Institute in 1998 and moved to Ashland, Oregon, in 2000. Christopher Lehmann-Haupt reviewed»\n",
+ "[3] «The Dancing Wu Li Masters | The Dancing Wu Li Masters The Dancing Wu Li Masters is a 1979 book by Gary Zukav, a popular science work exploring modern physics, and quantum phenomena in particular. It was awarded a 1980 U.S. National Book Award in category of Science. Although it explores empirical topics in modern physics research, \"The Dancing Wu Li Masters\" gained attention for leveraging metaphors taken from eastern spiritual movements, in particular the Huayen school of Buddhism with the monk Fazang's treatise on The Golden Lion, to explain quantum phenomena and has been regarded by some reviewers as a New Age work, although the»\n",
+ "[4] «The Dancing Wu Li Masters | a 1976 physics conference of eastern and western scientists at Esalen Institute, California; and he used the occasion as material for his book. At the conference, it was said that the Chinese term for physics is 'Wu Li', or \"patterns of organic energy.\" Zukav, among others, conceptualized 'physics' as the dance of the Wu Li Masters – teachers of physical essence. Zukav explains the concept further: The Dancing Wu Li Masters The Dancing Wu Li Masters is a 1979 book by Gary Zukav, a popular science work exploring modern physics, and quantum phenomena in particular. It was awarded a 1980»\n",
+ "\n",
+ "Question: Which award did the first book of Gary Zukav receive?\n",
+ "\n",
+ "Rationale: Let's think step by step. The first book is \"The Dancing Wu Li Masters\" and it was awarded a U.S. National Book Award.\n",
+ "\n",
+ "Answer: U.S. National Book Award\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context:\n",
+ "[1] «John du Pont | John du Pont John Eleuthère du Pont (November 22, 1938 – December 9, 2010) was an American philanthropist, heir to the Du Pont family fortune, and a convicted murderer. He had been a published ornithologist, philatelist, conchologist, sports enthusiast, and self-styled wrestling coach. He died in prison while serving a sentence of 30 years for the murder of Dave Schultz. In 1972, du Pont founded and directed the Delaware Museum of Natural History and contributed to Villanova University and other institutions. In the 1980s, he established a wrestling facility at his Foxcatcher Farm after becoming interested in the sport and»\n",
+ "[2] «Du Pont family | with political and business ventures, as well as philanthropic efforts on behalf of various organizations such as The Jockey Club, the Philadelphia Orchestra, the Delaware Art Museum, and the Grand Opera House. Two family members were the subjects of well-publicized criminal cases. Heir John Eleuthère du Pont was convicted of murdering wrestling coach Dave Schultz in 1996 and heir Robert H. Richards IV was convicted of raping his 3-year-old daughter in 2009. The family is depicted in the 2014 biographical film \"Foxcatcher\" with Steve Carell playing John Eleuthère du Pont and Vanessa Redgrave playing Jean du Pont, the wife of»\n",
+ "[3] «John du Pont | Foxcatcher team. Du Pont became a sponsor in wrestling, swimming, track, and the modern pentathlon. He was also involved in promoting a subset of the modern pentathlon (run, swim, shoot) as a separate event. He took up athletics and became a competitive wrestler in his 50s. His only prior wrestling experience was as a freshman in high school. He began competing again at the age of 55 in the 1992 Veteran's World Championships in Cali, Colombia; following that in 1993 in Toronto, Ontario; in 1994 in Rome, Italy; and in 1995 in Sofia, Bulgaria. In August 1988, a problem-plagued wrestling»\n",
+ "\n",
+ "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
+ "\n",
+ "Rationale: Let's think step by step. The question is asking for a wrestling team that the heir to the Du Pont family sponsored. In the context, it is mentioned that the heir, John du Pont, was interested in wrestling and even competed in wrestling competitions in his 50s. Furthermore, it is stated that he sponsored a wrestling team. Therefore, the answer must be a wrestling team.\n",
+ "\n",
+ "Answer: Foxcatcher team\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context:\n",
+ "[1] «Peter Outerbridge | Peter Outerbridge Peter Outerbridge (born June 30, 1966) is a Canadian actor, best known for his role as Ari Tasarov in the CW action series \"Nikita\", Dr. David Sandström in the TMN series \"ReGenesis\", Henrik \"Hank\" Johanssen in \"Orphan Black\", Bob Corbett in \"Bomb Girls\", William Easton in \"Saw VI\" and George Brown in the television film \"\". He also played the lead role of Detective William Murdoch in a three-episode mini-series, \"The Murdoch Mysteries\", in its initial run on Canadian television, with two episodes shown in 2004 and a third in 2005. Outerbridge was born and raised in Toronto,»\n",
+ "[2] «Except the Dying | 2008. On March 3, 2015, Acorn Media announced a re-release for all three movies, set for May 26, 2015. Except the Dying Except the Dying is a 2004 made-for-TV film starring Peter Outerbridge, Colm Meaney, Keeley Hawes, William B. Davis and Flora Montgomery. It was adapted by Janet MacLean from the novel of the same name by Maureen Jennings. In this film, William Murdoch is introduced, as a man of strong principles, who uses his unique abilities to solve crimes, sometimes using advanced science for his time. On the street of Toronto, in the 1890s, the naked body of a»\n",
+ "[3] «Saw VI | Saw VI Saw VI is a 2009 American horror film directed by Kevin Greutert from a screenplay written by Patrick Melton and Marcus Dunstan. It is the sixth installment in the \"Saw\" franchise and stars Tobin Bell, Costas Mandylor, Betsy Russell, Mark Rolston, Peter Outerbridge, and Shawnee Smith. It was produced by Mark Burg and Oren Koules of Twisted Pictures and distributed by Lionsgate. \"Saw VI\" continues the franchise's focus on the posthumous effects of the Jigsaw Killer and the progression of his successor, Mark Hoffman. In this film, Hoffman sets a series of traps for an insurance executive, William»\n",
+ "[4] «Saw VI | is a crafty move on the screenwriters' part, but once you get past that pseudo-ironic touch, this Saw is no more or less disturbing than any other in the series\". Saw VI Saw VI is a 2009 American horror film directed by Kevin Greutert from a screenplay written by Patrick Melton and Marcus Dunstan. It is the sixth installment in the \"Saw\" franchise and stars Tobin Bell, Costas Mandylor, Betsy Russell, Mark Rolston, Peter Outerbridge, and Shawnee Smith. It was produced by Mark Burg and Oren Koules of Twisted Pictures and distributed by Lionsgate. \"Saw VI\" continues the franchise's focus»\n",
+ "\n",
+ "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
+ "\n",
+ "Rationale: Let's think step by step. The question is asking for the director of the 2009 movie that features Peter Outerbridge as William Easton. We can use process of elimination to figure this out. In source 1, there is no mention of a 2009 movie. In source 2, the only mention of Peter Outerbridge is in the cast list for a 2004 made-for-TV film. In source 3, we see that Outerbridge was in a 2009 movie, Saw VI, and that it was directed by Kevin Greutert. This must be the correct answer, so we can check source 4 to confirm. In source 4, we see that Kevin Greutert is in fact the director of Saw VI.\n",
+ "\n",
+ "Answer: Kevin Greutert\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context:\n",
+ "[1] «The San Francisco mayoral election of 1971 was held on November 2, 1971, with incumbent Joseph Alioto being re-elected with 38.6 percent of the vote. Alioto was a slight favorite to win in spite of being scheduled to go on trial the following January on charges of conspiracy, mail fraud, and using the mail to promote bribery of a public official.»\n",
+ "[2] «Democratic Party (UK, 1998) | Democratic Party (UK, 1998) The Democratic Party was a political party in the United Kingdom, founded in November 1998 by Malvern businessman Geoff Southall, who also funded the party. It was described as \"right wing\" or \"hard right\" in news reports, but claimed to occupy the centre-ground of British politics. It aimed to reduce Britain's involvement with the European Union, opposed the adoption of the Euro, called for direct democracy, and argued for limits on immigration. The slogan was \"The will of the people NOT the party\". In 1999 it had a few hundred members, including previous supporters of James»\n",
+ "[3] «Democratic Party (Brazil, 1925-1934) | Democratic Party (Brazil, 1925-1934) The Democratic Party was a Brazilian political party, but state character, founded by dissidents of the Paulista Republican Party (PRP) on February 24, 1925, during the Old Republic and was its manifesto signed by 599 signatories. It was a representative organization of the traditional middle class, linked to coffee sectors, but especially the urban. In May 1926, the Youth Party (Youth Wing of dissenters, who had organized this caption for the federal elections of 1925), also dissident perrepismo, joined the Democratic Party. His most important names were Waldemar Ferreira, José Adriano Marrey Junior, Francisco Morato, Paulo»\n",
+ "[4] «Democratic Party (Turkey, 1970) | Democratic Party (Turkey, 1970) Democratic Party (Turkish: \"Demokratik Parti\", DP for short) was a former political party in Turkey founded on 18 December 1970 . It should not be confused with the historical Democrat Party or the current Democrat Party in Turkey. The historical Democrat Party which was the ruling party between 1950-1960 was closed by the military rule. In 1961 and 1965 elections most of ex Democratic Party adherents supported Justice Party, a party founded in 1961. After 1965 elections Justice Party came to power. But following 1969 elections in which Justice Party was again successful, tension between the»\n",
+ "[5] «Democratic Party (Japan, 1947) | Democratic Party (Japan, 1947) Democratic Party (民主党, Minshutō) was a centrist political party in Japan. It was founded in spring 1947 by merging the Progressive Party (Shinpo-tō) of Inukai Takeru with a faction of Liberal Party led by Hitoshi Ashida and obtained 124 seats in 1947 elections. The party had held seven seats in Tetsu Katayama's government in 1947-1948. For some months in 1948, party's leader Ashida was Prime minister. In March 1948, part of DP members led by Kijūrō Shidehara joined the Liberal Party to form the Democratic Liberal Party. In 1949 elections, the DP got 69 seats. The»\n",
+ "[6] «Democratic Party (United States) | Democratic Party (United States) The Democratic Party is one of the two major contemporary political parties in the United States, along with the Republican Party. Tracing its heritage back to Thomas Jefferson and James Madison's Democratic-Republican Party, the modern-day Democratic Party was founded around 1828 by supporters of Andrew Jackson, making it the world's oldest active political party. The Democrats' dominant worldview was once social conservatism and economic liberalism while populism was its leading characteristic in the rural South. In 1912, Theodore Roosevelt ran as a third-party candidate in the Progressive (\"Bull Moose\") Party, beginning a switch of political platforms»\n",
+ "\n",
+ "Question: What year was the party of the winner of the 1971 San Francisco mayoral election founded?\n",
+ "\n",
+ "Rationale: Let's think step by step.\u001b[32m The question is asking for the year the party of the mayor of San Francisco in 1971 was founded. In source 1, we see that the election was in 1971 and the winner was Joseph Alioto. In source 2, we see that the party was founded in 1998, which is not correct. In source 3, we see that the party was founded in 1925, which is not correct. In source 4, we see that the party was founded in 1970, which is not correct. In source 5, we see that the party was founded in 1947, which is not correct. In source 6, we see that the party was founded around 1828. This must be the correct answer, so we can check source 1 to confirm. In source 1\u001b[0m\u001b[31m \t (and 19 other completions)\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "('1828', None)"
+ ]
+ },
+ "execution_count": 25,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "multihop_QA_v2(dev[3].question), lm.inspect_history(n=3)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 26,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "100%|██████████| 13/13 [00:02<00:00, 5.38it/s]"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Answered 11 / 13 (84.6%) correctly.\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "\n"
+ ]
+ },
+ {
+ "data": {
+ "text/html": [
+ "<style type=\"text/css\" >\n",
+ " #T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2 th {\n",
+ " text-align: left;\n",
+ " } #T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2 td {\n",
+ " text-align: left;\n",
+ " }</style><table id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2\" ><thead> <tr> <th class=\"blank level0\" ></th> <th class=\"col_heading level0 col0\" >question</th> <th class=\"col_heading level0 col1\" >answer</th> <th class=\"col_heading level0 col2\" >prediction</th> <th class=\"col_heading level0 col3\" >correct</th> </tr></thead><tbody>\n",
+ " <tr>\n",
+ " <th id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
+ " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",
+ " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",
+ " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row0_col2\" class=\"data row0 col2\" >E. L. Doctorow</td>\n",
+ " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row0_col3\" class=\"data row0 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
+ " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",
+ " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",
+ " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row1_col2\" class=\"data row1 col2\" >The Killing Season</td>\n",
+ " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row1_col3\" class=\"data row1 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
+ " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",
+ " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",
+ " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row2_col2\" class=\"data row2 col2\" >Gainesville, Florida</td>\n",
+ " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row2_col3\" class=\"data row2 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
+ " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",
+ " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",
+ " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row3_col2\" class=\"data row3 col2\" >1828</td>\n",
+ " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row3_col3\" class=\"data row3 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
+ " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",
+ " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",
+ " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row4_col2\" class=\"data row4 col2\" >John Braine</td>\n",
+ " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row4_col3\" class=\"data row4 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
+ " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",
+ " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",
+ " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",
+ " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row5_col3\" class=\"data row5 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
+ " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",
+ " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",
+ " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row6_col2\" class=\"data row6 col2\" >Cheshire, UK</td>\n",
+ " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row6_col3\" class=\"data row6 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
+ " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",
+ " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",
+ " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row7_col2\" class=\"data row7 col2\" >February 13, 1980</td>\n",
+ " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row7_col3\" class=\"data row7 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
+ " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",
+ " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",
+ " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row8_col2\" class=\"data row8 col2\" >Dundalk</td>\n",
+ " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row8_col3\" class=\"data row8 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
+ " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",
+ " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",
+ " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row9_col2\" class=\"data row9 col2\" >Aerospace engineering</td>\n",
+ " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row9_col3\" class=\"data row9 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
+ " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",
+ " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",
+ " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row10_col2\" class=\"data row10 col2\" >Guildford, Surrey</td>\n",
+ " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row10_col3\" class=\"data row10 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
+ " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",
+ " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",
+ " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row11_col2\" class=\"data row11 col2\" >Portrait of a Marriage</td>\n",
+ " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row11_col3\" class=\"data row11 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
+ " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",
+ " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",
+ " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row12_col2\" class=\"data row12 col2\" >1442</td>\n",
+ " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row12_col3\" class=\"data row12 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " </tbody></table>"
+ ],
+ "text/plain": [
+ "<pandas.io.formats.style.Styler at 0x7f9ba36cfb50>"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/plain": [
+ "84.6"
+ ]
+ },
+ "execution_count": 26,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "evaluate(multihop_QA_v2, dev)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Ask your own questions\n",
+ "\n",
+ "You could also ask your own questions. This can be a productive vehicle to find problems in the programs above and implement patches for them! (Keep in mind that the search index we're using currently is from Dec 2018.)\n",
+ "\n",
+ "To do this, make sure you set your API key. This can be done by defining the `OPENAI_API_KEY` environemt variable or setting it at the beginning of the program (Cell #2)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 27,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "Write a search query that will help answer a complex question.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Question: ${the question to be answered}\n",
+ "Rationale: Let's think step by step. To answer this question, we first need to find out ${the missing information}\n",
+ "Search Query: ${a simple question for seeking the missing information}\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Question: Which award did the first book of Gary Zukav receive?\n",
+ "Rationale: Let's think step by step. To answer this question, we first need to find out the name of the first book of Gary Zukav.\n",
+ "Search Query: \"What is the name of Gary Zukav's first book?\"\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
+ "Rationale: Let's think step by step. To answer this question, we first need to find out who the heir to the Du Pont family fortune is.\n",
+ "Search Query: \"Heir to the Du Pont family fortune\"\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
+ "Rationale: Let's think step by step. To answer this question, we first need to find out the name of the 2009 movie featuring Peter Outerbridge as William Easton.\n",
+ "Search Query: \"2009 movie featuring Peter Outerbridge as William Easton\"\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Question: When was the creator of Hadoop given an award?\n",
+ "Rationale: Let's think step by step. To answer this question, we first need to find out\u001b[32m the name of the creator of Hadoop.\n",
+ "Search Query: \"Name of the creator of Hadoop\"\u001b[0m\u001b[31m \t (and 9 other completions)\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "Write a search query that will help answer a complex question.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context:\n",
+ "${sources that may contain relevant content}\n",
+ "\n",
+ "Question: ${the question to be answered}\n",
+ "\n",
+ "Rationale: Let's think step by step. Based on the context, we have learned the following. ${information from the context that provides useful clues}\n",
+ "\n",
+ "Search Query: ${a simple question for seeking the missing information}\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context:\n",
+ "[1] «Gary Zukav | Gary Zukav Gary Zukav (born October 17, 1942) is an American spiritual teacher and the author of four consecutive New York Times Best Sellers. Beginning in 1998, he appeared more than 30 times on \"The Oprah Winfrey Show\" to discuss transformation in human consciousness concepts presented in his book \"The Seat of the Soul\". His first book, \"The Dancing Wu Li Masters\" (1979), won a U.S. National Book Award. Gary Zukav was born in Port Arthur, Texas, and spent his early childhood in San Antonio and Houston. His family moved to Pittsburg, Kansas, while he was in fourth grade. In»\n",
+ "[2] «Gary Zukav | experience that led to his first book, \"Dancing Wu Li Masters\", written with extensive help from Jack Sarfatti and other physicists he met through Sarfatti as described in David Kaiser's book \"How the Hippies Saved Physics\". He later described this book as his \"first gift to Life\". In 1987 he moved to Mount Shasta, California, where he lived in a cabin as a self-described \"secular monk\" and spent extensive time in the surrounding wilderness. In 1993 he met Linda Francis. They co-founded the Seat of the Soul Institute in 1998 and moved to Ashland, Oregon, in 2000. Christopher Lehmann-Haupt reviewed»\n",
+ "\n",
+ "Question: Which award did the first book of Gary Zukav receive?\n",
+ "\n",
+ "Rationale: Let's think step by step. Based on the context, we have learned the following. Gary Zukav is an American spiritual teacher and the author of four consecutive New York Times Best Sellers. His first book, \"The Dancing Wu Li Masters\" was written with extensive help from Jack Sarfatti and other physicists.\n",
+ "\n",
+ "Search Query: Which award did \"The Dancing Wu Li Masters\" receive?\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context:\n",
+ "[1] «John du Pont | John du Pont John Eleuthère du Pont (November 22, 1938 – December 9, 2010) was an American philanthropist, heir to the Du Pont family fortune, and a convicted murderer. He had been a published ornithologist, philatelist, conchologist, sports enthusiast, and self-styled wrestling coach. He died in prison while serving a sentence of 30 years for the murder of Dave Schultz. In 1972, du Pont founded and directed the Delaware Museum of Natural History and contributed to Villanova University and other institutions. In the 1980s, he established a wrestling facility at his Foxcatcher Farm after becoming interested in the sport and»\n",
+ "[2] «Du Pont family | with political and business ventures, as well as philanthropic efforts on behalf of various organizations such as The Jockey Club, the Philadelphia Orchestra, the Delaware Art Museum, and the Grand Opera House. Two family members were the subjects of well-publicized criminal cases. Heir John Eleuthère du Pont was convicted of murdering wrestling coach Dave Schultz in 1996 and heir Robert H. Richards IV was convicted of raping his 3-year-old daughter in 2009. The family is depicted in the 2014 biographical film \"Foxcatcher\" with Steve Carell playing John Eleuthère du Pont and Vanessa Redgrave playing Jean du Pont, the wife of»\n",
+ "\n",
+ "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
+ "\n",
+ "Rationale: Let's think step by step. Based on the context, we have learned the following. The Du Pont family is a wealthy American family with a long history of business and philanthropic ventures. In the 1980s, one of the heirs, John Eleuthère du Pont, became interested in wrestling and established a wrestling facility at his farm.\n",
+ "\n",
+ "Search Query: Which wrestling team did John Eleuthère du Pont sponsor?\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context:\n",
+ "[1] «Peter Outerbridge | Peter Outerbridge Peter Outerbridge (born June 30, 1966) is a Canadian actor, best known for his role as Ari Tasarov in the CW action series \"Nikita\", Dr. David Sandström in the TMN series \"ReGenesis\", Henrik \"Hank\" Johanssen in \"Orphan Black\", Bob Corbett in \"Bomb Girls\", William Easton in \"Saw VI\" and George Brown in the television film \"\". He also played the lead role of Detective William Murdoch in a three-episode mini-series, \"The Murdoch Mysteries\", in its initial run on Canadian television, with two episodes shown in 2004 and a third in 2005. Outerbridge was born and raised in Toronto,»\n",
+ "[2] «Except the Dying | 2008. On March 3, 2015, Acorn Media announced a re-release for all three movies, set for May 26, 2015. Except the Dying Except the Dying is a 2004 made-for-TV film starring Peter Outerbridge, Colm Meaney, Keeley Hawes, William B. Davis and Flora Montgomery. It was adapted by Janet MacLean from the novel of the same name by Maureen Jennings. In this film, William Murdoch is introduced, as a man of strong principles, who uses his unique abilities to solve crimes, sometimes using advanced science for his time. On the street of Toronto, in the 1890s, the naked body of a»\n",
+ "\n",
+ "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
+ "\n",
+ "Rationale: Let's think step by step. Based on the context, we have learned the following. Outerbridge was born and raised in Toronto, he is a Canadian actor, best known for his role as Ari Tasarov in the CW action series \"Nikita\", Dr. David Sandström in the TMN series \"ReGenesis\", Henrik \"Hank\" Johanssen in \"Orphan Black\", Bob Corbett in \"Bomb Girls\", William Easton in \"Saw VI\" and George Brown in the television film.\n",
+ "\n",
+ "Search Query: Who was the director of the 2009 movie \"Saw VI\"?\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context:\n",
+ "[1] «Apache Hadoop | the Java programming language, with some native code in C and command line utilities written as shell scripts. Though MapReduce Java code is common, any programming language can be used with Hadoop Streaming to implement the map and reduce parts of the user's program. Other projects in the Hadoop ecosystem expose richer user interfaces. According to its co-founders, Doug Cutting and Mike Cafarella, the genesis of Hadoop was the Google File System paper that was published in October 2003. This paper spawned another one from Google \"MapReduce: Simplified Data Processing on Large Clusters\". Development started on the Apache Nutch project,»\n",
+ "[2] «Doug Cutting | Doug Cutting Douglass Read Cutting is a software designer and advocate and creator of open-source search technology. He founded Lucene and, with Mike Cafarella, Nutch, both open-source search technology projects which are now managed through the Apache Software Foundation. Cutting and Cafarella are also the co-founders of Apache Hadoop. Cutting holds a bachelor's degree from Stanford University. Prior to developing Lucene, Cutting held search technology positions at Xerox PARC where he worked on the Scatter/Gather algorithm and on computational stylistics. He also worked at Excite, where he was one of the chief designers of the search engine, and Apple Inc.,»\n",
+ "[3] «Doug Cutting | September 2010, he was elected its chairman. In 2015, O'Reilly awarded an open source award to Cutting. Doug Cutting Douglass Read Cutting is a software designer and advocate and creator of open-source search technology. He founded Lucene and, with Mike Cafarella, Nutch, both open-source search technology projects which are now managed through the Apache Software Foundation. Cutting and Cafarella are also the co-founders of Apache Hadoop. Cutting holds a bachelor's degree from Stanford University. Prior to developing Lucene, Cutting held search technology positions at Xerox PARC where he worked on the Scatter/Gather algorithm and on computational stylistics. He also worked»\n",
+ "[4] «Apache Hadoop | but was moved to the new Hadoop subproject in January 2006. Doug Cutting, who was working at Yahoo! at the time, named it after his son's toy elephant. The initial code that was factored out of Nutch consisted of about 5,000 lines of code for HDFS and about 6,000 lines of code for MapReduce. The first committer to add to the Hadoop project was Owen O'Malley (in March 2006); Hadoop 0.1.0 was released in April 2006. It continues to evolve through contributions that are being made to the project. Hadoop consists of the \"Hadoop Common\" package, which provides file system»\n",
+ "[5] «Doug Cutting | on the MapReduce algorithm, which allows very large scale computations to be trivially parallelized across large clusters of servers. Cutting and Cafarella, realizing the importance of this paper to extending Lucene into the realm of extremely large search problems, created the open-source Hadoop framework that allows applications based on the MapReduce paradigm to be run on large clusters of commodity hardware. Cutting was an employee of Yahoo!, where he led the Hadoop project full-time. He later went on to work for Cloudera. In July 2009, Cutting was elected to the board of directors of the Apache Software Foundation, and in»\n",
+ "\n",
+ "Question: When was the creator of Hadoop given an award?\n",
+ "\n",
+ "Rationale: Let's think step by step. Based on the context, we have learned the following.\u001b[32m Apache Hadoop is an open-source software framework for distributed storage and distributed processing of big data sets on computer clusters. The creator of Hadoop is Doug Cutting. In 2015, O'Reilly awarded an open source award to Cutting.\n",
+ "\n",
+ "Search Query: When was the creator of Hadoop given an award?\u001b[0m\u001b[31m \t (and 9 other completions)\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "Answer questions with short factoid answers.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Question: Who produced the album that included a re-recording of \"Lithium\"?\n",
+ "Answer: Butch Vig\n",
+ "\n",
+ "Question: What city was the victim of Joseph Druces working in?\n",
+ "Answer: Boston, Massachusetts\n",
+ "\n",
+ "Question: In what year was the star of To Hell and Back born?\n",
+ "Answer: 1925\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context:\n",
+ "${sources that may contain relevant content}\n",
+ "\n",
+ "Question: ${the question to be answered}\n",
+ "\n",
+ "Rationale: Let's think step by step. ${a step-by-step deduction that identifies the correct response, which will be provided below}\n",
+ "\n",
+ "Answer: ${a short factoid answer, often between 1 and 5 words}\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context:\n",
+ "[1] «Gary Zukav | Gary Zukav Gary Zukav (born October 17, 1942) is an American spiritual teacher and the author of four consecutive New York Times Best Sellers. Beginning in 1998, he appeared more than 30 times on \"The Oprah Winfrey Show\" to discuss transformation in human consciousness concepts presented in his book \"The Seat of the Soul\". His first book, \"The Dancing Wu Li Masters\" (1979), won a U.S. National Book Award. Gary Zukav was born in Port Arthur, Texas, and spent his early childhood in San Antonio and Houston. His family moved to Pittsburg, Kansas, while he was in fourth grade. In»\n",
+ "[2] «Gary Zukav | experience that led to his first book, \"Dancing Wu Li Masters\", written with extensive help from Jack Sarfatti and other physicists he met through Sarfatti as described in David Kaiser's book \"How the Hippies Saved Physics\". He later described this book as his \"first gift to Life\". In 1987 he moved to Mount Shasta, California, where he lived in a cabin as a self-described \"secular monk\" and spent extensive time in the surrounding wilderness. In 1993 he met Linda Francis. They co-founded the Seat of the Soul Institute in 1998 and moved to Ashland, Oregon, in 2000. Christopher Lehmann-Haupt reviewed»\n",
+ "[3] «The Dancing Wu Li Masters | The Dancing Wu Li Masters The Dancing Wu Li Masters is a 1979 book by Gary Zukav, a popular science work exploring modern physics, and quantum phenomena in particular. It was awarded a 1980 U.S. National Book Award in category of Science. Although it explores empirical topics in modern physics research, \"The Dancing Wu Li Masters\" gained attention for leveraging metaphors taken from eastern spiritual movements, in particular the Huayen school of Buddhism with the monk Fazang's treatise on The Golden Lion, to explain quantum phenomena and has been regarded by some reviewers as a New Age work, although the»\n",
+ "[4] «The Dancing Wu Li Masters | a 1976 physics conference of eastern and western scientists at Esalen Institute, California; and he used the occasion as material for his book. At the conference, it was said that the Chinese term for physics is 'Wu Li', or \"patterns of organic energy.\" Zukav, among others, conceptualized 'physics' as the dance of the Wu Li Masters – teachers of physical essence. Zukav explains the concept further: The Dancing Wu Li Masters The Dancing Wu Li Masters is a 1979 book by Gary Zukav, a popular science work exploring modern physics, and quantum phenomena in particular. It was awarded a 1980»\n",
+ "\n",
+ "Question: Which award did the first book of Gary Zukav receive?\n",
+ "\n",
+ "Rationale: Let's think step by step. The first book is \"The Dancing Wu Li Masters\" and it was awarded a U.S. National Book Award.\n",
+ "\n",
+ "Answer: U.S. National Book Award\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context:\n",
+ "[1] «John du Pont | John du Pont John Eleuthère du Pont (November 22, 1938 – December 9, 2010) was an American philanthropist, heir to the Du Pont family fortune, and a convicted murderer. He had been a published ornithologist, philatelist, conchologist, sports enthusiast, and self-styled wrestling coach. He died in prison while serving a sentence of 30 years for the murder of Dave Schultz. In 1972, du Pont founded and directed the Delaware Museum of Natural History and contributed to Villanova University and other institutions. In the 1980s, he established a wrestling facility at his Foxcatcher Farm after becoming interested in the sport and»\n",
+ "[2] «Du Pont family | with political and business ventures, as well as philanthropic efforts on behalf of various organizations such as The Jockey Club, the Philadelphia Orchestra, the Delaware Art Museum, and the Grand Opera House. Two family members were the subjects of well-publicized criminal cases. Heir John Eleuthère du Pont was convicted of murdering wrestling coach Dave Schultz in 1996 and heir Robert H. Richards IV was convicted of raping his 3-year-old daughter in 2009. The family is depicted in the 2014 biographical film \"Foxcatcher\" with Steve Carell playing John Eleuthère du Pont and Vanessa Redgrave playing Jean du Pont, the wife of»\n",
+ "[3] «John du Pont | Foxcatcher team. Du Pont became a sponsor in wrestling, swimming, track, and the modern pentathlon. He was also involved in promoting a subset of the modern pentathlon (run, swim, shoot) as a separate event. He took up athletics and became a competitive wrestler in his 50s. His only prior wrestling experience was as a freshman in high school. He began competing again at the age of 55 in the 1992 Veteran's World Championships in Cali, Colombia; following that in 1993 in Toronto, Ontario; in 1994 in Rome, Italy; and in 1995 in Sofia, Bulgaria. In August 1988, a problem-plagued wrestling»\n",
+ "\n",
+ "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
+ "\n",
+ "Rationale: Let's think step by step. The question is asking for a wrestling team that the heir to the Du Pont family sponsored. In the context, it is mentioned that the heir, John du Pont, was interested in wrestling and even competed in wrestling competitions in his 50s. Furthermore, it is stated that he sponsored a wrestling team. Therefore, the answer must be a wrestling team.\n",
+ "\n",
+ "Answer: Foxcatcher team\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context:\n",
+ "[1] «Peter Outerbridge | Peter Outerbridge Peter Outerbridge (born June 30, 1966) is a Canadian actor, best known for his role as Ari Tasarov in the CW action series \"Nikita\", Dr. David Sandström in the TMN series \"ReGenesis\", Henrik \"Hank\" Johanssen in \"Orphan Black\", Bob Corbett in \"Bomb Girls\", William Easton in \"Saw VI\" and George Brown in the television film \"\". He also played the lead role of Detective William Murdoch in a three-episode mini-series, \"The Murdoch Mysteries\", in its initial run on Canadian television, with two episodes shown in 2004 and a third in 2005. Outerbridge was born and raised in Toronto,»\n",
+ "[2] «Except the Dying | 2008. On March 3, 2015, Acorn Media announced a re-release for all three movies, set for May 26, 2015. Except the Dying Except the Dying is a 2004 made-for-TV film starring Peter Outerbridge, Colm Meaney, Keeley Hawes, William B. Davis and Flora Montgomery. It was adapted by Janet MacLean from the novel of the same name by Maureen Jennings. In this film, William Murdoch is introduced, as a man of strong principles, who uses his unique abilities to solve crimes, sometimes using advanced science for his time. On the street of Toronto, in the 1890s, the naked body of a»\n",
+ "[3] «Saw VI | Saw VI Saw VI is a 2009 American horror film directed by Kevin Greutert from a screenplay written by Patrick Melton and Marcus Dunstan. It is the sixth installment in the \"Saw\" franchise and stars Tobin Bell, Costas Mandylor, Betsy Russell, Mark Rolston, Peter Outerbridge, and Shawnee Smith. It was produced by Mark Burg and Oren Koules of Twisted Pictures and distributed by Lionsgate. \"Saw VI\" continues the franchise's focus on the posthumous effects of the Jigsaw Killer and the progression of his successor, Mark Hoffman. In this film, Hoffman sets a series of traps for an insurance executive, William»\n",
+ "[4] «Saw VI | is a crafty move on the screenwriters' part, but once you get past that pseudo-ironic touch, this Saw is no more or less disturbing than any other in the series\". Saw VI Saw VI is a 2009 American horror film directed by Kevin Greutert from a screenplay written by Patrick Melton and Marcus Dunstan. It is the sixth installment in the \"Saw\" franchise and stars Tobin Bell, Costas Mandylor, Betsy Russell, Mark Rolston, Peter Outerbridge, and Shawnee Smith. It was produced by Mark Burg and Oren Koules of Twisted Pictures and distributed by Lionsgate. \"Saw VI\" continues the franchise's focus»\n",
+ "\n",
+ "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
+ "\n",
+ "Rationale: Let's think step by step. The question is asking for the director of the 2009 movie that features Peter Outerbridge as William Easton. We can use process of elimination to figure this out. In source 1, there is no mention of a 2009 movie. In source 2, the only mention of Peter Outerbridge is in the cast list for a 2004 made-for-TV film. In source 3, we see that Outerbridge was in a 2009 movie, Saw VI, and that it was directed by Kevin Greutert. This must be the correct answer, so we can check source 4 to confirm. In source 4, we see that Kevin Greutert is in fact the director of Saw VI.\n",
+ "\n",
+ "Answer: Kevin Greutert\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context:\n",
+ "[1] «Apache Hadoop is an open-source software framework for distributed storage and distributed processing of big data sets on computer clusters. The creator of Hadoop is Doug Cutting. In 2015, O'Reilly awarded an open source award to Cutting.»\n",
+ "[2] «Doug Cutting | September 2010, he was elected its chairman. In 2015, O'Reilly awarded an open source award to Cutting. Doug Cutting Douglass Read Cutting is a software designer and advocate and creator of open-source search technology. He founded Lucene and, with Mike Cafarella, Nutch, both open-source search technology projects which are now managed through the Apache Software Foundation. Cutting and Cafarella are also the co-founders of Apache Hadoop. Cutting holds a bachelor's degree from Stanford University. Prior to developing Lucene, Cutting held search technology positions at Xerox PARC where he worked on the Scatter/Gather algorithm and on computational stylistics. He also worked»\n",
+ "[3] «Anoop Swarup | 1984. He initiated the UNDP-NYKS Poverty Alleviation Awards in 2000, was instrumental in launching a ‘National Think Tank’ in the Ministry of Finance in 2004 and campaign on ‘Indian Civilization through the Millennium’ (ICTM) in 2008 now an ongoing yearly event at Hastinapur. He founded the ‘Sambhav’ Trust in the Indian State of Rajasthan for de addiction in 2008. Anoop Swarup is associated with various organizations that include Board of Directors of OISCA Organization for Industrial, Spiritual and Cultural Advancement International, Tokyo Japan, United Nations Inter Agency working group on Population and Development as part of United Nations Development Assistance»\n",
+ "[4] «Apache Hadoop | but was moved to the new Hadoop subproject in January 2006. Doug Cutting, who was working at Yahoo! at the time, named it after his son's toy elephant. The initial code that was factored out of Nutch consisted of about 5,000 lines of code for HDFS and about 6,000 lines of code for MapReduce. The first committer to add to the Hadoop project was Owen O'Malley (in March 2006); Hadoop 0.1.0 was released in April 2006. It continues to evolve through contributions that are being made to the project. Hadoop consists of the \"Hadoop Common\" package, which provides file system»\n",
+ "[5] «Apache SystemML | November 2, 2015. On May 17, 2017, the Apache Software Foundation Board approved the graduation of Apache SystemML as an Apache Top Level Project. Apache SystemML Apache SystemML is a flexible machine learning system that automatically scales to Spark and Hadoop clusters. SystemML's distinguishing characteristics are: SystemML was created in 2010 by researchers at the IBM Almaden Research Center led by IBM Fellow Shivakumar Vaithyanathan. It was observed that data scientists would write machine learning algorithms in languages such as R and Python for small data. When it came time to scale to big data, a systems programmer would be»\n",
+ "[6] «Matei Zaharia | the Apache Mesos project. He also designed and implemented the core scheduling algorithms used in Apache Hadoop. He received two Best Paper awards at NSDI 2012 and SIGCOMM 2012, Honorable Mention for Community Award at NSDI 2012, and a Best Demo Award at SIGMOD 2012. Jointly with Reynold Xin, Parviz Deyhim, Xiangrui Meng, and Ali Ghodsi, he holds the 2014 world record in Daytona GraySort using Apache Spark. In 2015 he received the ACM Doctoral Dissertation Award. Matei Zaharia Matei Zaharia is a Romanian-Canadian computer scientist specializing in big data, distributed systems, and cloud computing. He is a co-founder and»\n",
+ "\n",
+ "Question: When was the creator of Hadoop given an award?\n",
+ "\n",
+ "Rationale: Let's think step by step.\u001b[32m The question is asking for the year when the creator of Hadoop was given an award. In source 1, we see that the creator of Hadoop is Doug Cutting and that in 2015, O'Reilly awarded an open source award to Cutting. Therefore, we can infer that the answer is 2015. We can check sources 2 and 3 to see if there is any mention of an award given to Doug Cutting in 2015, but since there is no mention, we can assume that source 1 is correct and the answer is 2015.\u001b[0m\u001b[31m \t (and 19 other completions)\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "('2015', None)"
+ ]
+ },
+ "execution_count": 27,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "multihop_QA_v2(\"When was the creator of Hadoop given an award?\"), lm.inspect_history(n=3)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Additional Programs...\n",
+ "\n",
+ "While Program 5 is undeniably better than, say, Program 1, it's far from perfect.\n",
+ "\n",
+ "Many different ideas can easily be implemented to make it more general or more powerful. Refer to the DSP paper for inspiration!\n",
+ "\n",
+ "One of your first improvements to Programs 4 and 5 would be to allow it to do more than two hops and to automatically stop when additional hops are not required.\n",
+ "\n",
+ "**Hint:** This is pretty easy to do. You can start with Program 4 and modify `hop_template` so its `SearchQuery` produces the string `N/A` when additional hops are not required.\n",
+ "\n",
+ "In the `multihop_search_v1` function, you can then break from the loop — i.e., `if completions.query == 'N/A': break` whenever this happens. How will you apply this to Program 5?\n",
+ "\n",
+ "```\n",
+ "SearchQueryWithStopping = dsp.Type(\n",
+ " prefix=\"Search Query:\",\n",
+ " desc=\"${a simple question for seeking the missing or required information --- say N/A if the context above contains all of the required information}\"\n",
+ ")\n",
+ "```"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Other incremental improvements will likely help program 5, including keeping the top passage from each hop in the context and/or the summary of each hop. To decide which changes to make, you can inspect the trace of the program when it makes mistakes!"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 28,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Program 4 with max_hops=4 and automatic stopping."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Additional Capabilities"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## more LLMs for use\n",
+ "Eventhough the most quality tested model is openai text-davinci, we also offer support for `openai chatgpt, cohere, and hf models`.\n",
+ "\n",
+ "`Note:` The quality and stability might be different based on the provider and the model. Please report any bugs that you come across."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# initializing chatgpt\n",
+ "\n",
+ "lm = dsp.GPT3(model='gpt-3.5-turbo', api_key=openai_key, model_type='chat')\n",
+ "\n",
+ "# initializing cohere\n",
+ "\n",
+ "lm = dsp.Cohere(model='command-xlarge-nightly')\n",
+ "\n",
+ "# initializing hf models\n",
+ "\n",
+ "lm = dsp.HFModel(model=\"google/flan-t5-base\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Re-Ranker\n",
+ "\n",
+ "You could also construct powerful retrieval programs that uses both `retriever and re-ranker model`.\n",
+ "\n",
+ "Currently, we support `Sentence-Transformers cross-encoder` for reranking, and an example initialisation and usage is provided below"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sbert_reranker = dsp.SentenceTransformersCrossEncoder(\"cross-encoder/ms-marco-MiniLM-L-12-v2\")\n",
+ "\n",
+ "dsp.settings.configure(lm=lm, rm=rm, reranker=sbert_reranker)\n",
+ "\n",
+ "# dsp.retrieve, and retrieveEnsemble already takes care of retrieving and re-ranking based on initiated rm, and reranker.\n",
+ "\n",
+ "reranked_passages = dsp.retrieve(\"When was the creator of Hadoop given an award?\", k=3)"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.9.4"
+ },
+ "vscode": {
+ "interpreter": {
+ "hash": "de72ef8826336f005e74d4a3f2b2dfb9239c3f097fd15aff69f3f6a4fc3bd8b8"
+ }
+ },
+ "widgets": {
+ "application/vnd.jupyter.widget-state+json": {
+ "state": {},
+ "version_major": 2,
+ "version_minor": 0
+ }
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}
diff --git a/demos/KNN Examples.ipynb b/dsp_v1_demos/knn_demonstrations_test.ipynb
similarity index 100%
rename from demos/KNN Examples.ipynb
rename to dsp_v1_demos/knn_demonstrations_test.ipynb
diff --git a/dspy/__init__.py b/dspy/__init__.py
new file mode 100644
index 000000000..62924665c
--- /dev/null
+++ b/dspy/__init__.py
@@ -0,0 +1,18 @@
+from .signatures import *
+
+from .retrieve import *
+from .predict import *
+from .primitives import *
+# from .evaluation import *
+
+
+# FIXME:
+
+
+import dsp
+
+settings = dsp.settings
+
+OpenAI = dsp.GPT3
+ColBERTv2 = dsp.ColBERTv2
+HFClientTGI = dsp.HFClientTGI
diff --git a/dspy/adapters/__init__.py b/dspy/adapters/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/dspy/adapters/basic_adapter.py b/dspy/adapters/basic_adapter.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/dspy/adapters/chatml_adapter.py b/dspy/adapters/chatml_adapter.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/dspy/adapters/llamachat_adapter.py b/dspy/adapters/llamachat_adapter.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/dspy/adapters/vicuna_adapter.py b/dspy/adapters/vicuna_adapter.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/dspy/datasets/__init__.py b/dspy/datasets/__init__.py
new file mode 100644
index 000000000..de40c1156
--- /dev/null
+++ b/dspy/datasets/__init__.py
@@ -0,0 +1,3 @@
+from .dataset import Dataset
+from .hotpotqa import HotPotQA
+from .colors import Colors
\ No newline at end of file
diff --git a/dspy/datasets/colors.py b/dspy/datasets/colors.py
new file mode 100644
index 000000000..cc04315d3
--- /dev/null
+++ b/dspy/datasets/colors.py
@@ -0,0 +1,32 @@
+import random
+from dspy.datasets.dataset import Dataset
+
+### A bunch of colors, originally from matplotlib
+all_colors = ['alice blue', 'dodger blue', 'light sky blue', 'deep sky blue', 'sky blue', 'steel blue', 'light steel blue', 'medium blue', 'navy blue', 'blue', 'royal blue', 'cadet blue', 'cornflower blue', 'medium slate blue', 'slate blue', 'dark slate blue', 'powder blue', 'turquoise', 'dark turquoise', 'medium turquoise', 'pale turquoise', 'light sea green', 'medium sea green', 'sea green', 'forest green', 'green yellow', 'lime green', 'dark green', 'green', 'lime', 'chartreuse', 'lawn green', 'yellow green', 'olive green', 'dark olive green', 'medium spring green', 'spring green', 'medium aquamarine', 'aquamarine', 'aqua', 'cyan', 'dark cyan', 'teal', 'medium orchid', 'dark orchid', 'orchid', 'blue violet', 'violet', 'dark violet', 'plum', 'thistle', 'magenta', 'fuchsia', 'dark magenta', 'medium purple', 'purple', 'rebecca purple', 'dark red', 'fire brick', 'indian red', 'light coral', 'dark salmon', 'light salmon', 'salmon', 'red', 'crimson', 'tomato', 'coral', 'orange red', 'dark orange', 'orange', 'yellow', 'gold', 'light goldenrod yellow', 'pale goldenrod', 'goldenrod', 'dark goldenrod', 'beige', 'moccasin', 'blanched almond', 'navajo white', 'antique white', 'bisque', 'burlywood', 'dark khaki', 'khaki', 'tan', 'wheat', 'snow', 'floral white', 'old lace', 'ivory', 'linen', 'seashell', 'honeydew', 'mint cream', 'azure', 'lavender', 'ghost white', 'white smoke', 'gainsboro', 'light gray', 'silver', 'dark gray', 'gray', 'dim gray', 'slate gray', 'light slate gray', 'dark slate gray', 'black', 'medium violet red', 'pale violet red', 'deep pink', 'hot pink', 'light pink', 'pink', 'peach puff', 'rosy brown', 'saddle brown', 'sandy brown', 'chocolate', 'peru', 'sienna', 'brown', 'maroon', 'white', 'misty rose', 'lavender blush', 'papaya whip', 'lemon chiffon', 'light yellow', 'corn silk', 'pale green', 'light green', 'olive drab', 'olive', 'dark sea green']
+
+class Colors(Dataset):
+ def __init__(self, sort_by_suffix=False, *args, **kwargs) -> None:
+ super().__init__(*args, **kwargs)
+
+ self.sort_by_suffix = sort_by_suffix
+ colors = self.sorted_by_suffix(all_colors)
+
+ train_size = int(len(colors) * 0.6) # chosen to ensure that similar colors aren't repeated between train and dev
+ train_colors, dev_colors = colors[:train_size], colors[train_size:]
+
+ self._train = [dict(color=color) for color in train_colors]
+ self._dev = [dict(color=color) for color in dev_colors]
+
+ random.Random(0).shuffle(self._train)
+ random.Random(0).shuffle(self._dev)
+
+ def sorted_by_suffix(self, colors):
+ if not self.sort_by_suffix:
+ return colors
+
+ if isinstance(colors[0], str):
+ sorted_colors = sorted(colors, key=lambda x: x[::-1])
+ else:
+ sorted_colors = sorted(colors, key=lambda x: x['color'][::-1])
+
+ return sorted_colors
diff --git a/dspy/datasets/dataset.py b/dspy/datasets/dataset.py
new file mode 100644
index 000000000..9cd279199
--- /dev/null
+++ b/dspy/datasets/dataset.py
@@ -0,0 +1,113 @@
+import uuid
+import random
+
+from dspy import Example
+from dsp.utils import dotdict
+
+class Dataset:
+ def __init__(self, train_seed=0, train_size=None, eval_seed=0, dev_size=None, test_size=None):
+ self.train_size = train_size
+ self.train_seed = train_seed
+ self.dev_size = dev_size
+ self.dev_seed = eval_seed
+ self.test_size = test_size
+ self.test_seed = eval_seed
+ self.do_shuffle = True
+
+ self.name = self.__class__.__name__
+
+ def reset_seeds(self, train_seed=None, train_size=None, eval_seed=None, dev_size=None, test_size=None):
+ self.train_size = train_size if train_size is not None else self.train_size
+ self.train_seed = train_seed if train_seed is not None else self.train_seed
+ self.dev_size = dev_size if dev_size is not None else self.dev_size
+ self.dev_seed = eval_seed if eval_seed is not None else self.dev_seed
+ self.test_size = test_size if test_size is not None else self.test_size
+ self.test_seed = eval_seed if eval_seed is not None else self.test_seed
+
+ if hasattr(self, '_train_'):
+ del self._train_
+
+ if hasattr(self, '_dev_'):
+ del self._dev_
+
+ if hasattr(self, '_test_'):
+ del self._test_
+
+ @property
+ def train(self):
+ if not hasattr(self, '_train_'):
+ self._train_ = self._shuffle_and_sample('train', self._train, self.train_size, self.train_seed)
+
+ return self._train_
+
+ @property
+ def dev(self):
+ if not hasattr(self, '_dev_'):
+ self._dev_ = self._shuffle_and_sample('dev', self._dev, self.dev_size, self.dev_seed)
+
+ return self._dev_
+
+ @property
+ def test(self):
+ if not hasattr(self, '_test_'):
+ self._test_ = self._shuffle_and_sample('test', self._test, self.test_size, self.test_seed)
+
+ return self._test_
+
+ def _shuffle_and_sample(self, split, data, size, seed=0):
+ '''
+ The setting (seed=s, size=N) is always a subset
+ of the setting (seed=s, size=M) for N < M.
+ '''
+
+ data = list(data)
+
+ # Shuffle the data irrespective of the requested size.
+ base_rng = random.Random(seed)
+
+ if self.do_shuffle:
+ base_rng.shuffle(data)
+
+ data = data[:size]
+ output = []
+
+ for example in data:
+ output.append(Example(**example, dspy_uuid=str(uuid.uuid4()), dspy_split=split))
+
+ # TODO: NOTE: Ideally we use these uuids for dedup internally, for demos and internal train/val splits.
+ # Now, some tasks (like convQA and Colors) have overlapping examples. Here, we should allow the user to give us
+ # a uuid field that would respect this in some way. This means that we need a more refined concept that
+ # uuid (each example is unique) and more like a group_uuid.
+
+ # rng = random.Random(seed)
+ # rng.shuffle(data)
+
+ return output
+
+ @classmethod
+ def prepare_by_seed(cls, train_seeds=[1,2,3,4,5], train_size=16, dev_size=1000,
+ divide_eval_per_seed=True, eval_seed=2023, **kwargs):
+
+ data_args = dotdict(train_size=train_size, eval_seed=eval_seed, dev_size=dev_size, test_size=0, **kwargs)
+ dataset = cls(**data_args)
+
+ eval_set = dataset.dev
+ eval_sets, train_sets = [], []
+
+ examples_per_seed = dev_size // len(train_seeds) if divide_eval_per_seed else dev_size
+ eval_offset = 0
+
+ for train_seed in train_seeds:
+ data_args.train_seed = train_seed
+ dataset.reset_seeds(**data_args)
+
+ eval_sets.append(eval_set[eval_offset:eval_offset+examples_per_seed])
+ train_sets.append(dataset.train)
+
+ assert len(eval_sets[-1]) == examples_per_seed, len(eval_sets[-1])
+ assert len(train_sets[-1]) == train_size, len(train_sets[-1])
+
+ if divide_eval_per_seed:
+ eval_offset += examples_per_seed
+
+ return dotdict(train_sets=train_sets, eval_sets=eval_sets)
diff --git a/dspy/datasets/hotpotqa.py b/dspy/datasets/hotpotqa.py
new file mode 100644
index 000000000..e1e0256ba
--- /dev/null
+++ b/dspy/datasets/hotpotqa.py
@@ -0,0 +1,80 @@
+import random
+
+from datasets import load_dataset
+from dspy.datasets.dataset import Dataset
+
+
+class HotPotQA(Dataset):
+ def __init__(self, *args, only_hard_examples=True, keep_details='dev_titles', larger_dev=True, **kwargs) -> None:
+ super().__init__(*args, **kwargs)
+ assert only_hard_examples, "Care must be taken when adding support for easy examples." \
+ "Dev must be all hard to match official dev, but training can be flexible."
+
+ hf_official_train = load_dataset("hotpot_qa", 'fullwiki', split='train')
+ hf_official_dev = load_dataset("hotpot_qa", 'fullwiki', split='validation')
+
+ official_train = []
+ for raw_example in hf_official_train:
+ if raw_example['level'] == 'hard':
+ if keep_details is True:
+ keys = ['id', 'question', 'answer', 'type', 'supporting_facts']
+ elif keep_details == 'dev_titles':
+ keys = ['question', 'answer', 'supporting_facts']
+ else:
+ keys = ['question', 'answer']
+
+ example = {k: raw_example[k] for k in keys}
+
+ if 'supporting_facts' in example:
+ example['gold_titles'] = set(example['supporting_facts']['title'])
+ del example['supporting_facts']
+
+ official_train.append(example)
+
+ rng = random.Random(0)
+ rng.shuffle(official_train)
+
+ if larger_dev is False: # default
+ self._train = official_train[:len(official_train)*90//100]
+ self._dev = official_train[len(official_train)*90//100:]
+ else: # new
+ self._train = official_train[:len(official_train)*75//100]
+ self._dev = official_train[len(official_train)*75//100:]
+
+ for example in self._train:
+ if keep_details == 'dev_titles':
+ del example['gold_titles']
+
+ test = []
+ for raw_example in hf_official_dev:
+ assert raw_example['level'] == 'hard'
+ example = {k: raw_example[k] for k in ['id', 'question', 'answer', 'type', 'supporting_facts']}
+ test.append(example)
+
+ self._test = test
+
+
+if __name__ == '__main__':
+ from dsp.utils import dotdict
+
+ data_args = dotdict(train_seed=1, train_size=16, eval_seed=2023, dev_size=200*5, test_size=0)
+ dataset = HotPotQA(**data_args)
+
+ print(dataset)
+ print(dataset.train[0].question)
+ print(dataset.train[15].question)
+
+ print(len(dataset.train), len(dataset.dev), len(dataset.test))
+
+ print(dataset.dev[0].question)
+ print(dataset.dev[340].question)
+ print(dataset.dev[937].question)
+
+"""
+What was the population of the city where Woodward Avenue ends in 2010?
+Where did the star , who is also an executive producer, of the Mick begin her carrer?
+16 1000 0
+Both London and German have seen attacks during war, there was one specific type of attack that Germany called the blitz, what did London call a similar attack?
+Pre-Madonna was a collection of demos by the singer who was a leading presence during the emergence of what network?
+Alan Mills composed the classic folk song that tells the story of what?
+"""
\ No newline at end of file
diff --git a/dspy/evaluate/__init__.py b/dspy/evaluate/__init__.py
new file mode 100644
index 000000000..1af0d8f5c
--- /dev/null
+++ b/dspy/evaluate/__init__.py
@@ -0,0 +1,2 @@
+from .evaluate import Evaluate
+from .metrics import *
diff --git a/dspy/evaluate/evaluate.py b/dspy/evaluate/evaluate.py
new file mode 100644
index 000000000..0d0e6729f
--- /dev/null
+++ b/dspy/evaluate/evaluate.py
@@ -0,0 +1,206 @@
+from openai import InvalidRequestError
+from openai.error import APIError
+
+import dsp
+import tqdm
+import threading
+import pandas as pd
+
+from IPython.display import display as ipython_display, HTML
+from concurrent.futures import ThreadPoolExecutor, as_completed
+
+from dsp.utils import EM, F1, HotPotF1
+from dsp.evaluation.utils import *
+
+"""
+TODO: The interface would be cleaner if display_progress=False, display_table=False was
+at __init__ time, along with the metric.
+
+But the program and devset (and maybe optional **kwargs to override the __init__ args)
+are given at __call__ time.
+
+That way we can reuse the Evaluate object within a program/notebook.
+
+TODO: Counting failures and having a max_failure count. When that is exceeded (also just at the end),
+we print the number of failures, the first N examples that failed, and the first N exceptions raised.
+
+TODO: When num_threads=1 don't create a thread pool. Just run the program in the main thread.
+"""
+
+
+class Evaluate:
+ def __init__(self, *, devset, metric=None, num_threads=1, display_progress=False, display_table=False, display=True):
+ self.devset = devset
+ self.metric = metric
+ self.num_threads = num_threads
+ self.display_progress = display_progress
+ self.display_table = display_table
+ self.display = display
+
+ def _execute_single_thread(self, wrapped_program, devset, display_progress):
+ ncorrect = 0
+ ntotal = 0
+ reordered_devset = []
+
+ pbar = tqdm.tqdm(total=len(devset), dynamic_ncols=True, disable=not display_progress)
+ for idx, arg in devset:
+ example_idx, example, prediction, score = wrapped_program(idx, arg)
+ reordered_devset.append((example_idx, example, prediction, score))
+ ncorrect += score
+ ntotal += 1
+ self._update_progress(pbar, ncorrect, ntotal)
+ pbar.close()
+
+ return reordered_devset, ncorrect, ntotal
+
+ def _execute_multi_thread(self, wrapped_program, devset, num_threads, display_progress):
+ ncorrect = 0
+ ntotal = 0
+ reordered_devset = []
+
+ with ThreadPoolExecutor(max_workers=num_threads) as executor:
+ futures = {executor.submit(wrapped_program, idx, arg) for idx, arg in devset}
+ pbar = tqdm.tqdm(total=len(devset), dynamic_ncols=True, disable=not display_progress)
+
+ for future in as_completed(futures):
+ example_idx, example, prediction, score = future.result()
+ reordered_devset.append((example_idx, example, prediction, score))
+ ncorrect += score
+ ntotal += 1
+ self._update_progress(pbar, ncorrect, ntotal)
+ pbar.close()
+
+ return reordered_devset, ncorrect, ntotal
+
+ def _update_progress(self, pbar, ncorrect, ntotal):
+ pbar.set_description(f"Average Metric: {ncorrect} / {ntotal} ({round(100 * ncorrect / ntotal, 1)})")
+ pbar.update()
+
+ def __call__(self, program, metric=None, devset=None, num_threads=None, display_progress=None, display_table=None, display=None):
+ metric = metric if metric is not None else self.metric
+ devset = devset if devset is not None else self.devset
+ num_threads = num_threads if num_threads is not None else self.num_threads
+ display_progress = display_progress if display_progress is not None else self.display_progress
+ display_table = display_table if display_table is not None else self.display_table
+
+ display = self.display if display is None else display
+ display_progress = display_progress and display
+ display_table = display_table if display else False
+
+ def wrapped_program(example_idx, example):
+ # NOTE: TODO: Won't work if threads create threads!
+ creating_new_thread = threading.get_ident() not in dsp.settings.stack_by_thread
+ if creating_new_thread:
+ dsp.settings.stack_by_thread[threading.get_ident()] = list(dsp.settings.main_stack)
+ # print(threading.get_ident(), dsp.settings.stack_by_thread[threading.get_ident()])
+
+ # print(type(example), example)
+
+ try:
+ prediction = program(**example.inputs())
+ score = metric(example, prediction) # FIXME: TODO: What's the right order? Maybe force name-based kwargs!
+ return example_idx, example, prediction, score
+ except Exception as e:
+ print(f"Error for example in dev set: \t\t {e}")
+ return example_idx, example, dict(), 0.0
+ finally:
+ if creating_new_thread:
+ del dsp.settings.stack_by_thread[threading.get_ident()]
+
+ devset = list(enumerate(devset))
+
+ if num_threads == 1:
+ reordered_devset, ncorrect, ntotal = self._execute_single_thread(wrapped_program, devset, display_progress)
+ else:
+ reordered_devset, ncorrect, ntotal = self._execute_multi_thread(wrapped_program, devset, num_threads, display_progress)
+
+ if display:
+ print(f"Average Metric: {ncorrect} / {ntotal} ({round(100 * ncorrect / ntotal, 1)}%)")
+
+ predicted_devset = sorted(reordered_devset)
+
+ # data = [{**example, **prediction, 'correct': score} for example, prediction, score in zip(reordered_devset, preds, scores)]
+ data = [merge_dicts(example, prediction) | {'correct': score} for _, example, prediction, score in predicted_devset]
+
+ df = pd.DataFrame(data)
+
+ # Truncate every cell in the DataFrame
+ df = df.applymap(truncate_cell)
+
+ # Rename the 'correct' column to the name of the metric function
+ metric_name = metric.__name__
+ df.rename(columns={'correct': metric_name}, inplace=True)
+
+ if display_table:
+ if isinstance(display_table, int):
+ df_to_display = df.head(display_table).copy()
+ truncated_rows = len(df) - display_table
+ else:
+ df_to_display = df.copy()
+ truncated_rows = 0
+
+ styled_df = configure_dataframe_display(df_to_display, metric_name)
+
+ ipython_display(styled_df)
+
+ if truncated_rows > 0:
+ # Simplified message about the truncated rows
+ message = f"""
+ <div style='
+ text-align: center;
+ font-size: 16px;
+ font-weight: bold;
+ color: #555;
+ margin: 10px 0;'>
+ ... {truncated_rows} more rows not displayed ...
+ </div>
+ """
+ ipython_display(HTML(message))
+
+
+ return round(100 * ncorrect / ntotal, 2)
+
+
+def merge_dicts(d1, d2):
+ merged = {}
+ for k, v in d1.items():
+ if k in d2:
+ merged[f"example_{k}"] = v
+ else:
+ merged[k] = v
+
+ for k, v in d2.items():
+ if k in d1:
+ merged[f"pred_{k}"] = v
+ else:
+ merged[k] = v
+
+ return merged
+
+
+def truncate_cell(content):
+ """Truncate content of a cell to 25 words."""
+ words = str(content).split()
+ if len(words) > 25:
+ return ' '.join(words[:25]) + '...'
+ return content
+
+def configure_dataframe_display(df, metric_name):
+ """Set various pandas display options for DataFrame."""
+ pd.options.display.max_colwidth = None
+ pd.set_option('display.max_colwidth', 15) # Adjust the number as needed
+ pd.set_option('display.width', 400) # Adjust
+
+ # df[metric_name] = df[metric_name].apply(lambda x: f'✔️ [{x}]' if x is True else f'❌ [{x}]')
+ df.loc[:, metric_name] = df[metric_name].apply(lambda x: f'✔️ [{x}]' if x is True else f'❌ [{x}]')
+
+ # Return styled DataFrame
+ return df.style.set_table_styles([
+ {'selector': 'th', 'props': [('text-align', 'left')]},
+ {'selector': 'td', 'props': [('text-align', 'left')]}
+ ]).set_properties(**{
+ 'text-align': 'left',
+ 'white-space': 'pre-wrap',
+ 'word-wrap': 'break-word',
+ 'max-width': '400px'
+ })
diff --git a/dspy/evaluate/metrics.py b/dspy/evaluate/metrics.py
new file mode 100644
index 000000000..da74dfc2f
--- /dev/null
+++ b/dspy/evaluate/metrics.py
@@ -0,0 +1,13 @@
+# TODO: This should move internally. Same for passage_match. dspy.metrics.answer_exact_match, dspy.metrics.answer_passage_match
+
+import dsp
+from dsp.utils import EM, normalize_text
+
+def answer_exact_match(example, pred, trace=None, frac=1.0):
+ return dsp.answer_match(pred.answer, [example.answer], frac=frac)
+
+answer_exact_match_str = dsp.answer_match
+
+def answer_passage_match(example, pred, trace=None):
+ return dsp.passage_match(pred.context, [example.answer])
+
diff --git a/dspy/predict/__init__.py b/dspy/predict/__init__.py
new file mode 100644
index 000000000..bda4e41a2
--- /dev/null
+++ b/dspy/predict/__init__.py
@@ -0,0 +1,3 @@
+from .predict import Predict
+from .chain_of_thought import ChainOfThought
+from .aggregation import majority
\ No newline at end of file
diff --git a/dspy/predict/aggregation.py b/dspy/predict/aggregation.py
new file mode 100644
index 000000000..4b44675e1
--- /dev/null
+++ b/dspy/predict/aggregation.py
@@ -0,0 +1,45 @@
+from dspy.primitives.prediction import Prediction, Completions
+from dsp.utils import normalize_text
+
+
+default_normalize = lambda s: normalize_text(s) or None
+
+
+def majority(prediction_or_completions, normalize=default_normalize, field=None):
+ """
+ Returns the most common completion for the target field (or the last field) in the signature.
+ When normalize returns None, that completion is ignored.
+ """
+
+ assert isinstance(prediction_or_completions, Prediction) or isinstance(prediction_or_completions, Completions)
+ input_type = type(prediction_or_completions)
+
+ # Get the completions
+ if isinstance(prediction_or_completions, Prediction):
+ completions = prediction_or_completions.completions
+ else:
+ completions = prediction_or_completions
+
+ # Normalize
+ field = field if field else completions.signature.fields[-1].output_variable
+ normalize = normalize if normalize else lambda x: x
+ normalized_values = [normalize(completion[field]) for completion in completions]
+ normalized_values_ = [x for x in normalized_values if x is not None]
+
+ # Count
+ value_counts = {}
+ for value in (normalized_values_ or normalized_values):
+ value_counts[value] = value_counts.get(value, 0) + 1
+
+ majority_value = max(value_counts, key=value_counts.get)
+
+ # Return the first completion with the majority value in the field
+ for completion in completions:
+ if normalize(completion[field]) == majority_value:
+ break
+
+ if input_type == Prediction:
+ return Prediction.from_completions([completion], signature=completions.signature)
+
+ return Completions([completion], signature=completions.signature)
+
diff --git a/dspy/predict/chain_of_thought.py b/dspy/predict/chain_of_thought.py
new file mode 100644
index 000000000..6d9c4d44c
--- /dev/null
+++ b/dspy/predict/chain_of_thought.py
@@ -0,0 +1,41 @@
+import dsp
+
+from .predict import Predict
+
+
+
+# TODO: FIXME: Insert this right before the *first* output field. Also rewrite this to use the new signature system.
+
+class ChainOfThought(Predict):
+ def __init__(self, signature, rationale_type=None, activated=True, **config):
+ super().__init__(signature, **config)
+
+ self.activated = activated
+
+ # signature = signature.signature
+ *keys, last_key = signature.kwargs.keys()
+
+ DEFAULT_RATIONALE_TYPE = dsp.Type(prefix="Reasoning: Let's think step by step in order to",
+ desc="${produce the " + last_key + "}. We ...")
+
+ rationale_type = rationale_type or DEFAULT_RATIONALE_TYPE
+
+ extended_kwargs = {key: signature.kwargs[key] for key in keys}
+ extended_kwargs.update({'rationale': rationale_type, last_key: signature.kwargs[last_key]})
+
+ self.extended_signature = dsp.Template(signature.instructions, **extended_kwargs)
+
+ def forward(self, **kwargs):
+ signature = self.signature
+
+ if self.activated is True or (self.activated is None and isinstance(dsp.settings.lm, dsp.GPT3)):
+ signature = self.extended_signature
+
+ return super().forward(signature=signature, **kwargs)
+
+
+"""
+TODO: In principle, we can update the field's prefix during forward too to fill any thing based on the input args.
+
+IF the user didn't overwrite our default rationale_type.
+"""
\ No newline at end of file
diff --git a/dspy/predict/parameter.py b/dspy/predict/parameter.py
new file mode 100644
index 000000000..1c87a6919
--- /dev/null
+++ b/dspy/predict/parameter.py
@@ -0,0 +1,5 @@
+class Parameter:
+ pass
+
+class Hyperparameter:
+ pass
diff --git a/dspy/predict/predict.py b/dspy/predict/predict.py
new file mode 100644
index 000000000..20f48049f
--- /dev/null
+++ b/dspy/predict/predict.py
@@ -0,0 +1,79 @@
+import dsp
+import random
+
+from dspy.predict.parameter import Parameter
+from dspy.primitives.prediction import Prediction
+
+
+class Predict(Parameter):
+ def __init__(self, signature, **config):
+ self.stage = random.randbytes(8).hex()
+ self.signature = signature #.signature
+ self.config = config
+ self.reset()
+
+ def reset(self):
+ self.lm = None
+ self.traces = []
+ self.train = []
+ self.demos = []
+
+ def dump_state(self):
+ state_keys = ["lm", "traces", "train", "demos"]
+ return {k: getattr(self, k) for k in state_keys}
+
+ def load_state(self, state):
+ for name, value in state.items():
+ setattr(self, name, value)
+
+ def __call__(self, **kwargs):
+ return self.forward(**kwargs)
+
+ def forward(self, **kwargs):
+ # Extract the three privileged keyword arguments.
+ signature = kwargs.pop("signature", self.signature)
+ demos = kwargs.pop("demos", self.demos)
+ config = dict(**self.config, **kwargs.pop("config", {}))
+
+ # If temperature is 0.0 but its n > 1, set temperature to 0.7.
+ temperature = config.get("temperature", None)
+ temperature = dsp.settings.lm.kwargs['temperature'] if temperature is None else temperature
+
+ num_generations = config.get("n", None)
+ num_generations = dsp.settings.lm.kwargs['n'] if num_generations is None else num_generations
+
+ if (temperature is None or temperature <= 0.15) and num_generations > 1:
+ config["temperature"] = 0.7
+ # print(f"#> Setting temperature to 0.7 since n={num_generations} and prior temperature={temperature}.")
+
+ # All of the other kwargs are presumed to fit a prefix of the signature.
+
+ x = dsp.Example(demos=demos, **kwargs)
+ x, C = dsp.generate(signature, **config)(x, stage=self.stage)
+
+ completions = []
+
+ for c in C:
+ completions.append({})
+ for field in signature.fields:
+ if field.output_variable not in kwargs.keys():
+ completions[-1][field.output_variable] = getattr(c, field.output_variable)
+
+ pred = Prediction.from_completions(completions, signature=signature)
+
+ if dsp.settings.trace is not None:
+ trace = dsp.settings.trace
+ trace.append((self, {**kwargs}, pred))
+
+ return pred
+
+ def __repr__(self):
+ return f"{self.__class__.__name__}({self.signature})"
+
+
+
+# TODO: get some defaults during init from the context window?
+# # TODO: FIXME: Hmm, I guess expected behavior is that contexts can
+# affect exeuction. Well, we need to determine whether context dominates, __init__ demoninates, or forward dominates.
+# Generally, unless overwritten, we'd see n=None, temperature=None.
+# That will eventually mean we have to learn them.
diff --git a/dspy/primitives/__init__.py b/dspy/primitives/__init__.py
new file mode 100644
index 000000000..f99f3af2d
--- /dev/null
+++ b/dspy/primitives/__init__.py
@@ -0,0 +1,3 @@
+from .example import *
+from .program import *
+from .prediction import *
\ No newline at end of file
diff --git a/dspy/primitives/box.py b/dspy/primitives/box.py
new file mode 100644
index 000000000..db1b51d8f
--- /dev/null
+++ b/dspy/primitives/box.py
@@ -0,0 +1,157 @@
+"""
+TODO: If we want to have Prediction::{**keys, completions, box} where box.{key} will behave as a value but also include
+the completions internally.
+
+The main thing left is to determine the semantic (and then implement them) for applying operations on the object.
+
+If we have a string (query) and completions (5 queries), and we modify the string, what happens to the completions?
+
+- Option 1: We modify the string and the (other) completions are not affected.
+- Option 2: We modify the string and the (other) completions too.
+- Option 3: We modify the string and the (other) completions are deleted.
+
+Option 2 seems most reasonable, but it depends on what metadata the box is going to store.
+
+It seems that a box fundamentally has two functions then:
+- Store a value and its "alternatives" (and hence allow transparent application over operations on value/alts)
+ - But not all operations can/should be applied on all as a map.
+ - I guess mainly "give me a string or list or dict or tuple or int or float" has to commit to a value.
+ - There also needs to be a .item().
+- Potentially track the "source" of the value (i.e., the predictor that generated it, and its inputs)
+- Give the value (eventually) to something that will consume the main value (implicitly or explicitly) or all/some of its alternatives explicitly.
+
+It might be wise to make this responsible for a smaller scope for now:
+
+- Just one string (and its alternatives).
+- No source tracking.
+- Allow operations on the string to map over the alternatives.
+- Seamless extraction at code boundaries.
+ - Basically, code will either treat this as string implicitly
+ (and hence only know about the one value, and on best effort basis we update the alternatives)
+ - Or code will explicitly work with the string or explicitly work with the full set of alternatives.
+
+- By default, all programs (and their sub-programs) will be running inside a context in which preserve_boxes=True.
+- But outside the program, once we see that none of the parent contexts have preserve_boxes=True, we can automatically
+ unpack all boxes before returning to user.
+
+Okay, so we'll have predictors return a `pred` in which `pred.query` is a box.
+
+You'd usually do one of:
+
+### Things that just give you one string
+ 1- Print `pred.query` or save it in a dict somewhere or a file somewhere.
+ 2- Call `pred.query.item()` to get the string explicitly.
+ 3- Modifications in freeform Python.
+ - Modify it by calling `pred.query = 'new query'` altogether.
+ - Modify it by doing `pred.query += 'new query'` or templating `f'{pred.query} new query'`.
+ - Other modifications are not allowed on strings (e.g., `pred.query[0] = 'a'` or `pred.query[0] += 'a'`).
+ - Cast to boolean after a comparison: `if pred.query == 'something': ...`
+ - Pytorch would say RuntimeError: Boolean value of Tensor with more than one value is ambiguous
+ - But we can keep the primary value and use that in the boolean.
+ - So technically, comparison can stick around, giving you multiple internal bools.
+
+Overall, I think it's coherent semantics, for the time being, to say that any of the above will just give you a string back and lose all tracking.
+
+
+### Things that give you a list of strings
+ 1- Explicitly asking for the candidates/completions.
+ 2- Then you could filter or map that list arbitrarily.
+
+In this case, it's just that Box will serve as syntactic sugar. If you don't want to think about `n` at all, you can
+pretend you have a string. If you do anything arbitrary on it, it indeed becomes a string.
+If you later decide to treat it as a list, it's easy to do so without losing that info when you say `pred.query`.
+
+### Things that are more interesting
+
+A) You can now pass pred.query to a DSPy predictor (or searcher, etc) and it can either naively work with the string,
+like pass it to a template, or it can explicitly ask for the list of candidates and do something with that.
+
+This will need a lot more string-specific operations though:
+- endswith, startswith, contains, split, strip, lower, upper, etc.
+- when doing ' '.join() must do map(str, values_to_join). No implicit __str__ conversion!
+- We can probably automate this by having a general fallback? That either returns one value or maps that over all of them.
+
+B) When you say dspy.assert pred.sentence1.endswith('blue'), it will actually check all the alternatives and locally filter them if possible.
+It may keep the bad ones somewhere just in case too.
+
+We could make this a little more explicit like dspy.assert(pred.sentence1, lambda x: x.endswith('blue'))
+
+C) When program_temperature is high, we can actually have some more interesting logic here. When you try to do things that are "selective",
+maybe we'll randomly give you one of the strings (that remain valid in the box, based on assertions).
+
+This could lead to interesting efficiency, because we can basically rerun the program, it'll still generate n=10 candidates,
+but each time it'll use a different one. So when branch_index changes, you get a new candidate each time, but it should be consistent in the same box.
+I guess this could be done by shuffling the same N=10 things. So basically, there's a user N and there's a system-level M.
+
+We can sometimes optimize things by doing M=5. So we'll generate Nx5 candidates in one or more calls (depending on value of Nx5).
+Then depending on the branch_idx, we'll return a fixed set of N candidates. But we have the rest.
+
+"""
+
+
+class BoxType(type):
+ # List of operations to override
+ ops = [
+ # Arithmetic operations
+ 'add', 'sub', 'mul', 'truediv', 'floordiv', 'mod', 'pow',
+ 'lshift', 'rshift', 'and', 'or', 'xor',
+ # 'r'-prefixed versions of arithmetic operations
+ 'radd', 'rsub', 'rmul', 'rtruediv', 'rfloordiv', 'rmod',
+ 'rpow', 'rlshift', 'rrshift', 'rand', 'ror', 'rxor',
+ # Sequence operations
+ 'getitem', 'setitem', 'delitem', 'contains',
+ # Unary and other operations
+ 'neg', 'pos', 'abs', 'invert', 'round', 'len',
+ 'getitem', 'setitem', 'delitem', 'contains', 'iter',
+ # Mappings operations (for dicts)
+ 'get', 'keys', 'values', 'items',
+ # Comparison
+ 'eq', 'ne', 'lt', 'le', 'gt', 'ge',
+ ]
+
+ def __init__(cls, name, bases, attrs):
+ def create_method(op):
+ def method(self, other=None):
+ if op in ['len', 'keys', 'values', 'items']:
+ return getattr(self._value, op)()
+ elif isinstance(other, Box):
+ return Box(getattr(self._value, f'__{op}__')(other._value))
+ elif other is not None:
+ return Box(getattr(self._value, f'__{op}__')(other))
+ else:
+ return NotImplemented
+ return method
+
+ for op in BoxType.ops:
+ setattr(cls, f'__{op}__', create_method(op))
+
+ super().__init__(name, bases, attrs)
+
+
+class Box(metaclass=BoxType):
+ def __init__(self, value, source=False):
+ self._value = value
+ self._source = source
+
+ def __repr__(self):
+ return repr(self._value)
+
+ def __str__(self):
+ return str(self._value)
+
+ def __bool__(self):
+ return bool(self._value)
+
+ # if method is missing just call it on the _value
+ def __getattr__(self, name):
+ return Box(getattr(self._value, name))
+
+ # # Unlike the others, this one collapses to a bool directly
+ # def __eq__(self, other):
+ # if isinstance(other, Box):
+ # return self._value == other._value
+ # else:
+ # return self._value == other
+
+ # def __ne__(self, other):
+ # return not self.__eq__(other)
diff --git a/dspy/primitives/example.py b/dspy/primitives/example.py
new file mode 100644
index 000000000..dbc16c1e1
--- /dev/null
+++ b/dspy/primitives/example.py
@@ -0,0 +1,107 @@
+import copy
+
+class Example:
+ def __init__(self, base=None, **kwargs):
+ # Internal storage and other attributes
+ self._store = {}
+ self._demos = []
+ self._input_keys = None
+
+ # Initialize from a base Example if provided
+ if base and isinstance(base, type(self)):
+ self._store = base._store.copy()
+
+ # Initialize from a dict if provided
+ elif base and isinstance(base, dict):
+ self._store = base.copy()
+
+ # Update with provided kwargs
+ self._store.update(kwargs)
+
+ def __getattr__(self, key):
+ if key.startswith('__') and key.endswith('__'):
+ raise AttributeError
+ if key in self._store:
+ return self._store[key]
+ raise AttributeError(f"'{type(self).__name__}' object has no attribute '{key}'")
+
+ def __setattr__(self, key, value):
+ if key.startswith('_') or key in dir(self.__class__):
+ super().__setattr__(key, value)
+ else:
+ self._store[key] = value
+
+ def __getitem__(self, key):
+ return self._store[key]
+
+ def __setitem__(self, key, value):
+ self._store[key] = value
+
+ def __delitem__(self, key):
+ del self._store[key]
+
+ def __contains__(self, key):
+ return key in self._store
+
+ def __len__(self):
+ return len([k for k in self._store if not k.startswith('dspy_')])
+
+ def __repr__(self):
+ # return f"Example({self._store})" + f" (input_keys={self._input_keys}, demos={self._demos})"
+ d = {k: v for k, v in self._store.items() if not k.startswith('dspy_')}
+ return f"Example({d})" + f" (input_keys={self._input_keys})"
+
+ def __str__(self):
+ return self.__repr__()
+
+ def __eq__(self, other):
+ return self._store == other._store
+
+ def __hash__(self):
+ return hash(tuple(self._store.items()))
+
+ def keys(self, include_dspy=False):
+ return [k for k in self._store.keys() if not k.startswith('dspy_') or include_dspy]
+
+ def values(self, include_dspy=False):
+ return [v for k, v in self._store.items() if not k.startswith('dspy_') or include_dspy]
+
+ def items(self, include_dspy=False):
+ return [(k, v) for k, v in self._store.items() if not k.startswith('dspy_') or include_dspy]
+
+ def get(self, key, default=None):
+ return self._store.get(key, default)
+
+ def with_inputs(self, *keys):
+ copied = self.copy()
+ copied._input_keys = set(keys)
+ return copied
+
+ def inputs(self):
+ if self._input_keys is None:
+ raise ValueError("Inputs have not been set for this example. Use `example.with_inputs()` to set them.")
+
+ # return items that are in input_keys
+ d = {key: self._store[key] for key in self._store if key in self._input_keys}
+ return type(self)(d)
+
+ def labels(self):
+ # return items that are NOT in input_keys
+ input_keys = self.inputs().keys()
+ d = {key: self._store[key] for key in self._store if key not in input_keys}
+ return type(self)(d)
+
+ def __iter__(self):
+ return iter(dict(self._store))
+
+ def copy(self, **kwargs):
+ return type(self)(base=self, **kwargs)
+
+ def without(self, *keys):
+ copied = self.copy()
+ for key in keys:
+ del copied[key]
+ return copied
+
+ def toDict(self):
+ return self._store.copy()
diff --git a/dspy/primitives/module.py b/dspy/primitives/module.py
new file mode 100644
index 000000000..2ad125d8f
--- /dev/null
+++ b/dspy/primitives/module.py
@@ -0,0 +1,69 @@
+import copy
+import ujson
+
+
+class BaseModule:
+ def __init__(self):
+ pass
+
+ def named_parameters(self):
+ """
+ Unlike PyTorch, handles (non-recursive) lists of parameters too.
+ """
+
+ from dspy.predict.parameter import Parameter
+
+ visited = set()
+ named_parameters = []
+
+ def add_parameter(param_name, param_value):
+ if isinstance(param_value, Parameter) and id(param_value) not in visited:
+ visited.add(id(param_value))
+ named_parameters.append((param_name, param_value))
+
+ for name, value in self.__dict__.items():
+ if isinstance(value, Parameter):
+ add_parameter(name, value)
+
+ elif isinstance(value, BaseModule):
+ for sub_name, param in value.named_parameters():
+ add_parameter(f"{name}.{sub_name}", param)
+
+ elif isinstance(value, (list, tuple)):
+ for idx, item in enumerate(value):
+ add_parameter(f"{name}[{idx}]", item)
+
+ elif isinstance(value, dict):
+ for key, item in value.items():
+ add_parameter(f"{name}['{key}']", item)
+
+ return named_parameters
+
+ def parameters(self):
+ return [param for _, param in self.named_parameters()]
+
+ def deepcopy(self):
+ return copy.deepcopy(self)
+
+ def reset_copy(self):
+ obj = copy.deepcopy(self)
+
+ for param in obj.parameters():
+ param.reset()
+
+ return obj
+
+ def dump_state(self):
+ return {name: param.dump_state() for name, param in self.named_parameters()}
+
+ def load_state(self, state):
+ for name, param in self.named_parameters():
+ param.load_state(state[name])
+
+ def save(self, path):
+ with open(path, "w") as f:
+ f.write(ujson.dumps(self.dump_state(), indent=2))
+
+ def load(self, path):
+ with open(path, "r") as f:
+ self.load_state(ujson.loads(f.read()))
diff --git a/dspy/primitives/prediction.py b/dspy/primitives/prediction.py
new file mode 100644
index 000000000..df653c1c4
--- /dev/null
+++ b/dspy/primitives/prediction.py
@@ -0,0 +1,90 @@
+from dspy.primitives.example import Example
+
+
+class Prediction(Example):
+ def __init__(self, *args, **kwargs):
+ super().__init__(*args, **kwargs)
+
+ del self._demos
+ del self._input_keys
+
+ self._completions = None
+
+ @classmethod
+ def from_completions(cls, list_or_dict, signature=None):
+ obj = cls()
+ obj._completions = Completions(list_or_dict, signature=signature)
+ obj._store = {k: v[0] for k, v in obj._completions.items()}
+
+ return obj
+
+ def __repr__(self):
+ store_repr = ',\n '.join(f"{k}={repr(v)}" for k, v in self._store.items())
+
+ if self._completions is None or len(self._completions) == 1:
+ return f"Prediction(\n {store_repr}\n)"
+
+ num_completions = len(self._completions)
+ return f"Prediction(\n {store_repr},\n completions=Completions(...)\n) ({num_completions-1} completions omitted)"
+
+ def __str__(self):
+ return self.__repr__()
+
+ @property
+ def completions(self):
+ return self._completions
+
+
+class Completions:
+ def __init__(self, list_or_dict, signature=None):
+ self.signature = signature
+
+ if isinstance(list_or_dict, list):
+ kwargs = {}
+ for arg in list_or_dict:
+ for k, v in arg.items():
+ kwargs.setdefault(k, []).append(v)
+ else:
+ kwargs = list_or_dict
+
+ assert all(isinstance(v, list) for v in kwargs.values()), "All values must be lists"
+
+ if kwargs:
+ length = len(next(iter(kwargs.values())))
+ assert all(len(v) == length for v in kwargs.values()), "All lists must have the same length"
+
+ self._completions = kwargs
+
+ def items(self):
+ return self._completions.items()
+
+ def __getitem__(self, key):
+ if isinstance(key, int):
+ if key < 0 or key >= len(self):
+ raise IndexError("Index out of range")
+
+ return Prediction(**{k: v[key] for k, v in self._completions.items()})
+
+ return self._completions[key]
+
+ def __getattr__(self, name):
+ if name in self._completions:
+ return self._completions[name]
+
+ raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
+
+ def __len__(self):
+ # Return the length of the list for one of the keys
+ # It assumes all lists have the same length
+ return len(next(iter(self._completions.values())))
+
+ def __contains__(self, key):
+ return key in self._completions
+
+ def __repr__(self):
+ items_repr = ',\n '.join(f"{k}={repr(v)}" for k, v in self._completions.items())
+ return f"Completions(\n {items_repr}\n)"
+
+ def __str__(self):
+ # return str(self._completions)
+ return self.__repr__()
diff --git a/dspy/primitives/program.py b/dspy/primitives/program.py
new file mode 100644
index 000000000..73dbee643
--- /dev/null
+++ b/dspy/primitives/program.py
@@ -0,0 +1,63 @@
+import copy
+
+from dspy.primitives.module import BaseModule
+
+
+class ProgramMeta(type):
+ pass
+ # def __call__(cls, *args, **kwargs):
+ # obj = super(ProgramMeta, cls).__call__(*args, **kwargs)
+
+ # if issubclass(cls, Program) and not getattr(obj, "_program_init_called", False):
+ # obj._base_init()
+ # obj._program_init_called = True
+ # return obj
+
+
+class Module(BaseModule, metaclass=ProgramMeta):
+ def _base_init(self):
+ self._compiled = False
+
+ def __init__(self):
+ self._compiled = False
+
+ def __call__(self, *args, **kwargs):
+ return self.forward(*args, **kwargs)
+
+ def named_predictors(self):
+ from dspy.predict.predict import Predict
+
+ named_parameters = self.named_parameters()
+ return [(name, param) for name, param in named_parameters if isinstance(param, Predict)]
+
+ def predictors(self):
+ return [param for _, param in self.named_predictors()]
+
+ def __repr__(self):
+ s = []
+
+ for name, param in self.named_predictors():
+ s.append(f"{name} = {param}")
+
+ return '\n'.join(s)
+
+ # def __deepcopy__(self, memo):
+ # # memo is a dict of id's to copies already made during the current call
+ # # Check if the object is already copied
+ # if id(self) in memo:
+ # return memo[id(self)]
+
+ # print(f"Deep copying {self.__class__.__name__}...")
+
+ # new_copy = copy.copy(self)
+ # memo[id(self)] = new_copy
+
+ # for k, v in self.__dict__.items():
+ # print(f"Copying attribute {k} of type {type(v)}...")
+ # setattr(new_copy, k, copy.deepcopy(v, memo))
+ # print("Done")
+
+ # return new_copy
+
+
+Program = Module
\ No newline at end of file
diff --git a/dspy/retrieve/__init__.py b/dspy/retrieve/__init__.py
new file mode 100644
index 000000000..1d1f9e8b7
--- /dev/null
+++ b/dspy/retrieve/__init__.py
@@ -0,0 +1 @@
+from .retrieve import Retrieve
\ No newline at end of file
diff --git a/dspy/retrieve/retrieve.py b/dspy/retrieve/retrieve.py
new file mode 100644
index 000000000..0d62cdbba
--- /dev/null
+++ b/dspy/retrieve/retrieve.py
@@ -0,0 +1,34 @@
+import dsp
+import random
+
+from dspy.predict.parameter import Parameter
+from dspy.primitives.prediction import Prediction
+
+
+class Retrieve(Parameter):
+ def __init__(self, k=3):
+ self.stage = random.randbytes(8).hex()
+ self.k = k
+
+ def reset(self):
+ pass
+
+ def dump_state(self):
+ state_keys = ["k"]
+ return {k: getattr(self, k) for k in state_keys}
+
+ def load_state(self, state):
+ for name, value in state.items():
+ setattr(self, name, value)
+
+ def __call__(self, *args, **kwargs):
+ return self.forward(*args, **kwargs)
+
+ def forward(self, query_or_queries):
+ queries = [query_or_queries] if isinstance(query_or_queries, str) else query_or_queries
+
+ passages = dsp.retrieveEnsemble(queries, k=self.k)
+ return Prediction(passages=passages)
+
+
+# TODO: Consider doing Prediction.from_completions with the individual sets of passages (per query) too.
\ No newline at end of file
diff --git a/dspy/signatures/__init__.py b/dspy/signatures/__init__.py
new file mode 100644
index 000000000..ba4637c83
--- /dev/null
+++ b/dspy/signatures/__init__.py
@@ -0,0 +1,2 @@
+from .field import *
+from .signature import *
diff --git a/dspy/signatures/field.py b/dspy/signatures/field.py
new file mode 100644
index 000000000..aa1793b6f
--- /dev/null
+++ b/dspy/signatures/field.py
@@ -0,0 +1,28 @@
+import re
+import dsp
+
+class Field:
+ """A more ergonomic datatype that infers prefix and desc if omitted."""
+ def __init__(self, *, prefix=None, desc=None, input, format=None):
+ self.prefix = prefix # This can be None initially and set later
+ self.desc = desc
+ self.format = format
+
+ def finalize(self, key, inferred_prefix):
+ """Set the prefix if it's not provided explicitly."""
+ if self.prefix is None:
+ self.prefix = inferred_prefix + ":"
+
+ if self.desc is None:
+ self.desc = f'${{{key}}}'
+
+ def __repr__(self):
+ return f"{self.__class__.__name__}(prefix={self.prefix}, desc={self.desc})"
+
+class InputField(Field):
+ def __init__(self, *, prefix=None, desc=None, format=None):
+ super().__init__(prefix=prefix, desc=desc, input=True, format=format)
+
+class OutputField(Field):
+ def __init__(self, *, prefix=None, desc=None, format=None):
+ super().__init__(prefix=prefix, desc=desc, input=False, format=format)
diff --git a/dspy/signatures/signature.py b/dspy/signatures/signature.py
new file mode 100644
index 000000000..f742addb8
--- /dev/null
+++ b/dspy/signatures/signature.py
@@ -0,0 +1,86 @@
+import re
+import dsp
+
+from .field import Field
+import threading
+
+class SignatureMeta(type):
+ _thread_local_storage = threading.local()
+
+ class _SignatureNamespace:
+ def __init__(self, fields):
+ for key, value in fields.items():
+ setattr(self, key, value)
+
+ def __new__(cls, name, bases, class_dict):
+ type_attributes = {}
+
+ for k, v in list(class_dict.items()):
+ if isinstance(v, Field):
+ v.finalize(k, infer_prefix(k))
+ type_attributes[k] = v
+ del class_dict[k]
+
+ instructions = class_dict.get('__doc__') or ""
+
+ new_class = super().__new__(cls, name, bases, class_dict)
+
+ # Attach the _SignatureNamespace directly to the class
+ setattr(new_class, 'signature', cls._SignatureNamespace(type_attributes))
+
+ # Create and attach the template directly to the class
+ setattr(new_class, '_template', dsp.Template(instructions=instructions, **type_attributes))
+
+ return new_class
+
+ def __call__(cls, *args, **kwargs):
+ return cls._template(*args, **kwargs)
+
+ def __getattr__(cls, attr):
+ # Redirect attribute access to the template object when accessed on the class directly
+ return getattr(cls._template, attr)
+
+ def __repr__(cls):
+ s = []
+
+ for name, field in cls.signature.__dict__.items():
+ s.append(f"- {name} = {field}")
+
+ return f'{cls.__name__}\n' + '\n'.join(s)
+
+class Signature(metaclass=SignatureMeta):
+ pass
+
+
+
+def infer_prefix(attribute_name: str) -> str:
+ """Infers a prefix from an attribute name."""
+
+ # Convert camelCase to snake_case, but handle sequences of capital letters properly
+ s1 = re.sub('(.)([A-Z][a-z]+)', r'\1_\2', attribute_name)
+ intermediate_name = re.sub('([a-z0-9])([A-Z])', r'\1_\2', s1)
+
+ # Insert underscores around numbers to ensure spaces in the final output
+ with_underscores_around_numbers = re.sub('([a-zA-Z])(\d)', r'\1_\2', intermediate_name)
+ with_underscores_around_numbers = re.sub('(\d)([a-zA-Z])', r'\1_\2', with_underscores_around_numbers)
+
+ # Convert snake_case to 'Proper Title Case', but ensure acronyms are uppercased
+ words = with_underscores_around_numbers.split('_')
+ title_cased_words = []
+ for word in words:
+ if word.isupper():
+ title_cased_words.append(word)
+ else:
+ title_cased_words.append(word.capitalize())
+
+ return ' '.join(title_cased_words)
+
+### Testing the function
+assert infer_prefix('someAttributeName42IsCool') == 'Some Attribute Name 42 Is Cool'
+assert infer_prefix('version2Update') == 'Version 2 Update'
+assert infer_prefix('modelT45Enhanced') == 'Model T 45 Enhanced'
+assert infer_prefix('someAttributeName') == 'Some Attribute Name'
+assert infer_prefix('some_attribute_name') == 'Some Attribute Name'
+assert infer_prefix('URLAddress') == 'URL Address'
+assert infer_prefix('isHTTPSecure') == 'Is HTTP Secure'
+assert infer_prefix('isHTTPSSecure123') == 'Is HTTPS Secure 123'
diff --git a/dspy/teleprompt/__init__.py b/dspy/teleprompt/__init__.py
new file mode 100644
index 000000000..3ceb081e3
--- /dev/null
+++ b/dspy/teleprompt/__init__.py
@@ -0,0 +1,4 @@
+from .teleprompt import *
+from .bootstrap import *
+from .vanilla import *
+from .random_search import *
\ No newline at end of file
diff --git a/dspy/teleprompt/bootstrap.py b/dspy/teleprompt/bootstrap.py
new file mode 100644
index 000000000..e61503470
--- /dev/null
+++ b/dspy/teleprompt/bootstrap.py
@@ -0,0 +1,154 @@
+import dsp
+import tqdm
+import random
+
+from dspy.primitives import Example
+
+from .teleprompt import Teleprompter
+from .vanilla import LabeledFewShot
+
+from dspy.evaluate.evaluate import Evaluate
+
+# TODO: metrics should return an object with __bool__ basically, but fine if they're more complex.
+# They can also be sortable.
+
+# TODO: Switch here from dsp.Example to dspy.Example. Right now, it's okay because it's internal only (predictors).
+# NOTE: Notice the places where we don't shuffle examples. I do like that this one doesn't shuffle.
+# Other ones that consider options may want to use both unshuffled and then shuffle a few times, when considering candidates.
+
+# TODO: the max_rounds should be supported here. First round is the same way it is now: minimal intervention.
+# Past that first round, though, we can set the temperature to be high and we need to use branch_idx to
+# get past the cache. In principle, we can also sample multiple outputs from the final generation step
+# (or even each step, in case the validation function just wants *one* thing that works, but nah)
+# and try them all. Having a pretty solid guess on the "final step" of each example isn't hard by the second round,
+# in the sense that we have the trace from the first round. (Yes it may change but that's an edge case that
+# won't hurt our "best effort" guarantees.)
+
+
+class BootstrapFewShot(Teleprompter):
+ def __init__(self, metric=None, teacher_settings={}, max_bootstrapped_demos=4, max_labeled_demos=16):
+ self.metric = metric
+ self.teacher_settings = teacher_settings
+
+ self.max_bootstrapped_demos = max_bootstrapped_demos
+ self.max_labeled_demos = max_labeled_demos
+ self.max_rounds = 1
+
+ def compile(self, student, *, teacher=None, trainset, valset=None):
+ self.trainset = trainset
+ self.valset = valset
+
+ self._prepare_student_and_teacher(student, teacher)
+ self._prepare_predictor_mappings()
+ self._bootstrap()
+
+ self.student = self._train()
+ self.student._compiled = True
+
+ return self.student
+
+ def _prepare_student_and_teacher(self, student, teacher):
+ self.student = student.reset_copy()
+ self.teacher = teacher.reset_copy() if teacher is not None else student.reset_copy()
+
+ assert self.student._compiled is False, "Student must be uncompiled."
+
+ if self.max_labeled_demos and self.teacher._compiled is False:
+ teleprompter = LabeledFewShot(k=self.max_labeled_demos)
+ self.teacher = teleprompter.compile(self.teacher.reset_copy(), trainset=self.trainset)
+
+ def _prepare_predictor_mappings(self):
+ name2predictor, predictor2name = {}, {}
+ student, teacher = self.student, self.teacher
+
+ for (name1, predictor1), (name2, predictor2) in zip(student.named_predictors(), teacher.named_predictors()):
+ assert name1 == name2, "Student and teacher must have the same program structure."
+ assert predictor1.signature == predictor2.signature, "Student and teacher must have the same signatures."
+ assert id(predictor1) != id(predictor2), "Student and teacher must be different objects."
+
+ name2predictor[name1] = None # dict(student=predictor1, teacher=predictor2)
+ predictor2name[id(predictor1)] = name1
+ predictor2name[id(predictor2)] = name2
+
+ self.name2predictor = name2predictor
+ self.predictor2name = predictor2name
+
+ def _bootstrap(self, *, max_bootsraps=None):
+ max_bootsraps = max_bootsraps or self.max_bootstrapped_demos
+
+ bootstrapped = {}
+ self.name2traces = {name: [] for name in self.name2predictor}
+
+ for round_idx in range(self.max_rounds):
+ for example_idx, example in enumerate(tqdm.tqdm(self.trainset)):
+ if example_idx not in bootstrapped:
+ success = self._bootstrap_one_example(example)
+
+ if success:
+ bootstrapped[example_idx] = True
+
+ if len(bootstrapped) >= max_bootsraps:
+ print(f'Bootstrapped {len(bootstrapped)} full traces after {example_idx+1} examples in round {round_idx}.')
+ break
+
+ # Unbootstrapped training examples
+
+ self.validation = [x for idx, x in enumerate(self.trainset) if idx not in bootstrapped]
+ random.Random(0).shuffle(self.validation)
+
+ self.validation = self.valset or self.validation
+
+ # NOTE: Can't yet use evaluate because we need to trace *per example*
+ # evaluate = Evaluate(program=self.teacher, metric=self.metric, num_threads=12)
+ # score = evaluate(self.metric, display_table=False, display_progress=True)
+
+ def _bootstrap_one_example(self, example):
+ name2traces = self.name2traces
+ teacher = self.teacher #.deepcopy()
+ predictor_cache = {}
+
+ with dsp.settings.context(trace=[], **self.teacher_settings):
+ for name, predictor in teacher.named_predictors():
+ predictor_cache[name] = predictor.demos
+ predictor.demos = [x for x in predictor.demos if x != example]
+
+ prediction = teacher(**example.inputs())
+ trace = dsp.settings.trace
+
+ for name, predictor in teacher.named_predictors():
+ predictor.demos = predictor_cache[name]
+
+ try:
+ success = (self.metric is None) or self.metric(example, prediction, trace)
+ except Exception as e:
+ success = False
+ print(f'Failed to evaluate example {example.dspy_uuid} with {self.metric} due to {e}.')
+
+ if success:
+ for step in trace:
+ predictor, inputs, outputs = step
+ demo = Example(augmented=True, dspy_uuid=example.dspy_uuid, **inputs, **outputs)
+
+ try:
+ predictor_name = self.predictor2name[id(predictor)]
+ except KeyError as e:
+ # TODO: Look closer into this. It's a bit tricky to reproduce.
+ print(f'Failed to find predictor {predictor} in {self.predictor2name}.')
+ print('Are you doing this in a notebook (Jupyter)? This might be caused by redefining values by rerunning cells.')
+ print('Try restarting the notebook, or open an issue.')
+ raise KeyError(f'Failed to find predictor {id(predictor)} {predictor} in {self.predictor2name}.') from e
+
+ name2traces[predictor_name].append(demo)
+
+ return success
+
+ def _train(self):
+ rng = random.Random(0)
+ raw_demos = self.validation
+
+ for name, predictor in self.student.named_predictors():
+ augmented_demos = self.name2traces[name][:self.max_bootstrapped_demos]
+ raw_demos = rng.sample(raw_demos, min(self.max_labeled_demos - len(augmented_demos), len(raw_demos)))
+ predictor.demos = augmented_demos + raw_demos
+
+ return self.student
diff --git a/dspy/teleprompt/random_search.py b/dspy/teleprompt/random_search.py
new file mode 100644
index 000000000..d77dccb50
--- /dev/null
+++ b/dspy/teleprompt/random_search.py
@@ -0,0 +1,117 @@
+import dsp
+import tqdm
+import random
+
+from .bootstrap import BootstrapFewShot
+from .vanilla import LabeledFewShot
+
+from dspy.evaluate.evaluate import Evaluate
+
+
+# TODO: Don't forget dealing with the raw demos.
+# TODO: Deal with the (pretty common) case of having a metric for filtering and a metric for eval
+
+# TODO: There's an extremely strong case (> 90%) to switch this to NOT inherit from BootstrapFewShot.
+# Instead, it should wrap it: during compilation just loop, create a copy to compile, shuffle the full/sampled
+# trainset and compile with that. This will also make it easier to use raw demos.
+# Once all versions exist, define the validation set and evaluate.
+
+# TODO: This function should take a max_budget and max_teacher_budget. That's in the number of program calls.
+# In this case, max_student_budget is max_budget - max_teacher_budget.
+# For max_teacher_budget, this will just limit the total number of things we bootstrap.
+# This can end up implicitly defining the number of candidate programs (i.e., stop when runs out). Cap at 16.
+# For max_student_budget, this will be a more upfront calculation.
+# Right now, it can also just induce the number of candidate programs. Later, it could be used more interestingly
+# for selective early stopping.
+# Progressive elimination sounds about right: after 50 examples, drop bottom third, after 100, another third, etc.
+# until only 3--5 are left for the end. Could also be systematic and add (earlier) stopping based on error bounds.
+# In general, though, the early filtering is just saying: either there are some really bad ones, or some really really
+# good ones, or most things are pretty close. In all of these cases, dropping the bottom third is not going to hurt.
+
+# Also, this function should be efficient in the following way:
+# seed -3, seed -2, and seed -1 so to speak should just be "zero shot", "labeled shots", and "bootstrap" without any tweaks.
+
+
+class BootstrapFewShotWithRandomSearch(BootstrapFewShot):
+ def __init__(self, metric, teacher_settings={}, max_bootstrapped_demos=4, max_labeled_demos=16, max_rounds=1, num_candidate_programs=16, num_threads=6):
+ self.metric = metric
+ self.teacher_settings = teacher_settings
+ self.max_rounds = max_rounds
+
+ self.num_threads = num_threads
+
+ self.min_num_samples = 1
+ self.max_num_samples = max_bootstrapped_demos
+ self.num_candidate_sets = num_candidate_programs
+ self.max_num_traces = 1 + int(max_bootstrapped_demos / 2.0 * self.num_candidate_sets)
+
+ # Semi-hacky way to get the parent class's _boostrap function to stop early.
+ self.max_bootstrapped_demos = self.max_num_traces
+ self.max_labeled_demos = max_labeled_demos
+
+ print("Going to sample between", self.min_num_samples, "and", self.max_num_samples, "traces per predictor.")
+ print("Going to sample", self.max_num_traces, "traces in total.")
+ print("Will attempt to train", self.num_candidate_sets, "candidate sets.")
+
+ # self.num_candidate_sets = 1
+
+ # import time
+ # time.sleep(10)
+
+ def _random_search_instance(self, idx):
+ print("Random search instance", idx)
+
+ rng = random.Random(idx)
+ program = self.student.deepcopy()
+
+ for predictor in program.predictors():
+ sample_size = rng.randint(self.min_num_samples, self.max_num_samples)
+ print(f"[{idx}] \t Sampling {sample_size} traces from {len(predictor.traces)} traces.")
+
+ augmented_demos = rng.sample(predictor.traces, min(sample_size, len(predictor.traces)))
+
+ # TODO: FIXME: Figuring out the raw demos here is a bit tricky. We can't just use the unused nor the unaugmented (validation) ones.
+ augmented_uuids = set([x.dspy_uuid for x in augmented_demos])
+ raw_demos_uuids = set([x.dspy_uuid for x in predictor.traces if x.dspy_uuid not in augmented_uuids])
+
+ raw_demos = [x for x in self.trainset if x.dspy_uuid in raw_demos_uuids]
+ raw_demos = rng.sample(raw_demos, min(self.max_labeled_demos - len(augmented_demos), len(raw_demos)))
+
+ print(f'Got {len(augmented_demos)} augmented demos and {len(raw_demos)} raw demos.')
+ predictor.demos = augmented_demos + raw_demos
+
+ evaluate = Evaluate(devset=self.validation, metric=self.metric, num_threads=self.num_threads, display_table=False, display_progress=True)
+ score = evaluate(program)
+
+ print('Score:', score, 'for set:', [len(predictor.demos) for predictor in program.predictors()])
+ # dsp.settings.lm.inspect_history(n=1)
+
+ return (score, program)
+
+ def _train(self):
+ for name, predictor in self.student.named_predictors():
+ predictor.traces = self.name2traces[name]
+ pass
+
+ self.candidate_sets = []
+
+ for candidate_set_idx in range(self.num_candidate_sets):
+ score, program = self._random_search_instance(candidate_set_idx)
+ self.candidate_sets.append((score, program))
+
+ best_score, best_program = max(self.candidate_sets, key=lambda x: x[0])
+ print('Best score:', best_score)
+
+ return best_program
+
+
+
+
+# sample between 4 and 10 examples from traces
+# TODO: FIXME: The max number of demos should be determined in part by the LM's tokenizer + max_length.
+# This does require excecuting the program, or at least the predictor.
+# # # # # # (Actually we can just combine the token counts of the traces, when formatted via signature/adapter).
+# Alternatively, we can keep track of the (zero-shot) number of tokens when we bootstrap.
+# As another option, we can just try a wide range and handle failures as penalties on the score.
+# The number "24" of traces to collect can also be affected. If we only need 3x10, some overlap is ok.
+# We can also consider having short_demos and long_demos.
diff --git a/dspy/teleprompt/teleprompt.py b/dspy/teleprompt/teleprompt.py
new file mode 100644
index 000000000..949a6f89c
--- /dev/null
+++ b/dspy/teleprompt/teleprompt.py
@@ -0,0 +1,10 @@
+import tqdm
+import random
+import dsp
+
+from dspy.evaluate.evaluate import Evaluate
+
+
+class Teleprompter:
+ def __init__(self):
+ pass
diff --git a/dspy/teleprompt/vanilla.py b/dspy/teleprompt/vanilla.py
new file mode 100644
index 000000000..619733691
--- /dev/null
+++ b/dspy/teleprompt/vanilla.py
@@ -0,0 +1,27 @@
+import dsp
+import random
+
+from .teleprompt import Teleprompter
+
+
+class LabeledFewShot(Teleprompter):
+ def __init__(self, k=16):
+ self.k = k
+
+ def compile(self, student, *, trainset):
+ self.student = student.reset_copy()
+ self.trainset = trainset
+
+ if len(self.trainset) == 0:
+ return self.student
+
+ rng = random.Random(0)
+
+ for predictor in self.student.predictors():
+ predictor.demos = rng.sample(self.trainset, min(self.k, len(self.trainset)))
+
+ return self.student
+
+# NOTE: I believe templatev2 keeps rdemos as long as they have the last field.
+# This may change later, especially with the introduction of required vs optional fields.
+# NOTE: Since we're relying on downstream code to handle the demos, this sampling may be sub-sampled.
diff --git a/intro.ipynb b/intro.ipynb
index 8fd0704d1..96d3dfeb3 100644
--- a/intro.ipynb
+++ b/intro.ipynb
@@ -4,88 +4,69 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "## 🎓 𝗗𝗦𝗣: 𝗗emonstrate–𝗦earch–𝗣redict\n",
+ "<img src=\"docs/images/DSPy_whitebg.png\" alt=\"DSPy7 Image\" height=\"150\"/>\n",
"\n",
- "This notebook illustrates the Demonstrate–Search–Predict (𝗗𝗦𝗣) framework on the multi-hop question answering (QA) task.\n",
+ "## **DSPy**: Programming with Foundation Models\n",
"\n",
- "Below, we'll build **five** small 𝗗𝗦𝗣 programs that implement strategies for this task. The programs range from extremely simple ones (programs 1 and 2) to moderate ones (programs 3 and 4). Program 5 considers a fairly advanced pipeline."
+ "!! TODO: Put a button for Colab"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "### Installation\n",
+ "This notebook introduces the **DSPy** framework for **Programming with Foundation Models**, i.e., language models (LMs) and retrieval models (RMs).\n",
"\n",
- "If you haven't installed **DSP** already, let's do that."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "try: # When on google Colab, let's clone the notebook so we download the cache.\n",
- " import google.colab \n",
- " !git -C dsp/ pull || git clone https://github.com/stanfordnlp/dsp\n",
- "except: pass\n",
+ "We built **DSPy** because we believe there is no ‟single best pipeline” for using LMs across tasks, and we believe you should not need to manually tweak prompts to get your pipelines to work well.\n",
"\n",
- "!pip install -U pip dsp-ml"
+ "To realize this, **DSPy** provides powerful abstractions for you to **build your own pipelines** for your own tasks using clean Pythonic operators over **modules that compose and learn**!"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "### Setting Up\n",
- "\n",
- "We'll start by setting up the language model (LM) and retrieval model (RM).\n",
+ "### 0] Setting Up & Installation\n",
"\n",
- "We will work with the **GPT-3.5** LM (`text-davinci-002`) and the **ColBERTv2** RM.\n",
+ "**DSPy** is from the authors of ColBERT and the Demonstrate-Search-Predict framework (i.e., DSP, which is the predecessor of **DSPy**). No background on DSP or ColBERT is needed at all, though!\n",
"\n",
- "To use GPT-3, you'll need an OpenAI key. For ColBERTv2, we've set up a server hosting a Wikipedia (Dec 2018) search index, so you don't need to worry about setting one up!\n",
- "\n",
- "To make things easy, we've set up a cache in this repository. _If you want to run this notebook without changing the code or examples, you don't need an API key. All examples are cached._"
+ "Let's begin by setting things up. The snippet below will also install **DSPy** if it's not there already."
]
},
{
"cell_type": "code",
- "execution_count": 2,
+ "execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%load_ext autoreload\n",
"%autoreload 2\n",
"\n",
- "try: import google.colab; root_path = 'dsp'\n",
- "except: root_path = '.'\n",
- "\n",
- "import os\n",
- "os.environ[\"DSP_NOTEBOOK_CACHEDIR\"] = os.path.join(root_path, 'cache')\n",
- "\n",
- "import dsp\n",
- "\n",
- "openai_key = os.getenv('OPENAI_API_KEY') # or replace with your API key (optional)\n",
- "colbert_server = 'http://ec2-44-228-128-229.us-west-2.compute.amazonaws.com:8893/api/search'\n",
- "\n",
- "lm = dsp.GPT3(model='text-davinci-002', api_key=openai_key)\n",
- "rm = dsp.ColBERTv2(url=colbert_server)\n",
- "\n",
- "dsp.settings.configure(lm=lm, rm=rm)"
+ "try: # When on google Colab, let's clone the notebook so we download the cache.\n",
+ " import google.colab\n",
+ " root_path = 'dspy'\n",
+ " !git -C dspy/ pull || git clone https://github.com/stanfordnlp/dspy\n",
+ "except:\n",
+ " root_path = '.'\n",
+ "\n",
+ "try:\n",
+ " import dspy\n",
+ "except Exception:\n",
+ " !pip install -U pip\n",
+ " !pip install -e ."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "### Task Examples\n",
+ "### 1] Getting Started\n",
"\n",
- "Next, let's look at a few examples of the task. Each example consists of a question and one or more gold answers.\n",
+ "We'll start by setting up the language model (LM) and retrieval model (RM). **DSPy** supports multiple API and local models. In this notebook, we'll work with GPT-3.5 (`gpt-3.5-turbo`) and the retriever `ColBERTv2`.\n",
"\n",
- "We have six training examples (`train`), which we'll feed into the programs. These will help define the task.\n",
+ "To make things easy, we've set up a ColBERTv2 server hosting a Wikipedia 2017 \"abstracts\" search index (i.e., containing first paragraph of each article from this [2017 dump](https://hotpotqa.github.io/wiki-readme.html)), so you don't need to worry about setting one up! It's free.\n",
"\n",
- "Notice that our examples only have input (`question`) and output (`answer`) fields. When our advanced programs build sophisticated pipelines, training \"demonstrations\" for other fields will be constructed automatically."
+ "**Note:** _If you want to run this notebook without changing the examples, you don't need an API key. All examples are already cached internally so you can inspect them!_"
]
},
{
@@ -94,399 +75,281 @@
"metadata": {},
"outputs": [],
"source": [
- "train = [('Who produced the album that included a re-recording of \"Lithium\"?', ['Butch Vig']),\n",
- " ('Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?', ['Kevin Greutert']),\n",
- " ('The heir to the Du Pont family fortune sponsored what wrestling team?', ['Foxcatcher', 'Team Foxcatcher', 'Foxcatcher Team']),\n",
- " ('In what year was the star of To Hell and Back born?', ['1925']),\n",
- " ('Which award did the first book of Gary Zukav receive?', ['U.S. National Book Award', 'National Book Award']),\n",
- " ('What city was the victim of Joseph Druces working in?', ['Boston, Massachusetts', 'Boston']),]\n",
+ "# Set up the cache for this notebook\n",
+ "import os; # os.environ[\"DSP_NOTEBOOK_CACHEDIR\"] = os.path.join(root_path, 'cache')\n",
+ "\n",
+ "import dspy\n",
+ "\n",
+ "turbo = dspy.OpenAI(model='gpt-3.5-turbo')\n",
+ "colbertv2_wiki17_abstracts = dspy.ColBERTv2(url='http://20.102.90.50:2017/wiki17_abstracts')\n",
"\n",
- "train = [dsp.Example(question=question, answer=answer) for question, answer in train]"
+ "dspy.settings.configure(lm=turbo, rm=colbertv2_wiki17_abstracts)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "The development examples (`dev`) will be used to assess the behavior of each program we build. Of course, this tiny set is not meant to be a reliable benchmark, but it'll be instructive to use it for illustration."
+ "In the last line above, we configure **DSPy** to use the turbo LM and the ColBERTv2 retriever (over Wikipedia 2017 abstracts) by default. This will be easy to overwrite for local parts of our programs if needed.\n",
+ "\n",
+ "!! TODO: Enable and test the notebook cache"
]
},
{
- "cell_type": "code",
- "execution_count": 4,
+ "cell_type": "markdown",
"metadata": {},
- "outputs": [],
"source": [
- "dev = [('Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?', ['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']),\n",
- " ('What documentary about the Gilgo Beach Killer debuted on A&E?', ['The Killing Season']),\n",
- " ('Right Back At It Again contains lyrics co-written by the singer born in what city?', ['Gainesville, Florida', 'Gainesville']),\n",
- " ('What year was the party of the winner of the 1971 San Francisco mayoral election founded?', ['1828']),\n",
- " ('Which author is English: John Braine or Studs Terkel?', ['John Braine']),\n",
- " ('Anthony Dirrell is the brother of which super middleweight title holder?', ['Andre Dirrell']),\n",
- " ('In which city is the sports nutrition business established by Oliver Cookson based ?', ['Cheshire', 'Cheshire, UK']),\n",
- " ('Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.', ['February 13, 1980']),\n",
- " ('Kyle Moran was born in the town on what river?', ['Castletown', 'Castletown River']),\n",
- " (\"What is the name of one branch of Robert D. Braun's speciality?\", ['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']),\n",
- " (\"Where was the actress who played the niece in the Priest film born?\", ['Surrey', 'Guildford, Surrey']),\n",
- " ('Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.', ['Portrait of a Marriage']),\n",
- " ('What year was the father of the Princes in the Tower born?', ['1442'])]\n",
- "\n",
- "dev = [dsp.Example(question=question, answer=answer) for question, answer in dev]"
+ "##### A word on the workflow"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "### Program 1: Vanilla GPT-3.5\n",
+ "You can build your own **DSPy programs** for various tasks, e.g., question answering, information extraction, or text-to-SQL.\n",
"\n",
- "Let's begin by implementing the simplest program for this task. We'll prompt **GPT-3.5** to answer each question based on its internal parameteric knowledge.\n",
+ "Whatever the task, the general workflow is:\n",
"\n",
- "We'll start by defining the `Template` that defines how we will communicate with the LM.\n",
- "\n",
- "Specifically, the question–answer template (`qa_template`) will include a question and a short answer for each example."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "metadata": {},
- "outputs": [],
- "source": [
- "Question = dsp.Type(prefix=\"Question:\", desc=\"${the question to be answered}\")\n",
- "Answer = dsp.Type(prefix=\"Answer:\", desc=\"${a short factoid answer, often between 1 and 5 words}\", format=dsp.format_answers)\n",
+ "1. **Collect a little bit of data.** Define examples of the inputs and outputs of your program (e.g., questions and their answers). This could just be a handful of quick examples you wrote down. If large datasets exist, the more the merrier!\n",
+ "1. **Write your program.** Define the modules (i.e., sub-tasks) of your program and the way they should interact together to solve your task.\n",
+ "1. **Define some validation logic.** What makes for a good run of your program? Maybe the answers need to have a certain length or stick to a particular format? Specify the logic that checks that.\n",
+ "1. **Compile!** Ask **DSPy** to _compile_ your program using your data. The compiler will use your data and validation logic to optimize your program (e.g., prompts and modules) so it's efficient and effective!\n",
+ "1. **Iterate.** Repeat the process by improving your data, program, validation, or by using more advanced features of the **DSPy** compiler.\n",
"\n",
- "qa_template = dsp.Template(instructions=\"Answer questions with short factoid answers.\", question=Question(), answer=Answer())"
+ "Let's now see this in action."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "Then, let's define the actual program, `vanilla_LM_QA`. It'll accept a string (`question`) and returns another string (its short `answer`)."
+ "### 2] Task Examples\n",
+ "\n",
+ "**DSPy** accomodates a wide variety of applications and tasks. **In this intro notebook, we will work on the example task of multi-hop question answering (QA).**\n",
+ "\n",
+ "Other notebooks and tutorials will present different tasks. Now, let us load a tiny sample from the HotPotQA multi-hop dataset."
]
},
{
"cell_type": "code",
- "execution_count": 6,
+ "execution_count": 4,
"metadata": {},
- "outputs": [],
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "(20, 50)"
+ ]
+ },
+ "execution_count": 4,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
"source": [
- "def vanilla_LM_QA(question: str) -> str:\n",
- " demos = dsp.sample(train, k=7)\n",
- " example = dsp.Example(question=question, demos=demos)\n",
+ "from dspy.datasets import HotPotQA\n",
+ "\n",
+ "# Load the dataset.\n",
+ "dataset = HotPotQA(train_seed=1, train_size=20, eval_seed=2023, dev_size=50, test_size=0)\n",
+ "\n",
+ "# Tell DSPy that the 'question' field is the input. Any other fields are labels and/or metadata.\n",
+ "trainset = [x.with_inputs('question') for x in dataset.train]\n",
+ "devset = [x.with_inputs('question') for x in dataset.dev]\n",
"\n",
- " example, completions = dsp.generate(qa_template)(example, stage='qa')\n",
- " return completions.answer"
+ "len(trainset), len(devset)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "Let's invoke the program on a sample question."
+ "We just loaded `trainset` (20 examples) and `devset` (50 examples). Each example in our **training set** contains just a **question** and its (human-annotated) **answer**.\n",
+ "\n",
+ "**DSPy** typically requires very minimal labeling. Whereas your pipeline may involve six or seven complex steps, you only need labels for the initial question and the final answer. **DSPy** will bootstrap any intermediate labels needed to support your pipeline. If you change your pipeline in any way, the data bootstrapped will change accordingly!\n",
+ "\n",
+ "Now, let's look at some data examples."
]
},
{
"cell_type": "code",
- "execution_count": 7,
+ "execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
- "Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?\n",
- "E. L. Doctorow\n"
+ "Question: At My Window was released by which American singer-songwriter?\n",
+ "Answer: John Townes Van Zandt\n"
]
}
],
"source": [
- "print(dev[0].question)\n",
- "print(vanilla_LM_QA(dev[0].question))"
+ "train_example = trainset[0]\n",
+ "print(f\"Question: {train_example.question}\")\n",
+ "print(f\"Answer: {train_example.answer}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "Let's inspect the last call to the LM to learn more about the behavior of the program."
+ "Examples in the **dev set** contain a third field, namely, **titles** of relevant Wikipedia articles. This is not essential but, for the sake of this intro, it'll help us get a sense of how well our programs are doing."
]
},
{
"cell_type": "code",
- "execution_count": 8,
+ "execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
- "\n",
- "\n",
- "\n",
- "\n",
- "Answer questions with short factoid answers.\n",
- "\n",
- "---\n",
- "\n",
- "Follow the following format.\n",
- "\n",
- "Question: ${the question to be answered}\n",
- "Answer: ${a short factoid answer, often between 1 and 5 words}\n",
- "\n",
- "---\n",
- "\n",
- "Question: Which award did the first book of Gary Zukav receive?\n",
- "Answer: U.S. National Book Award\n",
- "\n",
- "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
- "Answer: Foxcatcher\n",
- "\n",
- "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
- "Answer: Kevin Greutert\n",
- "\n",
- "Question: Who produced the album that included a re-recording of \"Lithium\"?\n",
- "Answer: Butch Vig\n",
- "\n",
- "Question: What city was the victim of Joseph Druces working in?\n",
- "Answer: Boston, Massachusetts\n",
- "\n",
- "Question: In what year was the star of To Hell and Back born?\n",
- "Answer: 1925\n",
- "\n",
- "Question: Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?\n",
- "Answer:\u001b[32m E. L. Doctorow\u001b[0m\n",
- "\n",
- "\n",
- "\n"
+ "Question: What is the nationality of the chef and restaurateur featured in Restaurant: Impossible?\n",
+ "Answer: English\n",
+ "Relevant Wikipedia Titles: {'Restaurant: Impossible', 'Robert Irvine'}\n"
]
}
],
"source": [
- "lm.inspect_history(n=1)"
+ "dev_example = devset[18]\n",
+ "print(f\"Question: {dev_example.question}\")\n",
+ "print(f\"Answer: {dev_example.answer}\")\n",
+ "print(f\"Relevant Wikipedia Titles: {dev_example.gold_titles}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "We can see that the program has organized the examples in a simple prompt and generated an answer for the `dev[0]` question we asked.\n",
- "\n",
- "Let's try on the remaining development examples."
+ "After loading the raw data, we'd applied `x.with_inputs('question')` to each example to tell **DSPy** that our input field in each example will be just `question`. Any other fields are labels or metadata that are not given to the system."
]
},
{
"cell_type": "code",
- "execution_count": 9,
+ "execution_count": 7,
"metadata": {},
"outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "100%|██████████| 13/13 [00:00<00:00, 456.27it/s]\n"
- ]
- },
{
"name": "stdout",
"output_type": "stream",
"text": [
- "Answered 3 / 13 (23.1%) correctly.\n"
+ "For this dataset, training examples have input keys ['question'] and label keys ['answer']\n",
+ "For this dataset, dev examples have input keys ['question'] and label keys ['answer', 'gold_titles']\n"
]
- },
- {
- "data": {
- "text/html": [
- "<style type=\"text/css\" >\n",
- " #T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2 th {\n",
- " text-align: left;\n",
- " } #T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2 td {\n",
- " text-align: left;\n",
- " }</style><table id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2\" ><thead> <tr> <th class=\"blank level0\" ></th> <th class=\"col_heading level0 col0\" >question</th> <th class=\"col_heading level0 col1\" >answer</th> <th class=\"col_heading level0 col2\" >prediction</th> <th class=\"col_heading level0 col3\" >correct</th> </tr></thead><tbody>\n",
- " <tr>\n",
- " <th id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row0_col2\" class=\"data row0 col2\" >E. L. Doctorow</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row0_col3\" class=\"data row0 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row1_col2\" class=\"data row1 col2\" >The Long Island Serial Killer</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row1_col3\" class=\"data row1 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row2_col2\" class=\"data row2 col2\" >Melbourne, Australia</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row2_col3\" class=\"data row2 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row3_col2\" class=\"data row3 col2\" >1966</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row3_col3\" class=\"data row3 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row4_col2\" class=\"data row4 col2\" >John Braine</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row4_col3\" class=\"data row4 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row5_col3\" class=\"data row5 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row6_col2\" class=\"data row6 col2\" >Manchester, England</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row6_col3\" class=\"data row6 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row7_col2\" class=\"data row7 col2\" >July 30, 1953</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row7_col3\" class=\"data row7 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row8_col2\" class=\"data row8 col2\" >Hudson River</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row8_col3\" class=\"data row8 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row9_col2\" class=\"data row9 col2\" >Aerospace engineering</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row9_col3\" class=\"data row9 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row10_col2\" class=\"data row10 col2\" >Hong Kong</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row10_col3\" class=\"data row10 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row11_col2\" class=\"data row11 col2\" >Venus</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row11_col3\" class=\"data row11 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row12_col2\" class=\"data row12 col2\" >1457</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row12_col3\" class=\"data row12 col3\" >❌</td>\n",
- " </tr>\n",
- " </tbody></table>"
- ],
- "text/plain": [
- "<pandas.io.formats.style.Styler at 0x7f9bd91ce1c0>"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- },
- {
- "data": {
- "text/plain": [
- "23.1"
- ]
- },
- "execution_count": 9,
- "metadata": {},
- "output_type": "execute_result"
}
],
"source": [
- "from dsp.evaluation.utils import evaluate\n",
+ "print(f\"For this dataset, training examples have input keys {train_example.inputs().keys()} and label keys {train_example.labels().keys()}\")\n",
+ "print(f\"For this dataset, dev examples have input keys {dev_example.inputs().keys()} and label keys {dev_example.labels().keys()}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Note that there's nothing special about the HotPotQA dataset: it's just a list of examples.\n",
+ "\n",
+ "You can define your own examples as below. A future notebook will guide you through creating your own data in unusual or data-scarce settings, which is a context where **DSPy** excels.\n",
+ "\n",
+ "```\n",
+ "dspy.Example(field1=value, field2=value2, ...)\n",
+ "```"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### 3] Building Blocks\n",
+ "\n",
+ "In **DSPy**, we will maintain a clean separation between **defining your modules in a declarative way** and **calling them in a pipeline to solve the task**.\n",
"\n",
- "evaluate(vanilla_LM_QA, dev)"
+ "This allows you to focus on the information flow of your pipeline. **DSPy** will then take your program and automatically optimize **how to prompt** (or finetune) LMs **for your particular pipeline** so it works well.\n",
+ "\n",
+ "If you have experience with PyTorch, you can think of DSPy as the PyTorch of the foundation model space. Before we see this in action, let's first understand some key pieces."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "### Program 2: Retrieve-then-Read w/ GPT-3.5\n",
+ "##### Using the Language Model: **Signatures** & **Predictors**\n",
"\n",
- "The program above relies entirely on the factual knowledge memorized by GPT-3. As we have seen, it's sometimes correct but it's far from reliable.\n",
+ "Every call to the LM in a **DSPy** program needs to have a **Signature**.\n",
"\n",
- "Let's try to address this. Our first improvement to the vanilla **GPT-3.5** strategy is to include search results in the LM prompt, often called a *retrieve-then-read* approach.\n",
+ "A signature consists of three simple elements:\n",
"\n",
- "In particular, we'll retrieve the most relevant passage to the question (according to the retriever) from the corpus and concatenate it into the prompt. This may help GPT-3.5 answer more factually.\n",
+ "- A minimal description of the sub-task the LM is supposed to solve.\n",
+ "- A description of one or more input fields (e.g., input question) that will we will give to the LM.\n",
+ "- A description of one or more output fields (e.g., the question's answer) that we will expect from the LM.\n",
"\n",
- "To do this, we'll need to modify the `Template`. In particular, we'll define a `qa_template_with_passages` template, which will include a `context` field in addition to a question and an answer."
+ "Let's define a simple signature for basic question answering."
]
},
{
"cell_type": "code",
- "execution_count": 10,
+ "execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
- "Context = dsp.Type(\n",
- " prefix=\"Context:\\n\",\n",
- " desc=\"${sources that may contain relevant content}\",\n",
- " format=dsp.passages2text\n",
- ")\n",
+ "class BasicQA(dspy.Signature):\n",
+ " \"\"\"Answer questions with short factoid answers.\"\"\"\n",
"\n",
- "qa_template_with_passages = dsp.Template(\n",
- " instructions=qa_template.instructions,\n",
- " context=Context(), question=Question(), answer=Answer()\n",
- ")"
+ " question = dspy.InputField()\n",
+ " answer = dspy.OutputField(desc=\"often between 1 and 5 words\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "Now let's modify the program. We'll call `dsp.retrieve`, which we hooked earlier in this notebook to a **ColBERTv2** retriever serving a Wikipedia (Dec 2018) index."
+ "In `BasicQA`, the docstring describes the sub-task here (i.e., answering questions). Each `InputField` or `OutputField` can optionally contain a description `desc` too. When it's not given, it's inferred from the field's name (e.g., `question`).\n",
+ "\n",
+ "Notice that there isn't anything special about this signature in **DSPy**. We can just as easily define a signature that takes a long snippet from a PDF and outputs structured information, for instance.\n",
+ "\n",
+ "Anyway, now that we have a signature, let's define and use a **Predictor**. A predictor is a module that knows how to use the LM to implement a signature. Importantly, predictors can **learn** to fit their behavior to the task!"
]
},
{
"cell_type": "code",
- "execution_count": 11,
+ "execution_count": 9,
"metadata": {},
- "outputs": [],
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Question: What is the nationality of the chef and restaurateur featured in Restaurant: Impossible?\n",
+ "Predicted Answer: American\n"
+ ]
+ }
+ ],
"source": [
- "def retrieve_then_read_QA(question: str) -> str:\n",
- " demos = dsp.sample(train, k=7)\n",
- " passages = dsp.retrieve(question, k=1)\n",
- " \n",
- " example = dsp.Example(question=question, context=passages, demos=demos)\n",
- " example, completions = dsp.generate(qa_template_with_passages)(example, stage='qa')\n",
+ "# Define the predictor.\n",
+ "generate_answer = dspy.Predict(BasicQA)\n",
+ "\n",
+ "# Call the predictor on a particular input.\n",
+ "pred = generate_answer(question=dev_example.question)\n",
"\n",
- " return completions.answer"
+ "# Print the input and the prediction.\n",
+ "print(f\"Question: {dev_example.question}\")\n",
+ "print(f\"Predicted Answer: {pred.answer}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "Let's invoke this on a sample question and let's inspect the last call to the LM to learn more about the behavior of the program."
+ "In the example above, we asked the predictor about the the chef featured in \"Restaurant: Impossible\". The model outputs an answer (\"American\").\n",
+ "\n",
+ "For visibility, we can inspect how this extremely basic predictor implemented our signature. Let's inspect the history of our LM (**turbo**)."
]
},
{
"cell_type": "code",
- "execution_count": 12,
+ "execution_count": 10,
"metadata": {},
"outputs": [
{
@@ -501,209 +364,129 @@
"\n",
"---\n",
"\n",
- "Question: Which award did the first book of Gary Zukav receive?\n",
- "Answer: U.S. National Book Award\n",
- "\n",
- "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
- "Answer: Foxcatcher\n",
- "\n",
- "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
- "Answer: Kevin Greutert\n",
- "\n",
- "Question: Who produced the album that included a re-recording of \"Lithium\"?\n",
- "Answer: Butch Vig\n",
- "\n",
- "Question: What city was the victim of Joseph Druces working in?\n",
- "Answer: Boston, Massachusetts\n",
- "\n",
- "Question: In what year was the star of To Hell and Back born?\n",
- "Answer: 1925\n",
- "\n",
- "---\n",
- "\n",
"Follow the following format.\n",
"\n",
- "Context:\n",
- "${sources that may contain relevant content}\n",
- "\n",
- "Question: ${the question to be answered}\n",
- "\n",
- "Answer: ${a short factoid answer, often between 1 and 5 words}\n",
+ "Question: ${question}\n",
+ "Answer: often between 1 and 5 words\n",
"\n",
"---\n",
"\n",
- "Context:\n",
- "«Julia Peterkin | Julia Peterkin Julia Peterkin (October 31, 1880 – August 10, 1961) was an American author from South Carolina. In 1929 she won the Pulitzer Prize for Novel/Literature, for her novel \"Scarlet Sister Mary.\" She wrote several novels about the plantation South, especially the Gullah people of the Low Country. She was one of the few white authors who wrote about the African-American experience. Julia Mood was born in Laurens County, South Carolina. Her father was a physician, and she was the third of his four children. Her mother died soon after her birth, and her father later married Janie Brogdon.»\n",
- "\n",
- "Question: Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?\n",
- "\n",
- "Answer:\u001b[32m E. L. Doctorow\u001b[0m\n",
+ "Question: What is the nationality of the chef and restaurateur featured in Restaurant: Impossible?\n",
+ "Answer:\u001b[32m American\u001b[0m\n",
"\n",
"\n",
"\n"
]
- },
- {
- "data": {
- "text/plain": [
- "('E. L. Doctorow', None)"
- ]
- },
- "execution_count": 12,
- "metadata": {},
- "output_type": "execute_result"
}
],
"source": [
- "retrieve_then_read_QA(dev[0].question), lm.inspect_history(n=1)"
+ "turbo.inspect_history(n=1)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "Now, let's evaluate this on the dev examples."
+ "It happens that this chef is both British and American, but we have no way of knowing if the model just guessed \"American\" because it's a common answer. In general, adding **retrieval** and **learning** will help the LM be more factual, and we'll explore this in a minute!\n",
+ "\n",
+ "But before we do that, how about we _just_ change the predictor? It would be nice to allow the model to elicit a chain of thought along with the prediction."
]
},
{
"cell_type": "code",
- "execution_count": 13,
+ "execution_count": 11,
"metadata": {},
"outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "100%|██████████| 13/13 [00:00<00:00, 376.92it/s]"
- ]
- },
{
"name": "stdout",
"output_type": "stream",
"text": [
- "Answered 5 / 13 (38.5%) correctly.\n"
+ "Question: What is the nationality of the chef and restaurateur featured in Restaurant: Impossible?\n",
+ "Thought: We know that the chef and restaurateur featured in Restaurant: Impossible is Robert Irvine.\n",
+ "Predicted Answer: British\n"
]
- },
+ }
+ ],
+ "source": [
+ "# Define the predictor. Notice we're just changing the class. The signature BasicQA is unchanged.\n",
+ "generate_answer_with_chain_of_thought = dspy.ChainOfThought(BasicQA)\n",
+ "\n",
+ "# Call the predictor on the same input.\n",
+ "pred = generate_answer_with_chain_of_thought(question=dev_example.question)\n",
+ "\n",
+ "# Print the input, the chain of thought, and the prediction.\n",
+ "print(f\"Question: {dev_example.question}\")\n",
+ "print(f\"Thought: {pred.rationale.split('.', 1)[1].strip()}\")\n",
+ "print(f\"Predicted Answer: {pred.answer}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "This is indeed a better answer: the model figures out that the chef in question is **Robert Irvine** and correctly identifies that he's British.\n",
+ "\n",
+ "These predictors (`dspy.Predict` and `dspy.ChainOfThought`) can be applied to _any_ signature. As we'll see below, they can also be optimized to learn from your data and validation logic."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "##### Using the Retrieval Model\n",
+ "\n",
+ "Using the retriever is pretty simple. A module `dspy.Retrieve(k)` will search for the top-`k` passages that match a given query.\n",
+ "\n",
+ "By default, this will use the retriever we configured at the top of this notebook, namely, ColBERTv2 over a Wikipedia index."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 12,
+ "metadata": {},
+ "outputs": [
{
- "name": "stderr",
+ "name": "stdout",
"output_type": "stream",
"text": [
+ "Top 3 passages for question: What is the nationality of the chef and restaurateur featured in Restaurant: Impossible? \n",
+ " ------------------------------ \n",
+ "\n",
+ "1] Restaurant: Impossible | Restaurant: Impossible is an American reality television series, featuring chef and restaurateur Robert Irvine, that aired on Food Network from 2011 to 2016. \n",
+ "\n",
+ "2] Jean Joho | Jean Joho is a French-American chef and restaurateur. He is chef/proprietor of Everest in Chicago (founded in 1986), Paris Club Bistro & Bar and Studio Paris in Chicago, The Eiffel Tower Restaurant in Las Vegas, and Brasserie JO in Boston. \n",
+ "\n",
+ "3] List of Restaurant: Impossible episodes | This is the list of the episodes for the American cooking and reality television series \"Restaurant Impossible\", produced by Food Network. The premise of the series is that within two days and on a budget of $10,000, celebrity chef Robert Irvine renovates a failing American restaurant with the goal of helping to restore it to profitability and prominence. Irvine is assisted by a designer (usually Taniya Nayak, Cheryl Torrenueva, or Lynn Keagan, but sometimes Vanessa De Leon, Krista Watterworth, Yvette Irene, or Nicole Faccuito), along with general contractor Tom Bury, who sometimes does double duty as both general contractor and designer. After assessing the problems with the restaurant, Robert Irvine typically creates a plan for the new decor, oversees the cleaning of the restaurant, reduces the size of the menu and improves the food, develops a promotional activity, educates the restaurant's owners, or trains the staff, as needed by each restaurant. \n",
"\n"
]
- },
- {
- "data": {
- "text/html": [
- "<style type=\"text/css\" >\n",
- " #T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2 th {\n",
- " text-align: left;\n",
- " } #T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2 td {\n",
- " text-align: left;\n",
- " }</style><table id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2\" ><thead> <tr> <th class=\"blank level0\" ></th> <th class=\"col_heading level0 col0\" >question</th> <th class=\"col_heading level0 col1\" >answer</th> <th class=\"col_heading level0 col2\" >prediction</th> <th class=\"col_heading level0 col3\" >correct</th> </tr></thead><tbody>\n",
- " <tr>\n",
- " <th id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row0_col2\" class=\"data row0 col2\" >E. L. Doctorow</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row0_col3\" class=\"data row0 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row1_col2\" class=\"data row1 col2\" >The Killing Season</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row1_col3\" class=\"data row1 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row2_col2\" class=\"data row2 col2\" >Daytona Beach, Florida.</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row2_col3\" class=\"data row2 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row3_col2\" class=\"data row3 col2\" >1854</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row3_col3\" class=\"data row3 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row4_col2\" class=\"data row4 col2\" >John Braine is English.</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row4_col3\" class=\"data row4 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row5_col3\" class=\"data row5 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row6_col2\" class=\"data row6 col2\" >Manchester, England</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row6_col3\" class=\"data row6 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row7_col2\" class=\"data row7 col2\" >August 5, 1962</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row7_col3\" class=\"data row7 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row8_col2\" class=\"data row8 col2\" >Dundalk, Ireland</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row8_col3\" class=\"data row8 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row9_col2\" class=\"data row9 col2\" >Jet Propulsion</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row9_col3\" class=\"data row9 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row10_col2\" class=\"data row10 col2\" >Enid, Oklahoma</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row10_col3\" class=\"data row10 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row11_col2\" class=\"data row11 col2\" >Portrait of a Marriage</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row11_col3\" class=\"data row11 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row12_col2\" class=\"data row12 col2\" >1442</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row12_col3\" class=\"data row12 col3\" >✔️</td>\n",
- " </tr>\n",
- " </tbody></table>"
- ],
- "text/plain": [
- "<pandas.io.formats.style.Styler at 0x7f9bb40bea30>"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- },
+ }
+ ],
+ "source": [
+ "retrieve = dspy.Retrieve(k=3)\n",
+ "topK_passages = retrieve(dev_example.question).passages\n",
+ "\n",
+ "print(f\"Top {retrieve.k} passages for question: {dev_example.question} \\n\", '-' * 30, '\\n')\n",
+ "\n",
+ "for idx, passage in enumerate(topK_passages):\n",
+ " print(f'{idx+1}]', passage, '\\n')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Feel free to any other queries you like."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 13,
+ "metadata": {},
+ "outputs": [
{
"data": {
"text/plain": [
- "38.5"
+ "'History of the FIFA World Cup | The FIFA World Cup was first held in 1930, when FIFA president Jules Rimet decided to stage an international football tournament. The inaugural edition, held in 1930, was contested as a final tournament of only thirteen teams invited by the organization. Since then, the World Cup has experienced successive expansions and format remodeling to its current 32-team final tournament preceded by a two-year qualifying process, involving over 200 teams from around the world.'"
]
},
"execution_count": 13,
@@ -712,24 +495,20 @@
}
],
"source": [
- "evaluate(retrieve_then_read_QA, dev)"
+ "retrieve(\"When was the first FIFA World Cup held?\").passages[0]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "### Program 3: Retrieve-then-Read w/ Self-Consistency\n",
+ "### 4] Program 1: Basic Retrieval-Augmented Generation (“RAG”)\n",
"\n",
- "It's clear that retrieval has the capacity to help **GPT-3.5** answer a larger number of questions more factually. However, these questions are too complicated for a single retrieved passage to suffice.\n",
+ "Let's define our first complete program for this task. We'll build a retrieval-augmented pipeline for answer generation.\n",
"\n",
- "In this program, we try to make some improvements to the pipeline from **Program 2**.\n",
+ "Given a question, we'll search for the top-3 passages in Wikipedia and then feed them as context for answer generation.\n",
"\n",
- "In particular:\n",
- "\n",
- "- We will include 5 passages (instead of a single passage) into the prompt.\n",
- "- We will ask GPT-3.5 to generate a Chain-of-Thought (CoT) rationale to more effectively extract the answer from the passages.\n",
- "- We will use Self-Consistency (SC) to marginalize this prediction across many chains of thought."
+ "Let's start by defining this signature: `context, question --> answer`."
]
},
{
@@ -738,26 +517,24 @@
"metadata": {},
"outputs": [],
"source": [
- "Rationale = dsp.Type(\n",
- " prefix=\"Rationale: Let's think step by step.\",\n",
- " desc=\"${a step-by-step deduction that identifies the correct response, which will be provided below}\"\n",
- ")\n",
+ "class GenerateAnswer(dspy.Signature):\n",
+ " \"\"\"Answer questions with short factoid answers.\"\"\"\n",
"\n",
- "qa_template_with_CoT = dsp.Template(\n",
- " instructions=qa_template.instructions,\n",
- " context=Context(), question=Question(), rationale=Rationale(), answer=Answer()\n",
- ")"
+ " context = dspy.InputField(desc=\"may contain relevant facts\")\n",
+ " question = dspy.InputField()\n",
+ " answer = dspy.OutputField(desc=\"often between 1 and 5 words\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "The snippet above updates our template to include a `rationale` field.\n",
+ "Great. Now let's define the actual program. This is a class that inherits from `dspy.Module`.\n",
"\n",
- "Below, we define the new program `retrieve_then_read_QA_v2`. In it, we define a new DSP _transformation_ `QA_predict`, which we will re-use in multiple programs.\n",
+ "It needs two methods:\n",
"\n",
- "The transformation `QA_predict` takes an `Example` (which has a question, context, and demonstrations) and generates an answer. It can be configured to use self-consistency (`sc=True`) or not."
+ "- The `__init__` method will simply declare the sub-modules it needs: `dspy.Retrieve` and `dspy.ChainOfThought`. The latter is defined to implement our `GenerateAnswer` signature.\n",
+ "- The `forward` method will describe the control flow of answering the question using the modules we have."
]
},
{
@@ -766,29 +543,44 @@
"metadata": {},
"outputs": [],
"source": [
- "@dsp.transformation\n",
- "def QA_predict(example: dsp.Example, sc=True):\n",
- " if sc:\n",
- " example, completions = dsp.generate(qa_template_with_CoT, n=20, temperature=0.7)(example, stage='qa')\n",
- " completions = dsp.majority(completions)\n",
- " else:\n",
- " example, completions = dsp.generate(qa_template_with_CoT)(example, stage='qa')\n",
- " \n",
- " return example.copy(answer=completions.answer)\n",
+ "class RAG(dspy.Module):\n",
+ " def __init__(self, num_passages=3):\n",
+ " super().__init__()\n",
"\n",
- "def retrieve_then_read_QA_v2(question: str) -> str:\n",
- " demos = dsp.sample(train, k=7)\n",
- " passages = dsp.retrieve(question, k=5)\n",
- " example = dsp.Example(question=question, context=passages, demos=demos)\n",
+ " self.retrieve = dspy.Retrieve(k=num_passages)\n",
+ " self.generate_answer = dspy.ChainOfThought(GenerateAnswer)\n",
" \n",
- " return QA_predict(example).answer"
+ " def forward(self, question):\n",
+ " context = self.retrieve(question).passages\n",
+ " prediction = self.generate_answer(context=context, question=question)\n",
+ " return dspy.Prediction(context=context, answer=prediction.answer)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "Let's inspect an example."
+ "##### Compiling the RAG program\n",
+ "\n",
+ "Having defined this program, let's now **compile** it. Compiling a program will update the parameters stored in each module. In our setting, this is primarily in the form of collecting and selecting good demonstrations for inclusion in your prompt(s).\n",
+ "\n",
+ "Compiling depends on three things:\n",
+ "\n",
+ "1. **A training set.** We'll just use our 20 question–answer examples from `trainset` above.\n",
+ "1. **A metric for validation.** We'll define a quick `validate_context_and_answer` that checks that the predicted answer is correct. It'll also check that the retrieved context does actually contain that answer.\n",
+ "1. **A specific teleprompter.** The **DSPy** compiler includes a number of **teleprompters** that can optimize your programs."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "**Teleprompters:** Teleprompters are powerful optimizers that can take any program and learn to bootstrap and select effective prompts for its modules. Hence the name, which means \"prompting at a distance\".\n",
+ "\n",
+ "Different teleprompters offer various tradeoffs in terms of how much they optimize cost versus quality, etc. We will use a simple default `BootstrapFewShot` in this notebook.\n",
+ "\n",
+ "\n",
+ "_If you're into analogies, you could think of this as your training data, your loss function, and your optimizer in a standard DNN supervised learning setup. Whereas SGD is a basic optimizer, there are more sophisticated (and more expensive!) ones like Adam or RMSProp._"
]
},
{
@@ -797,356 +589,89 @@
"metadata": {},
"outputs": [
{
- "name": "stdout",
+ "name": "stderr",
"output_type": "stream",
"text": [
- "\n",
- "\n",
- "\n",
- "\n",
- "Answer questions with short factoid answers.\n",
- "\n",
- "---\n",
- "\n",
- "Question: Which award did the first book of Gary Zukav receive?\n",
- "Answer: U.S. National Book Award\n",
- "\n",
- "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
- "Answer: Foxcatcher\n",
- "\n",
- "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
- "Answer: Kevin Greutert\n",
- "\n",
- "Question: Who produced the album that included a re-recording of \"Lithium\"?\n",
- "Answer: Butch Vig\n",
- "\n",
- "Question: What city was the victim of Joseph Druces working in?\n",
- "Answer: Boston, Massachusetts\n",
- "\n",
- "Question: In what year was the star of To Hell and Back born?\n",
- "Answer: 1925\n",
- "\n",
- "---\n",
- "\n",
- "Follow the following format.\n",
- "\n",
- "Context:\n",
- "${sources that may contain relevant content}\n",
- "\n",
- "Question: ${the question to be answered}\n",
- "\n",
- "Rationale: Let's think step by step. ${a step-by-step deduction that identifies the correct response, which will be provided below}\n",
- "\n",
- "Answer: ${a short factoid answer, often between 1 and 5 words}\n",
- "\n",
- "---\n",
- "\n",
- "Context:\n",
- "[1] «Right Back at It Again | Right Back at It Again \"Right Back at It Again\" is the second track and the first single from A Day to Remember's fifth album, \"Common Courtesy\" (2013). In October 20, 2015, the song was featured in Activision rhythm-music game, \"\". Vocalist, Jeremy McKinnon wrote the lyrics, while the music was written by McKinnon, former guitarist Tom Denney, guitarist Neil Westfall and producer Andrew Wade. \"Right Back at It Again\" almost wasn't included on the album as it was one of the excess songs the band had recorded, \"we realised that it sounded great, so on it went.\" \"Right Back»\n",
- "[2] «Right Back at It Again | at the Kerrang! Awards. Personnel per digital booklet. Right Back at It Again \"Right Back at It Again\" is the second track and the first single from A Day to Remember's fifth album, \"Common Courtesy\" (2013). In October 20, 2015, the song was featured in Activision rhythm-music game, \"\". Vocalist, Jeremy McKinnon wrote the lyrics, while the music was written by McKinnon, former guitarist Tom Denney, guitarist Neil Westfall and producer Andrew Wade. \"Right Back at It Again\" almost wasn't included on the album as it was one of the excess songs the band had recorded, \"we realised that it»\n",
- "[3] «Right Back at It Again | band played \"Right Back at It Again\" on both occasions. AbsolutePunk writer Thomas Nassiff said the \"mosh call\" lyric \"\"Brace for impact\"\" had \"pretty much ruin[ed] that track\". Tamsyn Wilce for Alter the Press! noted that the way \"Right Back at it Again\" follows on from opening track \"City of Ocala\" \"continues the catchy hardcore beats\" that is present in all of the band's albums. \"Rock Sound\"'s Andy Ritchie called the song \"unmistakably the 'All I Want' of 'Common Courtesy'\". A music video for the song was released on 19 December 2013. The music video was nominated for Best Video»\n",
- "[4] «Born Again in Dixieland | Born Again in Dixieland \"Born Again in Dixieland\" is a song co-written and recorded by Canadian country music artist Jason McCoy. Released in June 1997, it was the lead single from his album \"Playin' for Keeps\". The song reached #3 on the \"RPM\" Country Tracks chart in September 1997. The song won a SOCAN Song of the Year award. McCoy said that he started writing the song with Sheridan and he was playing songs for MCA Canada label and they asked if he had any other songs. He played a verse and the chorus from the song thinking it wasn't»\n",
- "[5] «Right Back Where We Started From | 1974. In a 3 May 2008 interview with Michael Shelley of WFMU, Edwards recalled that after hearing Maxine Nightingale sing on the session for Al Matthews' \"Fool\" that track's producer Pierre Tubbs had come up with \"Right Back Where We Started From\" as a good title for a song for Nightingale herself to record and had invited Edwards to co-write the song. Utilizing a tune which Edwards had written \"a couple of years before\", Tubbs and Edwards wrote \"Right Back Where We Started From\" in about seven minutes while driving to Charing Cross Hospital where Tubbs' wife Gabrielle (née Zimmerman)»\n",
- "\n",
- "Question: Right Back At It Again contains lyrics co-written by the singer born in what city?\n",
- "\n",
- "Rationale: Let's think step by step.\u001b[32m The song is by A Day to Remember, and the lyrics were co-written by the singer, Jeremy McKinnon. We just need to find out where McKinnon was born.\n",
- "\n",
- "Answer: Orlando, Florida\u001b[0m\u001b[31m \t (and 19 other completions)\u001b[0m\n",
- "\n",
- "\n",
- "\n"
+ " 50%|█████ | 10/20 [00:03<00:03, 2.66it/s]"
]
},
{
- "data": {
- "text/plain": [
- "('Ocala, Florida', None)"
- ]
- },
- "execution_count": 16,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "retrieve_then_read_QA_v2(dev[2].question), lm.inspect_history(n=1)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "And let's run this on the other dev examples."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 17,
- "metadata": {},
- "outputs": [
- {
- "name": "stderr",
+ "name": "stdout",
"output_type": "stream",
"text": [
- "100%|██████████| 13/13 [00:01<00:00, 11.23it/s]\n"
+ "Bootstrapped 4 full traces after 11 examples in round 0.\n"
]
},
{
- "name": "stdout",
+ "name": "stderr",
"output_type": "stream",
"text": [
- "Answered 7 / 13 (53.8%) correctly.\n"
+ "\n"
]
- },
- {
- "data": {
- "text/html": [
- "<style type=\"text/css\" >\n",
- " #T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2 th {\n",
- " text-align: left;\n",
- " } #T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2 td {\n",
- " text-align: left;\n",
- " }</style><table id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2\" ><thead> <tr> <th class=\"blank level0\" ></th> <th class=\"col_heading level0 col0\" >question</th> <th class=\"col_heading level0 col1\" >answer</th> <th class=\"col_heading level0 col2\" >prediction</th> <th class=\"col_heading level0 col3\" >correct</th> </tr></thead><tbody>\n",
- " <tr>\n",
- " <th id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row0_col2\" class=\"data row0 col2\" >E. L. Doctorow</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row0_col3\" class=\"data row0 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row1_col2\" class=\"data row1 col2\" >The Killing Season</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row1_col3\" class=\"data row1 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row2_col2\" class=\"data row2 col2\" >Ocala, Florida</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row2_col3\" class=\"data row2 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row3_col2\" class=\"data row3 col2\" >1824</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row3_col3\" class=\"data row3 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row4_col2\" class=\"data row4 col2\" >John Braine</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row4_col3\" class=\"data row4 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row5_col3\" class=\"data row5 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row6_col2\" class=\"data row6 col2\" >Cheshire, UK</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row6_col3\" class=\"data row6 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row7_col2\" class=\"data row7 col2\" >September 27, 1974</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row7_col3\" class=\"data row7 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row8_col2\" class=\"data row8 col2\" >Dundalk</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row8_col3\" class=\"data row8 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row9_col2\" class=\"data row9 col2\" >Aeronautics and Astronautics</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row9_col3\" class=\"data row9 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row10_col2\" class=\"data row10 col2\" >Italy</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row10_col3\" class=\"data row10 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row11_col2\" class=\"data row11 col2\" >Portrait of a Marriage</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row11_col3\" class=\"data row11 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row12_col2\" class=\"data row12 col2\" >1442</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row12_col3\" class=\"data row12 col3\" >✔️</td>\n",
- " </tr>\n",
- " </tbody></table>"
- ],
- "text/plain": [
- "<pandas.io.formats.style.Styler at 0x7f9baab4a850>"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- },
- {
- "data": {
- "text/plain": [
- "53.8"
- ]
- },
- "execution_count": 17,
- "metadata": {},
- "output_type": "execute_result"
}
],
"source": [
- "evaluate(retrieve_then_read_QA_v2, dev)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Program 4: Multi-Hop Retrieval\n",
- "\n",
- "From these simple programs, it becomes clear that a single search query is not enough for this task. This can be seen when trying to find the birth city of the writer of \"Right Back At It Again\". A search query identifies the author correctly as \"Jeremy McKinnon\", but it doesn't mention when he was born.\n",
- "\n",
- "The standard approach for this challenge in the retrieval-augmented NLP literature is to build multi-hop search systems, like GoldEn (Qi et al., 2019), IRRR (Qi et al., 2021) and Baleen (Khattab et al., 2021). These systems read the retrieved results and then generate additional queries to gather additional information if necessary."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "\n",
- "With **DSP**, we can easily simulate such systems in a few lines of code. Let's begin by defining two new templates.\n",
- "\n",
- "First, in `rewrite_template`, we will re-write the input question into a simple search query. This will serve as our \"first hop\" query."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 18,
- "metadata": {},
- "outputs": [],
- "source": [
- "SearchRationale = dsp.Type(\n",
- " prefix=\"Rationale: Let's think step by step. To answer this question, we first need to find out\",\n",
- " desc=\"${the missing information}\"\n",
- ")\n",
+ "from dspy.teleprompt import BootstrapFewShot\n",
"\n",
- "SearchQuery = dsp.Type(\n",
- " prefix=\"Search Query:\",\n",
- " desc=\"${a simple question for seeking the missing information}\"\n",
- ")\n",
+ "# Validation logic: check that the predicted answer is correct.\n",
+ "# Also check that the retrieved context does actually contain that answer.\n",
+ "def validate_context_and_answer(example, pred, trace=None):\n",
+ " answer_EM = dspy.evaluate.answer_exact_match(example, pred)\n",
+ " answer_PM = dspy.evaluate.answer_passage_match(example, pred)\n",
+ " return answer_EM and answer_PM\n",
"\n",
- "rewrite_template = dsp.Template(\n",
- " instructions=\"Write a search query that will help answer a complex question.\",\n",
- " question=Question(), rationale=SearchRationale(), query=SearchQuery()\n",
- ")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Next, in `hop_template`, we will use the retrieved information from earlier hops to generate additional search queries for missing information."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 19,
- "metadata": {},
- "outputs": [],
- "source": [
- "CondenseRationale = dsp.Type(\n",
- " prefix=\"Rationale: Let's think step by step. Based on the context, we have learned the following.\",\n",
- " desc=\"${information from the context that provides useful clues}\"\n",
- ")\n",
+ "# Set up a basic teleprompter, which will compile our RAG program.\n",
+ "teleprompter = BootstrapFewShot(metric=validate_context_and_answer)\n",
"\n",
- "hop_template = dsp.Template(\n",
- " instructions=rewrite_template.instructions,\n",
- " context=Context(), question=Question(), rationale=CondenseRationale(), query=SearchQuery()\n",
- ")"
+ "# Compile!\n",
+ "compiled_rag = teleprompter.compile(RAG(), trainset=trainset)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "Let's build our `multihop_QA_v1` program. The hallmark of this program is the search transformation called `multihop_search_v1`.\n",
- "\n",
- "This will largely simulate the IRRR system. In each hop, we will retrieve `k` (e.g., `k=2`) passages and concatenate them to the context retrieved from earlier hops to form a `context` chain.\n",
- "\n",
- "For simplicity, we will repeat this for a total of `max_hops` hops (e.g., `max_hops=2`). It's easy to add a stopping criteria in which the LM is asked whether more hops are required."
+ "Now that we've compiled our RAG program, let's try it out."
]
},
{
"cell_type": "code",
- "execution_count": 20,
+ "execution_count": 17,
"metadata": {},
- "outputs": [],
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Question: What castle did David Gregory inherit?\n",
+ "Predicted Answer: Kinnairdy Castle\n",
+ "Retrieved Contexts (truncated): ['David Gregory (physician) | David Gregory (20 December 1625 – 1720) was a Scottish physician and inventor. His surname is sometimes spelt as Gregorie, the original Scottish spelling. He inherited Kinn...', 'Gregory Tarchaneiotes | Gregory Tarchaneiotes (Greek: Γρηγόριος Ταρχανειώτης , Italian: \"Gregorio Tracanioto\" or \"Tracamoto\" ) was a \"protospatharius\" and the long-reigning catepan of Italy from 998 t...', 'David Gregory (mathematician) | David Gregory (originally spelt Gregorie) FRS (? 1659 – 10 October 1708) was a Scottish mathematician and astronomer. He was professor of mathematics at the University ...']\n"
+ ]
+ }
+ ],
"source": [
- "from dsp.utils import deduplicate\n",
- "\n",
- "@dsp.transformation\n",
- "def multihop_search_v1(example: dsp.Example, max_hops=2, k=2) -> dsp.Example:\n",
- " example.context = []\n",
- " \n",
- " for hop in range(max_hops):\n",
- " # Generate a query based\n",
- " template = rewrite_template if hop == 0 else hop_template\n",
- " example, completions = dsp.generate(template)(example, stage=f'h{hop}')\n",
- "\n",
- " # Retrieve k results based on the query generated\n",
- " passages = dsp.retrieve(completions.query, k=k)\n",
- "\n",
- " # Update the context by concatenating old and new passages\n",
- " example.context = deduplicate(example.context + passages)\n",
- "\n",
- " return example\n",
- "\n",
+ "# Ask any question you like to this simple RAG program.\n",
+ "my_question = \"What castle did David Gregory inherit?\"\n",
"\n",
- "def multihop_QA_v1(question: str) -> str:\n",
- " demos = dsp.sample(train, k=7)\n",
- " x = dsp.Example(question=question, demos=demos)\n",
- " \n",
- " x = multihop_search_v1(x)\n",
- " x = QA_predict(x, sc=False)\n",
+ "# Get the prediction. This contains `pred.context` and `pred.answer`.\n",
+ "pred = compiled_rag(my_question)\n",
"\n",
- " return x.answer"
+ "# Print the contexts and the answer.\n",
+ "print(f\"Question: {my_question}\")\n",
+ "print(f\"Predicted Answer: {pred.answer}\")\n",
+ "print(f\"Retrieved Contexts (truncated): {[c[:200] + '...' for c in pred.context]}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "Let's run this on one example and inspect the last 3 calls to the LM (i.e., search query 1, search query 2, and QA)."
+ "Excellent. How about we inspect the last prompt for the LM?"
]
},
{
"cell_type": "code",
- "execution_count": 21,
+ "execution_count": 18,
"metadata": {},
"outputs": [
{
@@ -1157,153 +682,365 @@
"\n",
"\n",
"\n",
- "Write a search query that will help answer a complex question.\n",
+ "Answer questions with short factoid answers.\n",
"\n",
"---\n",
"\n",
- "Follow the following format.\n",
+ "Question: At My Window was released by which American singer-songwriter?\n",
+ "Answer: John Townes Van Zandt\n",
"\n",
- "Question: ${the question to be answered}\n",
- "Rationale: Let's think step by step. To answer this question, we first need to find out ${the missing information}\n",
- "Search Query: ${a simple question for seeking the missing information}\n",
+ "Question: \"Everything Has Changed\" is a song from an album released under which record label ?\n",
+ "Answer: Big Machine Records\n",
"\n",
- "---\n",
+ "Question: The Victorians - Their Story In Pictures is a documentary series written by an author born in what year?\n",
+ "Answer: 1950\n",
+ "\n",
+ "Question: Which Pakistani cricket umpire who won 3 consecutive ICC umpire of the year awards in 2009, 2010, and 2011 will be in the ICC World Twenty20?\n",
+ "Answer: Aleem Sarwar Dar\n",
"\n",
- "Question: Right Back At It Again contains lyrics co-written by the singer born in what city?\n",
- "Rationale: Let's think step by step. To answer this question, we first need to find out\u001b[32m the name of the singer\n",
- "Search Query: Who sings Right Back At It Again\u001b[0m\n",
+ "Question: Having the combination of excellent foot speed and bat speed helped Eric Davis, create what kind of outfield for the Los Angeles Dodgers?\n",
+ "Answer: \"Outfield of Dreams\"\n",
"\n",
+ "Question: Who is older, Aleksandr Danilovich Aleksandrov or Anatoly Fomenko?\n",
+ "Answer: Aleksandr Danilovich Aleksandrov\n",
"\n",
+ "Question: The Organisation that allows a community to influence their operation or use and to enjoy the benefits arisingwas founded in what year?\n",
+ "Answer: 2010\n",
"\n",
+ "Question: Tombstone stared an actor born May 17, 1955 known as who?\n",
+ "Answer: Bill Paxton\n",
"\n",
+ "Question: In what year was the club founded that played Manchester City in the 1972 FA Charity Shield\n",
+ "Answer: 1874\n",
"\n",
+ "Question: which American actor was Candace Kita guest starred with\n",
+ "Answer: Bill Murray\n",
"\n",
+ "Question: Which is taller, the Empire State Building or the Bank of America Tower?\n",
+ "Answer: The Empire State Building\n",
"\n",
- "Write a search query that will help answer a complex question.\n",
+ "Question: Which company distributed this 1977 American animated film produced by Walt Disney Productions for which Sherman Brothers wrote songs?\n",
+ "Answer: Buena Vista Distribution\n",
"\n",
"---\n",
"\n",
"Follow the following format.\n",
"\n",
- "Context:\n",
- "${sources that may contain relevant content}\n",
+ "Context: may contain relevant facts\n",
"\n",
- "Question: ${the question to be answered}\n",
+ "Question: ${question}\n",
"\n",
- "Rationale: Let's think step by step. Based on the context, we have learned the following. ${information from the context that provides useful clues}\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
"\n",
- "Search Query: ${a simple question for seeking the missing information}\n",
+ "Answer: often between 1 and 5 words\n",
"\n",
"---\n",
"\n",
"Context:\n",
- "[1] «Right Back at It Again | at the Kerrang! Awards. Personnel per digital booklet. Right Back at It Again \"Right Back at It Again\" is the second track and the first single from A Day to Remember's fifth album, \"Common Courtesy\" (2013). In October 20, 2015, the song was featured in Activision rhythm-music game, \"\". Vocalist, Jeremy McKinnon wrote the lyrics, while the music was written by McKinnon, former guitarist Tom Denney, guitarist Neil Westfall and producer Andrew Wade. \"Right Back at It Again\" almost wasn't included on the album as it was one of the excess songs the band had recorded, \"we realised that it»\n",
- "[2] «Right Back at It Again | Right Back at It Again \"Right Back at It Again\" is the second track and the first single from A Day to Remember's fifth album, \"Common Courtesy\" (2013). In October 20, 2015, the song was featured in Activision rhythm-music game, \"\". Vocalist, Jeremy McKinnon wrote the lyrics, while the music was written by McKinnon, former guitarist Tom Denney, guitarist Neil Westfall and producer Andrew Wade. \"Right Back at It Again\" almost wasn't included on the album as it was one of the excess songs the band had recorded, \"we realised that it sounded great, so on it went.\" \"Right Back»\n",
- "\n",
- "Question: Right Back At It Again contains lyrics co-written by the singer born in what city?\n",
- "\n",
- "Rationale: Let's think step by step. Based on the context, we have learned the following.\u001b[32m The song is called \"Right Back at It Again\" and it is by the band A Day to Remember. The singer, Jeremy McKinnon, wrote the lyrics.\n",
+ "[1] «Tae Kwon Do Times | Tae Kwon Do Times is a magazine devoted to the martial art of taekwondo, and is published in the United States of America. While the title suggests that it focuses on taekwondo exclusively, the magazine also covers other Korean martial arts. \"Tae Kwon Do Times\" has published articles by a wide range of authors, including He-Young Kimm, Thomas Kurz, Scott Shaw, and Mark Van Schuyver.»\n",
+ "[2] «Kwon Tae-man | Kwon Tae-man (born 1941) was an early Korean hapkido practitioner and a pioneer of the art, first in Korea and then in the United States. He formed one of the earliest dojang's for hapkido in the United States in Torrance, California, and has been featured in many magazine articles promoting the art.»\n",
+ "[3] «Hee Il Cho | Cho Hee Il (born October 13, 1940) is a prominent Korean-American master of taekwondo, holding the rank of 9th \"dan\" in the martial art. He has written 11 martial art books, produced 70 martial art training videos, and has appeared on more than 70 martial arts magazine covers. Cho won several national and international competitions as a taekwondo competitor, and has appeared in several films, including \"Fight to Win\", \"Best of the Best\", \"Bloodsport II\", and \"Bloodsport III\". He founded the Action International Martial Arts Association (AIMAA) in 1980, and is its President. Cho is a member of both \"Black Belt\" magazine's Hall of Fame and \"Tae Kwon Do Times\" magazine's Hall of Fame.»\n",
"\n",
- "Search Query: Where was Jeremy McKinnon born?\u001b[0m\n",
+ "Question: Which magazine has published articles by Scott Shaw, Tae Kwon Do Times or Southwest Art?\n",
"\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know from the context that \"Tae Kwon Do Times\" is a magazine that covers taekwondo and other Korean martial arts. It has published articles by authors like Scott Shaw. On the other hand, there is no information about Southwest Art magazine in the context.\n",
"\n",
+ "Answer: Tae Kwon Do Times\n",
"\n",
+ "---\n",
"\n",
+ "Context:\n",
+ "[1] «Rosario Dawson | Rosario Isabel Dawson (born May 9, 1979) is an American actress, producer, singer, comic book writer, and political activist. She made her film debut in the 1995 teen drama \"Kids\". Her subsequent film roles include \"He Got Game\", \"Men in Black II\", \"25th Hour\", \"Rent\", \"Sin City\", \"Death Proof\", \"Seven Pounds\", \"\", and \"Top Five\". Dawson has also provided voice-over work for Disney and DC.»\n",
+ "[2] «Sarai Gonzalez | Sarai Isaura Gonzalez (born 2005) is an American Latina child actress who made her professional debut at the age of 11 on the Spanish-language \"\"Soy Yo\"\" (\"That's Me\") music video by Bomba Estéreo. Cast as a \"nerdy\" tween with a \"sassy\" and \"confident\" attitude, her performance turned her into a \"Latina icon\" for \"female empowerment, identity and self-worth\". She subsequently appeared in two get out the vote videos for Latinos in advance of the 2016 United States elections.»\n",
+ "[3] «Gabriela (2001 film) | Gabriela is a 2001 American romance film, starring Seidy Lopez in the title role alongside Jaime Gomez as her admirer Mike. The film has been cited as an inspiration behind the Premiere Weekend Club, which supports Latino film-making.»\n",
"\n",
+ "Question: Which American actress who made their film debut in the 1995 teen drama \"Kids\" was the co-founder of Voto Latino?\n",
"\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the actress made her film debut in 1995 and co-founded Voto Latino.\n",
"\n",
- "Answer questions with short factoid answers.\n",
+ "Answer: Rosario Dawson\n",
"\n",
"---\n",
"\n",
- "Question: Which award did the first book of Gary Zukav receive?\n",
- "Answer: U.S. National Book Award\n",
- "\n",
- "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
- "Answer: Foxcatcher\n",
- "\n",
- "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
- "Answer: Kevin Greutert\n",
+ "Context:\n",
+ "[1] «Battle of Kursk | The Battle of Kursk was a Second World War engagement between German and Soviet forces on the Eastern Front near Kursk (450 km south-west of Moscow) in the Soviet Union during July and August 1943. The battle began with the launch of the German offensive, Operation Citadel (German: \"Unternehmen Zitadelle\" ), on 5 July, which had the objective of pinching off the Kursk salient with attacks on the base of the salient from north and south simultaneously. After the German offensive stalled on the northern side of the salient, on 12 July the Soviets commenced their Kursk Strategic Offensive Operation with the launch of Operation Kutuzov (Russian: Кутузов ) against the rear of the German forces in the northern side. On the southern side, the Soviets also launched powerful counterattacks the same day, one of which led to a large armoured clash, the Battle of Prokhorovka. On 3 August, the Soviets began the second phase of the Kursk Strategic Offensive Operation with the launch of Operation Polkovodets Rumyantsev (Russian: Полководец Румянцев ) against the German forces in the southern side of the Kursk salient.»\n",
+ "[2] «Operation Mars | Operation Mars, also known as the Second Rzhev-Sychevka Offensive Operation (Russian: Вторая Ржевско-Сычёвская наступательная операция), was the codename for an offensive launched by Soviet forces against German forces during World War II. It took place between 25 November and 20 December 1942 around the Rzhev salient in the vicinity of Moscow.»\n",
+ "[3] «Kholm Pocket | The Kholm Pocket (German: \"Kessel von Cholm\" ; Russian: Холмский котёл ) was the name given for the encirclement of German troops by the Red Army around Kholm south of Leningrad, during World War II on the Eastern Front, from 23 January 1942 until 5 May 1942. A much larger pocket was simultaneously surrounded in Demyansk, about 100 km to the northeast. These were the results of German retreat following their defeat during the Battle of Moscow.»\n",
"\n",
- "Question: Who produced the album that included a re-recording of \"Lithium\"?\n",
- "Answer: Butch Vig\n",
+ "Question: What is the code name for the German offensive that started this Second World War engagement on the Eastern Front (a few hundred kilometers from Moscow) between Soviet and German forces, which included 102nd Infantry Division?\n",
"\n",
- "Question: What city was the victim of Joseph Druces working in?\n",
- "Answer: Boston, Massachusetts\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the German offensive that started the Battle of Kursk was called Operation Citadel.\n",
"\n",
- "Question: In what year was the star of To Hell and Back born?\n",
- "Answer: 1925\n",
+ "Answer: Operation Citadel\n",
"\n",
"---\n",
"\n",
- "Follow the following format.\n",
- "\n",
"Context:\n",
- "${sources that may contain relevant content}\n",
+ "[1] «Kerry Condon | Kerry Condon (born 4 January 1983) is an Irish television and film actress, best known for her role as Octavia of the Julii in the HBO/BBC series \"Rome,\" as Stacey Ehrmantraut in AMC's \"Better Call Saul\" and as the voice of F.R.I.D.A.Y. in various films in the Marvel Cinematic Universe. She is also the youngest actress ever to play Ophelia in a Royal Shakespeare Company production of \"Hamlet.\"»\n",
+ "[2] «Corona Riccardo | Corona Riccardo (c. 1878October 15, 1917) was an Italian born American actress who had a brief Broadway stage career before leaving to become a wife and mother. Born in Naples she came to acting in 1894 playing a Mexican girl in a play at the Empire Theatre. Wilson Barrett engaged her for a role in his play \"The Sign of the Cross\" which he took on tour of the United States. Riccardo played the role of Ancaria and later played Berenice in the same play. Robert B. Mantell in 1898 who struck by her beauty also cast her in two Shakespeare plays, \"Romeo and Juliet\" and \"Othello\". Author Lewis Strang writing in 1899 said Riccardo was the most promising actress in America at the time. Towards the end of 1898 Mantell chose her for another Shakespeare part, Ophelia im Hamlet. Afterwards she was due to join Augustin Daly's Theatre Company but Daly died in 1899. In 1899 she gained her biggest fame by playing Iras in the first stage production of Ben-Hur.»\n",
+ "[3] «Judi Dench | Dame Judith Olivia \"Judi\" Dench, {'1': \", '2': \", '3': \", '4': \"} (born 9 December 1934) is an English actress and author. Dench made her professional debut in 1957 with the Old Vic Company. Over the following few years, she performed in several of Shakespeare's plays in such roles as Ophelia in \"Hamlet\", Juliet in \"Romeo and Juliet\", and Lady Macbeth in \"Macbeth\". Although most of her work during this period was in theatre, she also branched into film work and won a BAFTA Award as Most Promising Newcomer. She drew strong reviews for her leading role in the musical \"Cabaret\" in 1968.»\n",
"\n",
- "Question: ${the question to be answered}\n",
+ "Question: Who acted in the shot film The Shore and is also the youngest actress ever to play Ophelia in a Royal Shakespeare Company production of \"Hamlet.\" ?\n",
"\n",
- "Rationale: Let's think step by step. ${a step-by-step deduction that identifies the correct response, which will be provided below}\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the actress we are looking for is the youngest actress ever to play Ophelia in a Royal Shakespeare Company production of \"Hamlet.\" We also know that she acted in the short film The Shore.\n",
"\n",
- "Answer: ${a short factoid answer, often between 1 and 5 words}\n",
+ "Answer: Kerry Condon\n",
"\n",
"---\n",
"\n",
"Context:\n",
- "[1] «Right Back at It Again | at the Kerrang! Awards. Personnel per digital booklet. Right Back at It Again \"Right Back at It Again\" is the second track and the first single from A Day to Remember's fifth album, \"Common Courtesy\" (2013). In October 20, 2015, the song was featured in Activision rhythm-music game, \"\". Vocalist, Jeremy McKinnon wrote the lyrics, while the music was written by McKinnon, former guitarist Tom Denney, guitarist Neil Westfall and producer Andrew Wade. \"Right Back at It Again\" almost wasn't included on the album as it was one of the excess songs the band had recorded, \"we realised that it»\n",
- "[2] «Right Back at It Again | Right Back at It Again \"Right Back at It Again\" is the second track and the first single from A Day to Remember's fifth album, \"Common Courtesy\" (2013). In October 20, 2015, the song was featured in Activision rhythm-music game, \"\". Vocalist, Jeremy McKinnon wrote the lyrics, while the music was written by McKinnon, former guitarist Tom Denney, guitarist Neil Westfall and producer Andrew Wade. \"Right Back at It Again\" almost wasn't included on the album as it was one of the excess songs the band had recorded, \"we realised that it sounded great, so on it went.\" \"Right Back»\n",
- "[3] «Jeremy McKinnon | Jeremy McKinnon Jeremy Wade McKinnon (born December 17, 1985) is an American musician and record producer, best known as a founding member and the lead vocalist of A Day to Remember. He has produced full-length albums for The Devil Wears Prada, The Ghost Inside, Neck Deep, and Wage War. Jeremy McKinnon was born in Gainesville, Florida and grew up with two sisters. His parents are originally from Brooklyn, New York and is of Irish and Italian descent. McKinnon first got a job at Boston Market and later went on to do construction work. McKinnon's interest in music came as a»\n",
- "[4] «Martin McKinnon | Martin McKinnon Martin \"Marty\" McKinnon (born 5 July 1975 in Adelaide) is a former Australian rules footballer who played with Adelaide, Geelong and the Brisbane Lions in the Australian Football League (AFL). McKinnon was recruited by Adelaide in the 1992 AFL Draft, with their first ever national draft pick. He was the youngest player on Adelaide's list at the time and played for Central District in the SANFL when not appearing with Adelaide. He made his AFL debut late in the 1994 AFL season and was a regular in the team the following year, playing mostly as a wingman. He»\n",
+ "[1] «David Gregory (physician) | David Gregory (20 December 1625 – 1720) was a Scottish physician and inventor. His surname is sometimes spelt as Gregorie, the original Scottish spelling. He inherited Kinnairdy Castle in 1664. Three of his twenty-nine children became mathematics professors. He is credited with inventing a military cannon that Isaac Newton described as \"being destructive to the human species\". Copies and details of the model no longer exist. Gregory's use of a barometer to predict farming-related weather conditions led him to be accused of witchcraft by Presbyterian ministers from Aberdeen, although he was never convicted.»\n",
+ "[2] «Gregory Tarchaneiotes | Gregory Tarchaneiotes (Greek: Γρηγόριος Ταρχανειώτης , Italian: \"Gregorio Tracanioto\" or \"Tracamoto\" ) was a \"protospatharius\" and the long-reigning catepan of Italy from 998 to 1006. In December 999, and again on February 2, 1002, he reinstituted and confirmed the possessions of the abbey and monks of Monte Cassino in Ascoli. In 1004, he fortified and expanded the castle of Dragonara on the Fortore. He gave it three circular towers and one square one. He also strengthened Lucera.»\n",
+ "[3] «David Gregory (mathematician) | David Gregory (originally spelt Gregorie) FRS (? 1659 – 10 October 1708) was a Scottish mathematician and astronomer. He was professor of mathematics at the University of Edinburgh, Savilian Professor of Astronomy at the University of Oxford, and a commentator on Isaac Newton's \"Principia\".»\n",
"\n",
- "Question: Right Back At It Again contains lyrics co-written by the singer born in what city?\n",
+ "Question: What castle did David Gregory inherit?\n",
"\n",
- "Rationale: Let's think step by step.\u001b[32m The song \"Right Back at It Again\" is co-written by the singer Jeremy McKinnon. A quick search of \"Jeremy McKinnon\" reveals that he was born in Gainesville, Florida.\n",
+ "Reasoning: Let's think step by step in order to\u001b[32m produce the answer. We know that David Gregory inherited a castle. The name of the castle is Kinnairdy Castle.\n",
"\n",
- "Answer: Gainesville, Florida\u001b[0m\n",
+ "Answer: Kinnairdy Castle\u001b[0m\n",
"\n",
"\n",
"\n"
]
+ }
+ ],
+ "source": [
+ "turbo.inspect_history(n=1)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Even though we haven't written any of this detailed demonstrations, we see that **DSPy** was able to bootstrap this 3,000 token prompt for **3-shot retrieval augmented generation with hard negative passages and chain of thought** from our extremely simple program.\n",
+ "\n",
+ "This illustrates the power of composition and learning. Of course, this was just generated by a particular teleprompter, which may or may not be perfect in each setting. As you'll see in **DSPy**, there is a large but systematic space of options you have to optimize and validate the quality and cost of your programs.\n",
+ "\n",
+ "If you're so inclined, you can easily inspect the learned objects themselves."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 19,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "generate_answer\n",
+ "Example({'augmented': True, 'context': ['Tae Kwon Do Times | Tae Kwon Do Times is a magazine devoted to the martial art of taekwondo, and is published in the United States of America. While the title suggests that it focuses on taekwondo exclusively, the magazine also covers other Korean martial arts. \"Tae Kwon Do Times\" has published articles by a wide range of authors, including He-Young Kimm, Thomas Kurz, Scott Shaw, and Mark Van Schuyver.', \"Kwon Tae-man | Kwon Tae-man (born 1941) was an early Korean hapkido practitioner and a pioneer of the art, first in Korea and then in the United States. He formed one of the earliest dojang's for hapkido in the United States in Torrance, California, and has been featured in many magazine articles promoting the art.\", 'Hee Il Cho | Cho Hee Il (born October 13, 1940) is a prominent Korean-American master of taekwondo, holding the rank of 9th \"dan\" in the martial art. He has written 11 martial art books, produced 70 martial art training videos, and has appeared on more than 70 martial arts magazine covers. Cho won several national and international competitions as a taekwondo competitor, and has appeared in several films, including \"Fight to Win\", \"Best of the Best\", \"Bloodsport II\", and \"Bloodsport III\". He founded the Action International Martial Arts Association (AIMAA) in 1980, and is its President. Cho is a member of both \"Black Belt\" magazine\\'s Hall of Fame and \"Tae Kwon Do Times\" magazine\\'s Hall of Fame.'], 'question': 'Which magazine has published articles by Scott Shaw, Tae Kwon Do Times or Southwest Art?', 'rationale': 'produce the answer. We know from the context that \"Tae Kwon Do Times\" is a magazine that covers taekwondo and other Korean martial arts. It has published articles by authors like Scott Shaw. On the other hand, there is no information about Southwest Art magazine in the context.', 'answer': 'Tae Kwon Do Times'}) (input_keys=None)\n",
+ "\n"
+ ]
+ }
+ ],
+ "source": [
+ "for name, parameter in compiled_rag.named_predictors():\n",
+ " print(name)\n",
+ " print(parameter.demos[0])\n",
+ " print()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "##### Evaluating the Answers\n",
+ "\n",
+ "We can now evaluate our `compiled_rag` program on the dev set. Of course, this tiny set is _not_ meant to be a reliable benchmark, but it'll be instructive to use it for illustration.\n",
+ "\n",
+ "For a start, let's evaluate the accuracy (exact match) of the predicted answer."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 20,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ " 0%| | 0/50 [00:00<?, ?it/s]"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 22 / 50 (44.0): 100%|██████████| 50/50 [00:16<00:00, 3.11it/s]"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 22 / 50 (44.0%)\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "\n"
+ ]
},
{
"data": {
+ "text/html": [
+ "<style type=\"text/css\">\n",
+ "#T_1be94 th {\n",
+ " text-align: left;\n",
+ "}\n",
+ "#T_1be94 td {\n",
+ " text-align: left;\n",
+ "}\n",
+ "#T_1be94_row0_col0, #T_1be94_row0_col1, #T_1be94_row0_col2, #T_1be94_row0_col3, #T_1be94_row0_col4, #T_1be94_row0_col5, #T_1be94_row1_col0, #T_1be94_row1_col1, #T_1be94_row1_col2, #T_1be94_row1_col3, #T_1be94_row1_col4, #T_1be94_row1_col5, #T_1be94_row2_col0, #T_1be94_row2_col1, #T_1be94_row2_col2, #T_1be94_row2_col3, #T_1be94_row2_col4, #T_1be94_row2_col5, #T_1be94_row3_col0, #T_1be94_row3_col1, #T_1be94_row3_col2, #T_1be94_row3_col3, #T_1be94_row3_col4, #T_1be94_row3_col5, #T_1be94_row4_col0, #T_1be94_row4_col1, #T_1be94_row4_col2, #T_1be94_row4_col3, #T_1be94_row4_col4, #T_1be94_row4_col5 {\n",
+ " text-align: left;\n",
+ " white-space: pre-wrap;\n",
+ " word-wrap: break-word;\n",
+ " max-width: 400px;\n",
+ "}\n",
+ "</style>\n",
+ "<table id=\"T_1be94\">\n",
+ " <thead>\n",
+ " <tr>\n",
+ " <th class=\"blank level0\" >&nbsp;</th>\n",
+ " <th id=\"T_1be94_level0_col0\" class=\"col_heading level0 col0\" >question</th>\n",
+ " <th id=\"T_1be94_level0_col1\" class=\"col_heading level0 col1\" >example_answer</th>\n",
+ " <th id=\"T_1be94_level0_col2\" class=\"col_heading level0 col2\" >gold_titles</th>\n",
+ " <th id=\"T_1be94_level0_col3\" class=\"col_heading level0 col3\" >context</th>\n",
+ " <th id=\"T_1be94_level0_col4\" class=\"col_heading level0 col4\" >pred_answer</th>\n",
+ " <th id=\"T_1be94_level0_col5\" class=\"col_heading level0 col5\" >answer_exact_match</th>\n",
+ " </tr>\n",
+ " </thead>\n",
+ " <tbody>\n",
+ " <tr>\n",
+ " <th id=\"T_1be94_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
+ " <td id=\"T_1be94_row0_col0\" class=\"data row0 col0\" >Are both Cangzhou and Qionghai in the Hebei province of China?</td>\n",
+ " <td id=\"T_1be94_row0_col1\" class=\"data row0 col1\" >no</td>\n",
+ " <td id=\"T_1be94_row0_col2\" class=\"data row0 col2\" >{'Qionghai', 'Cangzhou'}</td>\n",
+ " <td id=\"T_1be94_row0_col3\" class=\"data row0 col3\" >['Cangzhou | Cangzhou () is a prefecture-level city in eastern Hebei province, People\\'s Republic of China. At the 2010 census, Cangzhou\\'s built-up (\"or metro\") area...</td>\n",
+ " <td id=\"T_1be94_row0_col4\" class=\"data row0 col4\" >No</td>\n",
+ " <td id=\"T_1be94_row0_col5\" class=\"data row0 col5\" >✔️ [True]</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_1be94_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
+ " <td id=\"T_1be94_row1_col0\" class=\"data row1 col0\" >Who conducts the draft in which Marc-Andre Fleury was drafted to the Vegas Golden Knights for the 2017-18 season?</td>\n",
+ " <td id=\"T_1be94_row1_col1\" class=\"data row1 col1\" >National Hockey League</td>\n",
+ " <td id=\"T_1be94_row1_col2\" class=\"data row1 col2\" >{'2017 NHL Expansion Draft', '2017–18 Pittsburgh Penguins season'}</td>\n",
+ " <td id=\"T_1be94_row1_col3\" class=\"data row1 col3\" >['2017–18 Pittsburgh Penguins season | The 2017–18 Pittsburgh Penguins season will be the 51st season for the National Hockey League ice hockey team that was...</td>\n",
+ " <td id=\"T_1be94_row1_col4\" class=\"data row1 col4\" >National Hockey League</td>\n",
+ " <td id=\"T_1be94_row1_col5\" class=\"data row1 col5\" >✔️ [True]</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_1be94_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
+ " <td id=\"T_1be94_row2_col0\" class=\"data row2 col0\" >The Wings entered a new era, following the retirement of which Canadian retired professional ice hockey player and current general manager of the Tampa Bay...</td>\n",
+ " <td id=\"T_1be94_row2_col1\" class=\"data row2 col1\" >Steve Yzerman</td>\n",
+ " <td id=\"T_1be94_row2_col2\" class=\"data row2 col2\" >{'Steve Yzerman', '2006–07 Detroit Red Wings season'}</td>\n",
+ " <td id=\"T_1be94_row2_col3\" class=\"data row2 col3\" >['Steve Yzerman | Stephen Gregory \"Steve\" Yzerman ( ; born May 9, 1965) is a Canadian retired professional ice hockey player and current general manager...</td>\n",
+ " <td id=\"T_1be94_row2_col4\" class=\"data row2 col4\" >Steve Yzerman</td>\n",
+ " <td id=\"T_1be94_row2_col5\" class=\"data row2 col5\" >✔️ [True]</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_1be94_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
+ " <td id=\"T_1be94_row3_col0\" class=\"data row3 col0\" >What river is near the Crichton Collegiate Church?</td>\n",
+ " <td id=\"T_1be94_row3_col1\" class=\"data row3 col1\" >the River Tyne</td>\n",
+ " <td id=\"T_1be94_row3_col2\" class=\"data row3 col2\" >{'Crichton Collegiate Church', 'Crichton Castle'}</td>\n",
+ " <td id=\"T_1be94_row3_col3\" class=\"data row3 col3\" >[\"Crichton Collegiate Church | Crichton Collegiate Church is situated about 0.6 mi south west of the hamlet of Crichton in Midlothian, Scotland. Crichton itself is...</td>\n",
+ " <td id=\"T_1be94_row3_col4\" class=\"data row3 col4\" >River Tyne</td>\n",
+ " <td id=\"T_1be94_row3_col5\" class=\"data row3 col5\" >✔️ [True]</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_1be94_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
+ " <td id=\"T_1be94_row4_col0\" class=\"data row4 col0\" >In the 10th Century A.D. Ealhswith had a son called Æthelweard by which English king?</td>\n",
+ " <td id=\"T_1be94_row4_col1\" class=\"data row4 col1\" >King Alfred the Great</td>\n",
+ " <td id=\"T_1be94_row4_col2\" class=\"data row4 col2\" >{'Æthelweard (son of Alfred)', 'Ealhswith'}</td>\n",
+ " <td id=\"T_1be94_row4_col3\" class=\"data row4 col3\" >[\"Æthelweard of East Anglia | Æthelweard (died 854) was a 9th-century king of East Anglia, the long-lived Anglo-Saxon kingdom which today includes the English counties...</td>\n",
+ " <td id=\"T_1be94_row4_col4\" class=\"data row4 col4\" >King Alfred the Great</td>\n",
+ " <td id=\"T_1be94_row4_col5\" class=\"data row4 col5\" >✔️ [True]</td>\n",
+ " </tr>\n",
+ " </tbody>\n",
+ "</table>\n"
+ ],
"text/plain": [
- "('Gainesville, Florida', None)"
+ "<pandas.io.formats.style.Styler at 0x7f55c2c08d00>"
]
},
- "execution_count": 21,
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "\n",
+ " <div style='\n",
+ " text-align: center; \n",
+ " font-size: 16px; \n",
+ " font-weight: bold; \n",
+ " color: #555; \n",
+ " margin: 10px 0;'>\n",
+ " ... 45 more rows not displayed ...\n",
+ " </div>\n",
+ " "
+ ],
+ "text/plain": [
+ "<IPython.core.display.HTML object>"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/plain": [
+ "44.0"
+ ]
+ },
+ "execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
- "multihop_QA_v1(dev[2].question), lm.inspect_history(n=3)"
+ "from dspy.evaluate.evaluate import Evaluate\n",
+ "\n",
+ "# Set up the `evaluate_on_hotpotqa` function. We'll use this many times below.\n",
+ "evaluate_on_hotpotqa = Evaluate(devset=devset, num_threads=1, display_progress=True, display_table=5)\n",
+ "\n",
+ "# Evaluate the `compiled_rag` program with the `answer_exact_match` metric.\n",
+ "metric = dspy.evaluate.answer_exact_match\n",
+ "evaluate_on_hotpotqa(compiled_rag, metric=metric)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "Now, let's run this on the remaining dev examples."
+ "##### Evaluating the Retrieval\n",
+ "\n",
+ "It may also be instructive to look at the accuracy of retrieval. There are multiple ways to do this. Often, we can just check whether the rertieved passages contain the answer.\n",
+ "\n",
+ "That said, since our dev set includes the gold titles that should be retrieved, we can just use these here."
]
},
{
"cell_type": "code",
- "execution_count": 22,
+ "execution_count": 21,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
- "100%|██████████| 13/13 [00:00<00:00, 167.43it/s]"
+ "Average Metric: 13 / 50 (26.0): 100%|██████████| 50/50 [00:00<00:00, 444.20it/s]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
- "Answered 7 / 13 (53.8%) correctly.\n"
+ "Average Metric: 13 / 50 (26.0%)\n"
]
},
{
@@ -1316,107 +1053,83 @@
{
"data": {
"text/html": [
- "<style type=\"text/css\" >\n",
- " #T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2 th {\n",
- " text-align: left;\n",
- " } #T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2 td {\n",
- " text-align: left;\n",
- " }</style><table id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2\" ><thead> <tr> <th class=\"blank level0\" ></th> <th class=\"col_heading level0 col0\" >question</th> <th class=\"col_heading level0 col1\" >answer</th> <th class=\"col_heading level0 col2\" >prediction</th> <th class=\"col_heading level0 col3\" >correct</th> </tr></thead><tbody>\n",
- " <tr>\n",
- " <th id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row0_col2\" class=\"data row0 col2\" >E. L. Doctorow</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row0_col3\" class=\"data row0 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row1_col2\" class=\"data row1 col2\" >The Long Island Serial Killer: A Mother's Hunt for Justice</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row1_col3\" class=\"data row1 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row2_col2\" class=\"data row2 col2\" >Gainesville, Florida</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row2_col3\" class=\"data row2 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row3_col2\" class=\"data row3 col2\" >1947</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row3_col3\" class=\"data row3 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row4_col2\" class=\"data row4 col2\" >John Braine</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row4_col3\" class=\"data row4 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row5_col3\" class=\"data row5 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row6_col2\" class=\"data row6 col2\" >UK</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row6_col3\" class=\"data row6 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row7_col2\" class=\"data row7 col2\" >September 28, 1969</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row7_col3\" class=\"data row7 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row8_col2\" class=\"data row8 col2\" >Dundalk</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row8_col3\" class=\"data row8 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row9_col2\" class=\"data row9 col2\" >space technology</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row9_col3\" class=\"data row9 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row10_col2\" class=\"data row10 col2\" >Guildford, Surrey</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row10_col3\" class=\"data row10 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row11_col2\" class=\"data row11 col2\" >Portrait of a Marriage</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row11_col3\" class=\"data row11 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row12_col2\" class=\"data row12 col2\" >1442</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row12_col3\" class=\"data row12 col3\" >✔️</td>\n",
- " </tr>\n",
- " </tbody></table>"
+ "<style type=\"text/css\">\n",
+ "#T_eb418 th {\n",
+ " text-align: left;\n",
+ "}\n",
+ "#T_eb418 td {\n",
+ " text-align: left;\n",
+ "}\n",
+ "#T_eb418_row0_col0, #T_eb418_row0_col1, #T_eb418_row0_col2, #T_eb418_row0_col3, #T_eb418_row0_col4, #T_eb418_row0_col5, #T_eb418_row1_col0, #T_eb418_row1_col1, #T_eb418_row1_col2, #T_eb418_row1_col3, #T_eb418_row1_col4, #T_eb418_row1_col5, #T_eb418_row2_col0, #T_eb418_row2_col1, #T_eb418_row2_col2, #T_eb418_row2_col3, #T_eb418_row2_col4, #T_eb418_row2_col5, #T_eb418_row3_col0, #T_eb418_row3_col1, #T_eb418_row3_col2, #T_eb418_row3_col3, #T_eb418_row3_col4, #T_eb418_row3_col5, #T_eb418_row4_col0, #T_eb418_row4_col1, #T_eb418_row4_col2, #T_eb418_row4_col3, #T_eb418_row4_col4, #T_eb418_row4_col5 {\n",
+ " text-align: left;\n",
+ " white-space: pre-wrap;\n",
+ " word-wrap: break-word;\n",
+ " max-width: 400px;\n",
+ "}\n",
+ "</style>\n",
+ "<table id=\"T_eb418\">\n",
+ " <thead>\n",
+ " <tr>\n",
+ " <th class=\"blank level0\" >&nbsp;</th>\n",
+ " <th id=\"T_eb418_level0_col0\" class=\"col_heading level0 col0\" >question</th>\n",
+ " <th id=\"T_eb418_level0_col1\" class=\"col_heading level0 col1\" >example_answer</th>\n",
+ " <th id=\"T_eb418_level0_col2\" class=\"col_heading level0 col2\" >gold_titles</th>\n",
+ " <th id=\"T_eb418_level0_col3\" class=\"col_heading level0 col3\" >context</th>\n",
+ " <th id=\"T_eb418_level0_col4\" class=\"col_heading level0 col4\" >pred_answer</th>\n",
+ " <th id=\"T_eb418_level0_col5\" class=\"col_heading level0 col5\" >gold_passages_retrieved</th>\n",
+ " </tr>\n",
+ " </thead>\n",
+ " <tbody>\n",
+ " <tr>\n",
+ " <th id=\"T_eb418_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
+ " <td id=\"T_eb418_row0_col0\" class=\"data row0 col0\" >Are both Cangzhou and Qionghai in the Hebei province of China?</td>\n",
+ " <td id=\"T_eb418_row0_col1\" class=\"data row0 col1\" >no</td>\n",
+ " <td id=\"T_eb418_row0_col2\" class=\"data row0 col2\" >{'Qionghai', 'Cangzhou'}</td>\n",
+ " <td id=\"T_eb418_row0_col3\" class=\"data row0 col3\" >['Cangzhou | Cangzhou () is a prefecture-level city in eastern Hebei province, People\\'s Republic of China. At the 2010 census, Cangzhou\\'s built-up (\"or metro\") area...</td>\n",
+ " <td id=\"T_eb418_row0_col4\" class=\"data row0 col4\" >No</td>\n",
+ " <td id=\"T_eb418_row0_col5\" class=\"data row0 col5\" >❌ [False]</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_eb418_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
+ " <td id=\"T_eb418_row1_col0\" class=\"data row1 col0\" >Who conducts the draft in which Marc-Andre Fleury was drafted to the Vegas Golden Knights for the 2017-18 season?</td>\n",
+ " <td id=\"T_eb418_row1_col1\" class=\"data row1 col1\" >National Hockey League</td>\n",
+ " <td id=\"T_eb418_row1_col2\" class=\"data row1 col2\" >{'2017 NHL Expansion Draft', '2017–18 Pittsburgh Penguins season'}</td>\n",
+ " <td id=\"T_eb418_row1_col3\" class=\"data row1 col3\" >['2017–18 Pittsburgh Penguins season | The 2017–18 Pittsburgh Penguins season will be the 51st season for the National Hockey League ice hockey team that was...</td>\n",
+ " <td id=\"T_eb418_row1_col4\" class=\"data row1 col4\" >National Hockey League</td>\n",
+ " <td id=\"T_eb418_row1_col5\" class=\"data row1 col5\" >✔️ [True]</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_eb418_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
+ " <td id=\"T_eb418_row2_col0\" class=\"data row2 col0\" >The Wings entered a new era, following the retirement of which Canadian retired professional ice hockey player and current general manager of the Tampa Bay...</td>\n",
+ " <td id=\"T_eb418_row2_col1\" class=\"data row2 col1\" >Steve Yzerman</td>\n",
+ " <td id=\"T_eb418_row2_col2\" class=\"data row2 col2\" >{'Steve Yzerman', '2006–07 Detroit Red Wings season'}</td>\n",
+ " <td id=\"T_eb418_row2_col3\" class=\"data row2 col3\" >['Steve Yzerman | Stephen Gregory \"Steve\" Yzerman ( ; born May 9, 1965) is a Canadian retired professional ice hockey player and current general manager...</td>\n",
+ " <td id=\"T_eb418_row2_col4\" class=\"data row2 col4\" >Steve Yzerman</td>\n",
+ " <td id=\"T_eb418_row2_col5\" class=\"data row2 col5\" >✔️ [True]</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_eb418_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
+ " <td id=\"T_eb418_row3_col0\" class=\"data row3 col0\" >What river is near the Crichton Collegiate Church?</td>\n",
+ " <td id=\"T_eb418_row3_col1\" class=\"data row3 col1\" >the River Tyne</td>\n",
+ " <td id=\"T_eb418_row3_col2\" class=\"data row3 col2\" >{'Crichton Collegiate Church', 'Crichton Castle'}</td>\n",
+ " <td id=\"T_eb418_row3_col3\" class=\"data row3 col3\" >[\"Crichton Collegiate Church | Crichton Collegiate Church is situated about 0.6 mi south west of the hamlet of Crichton in Midlothian, Scotland. Crichton itself is...</td>\n",
+ " <td id=\"T_eb418_row3_col4\" class=\"data row3 col4\" >River Tyne</td>\n",
+ " <td id=\"T_eb418_row3_col5\" class=\"data row3 col5\" >✔️ [True]</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_eb418_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
+ " <td id=\"T_eb418_row4_col0\" class=\"data row4 col0\" >In the 10th Century A.D. Ealhswith had a son called Æthelweard by which English king?</td>\n",
+ " <td id=\"T_eb418_row4_col1\" class=\"data row4 col1\" >King Alfred the Great</td>\n",
+ " <td id=\"T_eb418_row4_col2\" class=\"data row4 col2\" >{'Æthelweard (son of Alfred)', 'Ealhswith'}</td>\n",
+ " <td id=\"T_eb418_row4_col3\" class=\"data row4 col3\" >[\"Æthelweard of East Anglia | Æthelweard (died 854) was a 9th-century king of East Anglia, the long-lived Anglo-Saxon kingdom which today includes the English counties...</td>\n",
+ " <td id=\"T_eb418_row4_col4\" class=\"data row4 col4\" >King Alfred the Great</td>\n",
+ " <td id=\"T_eb418_row4_col5\" class=\"data row4 col5\" >❌ [False]</td>\n",
+ " </tr>\n",
+ " </tbody>\n",
+ "</table>\n"
],
"text/plain": [
- "<pandas.io.formats.style.Styler at 0x7f9bb041dc10>"
+ "<pandas.io.formats.style.Styler at 0x7f559c5b9f70>"
]
},
"metadata": {},
@@ -1424,39 +1137,80 @@
},
{
"data": {
+ "text/html": [
+ "\n",
+ " <div style='\n",
+ " text-align: center; \n",
+ " font-size: 16px; \n",
+ " font-weight: bold; \n",
+ " color: #555; \n",
+ " margin: 10px 0;'>\n",
+ " ... 45 more rows not displayed ...\n",
+ " </div>\n",
+ " "
+ ],
"text/plain": [
- "53.8"
+ "<IPython.core.display.HTML object>"
]
},
- "execution_count": 22,
"metadata": {},
- "output_type": "execute_result"
+ "output_type": "display_data"
}
],
"source": [
- "evaluate(multihop_QA_v1, dev)"
+ "def gold_passages_retrieved(example, pred, trace=None):\n",
+ " gold_titles = set(map(dspy.evaluate.normalize_text, example['gold_titles']))\n",
+ " found_titles = set(map(dspy.evaluate.normalize_text, [c.split(' | ')[0] for c in pred.context]))\n",
+ "\n",
+ " return gold_titles.issubset(found_titles)\n",
+ "\n",
+ "compiled_rag_retrieval_score = evaluate_on_hotpotqa(compiled_rag, metric=gold_passages_retrieved)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Although this simple `compiled_rag` program is able to answer a decent fraction of the questions correctly (on this tiny set, over 40%), the quality of retrieval is much lower.\n",
+ "\n",
+ "This potentially suggests that the LM is often relying on the knowledge it memorized during training to answer questions. To address this weak retrieval, let's explore a second program that involves more advanced search behavior."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "### Program 5: Multi-Hop Condensed Retrieval w/ Automatic Demos and Query Fusion\n",
+ "### 5] Program 2: Multi-Hop Search (“Baleen”)\n",
"\n",
- "Through Program 4, we've begun to explore some of the power of the DSP abstraction. However, if you look closely, you will see a few downsides of the previous approach:\n",
+ "From exploring the harder questions in the training/dev sets, it becomes clear that a single search query is often not enough for this task. For instance, this can be seen when a question ask about, say, the birth city of the writer of \"Right Back At It Again\". A search query identifies the author correctly as \"Jeremy McKinnon\", but it wouldn't figure out when he was born.\n",
"\n",
- "1. The search transformations invoke the LM without any demonstrations in the prompt. That is because we only have training data for the question–answer pairs and not for the intermediate labels (e.g., search queries).\n",
- "2. The QA prompt uses passages (`context`) and a Chain-of-Thought (`rationale`) for the question to be answered. However, the training demonstrations include neither context nor CoT because they aren't available in our labels.\n",
- "3. The search transformations commit to a single query per hop, which may single out an unproductive chain of passages and hence fail to uncover relevant information."
+ "The standard approach for this challenge in the retrieval-augmented NLP literature is to build multi-hop search systems, like GoldEn (Qi et al., 2019) and Baleen (Khattab et al., 2021). These systems read the retrieved results and then generate additional queries to gather additional information if necessary. Using **DSPy**, we can easily simulate such systems in a few lines of code.\n",
+ "\n",
+ "\n",
+ "We'll still use the `GenerateAnswer` signature from the RAG implementation above. All we need now is a **signature** for the \"hop\" behavior: taking some partial context and a question, generate a search query to find missing information."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 22,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "class GenerateSearchQuery(dspy.Signature):\n",
+ " \"\"\"Write a simple search query that will help answer a complex question.\"\"\"\n",
+ "\n",
+ " context = dspy.InputField(desc=\"may contain relevant facts\")\n",
+ " question = dspy.InputField()\n",
+ " query = dspy.OutputField()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "We address these problems automatically in the program below.\n",
+ "Note: We could have written `context = GenerateAnswer.signature.context` to avoid duplicating the description of the `context` field.\n",
"\n",
- "In it, we begin by defining `multihop_demonstrate` (which uses `multihop_attempt`) to automatically **annotate** demonstrations for the complex multi-hop pipeline. These demonstrations will be provided to the LM when it's invoked for each transformation."
+ "Now, let's define the program itself `SimplifiedBaleen`. There are many possible ways to implement this, but we'll keep this version down to the key elements for simplicity."
]
},
{
@@ -1465,76 +1219,103 @@
"metadata": {},
"outputs": [],
"source": [
- "@dsp.transformation\n",
- "def multihop_attempt(d: dsp.Example) -> dsp.Example:\n",
- " # Prepare unaugmented demonstrations for the example.\n",
- " x = dsp.Example(question=d.question, demos=dsp.all_but(train, d))\n",
- " \n",
- " # Search. And skip examples where search fails.\n",
- " # Annotate demonstrations for multihop_search_v2 with the simpler multihop_search_v1 pipeline.\n",
- " x = multihop_search_v1(x)\n",
- " if not dsp.passage_match(x.context, d.answer): return None\n",
- " \n",
- " # Predict. And skip examples where predict fails.\n",
- " x = QA_predict(x, sc=False)\n",
- " if not dsp.answer_match(x.answer, d.answer): return None\n",
+ "from dsp.utils import deduplicate\n",
+ "\n",
+ "class SimplifiedBaleen(dspy.Module):\n",
+ " def __init__(self, passages_per_hop=3, max_hops=2):\n",
+ " super().__init__()\n",
+ "\n",
+ " self.generate_query = [dspy.ChainOfThought(GenerateSearchQuery) for _ in range(max_hops)]\n",
+ " self.retrieve = dspy.Retrieve(k=passages_per_hop)\n",
+ " self.generate_answer = dspy.ChainOfThought(GenerateAnswer)\n",
+ " self.max_hops = max_hops\n",
" \n",
- " return d.copy(**x)\n",
+ " def forward(self, question):\n",
+ " context = []\n",
+ " \n",
+ " for hop in range(self.max_hops):\n",
+ " query = self.generate_query[hop](context=context, question=question).query\n",
+ " passages = self.retrieve(query).passages\n",
+ " context = deduplicate(context + passages)\n",
"\n",
- "@dsp.transformation\n",
- "def multihop_demonstrate(x: dsp.Example) -> dsp.Example:\n",
- " demos = dsp.sample(train, k=7)\n",
- " x.demos = dsp.annotate(multihop_attempt)(demos, k=3, return_all=True)\n",
- " return x"
+ " pred = self.generate_answer(context=context, question=question)\n",
+ " return dspy.Prediction(context=context, answer=pred.answer)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "We now implement `multihop_search_v2` as part of `multihop_QA_v2`.\n",
+ "As we can see, the `__init__` method defines a few key sub-modules:\n",
"\n",
- "In addition to the changes mentioned earlier, this program simulates the Baleen system (Khattab et al., 2021) in a few lines of code.\n",
+ "- **generate_query**: For each hop, we will have one `dspy.ChainOfThought` predictor with the `GenerateSearchQuery` signature.\n",
+ "- **retrieve**: This module will do the actual search, using the generated queries.\n",
+ "- **generate_answer**: This `dspy.Predict` module will be used after all the search steps. It has a `GenerateAnswer`, to actually produce an answer.\n",
"\n",
- "In each retrieval hop (after the very first hop), the summary of the previous hop(s) is included in the prompt. This allows us to efficiently read a larger number of passages from the current hop."
+ "The `forward` method uses these sub-modules in simple control flow.\n",
+ "\n",
+ "1. First, we'll loop up to `self.max_hops` times.\n",
+ "1. In each iteration, we'll generate a search query using the predictor at `self.generate_query[hop]`.\n",
+ "1. We'll retrieve the top-k passages using that query.\n",
+ "1. We'll add the (deduplicated) passages to our accumulator of `context`.\n",
+ "1. After the loop, we'll use `self.generate_answer` to produce an answer.\n",
+ "1. We'll return a prediction with the retrieved `context` and predicted `answer`."
]
},
{
- "cell_type": "code",
- "execution_count": 24,
+ "cell_type": "markdown",
"metadata": {},
- "outputs": [],
"source": [
- "@dsp.transformation\n",
- "def multihop_search_v2(example: dsp.Example, max_hops=2, k=5) -> dsp.Example:\n",
- " example.context = []\n",
+ "##### Inspect the zero-shot version of the Baleen program\n",
"\n",
- " for hop in range(max_hops):\n",
- " # Generate queries\n",
- " template = rewrite_template if hop == 0 else hop_template\n",
- " example, completions = dsp.generate(template, n=10, temperature=0.7)(example, stage=f'h{hop}')\n",
- " \n",
- " # Collect the queries and search with result fusion\n",
- " queries = [c.query for c in completions] + [example.question]\n",
- " example.context = dsp.retrieveEnsemble(queries, k=k)\n",
+ "We will also compile this program shortly. But, before that, we can try it out in a \"zero-shot\" setting (i.e., without any compilation).\n",
"\n",
- " # Arrange the passages for the next hop\n",
- " if hop > 0:\n",
- " example.context = [completions[0].rationale] + example.context\n",
- " \n",
- " return example\n",
+ "Using a program in zero-shot (uncompiled) setting doesn't mean that quality will be bad. It just means that we're bottlenecked directly by the reliability of the underlying LM to understand our sub-tasks from minimal instructions.\n",
"\n",
- "def multihop_QA_v2(question: str) -> str:\n",
- " x = dsp.Example(question=question)\n",
- " x = multihop_demonstrate(x)\n",
- " x = multihop_search_v2(x)\n",
- " x = QA_predict(x)\n",
- " return x.answer"
+ "This is often just fine when using the most expensive/powerful models (e.g., GPT-4) on the easiest and most standard tasks (e.g., answering simple questions about popular entities).\n",
+ "\n",
+ "However, a zero-shot approach quickly falls short for more specialized tasks, for novel domains/settings, and for more efficient (or open) models. **DSPy** can help you in all of these settings."
]
},
{
"cell_type": "code",
- "execution_count": 25,
+ "execution_count": 24,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Question: How many storeys are in the castle that David Gregory inherited?\n",
+ "Predicted Answer: five\n",
+ "Retrieved Contexts (truncated): ['David Gregory (physician) | David Gregory (20 December 1625 – 1720) was a Scottish physician and inventor. His surname is sometimes spelt as Gregorie, the original Scottish spelling. He inherited Kinn...', 'The Boleyn Inheritance | The Boleyn Inheritance is a novel by British author Philippa Gregory which was first published in 2006. It is a direct sequel to her previous novel \"The Other Boleyn Girl,\" an...', 'Gregory of Gaeta | Gregory was the Duke of Gaeta from 963 until his death. He was the second son of Docibilis II of Gaeta and his wife Orania. He succeeded his brother John II, who had left only daugh...', 'Kinnairdy Castle | Kinnairdy Castle is a tower house, having five storeys and a garret, two miles south of Aberchirder, Aberdeenshire, Scotland. The alternative name is Old Kinnairdy....', 'Kinnaird Head | Kinnaird Head (Scottish Gaelic: \"An Ceann Àrd\" , \"high headland\") is a headland projecting into the North Sea, within the town of Fraserburgh, Aberdeenshire on the east coast of Scotla...', 'Kinnaird Castle, Brechin | Kinnaird Castle is a 15th-century castle in Angus, Scotland. The castle has been home to the Carnegie family, the Earl of Southesk, for more than 600 years....']\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Ask any question you like to this simple RAG program.\n",
+ "my_question = \"How many storeys are in the castle that David Gregory inherited?\"\n",
+ "\n",
+ "# Get the prediction. This contains `pred.context` and `pred.answer`.\n",
+ "uncompiled_baleen = SimplifiedBaleen() # uncompiled (i.e., zero-shot) program\n",
+ "pred = uncompiled_baleen(my_question)\n",
+ "\n",
+ "# Print the contexts and the answer.\n",
+ "print(f\"Question: {my_question}\")\n",
+ "print(f\"Predicted Answer: {pred.answer}\")\n",
+ "print(f\"Retrieved Contexts (truncated): {[c[:200] + '...' for c in pred.context]}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's inspect the last **three** calls to the LM (i.e., generating the first hop's query, generating the second hop's query, and generating the answer)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 25,
"metadata": {},
"outputs": [
{
@@ -1545,111 +1326,62 @@
"\n",
"\n",
"\n",
- "Write a search query that will help answer a complex question.\n",
- "\n",
- "---\n",
- "\n",
- "Follow the following format.\n",
- "\n",
- "Question: ${the question to be answered}\n",
- "Rationale: Let's think step by step. To answer this question, we first need to find out ${the missing information}\n",
- "Search Query: ${a simple question for seeking the missing information}\n",
- "\n",
- "---\n",
- "\n",
- "Question: Which award did the first book of Gary Zukav receive?\n",
- "Rationale: Let's think step by step. To answer this question, we first need to find out the name of the first book of Gary Zukav.\n",
- "Search Query: \"What is the name of Gary Zukav's first book?\"\n",
- "\n",
- "---\n",
- "\n",
- "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
- "Rationale: Let's think step by step. To answer this question, we first need to find out who the heir to the Du Pont family fortune is.\n",
- "Search Query: \"Heir to the Du Pont family fortune\"\n",
- "\n",
- "---\n",
- "\n",
- "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
- "Rationale: Let's think step by step. To answer this question, we first need to find out the name of the 2009 movie featuring Peter Outerbridge as William Easton.\n",
- "Search Query: \"2009 movie featuring Peter Outerbridge as William Easton\"\n",
+ "Write a simple search query that will help answer a complex question.\n",
"\n",
"---\n",
"\n",
- "Question: What year was the party of the winner of the 1971 San Francisco mayoral election founded?\n",
- "Rationale: Let's think step by step. To answer this question, we first need to find out\u001b[32m the name of the party of the winner of the 1971 San Francisco mayoral election.\n",
- "Search Query: \"Party of the winner of the 1971 San Francisco mayoral election\"\u001b[0m\u001b[31m \t (and 9 other completions)\u001b[0m\n",
- "\n",
- "\n",
- "\n",
+ "Follow the following format.\n",
"\n",
+ "Context: may contain relevant facts\n",
"\n",
+ "Question: ${question}\n",
"\n",
+ "Reasoning: Let's think step by step in order to ${produce the query}. We ...\n",
"\n",
- "Write a search query that will help answer a complex question.\n",
+ "Query: ${query}\n",
"\n",
"---\n",
"\n",
- "Follow the following format.\n",
- "\n",
- "Context:\n",
- "${sources that may contain relevant content}\n",
- "\n",
- "Question: ${the question to be answered}\n",
- "\n",
- "Rationale: Let's think step by step. Based on the context, we have learned the following. ${information from the context that provides useful clues}\n",
- "\n",
- "Search Query: ${a simple question for seeking the missing information}\n",
+ "Context: N/A\n",
"\n",
- "---\n",
+ "Question: How many storeys are in the castle that David Gregory inherited?\n",
"\n",
- "Context:\n",
- "[1] «Gary Zukav | Gary Zukav Gary Zukav (born October 17, 1942) is an American spiritual teacher and the author of four consecutive New York Times Best Sellers. Beginning in 1998, he appeared more than 30 times on \"The Oprah Winfrey Show\" to discuss transformation in human consciousness concepts presented in his book \"The Seat of the Soul\". His first book, \"The Dancing Wu Li Masters\" (1979), won a U.S. National Book Award. Gary Zukav was born in Port Arthur, Texas, and spent his early childhood in San Antonio and Houston. His family moved to Pittsburg, Kansas, while he was in fourth grade. In»\n",
- "[2] «Gary Zukav | experience that led to his first book, \"Dancing Wu Li Masters\", written with extensive help from Jack Sarfatti and other physicists he met through Sarfatti as described in David Kaiser's book \"How the Hippies Saved Physics\". He later described this book as his \"first gift to Life\". In 1987 he moved to Mount Shasta, California, where he lived in a cabin as a self-described \"secular monk\" and spent extensive time in the surrounding wilderness. In 1993 he met Linda Francis. They co-founded the Seat of the Soul Institute in 1998 and moved to Ashland, Oregon, in 2000. Christopher Lehmann-Haupt reviewed»\n",
+ "Reasoning: Let's think step by step in order to\u001b[32m find the answer to this question. First, we need to find information about David Gregory and the castle he inherited. Then, we can search for details about the castle's architecture or any historical records that mention the number of storeys.\n",
"\n",
- "Question: Which award did the first book of Gary Zukav receive?\n",
+ "Query: \"David Gregory castle inheritance\"\u001b[0m\n",
"\n",
- "Rationale: Let's think step by step. Based on the context, we have learned the following. Gary Zukav is an American spiritual teacher and the author of four consecutive New York Times Best Sellers. His first book, \"The Dancing Wu Li Masters\" was written with extensive help from Jack Sarfatti and other physicists.\n",
"\n",
- "Search Query: Which award did \"The Dancing Wu Li Masters\" receive?\n",
"\n",
- "---\n",
"\n",
- "Context:\n",
- "[1] «John du Pont | John du Pont John Eleuthère du Pont (November 22, 1938 – December 9, 2010) was an American philanthropist, heir to the Du Pont family fortune, and a convicted murderer. He had been a published ornithologist, philatelist, conchologist, sports enthusiast, and self-styled wrestling coach. He died in prison while serving a sentence of 30 years for the murder of Dave Schultz. In 1972, du Pont founded and directed the Delaware Museum of Natural History and contributed to Villanova University and other institutions. In the 1980s, he established a wrestling facility at his Foxcatcher Farm after becoming interested in the sport and»\n",
- "[2] «Du Pont family | with political and business ventures, as well as philanthropic efforts on behalf of various organizations such as The Jockey Club, the Philadelphia Orchestra, the Delaware Art Museum, and the Grand Opera House. Two family members were the subjects of well-publicized criminal cases. Heir John Eleuthère du Pont was convicted of murdering wrestling coach Dave Schultz in 1996 and heir Robert H. Richards IV was convicted of raping his 3-year-old daughter in 2009. The family is depicted in the 2014 biographical film \"Foxcatcher\" with Steve Carell playing John Eleuthère du Pont and Vanessa Redgrave playing Jean du Pont, the wife of»\n",
"\n",
- "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
"\n",
- "Rationale: Let's think step by step. Based on the context, we have learned the following. The Du Pont family is a wealthy American family with a long history of business and philanthropic ventures. In the 1980s, one of the heirs, John Eleuthère du Pont, became interested in wrestling and established a wrestling facility at his farm.\n",
"\n",
- "Search Query: Which wrestling team did John Eleuthère du Pont sponsor?\n",
+ "Write a simple search query that will help answer a complex question.\n",
"\n",
"---\n",
"\n",
- "Context:\n",
- "[1] «Peter Outerbridge | Peter Outerbridge Peter Outerbridge (born June 30, 1966) is a Canadian actor, best known for his role as Ari Tasarov in the CW action series \"Nikita\", Dr. David Sandström in the TMN series \"ReGenesis\", Henrik \"Hank\" Johanssen in \"Orphan Black\", Bob Corbett in \"Bomb Girls\", William Easton in \"Saw VI\" and George Brown in the television film \"\". He also played the lead role of Detective William Murdoch in a three-episode mini-series, \"The Murdoch Mysteries\", in its initial run on Canadian television, with two episodes shown in 2004 and a third in 2005. Outerbridge was born and raised in Toronto,»\n",
- "[2] «Except the Dying | 2008. On March 3, 2015, Acorn Media announced a re-release for all three movies, set for May 26, 2015. Except the Dying Except the Dying is a 2004 made-for-TV film starring Peter Outerbridge, Colm Meaney, Keeley Hawes, William B. Davis and Flora Montgomery. It was adapted by Janet MacLean from the novel of the same name by Maureen Jennings. In this film, William Murdoch is introduced, as a man of strong principles, who uses his unique abilities to solve crimes, sometimes using advanced science for his time. On the street of Toronto, in the 1890s, the naked body of a»\n",
+ "Follow the following format.\n",
"\n",
- "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
+ "Context: may contain relevant facts\n",
"\n",
- "Rationale: Let's think step by step. Based on the context, we have learned the following. Outerbridge was born and raised in Toronto, he is a Canadian actor, best known for his role as Ari Tasarov in the CW action series \"Nikita\", Dr. David Sandström in the TMN series \"ReGenesis\", Henrik \"Hank\" Johanssen in \"Orphan Black\", Bob Corbett in \"Bomb Girls\", William Easton in \"Saw VI\" and George Brown in the television film.\n",
+ "Question: ${question}\n",
"\n",
- "Search Query: Who was the director of the 2009 movie \"Saw VI\"?\n",
+ "Reasoning: Let's think step by step in order to ${produce the query}. We ...\n",
+ "\n",
+ "Query: ${query}\n",
"\n",
"---\n",
"\n",
"Context:\n",
- "[1] «1971 San Francisco mayoral election | 1971 San Francisco mayoral election The San Francisco mayoral election of 1971 was held on November 2, 1971, with incumbent Joseph Alioto being re-elected with 38.6 percent of the vote, from among 11 candidates, there being no provision for a runoff. The three major contenders—Joe Alioto; Harold Dobbs, former member of the Board of Supervisors; and Dianne Feinstein, president of the Board of Supervisors—tried to outdo one another as advocates for law and order. Alioto challenged Dobbs to commit to keeping Police Chief Donald Scott and accused Feinstein of wanting to create a civilian review board to investigate police conduct.»\n",
- "[2] «1971 San Francisco mayoral election | the November 2 election was the re-election of Mayor Joseph Alioto by a 3–2 margin over his nearest challenger. Alioto quoted Jackie Gleason—\"How sweet it is!\"—at a victory celebration and referred to his federal indictment as an attempt by the \"national administration\" to \"knock out a mayor\". Among other races on the ballot was Richard Hongisto's surprise victory for sheriff over incumbent Matthew Carberry. On the ballot was a referendum limiting new-building heights to six stories, which was defeated. A referendum to make the school board an elective one, it being the only appointive board in the state, passed. The»\n",
- "[3] «1971 San Francisco mayoral election | school-board measure was regarded as a reaction to busing initiated by the school board under court order. Another measure approved was to keep cable car service at no less than current levels. 1971 San Francisco mayoral election The San Francisco mayoral election of 1971 was held on November 2, 1971, with incumbent Joseph Alioto being re-elected with 38.6 percent of the vote, from among 11 candidates, there being no provision for a runoff. The three major contenders—Joe Alioto; Harold Dobbs, former member of the Board of Supervisors; and Dianne Feinstein, president of the Board of Supervisors—tried to outdo one another»\n",
- "[4] «1971 San Francisco mayoral election | achieve ethnic balance in elementary schools. Former newspaper editor Scott Newhall proposed staging a world ecology fair and replacing California's water plan by importing icebergs, and imputed little difference between the top three contenders, whom he lumped together as \"Joe Dobbstein\". Alioto was a slight favorite to win in spite of being scheduled to go on trial the following January on charges of conspiracy, mail fraud, and using the mail to promote bribery of a public official. As of November, Dobbs, the only Republican in the race, had passed Feinstein in polls and was gaining on Alioto. The result of»\n",
- "[5] «2003 San Francisco mayoral election | against leftist/progressive Gonzalez, a member of the Green Party. It was the first mayoral election in San Francisco that a Green Party candidate took a noticeable amount of the vote. The election was close, with Gonzalez leading in the polls and winning the popular vote among ballots cast on election day, while Newsom had a larger lead on absentee ballots. 2003 San Francisco mayoral election The 2003 San Francisco mayoral election occurred on November 4, 2003. The incumbent, Willie Brown, was termed out of office and could not seek a third term. The general election included three top candidates including»\n",
+ "[1] «David Gregory (physician) | David Gregory (20 December 1625 – 1720) was a Scottish physician and inventor. His surname is sometimes spelt as Gregorie, the original Scottish spelling. He inherited Kinnairdy Castle in 1664. Three of his twenty-nine children became mathematics professors. He is credited with inventing a military cannon that Isaac Newton described as \"being destructive to the human species\". Copies and details of the model no longer exist. Gregory's use of a barometer to predict farming-related weather conditions led him to be accused of witchcraft by Presbyterian ministers from Aberdeen, although he was never convicted.»\n",
+ "[2] «The Boleyn Inheritance | The Boleyn Inheritance is a novel by British author Philippa Gregory which was first published in 2006. It is a direct sequel to her previous novel \"The Other Boleyn Girl,\" and one of the additions to her six-part series on the Tudor royals. (The other titles in the series are \"The Constant Princess,\" \"The Queen's Fool,\" \"The Virgin's Lover,and The Other Queen.\") * The novel is told through the first-person narratives of – Anne of Cleves, Katherine Howard, and Jane Boleyn, who was mentioned in \"The Other Boleyn Girl.\" It covers a period from 1539 until 1542 and chronicles the fourth and fifth marriages of King Henry VIII of England.»\n",
+ "[3] «Gregory of Gaeta | Gregory was the Duke of Gaeta from 963 until his death. He was the second son of Docibilis II of Gaeta and his wife Orania. He succeeded his brother John II, who had left only daughters. Gregory rapidly depleted the \"publicum\" (public land) of the Duchy of Gaeta by doling it out to family members as grants. Gregory disappears from the records in 964 and was succeeded by his younger brother Marinus of Fondi over the heads of his three sons. It is possible that there was an internal power struggle between factions of the Docibilan family and that Gregory was forced out. On the other hand, perhaps he died and his sons fought a losing battle for their inheritance to Gaeta.»\n",
"\n",
- "Question: What year was the party of the winner of the 1971 San Francisco mayoral election founded?\n",
+ "Question: How many storeys are in the castle that David Gregory inherited?\n",
"\n",
- "Rationale: Let's think step by step. Based on the context, we have learned the following.\u001b[32m The San Francisco mayoral election of 1971 was held on November 2, 1971, with incumbent Joseph Alioto being re-elected with 38.6 percent of the vote. Alioto was a slight favorite to win in spite of being scheduled to go on trial the following January on charges of conspiracy, mail fraud, and using the mail to promote bribery of a public official.\n",
+ "Reasoning: Let's think step by step in order to\u001b[32m produce the query. We know that David Gregory inherited Kinnairdy Castle, so we need to find information about the castle and its characteristics.\n",
"\n",
- "Search Query: What year was the Democratic Party founded?\u001b[0m\u001b[31m \t (and 9 other completions)\u001b[0m\n",
+ "Query: \"Kinnairdy Castle number of storeys\"\u001b[0m\n",
"\n",
"\n",
"\n",
@@ -1661,119 +1393,153 @@
"\n",
"---\n",
"\n",
- "Question: Who produced the album that included a re-recording of \"Lithium\"?\n",
- "Answer: Butch Vig\n",
- "\n",
- "Question: What city was the victim of Joseph Druces working in?\n",
- "Answer: Boston, Massachusetts\n",
- "\n",
- "Question: In what year was the star of To Hell and Back born?\n",
- "Answer: 1925\n",
- "\n",
- "---\n",
- "\n",
"Follow the following format.\n",
"\n",
- "Context:\n",
- "${sources that may contain relevant content}\n",
- "\n",
- "Question: ${the question to be answered}\n",
- "\n",
- "Rationale: Let's think step by step. ${a step-by-step deduction that identifies the correct response, which will be provided below}\n",
- "\n",
- "Answer: ${a short factoid answer, often between 1 and 5 words}\n",
+ "Context: may contain relevant facts\n",
"\n",
- "---\n",
- "\n",
- "Context:\n",
- "[1] «Gary Zukav | Gary Zukav Gary Zukav (born October 17, 1942) is an American spiritual teacher and the author of four consecutive New York Times Best Sellers. Beginning in 1998, he appeared more than 30 times on \"The Oprah Winfrey Show\" to discuss transformation in human consciousness concepts presented in his book \"The Seat of the Soul\". His first book, \"The Dancing Wu Li Masters\" (1979), won a U.S. National Book Award. Gary Zukav was born in Port Arthur, Texas, and spent his early childhood in San Antonio and Houston. His family moved to Pittsburg, Kansas, while he was in fourth grade. In»\n",
- "[2] «Gary Zukav | experience that led to his first book, \"Dancing Wu Li Masters\", written with extensive help from Jack Sarfatti and other physicists he met through Sarfatti as described in David Kaiser's book \"How the Hippies Saved Physics\". He later described this book as his \"first gift to Life\". In 1987 he moved to Mount Shasta, California, where he lived in a cabin as a self-described \"secular monk\" and spent extensive time in the surrounding wilderness. In 1993 he met Linda Francis. They co-founded the Seat of the Soul Institute in 1998 and moved to Ashland, Oregon, in 2000. Christopher Lehmann-Haupt reviewed»\n",
- "[3] «The Dancing Wu Li Masters | The Dancing Wu Li Masters The Dancing Wu Li Masters is a 1979 book by Gary Zukav, a popular science work exploring modern physics, and quantum phenomena in particular. It was awarded a 1980 U.S. National Book Award in category of Science. Although it explores empirical topics in modern physics research, \"The Dancing Wu Li Masters\" gained attention for leveraging metaphors taken from eastern spiritual movements, in particular the Huayen school of Buddhism with the monk Fazang's treatise on The Golden Lion, to explain quantum phenomena and has been regarded by some reviewers as a New Age work, although the»\n",
- "[4] «The Dancing Wu Li Masters | a 1976 physics conference of eastern and western scientists at Esalen Institute, California; and he used the occasion as material for his book. At the conference, it was said that the Chinese term for physics is 'Wu Li', or \"patterns of organic energy.\" Zukav, among others, conceptualized 'physics' as the dance of the Wu Li Masters – teachers of physical essence. Zukav explains the concept further: The Dancing Wu Li Masters The Dancing Wu Li Masters is a 1979 book by Gary Zukav, a popular science work exploring modern physics, and quantum phenomena in particular. It was awarded a 1980»\n",
- "\n",
- "Question: Which award did the first book of Gary Zukav receive?\n",
- "\n",
- "Rationale: Let's think step by step. The first book is \"The Dancing Wu Li Masters\" and it was awarded a U.S. National Book Award.\n",
- "\n",
- "Answer: U.S. National Book Award\n",
- "\n",
- "---\n",
+ "Question: ${question}\n",
"\n",
- "Context:\n",
- "[1] «John du Pont | John du Pont John Eleuthère du Pont (November 22, 1938 – December 9, 2010) was an American philanthropist, heir to the Du Pont family fortune, and a convicted murderer. He had been a published ornithologist, philatelist, conchologist, sports enthusiast, and self-styled wrestling coach. He died in prison while serving a sentence of 30 years for the murder of Dave Schultz. In 1972, du Pont founded and directed the Delaware Museum of Natural History and contributed to Villanova University and other institutions. In the 1980s, he established a wrestling facility at his Foxcatcher Farm after becoming interested in the sport and»\n",
- "[2] «Du Pont family | with political and business ventures, as well as philanthropic efforts on behalf of various organizations such as The Jockey Club, the Philadelphia Orchestra, the Delaware Art Museum, and the Grand Opera House. Two family members were the subjects of well-publicized criminal cases. Heir John Eleuthère du Pont was convicted of murdering wrestling coach Dave Schultz in 1996 and heir Robert H. Richards IV was convicted of raping his 3-year-old daughter in 2009. The family is depicted in the 2014 biographical film \"Foxcatcher\" with Steve Carell playing John Eleuthère du Pont and Vanessa Redgrave playing Jean du Pont, the wife of»\n",
- "[3] «John du Pont | Foxcatcher team. Du Pont became a sponsor in wrestling, swimming, track, and the modern pentathlon. He was also involved in promoting a subset of the modern pentathlon (run, swim, shoot) as a separate event. He took up athletics and became a competitive wrestler in his 50s. His only prior wrestling experience was as a freshman in high school. He began competing again at the age of 55 in the 1992 Veteran's World Championships in Cali, Colombia; following that in 1993 in Toronto, Ontario; in 1994 in Rome, Italy; and in 1995 in Sofia, Bulgaria. In August 1988, a problem-plagued wrestling»\n",
- "\n",
- "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
"\n",
- "Rationale: Let's think step by step. The question is asking for a wrestling team that the heir to the Du Pont family sponsored. In the context, it is mentioned that the heir, John du Pont, was interested in wrestling and even competed in wrestling competitions in his 50s. Furthermore, it is stated that he sponsored a wrestling team. Therefore, the answer must be a wrestling team.\n",
- "\n",
- "Answer: Foxcatcher team\n",
+ "Answer: often between 1 and 5 words\n",
"\n",
"---\n",
"\n",
"Context:\n",
- "[1] «Peter Outerbridge | Peter Outerbridge Peter Outerbridge (born June 30, 1966) is a Canadian actor, best known for his role as Ari Tasarov in the CW action series \"Nikita\", Dr. David Sandström in the TMN series \"ReGenesis\", Henrik \"Hank\" Johanssen in \"Orphan Black\", Bob Corbett in \"Bomb Girls\", William Easton in \"Saw VI\" and George Brown in the television film \"\". He also played the lead role of Detective William Murdoch in a three-episode mini-series, \"The Murdoch Mysteries\", in its initial run on Canadian television, with two episodes shown in 2004 and a third in 2005. Outerbridge was born and raised in Toronto,»\n",
- "[2] «Except the Dying | 2008. On March 3, 2015, Acorn Media announced a re-release for all three movies, set for May 26, 2015. Except the Dying Except the Dying is a 2004 made-for-TV film starring Peter Outerbridge, Colm Meaney, Keeley Hawes, William B. Davis and Flora Montgomery. It was adapted by Janet MacLean from the novel of the same name by Maureen Jennings. In this film, William Murdoch is introduced, as a man of strong principles, who uses his unique abilities to solve crimes, sometimes using advanced science for his time. On the street of Toronto, in the 1890s, the naked body of a»\n",
- "[3] «Saw VI | Saw VI Saw VI is a 2009 American horror film directed by Kevin Greutert from a screenplay written by Patrick Melton and Marcus Dunstan. It is the sixth installment in the \"Saw\" franchise and stars Tobin Bell, Costas Mandylor, Betsy Russell, Mark Rolston, Peter Outerbridge, and Shawnee Smith. It was produced by Mark Burg and Oren Koules of Twisted Pictures and distributed by Lionsgate. \"Saw VI\" continues the franchise's focus on the posthumous effects of the Jigsaw Killer and the progression of his successor, Mark Hoffman. In this film, Hoffman sets a series of traps for an insurance executive, William»\n",
- "[4] «Saw VI | is a crafty move on the screenwriters' part, but once you get past that pseudo-ironic touch, this Saw is no more or less disturbing than any other in the series\". Saw VI Saw VI is a 2009 American horror film directed by Kevin Greutert from a screenplay written by Patrick Melton and Marcus Dunstan. It is the sixth installment in the \"Saw\" franchise and stars Tobin Bell, Costas Mandylor, Betsy Russell, Mark Rolston, Peter Outerbridge, and Shawnee Smith. It was produced by Mark Burg and Oren Koules of Twisted Pictures and distributed by Lionsgate. \"Saw VI\" continues the franchise's focus»\n",
- "\n",
- "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
- "\n",
- "Rationale: Let's think step by step. The question is asking for the director of the 2009 movie that features Peter Outerbridge as William Easton. We can use process of elimination to figure this out. In source 1, there is no mention of a 2009 movie. In source 2, the only mention of Peter Outerbridge is in the cast list for a 2004 made-for-TV film. In source 3, we see that Outerbridge was in a 2009 movie, Saw VI, and that it was directed by Kevin Greutert. This must be the correct answer, so we can check source 4 to confirm. In source 4, we see that Kevin Greutert is in fact the director of Saw VI.\n",
- "\n",
- "Answer: Kevin Greutert\n",
- "\n",
- "---\n",
+ "[1] «David Gregory (physician) | David Gregory (20 December 1625 – 1720) was a Scottish physician and inventor. His surname is sometimes spelt as Gregorie, the original Scottish spelling. He inherited Kinnairdy Castle in 1664. Three of his twenty-nine children became mathematics professors. He is credited with inventing a military cannon that Isaac Newton described as \"being destructive to the human species\". Copies and details of the model no longer exist. Gregory's use of a barometer to predict farming-related weather conditions led him to be accused of witchcraft by Presbyterian ministers from Aberdeen, although he was never convicted.»\n",
+ "[2] «The Boleyn Inheritance | The Boleyn Inheritance is a novel by British author Philippa Gregory which was first published in 2006. It is a direct sequel to her previous novel \"The Other Boleyn Girl,\" and one of the additions to her six-part series on the Tudor royals. (The other titles in the series are \"The Constant Princess,\" \"The Queen's Fool,\" \"The Virgin's Lover,and The Other Queen.\") * The novel is told through the first-person narratives of – Anne of Cleves, Katherine Howard, and Jane Boleyn, who was mentioned in \"The Other Boleyn Girl.\" It covers a period from 1539 until 1542 and chronicles the fourth and fifth marriages of King Henry VIII of England.»\n",
+ "[3] «Gregory of Gaeta | Gregory was the Duke of Gaeta from 963 until his death. He was the second son of Docibilis II of Gaeta and his wife Orania. He succeeded his brother John II, who had left only daughters. Gregory rapidly depleted the \"publicum\" (public land) of the Duchy of Gaeta by doling it out to family members as grants. Gregory disappears from the records in 964 and was succeeded by his younger brother Marinus of Fondi over the heads of his three sons. It is possible that there was an internal power struggle between factions of the Docibilan family and that Gregory was forced out. On the other hand, perhaps he died and his sons fought a losing battle for their inheritance to Gaeta.»\n",
+ "[4] «Kinnairdy Castle | Kinnairdy Castle is a tower house, having five storeys and a garret, two miles south of Aberchirder, Aberdeenshire, Scotland. The alternative name is Old Kinnairdy.»\n",
+ "[5] «Kinnaird Head | Kinnaird Head (Scottish Gaelic: \"An Ceann Àrd\" , \"high headland\") is a headland projecting into the North Sea, within the town of Fraserburgh, Aberdeenshire on the east coast of Scotland. The 16th-century Kinnaird Castle was converted in 1787 for use as the Kinnaird Head Lighthouse, the first lighthouse in Scotland to be lit by the Commissioners of Northern Lights. Kinnaird Castle and the nearby Winetower were described by W. Douglas Simpson as two of the nine castles of the Knuckle, referring to the rocky headland of north-east Aberdeenshire. Both buildings are category A listed buildings.»\n",
+ "[6] «Kinnaird Castle, Brechin | Kinnaird Castle is a 15th-century castle in Angus, Scotland. The castle has been home to the Carnegie family, the Earl of Southesk, for more than 600 years.»\n",
"\n",
- "Context:\n",
- "[1] «The San Francisco mayoral election of 1971 was held on November 2, 1971, with incumbent Joseph Alioto being re-elected with 38.6 percent of the vote. Alioto was a slight favorite to win in spite of being scheduled to go on trial the following January on charges of conspiracy, mail fraud, and using the mail to promote bribery of a public official.»\n",
- "[2] «Democratic Party (UK, 1998) | Democratic Party (UK, 1998) The Democratic Party was a political party in the United Kingdom, founded in November 1998 by Malvern businessman Geoff Southall, who also funded the party. It was described as \"right wing\" or \"hard right\" in news reports, but claimed to occupy the centre-ground of British politics. It aimed to reduce Britain's involvement with the European Union, opposed the adoption of the Euro, called for direct democracy, and argued for limits on immigration. The slogan was \"The will of the people NOT the party\". In 1999 it had a few hundred members, including previous supporters of James»\n",
- "[3] «Democratic Party (Brazil, 1925-1934) | Democratic Party (Brazil, 1925-1934) The Democratic Party was a Brazilian political party, but state character, founded by dissidents of the Paulista Republican Party (PRP) on February 24, 1925, during the Old Republic and was its manifesto signed by 599 signatories. It was a representative organization of the traditional middle class, linked to coffee sectors, but especially the urban. In May 1926, the Youth Party (Youth Wing of dissenters, who had organized this caption for the federal elections of 1925), also dissident perrepismo, joined the Democratic Party. His most important names were Waldemar Ferreira, José Adriano Marrey Junior, Francisco Morato, Paulo»\n",
- "[4] «Democratic Party (Turkey, 1970) | Democratic Party (Turkey, 1970) Democratic Party (Turkish: \"Demokratik Parti\", DP for short) was a former political party in Turkey founded on 18 December 1970 . It should not be confused with the historical Democrat Party or the current Democrat Party in Turkey. The historical Democrat Party which was the ruling party between 1950-1960 was closed by the military rule. In 1961 and 1965 elections most of ex Democratic Party adherents supported Justice Party, a party founded in 1961. After 1965 elections Justice Party came to power. But following 1969 elections in which Justice Party was again successful, tension between the»\n",
- "[5] «Democratic Party (Japan, 1947) | Democratic Party (Japan, 1947) Democratic Party (民主党, Minshutō) was a centrist political party in Japan. It was founded in spring 1947 by merging the Progressive Party (Shinpo-tō) of Inukai Takeru with a faction of Liberal Party led by Hitoshi Ashida and obtained 124 seats in 1947 elections. The party had held seven seats in Tetsu Katayama's government in 1947-1948. For some months in 1948, party's leader Ashida was Prime minister. In March 1948, part of DP members led by Kijūrō Shidehara joined the Liberal Party to form the Democratic Liberal Party. In 1949 elections, the DP got 69 seats. The»\n",
- "[6] «Democratic Party (United States) | Democratic Party (United States) The Democratic Party is one of the two major contemporary political parties in the United States, along with the Republican Party. Tracing its heritage back to Thomas Jefferson and James Madison's Democratic-Republican Party, the modern-day Democratic Party was founded around 1828 by supporters of Andrew Jackson, making it the world's oldest active political party. The Democrats' dominant worldview was once social conservatism and economic liberalism while populism was its leading characteristic in the rural South. In 1912, Theodore Roosevelt ran as a third-party candidate in the Progressive (\"Bull Moose\") Party, beginning a switch of political platforms»\n",
+ "Question: How many storeys are in the castle that David Gregory inherited?\n",
"\n",
- "Question: What year was the party of the winner of the 1971 San Francisco mayoral election founded?\n",
+ "Reasoning: Let's think step by step in order to\u001b[32m produce the answer. We know from the context that David Gregory inherited Kinnairdy Castle. \n",
"\n",
- "Rationale: Let's think step by step.\u001b[32m The question is asking for the year the party of the mayor of San Francisco in 1971 was founded. In source 1, we see that the election was in 1971 and the winner was Joseph Alioto. In source 2, we see that the party was founded in 1998, which is not correct. In source 3, we see that the party was founded in 1925, which is not correct. In source 4, we see that the party was founded in 1970, which is not correct. In source 5, we see that the party was founded in 1947, which is not correct. In source 6, we see that the party was founded around 1828. This must be the correct answer, so we can check source 1 to confirm. In source 1\u001b[0m\u001b[31m \t (and 19 other completions)\u001b[0m\n",
+ "Answer: five\u001b[0m\n",
"\n",
"\n",
"\n"
]
+ }
+ ],
+ "source": [
+ "turbo.inspect_history(n=3)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "##### Compiling the Baleen program\n",
+ "\n",
+ "Now is the time to compile our multi-hop (`SimplifiedBaleen`) program.\n",
+ "\n",
+ "We will first define our validation logic, which will simply require that:\n",
+ "\n",
+ "- The predicted answer matches the gold answer.\n",
+ "- The retrieved context contains the gold answer.\n",
+ "- None of the generated queries is rambling (i.e., none exceeds 100 characters in length).\n",
+ "- None of the generated queries is roughly repeated (i.e., none is within 0.8 or higher F1 score of earlier queries)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 26,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def validate_context_and_answer_and_hops(example, pred, trace=None):\n",
+ " if not dspy.evaluate.answer_exact_match(example, pred): return False\n",
+ " if not dspy.evaluate.answer_passage_match(example, pred): return False\n",
+ "\n",
+ " hops = [example.question] + [outputs.query for *_, outputs in trace if 'query' in outputs]\n",
+ "\n",
+ " if max([len(h) for h in hops]) > 100: return False\n",
+ " if any(dspy.evaluate.answer_exact_match_str(hops[idx], hops[:idx], frac=0.8) for idx in range(2, len(hops))): return False\n",
+ "\n",
+ " return True"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Like we did for RAG, we'll use one of the most basic teleprompters in **DSPy**, namely, `BootstrapFewShot`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 27,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ " 0%| | 0/20 [00:00<?, ?it/s]"
+ ]
},
{
- "data": {
- "text/plain": [
- "('1828', None)"
- ]
- },
- "execution_count": 25,
- "metadata": {},
- "output_type": "execute_result"
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "100%|██████████| 20/20 [00:12<00:00, 1.61it/s]\n"
+ ]
}
],
"source": [
- "multihop_QA_v2(dev[3].question), lm.inspect_history(n=3)"
+ "teleprompter = BootstrapFewShot(metric=validate_context_and_answer_and_hops)\n",
+ "compiled_baleen = teleprompter.compile(SimplifiedBaleen(), teacher=SimplifiedBaleen(passages_per_hop=2), trainset=trainset)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "##### Evaluating the Retrieval\n",
+ "\n",
+ "Earlier, it appeared like our simple RAG program was not very effective at finding all evidence required for answering each question. Is this resolved by the adding some extra steps in the `forward` function of `SimplifiedBaleen`? What about compiling, does it help for that? \n",
+ "\n",
+ "The answer for these questions is not always going to be obvious. However, **DSPy** makes it extremely easy to try many diverse approaches with minimal effort.\n",
+ "\n",
+ "Let's evaluate the quality of retrieval of our compiled and uncompiled Baleen pipelines!"
]
},
{
"cell_type": "code",
- "execution_count": 26,
+ "execution_count": 28,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "uncompiled_baleen_retrieval_score = evaluate_on_hotpotqa(uncompiled_baleen, metric=gold_passages_retrieved, display=False)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 29,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
- "100%|██████████| 13/13 [00:02<00:00, 5.38it/s]"
+ " 0%| | 0/50 [00:00<?, ?it/s]"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Average Metric: 30 / 50 (60.0): 100%|██████████| 50/50 [00:27<00:00, 1.80it/s]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
- "Answered 11 / 13 (84.6%) correctly.\n"
+ "Average Metric: 30 / 50 (60.0%)\n"
]
},
{
@@ -1786,107 +1552,83 @@
{
"data": {
"text/html": [
- "<style type=\"text/css\" >\n",
- " #T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2 th {\n",
- " text-align: left;\n",
- " } #T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2 td {\n",
- " text-align: left;\n",
- " }</style><table id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2\" ><thead> <tr> <th class=\"blank level0\" ></th> <th class=\"col_heading level0 col0\" >question</th> <th class=\"col_heading level0 col1\" >answer</th> <th class=\"col_heading level0 col2\" >prediction</th> <th class=\"col_heading level0 col3\" >correct</th> </tr></thead><tbody>\n",
- " <tr>\n",
- " <th id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row0_col2\" class=\"data row0 col2\" >E. L. Doctorow</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row0_col3\" class=\"data row0 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row1_col2\" class=\"data row1 col2\" >The Killing Season</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row1_col3\" class=\"data row1 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row2_col2\" class=\"data row2 col2\" >Gainesville, Florida</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row2_col3\" class=\"data row2 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row3_col2\" class=\"data row3 col2\" >1828</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row3_col3\" class=\"data row3 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row4_col2\" class=\"data row4 col2\" >John Braine</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row4_col3\" class=\"data row4 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row5_col3\" class=\"data row5 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row6_col2\" class=\"data row6 col2\" >Cheshire, UK</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row6_col3\" class=\"data row6 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row7_col2\" class=\"data row7 col2\" >February 13, 1980</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row7_col3\" class=\"data row7 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row8_col2\" class=\"data row8 col2\" >Dundalk</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row8_col3\" class=\"data row8 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row9_col2\" class=\"data row9 col2\" >Aerospace engineering</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row9_col3\" class=\"data row9 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row10_col2\" class=\"data row10 col2\" >Guildford, Surrey</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row10_col3\" class=\"data row10 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row11_col2\" class=\"data row11 col2\" >Portrait of a Marriage</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row11_col3\" class=\"data row11 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row12_col2\" class=\"data row12 col2\" >1442</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row12_col3\" class=\"data row12 col3\" >✔️</td>\n",
- " </tr>\n",
- " </tbody></table>"
+ "<style type=\"text/css\">\n",
+ "#T_c548d th {\n",
+ " text-align: left;\n",
+ "}\n",
+ "#T_c548d td {\n",
+ " text-align: left;\n",
+ "}\n",
+ "#T_c548d_row0_col0, #T_c548d_row0_col1, #T_c548d_row0_col2, #T_c548d_row0_col3, #T_c548d_row0_col4, #T_c548d_row0_col5, #T_c548d_row1_col0, #T_c548d_row1_col1, #T_c548d_row1_col2, #T_c548d_row1_col3, #T_c548d_row1_col4, #T_c548d_row1_col5, #T_c548d_row2_col0, #T_c548d_row2_col1, #T_c548d_row2_col2, #T_c548d_row2_col3, #T_c548d_row2_col4, #T_c548d_row2_col5, #T_c548d_row3_col0, #T_c548d_row3_col1, #T_c548d_row3_col2, #T_c548d_row3_col3, #T_c548d_row3_col4, #T_c548d_row3_col5, #T_c548d_row4_col0, #T_c548d_row4_col1, #T_c548d_row4_col2, #T_c548d_row4_col3, #T_c548d_row4_col4, #T_c548d_row4_col5 {\n",
+ " text-align: left;\n",
+ " white-space: pre-wrap;\n",
+ " word-wrap: break-word;\n",
+ " max-width: 400px;\n",
+ "}\n",
+ "</style>\n",
+ "<table id=\"T_c548d\">\n",
+ " <thead>\n",
+ " <tr>\n",
+ " <th class=\"blank level0\" >&nbsp;</th>\n",
+ " <th id=\"T_c548d_level0_col0\" class=\"col_heading level0 col0\" >question</th>\n",
+ " <th id=\"T_c548d_level0_col1\" class=\"col_heading level0 col1\" >example_answer</th>\n",
+ " <th id=\"T_c548d_level0_col2\" class=\"col_heading level0 col2\" >gold_titles</th>\n",
+ " <th id=\"T_c548d_level0_col3\" class=\"col_heading level0 col3\" >context</th>\n",
+ " <th id=\"T_c548d_level0_col4\" class=\"col_heading level0 col4\" >pred_answer</th>\n",
+ " <th id=\"T_c548d_level0_col5\" class=\"col_heading level0 col5\" >gold_passages_retrieved</th>\n",
+ " </tr>\n",
+ " </thead>\n",
+ " <tbody>\n",
+ " <tr>\n",
+ " <th id=\"T_c548d_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
+ " <td id=\"T_c548d_row0_col0\" class=\"data row0 col0\" >Are both Cangzhou and Qionghai in the Hebei province of China?</td>\n",
+ " <td id=\"T_c548d_row0_col1\" class=\"data row0 col1\" >no</td>\n",
+ " <td id=\"T_c548d_row0_col2\" class=\"data row0 col2\" >{'Qionghai', 'Cangzhou'}</td>\n",
+ " <td id=\"T_c548d_row0_col3\" class=\"data row0 col3\" >['Cangzhou | Cangzhou () is a prefecture-level city in eastern Hebei province, People\\'s Republic of China. At the 2010 census, Cangzhou\\'s built-up (\"or metro\") area...</td>\n",
+ " <td id=\"T_c548d_row0_col4\" class=\"data row0 col4\" >No</td>\n",
+ " <td id=\"T_c548d_row0_col5\" class=\"data row0 col5\" >✔️ [True]</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_c548d_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
+ " <td id=\"T_c548d_row1_col0\" class=\"data row1 col0\" >Who conducts the draft in which Marc-Andre Fleury was drafted to the Vegas Golden Knights for the 2017-18 season?</td>\n",
+ " <td id=\"T_c548d_row1_col1\" class=\"data row1 col1\" >National Hockey League</td>\n",
+ " <td id=\"T_c548d_row1_col2\" class=\"data row1 col2\" >{'2017 NHL Expansion Draft', '2017–18 Pittsburgh Penguins season'}</td>\n",
+ " <td id=\"T_c548d_row1_col3\" class=\"data row1 col3\" >[\"2017 NHL Expansion Draft | The 2017 NHL Expansion Draft was an expansion draft conducted by the National Hockey League on June 18–20, 2017 to...</td>\n",
+ " <td id=\"T_c548d_row1_col4\" class=\"data row1 col4\" >National Hockey League (NHL)</td>\n",
+ " <td id=\"T_c548d_row1_col5\" class=\"data row1 col5\" >❌ [False]</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_c548d_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
+ " <td id=\"T_c548d_row2_col0\" class=\"data row2 col0\" >The Wings entered a new era, following the retirement of which Canadian retired professional ice hockey player and current general manager of the Tampa Bay...</td>\n",
+ " <td id=\"T_c548d_row2_col1\" class=\"data row2 col1\" >Steve Yzerman</td>\n",
+ " <td id=\"T_c548d_row2_col2\" class=\"data row2 col2\" >{'Steve Yzerman', '2006–07 Detroit Red Wings season'}</td>\n",
+ " <td id=\"T_c548d_row2_col3\" class=\"data row2 col3\" >['List of Tampa Bay Lightning general managers | The Tampa Bay Lightning are an American professional ice hockey team based in Tampa, Florida. They play...</td>\n",
+ " <td id=\"T_c548d_row2_col4\" class=\"data row2 col4\" >Steve Yzerman</td>\n",
+ " <td id=\"T_c548d_row2_col5\" class=\"data row2 col5\" >❌ [False]</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_c548d_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
+ " <td id=\"T_c548d_row3_col0\" class=\"data row3 col0\" >What river is near the Crichton Collegiate Church?</td>\n",
+ " <td id=\"T_c548d_row3_col1\" class=\"data row3 col1\" >the River Tyne</td>\n",
+ " <td id=\"T_c548d_row3_col2\" class=\"data row3 col2\" >{'Crichton Collegiate Church', 'Crichton Castle'}</td>\n",
+ " <td id=\"T_c548d_row3_col3\" class=\"data row3 col3\" >[\"Crichton Collegiate Church | Crichton Collegiate Church is situated about 0.6 mi south west of the hamlet of Crichton in Midlothian, Scotland. Crichton itself is...</td>\n",
+ " <td id=\"T_c548d_row3_col4\" class=\"data row3 col4\" >River Tyne</td>\n",
+ " <td id=\"T_c548d_row3_col5\" class=\"data row3 col5\" >✔️ [True]</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_c548d_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
+ " <td id=\"T_c548d_row4_col0\" class=\"data row4 col0\" >In the 10th Century A.D. Ealhswith had a son called Æthelweard by which English king?</td>\n",
+ " <td id=\"T_c548d_row4_col1\" class=\"data row4 col1\" >King Alfred the Great</td>\n",
+ " <td id=\"T_c548d_row4_col2\" class=\"data row4 col2\" >{'Æthelweard (son of Alfred)', 'Ealhswith'}</td>\n",
+ " <td id=\"T_c548d_row4_col3\" class=\"data row4 col3\" >['Æthelweard (son of Alfred) | Æthelweard (d. 920 or 922) was the younger son of King Alfred the Great and Ealhswith.', 'Æthelred the Unready |...</td>\n",
+ " <td id=\"T_c548d_row4_col4\" class=\"data row4 col4\" >King Alfred the Great</td>\n",
+ " <td id=\"T_c548d_row4_col5\" class=\"data row4 col5\" >❌ [False]</td>\n",
+ " </tr>\n",
+ " </tbody>\n",
+ "</table>\n"
],
"text/plain": [
- "<pandas.io.formats.style.Styler at 0x7f9ba36cfb50>"
+ "<pandas.io.formats.style.Styler at 0x7f559c5b9760>"
]
},
"metadata": {},
@@ -1894,33 +1636,72 @@
},
{
"data": {
+ "text/html": [
+ "\n",
+ " <div style='\n",
+ " text-align: center; \n",
+ " font-size: 16px; \n",
+ " font-weight: bold; \n",
+ " color: #555; \n",
+ " margin: 10px 0;'>\n",
+ " ... 45 more rows not displayed ...\n",
+ " </div>\n",
+ " "
+ ],
"text/plain": [
- "84.6"
+ "<IPython.core.display.HTML object>"
]
},
- "execution_count": 26,
"metadata": {},
- "output_type": "execute_result"
+ "output_type": "display_data"
}
],
"source": [
- "evaluate(multihop_QA_v2, dev)"
+ "compiled_baleen_retrieval_score = evaluate_on_hotpotqa(compiled_baleen, metric=gold_passages_retrieved)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 30,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "## Retrieval Score for RAG: 26.0\n",
+ "## Retrieval Score for uncompiled Baleen: 36.0\n",
+ "## Retrieval Score for compiled Baleen: 60.0\n"
+ ]
+ }
+ ],
+ "source": [
+ "print(f\"## Retrieval Score for RAG: {compiled_rag_retrieval_score}\") # note that for RAG, compilation has no effect on the retrieval step\n",
+ "print(f\"## Retrieval Score for uncompiled Baleen: {uncompiled_baleen_retrieval_score}\")\n",
+ "print(f\"## Retrieval Score for compiled Baleen: {compiled_baleen_retrieval_score}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "### Ask your own questions\n",
+ "Excellent! There might be something to this compiled, multi-hop program then. But this is far from all you can do: **DSPy** gives you a clean space of composable operators to deal with any shortcomings you see.\n",
+ "\n",
+ "We can inspect a few concrete examples. If we see failure causes, we can:\n",
+ "\n",
+ "1. Expand our pipeline by using additional sub-modules (e.g., maybe summarize after retrieval?)\n",
+ "1. Modify our pipeline by using more complex logic (e.g., maybe we need to break out of the multi-hop loop if we found all information we need?) \n",
+ "1. Refine our validation logic (e.g., maybe use a metric that use a second **DSPy** program to do the answer evaluation, instead of relying on strict string matching)\n",
+ "1. Use a different teleprompter to optimize your pipeline more aggressively.\n",
+ "1. Add more or better training examples!\n",
"\n",
- "You could also ask your own questions. This can be a productive vehicle to find problems in the programs above and implement patches for them! (Keep in mind that the search index we're using currently is from Dec 2018.)\n",
"\n",
- "To do this, make sure you set your API key. This can be done by defining the `OPENAI_API_KEY` environemt variable or setting it at the beginning of the program (Cell #2)."
+ "Or, if you really want, we can tweak the descriptions in the Signatures we use in your program to make them more precisely suited for their sub-tasks. This is akin to prompt engineering and should be a final resort, given the other powerful options that **DSPy** gives us!"
]
},
{
"cell_type": "code",
- "execution_count": 27,
+ "execution_count": 31,
"metadata": {},
"outputs": [
{
@@ -1931,39 +1712,59 @@
"\n",
"\n",
"\n",
- "Write a search query that will help answer a complex question.\n",
+ "Write a simple search query that will help answer a complex question.\n",
"\n",
"---\n",
"\n",
"Follow the following format.\n",
"\n",
- "Question: ${the question to be answered}\n",
- "Rationale: Let's think step by step. To answer this question, we first need to find out ${the missing information}\n",
- "Search Query: ${a simple question for seeking the missing information}\n",
+ "Context: may contain relevant facts\n",
+ "\n",
+ "Question: ${question}\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to ${produce the query}. We ...\n",
+ "\n",
+ "Query: ${query}\n",
"\n",
"---\n",
"\n",
- "Question: Which award did the first book of Gary Zukav receive?\n",
- "Rationale: Let's think step by step. To answer this question, we first need to find out the name of the first book of Gary Zukav.\n",
- "Search Query: \"What is the name of Gary Zukav's first book?\"\n",
+ "Context: N/A\n",
+ "\n",
+ "Question: In what year was the club founded that played Manchester City in the 1972 FA Charity Shield\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the query. We know that the FA Charity Shield is an annual football match played in England between the winners of the previous season's Premier League and FA Cup. In this case, we are looking for the year when Manchester City played against a specific club in the 1972 FA Charity Shield. To find this information, we can search for the history of the FA Charity Shield and the teams that participated in the 1972 edition.\n",
+ "\n",
+ "Query: \"History of FA Charity Shield 1972\"\n",
"\n",
"---\n",
"\n",
- "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
- "Rationale: Let's think step by step. To answer this question, we first need to find out who the heir to the Du Pont family fortune is.\n",
- "Search Query: \"Heir to the Du Pont family fortune\"\n",
+ "Context: N/A\n",
+ "\n",
+ "Question: Which is taller, the Empire State Building or the Bank of America Tower?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the query. We need to find the heights of both buildings and compare them.\n",
+ "\n",
+ "Query: \"height of Empire State Building\" OR \"height of Bank of America Tower\"\n",
"\n",
"---\n",
"\n",
- "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
- "Rationale: Let's think step by step. To answer this question, we first need to find out the name of the 2009 movie featuring Peter Outerbridge as William Easton.\n",
- "Search Query: \"2009 movie featuring Peter Outerbridge as William Easton\"\n",
+ "Context: N/A\n",
+ "\n",
+ "Question: Who is older, Aleksandr Danilovich Aleksandrov or Anatoly Fomenko?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to produce the query. We can search for the birth dates of both Aleksandr Danilovich Aleksandrov and Anatoly Fomenko and compare them to determine who is older.\n",
+ "\n",
+ "Query: \"Birth date Aleksandr Danilovich Aleksandrov\" \"Birth date Anatoly Fomenko\"\n",
"\n",
"---\n",
"\n",
- "Question: When was the creator of Hadoop given an award?\n",
- "Rationale: Let's think step by step. To answer this question, we first need to find out\u001b[32m the name of the creator of Hadoop.\n",
- "Search Query: \"Name of the creator of Hadoop\"\u001b[0m\u001b[31m \t (and 9 other completions)\u001b[0m\n",
+ "Context: N/A\n",
+ "\n",
+ "Question: How many storeys are in the castle that David Gregory inherited?\n",
+ "\n",
+ "Reasoning: Let's think step by step in order to\u001b[32m produce the query. We need to find information about the castle that David Gregory inherited and determine the number of storeys it has.\n",
+ "\n",
+ "Query: \"Castle inherited by David Gregory number of storeys\"\u001b[0m\n",
"\n",
"\n",
"\n",
@@ -1971,71 +1772,68 @@
"\n",
"\n",
"\n",
- "Write a search query that will help answer a complex question.\n",
+ "Write a simple search query that will help answer a complex question.\n",
"\n",
"---\n",
"\n",
"Follow the following format.\n",
"\n",
- "Context:\n",
- "${sources that may contain relevant content}\n",
+ "Context: may contain relevant facts\n",
"\n",
- "Question: ${the question to be answered}\n",
+ "Question: ${question}\n",
"\n",
- "Rationale: Let's think step by step. Based on the context, we have learned the following. ${information from the context that provides useful clues}\n",
+ "Reasoning: Let's think step by step in order to ${produce the query}. We ...\n",
"\n",
- "Search Query: ${a simple question for seeking the missing information}\n",
+ "Query: ${query}\n",
"\n",
"---\n",
"\n",
"Context:\n",
- "[1] «Gary Zukav | Gary Zukav Gary Zukav (born October 17, 1942) is an American spiritual teacher and the author of four consecutive New York Times Best Sellers. Beginning in 1998, he appeared more than 30 times on \"The Oprah Winfrey Show\" to discuss transformation in human consciousness concepts presented in his book \"The Seat of the Soul\". His first book, \"The Dancing Wu Li Masters\" (1979), won a U.S. National Book Award. Gary Zukav was born in Port Arthur, Texas, and spent his early childhood in San Antonio and Houston. His family moved to Pittsburg, Kansas, while he was in fourth grade. In»\n",
- "[2] «Gary Zukav | experience that led to his first book, \"Dancing Wu Li Masters\", written with extensive help from Jack Sarfatti and other physicists he met through Sarfatti as described in David Kaiser's book \"How the Hippies Saved Physics\". He later described this book as his \"first gift to Life\". In 1987 he moved to Mount Shasta, California, where he lived in a cabin as a self-described \"secular monk\" and spent extensive time in the surrounding wilderness. In 1993 he met Linda Francis. They co-founded the Seat of the Soul Institute in 1998 and moved to Ashland, Oregon, in 2000. Christopher Lehmann-Haupt reviewed»\n",
+ "[1] «1972 FA Charity Shield | The 1972 FA Charity Shield was contested between Manchester City and Aston Villa.»\n",
+ "[2] «1971 FA Charity Shield | The 1971 FA Charity Shield was a football match between Leicester City and Liverpool at Filbert Street on Saturday 7 August 1971.»\n",
"\n",
- "Question: Which award did the first book of Gary Zukav receive?\n",
+ "Question: In what year was the club founded that played Manchester City in the 1972 FA Charity Shield\n",
"\n",
- "Rationale: Let's think step by step. Based on the context, we have learned the following. Gary Zukav is an American spiritual teacher and the author of four consecutive New York Times Best Sellers. His first book, \"The Dancing Wu Li Masters\" was written with extensive help from Jack Sarfatti and other physicists.\n",
+ "Reasoning: Let's think step by step in order to produce the query. We know that the 1972 FA Charity Shield was contested between Manchester City and Aston Villa. To find out the year the club founded that played against Manchester City, we need to search for the founding year of Aston Villa.\n",
"\n",
- "Search Query: Which award did \"The Dancing Wu Li Masters\" receive?\n",
+ "Query: \"Aston Villa founding year\"\n",
"\n",
"---\n",
"\n",
"Context:\n",
- "[1] «John du Pont | John du Pont John Eleuthère du Pont (November 22, 1938 – December 9, 2010) was an American philanthropist, heir to the Du Pont family fortune, and a convicted murderer. He had been a published ornithologist, philatelist, conchologist, sports enthusiast, and self-styled wrestling coach. He died in prison while serving a sentence of 30 years for the murder of Dave Schultz. In 1972, du Pont founded and directed the Delaware Museum of Natural History and contributed to Villanova University and other institutions. In the 1980s, he established a wrestling facility at his Foxcatcher Farm after becoming interested in the sport and»\n",
- "[2] «Du Pont family | with political and business ventures, as well as philanthropic efforts on behalf of various organizations such as The Jockey Club, the Philadelphia Orchestra, the Delaware Art Museum, and the Grand Opera House. Two family members were the subjects of well-publicized criminal cases. Heir John Eleuthère du Pont was convicted of murdering wrestling coach Dave Schultz in 1996 and heir Robert H. Richards IV was convicted of raping his 3-year-old daughter in 2009. The family is depicted in the 2014 biographical film \"Foxcatcher\" with Steve Carell playing John Eleuthère du Pont and Vanessa Redgrave playing Jean du Pont, the wife of»\n",
+ "[1] «Empire State Building | The Empire State Building is a 102-story skyscraper located on Fifth Avenue between West 33rd and 34th Streets in Midtown, Manhattan, New York City. It has a roof height of 1,250 feet (381 m), and with its antenna included, it stands a total of 1454 ft tall. Its name is derived from the nickname for New York, the Empire State.»\n",
+ "[2] «Bank of America Plaza (Atlanta) | Bank of America Plaza is a skyscraper located in between Midtown Atlanta and Downtown Atlanta. At 312 m , the tower is the 96th-tallest building in the world. It is the 14th tallest building in the U.S., the tallest building in Georgia and the tallest building in any U.S. state capital, overtaking the 250 m (820 ft), 50 story One Atlantic Center in height, which previously held the record as Georgia's tallest building. It has 55 stories of office space and was completed in 1992, when it was called NationsBank Plaza. Originally intended to be the headquarters for Citizens & Southern National Bank (which merged with Sovran Bank during construction), it became NationsBank's property following its formation in the 1991 hostile takeover of C&S/Sovran by NCNB.»\n",
"\n",
- "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
+ "Question: Which is taller, the Empire State Building or the Bank of America Tower?\n",
"\n",
- "Rationale: Let's think step by step. Based on the context, we have learned the following. The Du Pont family is a wealthy American family with a long history of business and philanthropic ventures. In the 1980s, one of the heirs, John Eleuthère du Pont, became interested in wrestling and established a wrestling facility at his farm.\n",
+ "Reasoning: Let's think step by step in order to answer the question. We know that the Empire State Building has a roof height of 1,250 feet and a total height of 1,454 feet including its antenna. On the other hand, the Bank of America Plaza in Atlanta is 312 meters tall. To compare the heights of the two buildings, we need to convert the height of the Bank of America Plaza from meters to feet.\n",
"\n",
- "Search Query: Which wrestling team did John Eleuthère du Pont sponsor?\n",
+ "Query: \"Convert 312 meters to feet\"\n",
"\n",
"---\n",
"\n",
"Context:\n",
- "[1] «Peter Outerbridge | Peter Outerbridge Peter Outerbridge (born June 30, 1966) is a Canadian actor, best known for his role as Ari Tasarov in the CW action series \"Nikita\", Dr. David Sandström in the TMN series \"ReGenesis\", Henrik \"Hank\" Johanssen in \"Orphan Black\", Bob Corbett in \"Bomb Girls\", William Easton in \"Saw VI\" and George Brown in the television film \"\". He also played the lead role of Detective William Murdoch in a three-episode mini-series, \"The Murdoch Mysteries\", in its initial run on Canadian television, with two episodes shown in 2004 and a third in 2005. Outerbridge was born and raised in Toronto,»\n",
- "[2] «Except the Dying | 2008. On March 3, 2015, Acorn Media announced a re-release for all three movies, set for May 26, 2015. Except the Dying Except the Dying is a 2004 made-for-TV film starring Peter Outerbridge, Colm Meaney, Keeley Hawes, William B. Davis and Flora Montgomery. It was adapted by Janet MacLean from the novel of the same name by Maureen Jennings. In this film, William Murdoch is introduced, as a man of strong principles, who uses his unique abilities to solve crimes, sometimes using advanced science for his time. On the street of Toronto, in the 1890s, the naked body of a»\n",
+ "[1] «Aleksandr Danilovich Aleksandrov | Aleksandr Danilovich Aleksandrov (Russian: Алекса́ндр Дани́лович Алекса́ндров , alternative transliterations: \"Alexandr\" or \"Alexander\" (first name), and \"Alexandrov\" (last name)) (August 4, 1912 – July 27, 1999), was a Soviet/Russian mathematician, physicist, philosopher and mountaineer.»\n",
+ "[2] «Aleksandr Pavlovich Aleksandrov | Aleksandr Pavlovich Aleksandrov (Russian: Александр Павлович Александров ; born February 20, 1943) is a former Soviet cosmonaut and twice Hero of the Soviet Union (November 23, 1983 and December 29, 1987).»\n",
"\n",
- "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
+ "Question: Who is older, Aleksandr Danilovich Aleksandrov or Anatoly Fomenko?\n",
"\n",
- "Rationale: Let's think step by step. Based on the context, we have learned the following. Outerbridge was born and raised in Toronto, he is a Canadian actor, best known for his role as Ari Tasarov in the CW action series \"Nikita\", Dr. David Sandström in the TMN series \"ReGenesis\", Henrik \"Hank\" Johanssen in \"Orphan Black\", Bob Corbett in \"Bomb Girls\", William Easton in \"Saw VI\" and George Brown in the television film.\n",
+ "Reasoning: Let's think step by step in order to produce the query. We know that Aleksandr Danilovich Aleksandrov was born on August 4, 1912, but we don't have information about Anatoly Fomenko's birthdate. To find out who is older, we need to compare their birthdates.\n",
"\n",
- "Search Query: Who was the director of the 2009 movie \"Saw VI\"?\n",
+ "Query: \"Anatoly Fomenko birthdate\"\n",
"\n",
"---\n",
"\n",
"Context:\n",
- "[1] «Apache Hadoop | the Java programming language, with some native code in C and command line utilities written as shell scripts. Though MapReduce Java code is common, any programming language can be used with Hadoop Streaming to implement the map and reduce parts of the user's program. Other projects in the Hadoop ecosystem expose richer user interfaces. According to its co-founders, Doug Cutting and Mike Cafarella, the genesis of Hadoop was the Google File System paper that was published in October 2003. This paper spawned another one from Google \"MapReduce: Simplified Data Processing on Large Clusters\". Development started on the Apache Nutch project,»\n",
- "[2] «Doug Cutting | Doug Cutting Douglass Read Cutting is a software designer and advocate and creator of open-source search technology. He founded Lucene and, with Mike Cafarella, Nutch, both open-source search technology projects which are now managed through the Apache Software Foundation. Cutting and Cafarella are also the co-founders of Apache Hadoop. Cutting holds a bachelor's degree from Stanford University. Prior to developing Lucene, Cutting held search technology positions at Xerox PARC where he worked on the Scatter/Gather algorithm and on computational stylistics. He also worked at Excite, where he was one of the chief designers of the search engine, and Apple Inc.,»\n",
- "[3] «Doug Cutting | September 2010, he was elected its chairman. In 2015, O'Reilly awarded an open source award to Cutting. Doug Cutting Douglass Read Cutting is a software designer and advocate and creator of open-source search technology. He founded Lucene and, with Mike Cafarella, Nutch, both open-source search technology projects which are now managed through the Apache Software Foundation. Cutting and Cafarella are also the co-founders of Apache Hadoop. Cutting holds a bachelor's degree from Stanford University. Prior to developing Lucene, Cutting held search technology positions at Xerox PARC where he worked on the Scatter/Gather algorithm and on computational stylistics. He also worked»\n",
- "[4] «Apache Hadoop | but was moved to the new Hadoop subproject in January 2006. Doug Cutting, who was working at Yahoo! at the time, named it after his son's toy elephant. The initial code that was factored out of Nutch consisted of about 5,000 lines of code for HDFS and about 6,000 lines of code for MapReduce. The first committer to add to the Hadoop project was Owen O'Malley (in March 2006); Hadoop 0.1.0 was released in April 2006. It continues to evolve through contributions that are being made to the project. Hadoop consists of the \"Hadoop Common\" package, which provides file system»\n",
- "[5] «Doug Cutting | on the MapReduce algorithm, which allows very large scale computations to be trivially parallelized across large clusters of servers. Cutting and Cafarella, realizing the importance of this paper to extending Lucene into the realm of extremely large search problems, created the open-source Hadoop framework that allows applications based on the MapReduce paradigm to be run on large clusters of commodity hardware. Cutting was an employee of Yahoo!, where he led the Hadoop project full-time. He later went on to work for Cloudera. In July 2009, Cutting was elected to the board of directors of the Apache Software Foundation, and in»\n",
+ "[1] «David Gregory (physician) | David Gregory (20 December 1625 – 1720) was a Scottish physician and inventor. His surname is sometimes spelt as Gregorie, the original Scottish spelling. He inherited Kinnairdy Castle in 1664. Three of his twenty-nine children became mathematics professors. He is credited with inventing a military cannon that Isaac Newton described as \"being destructive to the human species\". Copies and details of the model no longer exist. Gregory's use of a barometer to predict farming-related weather conditions led him to be accused of witchcraft by Presbyterian ministers from Aberdeen, although he was never convicted.»\n",
+ "[2] «Gregory Tarchaneiotes | Gregory Tarchaneiotes (Greek: Γρηγόριος Ταρχανειώτης , Italian: \"Gregorio Tracanioto\" or \"Tracamoto\" ) was a \"protospatharius\" and the long-reigning catepan of Italy from 998 to 1006. In December 999, and again on February 2, 1002, he reinstituted and confirmed the possessions of the abbey and monks of Monte Cassino in Ascoli. In 1004, he fortified and expanded the castle of Dragonara on the Fortore. He gave it three circular towers and one square one. He also strengthened Lucera.»\n",
+ "[3] «Gregory Parsloe-Parsloe | Sir Gregory Parsloe-Parsloe, 7th Baronet is a fictional character from the Blandings stories of P. G. Wodehouse. The seventh Baronet, who resides at Matchingham Hall, he is the son of the Very Reverend Dean Parsloe-Parsloe and is the rival and enemy of Lord Emsworth, master of Blandings Castle.»\n",
"\n",
- "Question: When was the creator of Hadoop given an award?\n",
+ "Question: How many storeys are in the castle that David Gregory inherited?\n",
"\n",
- "Rationale: Let's think step by step. Based on the context, we have learned the following.\u001b[32m Apache Hadoop is an open-source software framework for distributed storage and distributed processing of big data sets on computer clusters. The creator of Hadoop is Doug Cutting. In 2015, O'Reilly awarded an open source award to Cutting.\n",
+ "Reasoning: Let's think step by step in order to\u001b[32m produce the query. We know that David Gregory inherited Kinnairdy Castle and that he had three of his twenty-nine children become mathematics professors. To find out how many storeys are in the castle, we need to search for information about Kinnairdy Castle.\n",
"\n",
- "Search Query: When was the creator of Hadoop given an award?\u001b[0m\u001b[31m \t (and 9 other completions)\u001b[0m\n",
+ "Query: \"Kinnairdy Castle number of storeys\"\u001b[0m\n",
"\n",
"\n",
"\n",
@@ -2047,100 +1845,123 @@
"\n",
"---\n",
"\n",
- "Question: Who produced the album that included a re-recording of \"Lithium\"?\n",
- "Answer: Butch Vig\n",
+ "Question: The Organisation that allows a community to influence their operation or use and to enjoy the benefits arisingwas founded in what year?\n",
+ "Answer: 2010\n",
+ "\n",
+ "Question: On the coast of what ocean is the birthplace of Diogal Sakho?\n",
+ "Answer: Atlantic\n",
+ "\n",
+ "Question: Which company distributed this 1977 American animated film produced by Walt Disney Productions for which Sherman Brothers wrote songs?\n",
+ "Answer: Buena Vista Distribution\n",
+ "\n",
+ "Question: Which magazine has published articles by Scott Shaw, Tae Kwon Do Times or Southwest Art?\n",
+ "Answer: Tae Kwon Do Times\n",
+ "\n",
+ "Question: Which American actress who made their film debut in the 1995 teen drama \"Kids\" was the co-founder of Voto Latino?\n",
+ "Answer: Rosario Dawson\n",
+ "\n",
+ "Question: Who acted in the shot film The Shore and is also the youngest actress ever to play Ophelia in a Royal Shakespeare Company production of \"Hamlet.\" ?\n",
+ "Answer: Kerry Condon\n",
+ "\n",
+ "Question: which American actor was Candace Kita guest starred with\n",
+ "Answer: Bill Murray\n",
+ "\n",
+ "Question: \"Everything Has Changed\" is a song from an album released under which record label ?\n",
+ "Answer: Big Machine Records\n",
"\n",
- "Question: What city was the victim of Joseph Druces working in?\n",
- "Answer: Boston, Massachusetts\n",
+ "Question: Tombstone stared an actor born May 17, 1955 known as who?\n",
+ "Answer: Bill Paxton\n",
"\n",
- "Question: In what year was the star of To Hell and Back born?\n",
- "Answer: 1925\n",
+ "Question: Which of these publications was most recently published, Who Put the Bomp or Self?\n",
+ "Answer: Self\n",
+ "\n",
+ "Question: What is the code name for the German offensive that started this Second World War engagement on the Eastern Front (a few hundred kilometers from Moscow) between Soviet and German forces, which included 102nd Infantry Division?\n",
+ "Answer: Operation Citadel\n",
+ "\n",
+ "Question: Samantha Cristoforetti and Mark Shuttleworth are both best known for being first in their field to go where?\n",
+ "Answer: space\n",
+ "\n",
+ "Question: Having the combination of excellent foot speed and bat speed helped Eric Davis, create what kind of outfield for the Los Angeles Dodgers?\n",
+ "Answer: \"Outfield of Dreams\"\n",
"\n",
"---\n",
"\n",
"Follow the following format.\n",
"\n",
- "Context:\n",
- "${sources that may contain relevant content}\n",
+ "Context: may contain relevant facts\n",
"\n",
- "Question: ${the question to be answered}\n",
+ "Question: ${question}\n",
"\n",
- "Rationale: Let's think step by step. ${a step-by-step deduction that identifies the correct response, which will be provided below}\n",
+ "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
"\n",
- "Answer: ${a short factoid answer, often between 1 and 5 words}\n",
+ "Answer: often between 1 and 5 words\n",
"\n",
"---\n",
"\n",
"Context:\n",
- "[1] «Gary Zukav | Gary Zukav Gary Zukav (born October 17, 1942) is an American spiritual teacher and the author of four consecutive New York Times Best Sellers. Beginning in 1998, he appeared more than 30 times on \"The Oprah Winfrey Show\" to discuss transformation in human consciousness concepts presented in his book \"The Seat of the Soul\". His first book, \"The Dancing Wu Li Masters\" (1979), won a U.S. National Book Award. Gary Zukav was born in Port Arthur, Texas, and spent his early childhood in San Antonio and Houston. His family moved to Pittsburg, Kansas, while he was in fourth grade. In»\n",
- "[2] «Gary Zukav | experience that led to his first book, \"Dancing Wu Li Masters\", written with extensive help from Jack Sarfatti and other physicists he met through Sarfatti as described in David Kaiser's book \"How the Hippies Saved Physics\". He later described this book as his \"first gift to Life\". In 1987 he moved to Mount Shasta, California, where he lived in a cabin as a self-described \"secular monk\" and spent extensive time in the surrounding wilderness. In 1993 he met Linda Francis. They co-founded the Seat of the Soul Institute in 1998 and moved to Ashland, Oregon, in 2000. Christopher Lehmann-Haupt reviewed»\n",
- "[3] «The Dancing Wu Li Masters | The Dancing Wu Li Masters The Dancing Wu Li Masters is a 1979 book by Gary Zukav, a popular science work exploring modern physics, and quantum phenomena in particular. It was awarded a 1980 U.S. National Book Award in category of Science. Although it explores empirical topics in modern physics research, \"The Dancing Wu Li Masters\" gained attention for leveraging metaphors taken from eastern spiritual movements, in particular the Huayen school of Buddhism with the monk Fazang's treatise on The Golden Lion, to explain quantum phenomena and has been regarded by some reviewers as a New Age work, although the»\n",
- "[4] «The Dancing Wu Li Masters | a 1976 physics conference of eastern and western scientists at Esalen Institute, California; and he used the occasion as material for his book. At the conference, it was said that the Chinese term for physics is 'Wu Li', or \"patterns of organic energy.\" Zukav, among others, conceptualized 'physics' as the dance of the Wu Li Masters – teachers of physical essence. Zukav explains the concept further: The Dancing Wu Li Masters The Dancing Wu Li Masters is a 1979 book by Gary Zukav, a popular science work exploring modern physics, and quantum phenomena in particular. It was awarded a 1980»\n",
+ "[1] «1972 FA Charity Shield | The 1972 FA Charity Shield was contested between Manchester City and Aston Villa.»\n",
+ "[2] «1971 FA Charity Shield | The 1971 FA Charity Shield was a football match between Leicester City and Liverpool at Filbert Street on Saturday 7 August 1971.»\n",
+ "[3] «1896–97 Aston Villa F.C. season | The 1896-87 season was Aston Villa's ninth season in the Football League since being one of its 12 founding members in 1888.»\n",
+ "[4] «List of Aston Villa F.C. seasons | This is a list of seasons played by Aston Villa Football Club in English and European football, from 1879 (the year of the club's first FA Cup entry) to the most recent completed season. Aston Villa football club was founded in March, 1874, by members of the Villa Cross Wesleyan Chapel in Aston. Throughout the 1870s Aston Villa played a small amount of games. At least one game, against Aston Brook St Mary's was played with one half under Rugby rules and the other under football rules. In the 1880s the game became more formalised and in 1888, William McGregor formed the Football League with 11 other clubs.»\n",
"\n",
- "Question: Which award did the first book of Gary Zukav receive?\n",
+ "Question: In what year was the club founded that played Manchester City in the 1972 FA Charity Shield\n",
"\n",
- "Rationale: Let's think step by step. The first book is \"The Dancing Wu Li Masters\" and it was awarded a U.S. National Book Award.\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the 1972 FA Charity Shield was contested between Manchester City and Aston Villa. According to the context, Aston Villa Football Club was founded in March, 1874.\n",
"\n",
- "Answer: U.S. National Book Award\n",
+ "Answer: 1874\n",
"\n",
"---\n",
"\n",
"Context:\n",
- "[1] «John du Pont | John du Pont John Eleuthère du Pont (November 22, 1938 – December 9, 2010) was an American philanthropist, heir to the Du Pont family fortune, and a convicted murderer. He had been a published ornithologist, philatelist, conchologist, sports enthusiast, and self-styled wrestling coach. He died in prison while serving a sentence of 30 years for the murder of Dave Schultz. In 1972, du Pont founded and directed the Delaware Museum of Natural History and contributed to Villanova University and other institutions. In the 1980s, he established a wrestling facility at his Foxcatcher Farm after becoming interested in the sport and»\n",
- "[2] «Du Pont family | with political and business ventures, as well as philanthropic efforts on behalf of various organizations such as The Jockey Club, the Philadelphia Orchestra, the Delaware Art Museum, and the Grand Opera House. Two family members were the subjects of well-publicized criminal cases. Heir John Eleuthère du Pont was convicted of murdering wrestling coach Dave Schultz in 1996 and heir Robert H. Richards IV was convicted of raping his 3-year-old daughter in 2009. The family is depicted in the 2014 biographical film \"Foxcatcher\" with Steve Carell playing John Eleuthère du Pont and Vanessa Redgrave playing Jean du Pont, the wife of»\n",
- "[3] «John du Pont | Foxcatcher team. Du Pont became a sponsor in wrestling, swimming, track, and the modern pentathlon. He was also involved in promoting a subset of the modern pentathlon (run, swim, shoot) as a separate event. He took up athletics and became a competitive wrestler in his 50s. His only prior wrestling experience was as a freshman in high school. He began competing again at the age of 55 in the 1992 Veteran's World Championships in Cali, Colombia; following that in 1993 in Toronto, Ontario; in 1994 in Rome, Italy; and in 1995 in Sofia, Bulgaria. In August 1988, a problem-plagued wrestling»\n",
+ "[1] «Empire State Building | The Empire State Building is a 102-story skyscraper located on Fifth Avenue between West 33rd and 34th Streets in Midtown, Manhattan, New York City. It has a roof height of 1,250 feet (381 m), and with its antenna included, it stands a total of 1454 ft tall. Its name is derived from the nickname for New York, the Empire State.»\n",
+ "[2] «Bank of America Plaza (Atlanta) | Bank of America Plaza is a skyscraper located in between Midtown Atlanta and Downtown Atlanta. At 312 m , the tower is the 96th-tallest building in the world. It is the 14th tallest building in the U.S., the tallest building in Georgia and the tallest building in any U.S. state capital, overtaking the 250 m (820 ft), 50 story One Atlantic Center in height, which previously held the record as Georgia's tallest building. It has 55 stories of office space and was completed in 1992, when it was called NationsBank Plaza. Originally intended to be the headquarters for Citizens & Southern National Bank (which merged with Sovran Bank during construction), it became NationsBank's property following its formation in the 1991 hostile takeover of C&S/Sovran by NCNB.»\n",
+ "[3] «Foot (unit) | The foot ( feet; abbreviation: ft; symbol: ′, the prime symbol) is a unit of length in the imperial and US customary systems of measurement. Since 1959, both units have been defined by international agreement as equivalent to 0.3048 meters exactly. In both systems, the foot comprises 12 inches and three feet compose a yard.»\n",
+ "[4] «Pentameter | Pentameter (from Greek: πεντάμετρος - 'measuring five (feet)') is a poetic meter. А poem is said to be written in a particular pentameter when the lines of the poem have the length of five feet, where 'foot' is a combination of a particular number (1 or 2) of unstressed (or weak) syllables and a stressed (or strong) syllable. Depending on the pattern of feet, pentameter can be iambic (one of three two-syllable meters alongside trochaic and spondaic) or dactylic (one of two three-syllable meters alongside anapestic) (see links below).»\n",
"\n",
- "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
+ "Question: Which is taller, the Empire State Building or the Bank of America Tower?\n",
"\n",
- "Rationale: Let's think step by step. The question is asking for a wrestling team that the heir to the Du Pont family sponsored. In the context, it is mentioned that the heir, John du Pont, was interested in wrestling and even competed in wrestling competitions in his 50s. Furthermore, it is stated that he sponsored a wrestling team. Therefore, the answer must be a wrestling team.\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that the Empire State Building has a roof height of 1,250 feet and with its antenna included, it stands a total of 1,454 feet tall. On the other hand, the Bank of America Plaza in Atlanta is 312 meters tall, which is approximately 1,024 feet. Therefore, the Empire State Building is taller than the Bank of America Plaza.\n",
"\n",
- "Answer: Foxcatcher team\n",
+ "Answer: Empire State Building\n",
"\n",
"---\n",
"\n",
"Context:\n",
- "[1] «Peter Outerbridge | Peter Outerbridge Peter Outerbridge (born June 30, 1966) is a Canadian actor, best known for his role as Ari Tasarov in the CW action series \"Nikita\", Dr. David Sandström in the TMN series \"ReGenesis\", Henrik \"Hank\" Johanssen in \"Orphan Black\", Bob Corbett in \"Bomb Girls\", William Easton in \"Saw VI\" and George Brown in the television film \"\". He also played the lead role of Detective William Murdoch in a three-episode mini-series, \"The Murdoch Mysteries\", in its initial run on Canadian television, with two episodes shown in 2004 and a third in 2005. Outerbridge was born and raised in Toronto,»\n",
- "[2] «Except the Dying | 2008. On March 3, 2015, Acorn Media announced a re-release for all three movies, set for May 26, 2015. Except the Dying Except the Dying is a 2004 made-for-TV film starring Peter Outerbridge, Colm Meaney, Keeley Hawes, William B. Davis and Flora Montgomery. It was adapted by Janet MacLean from the novel of the same name by Maureen Jennings. In this film, William Murdoch is introduced, as a man of strong principles, who uses his unique abilities to solve crimes, sometimes using advanced science for his time. On the street of Toronto, in the 1890s, the naked body of a»\n",
- "[3] «Saw VI | Saw VI Saw VI is a 2009 American horror film directed by Kevin Greutert from a screenplay written by Patrick Melton and Marcus Dunstan. It is the sixth installment in the \"Saw\" franchise and stars Tobin Bell, Costas Mandylor, Betsy Russell, Mark Rolston, Peter Outerbridge, and Shawnee Smith. It was produced by Mark Burg and Oren Koules of Twisted Pictures and distributed by Lionsgate. \"Saw VI\" continues the franchise's focus on the posthumous effects of the Jigsaw Killer and the progression of his successor, Mark Hoffman. In this film, Hoffman sets a series of traps for an insurance executive, William»\n",
- "[4] «Saw VI | is a crafty move on the screenwriters' part, but once you get past that pseudo-ironic touch, this Saw is no more or less disturbing than any other in the series\". Saw VI Saw VI is a 2009 American horror film directed by Kevin Greutert from a screenplay written by Patrick Melton and Marcus Dunstan. It is the sixth installment in the \"Saw\" franchise and stars Tobin Bell, Costas Mandylor, Betsy Russell, Mark Rolston, Peter Outerbridge, and Shawnee Smith. It was produced by Mark Burg and Oren Koules of Twisted Pictures and distributed by Lionsgate. \"Saw VI\" continues the franchise's focus»\n",
+ "[1] «Aleksandr Danilovich Aleksandrov | Aleksandr Danilovich Aleksandrov (Russian: Алекса́ндр Дани́лович Алекса́ндров , alternative transliterations: \"Alexandr\" or \"Alexander\" (first name), and \"Alexandrov\" (last name)) (August 4, 1912 – July 27, 1999), was a Soviet/Russian mathematician, physicist, philosopher and mountaineer.»\n",
+ "[2] «Aleksandr Pavlovich Aleksandrov | Aleksandr Pavlovich Aleksandrov (Russian: Александр Павлович Александров ; born February 20, 1943) is a former Soviet cosmonaut and twice Hero of the Soviet Union (November 23, 1983 and December 29, 1987).»\n",
+ "[3] «Anatoly Fomenko | Anatoly Timofeevich Fomenko (Russian: Анато́лий Тимофе́евич Фоме́нко ) (born 13 March 1945 in Stalino, USSR) is a Soviet and Russian mathematician, professor at Moscow State University, well known as a topologist, and a member of the Russian Academy of Sciences. He is author of a pseudoscientific theory known as New Chronology. He is also a member of the Russian Academy of Natural Sciences (1991).»\n",
+ "[4] «Semyon Fomin | Semyon Anatolyevich Fomin (Russian: Семён Анатольевич Фомин ; born 10 January 1989) is a Russian professional footballer. He plays as a midfielder for FC Luch-Energiya Vladivostok.»\n",
"\n",
- "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
+ "Question: Who is older, Aleksandr Danilovich Aleksandrov or Anatoly Fomenko?\n",
"\n",
- "Rationale: Let's think step by step. The question is asking for the director of the 2009 movie that features Peter Outerbridge as William Easton. We can use process of elimination to figure this out. In source 1, there is no mention of a 2009 movie. In source 2, the only mention of Peter Outerbridge is in the cast list for a 2004 made-for-TV film. In source 3, we see that Outerbridge was in a 2009 movie, Saw VI, and that it was directed by Kevin Greutert. This must be the correct answer, so we can check source 4 to confirm. In source 4, we see that Kevin Greutert is in fact the director of Saw VI.\n",
+ "Reasoning: Let's think step by step in order to produce the answer. We know that Aleksandr Danilovich Aleksandrov was born on August 4, 1912, and Anatoly Fomenko was born on March 13, 1945. Therefore, Aleksandr Danilovich Aleksandrov is older.\n",
"\n",
- "Answer: Kevin Greutert\n",
+ "Answer: Aleksandr Danilovich Aleksandrov\n",
"\n",
"---\n",
"\n",
"Context:\n",
- "[1] «Apache Hadoop is an open-source software framework for distributed storage and distributed processing of big data sets on computer clusters. The creator of Hadoop is Doug Cutting. In 2015, O'Reilly awarded an open source award to Cutting.»\n",
- "[2] «Doug Cutting | September 2010, he was elected its chairman. In 2015, O'Reilly awarded an open source award to Cutting. Doug Cutting Douglass Read Cutting is a software designer and advocate and creator of open-source search technology. He founded Lucene and, with Mike Cafarella, Nutch, both open-source search technology projects which are now managed through the Apache Software Foundation. Cutting and Cafarella are also the co-founders of Apache Hadoop. Cutting holds a bachelor's degree from Stanford University. Prior to developing Lucene, Cutting held search technology positions at Xerox PARC where he worked on the Scatter/Gather algorithm and on computational stylistics. He also worked»\n",
- "[3] «Anoop Swarup | 1984. He initiated the UNDP-NYKS Poverty Alleviation Awards in 2000, was instrumental in launching a ‘National Think Tank’ in the Ministry of Finance in 2004 and campaign on ‘Indian Civilization through the Millennium’ (ICTM) in 2008 now an ongoing yearly event at Hastinapur. He founded the ‘Sambhav’ Trust in the Indian State of Rajasthan for de addiction in 2008. Anoop Swarup is associated with various organizations that include Board of Directors of OISCA Organization for Industrial, Spiritual and Cultural Advancement International, Tokyo Japan, United Nations Inter Agency working group on Population and Development as part of United Nations Development Assistance»\n",
- "[4] «Apache Hadoop | but was moved to the new Hadoop subproject in January 2006. Doug Cutting, who was working at Yahoo! at the time, named it after his son's toy elephant. The initial code that was factored out of Nutch consisted of about 5,000 lines of code for HDFS and about 6,000 lines of code for MapReduce. The first committer to add to the Hadoop project was Owen O'Malley (in March 2006); Hadoop 0.1.0 was released in April 2006. It continues to evolve through contributions that are being made to the project. Hadoop consists of the \"Hadoop Common\" package, which provides file system»\n",
- "[5] «Apache SystemML | November 2, 2015. On May 17, 2017, the Apache Software Foundation Board approved the graduation of Apache SystemML as an Apache Top Level Project. Apache SystemML Apache SystemML is a flexible machine learning system that automatically scales to Spark and Hadoop clusters. SystemML's distinguishing characteristics are: SystemML was created in 2010 by researchers at the IBM Almaden Research Center led by IBM Fellow Shivakumar Vaithyanathan. It was observed that data scientists would write machine learning algorithms in languages such as R and Python for small data. When it came time to scale to big data, a systems programmer would be»\n",
- "[6] «Matei Zaharia | the Apache Mesos project. He also designed and implemented the core scheduling algorithms used in Apache Hadoop. He received two Best Paper awards at NSDI 2012 and SIGCOMM 2012, Honorable Mention for Community Award at NSDI 2012, and a Best Demo Award at SIGMOD 2012. Jointly with Reynold Xin, Parviz Deyhim, Xiangrui Meng, and Ali Ghodsi, he holds the 2014 world record in Daytona GraySort using Apache Spark. In 2015 he received the ACM Doctoral Dissertation Award. Matei Zaharia Matei Zaharia is a Romanian-Canadian computer scientist specializing in big data, distributed systems, and cloud computing. He is a co-founder and»\n",
+ "[1] «David Gregory (physician) | David Gregory (20 December 1625 – 1720) was a Scottish physician and inventor. His surname is sometimes spelt as Gregorie, the original Scottish spelling. He inherited Kinnairdy Castle in 1664. Three of his twenty-nine children became mathematics professors. He is credited with inventing a military cannon that Isaac Newton described as \"being destructive to the human species\". Copies and details of the model no longer exist. Gregory's use of a barometer to predict farming-related weather conditions led him to be accused of witchcraft by Presbyterian ministers from Aberdeen, although he was never convicted.»\n",
+ "[2] «Gregory Tarchaneiotes | Gregory Tarchaneiotes (Greek: Γρηγόριος Ταρχανειώτης , Italian: \"Gregorio Tracanioto\" or \"Tracamoto\" ) was a \"protospatharius\" and the long-reigning catepan of Italy from 998 to 1006. In December 999, and again on February 2, 1002, he reinstituted and confirmed the possessions of the abbey and monks of Monte Cassino in Ascoli. In 1004, he fortified and expanded the castle of Dragonara on the Fortore. He gave it three circular towers and one square one. He also strengthened Lucera.»\n",
+ "[3] «Gregory Parsloe-Parsloe | Sir Gregory Parsloe-Parsloe, 7th Baronet is a fictional character from the Blandings stories of P. G. Wodehouse. The seventh Baronet, who resides at Matchingham Hall, he is the son of the Very Reverend Dean Parsloe-Parsloe and is the rival and enemy of Lord Emsworth, master of Blandings Castle.»\n",
+ "[4] «Kinnairdy Castle | Kinnairdy Castle is a tower house, having five storeys and a garret, two miles south of Aberchirder, Aberdeenshire, Scotland. The alternative name is Old Kinnairdy.»\n",
+ "[5] «Kinnaird Head | Kinnaird Head (Scottish Gaelic: \"An Ceann Àrd\" , \"high headland\") is a headland projecting into the North Sea, within the town of Fraserburgh, Aberdeenshire on the east coast of Scotland. The 16th-century Kinnaird Castle was converted in 1787 for use as the Kinnaird Head Lighthouse, the first lighthouse in Scotland to be lit by the Commissioners of Northern Lights. Kinnaird Castle and the nearby Winetower were described by W. Douglas Simpson as two of the nine castles of the Knuckle, referring to the rocky headland of north-east Aberdeenshire. Both buildings are category A listed buildings.»\n",
+ "[6] «Kinnaird Castle, Brechin | Kinnaird Castle is a 15th-century castle in Angus, Scotland. The castle has been home to the Carnegie family, the Earl of Southesk, for more than 600 years.»\n",
"\n",
- "Question: When was the creator of Hadoop given an award?\n",
+ "Question: How many storeys are in the castle that David Gregory inherited?\n",
"\n",
- "Rationale: Let's think step by step.\u001b[32m The question is asking for the year when the creator of Hadoop was given an award. In source 1, we see that the creator of Hadoop is Doug Cutting and that in 2015, O'Reilly awarded an open source award to Cutting. Therefore, we can infer that the answer is 2015. We can check sources 2 and 3 to see if there is any mention of an award given to Doug Cutting in 2015, but since there is no mention, we can assume that source 1 is correct and the answer is 2015.\u001b[0m\u001b[31m \t (and 19 other completions)\u001b[0m\n",
+ "Reasoning: Let's think step by step in order to\u001b[32m produce the answer. We know that David Gregory inherited Kinnairdy Castle. According to the context, Kinnairdy Castle is a tower house with five storeys and a garret.\n",
+ "\n",
+ "Answer: Five\u001b[0m\n",
"\n",
"\n",
"\n"
]
- },
- {
- "data": {
- "text/plain": [
- "('2015', None)"
- ]
- },
- "execution_count": 27,
- "metadata": {},
- "output_type": "execute_result"
}
],
"source": [
- "multihop_QA_v2(\"When was the creator of Hadoop given an award?\"), lm.inspect_history(n=3)"
+ "compiled_baleen(\"How many storeys are in the castle that David Gregory inherited?\")\n",
+ "turbo.inspect_history(n=3)"
]
},
{
@@ -2149,113 +1970,11 @@
"metadata": {},
"outputs": [],
"source": []
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Additional Programs...\n",
- "\n",
- "While Program 5 is undeniably better than, say, Program 1, it's far from perfect.\n",
- "\n",
- "Many different ideas can easily be implemented to make it more general or more powerful. Refer to the DSP paper for inspiration!\n",
- "\n",
- "One of your first improvements to Programs 4 and 5 would be to allow it to do more than two hops and to automatically stop when additional hops are not required.\n",
- "\n",
- "**Hint:** This is pretty easy to do. You can start with Program 4 and modify `hop_template` so its `SearchQuery` produces the string `N/A` when additional hops are not required.\n",
- "\n",
- "In the `multihop_search_v1` function, you can then break from the loop — i.e., `if completions.query == 'N/A': break` whenever this happens. How will you apply this to Program 5?\n",
- "\n",
- "```\n",
- "SearchQueryWithStopping = dsp.Type(\n",
- " prefix=\"Search Query:\",\n",
- " desc=\"${a simple question for seeking the missing or required information --- say N/A if the context above contains all of the required information}\"\n",
- ")\n",
- "```"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Other incremental improvements will likely help program 5, including keeping the top passage from each hop in the context and/or the summary of each hop. To decide which changes to make, you can inspect the trace of the program when it makes mistakes!"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 28,
- "metadata": {},
- "outputs": [],
- "source": [
- "# Program 4 with max_hops=4 and automatic stopping."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# Additional Capabilities"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## more LLMs for use\n",
- "Eventhough the most quality tested model is openai text-davinci, we also offer support for `openai chatgpt, cohere, and hf models`.\n",
- "\n",
- "`Note:` The quality and stability might be different based on the provider and the model. Please report any bugs that you come across."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# initializing chatgpt\n",
- "\n",
- "lm = dsp.GPT3(model='gpt-3.5-turbo', api_key=openai_key, model_type='chat')\n",
- "\n",
- "# initializing cohere\n",
- "\n",
- "lm = dsp.Cohere(model='command-xlarge-nightly')\n",
- "\n",
- "# initializing hf models\n",
- "\n",
- "lm = dsp.HFModel(model=\"google/flan-t5-base\")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Re-Ranker\n",
- "\n",
- "You could also construct powerful retrieval programs that uses both `retriever and re-ranker model`.\n",
- "\n",
- "Currently, we support `Sentence-Transformers cross-encoder` for reranking, and an example initialisation and usage is provided below"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "sbert_reranker = dsp.SentenceTransformersCrossEncoder(\"cross-encoder/ms-marco-MiniLM-L-12-v2\")\n",
- "\n",
- "dsp.settings.configure(lm=lm, rm=rm, reranker=sbert_reranker)\n",
- "\n",
- "# dsp.retrieve, and retrieveEnsemble already takes care of retrieving and re-ranking based on initiated rm, and reranker.\n",
- "\n",
- "reranked_passages = dsp.retrieve(\"When was the creator of Hadoop given an award?\", k=3)"
- ]
}
],
"metadata": {
"kernelspec": {
- "display_name": "Python 3 (ipykernel)",
+ "display_name": "py39",
"language": "python",
"name": "python3"
},
@@ -2269,21 +1988,10 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
- "version": "3.9.4"
- },
- "vscode": {
- "interpreter": {
- "hash": "de72ef8826336f005e74d4a3f2b2dfb9239c3f097fd15aff69f3f6a4fc3bd8b8"
- }
- },
- "widgets": {
- "application/vnd.jupyter.widget-state+json": {
- "state": {},
- "version_major": 2,
- "version_minor": 0
- }
- }
+ "version": "3.9.17"
+ },
+ "orig_nbformat": 4
},
"nbformat": 4,
- "nbformat_minor": 4
+ "nbformat_minor": 2
}
diff --git a/setup.py b/setup.py
index ec76b448e..09a836deb 100644
--- a/setup.py
+++ b/setup.py
@@ -2,13 +2,13 @@ from setuptools import setup, find_packages
setup(
name="dspy-ai",
- version="0.1.9",
- description="Demonstrate-Search-Predict",
+ version="2.0.0-alpha",
+ description="DSPy",
url="https://github.com/stanfordnlp/dsp",
author="Omar Khattab",
author_email="[email protected]",
license="MIT License",
- packages=find_packages(),
+ packages=find_packages(include=['dsp', 'dspy']),
python_requires='>=3.9',
install_requires=[
"backoff",