An LLM-powered knowledge curation system that researches a topic and generates a full-length report with citations.
Python agentic-ragdeep-researchemnlp2024knowledge-curationlarge-language-modelsnaaclnlpreport-generationretrieval-augmented-generation
bump up version; add encoder class document; update readme
Yucheng-Jiangfaff1f7f3b36
diff --git a/README.md b/README.mdindex c6263c2..7fb8106 100644--- a/README.md+++ b/README.md@@ -7,9 +7,10 @@<p align="center">| <a href="http://storm.genie.stanford.edu"><b>Research preview</b></a> | <a href="https://arxiv.org/abs/2402.14207"><b>STORM Paper</b></a>| <a href="https://www.arxiv.org/abs/2408.15232"><b>Co-STORM Paper</b></a> | <a href="https://storm-project.stanford.edu/"><b>Website</b></a> |</p>-**Latest News** 🔥+- [2025/01] We add [litellm](https://github.com/BerriAI/litellm) integration for language models and embedding models+- [2024/09] Co-STORM codebase is now released and integrated into `knowledge-storm` python package v1.0.0. Run `pip install knowledge-storm --upgrade` to check it out.- [2024/09] We introduce collaborative STORM (Co-STORM) to support human-AI collaborative knowledge curation! [Co-STORM Paper](https://www.arxiv.org/abs/2408.15232) has been accepted to EMNLP 2024 main conference.@@ -92,10 +93,11 @@ You could also install the source code which allows you to modify the behavior oCurrently, our package support:-- `OpenAIModel`, `AzureOpenAIModel`, `ClaudeModel`, `VLLMClient`, `TGIClient`, `TogetherClient`, `OllamaClient`, `GoogleModel`, `DeepSeekModel`, `GroqModel` as language model components-- `YouRM`, `BingSearch`, `VectorRM`, `SerperRM`, `BraveRM`, `SearXNG`, `DuckDuckGoSearchRM`, `TavilySearchRM`, `GoogleSearch`, and `AzureAISearch` as retrieval module components+- Language model components: All language models supported by litellm as listed [here](https://docs.litellm.ai/docs/providers)+- Embedding model components: All embedding models supported by litellm as listed [here](https://docs.litellm.ai/docs/embedding/supported_embedding)+- retrieval module components: `YouRM`, `BingSearch`, `VectorRM`, `SerperRM`, `BraveRM`, `SearXNG`, `DuckDuckGoSearchRM`, `TavilySearchRM`, `GoogleSearch`, and `AzureAISearch` as-:star2: **PRs for integrating more language models into [knowledge_storm/lm.py](knowledge_storm/lm.py) and search engines/retrievers into [knowledge_storm/rm.py](knowledge_storm/rm.py) are highly appreciated!**+:star2: **PRs for integrating more search engines/retrievers into [knowledge_storm/rm.py](knowledge_storm/rm.py) are highly appreciated!**Both STORM and Co-STORM are working in the information curation layer, you need to set up the information retrieval module and language model module to create their `Runner` classes respectively.@@ -106,7 +108,7 @@ The STORM knowledge curation engine is defined as a simple Python `STORMWikiRunn```pythonimport osfrom knowledge_storm import STORMWikiRunnerArguments, STORMWikiRunner, STORMWikiLMConfigs-from knowledge_storm.lm import OpenAIModel+from knowledge_storm.lm import LitellmModelfrom knowledge_storm.rm import YouRMlm_configs = STORMWikiLMConfigs()@@ -118,8 +120,8 @@ openai_kwargs = {# STORM is a LM system so different components can be powered by different models to reach a good balance between cost and quality.# For a good practice, choose a cheaper/faster model for `conv_simulator_lm` which is used to split queries, synthesize answers in the conversation.# Choose a more powerful model for `article_gen_lm` to generate verifiable text with citations.-gpt_35 = OpenAIModel(model='gpt-3.5-turbo', max_tokens=500, **openai_kwargs)-gpt_4 = OpenAIModel(model='gpt-4o', max_tokens=3000, **openai_kwargs)+gpt_35 = LitellmModel(model='gpt-3.5-turbo', max_tokens=500, **openai_kwargs)+gpt_4 = LitellmModel(model='gpt-4o', max_tokens=3000, **openai_kwargs)lm_configs.set_conv_simulator_lm(gpt_35)lm_configs.set_question_asker_lm(gpt_35)lm_configs.set_outline_gen_lm(gpt_4)@@ -155,7 +157,7 @@ The Co-STORM knowledge curation engine is defined as a simple Python `CoStormRun```pythonfrom knowledge_storm.collaborative_storm.engine import CollaborativeStormLMConfigs, RunnerArgument, CoStormRunner-from knowledge_storm.lm import OpenAIModel+from knowledge_storm.lm import LitellmModelfrom knowledge_storm.logging_wrapper import LoggingWrapperfrom knowledge_storm.rm import BingSearch@@ -168,12 +170,12 @@ openai_kwargs = {"top_p": 0.9,"api_base": None,}-question_answering_lm = OpenAIModel(model=gpt_4o_model_name, max_tokens=1000, **openai_kwargs)-discourse_manage_lm = OpenAIModel(model=gpt_4o_model_name, max_tokens=500, **openai_kwargs)-utterance_polishing_lm = OpenAIModel(model=gpt_4o_model_name, max_tokens=2000, **openai_kwargs)-warmstart_outline_gen_lm = OpenAIModel(model=gpt_4o_model_name, max_tokens=500, **openai_kwargs)-question_asking_lm = OpenAIModel(model=gpt_4o_model_name, max_tokens=300, **openai_kwargs)-knowledge_base_lm = OpenAIModel(model=gpt_4o_model_name, max_tokens=1000, **openai_kwargs)+question_answering_lm = LitellmModel(model=gpt_4o_model_name, max_tokens=1000, **openai_kwargs)+discourse_manage_lm = LitellmModel(model=gpt_4o_model_name, max_tokens=500, **openai_kwargs)+utterance_polishing_lm = LitellmModel(model=gpt_4o_model_name, max_tokens=2000, **openai_kwargs)+warmstart_outline_gen_lm = LitellmModel(model=gpt_4o_model_name, max_tokens=500, **openai_kwargs)+question_asking_lm = LitellmModel(model=gpt_4o_model_name, max_tokens=300, **openai_kwargs)+knowledge_base_lm = LitellmModel(model=gpt_4o_model_name, max_tokens=1000, **openai_kwargs)lm_config.set_question_answering_lm(question_answering_lm)lm_config.set_discourse_manage_lm(discourse_manage_lm)@@ -222,6 +224,7 @@ We provide scripts in our [examples folder](examples) as a quick start to run STWe suggest using `secrets.toml` to set up the API keys. Create a file `secrets.toml` under the root directory and add the following content:```shell+# ============ language model configurations ============# Set up OpenAI API key.OPENAI_API_KEY="your_openai_api_key"# If you are using the API service provided by OpenAI, include the following line:@@ -230,15 +233,10 @@ OPENAI_API_TYPE="openai"OPENAI_API_TYPE="azure"AZURE_API_BASE="your_azure_api_base_url"AZURE_API_VERSION="your_azure_api_version"-# Set up You.com search API key.-YDC_API_KEY="your_youcom_api_key"-```--for **Co-STORM**, please also add following-```-# if use openai encoder-ENCODER_API_TYPE="openai"-# or ENCODER_API_TYPE="azure" if use azure openai encoder+# ============ retriever configurations ============+BING_SEARCH_API_KEY="your_bing_search_api_key" # if using bing search+# ============ encoder configurations ============+ENCODER_API_TYPE="openai" # if using openai encoder```### STORM examples@@ -249,7 +247,7 @@ Run the following command.```bashpython examples/storm_examples/run_storm_wiki_gpt.py \--output-dir $OUTPUT_DIR \- --retriever you \+ --retriever bing \--do-research \--do-generate-outline \--do-generate-article \@@ -328,20 +326,44 @@ We are very grateful to [Michelle Lam](https://michelle123lam.github.io/) for de## CitationPlease cite our paper if you use this code or part of it in your work:```bibtex-@misc{jiang2024unknownunknowns,- title={Into the Unknown Unknowns: Engaged Human Learning through Participation in Language Model Agent Conversations},- author={Yucheng Jiang and Yijia Shao and Dekun Ma and Sina J. Semnani and Monica S. Lam},- year={2024},- eprint={2408.15232},- archivePrefix={arXiv},- primaryClass={cs.CL},- url={https://arxiv.org/abs/2408.15232},+@inproceedings{jiang-etal-2024-unknown,+ title = "Into the Unknown Unknowns: Engaged Human Learning through Participation in Language Model Agent Conversations",+ author = "Jiang, Yucheng and+ Shao, Yijia and+ Ma, Dekun and+ Semnani, Sina and+ Lam, Monica",+ editor = "Al-Onaizan, Yaser and+ Bansal, Mohit and+ Chen, Yun-Nung",+ booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing",+ month = nov,+ year = "2024",+ address = "Miami, Florida, USA",+ publisher = "Association for Computational Linguistics",+ url = "https://aclanthology.org/2024.emnlp-main.554/",+ doi = "10.18653/v1/2024.emnlp-main.554",+ pages = "9917--9955",}-@inproceedings{shao2024assisting,- title={{Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models}},- author={Yijia Shao and Yucheng Jiang and Theodore A. Kanell and Peter Xu and Omar Khattab and Monica S. Lam},- year={2024},- booktitle={Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)}+@inproceedings{shao-etal-2024-assisting,+ title = "Assisting in Writing {W}ikipedia-like Articles From Scratch with Large Language Models",+ author = "Shao, Yijia and+ Jiang, Yucheng and+ Kanell, Theodore and+ Xu, Peter and+ Khattab, Omar and+ Lam, Monica",+ editor = "Duh, Kevin and+ Gomez, Helena and+ Bethard, Steven",+ booktitle = "Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)",+ month = jun,+ year = "2024",+ address = "Mexico City, Mexico",+ publisher = "Association for Computational Linguistics",+ url = "https://aclanthology.org/2024.naacl-long.347/",+ doi = "10.18653/v1/2024.naacl-long.347",+ pages = "6252--6278",}```diff --git a/knowledge_storm/__init__.py b/knowledge_storm/__init__.pyindex 1033895..8df0bb0 100644--- a/knowledge_storm/__init__.py+++ b/knowledge_storm/__init__.py@@ -7,4 +7,4 @@ from .rm import *from .utils import *from .dataclass import *-__version__ = "1.0.1"+__version__ = "1.1.1"diff --git a/knowledge_storm/encoder.py b/knowledge_storm/encoder.pyindex 0283a70..601da99 100644--- a/knowledge_storm/encoder.py+++ b/knowledge_storm/encoder.py@@ -34,6 +34,25 @@ except ImportError:class Encoder:+ """+ A wrapper class for the LiteLLM embedding model, designed to handle embedding+ generation tasks efficiently. It supports parallel processing and local caching of+ embedding results for improved performance.++ The Encoder utilizes the LiteLLM library to interact with various embedding models,+ such as OpenAI and Azure embeddings. Users can specify the desired encoder type and+ provide relevant API credentials during initialization.++ Features:+ - Support for multiple embedding models (e.g., OpenAI, Azure).+ - Parallel processing for faster embedding generation.+ - Local disk caching to store and reuse embedding results.+ - Total token usage tracking for cost monitoring.++ Note:+ Refer to the LiteLLM documentation for details on supported embedding models:+ https://docs.litellm.ai/docs/embedding/supported_embedding+ """def __init__(self,encoder_type: Optional[str] = None,diff --git a/knowledge_storm/lm.py b/knowledge_storm/lm.pyindex 6032589..78062d5 100644--- a/knowledge_storm/lm.py+++ b/knowledge_storm/lm.py@@ -184,6 +184,8 @@ def _inspect_history(lm, n: int = 1):print(_red(choices_text, end=""))print("\n\n\n")++############################@@ -262,9 +264,11 @@ class LitellmModel(LM):return outputs+# ========================================================================-# following LM models are deprecated.-# Remained in this file for reproduction prorpose+# The following language model classes were deprecated after v1.0.2.+# They remain in this file for backward compatibility but will no longer be maintained.+class OpenAIModel(dspy.OpenAI):"""A wrapper class for dspy.OpenAI."""@@ -1273,4 +1277,6 @@ class GoogleModel(dspy.dsp.modules.lm.LM):completions.append(response.parts[0].text)return completions++# ========================================================================diff --git a/setup.py b/setup.pyindex 0890bb7..4f5d453 100644--- a/setup.py+++ b/setup.py@@ -16,7 +16,7 @@ with open("requirements.txt", encoding="utf-8") as f:setup(name="knowledge-storm",- version="1.0.1",+ version="1.1.1",author="Yijia Shao, Yucheng Jiang",author_email="[email protected], [email protected]",description="STORM: A language model-powered knowledge curation engine.",