An LLM-powered knowledge curation system that researches a topic and generates a full-length report with citations.
Python agentic-ragdeep-researchemnlp2024knowledge-curationlarge-language-modelsnaaclnlpreport-generationretrieval-augmented-generation
Require users to provide unique urls when using VectorRM.
AMMAS1e31dcf1a0c89
diff --git a/examples/helper/process_kaggle_arxiv_abstract_dataset.py b/examples/helper/process_kaggle_arxiv_abstract_dataset.pyindex f83012c..4cb885c 100644--- a/examples/helper/process_kaggle_arxiv_abstract_dataset.py+++ b/examples/helper/process_kaggle_arxiv_abstract_dataset.py@@ -21,7 +21,7 @@ if __name__ == "__main__":# Reformat the dataset to match the VectorRM input format.df.rename(columns={"abstracts": "content", "titles": "title"}, inplace=True)- df['url'] = ['uid_' + str(idx) for idx in range(len(df))]+ df['url'] = ['uid_' + str(idx) for idx in range(len(df))] # Ensure the url is unique.df['description'] = ''print(f'The downsampled dataset has {len(df)} samples.')diff --git a/examples/run_storm_wiki_gpt_with_VectorRM.py b/examples/run_storm_wiki_gpt_with_VectorRM.pyindex 9655245..c5dd435 100644--- a/examples/run_storm_wiki_gpt_with_VectorRM.py+++ b/examples/run_storm_wiki_gpt_with_VectorRM.py@@ -8,13 +8,11 @@ You need to set up the following environment variables to run this script:You will also need an existing Qdrant vector store either saved in a folder locally offline or in a server online.If not, then you would need a CSV file with documents, and the script is going to create the vector store for you.The CSV should be in the following format:-Content | Title | URL | Description+content | title | url | descriptionI am a document. | Document 1 | docu-n-112 | A self-explanatory document.I am another document. | Document 2 | docu-l-13 | Another self-explanatory document.-Notice that the URL can be an identifier for the document for any internal use.-The Title, URL, and Description columns are optional. If not provided, the script will use default empty values.-The content column is crucial and should be provided.+Notice that the URL will be a unique identifier for the document so ensure different documents have different urls.Output will be structured as belowargs.output_dir/diff --git a/src/rm.py b/src/rm.pyindex a0a6a8d..5126aa5 100644--- a/src/rm.py+++ b/src/rm.py@@ -171,8 +171,8 @@ class VectorRM(dspy.Retrieve):To be compatible with STORM, the custom documents should have the following fields:- content: The main text content of the document.- title: The title of the document.- - url: The URL of the document. STORM use url as the unique identifier of the document.- If not provided, a random string will be generated as the url.+ - url: The URL of the document. STORM use url as the unique identifier of the document, so ensure different+ documents have different urls.- description (optional): The description of the document.The documents should be stored in a CSV file."""