OpenRAG is a comprehensive, single package Retrieval-Augmented Generation platform built on Langflow, Docling, and Opensearch.
Python
sync and manage cloud storage connectors
April M41d1bbcd8c3d
diff --git a/docs/docs/core-components/ingestion.mdx b/docs/docs/core-components/ingestion.mdxindex 456f08ded..39c36099d 100644--- a/docs/docs/core-components/ingestion.mdx+++ b/docs/docs/core-components/ingestion.mdx@@ -67,8 +67,6 @@ When ingesting from the local file system or cloud storage connectors, OpenRAG s* .xls* .xlsx-<!-- TBD: .vtt, .mdx supported? -->-### Ingest local files temporarily<PartialTempKnowledge />@@ -79,19 +77,23 @@ To ingest knowledge from cloud storage using an OpenRAG cloud storage connector,1. [Configure a cloud storage connector](/knowledge-connectors).-2. Click <Icon name="Library" aria-hidden="true"/> **Knowledge** to view your OpenSearch knowledge base.+2. On the <Icon name="Library" aria-hidden="true"/> **Knowledge** or <Icon name="Settings2" aria-hidden="true"/> **Settings** page, click **Add Knowledge** to ingest files and folders from your connected cloud storage.-3. Click **Add Knowledge**, and then select a storage provider.+After triggering ingestion, the selected files are processed in the background through the **OpenSearch Ingestion** flow.-4. On the **Add Cloud Knowledge** page, click **Add Files**, and then select the files and folders to ingest from the connected storage.+<PartialIngestionFlow />-5. Click **Ingest Files**.+You can [monitor ingestion](#monitor-ingestion) to see the progress of the uploads and check for failed uploads.-The selected files are processed in the background through the **OpenSearch Ingestion** flow.+### Sync cloud storage-<PartialIngestionFlow />+:::info+If you have a lot of files in cloud storage, be aware that synchronization can trigger a large number of ingestions.+This can take a long time to complete, and timeouts can occur.+For more information, see [Troubleshoot document ingestion or similarity search issues](/support/troubleshoot#document-ingestion-or-similarity-search-issues).+:::-You can [monitor ingestion](#monitor-ingestion) to see the progress of the uploads and check for failed uploads.+On the <Icon name="Library" aria-hidden="true"/> **Knowledge** or <Icon name="Settings2" aria-hidden="true"/> **Settings** page, click <Icon name="Settings2" aria-hidden="true"/> **Sync** to synchronize your cloud storage and your OpenRAG knowledge base.## Ingest knowledge from URLs {#url-flow}diff --git a/docs/docs/core-components/knowledge-browse.mdx b/docs/docs/core-components/knowledge-browse.mdxindex 3b8380bb0..2ed6dac76 100644--- a/docs/docs/core-components/knowledge-browse.mdx+++ b/docs/docs/core-components/knowledge-browse.mdx@@ -66,6 +66,16 @@ If you [delete these documents](#delete-knowledge), then you won't be able to asIt is recommended that you keep these documents, and use [filters](/knowledge-filters) to separate them from your other knowledge.An **OpenRAG Docs** filter is created automatically for these documents.+### Get latest OpenRAG documentation++On the **Knowledge** page, click <Icon name="Library" aria-hidden="true"/> **Fetch latest docs** to ingest the latest OpenRAG documentation from docs.openrag.com (this website).++:::tip+**Fetch latest docs** _only_ gets the latest OpenRAG documentation.++To update docs ingested from cloud storage connectors], see [Configure connectors](/knowledge-connectors).+:::+## Add knowledgeAdd knowledge to your knowledge base by [ingesting documents](/ingestion).diff --git a/docs/docs/core-components/knowledge-connectors.mdx b/docs/docs/core-components/knowledge-connectors.mdxindex 52394aaf6..afbcbddee 100644--- a/docs/docs/core-components/knowledge-connectors.mdx+++ b/docs/docs/core-components/knowledge-connectors.mdx@@ -15,9 +15,6 @@ OpenRAG uses OAuth credentials to authorize access to cloud storage services soIf you provide Google OAuth credentials, these credentials are also used to enable OAuth mode for OpenRAG and your OpenSearch knowledge base.-<!-- TBD: separation of pure OAuth from cloud storage connector OAuth apps -->-<!-- TBD: Disconnect/reconnect connectors -->-## OAuth credentials and OpenSearch authentication modes {#auth}<PartialOpensearchAuthModes />@@ -128,12 +125,11 @@ You can do this during [initial set up](/docker#setup), or you can add the crede## Ingest documents with cloud connectors-See [Ingest files from cloud storage](/ingestion#oauth-ingestion).+For more information, see [Ingest files from cloud storage](/ingestion#oauth-ingestion).-## Disconnect and reconnect cloud storage connectors+## Manage cloud storage connectors-In OpenRAG, click <Icon name="Settings2" aria-hidden="true"/> **Settings** to manage cloud storage connections.-You can connect, disconnect, reconnect, and trigger ingestion for each configured connector.+In OpenRAG, click <Icon name="Settings2" aria-hidden="true"/> **Settings** to connect, disconnect, reconnect, and edit connector configuration details for each cloud storage connector.## See also