Skip to content

OpenRAG is a comprehensive, single package Retrieval-Augmented Generation platform built on Langflow, Docling, and Opensearch.

Python

finish ingestion and connectors pages

April Ma521dba6484a
diff --git a/docs/docs/core-components/ingestion.mdx b/docs/docs/core-components/ingestion.mdx
index 2045a398b..51fd33cee 100644
--- a/docs/docs/core-components/ingestion.mdx
+++ b/docs/docs/core-components/ingestion.mdx
@@ -6,32 +6,20 @@ slug: /ingestion
import Icon from "@site/src/components/icon/icon";
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
-import PartialTempKnowledge from '@site/docs/_partial-temp-knowledge.mdx';
import PartialIngestionFlow from '@site/docs/_partial-ingestion-flow.mdx';
-import PartialDockerComposeUp from '@site/docs/_partial-docker-compose-up.mdx';
-import PartialDockerStopAll from '@site/docs/_partial-docker-stop-all.mdx';
-import PartialOAuthCredentials from '@site/docs/_partial-oauth-credentials.mdx';
-
-
-<!--
-Ingest knowledge (existing page)
-Connector setup will be moved to a different page.
-See also: Configure connectors, browse and manage knowledge
-TBD: Supported file types
-
-check/update links
--->
+import PartialTempKnowledge from '@site/docs/_partial-temp-knowledge.mdx';
Upload documents to your [OpenRAG OpenSearch knowledge base](/knowledge) to populate your knowledge base with unique content, such as your own company documents, research papers, or websites.
-Documents are processed through OpenRAG's knowledge ingestion flows with Docling.
OpenRAG can ingest knowledge from direct file uploads, URLs, and cloud storage connectors.
Knowledge ingestion is powered by OpenRAG's built-in knowledge ingestion flows that use Docling to process documents before storing the documents in your OpenSearch knowledge base.
-During ingestion, documents are broken into smaller chunks of content that are then embedded using your selected [embedding model](/ingestion-configure#set-the-embedding-model-and-dimensions).
-Then, the chunks, embeddings, and associated metadata (which connects chunks of the same document) are stored in your OpenSearch knowledge base.
-To modify chunking behavior and other ingestion settings, see [Configure ingestion](/ingestion-configure) and [Inspect and modify flows](/agents#inspect-and-modify-flows).
+During ingestion, documents are broken into smaller chunks of content.
+Embeddings are generated for each chunk so the documents can be retrieved through similarity search during [chat](/chat), which is a standard retrieval augmented generation (RAG) pattern.
+The chunks, embeddings, and associated metadata (which connect chunks of the same document) are stored in your OpenSearch knowledge base.
+
+To modify chunking behavior, embedding model, and other ingestion settings, see [Configure ingestion](/ingestion-configure).
## Ingest local files and folders
@@ -82,117 +70,23 @@ When ingesting from the local file system or cloud storage connectors, OpenRAG s
<!-- TBD: .vtt, .mdx supported? -->
-## Ingest local files temporarily
+### Ingest local files temporarily
<PartialTempKnowledge />
-## Ingest files with cloud storage connectors {#oauth-ingestion}
-
-OpenRAG can use cloud storage connectors to ingest documents from the following external services:
-
-* AWS S3
-* Google Drive
-* Microsoft OneDrive
-* Microsoft Sharepoint
-
-These connectors enable seamless ingestion of files from cloud storage to your OpenRAG knowledge base.
-
-OAuth credentials are used to authorize access to cloud storage services.
-
-### Configure cloud storage connectors
-
-Before you can ingest documents from cloud storage, you must authorize OpenRAG's access to your cloud storage services.
-
-Typically, this requires that you register OpenRAG as an OAuth application in your cloud provider, and then obtain the app's OAuth credentials, such as a client ID and secret key.
-To enable multiple connectors, you must register an app and generate credentials for each provider.
-
-Then, add the OAuth credentials to your OpenRAG configuration:
-
-<Tabs>
-<TabItem value="TUI" label="TUI-managed services" default>
-
-If you use the [Terminal User Interface (TUI)](/tui) to manage your OpenRAG services, enter OAuth credentials on the **Advanced Setup** page.
-You can do this during [installation](/install#setup), or you can add the credentials afterwards:
-
-1. If OpenRAG is running, click **Stop All Services** in the TUI.
-
-2. Open the **Advanced Setup** page, and then add the OAuth credentials for the cloud storage providers that you want to use under **API Keys**:
-
- <PartialOAuthCredentials />
-
-3. For each connector you configured, register the redirect URIs shown in the TUI in your OAuth apps.
-
- The redirect URIs are used for the cloud storage connector webhooks. For Google, the redirect URIs are also used to redirect users back to OpenRAG after they sign in.
-
-4. Optional: Under **Others**, set the **Webhook Base URL** to the base address for your OAuth connector endpoints. If set, the OAuth connector webhook URLs are constructed as `WEBHOOK_BASE_URL/connectors/${provider}/webhook`. This option is required to enable automatic ingestion from cloud storage.
-
-5. Click **Save Configuration** to add the OAuth credentials to your [OpenRAG `.env` file](/reference/configuration).
-
-6. Click **Start Services** to restart the OpenRAG containers with the new configuration.
-
-7. Launch the OpenRAG app.
-
- If you provided Google OAuth credentials, you must sign in with Google before you are redirected to your OpenRAG instance.
-
-</TabItem>
-<TabItem value="env" label="Self-managed services">
-
-If you [installed OpenRAG with self-managed services](/docker), set OAuth credentials in your [OpenRAG `.env` file](/reference/configuration).
+## Ingest files from cloud storage {#oauth-ingestion}
-You can do this during [initial set up](/docker#setup), or you can add the credentials afterwards:
+To ingest knowledge from cloud storage using an OpenRAG cloud storage connector, do the following:
-1. Stop all OpenRAG containers:
+1. [Configure a cloud storage connector](/knowledge-connectors).
- <PartialDockerStopAll />
+2. Click <Icon name="Library" aria-hidden="true"/> **Knowledge** to view your OpenSearch knowledge base.
-2. Edit your OpenRAG `.env` file, and then add the [OAuth and cloud storage environment variables](/reference/configuration#oauth-and-cloud-storage-connector-settings) for the providers that you want to use:
+3. Click **Add Knowledge**, and then select a storage provider.
- ```env
- GOOGLE_OAUTH_CLIENT_ID=
- GOOGLE_OAUTH_CLIENT_SECRET=
+4. On the **Add Cloud Knowledge** page, click **Add Files**, and then select the files and folders to ingest from the connected storage.
- MICROSOFT_GRAPH_OAUTH_CLIENT_ID=
- MICROSOFT_GRAPH_OAUTH_CLIENT_SECRET=
-
- AWS_ACCESS_KEY_ID=
- AWS_SECRET_ACCESS_KEY=
- ```
-
- <PartialOAuthCredentials />
-
-3. Optional: Set the `WEBHOOK_BASE_URL` to the base address for your OAuth connector endpoints. If set, the OAuth connector webhook URLs are constructed as `WEBHOOK_BASE_URL/connectors/${provider}/webhook`. This option is required to enable automatic ingestion from cloud storage.
-
-4. Save the `.env` file.
-
-5. For each connector, you must register the OpenRAG redirect URIs in your OAuth apps:
-
- * Local deployments: `http://localhost:3000/auth/callback`
- * Production deployments: `https://your-domain.com/auth/callback`
-
- The redirect URIs are used for the cloud storage connector webhooks. For Google, the redirect URIs are also used to redirect users back to OpenRAG after they sign in.
-
-6. Restart your OpenRAG containers:
-
- <PartialDockerComposeUp />
-
-7. Access the OpenRAG frontend at `http://localhost:3000`.
-
- If you provided Google OAuth credentials, you must sign in with Google before you are redirected to your OpenRAG instance.
-
-</TabItem>
-</Tabs>
-
-### Ingest files from cloud storage
-
-To ingest knowledge with a cloud storage connector, do the following:
-
-1. Click <Icon name="Library" aria-hidden="true"/> **Knowledge** to view your OpenSearch knowledge base.
-
-2. Click **Add Knowledge**, and then select a storage provider.
-
-3. On the **Add Cloud Knowledge** page, click **Add Files**, and then select the files and folders to ingest from the connected storage.
-
-4. Click **Ingest Files**.
+5. Click **Ingest Files**.
The selected files are processed in the background through the **OpenSearch Ingestion** flow.
@@ -249,6 +143,14 @@ For each task, depending on its state, you can find the task ID, start time, dur
### Ingestion performance expectations
+Ingestion performance depends on many factors, such as the number and size of files ingested, file types, file contents, embedding model, chunk size, and hardware resources.
+
+Particularly when ingesting folders and very large files, such as PDFs with more than 300 pages, ingestion can take a long time or time out.
+For more information, see [Troubleshoot document ingestion or similarity search issues](/support/troubleshoot#document-ingestion-or-similarity-search-issues)
+
+<details>
+<summary>Example: Ingestion performance test</summary>
+
The following performance test was conducted with Docling Serve.
On a local VM with 7 vCPUs and 8 GiB RAM, OpenRAG ingested approximately 5.03 GB across 1,083 files in about 42 minutes.
@@ -260,9 +162,6 @@ Throughput scales with CPU cores, memory, storage speed, and configuration choic
This test returned 12 error, approximately 1.1 percent of the total files ingested.
All errors were file-specific, and they didn't stop the pipeline.
-<details>
-<summary>Ingestion performance test details</summary>
-
* Ingestion dataset:
* Total files: 1,083 items mounted
@@ -298,8 +197,7 @@ All errors were file-specific, and they didn't stop the pipeline.
* [About knowledge](/knowledge)
* [Configure ingestion](/ingestion-configure)
-* [Browse and manage knowledge](/knowledge-browse)
* [Configure connectors](/knowledge-connectors)
+* [Browse and manage knowledge](/knowledge-browse)
* [Chat with knowledge](/chat)
-* [Inspect and modify flows](/agents#inspect-and-modify-flows)
* [Troubleshoot document ingestion or similarity search issues](/support/troubleshoot#document-ingestion-or-similarity-search-issues)
\ No newline at end of file
diff --git a/docs/docs/core-components/knowledge-connectors.mdx b/docs/docs/core-components/knowledge-connectors.mdx
index a25dc42c9..b56b2fb71 100644
--- a/docs/docs/core-components/knowledge-connectors.mdx
+++ b/docs/docs/core-components/knowledge-connectors.mdx
@@ -4,30 +4,122 @@ slug: /knowledge-connectors
---
import Icon from "@site/src/components/icon/icon";
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
import PartialOpensearchAuthModes from '@site/docs/_partial-opensearch-auth-modes.mdx';
+import PartialOAuthCredentials from '@site/docs/_partial-oauth-credentials.mdx';
+import PartialDockerStopAll from '@site/docs/_partial-docker-stop-all.mdx';
+import PartialDockerComposeUp from '@site/docs/_partial-docker-compose-up.mdx';
-<!--
-Configure connectors (new page)
-Oauth mode
-Cloud storage connector setup
-TBD: disconnector/reconnect connectors, better separation of Oauth providers vs cloud storage access
-See also: About knowledge, ingest knowledge
+OpenRAG uses OAuth credentials to authorize access to cloud storage services so you can ingest documents from cloud storage.
-check/update links
--->
+If you provide Google OAuth credentials, these credentials are also used to enable OAuth mode for OpenRAG and your OpenSearch knowledge base.
-## OAuth credentials and OpenSearch authentication modes
+<!-- TBD: separation of pure OAuth from cloud storage connector OAuth apps -->
+<!-- TBD: Disconnect/reconnect connectors -->
+
+## OAuth credentials and OpenSearch authentication modes {#auth}
<PartialOpensearchAuthModes />
## Cloud storage connectors
+You can use OpenRAG's cloud storage connectors to [ingest files from cloud storage](/ingestion#oauth-ingestion).
+Specifically, you can connect to the following services:
+
+* AWS S3
+* Google Drive
+* Microsoft OneDrive
+* Microsoft Sharepoint
+
+To configure a cloud storage connector, you must register an OAuth app, get OAuth credentials, and then add the OAuth credentials to your OpenRAG configuration.
+
+### Register an OAuth app and generate credentials
+
+Register OpenRAG as an OAuth application in your cloud provider.
+Then, obtain OAuth credentials for the app, such as a client ID and secret key.
+
+To enable multiple connectors, you must register an app and generate credentials for each provider.
+
+### Add OAuth credentials to OpenRAG
+
+<Tabs>
+<TabItem value="TUI" label="TUI-managed services" default>
+
+If you use the [Terminal User Interface (TUI)](/tui) to manage your OpenRAG services, enter OAuth credentials on the **Advanced Setup** page.
+You can do this during [installation](/install#setup), or you can add the credentials afterwards:
+
+1. If OpenRAG is running, click **Stop All Services** in the TUI.
+
+2. Open the **Advanced Setup** page, and then add the OAuth credentials for the cloud storage providers that you want to use under **API Keys**:
+
+ <PartialOAuthCredentials />
+
+3. For each connector you configured, register the redirect URIs shown in the TUI in your OAuth apps.
+
+ The redirect URIs are used for the cloud storage connector webhooks. For Google, the redirect URIs are also used to redirect users back to OpenRAG after they sign in.
+
+4. Optional: Under **Others**, set the **Webhook Base URL** to the base address for your OAuth connector endpoints. If set, the OAuth connector webhook URLs are constructed as `WEBHOOK_BASE_URL/connectors/${provider}/webhook`. This option is required to enable automatic ingestion from cloud storage.
+
+5. Click **Save Configuration** to add the OAuth credentials to your [OpenRAG `.env` file](/reference/configuration).
+
+6. Click **Start Services** to restart the OpenRAG containers with the new configuration.
+
+7. Launch the OpenRAG app.
+
+ If you provided Google OAuth credentials, you must sign in with Google before you are redirected to your OpenRAG instance.
+
+</TabItem>
+<TabItem value="env" label="Self-managed services">
+
+If you [installed OpenRAG with self-managed services](/docker), set OAuth credentials in your [OpenRAG `.env` file](/reference/configuration).
+
+You can do this during [initial set up](/docker#setup), or you can add the credentials afterwards:
+
+1. Stop all OpenRAG containers:
+
+ <PartialDockerStopAll />
+
+2. Edit your OpenRAG `.env` file, and then add the [OAuth and cloud storage environment variables](/reference/configuration#oauth-and-cloud-storage-connector-settings) for the providers that you want to use:
+
+ ```env
+ GOOGLE_OAUTH_CLIENT_ID=
+ GOOGLE_OAUTH_CLIENT_SECRET=
+
+ MICROSOFT_GRAPH_OAUTH_CLIENT_ID=
+ MICROSOFT_GRAPH_OAUTH_CLIENT_SECRET=
+
+ AWS_ACCESS_KEY_ID=
+ AWS_SECRET_ACCESS_KEY=
+ ```
+
+ <PartialOAuthCredentials />
+
+3. Optional: Set the `WEBHOOK_BASE_URL` to the base address for your OAuth connector endpoints. If set, the OAuth connector webhook URLs are constructed as `WEBHOOK_BASE_URL/connectors/${provider}/webhook`. This option is required to enable automatic ingestion from cloud storage.
+
+4. Save the `.env` file.
+
+5. For each connector, you must register the OpenRAG redirect URIs in your OAuth apps:
+
+ * Local deployments: `http://localhost:3000/auth/callback`
+ * Production deployments: `https://your-domain.com/auth/callback`
+
+ The redirect URIs are used for the cloud storage connector webhooks. For Google, the redirect URIs are also used to redirect users back to OpenRAG after they sign in.
+
+6. Restart your OpenRAG containers:
+
+ <PartialDockerComposeUp />
+
+7. Access the OpenRAG frontend at `http://localhost:3000`.
+
+ If you provided Google OAuth credentials, you must sign in with Google before you are redirected to your OpenRAG instance.
+
+</TabItem>
+</Tabs>
## See also
* [About knowledge](/knowledge)
* [Ingest knowledge](/ingestion)
* [Configure ingestion](/ingestion-configure)
-* [Browse and manage knowledge](/knowledge-browse)
-
-<!-- TBD - separation of pure OAuth from cloud storage connector OAuth apps -->
\ No newline at end of file
+* [Browse and manage knowledge](/knowledge-browse)
\ No newline at end of file