OpenRAG is a comprehensive, single package Retrieval-Augmented Generation platform built on Langflow, Docling, and Opensearch.
Python
finish ingestion and connectors pages
April Ma521dba6484a
diff --git a/docs/docs/core-components/ingestion.mdx b/docs/docs/core-components/ingestion.mdxindex 2045a398b..51fd33cee 100644--- a/docs/docs/core-components/ingestion.mdx+++ b/docs/docs/core-components/ingestion.mdx@@ -6,32 +6,20 @@ slug: /ingestionimport Icon from "@site/src/components/icon/icon";import Tabs from '@theme/Tabs';import TabItem from '@theme/TabItem';-import PartialTempKnowledge from '@site/docs/_partial-temp-knowledge.mdx';import PartialIngestionFlow from '@site/docs/_partial-ingestion-flow.mdx';-import PartialDockerComposeUp from '@site/docs/_partial-docker-compose-up.mdx';-import PartialDockerStopAll from '@site/docs/_partial-docker-stop-all.mdx';-import PartialOAuthCredentials from '@site/docs/_partial-oauth-credentials.mdx';---<!---Ingest knowledge (existing page)-Connector setup will be moved to a different page.-See also: Configure connectors, browse and manage knowledge-TBD: Supported file types--check/update links--->+import PartialTempKnowledge from '@site/docs/_partial-temp-knowledge.mdx';Upload documents to your [OpenRAG OpenSearch knowledge base](/knowledge) to populate your knowledge base with unique content, such as your own company documents, research papers, or websites.-Documents are processed through OpenRAG's knowledge ingestion flows with Docling.OpenRAG can ingest knowledge from direct file uploads, URLs, and cloud storage connectors.Knowledge ingestion is powered by OpenRAG's built-in knowledge ingestion flows that use Docling to process documents before storing the documents in your OpenSearch knowledge base.-During ingestion, documents are broken into smaller chunks of content that are then embedded using your selected [embedding model](/ingestion-configure#set-the-embedding-model-and-dimensions).-Then, the chunks, embeddings, and associated metadata (which connects chunks of the same document) are stored in your OpenSearch knowledge base.-To modify chunking behavior and other ingestion settings, see [Configure ingestion](/ingestion-configure) and [Inspect and modify flows](/agents#inspect-and-modify-flows).+During ingestion, documents are broken into smaller chunks of content.+Embeddings are generated for each chunk so the documents can be retrieved through similarity search during [chat](/chat), which is a standard retrieval augmented generation (RAG) pattern.+The chunks, embeddings, and associated metadata (which connect chunks of the same document) are stored in your OpenSearch knowledge base.++To modify chunking behavior, embedding model, and other ingestion settings, see [Configure ingestion](/ingestion-configure).## Ingest local files and folders@@ -82,117 +70,23 @@ When ingesting from the local file system or cloud storage connectors, OpenRAG s<!-- TBD: .vtt, .mdx supported? -->-## Ingest local files temporarily+### Ingest local files temporarily<PartialTempKnowledge />-## Ingest files with cloud storage connectors {#oauth-ingestion}--OpenRAG can use cloud storage connectors to ingest documents from the following external services:--* AWS S3-* Google Drive-* Microsoft OneDrive-* Microsoft Sharepoint--These connectors enable seamless ingestion of files from cloud storage to your OpenRAG knowledge base.--OAuth credentials are used to authorize access to cloud storage services.--### Configure cloud storage connectors--Before you can ingest documents from cloud storage, you must authorize OpenRAG's access to your cloud storage services.--Typically, this requires that you register OpenRAG as an OAuth application in your cloud provider, and then obtain the app's OAuth credentials, such as a client ID and secret key.-To enable multiple connectors, you must register an app and generate credentials for each provider.--Then, add the OAuth credentials to your OpenRAG configuration:--<Tabs>-<TabItem value="TUI" label="TUI-managed services" default>--If you use the [Terminal User Interface (TUI)](/tui) to manage your OpenRAG services, enter OAuth credentials on the **Advanced Setup** page.-You can do this during [installation](/install#setup), or you can add the credentials afterwards:--1. If OpenRAG is running, click **Stop All Services** in the TUI.--2. Open the **Advanced Setup** page, and then add the OAuth credentials for the cloud storage providers that you want to use under **API Keys**:-- <PartialOAuthCredentials />--3. For each connector you configured, register the redirect URIs shown in the TUI in your OAuth apps.-- The redirect URIs are used for the cloud storage connector webhooks. For Google, the redirect URIs are also used to redirect users back to OpenRAG after they sign in.--4. Optional: Under **Others**, set the **Webhook Base URL** to the base address for your OAuth connector endpoints. If set, the OAuth connector webhook URLs are constructed as `WEBHOOK_BASE_URL/connectors/${provider}/webhook`. This option is required to enable automatic ingestion from cloud storage.--5. Click **Save Configuration** to add the OAuth credentials to your [OpenRAG `.env` file](/reference/configuration).--6. Click **Start Services** to restart the OpenRAG containers with the new configuration.--7. Launch the OpenRAG app.-- If you provided Google OAuth credentials, you must sign in with Google before you are redirected to your OpenRAG instance.--</TabItem>-<TabItem value="env" label="Self-managed services">--If you [installed OpenRAG with self-managed services](/docker), set OAuth credentials in your [OpenRAG `.env` file](/reference/configuration).+## Ingest files from cloud storage {#oauth-ingestion}-You can do this during [initial set up](/docker#setup), or you can add the credentials afterwards:+To ingest knowledge from cloud storage using an OpenRAG cloud storage connector, do the following:-1. Stop all OpenRAG containers:+1. [Configure a cloud storage connector](/knowledge-connectors).- <PartialDockerStopAll />+2. Click <Icon name="Library" aria-hidden="true"/> **Knowledge** to view your OpenSearch knowledge base.-2. Edit your OpenRAG `.env` file, and then add the [OAuth and cloud storage environment variables](/reference/configuration#oauth-and-cloud-storage-connector-settings) for the providers that you want to use:+3. Click **Add Knowledge**, and then select a storage provider.- ```env- GOOGLE_OAUTH_CLIENT_ID=- GOOGLE_OAUTH_CLIENT_SECRET=+4. On the **Add Cloud Knowledge** page, click **Add Files**, and then select the files and folders to ingest from the connected storage.- MICROSOFT_GRAPH_OAUTH_CLIENT_ID=- MICROSOFT_GRAPH_OAUTH_CLIENT_SECRET=-- AWS_ACCESS_KEY_ID=- AWS_SECRET_ACCESS_KEY=- ```-- <PartialOAuthCredentials />--3. Optional: Set the `WEBHOOK_BASE_URL` to the base address for your OAuth connector endpoints. If set, the OAuth connector webhook URLs are constructed as `WEBHOOK_BASE_URL/connectors/${provider}/webhook`. This option is required to enable automatic ingestion from cloud storage.--4. Save the `.env` file.--5. For each connector, you must register the OpenRAG redirect URIs in your OAuth apps:-- * Local deployments: `http://localhost:3000/auth/callback`- * Production deployments: `https://your-domain.com/auth/callback`-- The redirect URIs are used for the cloud storage connector webhooks. For Google, the redirect URIs are also used to redirect users back to OpenRAG after they sign in.--6. Restart your OpenRAG containers:-- <PartialDockerComposeUp />--7. Access the OpenRAG frontend at `http://localhost:3000`.-- If you provided Google OAuth credentials, you must sign in with Google before you are redirected to your OpenRAG instance.--</TabItem>-</Tabs>--### Ingest files from cloud storage--To ingest knowledge with a cloud storage connector, do the following:--1. Click <Icon name="Library" aria-hidden="true"/> **Knowledge** to view your OpenSearch knowledge base.--2. Click **Add Knowledge**, and then select a storage provider.--3. On the **Add Cloud Knowledge** page, click **Add Files**, and then select the files and folders to ingest from the connected storage.--4. Click **Ingest Files**.+5. Click **Ingest Files**.The selected files are processed in the background through the **OpenSearch Ingestion** flow.@@ -249,6 +143,14 @@ For each task, depending on its state, you can find the task ID, start time, dur### Ingestion performance expectations+Ingestion performance depends on many factors, such as the number and size of files ingested, file types, file contents, embedding model, chunk size, and hardware resources.++Particularly when ingesting folders and very large files, such as PDFs with more than 300 pages, ingestion can take a long time or time out.+For more information, see [Troubleshoot document ingestion or similarity search issues](/support/troubleshoot#document-ingestion-or-similarity-search-issues)++<details>+<summary>Example: Ingestion performance test</summary>+The following performance test was conducted with Docling Serve.On a local VM with 7 vCPUs and 8 GiB RAM, OpenRAG ingested approximately 5.03 GB across 1,083 files in about 42 minutes.@@ -260,9 +162,6 @@ Throughput scales with CPU cores, memory, storage speed, and configuration choicThis test returned 12 error, approximately 1.1 percent of the total files ingested.All errors were file-specific, and they didn't stop the pipeline.-<details>-<summary>Ingestion performance test details</summary>-* Ingestion dataset:* Total files: 1,083 items mounted@@ -298,8 +197,7 @@ All errors were file-specific, and they didn't stop the pipeline.* [About knowledge](/knowledge)* [Configure ingestion](/ingestion-configure)-* [Browse and manage knowledge](/knowledge-browse)* [Configure connectors](/knowledge-connectors)+* [Browse and manage knowledge](/knowledge-browse)* [Chat with knowledge](/chat)-* [Inspect and modify flows](/agents#inspect-and-modify-flows)* [Troubleshoot document ingestion or similarity search issues](/support/troubleshoot#document-ingestion-or-similarity-search-issues)\ No newline at end of filediff --git a/docs/docs/core-components/knowledge-connectors.mdx b/docs/docs/core-components/knowledge-connectors.mdxindex a25dc42c9..b56b2fb71 100644--- a/docs/docs/core-components/knowledge-connectors.mdx+++ b/docs/docs/core-components/knowledge-connectors.mdx@@ -4,30 +4,122 @@ slug: /knowledge-connectors---import Icon from "@site/src/components/icon/icon";+import Tabs from '@theme/Tabs';+import TabItem from '@theme/TabItem';import PartialOpensearchAuthModes from '@site/docs/_partial-opensearch-auth-modes.mdx';+import PartialOAuthCredentials from '@site/docs/_partial-oauth-credentials.mdx';+import PartialDockerStopAll from '@site/docs/_partial-docker-stop-all.mdx';+import PartialDockerComposeUp from '@site/docs/_partial-docker-compose-up.mdx';-<!---Configure connectors (new page)-Oauth mode-Cloud storage connector setup-TBD: disconnector/reconnect connectors, better separation of Oauth providers vs cloud storage access-See also: About knowledge, ingest knowledge+OpenRAG uses OAuth credentials to authorize access to cloud storage services so you can ingest documents from cloud storage.-check/update links--->+If you provide Google OAuth credentials, these credentials are also used to enable OAuth mode for OpenRAG and your OpenSearch knowledge base.-## OAuth credentials and OpenSearch authentication modes+<!-- TBD: separation of pure OAuth from cloud storage connector OAuth apps -->+<!-- TBD: Disconnect/reconnect connectors -->++## OAuth credentials and OpenSearch authentication modes {#auth}<PartialOpensearchAuthModes />## Cloud storage connectors+You can use OpenRAG's cloud storage connectors to [ingest files from cloud storage](/ingestion#oauth-ingestion).+Specifically, you can connect to the following services:++* AWS S3+* Google Drive+* Microsoft OneDrive+* Microsoft Sharepoint++To configure a cloud storage connector, you must register an OAuth app, get OAuth credentials, and then add the OAuth credentials to your OpenRAG configuration.++### Register an OAuth app and generate credentials++Register OpenRAG as an OAuth application in your cloud provider.+Then, obtain OAuth credentials for the app, such as a client ID and secret key.++To enable multiple connectors, you must register an app and generate credentials for each provider.++### Add OAuth credentials to OpenRAG++<Tabs>+<TabItem value="TUI" label="TUI-managed services" default>++If you use the [Terminal User Interface (TUI)](/tui) to manage your OpenRAG services, enter OAuth credentials on the **Advanced Setup** page.+You can do this during [installation](/install#setup), or you can add the credentials afterwards:++1. If OpenRAG is running, click **Stop All Services** in the TUI.++2. Open the **Advanced Setup** page, and then add the OAuth credentials for the cloud storage providers that you want to use under **API Keys**:++ <PartialOAuthCredentials />++3. For each connector you configured, register the redirect URIs shown in the TUI in your OAuth apps.++ The redirect URIs are used for the cloud storage connector webhooks. For Google, the redirect URIs are also used to redirect users back to OpenRAG after they sign in.++4. Optional: Under **Others**, set the **Webhook Base URL** to the base address for your OAuth connector endpoints. If set, the OAuth connector webhook URLs are constructed as `WEBHOOK_BASE_URL/connectors/${provider}/webhook`. This option is required to enable automatic ingestion from cloud storage.++5. Click **Save Configuration** to add the OAuth credentials to your [OpenRAG `.env` file](/reference/configuration).++6. Click **Start Services** to restart the OpenRAG containers with the new configuration.++7. Launch the OpenRAG app.++ If you provided Google OAuth credentials, you must sign in with Google before you are redirected to your OpenRAG instance.++</TabItem>+<TabItem value="env" label="Self-managed services">++If you [installed OpenRAG with self-managed services](/docker), set OAuth credentials in your [OpenRAG `.env` file](/reference/configuration).++You can do this during [initial set up](/docker#setup), or you can add the credentials afterwards:++1. Stop all OpenRAG containers:++ <PartialDockerStopAll />++2. Edit your OpenRAG `.env` file, and then add the [OAuth and cloud storage environment variables](/reference/configuration#oauth-and-cloud-storage-connector-settings) for the providers that you want to use:++ ```env+ GOOGLE_OAUTH_CLIENT_ID=+ GOOGLE_OAUTH_CLIENT_SECRET=++ MICROSOFT_GRAPH_OAUTH_CLIENT_ID=+ MICROSOFT_GRAPH_OAUTH_CLIENT_SECRET=++ AWS_ACCESS_KEY_ID=+ AWS_SECRET_ACCESS_KEY=+ ```++ <PartialOAuthCredentials />++3. Optional: Set the `WEBHOOK_BASE_URL` to the base address for your OAuth connector endpoints. If set, the OAuth connector webhook URLs are constructed as `WEBHOOK_BASE_URL/connectors/${provider}/webhook`. This option is required to enable automatic ingestion from cloud storage.++4. Save the `.env` file.++5. For each connector, you must register the OpenRAG redirect URIs in your OAuth apps:++ * Local deployments: `http://localhost:3000/auth/callback`+ * Production deployments: `https://your-domain.com/auth/callback`++ The redirect URIs are used for the cloud storage connector webhooks. For Google, the redirect URIs are also used to redirect users back to OpenRAG after they sign in.++6. Restart your OpenRAG containers:++ <PartialDockerComposeUp />++7. Access the OpenRAG frontend at `http://localhost:3000`.++ If you provided Google OAuth credentials, you must sign in with Google before you are redirected to your OpenRAG instance.++</TabItem>+</Tabs>## See also* [About knowledge](/knowledge)* [Ingest knowledge](/ingestion)* [Configure ingestion](/ingestion-configure)-* [Browse and manage knowledge](/knowledge-browse)--<!-- TBD - separation of pure OAuth from cloud storage connector OAuth apps -->\ No newline at end of file+* [Browse and manage knowledge](/knowledge-browse)\ No newline at end of file