OpenRAG is a comprehensive, single package Retrieval-Augmented Generation platform built on Langflow, Docling, and Opensearch.
Python
start moving content, update links
April M2705e1f0bddf
diff --git a/docs/docs/_partial-ingestion-flow.mdx b/docs/docs/_partial-ingestion-flow.mdxindex 361116181..24a874753 100644--- a/docs/docs/_partial-ingestion-flow.mdx+++ b/docs/docs/_partial-ingestion-flow.mdx@@ -11,13 +11,13 @@ The **OpenSearch Ingestion** flow is comprised of several components that work t* [**Docling Serve** component](https://docs.langflow.org/bundles-docling#docling-serve): Ingests files and processes them by connecting to OpenRAG's local Docling Serve service. The output is `DoclingDocument` data that contains the extracted text and metadata from the documents.* [**Export DoclingDocument** component](https://docs.langflow.org/bundles-docling#export-doclingdocument): Exports processed `DoclingDocument` data to Markdown format with image placeholders. This conversion standardizes the document data in preparation for further processing.* [**DataFrame Operations** component](https://docs.langflow.org/dataframe-operations): Three of these components run sequentially to add metadata to the document data: `filename`, `file_size`, and `mimetype`.-* [**Split Text** component](https://docs.langflow.org/split-text): Splits the processed text into chunks, based on the configured [chunk size and overlap settings](/knowledge#knowledge-ingestion-settings).+* [**Split Text** component](https://docs.langflow.org/split-text): Splits the processed text into chunks, based on the configured [chunk size and overlap settings](/ingestion-configure).* **Secret Input** component: If needed, four of these components securely fetch the [OAuth authentication](/knowledge#auth) configuration variables: `CONNECTOR_TYPE`, `OWNER`, `OWNER_EMAIL`, and `OWNER_NAME`.* **Create Data** component: Combines the authentication credentials from the **Secret Input** components into a structured data object that is associated with the document embeddings.-* [**Embedding Model** component](https://docs.langflow.org/components-embedding-models): Generates vector embeddings using your selected [embedding model](/knowledge#set-the-embedding-model-and-dimensions).-* [**OpenSearch** component](https://docs.langflow.org/bundles-elastic#opensearch): Stores the processed documents and their embeddings in a `documents` index of your OpenRAG [OpenSearch knowledge base](/knowledge).+* [**Embedding Model** component](https://docs.langflow.org/components-embedding-models): Generates vector embeddings using your selected [embedding model](/ingestion-configure#set-the-embedding-model-and-dimensions).+* [**OpenSearch** component](https://docs.langflow.org/bundles-elastic#opensearch): Stores the processed documents and their embeddings in your OpenRAG [OpenSearch knowledge base](/knowledge).- The default address for the OpenSearch instance is `https://opensearch:9200`. To change this address, edit the `OPENSEARCH_PORT` [environment variable](/reference/configuration#opensearch-settings).+ The default address for the OpenSearch knowledge base is `https://opensearch:9200`. To change this address, edit the `OPENSEARCH_PORT` [environment variable](/reference/configuration#opensearch-settings).The default authentication method is JSON Web Token (JWT) authentication. If you [edit the flow](/agents#inspect-and-modify-flows), you can select `basic` auth mode, which uses the `OPENSEARCH_USERNAME` and `OPENSEARCH_PASSWORD` [environment variables](/reference/configuration#opensearch-settings) for authentication instead of JWT.diff --git a/docs/docs/_partial-onboarding.mdx b/docs/docs/_partial-onboarding.mdxindex 3e05ebdfb..d34b1a1db 100644--- a/docs/docs/_partial-onboarding.mdx+++ b/docs/docs/_partial-onboarding.mdx@@ -34,7 +34,7 @@ For information about another provider's credentials and settings, see the instr5. Click **Complete**.- After you configure the embedding model, OpenRAG uses your credentials and models to ingest some [initial documents](/knowledge#default-documents). This tests the connection, and it allows you to ask OpenRAG about itself in the [**Chat**](/chat).+ After you configure the embedding model, OpenRAG uses your credentials and models to ingest some [initial documents](/knowledge-browse#default-documents). This tests the connection, and it allows you to ask OpenRAG about itself in the [**Chat**](/chat).If there is a problem with the model configuration, an error occurs and you are redirected back to the application onboarding screen.Verify that the credential is valid and has access to the selected model, and then click **Complete** to retry ingestion.@@ -66,7 +66,7 @@ For information about another provider's credentials and settings, see the instr6. Click **Complete**.- After you configure the embedding model, OpenRAG uses your credentials and models to ingest some [initial documents](/knowledge#default-documents). This tests the connection, and it allows you to ask OpenRAG about itself in the [**Chat**](/chat).+ After you configure the embedding model, OpenRAG uses your credentials and models to ingest some [initial documents](/knowledge-browse#default-documents). This tests the connection, and it allows you to ask OpenRAG about itself in the [**Chat**](/chat).If there is a problem with the model configuration, an error occurs and you are redirected back to the application onboarding screen.Verify that the credentials are valid and have access to the selected model, and then click **Complete** to retry ingestion.@@ -111,7 +111,7 @@ For information about another provider's credentials and settings, see the instr6. Click **Complete**.- After you configure the embedding model, OpenRAG uses your credentials and models to ingest some [initial documents](/knowledge#default-documents). This tests the connection, and it allows you to ask OpenRAG about itself in the [**Chat**](/chat).+ After you configure the embedding model, OpenRAG uses your credentials and models to ingest some [initial documents](/knowledge-browse#default-documents). This tests the connection, and it allows you to ask OpenRAG about itself in the [**Chat**](/chat).If there is a problem with the model configuration, an error occurs and you are redirected back to the application onboarding screen.Verify that the server address is valid, and that the selected model is running on the server.Then, click **Complete** to retry ingestion.@@ -136,7 +136,7 @@ For information about another provider's credentials and settings, see the instr5. Click **Complete**.- After you configure the embedding model, OpenRAG uses your credentials and models to ingest some [initial documents](/knowledge#default-documents). This tests the connection, and it allows you to ask OpenRAG about itself in the [**Chat**](/chat).+ After you configure the embedding model, OpenRAG uses your credentials and models to ingest some [initial documents](/knowledge-browse#default-documents). This tests the connection, and it allows you to ask OpenRAG about itself in the [**Chat**](/chat).If there is a problem with the model configuration, an error occurs and you are redirected back to the application onboarding screen.Verify that the credential is valid and has access to the selected model, and then click **Complete** to retry ingestion.diff --git a/docs/docs/_partial-setup.mdx b/docs/docs/_partial-setup.mdxindex b6d675d91..b698d848d 100644--- a/docs/docs/_partial-setup.mdx+++ b/docs/docs/_partial-setup.mdx@@ -56,7 +56,7 @@ import PartialOAuthCredentials from '@site/docs/_partial-oauth-credentials.mdx';6. Optional: Under **Others**, you can edit the following settings if needed:- * **Documents Paths**: Use the default path, or provide a path to a directory where you want OpenRAG to look for documents to ingest into your [knowledge base](/knowledge).+ * **Documents Paths**: Set the [local documents path](/ingestion-configure#set-the-local-documents-path). Use the default path, or provide a path to a directory where you want OpenRAG to look for documents to ingest into your [knowledge base](/knowledge).* **OpenSearch Data Path**: Use the default path, or specify the path where you want OpenRAG to create your OpenSearch index.* **Langflow Public URL (Advanced Setup only)** : Sets the base address to access the Langflow web interface where users interact with the Langflow editor in a browser. You must set this value to run Langflow on a non-default port (`7860`)* **Webhook Base URL (Advanced Setup only)**: If you entered OAuth credentials, you can set the base address for your OAuth connector endpoints. If set, the OAuth connector webhook URLs are constructed as `WEBHOOK_BASE_URL/connectors/${provider}/webhook`. This option is required to enable automatic ingestion from cloud storage.diff --git a/docs/docs/core-components/ingestion-configure.mdx b/docs/docs/core-components/ingestion-configure.mdxindex 27ceeaf55..eec8fecff 100644--- a/docs/docs/core-components/ingestion-configure.mdx+++ b/docs/docs/core-components/ingestion-configure.mdx@@ -1,4 +1,114 @@---title: Configure ingestionslug: /ingestion-configure----\ No newline at end of file+---++import Icon from "@site/src/components/icon/icon";++The knowledge ingestion settings determine how documents are processed when ingested into your [knowledge base](/knowledge).+This includes configurations such as chunking strategies, embedding models, and image handling.++## Existing documents aren't reprocessed after changing ingestion settings++:::warning+Changes to knowledge ingestion settings only apply to documents that you upload after making changes.+Documents uploaded before changing these settings aren't reprocessed.+:::++After changing knowledge ingestion settings, you must determine if you need to reupload any documents to be consistent with the new settings.++It isn't always necessary to reupload documents after changing knowledge ingestion settings.+For example, it is typical to upload some documents with OCR enabled and others without OCR enabled.++If needed, you can use [filters](/knowledge-filters) to separate documents that you uploaded with different settings, such as different embedding models.++## Set the embedding model and dimensions {#set-the-embedding-model-and-dimensions}++When you [install OpenRAG](/install-options), you select at least one embedding model during the [application onboarding process](/install#application-onboarding).+OpenRAG automatically detects and configures the appropriate vector dimensions for your selected embedding model, ensuring optimal search performance and compatibility.++In the OpenRAG repository, you can find the complete list of supported models in [`models_service.py`](https://github.com/langflow-ai/openrag/blob/main/src/services/models_service.py) and the corresponding vector dimensions in [`settings.py`](https://github.com/langflow-ai/openrag/blob/main/src/config/settings.py).++During the application onboarding process, you can select from the supported models.+The default embedding dimension is `1536`, and the default model is the OpenAI `text-embedding-3-small`.++If you want to use an unsupported model, you must manually set the model in your [OpenRAG `.env` file](/reference/configuration).+If you use an unsupported embedding model that doesn't have defined dimensions in `settings.py`, then OpenRAG falls back to the default dimensions (1536) and logs a warning. OpenRAG's OpenSearch knowledge base and flows continue to work, but [similarity search](https://www.ibm.com/think/topics/vector-search) quality can be affected if the actual model dimensions aren't 1536.++To change the embedding model after onboarding, modify the embedding model configuration on the OpenRAG **Settings** page or in your [OpenRAG `.env` file](/reference/configuration).+This ensures that all relevant [OpenRAG flows](/agents) are updated to use the new embedding model configuration.++If you edit these settings in the `.env` file, you must [stop and restart the OpenRAG containers](/manage-services#stop-and-start-containers) to apply the changes.++## Set Docling parameters++OpenRAG uses [Docling](https://docling-project.github.io/docling/) for document ingestion because it supports many file formats, processes tables and images well, and performs efficiently.++When you [upload documents](/ingestion), Docling processes the files, splits them into chunks, and stores them as separate, structured documents in your OpenSearch knowledge base.++### Select a Docling implementation {#select-a-docling-implementation}++You can use either Docling Serve or OpenRAG's built-in Docling ingestion pipeline to process documents.++* **Docling Serve ingestion**: By default, OpenRAG uses [Docling Serve](https://github.com/docling-project/docling-serve).+It starts a local `docling serve` process, and then runs Docling ingestion through the Docling Serve API.++ To use a remote `docling serve` instance or your own local instance, set `DOCLING_SERVE_URL=http://**HOST_IP**:5001` in your [OpenRAG `.env` file](/reference/configuration#document-processing-settings).+ The service must run on port 5001.++* **Built-in Docling ingestion**: If you want to use OpenRAG's built-in Docling ingestion pipeline instead of the separate Docling Serve service, set `DISABLE_INGEST_WITH_LANGFLOW=true` in your [OpenRAG `.env` file](/reference/configuration#document-processing-settings).+The built-in pipeline uses the Docling processor directly instead of through the Docling Serve API.+For the underlying functionality, see [`processors.py`](https://github.com/langflow-ai/openrag/blob/main/src/models/processors.py#L58) in the OpenRAG repository.++### Configure Docling ingestion settings++To modify the Docling document processing and embedding parameters, click <Icon name="Settings2" aria-hidden="true"/> **Settings** in OpenRAG, and then find the **Knowledge Ingest** section.++:::tip+The TUI warns you if `docling serve` isn't running.+For information about starting and stopping OpenRAG native services, like Docling, see [Manage OpenRAG services](/manage-services).+:::++You can edit the following parameters:++* **Embedding model**: Select the model to use to generate vector embeddings for your documents.++ This is initially set during installation.+ The recommended way to change this setting is in the OpenRAG <Icon name="Settings2" aria-hidden="true"/> **Settings** or your [OpenRAG `.env` file](/reference/configuration).+ This ensures that all relevant [OpenRAG flows](/agents) are updated to use the new embedding model configuration.++ If you uploaded documents prior to changing the embedding model, you can [create filters](/knowledge-filters) to separate documents embedded with different models, or you can reupload all documents to regenerate embeddings with the new model.+ If you want to use multiple embeddings models, similarity search (in the **Chat**) can take longer as it searches each model's embeddings separately.++* **Chunk size**: Set the number of characters for each text chunk when breaking down a file.+Larger chunks yield more context per chunk, but can include irrelevant information. Smaller chunks yield more precise semantic search, but can lack context.+The default value is 1000 characters, which is usually a good balance between context and precision.++* **Chunk overlap**: Set the number of characters to overlap over chunk boundaries.+Use larger overlap values for documents where context is most important. Use smaller overlap values for simpler documents or when optimization is most important.+The default value is 200 characters, which represents an overlap of 20 percent with the default **Chunk size** of 1000. This is suitable for general use. For faster processing, decrease the overlap to approximately 10 percent. For more complex documents where you need to preserve context across chunks, increase it to approximately 40 percent.++* **Table structure**: Enables Docling's [`DocumentConverter`](https://docling-project.github.io/docling/reference/document_converter/) tool for parsing tables. Instead of treating tables as plain text, tables are output as structured table data with preserved relationships and metadata. This option is enabled by default.++* **OCR**: Enables Optical Character Recognition (OCR) processing when extracting text from images and ingesting scanned documents. This setting is best suited for processing text-based documents faster with Docling's [`DocumentConverter`](https://docling-project.github.io/docling/reference/document_converter/). Images are ignored and not processed.++ This option is disabled by default. Enabling OCR can slow ingestion performance.++ If OpenRAG detects that the local machine is running on macOS, OpenRAG uses the [ocrmac](https://www.piwheels.org/project/ocrmac/) OCR engine. Other platforms use [easyocr](https://www.jaided.ai/easyocr/).++* **Picture descriptions**: Only applicable if **OCR** is enabled. Adds image descriptions generated by the [`SmolVLM-256M-Instruct`](https://huggingface.co/HuggingFaceTB/SmolVLM-Instruct) model. Enabling picture descriptions can slow ingestion performance.++## Set the local documents path {#set-the-local-documents-path}++The default path for local uploads is `~/.openrag/documents`. This is mounted to the `/app/openrag-documents/` directory inside the OpenRAG container. Files added to the host or container directory are visible in both locations.++To change this location, modify the **Documents Paths** variable in either the [**Basic/Advanced Setup** menu](/install#setup) or in your [OpenRAG `.env` file](/reference/configuration).++## See also++* [About knowledge](/knowledge)+* [Ingest knowledge](/ingestion)+* [Browse and manage knowledge](/knowledge-browse)+* [Configure connectors](/knowledge-connectors)+* [Inspect and modify flows](/agents#inspect-and-modify-flows)+* [Troubleshoot document ingestion or similarity search issues](/support/troubleshoot#document-ingestion-or-similarity-search-issues)\ No newline at end of filediff --git a/docs/docs/core-components/ingestion.mdx b/docs/docs/core-components/ingestion.mdxindex 491ecfc0f..e9e0c5a80 100644--- a/docs/docs/core-components/ingestion.mdx+++ b/docs/docs/core-components/ingestion.mdx@@ -12,16 +12,16 @@ import PartialDockerComposeUp from '@site/docs/_partial-docker-compose-up.mdx';import PartialDockerStopAll from '@site/docs/_partial-docker-stop-all.mdx';import PartialOAuthCredentials from '@site/docs/_partial-oauth-credentials.mdx';-Upload documents to your [OpenRAG OpenSearch instance](/knowledge) to populate your knowledge base with unique content, such as your own company documents, research papers, or websites.+Upload documents to your [OpenRAG OpenSearch knowledge base](/knowledge) to populate your knowledge base with unique content, such as your own company documents, research papers, or websites.Documents are processed through OpenRAG's knowledge ingestion flows with Docling.OpenRAG can ingest knowledge from direct file uploads, URLs, and cloud storage connectors.-Knowledge ingestion is powered by OpenRAG's built-in knowledge ingestion flows that use Docling to process documents before storing the documents in your OpenSearch database.-During ingestion, documents are broken into smaller chunks of content that are then embedded using your selected [embedding model](/knowledge#set-the-embedding-model-and-dimensions).-Then, the chunks, embeddings, and associated metadata (which connects chunks of the same document) are stored in your OpenSearch database.+Knowledge ingestion is powered by OpenRAG's built-in knowledge ingestion flows that use Docling to process documents before storing the documents in your OpenSearch knowledge base.+During ingestion, documents are broken into smaller chunks of content that are then embedded using your selected [embedding model](/ingestion-configure#set-the-embedding-model-and-dimensions).+Then, the chunks, embeddings, and associated metadata (which connects chunks of the same document) are stored in your OpenSearch knowledge base.-To modify chunking behavior and other ingestion settings, see [Knowledge ingestion settings](/knowledge#knowledge-ingestion-settings) and [Inspect and modify flows](/agents#inspect-and-modify-flows).+To modify chunking behavior and other ingestion settings, see [Configure ingestion](/ingestion-configure) and [Inspect and modify flows](/agents#inspect-and-modify-flows).## Ingest local files and folders@@ -34,7 +34,7 @@ You can upload files and folders from your local machine to your knowledge base:3. To upload one file, click <Icon name="File" aria-hidden="true"/> **File**. To upload all documents in a folder, click <Icon name="Folder" aria-hidden="true"/> **Folder**.The default path is `~/.openrag/documents`.- To change this path, see [Set the local documents path](/knowledge#set-the-local-documents-path).+ To change this path, see [Set the local documents path](/ingestion-configure#set-the-local-documents-path).The selected files are processed in the background through the **OpenSearch Ingestion** flow.@@ -286,8 +286,10 @@ All errors were file-specific, and they didn't stop the pipeline.## See also-* [Configure knowledge](/knowledge)-* [Filter knowledge](/knowledge-filters)+* [About knowledge](/knowledge)+* [Configure ingestion](/ingestion-configure)+* [Browse and manage knowledge](/knowledge-browse)+* [Configure connectors](/knowledge-connectors)* [Chat with knowledge](/chat)* [Inspect and modify flows](/agents#inspect-and-modify-flows)* [Troubleshoot document ingestion or similarity search issues](/support/troubleshoot#document-ingestion-or-similarity-search-issues)\ No newline at end of filediff --git a/docs/docs/core-components/knowledge-browse.mdx b/docs/docs/core-components/knowledge-browse.mdxindex 36e8e9236..40f800b9d 100644--- a/docs/docs/core-components/knowledge-browse.mdx+++ b/docs/docs/core-components/knowledge-browse.mdx@@ -1,4 +1,85 @@---title: Browse and manage knowledgeslug: /knowledge-browse----\ No newline at end of file+---++import Icon from "@site/src/components/icon/icon";++The **Knowledge** page lists the documents OpenRAG has ingested into your [OpenSearch knowledge base](/knowledge), specifically in an [OpenSearch index](/knowledge#index) named `documents`.++To explore the raw contents of your knowledge base, click <Icon name="Library" aria-hidden="true"/> **Knowledge** to get a list of all ingested documents.++## Inspect knowledge++For each document, the **Knowledge** page provides the following information:++* **Source**: Name of the ingested content, such as the file name.++* **Size**++* **Type**++* **Owner**: User that uploaded the document.++ <PartialAnonymousUserOwner />++* **Chunks**: Number of chunks created by splitting the document during ingestion.++ Click a document to view the individual chunks and technical details related to chunking.+ If the chunks seem incorrect or incomplete, see [Troubleshoot document ingestion or similarity search issues](/support/troubleshoot#document-ingestion-or-similarity-search-issues).++* **Avg score**: Average similarity score across all chunks of the document.++ If you [search the knowledge base](#search-knowledge), the **Avg score** column shows the similarity score for your search query or filter.++* **Embedding model** and **Dimensions**: The embedding model and dimensions used to embed the chunks.++* **Status**: Status of document ingestion.+If ingestion is complete and successful, then the status is **Active**.+For more information, see [Monitor ingestion](/ingestion#monitor-ingestion).++## Search knowledge {#search-knowledge}++You can use the search field on the **Knowledge** page to find documents using semantic search and knowledge filters:++To search all documents, enter a search string in the search field, and then press <kbd>Enter</kbd>.++To apply a [knowledge filter](/knowledge-filters), select the filter from the **Knowledge Filters** list.+The filter settings pane opens, and the filter appears in the search field.+To remove the filter, close the filter settings pane or clear the filter from the search field.++You can use the filter alone or in combination with a search string.+If a knowledge filter has a **Search Query**, that query is applied in addition to any text string you enter in the search field.++Only one filter can be applied at a time.++## Default documents {#default-documents}++By default, OpenRAG includes some initial documents about OpenRAG.+These documents are ingested automatically during the [application onboarding process](/install#application-onboarding).++You can use these documents to ask OpenRAG about itself, or to test the [**Chat**](/chat) feature before uploading your own documents.++If you [delete these documents](#delete-knowledge), then you won't be able to ask OpenRAG about itself and it's own functionality.+It is recommended that you keep these documents, and use [filters](/knowledge-filters) to separate them from your other knowledge.+An **OpenRAG Docs** filter is created automatically for these documents.++## Delete knowledge {#delete-knowledge}++:::warning+This is a destructive operation that cannot be undone.+:::++To delete documents from your knowledge base, click <Icon name="Library" aria-hidden="true"/> **Knowledge**, use the checkboxes to select one or more documents, and then click **Delete**.+If you select the checkbox at the top of the list, all documents are selected and your entire knowledge base will be deleted.++To delete an individual document, you can also click <Icon name="Ellipsis" aria-hidden="true"/> **More** next to that document, and then select **Delete**.++To completely clear your entire knowledge base and OpenSearch index, [reset your OpenRAG containers](/manage-services#reset-containers) or [reinstall OpenRAG](/reinstall).++## See also++* [About knowledge](/knowledge)+* [Ingest knowledge](/ingestion)+* [Filter knowledge](/knowledge-filters)+* [Troubleshoot document ingestion or similarity search issues](/support/troubleshoot#document-ingestion-or-similarity-search-issues)\ No newline at end of filediff --git a/docs/docs/core-components/knowledge-connectors.mdx b/docs/docs/core-components/knowledge-connectors.mdxindex 415dce057..f9cda6afd 100644--- a/docs/docs/core-components/knowledge-connectors.mdx+++ b/docs/docs/core-components/knowledge-connectors.mdx@@ -3,6 +3,7 @@ title: Configure connectorsslug: /knowledge-connectors---+import Icon from "@site/src/components/icon/icon";<!--Proposed plan to reorganize the knowledge content again:@@ -34,10 +35,17 @@ See also: Local doc path setting, filter knowledge, ingest knowledgeFilter knowledge (existing page)Configure connectors (new page)-Oauth mode (maybe use a partial)+Oauth mode (maybe use a partial, maybe change the /knowledge#auth links to point here)Cloud storage connector setupTBD: disconnector/reconnect connectors, better separation of Oauth providers vs cloud storage accessSee also: About knowledge, ingest knowledgeRelated tasks: Check all links to the 3 existing pages, check all anchors and sections without dedicated anchors, check nav titles in sidebar.js.- -->\ No newline at end of file+ -->++## See also++* [About knowledge](/knowledge)+* [Ingest knowledge](/ingestion)+* [Configure ingestion](/ingestion-configure)+* [Browse and manage knowledge](/knowledge-browse)\ No newline at end of filediff --git a/docs/docs/core-components/knowledge-filters.mdx b/docs/docs/core-components/knowledge-filters.mdxindex e90fd882b..ac059216d 100644--- a/docs/docs/core-components/knowledge-filters.mdx+++ b/docs/docs/core-components/knowledge-filters.mdx@@ -75,7 +75,7 @@ On the filter settings pane, edit the filter as desired, and then click **UpdateIn the OpenRAG <Icon name="MessageSquare" aria-hidden="true"/> **Chat**, click <Icon name="Funnel" aria-hidden="true"/> **Filter**, and then select the filter to apply.Chat filters apply to one chat session only.-You can also use filters when [browsing your knowledge base](/knowledge#browse-knowledge).+You can also use filters when [browsing your knowledge base](/knowledge-browse).This is a helpful way to test filters and manage knowledge bases that have many documents.## Delete a filter@@ -84,4 +84,11 @@ This is a helpful way to test filters and manage knowledge bases that have many2. In the **Knowledge Filters** list, click the filter that you want to delete.-3. In the filter settings pane, click **Delete Filter**.\ No newline at end of file+3. In the filter settings pane, click **Delete Filter**.++## See also++* [About knowledge](/knowledge)+* [Browse and manage knowledge](/knowledge-browse)+* [Configure connectors](/knowledge-connectors)+* [Chat with knowledge](/chat)\ No newline at end of filediff --git a/docs/docs/core-components/knowledge.mdx b/docs/docs/core-components/knowledge.mdxindex db72a0b8e..3c85f2504 100644--- a/docs/docs/core-components/knowledge.mdx+++ b/docs/docs/core-components/knowledge.mdx@@ -6,6 +6,13 @@ slug: /knowledgeimport Icon from "@site/src/components/icon/icon";import PartialAnonymousUserOwner from '@site/docs/_partial-anonymous-user-owner.mdx';++## What is knowledge+## The role of OpenSearch+## The index (only 1, possibly soon easier to rename)+## Auth modes (maybe use a partial)++OpenRAG includes a built-in [OpenSearch](https://docs.opensearch.org/latest/) instance that serves as the underlying datastore for your _knowledge_ (documents).This specialized database is used to store and retrieve your documents and the associated vector data (embeddings).@@ -14,71 +21,10 @@ The documents in your OpenSearch knowledge base provide specialized context in aYou can [upload documents](/ingestion) from a variety of sources to populate your knowledge base with unique content, such as your own company documents, research papers, or websites.Documents are processed through OpenRAG's knowledge ingestion flows with Docling.-Then, the [OpenRAG **Chat**](/chat) can run [similarity searches](https://www.ibm.com/think/topics/vector-search) against your OpenSearch database to retrieve relevant information and generate context-aware responses.+Then, the [OpenRAG **Chat**](/chat) can run [similarity searches](https://www.ibm.com/think/topics/vector-search) against your OpenSearch knowledge base to retrieve relevant information and generate context-aware responses.You can configure how documents are ingested and how the **Chat** interacts with your knowledge base.-## Browse knowledge {#browse-knowledge}--The **Knowledge** page lists the documents OpenRAG has ingested into your OpenSearch database, specifically in an [OpenSearch index](https://docs.opensearch.org/latest/getting-started/intro/#index) named `documents`.--To explore the raw contents of your knowledge base, click <Icon name="Library" aria-hidden="true"/> **Knowledge** to get a list of all ingested documents.--### Inspect knowledge--For each document, the **Knowledge** page provides the following information:--* **Source**: Name of the ingested content, such as the file name.--* **Size**--* **Type**--* **Owner**: User that uploaded the document.-- <PartialAnonymousUserOwner />--* **Chunks**: Number of chunks created by splitting the document during ingestion.-- Click a document to view the individual chunks and technical details related to chunking.- If the chunks seem incorrect or incomplete, see [Troubleshoot document ingestion or similarity search issues](/support/troubleshoot#document-ingestion-or-similarity-search-issues).--* **Avg score**: Average similarity score across all chunks of the document.-- If you [search the knowledge base](#search-knowledge), the **Avg score** column shows the similarity score for your search query or filter.--* **Embedding model** and **Dimensions**: The embedding model and dimensions used to embed the chunks.--* **Status**: Status of document ingestion.-If ingestion is complete and successful, then the status is **Active**.-For more information, see [Monitor ingestion](/ingestion#monitor-ingestion).--### Search knowledge {#search-knowledge}--You can use the search field on the **Knowledge** page to find documents using semantic search and knowledge filters:--To search all documents, enter a search string in the search field, and then press <kbd>Enter</kbd>.--To apply a [knowledge filter](/knowledge-filters), select the filter from the **Knowledge Filters** list.-The filter settings pane opens, and the filter appears in the search field.-To remove the filter, close the filter settings pane or clear the filter from the search field.--You can use the filter alone or in combination with a search string.-If a knowledge filter has a **Search Query**, that query is applied in addition to any text string you enter in the search field.--Only one filter can be applied at a time.--### Default documents {#default-documents}--By default, OpenRAG includes some initial documents about OpenRAG.-These documents are ingested automatically during the [application onboarding process](/install#application-onboarding).--You can use these documents to ask OpenRAG about itself, or to test the [**Chat**](/chat) feature before uploading your own documents.--If you [delete these documents](#delete-knowledge), then you won't be able to ask OpenRAG about itself and it's own functionality.-It is recommended that you keep these documents, and use [filters](/knowledge-filters) to separate them from your other knowledge.-An **OpenRAG Docs** filter is created automatically for these documents.-## OpenSearch authentication and document access {#auth}When you [install OpenRAG](/install-options), you provide the initial configuration values for your OpenRAG services, including authentication credentials for OpenSearch and OAuth connectors.@@ -89,18 +35,18 @@ Google is the only supported OAuth provider for OpenRAG.Other [OAuth credentials](/reference/configuration#oauth-and-cloud-storage-connector-settings) are used only to authorize cloud storage connectors.:::-The presence of Google OAuth credentials determines how OpenRAG authenticates with your deployment's OpenSearch instance, and how it controls user access to documents in your knowledge base:+The presence of Google OAuth credentials determines how OpenRAG authenticates with your deployment's OpenSearch knowledge base, and how it controls user access to documents in your knowledge base:-* **No-auth mode**: If you don't provide Google OAuth credentials, then the OpenRAG OpenSearch instance runs in no-auth mode. This mode uses one anonymous JWT token for OpenSearch authentication. There is no differentiation between users; all users that access your OpenRAG instance can access all documents uploaded to your knowledge base.+* **No-auth mode**: If you don't provide Google OAuth credentials, then the OpenRAG OpenSearch knowledge base runs in no-auth mode. This mode uses one anonymous JWT token for OpenSearch authentication. There is no differentiation between users; all users that access your OpenRAG instance can access all documents uploaded to your knowledge base.-* **OAuth mode**: If you provide Google OAuth credentials, then the OpenRAG OpenSearch instance runs in OAuth mode. This mode uses a unique JWT token for each OpenRAG user, and each document is tagged with user ownership. Documents are filtered by user owner; users see only the documents that they uploaded or have access to through their cloud storage accounts.+* **OAuth mode**: If you provide Google OAuth credentials, then the OpenRAG OpenSearch knowledge base runs in OAuth mode. This mode uses a unique JWT token for each OpenRAG user, and each document is tagged with user ownership. Documents are filtered by user owner; users see only the documents that they uploaded or have access to through their cloud storage accounts.:::tipTo enable OAuth mode after initial setup, you must [configure the Google Drive cloud storage connector for document ingestion](/ingestion#oauth-ingestion).This is because OpenRAG uses Google OAuth credentials for both OAuth mode _and_ the Google Drive cloud storage connector.:::-## OpenSearch indexes+## OpenSearch index {#index}An [OpenSearch index](https://docs.opensearch.org/latest/getting-started/intro/#index) is a collection of documents in an OpenSearch database.@@ -110,121 +56,12 @@ It is possible to change the index name by [editing the ingestion flow](/agents#However, this can impact dependent processes, such as the [filters](/knowledge-filters) and [**Chat**](/chat), that reference the `documents` index by default.Make sure you edit other flows as needed to ensure all processes use the same index name.-If you encounter errors or unexpected behavior after changing the index name, you can [revert the flows to their original configuration](/agents#revert-a-built-in-flow-to-its-original-configuration), or [delete knowledge](/knowledge#delete-knowledge) to clear the existing documents from your knowledge base.--## Knowledge ingestion settings {#knowledge-ingestion-settings}--:::warning-Knowledge ingestion settings apply to documents you upload after making the changes.-Documents uploaded before changing these settings aren't reprocessed.-:::--After changing knowledge ingestion settings, you must determine if you need to reupload any documents to be consistent with the new settings.--It isn't always necessary to reupload documents after changing knowledge ingestion settings.-For example, it is typical to upload some documents with OCR enabled and others without OCR enabled.--If needed, you can use [filters](/knowledge-filters) to separate documents that you uploaded with different settings, such as different embedding models.--### Set the embedding model and dimensions {#set-the-embedding-model-and-dimensions}--When you [install OpenRAG](/install-options), you select at least one embedding model during the [application onboarding process](/install#application-onboarding).-OpenRAG automatically detects and configures the appropriate vector dimensions for your selected embedding model, ensuring optimal search performance and compatibility.--In the OpenRAG repository, you can find the complete list of supported models in [`models_service.py`](https://github.com/langflow-ai/openrag/blob/main/src/services/models_service.py) and the corresponding vector dimensions in [`settings.py`](https://github.com/langflow-ai/openrag/blob/main/src/config/settings.py).--During the application onboarding process, you can select from the supported models.-The default embedding dimension is `1536`, and the default model is the OpenAI `text-embedding-3-small`.--If you want to use an unsupported model, you must manually set the model in your [OpenRAG `.env` file](/reference/configuration).-If you use an unsupported embedding model that doesn't have defined dimensions in `settings.py`, then OpenRAG falls back to the default dimensions (1536) and logs a warning. OpenRAG's OpenSearch instance and flows continue to work, but [similarity search](https://www.ibm.com/think/topics/vector-search) quality can be affected if the actual model dimensions aren't 1536.--To change the embedding model after onboarding, modify the embedding model configuration on the OpenRAG **Settings** page or in your [OpenRAG `.env` file](/reference/configuration).-This ensures that all relevant [OpenRAG flows](/agents) are updated to use the new embedding model configuration.--If you edit these settings in the `.env` file, you must [stop and restart the OpenRAG containers](/manage-services#stop-and-start-containers) to apply the changes.--### Set Docling parameters--OpenRAG uses [Docling](https://docling-project.github.io/docling/) for document ingestion because it supports many file formats, processes tables and images well, and performs efficiently.--When you [upload documents](/ingestion), Docling processes the files, splits them into chunks, and stores them as separate, structured documents in your OpenSearch knowledge base.--#### Select a Docling implementation {#select-a-docling-implementation}--You can use either Docling Serve or OpenRAG's built-in Docling ingestion pipeline to process documents.--* **Docling Serve ingestion**: By default, OpenRAG uses [Docling Serve](https://github.com/docling-project/docling-serve).-It starts a local `docling serve` process, and then runs Docling ingestion through the Docling Serve API.-- To use a remote `docling serve` instance or your own local instance, set `DOCLING_SERVE_URL=http://**HOST_IP**:5001` in your [OpenRAG `.env` file](/reference/configuration#document-processing-settings).- The service must run on port 5001.--* **Built-in Docling ingestion**: If you want to use OpenRAG's built-in Docling ingestion pipeline instead of the separate Docling Serve service, set `DISABLE_INGEST_WITH_LANGFLOW=true` in your [OpenRAG `.env` file](/reference/configuration#document-processing-settings).-The built-in pipeline uses the Docling processor directly instead of through the Docling Serve API.-For the underlying functionality, see [`processors.py`](https://github.com/langflow-ai/openrag/blob/main/src/models/processors.py#L58) in the OpenRAG repository.--#### Configure Docling ingestion settings--To modify the Docling document processing and embedding parameters, click <Icon name="Settings2" aria-hidden="true"/> **Settings** in OpenRAG, and then find the **Knowledge Ingest** section.--:::tip-The TUI warns you if `docling serve` isn't running.-For information about starting and stopping OpenRAG native services, like Docling, see [Manage OpenRAG services](/manage-services).-:::--You can edit the following parameters:--* **Embedding model**: Select the model to use to generate vector embeddings for your documents.-- This is initially set during installation.- The recommended way to change this setting is in the OpenRAG <Icon name="Settings2" aria-hidden="true"/> **Settings** or your [OpenRAG `.env` file](/reference/configuration).- This ensures that all relevant [OpenRAG flows](/agents) are updated to use the new embedding model configuration.-- If you uploaded documents prior to changing the embedding model, you can [create filters](/knowledge-filters) to separate documents embedded with different models, or you can reupload all documents to regenerate embeddings with the new model.- If you want to use multiple embeddings models, similarity search (in the **Chat**) can take longer as it searches each model's embeddings separately.--* **Chunk size**: Set the number of characters for each text chunk when breaking down a file.-Larger chunks yield more context per chunk, but can include irrelevant information. Smaller chunks yield more precise semantic search, but can lack context.-The default value is 1000 characters, which is usually a good balance between context and precision.--* **Chunk overlap**: Set the number of characters to overlap over chunk boundaries.-Use larger overlap values for documents where context is most important. Use smaller overlap values for simpler documents or when optimization is most important.-The default value is 200 characters, which represents an overlap of 20 percent with the default **Chunk size** of 1000. This is suitable for general use. For faster processing, decrease the overlap to approximately 10 percent. For more complex documents where you need to preserve context across chunks, increase it to approximately 40 percent.--* **Table structure**: Enables Docling's [`DocumentConverter`](https://docling-project.github.io/docling/reference/document_converter/) tool for parsing tables. Instead of treating tables as plain text, tables are output as structured table data with preserved relationships and metadata. This option is enabled by default.--* **OCR**: Enables Optical Character Recognition (OCR) processing when extracting text from images and ingesting scanned documents. This setting is best suited for processing text-based documents faster with Docling's [`DocumentConverter`](https://docling-project.github.io/docling/reference/document_converter/). Images are ignored and not processed.-- This option is disabled by default. Enabling OCR can slow ingestion performance.-- If OpenRAG detects that the local machine is running on macOS, OpenRAG uses the [ocrmac](https://www.piwheels.org/project/ocrmac/) OCR engine. Other platforms use [easyocr](https://www.jaided.ai/easyocr/).--* **Picture descriptions**: Only applicable if **OCR** is enabled. Adds image descriptions generated by the [`SmolVLM-256M-Instruct`](https://huggingface.co/HuggingFaceTB/SmolVLM-Instruct) model. Enabling picture descriptions can slow ingestion performance.--### Set the local documents path {#set-the-local-documents-path}--The default path for local uploads is `~/.openrag/documents`. This is mounted to the `/app/openrag-documents/` directory inside the OpenRAG container. Files added to the host or container directory are visible in both locations.--To change this location, modify the **Documents Paths** variable in either the [**Basic/Advanced Setup** menu](/install#setup) or in your [OpenRAG `.env` file](/reference/configuration).--## Delete knowledge {#delete-knowledge}--:::warning-This is a destructive operation that cannot be undone.-:::--To delete documents from your knowledge base, click <Icon name="Library" aria-hidden="true"/> **Knowledge**, use the checkboxes to select one or more documents, and then click **Delete**.-If you select the checkbox at the top of the list, all documents are selected and your entire knowledge base will be deleted.--To delete an individual document, you can also click <Icon name="Ellipsis" aria-hidden="true"/> **More** next to that document, and then select **Delete**.--To completely clear your entire knowledge base and OpenSearch index, [reset your OpenRAG containers](/manage-services#reset-containers) or [reinstall OpenRAG](/reinstall).+If you encounter errors or unexpected behavior after changing the index name, you can [revert the flows to their original configuration](/agents#revert-a-built-in-flow-to-its-original-configuration), or [delete knowledge](/knowledge-browse#delete-knowledge) to clear the existing documents from your knowledge base.## See also* [Ingest knowledge](/ingestion)-* [Filter knowledge](/knowledge-filters)+* [Configure ingestion](/ingestion-configure)+* [Browse and manage knowledge](/knowledge-browse)* [Chat with knowledge](/chat)-* [Inspect and modify flows](/agents#inspect-and-modify-flows)* [Troubleshoot document ingestion or similarity search issues](/support/troubleshoot#document-ingestion-or-similarity-search-issues)\ No newline at end of filediff --git a/docs/docs/get-started/docker.mdx b/docs/docs/get-started/docker.mdxindex f5320bef3..44fe19cdd 100644--- a/docs/docs/get-started/docker.mdx+++ b/docs/docs/get-started/docker.mdx@@ -131,7 +131,7 @@ The following variables are required or recommended:Additionally, this enables the [MLX framework](https://opensource.apple.com/projects/mlx/) for accelerated performance on Apple Silicon Mac machines.:::tip- If you don't want to use the default Docling Serve implementation, see [Select a Docling implementation](/knowledge#select-a-docling-implementation).+ If you don't want to use the default Docling Serve implementation, see [Select a Docling implementation](/ingestion-configure#select-a-docling-implementation).:::2. Confirm `docling serve` is running.diff --git a/docs/docs/get-started/quickstart.mdx b/docs/docs/get-started/quickstart.mdxindex 880b59393..23bea110b 100644--- a/docs/docs/get-started/quickstart.mdx+++ b/docs/docs/get-started/quickstart.mdx@@ -90,8 +90,8 @@ For example: `What documents are available to you?`The agent responds with a summary of OpenRAG's default documents.-3. To verify the agent's response, click <Icon name="Library" aria-hidden="true"/> **Knowledge** to view the documents stored in the OpenRAG OpenSearch database.-You can click a document to view the chunks of the document as they are stored in the database.+3. To verify the agent's response, click <Icon name="Library" aria-hidden="true"/> **Knowledge** to view the documents stored in the [OpenRAG OpenSearch knowledge base](/knowledge).+You can click a document to view the chunks of the document as they are stored in the knowledge base.4. Click **Add Knowledge** to add your own documents to your OpenRAG knowledge base.diff --git a/docs/docs/reference/configuration.mdx b/docs/docs/reference/configuration.mdxindex ce82055c3..c22a231a1 100644--- a/docs/docs/reference/configuration.mdx+++ b/docs/docs/reference/configuration.mdx@@ -67,15 +67,15 @@ These variables are initially set during the [application onboarding process](/iControls some aspects of how OpenRAG [processes and ingests documents](/ingestion) into your knowledge base.-For the embedding model and Docling engine variables that you can set on the OpenRAG **Settings** page, see [Knowledge ingestion settings](/knowledge#knowledge-ingestion-settings).+For the embedding model and Docling engine variables that you can set on the OpenRAG **Settings** page, see [Configure ingestion](/ingestion-configure).For Langflow flow IDs and Langflow timeout settings, see [Langflow settings](#langflow-settings).| Variable | Default | Description ||----------|---------|-------------|-| `DISABLE_INGEST_WITH_LANGFLOW` | `false` | Disable Langflow ingestion pipeline if you don't want to use the default [Docling Serve instance](/knowledge#select-a-docling-implementation). |-| `DOCLING_SERVE_URL` | `http://HOST_IP:5001` | URL for the [Docling Serve instance](/knowledge#select-a-docling-implementation). By default, OpenRAG starts a local `docling serve` process and auto-detects the host. To use your own local or remote Docling Serve instance, set this variable to the full path to the target instance. The service must run on port 5001. |-| `OPENRAG_DOCUMENTS_PATH` | `~/.openrag/documents` | The [local documents path](/knowledge#set-the-local-documents-path) for ingestion. |+| `DISABLE_INGEST_WITH_LANGFLOW` | `false` | Disable Langflow ingestion pipeline if you don't want to use the default [Docling Serve instance](/ingestion-configure#select-a-docling-implementation). |+| `DOCLING_SERVE_URL` | `http://HOST_IP:5001` | URL for the [Docling Serve instance](/ingestion-configure#select-a-docling-implementation). By default, OpenRAG starts a local `docling serve` process and auto-detects the host. To use your own local or remote Docling Serve instance, set this variable to the full path to the target instance. The service must run on port 5001. |+| `OPENRAG_DOCUMENTS_PATH` | `~/.openrag/documents` | The [local documents path](/ingestion-configure#set-the-local-documents-path) for ingestion. || `HOST_DOCKER_INTERNAL` | `host.docker.internal` | Host address for accessing services running on the host machine from within Docker containers. This is used to connect to the local Docling Serve instance started by OpenRAG. If your system uses a different hostname for this purpose, set this variable accordingly. |## Langflow settings {#langflow-settings}diff --git a/docs/docs/support/troubleshoot.mdx b/docs/docs/support/troubleshoot.mdxindex 67c24528c..dcce6b9c7 100644--- a/docs/docs/support/troubleshoot.mdx+++ b/docs/docs/support/troubleshoot.mdx@@ -208,19 +208,19 @@ This cache can become stale, producing errors like missing dependencies.If you install OpenRAG with `uv`, dependencies are synced directly from your `pyproject.toml` file.This should automatically install `easyocr` because `easyocr` is included as a dependency in OpenRAG's `pyproject.toml`.-If you don't need OCR, you can disable OCR-based processing in your [ingestion settings](/knowledge#knowledge-ingestion-settings) to avoid requiring `easyocr`.+If you don't need OCR, you can disable OCR-based processing in your [ingestion settings](/ingestion-configure) to avoid requiring `easyocr`.### Problems when referencing documents in chat-If the OpenRAG **Chat** doesn't seem to use your documents correctly, [browse your knowledge base](/knowledge#browse-knowledge) to confirm that the documents are uploaded in full, and the chunks are correct.+If the OpenRAG **Chat** doesn't seem to use your documents correctly, [browse your knowledge base](/knowledge-browse) to confirm that the documents are uploaded in full, and the chunks are correct.If the documents are present and well-formed, check your [knowledge filters](/knowledge-filters).If you applied a filter to the chat, make sure the expected documents aren't excluded by the filter settings.-You can test this by applying the filter when you [browse the knowledge base](/knowledge#browse-knowledge).+You can test this by applying the filter when you [browse the knowledge base](/knowledge-browse).If the filter excludes any documents, the agent cannot access those documents.Be aware that some settings create dynamic filters that don't always produce the same results, such as a **Search query** combined with a low **Response limit**.-If the document chunks have missing, incorrect, or unexpected text, you must [delete the documents](/knowledge#delete-knowledge) from your knowledge base, modify the [ingestion parameters](/knowledge#knowledge-ingestion-settings) or the documents themselves, and then reingest the documents.+If the document chunks have missing, incorrect, or unexpected text, you must [delete the documents](/knowledge-browse#delete-knowledge) from your knowledge base, modify the [ingestion settings](/ingestion-configure) or the documents themselves, and then reingest the documents.For example:* Break combined documents into separate files for better metadata context.@@ -236,8 +236,8 @@ The following issues can occur when using the [OpenRAG **Chat**](/chat) feature.In the **Chat**, click <Icon name="Gear" aria-hidden="true"/> **Function Call: search_documents (tool_call)** to view the log of tool calls made by the agent.This can shows you how the agent used particular tools and the reasoning.-Click <Icon name="Library" aria-hidden="true"/> **Knowledge** to confirm that the documents are present in the OpenRAG OpenSearch database, and then click each document to see how the document was chunked.-If a document was chunked improperly, you might need to tweak the [ingestion settings](/ingestion) or modify and reupload the document.+Click <Icon name="Library" aria-hidden="true"/> **Knowledge** to confirm that the documents are present in the OpenRAG OpenSearch knowledge base, and then click each document to see how the document was chunked.+If a document was chunked improperly, you might need to tweak the [ingestion settings](/ingestion-configure) or modify and reupload the document.See also [Document ingestion or similarity search issues](#document-ingestion-or-similarity-search-issues).