The agent engineering platform.
Python agentsaiai-agentsanthropicchatgptdeepagentsenterpriseframeworkgeminigenerative-ailangchainlanggraphllmmultiagentopen-sourceopenaipydanticpythonragtypescript
community[minor]: added new document loaders based on dedoc library (#24303)
Alexander Golodkov2a70a07aad61
diff --git a/docs/docs/integrations/document_loaders/dedoc.ipynb b/docs/docs/integrations/document_loaders/dedoc.ipynbnew file mode 100644index 0000000000..084537141f--- /dev/null+++ b/docs/docs/integrations/document_loaders/dedoc.ipynb@@ -0,0 +1,484 @@+{+ "cells": [+ {+ "cell_type": "markdown",+ "id": "6b74f73d-1763-42d0-9c24-8f65f445bb72",+ "metadata": {},+ "source": [+ "# Dedoc\n",+ "\n",+ "This sample demonstrates the use of `Dedoc` in combination with `LangChain` as a `DocumentLoader`.\n",+ "\n",+ "## Overview\n",+ "\n",+ "[Dedoc](https://dedoc.readthedocs.io) is an [open-source](https://github.com/ispras/dedoc)\n",+ "library/service that extracts texts, tables, attached files and document structure\n",+ "(e.g., titles, list items, etc.) from files of various formats.\n",+ "\n",+ "`Dedoc` supports `DOCX`, `XLSX`, `PPTX`, `EML`, `HTML`, `PDF`, images and more.\n",+ "Full list of supported formats can be found [here](https://dedoc.readthedocs.io/en/latest/#id1).\n",+ "\n",+ "\n",+ "### Integration details\n",+ "\n",+ "| Class | Package | Local | Serializable | JS support |\n",+ "|:-----------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------|:-----:|:------------:|:----------:|\n",+ "| [DedocFileLoader](https://api.python.langchain.com/en/latest/document_loaders/langchain_community.document_loaders.dedoc.DedocFileLoader.html) | [langchain_community](https://api.python.langchain.com/en/latest/community_api_reference.html) | ❌ | beta | ❌ |\n",+ "| [DedocPDFLoader](https://api.python.langchain.com/en/latest/document_loaders/langchain_community.document_loaders.pdf.DedocPDFLoader.html) | [langchain_community](https://api.python.langchain.com/en/latest/community_api_reference.html) | ❌ | beta | ❌ | \n",+ "| [DedocAPIFileLoader](https://api.python.langchain.com/en/latest/document_loaders/langchain_community.document_loaders.dedoc.DedocAPIFileLoader.html) | [langchain_community](https://api.python.langchain.com/en/latest/community_api_reference.html) | ❌ | beta | ❌ | \n",+ "\n",+ "\n",+ "### Loader features\n",+ "\n",+ "Methods for lazy loading and async loading are available, but in fact, document loading is executed synchronously.\n",+ "\n",+ "| Source | Document Lazy Loading | Async Support |\n",+ "|:------------------:|:---------------------:|:-------------:| \n",+ "| DedocFileLoader | ❌ | ❌ |\n",+ "| DedocPDFLoader | ❌ | ❌ | \n",+ "| DedocAPIFileLoader | ❌ | ❌ | \n",+ "\n",+ "## Setup\n",+ "\n",+ "* To access `DedocFileLoader` and `DedocPDFLoader` document loaders, you'll need to install the `dedoc` integration package.\n",+ "* To access `DedocAPIFileLoader`, you'll need to run the `Dedoc` service, e.g. `Docker` container (please see [the documentation](https://dedoc.readthedocs.io/en/latest/getting_started/installation.html#install-and-run-dedoc-using-docker) \n",+ "for more details):\n",+ "\n",+ "```bash\n",+ "docker pull dedocproject/dedoc\n",+ "docker run -p 1231:1231\n",+ "```\n",+ "\n",+ "`Dedoc` installation instruction is given [here](https://dedoc.readthedocs.io/en/latest/getting_started/installation.html)."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 1,+ "id": "511c109d-a5c3-42ba-914e-5d1b385bc40f",+ "metadata": {},+ "outputs": [+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "Note: you may need to restart the kernel to use updated packages.\n"+ ]+ }+ ],+ "source": [+ "# Install package\n",+ "%pip install --quiet \"dedoc[torch]\""+ ]+ },+ {+ "cell_type": "markdown",+ "id": "6820c0e9-d56d-4899-b8c8-374760360e2b",+ "metadata": {},+ "source": [+ "## Instantiation"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 2,+ "id": "c1f98cae-71ec-4d60-87fb-96c1a76851d8",+ "metadata": {},+ "outputs": [],+ "source": [+ "from langchain_community.document_loaders import DedocFileLoader\n",+ "\n",+ "loader = DedocFileLoader(\"./example_data/state_of_the_union.txt\")"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "5d7bc2b3-73a0-4cd6-8014-cc7184aa9d4a",+ "metadata": {},+ "source": [+ "## Load"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 3,+ "id": "b9097c14-6168-4726-819e-24abb9a63b13",+ "metadata": {},+ "outputs": [+ {+ "data": {+ "text/plain": [+ "'\\nMadam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and t'"+ ]+ },+ "execution_count": 3,+ "metadata": {},+ "output_type": "execute_result"+ }+ ],+ "source": [+ "docs = loader.load()\n",+ "docs[0].page_content[:100]"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "9ed8bd46-0047-4ccc-b2d6-beb7761f7312",+ "metadata": {},+ "source": [+ "## Lazy Load"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 4,+ "id": "6ae12d7e-8105-4bbe-9031-0e968475f6bf",+ "metadata": {},+ "outputs": [+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "\n",+ "Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and t\n"+ ]+ }+ ],+ "source": [+ "docs = loader.lazy_load()\n",+ "\n",+ "for doc in docs:\n",+ " print(doc.page_content[:100])\n",+ " break"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "8772ae40-6239-4751-bb2d-b4a9415c1ad1",+ "metadata": {},+ "source": [+ "## API reference\n",+ "\n",+ "For detailed information on configuring and calling `Dedoc` loaders, please see the API references: \n",+ "\n",+ "* https://api.python.langchain.com/en/latest/document_loaders/langchain_community.document_loaders.dedoc.DedocFileLoader.html\n",+ "* https://api.python.langchain.com/en/latest/document_loaders/langchain_community.document_loaders.pdf.DedocPDFLoader.html\n",+ "* https://api.python.langchain.com/en/latest/document_loaders/langchain_community.document_loaders.dedoc.DedocAPIFileLoader.html"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "c4d5e702-0e21-4cad-a4c3-b9b3bff77203",+ "metadata": {},+ "source": [+ "## Loading any file\n",+ "\n",+ "For automatic handling of any file in a [supported format](https://dedoc.readthedocs.io/en/latest/#id1),\n",+ "`DedocFileLoader` can be useful.\n",+ "The file loader automatically detects the file type with a correct extension.\n",+ "\n",+ "File parsing process can be configured through `dedoc_kwargs` during the `DedocFileLoader` class initialization.\n",+ "Here the basic examples of some options usage are given, \n",+ "please see the documentation of `DedocFileLoader` and \n",+ "[dedoc documentation](https://dedoc.readthedocs.io/en/latest/parameters/parameters.html) \n",+ "to get more details about configuration parameters."+ ]+ },+ {+ "cell_type": "markdown",+ "id": "de97d0ed-d6b1-44e0-b392-1f3d89c762f9",+ "metadata": {},+ "source": [+ "### Basic example"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 5,+ "id": "50ffeeee-db12-4801-b208-7e32ea3d72ad",+ "metadata": {},+ "outputs": [+ {+ "data": {+ "text/plain": [+ "'\\nMadam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans. \\n\\n\\n\\nLast year COVID-19 kept us apart. This year we are finally together again. \\n\\n\\n\\nTonight, we meet as Democrats Republicans and Independents. But most importantly as Americans. \\n\\n\\n\\nWith a duty to one another to the American people to '"+ ]+ },+ "execution_count": 5,+ "metadata": {},+ "output_type": "execute_result"+ }+ ],+ "source": [+ "from langchain_community.document_loaders import DedocFileLoader\n",+ "\n",+ "loader = DedocFileLoader(\"./example_data/state_of_the_union.txt\")\n",+ "\n",+ "docs = loader.load()\n",+ "\n",+ "docs[0].page_content[:400]"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "457e5d4c-a4ee-4f31-ae74-3f75a1bbd0af",+ "metadata": {},+ "source": [+ "### Modes of split\n",+ "\n",+ "`DedocFileLoader` supports different types of document splitting into parts (each part is returned separately).\n",+ "For this purpose, `split` parameter is used with the following options:\n",+ "* `document` (default value): document text is returned as a single langchain `Document` object (don't split);\n",+ "* `page`: split document text into pages (works for `PDF`, `DJVU`, `PPTX`, `PPT`, `ODP`);\n",+ "* `node`: split document text into `Dedoc` tree nodes (title nodes, list item nodes, raw text nodes);\n",+ "* `line`: split document text into textual lines."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 6,+ "id": "eec54d31-ae7a-4a3c-aa10-4ae276b1e4c4",+ "metadata": {},+ "outputs": [+ {+ "data": {+ "text/plain": [+ "2"+ ]+ },+ "execution_count": 6,+ "metadata": {},+ "output_type": "execute_result"+ }+ ],+ "source": [+ "loader = DedocFileLoader(\n",+ " \"./example_data/layout-parser-paper.pdf\",\n",+ " split=\"page\",\n",+ " pages=\":2\",\n",+ ")\n",+ "\n",+ "docs = loader.load()\n",+ "\n",+ "len(docs)"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "61e11769-4780-4f77-b10e-27db6936f226",+ "metadata": {},+ "source": [+ "### Handling tables\n",+ "\n",+ "`DedocFileLoader` supports tables handling when `with_tables` parameter is \n",+ "set to `True` during loader initialization (`with_tables=True` by default). \n",+ "\n",+ "Tables are not split - each table corresponds to one langchain `Document` object.\n",+ "For tables, `Document` object has additional `metadata` fields `type=\"table\"` \n",+ "and `text_as_html` with table `HTML` representation."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 7,+ "id": "bbeb2f8a-ac5e-4b59-8026-7ea3fc14c928",+ "metadata": {},+ "outputs": [+ {+ "data": {+ "text/plain": [+ "('table',\n",+ " '<table border=\"1\" style=\"border-collapse: collapse; width: 100%;\">\\n<tbody>\\n<tr>\\n<td colspan=\"1\" rowspan=\"1\">Team</td>\\n<td colspan=\"1\" rowspan=\"1\"> "Payroll (millions)"</td>\\n<td colspan=\"1\" r')"+ ]+ },+ "execution_count": 7,+ "metadata": {},+ "output_type": "execute_result"+ }+ ],+ "source": [+ "loader = DedocFileLoader(\"./example_data/mlb_teams_2012.csv\")\n",+ "\n",+ "docs = loader.load()\n",+ "\n",+ "docs[1].metadata[\"type\"], docs[1].metadata[\"text_as_html\"][:200]"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "b4a2b872-2aba-4e4c-8b2f-83a5a81ee1da",+ "metadata": {},+ "source": [+ "### Handling attached files\n",+ "\n",+ "`DedocFileLoader` supports attached files handling when `with_attachments` is set \n",+ "to `True` during loader initialization (`with_attachments=False` by default). \n",+ "\n",+ "Attachments are split according to the `split` parameter.\n",+ "For attachments, langchain `Document` object has an additional metadata \n",+ "field `type=\"attachment\"`."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 8,+ "id": "bb9d6c1c-e24c-4979-88a0-38d54abd6332",+ "metadata": {},+ "outputs": [+ {+ "data": {+ "text/plain": [+ "('attachment',\n",+ " '\\nContent-Type\\nmultipart/mixed; boundary=\"0000000000005d654405f082adb7\"\\nDate\\nFri, 23 Dec 2022 12:08:48 -0600\\nFrom\\nMallori Harrell <[email protected]>\\nMIME-Version\\n1.0\\nMessage-ID\\n<CAPgNNXSzLVJ-d1OCX_TjFgJU7ugtQrjFybPtAMmmYZzphxNFYg@mail.gmail.com>\\nSubject\\nFake email with attachment\\nTo\\nMallori Harrell <[email protected]>')"+ ]+ },+ "execution_count": 8,+ "metadata": {},+ "output_type": "execute_result"+ }+ ],+ "source": [+ "loader = DedocFileLoader(\n",+ " \"./example_data/fake-email-attachment.eml\",\n",+ " with_attachments=True,\n",+ ")\n",+ "\n",+ "docs = loader.load()\n",+ "\n",+ "docs[1].metadata[\"type\"], docs[1].page_content"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "d435c3f6-703a-4064-8307-ace140de967a",+ "metadata": {},+ "source": [+ "## Loading PDF file\n",+ "\n",+ "If you want to handle only `PDF` documents, you can use `DedocPDFLoader` with only `PDF` support.\n",+ "The loader supports the same parameters for document split, tables and attachments extraction.\n",+ "\n",+ "`Dedoc` can extract `PDF` with or without a textual layer, \n",+ "as well as automatically detect its presence and correctness.\n",+ "Several `PDF` handlers are available, you can use `pdf_with_text_layer` \n",+ "parameter to choose one of them.\n",+ "Please see [parameters description](https://dedoc.readthedocs.io/en/latest/parameters/pdf_handling.html) \n",+ "to get more details.\n",+ "\n",+ "For `PDF` without a textual layer, `Tesseract OCR` and its language packages should be installed.\n",+ "In this case, [the instruction](https://dedoc.readthedocs.io/en/latest/tutorials/add_new_language.html) can be useful."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 9,+ "id": "0103a7f3-6b5e-4444-8f4d-83dd3724a9af",+ "metadata": {},+ "outputs": [+ {+ "data": {+ "text/plain": [+ "'\\n2\\n\\nZ. Shen et al.\\n\\n37], layout detection [38, 22], table detection [26], and scene text detection [4].\\n\\nA generalized learning-based framework dramatically reduces the need for the\\n\\nmanual specification of complicated rules, which is the status quo with traditional\\n\\nmethods. DL has the potential to transform DIA pipelines and benefit a broad\\n\\nspectrum of large-scale document digitization projects.\\n'"+ ]+ },+ "execution_count": 9,+ "metadata": {},+ "output_type": "execute_result"+ }+ ],+ "source": [+ "from langchain_community.document_loaders import DedocPDFLoader\n",+ "\n",+ "loader = DedocPDFLoader(\n",+ " \"./example_data/layout-parser-paper.pdf\", pdf_with_text_layer=\"true\", pages=\"2:2\"\n",+ ")\n",+ "\n",+ "docs = loader.load()\n",+ "\n",+ "docs[0].page_content[:400]"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "13061995-1805-40c2-a77a-a6cd80999e20",+ "metadata": {},+ "source": [+ "## Dedoc API\n",+ "\n",+ "If you want to get up and running with less set up, you can use `Dedoc` as a service.\n",+ "**`DedocAPIFileLoader` can be used without installation of `dedoc` library.**\n",+ "The loader supports the same parameters as `DedocFileLoader` and\n",+ "also automatically detects input file types.\n",+ "\n",+ "To use `DedocAPIFileLoader`, you should run the `Dedoc` service, e.g. `Docker` container (please see [the documentation](https://dedoc.readthedocs.io/en/latest/getting_started/installation.html#install-and-run-dedoc-using-docker) \n",+ "for more details):\n",+ "\n",+ "```bash\n",+ "docker pull dedocproject/dedoc\n",+ "docker run -p 1231:1231\n",+ "```\n",+ "\n",+ "Please do not use our demo URL `https://dedoc-readme.hf.space` in your code."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 10,+ "id": "211fc0b5-6080-4974-a6c1-f982bafd87d6",+ "metadata": {},+ "outputs": [+ {+ "data": {+ "text/plain": [+ "'\\nMadam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans. \\n\\n\\n\\nLast year COVID-19 kept us apart. This year we are finally together again. \\n\\n\\n\\nTonight, we meet as Democrats Republicans and Independents. But most importantly as Americans. \\n\\n\\n\\nWith a duty to one another to the American people to '"+ ]+ },+ "execution_count": 10,+ "metadata": {},+ "output_type": "execute_result"+ }+ ],+ "source": [+ "from langchain_community.document_loaders import DedocAPIFileLoader\n",+ "\n",+ "loader = DedocAPIFileLoader(\n",+ " \"./example_data/state_of_the_union.txt\",\n",+ " url=\"https://dedoc-readme.hf.space\",\n",+ ")\n",+ "\n",+ "docs = loader.load()\n",+ "\n",+ "docs[0].page_content[:400]"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "id": "faaff475-5209-436f-bcde-97d58daed05c",+ "metadata": {},+ "outputs": [],+ "source": []+ }+ ],+ "metadata": {+ "kernelspec": {+ "display_name": "Python 3 (ipykernel)",+ "language": "python",+ "name": "python3"+ },+ "language_info": {+ "codemirror_mode": {+ "name": "ipython",+ "version": 3+ },+ "file_extension": ".py",+ "mimetype": "text/x-python",+ "name": "python",+ "nbconvert_exporter": "python",+ "pygments_lexer": "ipython3",+ "version": "3.9.19"+ }+ },+ "nbformat": 4,+ "nbformat_minor": 5+}diff --git a/docs/docs/integrations/providers/dedoc.mdx b/docs/docs/integrations/providers/dedoc.mdxnew file mode 100644index 0000000000..3f2aaa206e--- /dev/null+++ b/docs/docs/integrations/providers/dedoc.mdx@@ -0,0 +1,56 @@+# Dedoc++>[Dedoc](https://dedoc.readthedocs.io) is an [open-source](https://github.com/ispras/dedoc)+library/service that extracts texts, tables, attached files and document structure+(e.g., titles, list items, etc.) from files of various formats.++`Dedoc` supports `DOCX`, `XLSX`, `PPTX`, `EML`, `HTML`, `PDF`, images and more.+Full list of supported formats can be found [here](https://dedoc.readthedocs.io/en/latest/#id1).++## Installation and Setup++### Dedoc library++You can install `Dedoc` using `pip`.+In this case, you will need to install dependencies,+please go [here](https://dedoc.readthedocs.io/en/latest/getting_started/installation.html)+to get more information.++```bash+pip install dedoc+```++### Dedoc API++If you are going to use `Dedoc` API, you don't need to install `dedoc` library.+In this case, you should run the `Dedoc` service, e.g. `Docker` container (please see+[the documentation](https://dedoc.readthedocs.io/en/latest/getting_started/installation.html#install-and-run-dedoc-using-docker)+for more details):++```bash+docker pull dedocproject/dedoc+docker run -p 1231:1231+```++## Document Loader++* For handling files of any formats (supported by `Dedoc`), you can use `DedocFileLoader`:++ ```python+ from langchain_community.document_loaders import DedocFileLoader+ ```++* For handling PDF files (with or without a textual layer), you can use `DedocPDFLoader`:++ ```python+ from langchain_community.document_loaders import DedocPDFLoader+ ```++* For handling files of any formats without library installation,+you can use `Dedoc API` with `DedocAPIFileLoader`:++ ```python+ from langchain_community.document_loaders import DedocAPIFileLoader+ ```++Please see a [usage example](/docs/integrations/document_loaders/dedoc) for more details.diff --git a/libs/community/extended_testing_deps.txt b/libs/community/extended_testing_deps.txtindex 198e8954f8..55335cad97 100644--- a/libs/community/extended_testing_deps.txt+++ b/libs/community/extended_testing_deps.txt@@ -16,6 +16,7 @@ cloudpickle>=2.0.0cohere>=4,<6databricks-vectorsearch>=0.21,<0.22datasets>=2.15.0,<3+dedoc>=2.2.6,<3dgml-utils>=0.3.0,<0.4elasticsearch>=8.12.0,<9esprima>=4.0.1,<5diff --git a/libs/community/langchain_community/document_loaders/__init__.py b/libs/community/langchain_community/document_loaders/__init__.pyindex 7f111b1aae..e03f769312 100644--- a/libs/community/langchain_community/document_loaders/__init__.py+++ b/libs/community/langchain_community/document_loaders/__init__.py@@ -142,6 +142,10 @@ if TYPE_CHECKING:from langchain_community.document_loaders.dataframe import (DataFrameLoader,)+ from langchain_community.document_loaders.dedoc import (+ DedocAPIFileLoader,+ DedocFileLoader,+ )from langchain_community.document_loaders.diffbot import (DiffbotLoader,)@@ -340,6 +344,7 @@ if TYPE_CHECKING:)from langchain_community.document_loaders.pdf import (AmazonTextractPDFLoader,+ DedocPDFLoader,MathpixPDFLoader,OnlinePDFLoader,PagedPDFSplitter,@@ -570,6 +575,9 @@ _module_lookup = {"CubeSemanticLoader": "langchain_community.document_loaders.cube_semantic","DataFrameLoader": "langchain_community.document_loaders.dataframe","DatadogLogsLoader": "langchain_community.document_loaders.datadog_logs",+ "DedocAPIFileLoader": "langchain_community.document_loaders.dedoc",+ "DedocFileLoader": "langchain_community.document_loaders.dedoc",+ "DedocPDFLoader": "langchain_community.document_loaders.pdf","DiffbotLoader": "langchain_community.document_loaders.diffbot","DirectoryLoader": "langchain_community.document_loaders.directory","DiscordChatLoader": "langchain_community.document_loaders.discord",@@ -771,6 +779,9 @@ __all__ = ["CubeSemanticLoader","DataFrameLoader","DatadogLogsLoader",+ "DedocAPIFileLoader",+ "DedocFileLoader",+ "DedocPDFLoader","DiffbotLoader","DirectoryLoader","DiscordChatLoader",diff --git a/libs/community/langchain_community/document_loaders/dedoc.py b/libs/community/langchain_community/document_loaders/dedoc.pynew file mode 100644index 0000000000..ed8ebc11d7--- /dev/null+++ b/libs/community/langchain_community/document_loaders/dedoc.py@@ -0,0 +1,546 @@+import html+import json+import os+from abc import ABC, abstractmethod+from typing import (+ Dict,+ Iterator,+ Optional,+ Tuple,+ Union,+)++from langchain_core.documents import Document++from langchain_community.document_loaders.base import BaseLoader+++class DedocBaseLoader(BaseLoader, ABC):+ """+ Base Loader that uses `dedoc` (https://dedoc.readthedocs.io).++ Loader enables extracting text, tables and attached files from the given file:+ * `Text` can be split by pages, `dedoc` tree nodes, textual lines+ (according to the `split` parameter).+ * `Attached files` (when with_attachments=True)+ are split according to the `split` parameter.+ For attachments, langchain Document object has an additional metadata field+ `type`="attachment".+ * `Tables` (when with_tables=True) are not split - each table corresponds to one+ langchain Document object.+ For tables, Document object has additional metadata fields `type`="table"+ and `text_as_html` with table HTML representation.+ """++ def __init__(+ self,+ file_path: str,+ *,+ split: str = "document",+ with_tables: bool = True,+ with_attachments: Union[str, bool] = False,+ recursion_deep_attachments: int = 10,+ pdf_with_text_layer: str = "auto_tabby",+ language: str = "rus+eng",+ pages: str = ":",+ is_one_column_document: str = "auto",+ document_orientation: str = "auto",+ need_header_footer_analysis: Union[str, bool] = False,+ need_binarization: Union[str, bool] = False,+ need_pdf_table_analysis: Union[str, bool] = True,+ delimiter: Optional[str] = None,+ encoding: Optional[str] = None,+ ) -> None:+ """+ Initialize with file path and parsing parameters.++ Args:+ file_path: path to the file for processing+ split: type of document splitting into parts (each part is returned+ separately), default value "document"+ "document": document text is returned as a single langchain Document+ object (don't split)+ "page": split document text into pages (works for PDF, DJVU, PPTX, PPT,+ ODP)+ "node": split document text into tree nodes (title nodes, list item+ nodes, raw text nodes)+ "line": split document text into lines+ with_tables: add tables to the result - each table is returned as a single+ langchain Document object++ Parameters used for document parsing via `dedoc`+ (https://dedoc.readthedocs.io/en/latest/parameters/parameters.html):++ with_attachments: enable attached files extraction+ recursion_deep_attachments: recursion level for attached files+ extraction, works only when with_attachments==True+ pdf_with_text_layer: type of handler for parsing PDF documents,+ available options+ ["true", "false", "tabby", "auto", "auto_tabby" (default)]+ language: language of the document for PDF without a textual layer and+ images, available options ["eng", "rus", "rus+eng" (default)],+ the list of languages can be extended, please see+ https://dedoc.readthedocs.io/en/latest/tutorials/add_new_language.html+ pages: page slice to define the reading range for parsing PDF documents+ is_one_column_document: detect number of columns for PDF without+ a textual layer and images, available options+ ["true", "false", "auto" (default)]+ document_orientation: fix document orientation (90, 180, 270 degrees)+ for PDF without a textual layer and images, available options+ ["auto" (default), "no_change"]+ need_header_footer_analysis: remove headers and footers from the output+ result for parsing PDF and images+ need_binarization: clean pages background (binarize) for PDF without a+ textual layer and images+ need_pdf_table_analysis: parse tables for PDF without a textual layer+ and images+ delimiter: column separator for CSV, TSV files+ encoding: encoding of TXT, CSV, TSV+ """+ self.parsing_parameters = {+ key: value+ for key, value in locals().items()+ if key not in {"self", "file_path", "split", "with_tables"}+ }+ self.valid_split_values = {"document", "page", "node", "line"}+ if split not in self.valid_split_values:+ raise ValueError(+ f"Got {split} for `split`, but should be one of "+ f"`{self.valid_split_values}`"+ )+ self.split = split+ self.with_tables = with_tables+ self.file_path = file_path++ structure_type = "tree" if self.split == "node" else "linear"+ self.parsing_parameters["structure_type"] = structure_type+ self.parsing_parameters["need_content_analysis"] = with_attachments++ def lazy_load(self) -> Iterator[Document]:+ """Lazily load documents."""+ import tempfile++ try:+ from dedoc import DedocManager+ except ImportError:+ raise ImportError(+ "`dedoc` package not found, please install it with `pip install dedoc`"+ )+ dedoc_manager = DedocManager(manager_config=self._make_config())+ dedoc_manager.config["logger"].disabled = True++ with tempfile.TemporaryDirectory() as tmpdir:+ document_tree = dedoc_manager.parse(+ file_path=self.file_path,+ parameters={**self.parsing_parameters, "attachments_dir": tmpdir},+ )+ yield from self._split_document(+ document_tree=document_tree.to_api_schema().dict(), split=self.split+ )++ @abstractmethod+ def _make_config(self) -> dict:+ """+ Make configuration for DedocManager according to the file extension and+ parsing parameters.+ """+ pass++ def _json2txt(self, paragraph: dict) -> str:+ """Get text (recursively) of the document tree node."""+ subparagraphs_text = "\n".join(+ [+ self._json2txt(subparagraph)+ for subparagraph in paragraph["subparagraphs"]+ ]+ )+ text = (+ f"{paragraph['text']}\n{subparagraphs_text}"+ if subparagraphs_text+ else paragraph["text"]+ )+ return text++ def _parse_subparagraphs(+ self, document_tree: dict, document_metadata: dict+ ) -> Iterator[Document]:+ """Parse recursively document tree obtained by `dedoc`."""+ if len(document_tree["subparagraphs"]) > 0:+ for subparagraph in document_tree["subparagraphs"]:+ yield from self._parse_subparagraphs(+ document_tree=subparagraph, document_metadata=document_metadata+ )+ else:+ yield Document(+ page_content=document_tree["text"],+ metadata={**document_metadata, **document_tree["metadata"]},+ )++ def _split_document(+ self,+ document_tree: dict,+ split: str,+ additional_metadata: Optional[dict] = None,+ ) -> Iterator[Document]:+ """Split document into parts according to the `split` parameter."""+ document_metadata = document_tree["metadata"]+ if additional_metadata:+ document_metadata = {**document_metadata, **additional_metadata}++ if split == "document":+ text = self._json2txt(paragraph=document_tree["content"]["structure"])+ yield Document(page_content=text, metadata=document_metadata)++ elif split == "page":+ nodes = document_tree["content"]["structure"]["subparagraphs"]+ page_id = nodes[0]["metadata"]["page_id"]+ page_text = ""++ for node in nodes:+ if node["metadata"]["page_id"] == page_id:+ page_text += self._json2txt(node)+ else:+ yield Document(+ page_content=page_text,+ metadata={**document_metadata, "page_id": page_id},+ )+ page_id = node["metadata"]["page_id"]+ page_text = self._json2txt(node)++ yield Document(+ page_content=page_text,+ metadata={**document_metadata, "page_id": page_id},+ )++ elif split == "line":+ for node in document_tree["content"]["structure"]["subparagraphs"]:+ line_metadata = node["metadata"]+ yield Document(+ page_content=self._json2txt(node),+ metadata={**document_metadata, **line_metadata},+ )++ elif split == "node":+ yield from self._parse_subparagraphs(+ document_tree=document_tree["content"]["structure"],+ document_metadata=document_metadata,+ )++ else:+ raise ValueError(+ f"Got {split} for `split`, but should be one of "+ f"`{self.valid_split_values}`"+ )++ if self.with_tables:+ for table in document_tree["content"]["tables"]:+ table_text, table_html = self._get_table(table)+ yield Document(+ page_content=table_text,+ metadata={+ **table["metadata"],+ "type": "table",+ "text_as_html": table_html,+ },+ )++ for attachment in document_tree["attachments"]:+ yield from self._split_document(+ document_tree=attachment,+ split=self.split,+ additional_metadata={"type": "attachment"},+ )++ def _get_table(self, table: dict) -> Tuple[str, str]:+ """Get text and HTML representation of the table."""+ table_text = ""+ for row in table["cells"]:+ for cell in row:+ table_text += " ".join(line["text"] for line in cell["lines"])+ table_text += "\t"+ table_text += "\n"++ table_html = (+ '<table border="1" style="border-collapse: collapse; width: 100%;'+ '">\n<tbody>\n'+ )+ for row in table["cells"]:+ table_html += "<tr>\n"+ for cell in row:+ cell_text = "\n".join(line["text"] for line in cell["lines"])+ cell_text = html.escape(cell_text)+ table_html += "<td"+ if cell["invisible"]:+ table_html += ' style="display: none" '+ table_html += (+ f' colspan="{cell["colspan"]}" rowspan='+ f'"{cell["rowspan"]}">{cell_text}</td>\n'+ )+ table_html += "</tr>\n"+ table_html += "</tbody>\n</table>"++ return table_text, table_html+++class DedocFileLoader(DedocBaseLoader):+ """+ DedocFileLoader document loader integration to load files using `dedoc`.++ The file loader automatically detects the file type (with the correct extension).+ The list of supported file types is gives at+ https://dedoc.readthedocs.io/en/latest/index.html#id1.+ Please see the documentation of DedocBaseLoader to get more details.++ Setup:+ Install ``dedoc`` package.++ .. code-block:: bash++ pip install -U dedoc++ Instantiate:+ .. code-block:: python++ from langchain_community.document_loaders import DedocFileLoader++ loader = DedocFileLoader(+ file_path="example.pdf",+ # split=...,+ # with_tables=...,+ # pdf_with_text_layer=...,+ # pages=...,+ # ...+ )++ Load:+ .. code-block:: python++ docs = loader.load()+ print(docs[0].page_content[:100])+ print(docs[0].metadata)++ .. code-block:: python++ Some text+ {+ 'file_name': 'example.pdf',+ 'file_type': 'application/pdf',+ # ...+ }++ Lazy load:+ .. code-block:: python++ docs = []+ docs_lazy = loader.lazy_load()++ for doc in docs_lazy:+ docs.append(doc)+ print(docs[0].page_content[:100])+ print(docs[0].metadata)++ .. code-block:: python++ Some text+ {+ 'file_name': 'example.pdf',+ 'file_type': 'application/pdf',+ # ...+ }+ """++ def _make_config(self) -> dict:+ from dedoc.utils.langchain import make_manager_config++ return make_manager_config(+ file_path=self.file_path,+ parsing_params=self.parsing_parameters,+ split=self.split,+ )+++class DedocAPIFileLoader(DedocBaseLoader):+ """+ Load files using `dedoc` API.+ The file loader automatically detects the file type (even with the wrong extension).+ By default, the loader makes a call to the locally hosted `dedoc` API.+ More information about `dedoc` API can be found in `dedoc` documentation:+ https://dedoc.readthedocs.io/en/latest/dedoc_api_usage/api.html++ Please see the documentation of DedocBaseLoader to get more details.++ Setup:+ You don't need to install `dedoc` library for using this loader.+ Instead, the `dedoc` API needs to be run.+ You may use Docker container for this purpose.+ Please see `dedoc` documentation for more details:+ https://dedoc.readthedocs.io/en/latest/getting_started/installation.html#install-and-run-dedoc-using-docker++ .. code-block:: bash++ docker pull dedocproject/dedoc+ docker run -p 1231:1231++ Instantiate:+ .. code-block:: python++ from langchain_community.document_loaders import DedocAPIFileLoader++ loader = DedocAPIFileLoader(+ file_path="example.pdf",+ # url=...,+ # split=...,+ # with_tables=...,+ # pdf_with_text_layer=...,+ # pages=...,+ # ...+ )++ Load:+ .. code-block:: python++ docs = loader.load()+ print(docs[0].page_content[:100])+ print(docs[0].metadata)++ .. code-block:: python++ Some text+ {+ 'file_name': 'example.pdf',+ 'file_type': 'application/pdf',+ # ...+ }++ Lazy load:+ .. code-block:: python++ docs = []+ docs_lazy = loader.lazy_load()++ for doc in docs_lazy:+ docs.append(doc)+ print(docs[0].page_content[:100])+ print(docs[0].metadata)++ .. code-block:: python++ Some text+ {+ 'file_name': 'example.pdf',+ 'file_type': 'application/pdf',+ # ...+ }+ """++ def __init__(+ self,+ file_path: str,+ *,+ url: str = "http://0.0.0.0:1231",+ split: str = "document",+ with_tables: bool = True,+ with_attachments: Union[str, bool] = False,+ recursion_deep_attachments: int = 10,+ pdf_with_text_layer: str = "auto_tabby",+ language: str = "rus+eng",+ pages: str = ":",+ is_one_column_document: str = "auto",+ document_orientation: str = "auto",+ need_header_footer_analysis: Union[str, bool] = False,+ need_binarization: Union[str, bool] = False,+ need_pdf_table_analysis: Union[str, bool] = True,+ delimiter: Optional[str] = None,+ encoding: Optional[str] = None,+ ) -> None:+ """Initialize with file path, API url and parsing parameters.++ Args:+ file_path: path to the file for processing+ url: URL to call `dedoc` API+ split: type of document splitting into parts (each part is returned+ separately), default value "document"+ "document": document is returned as a single langchain Document object+ (don't split)+ "page": split document into pages (works for PDF, DJVU, PPTX, PPT, ODP)+ "node": split document into tree nodes (title nodes, list item nodes,+ raw text nodes)+ "line": split document into lines+ with_tables: add tables to the result - each table is returned as a single+ langchain Document object++ Parameters used for document parsing via `dedoc`+ (https://dedoc.readthedocs.io/en/latest/parameters/parameters.html):++ with_attachments: enable attached files extraction+ recursion_deep_attachments: recursion level for attached files+ extraction, works only when with_attachments==True+ pdf_with_text_layer: type of handler for parsing PDF documents,+ available options+ ["true", "false", "tabby", "auto", "auto_tabby" (default)]+ language: language of the document for PDF without a textual layer and+ images, available options ["eng", "rus", "rus+eng" (default)],+ the list of languages can be extended, please see+ https://dedoc.readthedocs.io/en/latest/tutorials/add_new_language.html+ pages: page slice to define the reading range for parsing PDF documents+ is_one_column_document: detect number of columns for PDF without+ a textual layer and images, available options+ ["true", "false", "auto" (default)]+ document_orientation: fix document orientation (90, 180, 270 degrees)+ for PDF without a textual layer and images, available options+ ["auto" (default), "no_change"]+ need_header_footer_analysis: remove headers and footers from the output+ result for parsing PDF and images+ need_binarization: clean pages background (binarize) for PDF without a+ textual layer and images+ need_pdf_table_analysis: parse tables for PDF without a textual layer+ and images+ delimiter: column separator for CSV, TSV files+ encoding: encoding of TXT, CSV, TSV+ """+ super().__init__(+ file_path=file_path,+ split=split,+ with_tables=with_tables,+ with_attachments=with_attachments,+ recursion_deep_attachments=recursion_deep_attachments,+ pdf_with_text_layer=pdf_with_text_layer,+ language=language,+ pages=pages,+ is_one_column_document=is_one_column_document,+ document_orientation=document_orientation,+ need_header_footer_analysis=need_header_footer_analysis,+ need_binarization=need_binarization,+ need_pdf_table_analysis=need_pdf_table_analysis,+ delimiter=delimiter,+ encoding=encoding,+ )+ self.url = url+ self.parsing_parameters["return_format"] = "json"++ def lazy_load(self) -> Iterator[Document]:+ """Lazily load documents."""+ doc_tree = self._send_file(+ url=self.url, file_path=self.file_path, parameters=self.parsing_parameters+ )+ yield from self._split_document(document_tree=doc_tree, split=self.split)++ def _make_config(self) -> dict:+ return {}++ def _send_file(+ self, url: str, file_path: str, parameters: dict+ ) -> Dict[str, Union[list, dict, str]]:+ """Send POST-request to `dedoc` API and return the results"""+ import requests++ file_name = os.path.basename(file_path)+ with open(file_path, "rb") as file:+ files = {"file": (file_name, file)}+ r = requests.post(f"{url}/upload", files=files, data=parameters)++ if r.status_code != 200:+ raise ValueError(f"Error during file handling: {r.content.decode()}")++ result = json.loads(r.content.decode())+ return resultdiff --git a/libs/community/langchain_community/document_loaders/pdf.py b/libs/community/langchain_community/document_loaders/pdf.pyindex 02c416a135..ea4485e589 100644--- a/libs/community/langchain_community/document_loaders/pdf.py+++ b/libs/community/langchain_community/document_loaders/pdf.py@@ -26,6 +26,7 @@ from langchain_core.utils import get_from_dict_or_envfrom langchain_community.document_loaders.base import BaseLoaderfrom langchain_community.document_loaders.blob_loaders import Blob+from langchain_community.document_loaders.dedoc import DedocBaseLoaderfrom langchain_community.document_loaders.parsers.pdf import (AmazonTextractPDFParser,DocumentIntelligenceParser,@@ -738,6 +739,104 @@ class AmazonTextractPDFLoader(BasePDFLoader):raise ValueError(f"unsupported mime type: {blob.mimetype}") # type: ignore[attr-defined]+class DedocPDFLoader(DedocBaseLoader):+ """+ DedocPDFLoader document loader integration to load PDF files using `dedoc`.+ The file loader can automatically detect the correctness of a textual layer in the+ PDF document.+ Note that `__init__` method supports parameters that differ from ones of+ DedocBaseLoader.++ Setup:+ Install ``dedoc`` package.++ .. code-block:: bash++ pip install -U dedoc++ Instantiate:+ .. code-block:: python++ from langchain_community.document_loaders import DedocPDFLoader++ loader = DedocPDFLoader(+ file_path="example.pdf",+ # split=...,+ # with_tables=...,+ # pdf_with_text_layer=...,+ # pages=...,+ # ...+ )++ Load:+ .. code-block:: python++ docs = loader.load()+ print(docs[0].page_content[:100])+ print(docs[0].metadata)++ .. code-block:: python++ Some text+ {+ 'file_name': 'example.pdf',+ 'file_type': 'application/pdf',+ # ...+ }++ Lazy load:+ .. code-block:: python++ docs = []+ docs_lazy = loader.lazy_load()++ for doc in docs_lazy:+ docs.append(doc)+ print(docs[0].page_content[:100])+ print(docs[0].metadata)++ .. code-block:: python++ Some text+ {+ 'file_name': 'example.pdf',+ 'file_type': 'application/pdf',+ # ...+ }++ Parameters used for document parsing via `dedoc`+ (https://dedoc.readthedocs.io/en/latest/parameters/pdf_handling.html):++ with_attachments: enable attached files extraction+ recursion_deep_attachments: recursion level for attached files extraction,+ works only when with_attachments==True+ pdf_with_text_layer: type of handler for parsing, available options+ ["true", "false", "tabby", "auto", "auto_tabby" (default)]+ language: language of the document for PDF without a textual layer,+ available options ["eng", "rus", "rus+eng" (default)], the list of+ languages can be extended, please see+ https://dedoc.readthedocs.io/en/latest/tutorials/add_new_language.html+ pages: page slice to define the reading range for parsing+ is_one_column_document: detect number of columns for PDF without a textual+ layer, available options ["true", "false", "auto" (default)]+ document_orientation: fix document orientation (90, 180, 270 degrees) for PDF+ without a textual layer, available options ["auto" (default), "no_change"]+ need_header_footer_analysis: remove headers and footers from the output result+ need_binarization: clean pages background (binarize) for PDF without a textual+ layer+ need_pdf_table_analysis: parse tables for PDF without a textual layer+ """++ def _make_config(self) -> dict:+ from dedoc.utils.langchain import make_manager_pdf_config++ return make_manager_pdf_config(+ file_path=self.file_path,+ parsing_params=self.parsing_parameters,+ split=self.split,+ )++class DocumentIntelligenceLoader(BasePDFLoader):"""Load a PDF with Azure Document Intelligence"""diff --git a/libs/community/tests/integration_tests/document_loaders/test_dedoc.py b/libs/community/tests/integration_tests/document_loaders/test_dedoc.pynew file mode 100644index 0000000000..3499167a9f--- /dev/null+++ b/libs/community/tests/integration_tests/document_loaders/test_dedoc.py@@ -0,0 +1,146 @@+import os+from pathlib import Path++from langchain_community.document_loaders import (+ DedocAPIFileLoader,+ DedocFileLoader,+ DedocPDFLoader,+)++EXAMPLE_DOCS_DIRECTORY = str(Path(__file__).parent.parent / "examples/")++FILE_NAMES = [+ "example.html",+ "example.json",+ "fake-email-attachment.eml",+ "layout-parser-paper.pdf",+ "slack_export.zip",+ "stanley-cups.csv",+ "stanley-cups.xlsx",+ "whatsapp_chat.txt",+]+++def test_dedoc_file_loader() -> None:+ for file_name in FILE_NAMES:+ file_path = os.path.join(EXAMPLE_DOCS_DIRECTORY, file_name)+ loader = DedocFileLoader(+ file_path,+ split="document",+ with_tables=False,+ pdf_with_text_layer="tabby",+ pages=":1",+ )+ docs = loader.load()++ assert len(docs) == 1+++def test_dedoc_pdf_loader() -> None:+ file_name = "layout-parser-paper.pdf"+ for mode in ("true", "tabby"):+ file_path = os.path.join(EXAMPLE_DOCS_DIRECTORY, file_name)+ loader = DedocPDFLoader(+ file_path,+ split="document",+ with_tables=False,+ pdf_with_text_layer=mode,+ pages=":1",+ )+ docs = loader.load()++ assert len(docs) == 1+++def test_dedoc_content_html() -> None:+ file_name = "example.html"+ file_path = os.path.join(EXAMPLE_DOCS_DIRECTORY, file_name)+ loader = DedocFileLoader(+ file_path,+ split="line",+ with_tables=False,+ )+ docs = loader.load()++ assert docs[0].metadata["file_name"] == "example.html"+ assert docs[0].metadata["file_type"] == "text/html"+ assert "Instead of drinking water from the cat bowl" in docs[0].page_content+ assert "Chase the red dot" not in docs[0].page_content+++def test_dedoc_content_pdf() -> None:+ file_name = "layout-parser-paper.pdf"+ file_path = os.path.join(EXAMPLE_DOCS_DIRECTORY, file_name)+ loader = DedocFileLoader(+ file_path, split="page", pdf_with_text_layer="tabby", pages=":5"+ )+ docs = loader.load()+ table_list = [item for item in docs if item.metadata.get("type", "") == "table"]++ assert len(docs) == 6+ assert docs[0].metadata["file_name"] == "layout-parser-paper.pdf"+ assert docs[0].metadata["file_type"] == "application/pdf"+ assert "This paper introduces LayoutParser, an open-source" in docs[0].page_content+ assert "layout detection [38, 22], table detection [26]" in docs[1].page_content+ assert "LayoutParser: A Unified Toolkit for DL-Based DIA" in docs[2].page_content+ assert len(table_list) > 0+ assert (+ '\n<tbody>\n<tr>\n<td colspan="1" rowspan="1">'+ in table_list[0].metadata["text_as_html"]+ )+++def test_dedoc_content_json() -> None:+ file_name = "example.json"+ file_path = os.path.join(EXAMPLE_DOCS_DIRECTORY, file_name)+ loader = DedocFileLoader(file_path, split="node")+ docs = loader.load()++ assert len(docs) == 11+ assert docs[0].metadata["file_name"] == "example.json"+ assert docs[0].metadata["file_type"] == "application/json"+ assert "Bye!" in docs[0].page_content+++def test_dedoc_content_txt() -> None:+ file_name = "whatsapp_chat.txt"+ file_path = os.path.join(EXAMPLE_DOCS_DIRECTORY, file_name)+ loader = DedocFileLoader(file_path, split="line")+ docs = loader.load()++ assert len(docs) == 10+ assert docs[0].metadata["file_name"] == "whatsapp_chat.txt"+ assert docs[0].metadata["file_type"] == "text/plain"+ assert "[05.05.23, 15:48:11] James: Hi here" in docs[0].page_content+ assert "[11/8/21, 9:41:32 AM] User name: Message 123" in docs[1].page_content+ assert "1/23/23, 3:19 AM - User 2: Bye!" in docs[2].page_content+++def test_dedoc_table_handling() -> None:+ file_name = "stanley-cups.csv"+ file_path = os.path.join(EXAMPLE_DOCS_DIRECTORY, file_name)+ loader = DedocFileLoader(file_path, split="document")+ docs = loader.load()++ assert len(docs) == 2+ assert docs[0].metadata["file_name"] == "stanley-cups.csv"+ assert docs[0].metadata["file_type"] == "text/csv"+ assert docs[1].metadata["type"] == "table"+ assert '<td colspan="1" rowspan="1">1</td>' in docs[1].metadata["text_as_html"]+ assert "Maple Leafs\tTOR\t13" in docs[1].page_content+++def test_dedoc_api_file_loader() -> None:+ file_name = "whatsapp_chat.txt"+ file_path = os.path.join(EXAMPLE_DOCS_DIRECTORY, file_name)+ loader = DedocAPIFileLoader(+ file_path, split="line", url="https://dedoc-readme.hf.space"+ )+ docs = loader.load()++ assert len(docs) == 10+ assert docs[0].metadata["file_name"] == "whatsapp_chat.txt"+ assert docs[0].metadata["file_type"] == "text/plain"+ assert "[05.05.23, 15:48:11] James: Hi here" in docs[0].page_content+ assert "[11/8/21, 9:41:32 AM] User name: Message 123" in docs[1].page_content+ assert "1/23/23, 3:19 AM - User 2: Bye!" in docs[2].page_contentdiff --git a/libs/community/tests/unit_tests/document_loaders/test_imports.py b/libs/community/tests/unit_tests/document_loaders/test_imports.pyindex 0f8628d20e..5cd9ce3d40 100644--- a/libs/community/tests/unit_tests/document_loaders/test_imports.py+++ b/libs/community/tests/unit_tests/document_loaders/test_imports.py@@ -51,6 +51,9 @@ EXPECTED_ALL = ["CubeSemanticLoader","DataFrameLoader","DatadogLogsLoader",+ "DedocAPIFileLoader",+ "DedocFileLoader",+ "DedocPDFLoader","PebbloSafeLoader","DiffbotLoader","DirectoryLoader",