Skip to content

The agent engineering platform.

community[minor]: added new document loaders based on dedoc library (#24303)

Alexander Golodkov2a70a07aad61
diff --git a/docs/docs/integrations/document_loaders/dedoc.ipynb b/docs/docs/integrations/document_loaders/dedoc.ipynb
new file mode 100644
index 0000000000..084537141f
--- /dev/null
+++ b/docs/docs/integrations/document_loaders/dedoc.ipynb
@@ -0,0 +1,484 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "6b74f73d-1763-42d0-9c24-8f65f445bb72",
+ "metadata": {},
+ "source": [
+ "# Dedoc\n",
+ "\n",
+ "This sample demonstrates the use of `Dedoc` in combination with `LangChain` as a `DocumentLoader`.\n",
+ "\n",
+ "## Overview\n",
+ "\n",
+ "[Dedoc](https://dedoc.readthedocs.io) is an [open-source](https://github.com/ispras/dedoc)\n",
+ "library/service that extracts texts, tables, attached files and document structure\n",
+ "(e.g., titles, list items, etc.) from files of various formats.\n",
+ "\n",
+ "`Dedoc` supports `DOCX`, `XLSX`, `PPTX`, `EML`, `HTML`, `PDF`, images and more.\n",
+ "Full list of supported formats can be found [here](https://dedoc.readthedocs.io/en/latest/#id1).\n",
+ "\n",
+ "\n",
+ "### Integration details\n",
+ "\n",
+ "| Class | Package | Local | Serializable | JS support |\n",
+ "|:-----------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------|:-----:|:------------:|:----------:|\n",
+ "| [DedocFileLoader](https://api.python.langchain.com/en/latest/document_loaders/langchain_community.document_loaders.dedoc.DedocFileLoader.html) | [langchain_community](https://api.python.langchain.com/en/latest/community_api_reference.html) | ❌ | beta | ❌ |\n",
+ "| [DedocPDFLoader](https://api.python.langchain.com/en/latest/document_loaders/langchain_community.document_loaders.pdf.DedocPDFLoader.html) | [langchain_community](https://api.python.langchain.com/en/latest/community_api_reference.html) | ❌ | beta | ❌ | \n",
+ "| [DedocAPIFileLoader](https://api.python.langchain.com/en/latest/document_loaders/langchain_community.document_loaders.dedoc.DedocAPIFileLoader.html) | [langchain_community](https://api.python.langchain.com/en/latest/community_api_reference.html) | ❌ | beta | ❌ | \n",
+ "\n",
+ "\n",
+ "### Loader features\n",
+ "\n",
+ "Methods for lazy loading and async loading are available, but in fact, document loading is executed synchronously.\n",
+ "\n",
+ "| Source | Document Lazy Loading | Async Support |\n",
+ "|:------------------:|:---------------------:|:-------------:| \n",
+ "| DedocFileLoader | ❌ | ❌ |\n",
+ "| DedocPDFLoader | ❌ | ❌ | \n",
+ "| DedocAPIFileLoader | ❌ | ❌ | \n",
+ "\n",
+ "## Setup\n",
+ "\n",
+ "* To access `DedocFileLoader` and `DedocPDFLoader` document loaders, you'll need to install the `dedoc` integration package.\n",
+ "* To access `DedocAPIFileLoader`, you'll need to run the `Dedoc` service, e.g. `Docker` container (please see [the documentation](https://dedoc.readthedocs.io/en/latest/getting_started/installation.html#install-and-run-dedoc-using-docker) \n",
+ "for more details):\n",
+ "\n",
+ "```bash\n",
+ "docker pull dedocproject/dedoc\n",
+ "docker run -p 1231:1231\n",
+ "```\n",
+ "\n",
+ "`Dedoc` installation instruction is given [here](https://dedoc.readthedocs.io/en/latest/getting_started/installation.html)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "id": "511c109d-a5c3-42ba-914e-5d1b385bc40f",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Note: you may need to restart the kernel to use updated packages.\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Install package\n",
+ "%pip install --quiet \"dedoc[torch]\""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "6820c0e9-d56d-4899-b8c8-374760360e2b",
+ "metadata": {},
+ "source": [
+ "## Instantiation"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "id": "c1f98cae-71ec-4d60-87fb-96c1a76851d8",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from langchain_community.document_loaders import DedocFileLoader\n",
+ "\n",
+ "loader = DedocFileLoader(\"./example_data/state_of_the_union.txt\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "5d7bc2b3-73a0-4cd6-8014-cc7184aa9d4a",
+ "metadata": {},
+ "source": [
+ "## Load"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "id": "b9097c14-6168-4726-819e-24abb9a63b13",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "'\\nMadam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and t'"
+ ]
+ },
+ "execution_count": 3,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "docs = loader.load()\n",
+ "docs[0].page_content[:100]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "9ed8bd46-0047-4ccc-b2d6-beb7761f7312",
+ "metadata": {},
+ "source": [
+ "## Lazy Load"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "id": "6ae12d7e-8105-4bbe-9031-0e968475f6bf",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and t\n"
+ ]
+ }
+ ],
+ "source": [
+ "docs = loader.lazy_load()\n",
+ "\n",
+ "for doc in docs:\n",
+ " print(doc.page_content[:100])\n",
+ " break"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "8772ae40-6239-4751-bb2d-b4a9415c1ad1",
+ "metadata": {},
+ "source": [
+ "## API reference\n",
+ "\n",
+ "For detailed information on configuring and calling `Dedoc` loaders, please see the API references: \n",
+ "\n",
+ "* https://api.python.langchain.com/en/latest/document_loaders/langchain_community.document_loaders.dedoc.DedocFileLoader.html\n",
+ "* https://api.python.langchain.com/en/latest/document_loaders/langchain_community.document_loaders.pdf.DedocPDFLoader.html\n",
+ "* https://api.python.langchain.com/en/latest/document_loaders/langchain_community.document_loaders.dedoc.DedocAPIFileLoader.html"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "c4d5e702-0e21-4cad-a4c3-b9b3bff77203",
+ "metadata": {},
+ "source": [
+ "## Loading any file\n",
+ "\n",
+ "For automatic handling of any file in a [supported format](https://dedoc.readthedocs.io/en/latest/#id1),\n",
+ "`DedocFileLoader` can be useful.\n",
+ "The file loader automatically detects the file type with a correct extension.\n",
+ "\n",
+ "File parsing process can be configured through `dedoc_kwargs` during the `DedocFileLoader` class initialization.\n",
+ "Here the basic examples of some options usage are given, \n",
+ "please see the documentation of `DedocFileLoader` and \n",
+ "[dedoc documentation](https://dedoc.readthedocs.io/en/latest/parameters/parameters.html) \n",
+ "to get more details about configuration parameters."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "de97d0ed-d6b1-44e0-b392-1f3d89c762f9",
+ "metadata": {},
+ "source": [
+ "### Basic example"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "id": "50ffeeee-db12-4801-b208-7e32ea3d72ad",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "'\\nMadam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans. \\n\\n\\n\\nLast year COVID-19 kept us apart. This year we are finally together again. \\n\\n\\n\\nTonight, we meet as Democrats Republicans and Independents. But most importantly as Americans. \\n\\n\\n\\nWith a duty to one another to the American people to '"
+ ]
+ },
+ "execution_count": 5,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "from langchain_community.document_loaders import DedocFileLoader\n",
+ "\n",
+ "loader = DedocFileLoader(\"./example_data/state_of_the_union.txt\")\n",
+ "\n",
+ "docs = loader.load()\n",
+ "\n",
+ "docs[0].page_content[:400]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "457e5d4c-a4ee-4f31-ae74-3f75a1bbd0af",
+ "metadata": {},
+ "source": [
+ "### Modes of split\n",
+ "\n",
+ "`DedocFileLoader` supports different types of document splitting into parts (each part is returned separately).\n",
+ "For this purpose, `split` parameter is used with the following options:\n",
+ "* `document` (default value): document text is returned as a single langchain `Document` object (don't split);\n",
+ "* `page`: split document text into pages (works for `PDF`, `DJVU`, `PPTX`, `PPT`, `ODP`);\n",
+ "* `node`: split document text into `Dedoc` tree nodes (title nodes, list item nodes, raw text nodes);\n",
+ "* `line`: split document text into textual lines."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "id": "eec54d31-ae7a-4a3c-aa10-4ae276b1e4c4",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "2"
+ ]
+ },
+ "execution_count": 6,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "loader = DedocFileLoader(\n",
+ " \"./example_data/layout-parser-paper.pdf\",\n",
+ " split=\"page\",\n",
+ " pages=\":2\",\n",
+ ")\n",
+ "\n",
+ "docs = loader.load()\n",
+ "\n",
+ "len(docs)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "61e11769-4780-4f77-b10e-27db6936f226",
+ "metadata": {},
+ "source": [
+ "### Handling tables\n",
+ "\n",
+ "`DedocFileLoader` supports tables handling when `with_tables` parameter is \n",
+ "set to `True` during loader initialization (`with_tables=True` by default). \n",
+ "\n",
+ "Tables are not split - each table corresponds to one langchain `Document` object.\n",
+ "For tables, `Document` object has additional `metadata` fields `type=\"table\"` \n",
+ "and `text_as_html` with table `HTML` representation."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "id": "bbeb2f8a-ac5e-4b59-8026-7ea3fc14c928",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "('table',\n",
+ " '<table border=\"1\" style=\"border-collapse: collapse; width: 100%;\">\\n<tbody>\\n<tr>\\n<td colspan=\"1\" rowspan=\"1\">Team</td>\\n<td colspan=\"1\" rowspan=\"1\"> &quot;Payroll (millions)&quot;</td>\\n<td colspan=\"1\" r')"
+ ]
+ },
+ "execution_count": 7,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "loader = DedocFileLoader(\"./example_data/mlb_teams_2012.csv\")\n",
+ "\n",
+ "docs = loader.load()\n",
+ "\n",
+ "docs[1].metadata[\"type\"], docs[1].metadata[\"text_as_html\"][:200]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b4a2b872-2aba-4e4c-8b2f-83a5a81ee1da",
+ "metadata": {},
+ "source": [
+ "### Handling attached files\n",
+ "\n",
+ "`DedocFileLoader` supports attached files handling when `with_attachments` is set \n",
+ "to `True` during loader initialization (`with_attachments=False` by default). \n",
+ "\n",
+ "Attachments are split according to the `split` parameter.\n",
+ "For attachments, langchain `Document` object has an additional metadata \n",
+ "field `type=\"attachment\"`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "id": "bb9d6c1c-e24c-4979-88a0-38d54abd6332",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "('attachment',\n",
+ " '\\nContent-Type\\nmultipart/mixed; boundary=\"0000000000005d654405f082adb7\"\\nDate\\nFri, 23 Dec 2022 12:08:48 -0600\\nFrom\\nMallori Harrell <[email protected]>\\nMIME-Version\\n1.0\\nMessage-ID\\n<CAPgNNXSzLVJ-d1OCX_TjFgJU7ugtQrjFybPtAMmmYZzphxNFYg@mail.gmail.com>\\nSubject\\nFake email with attachment\\nTo\\nMallori Harrell <[email protected]>')"
+ ]
+ },
+ "execution_count": 8,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "loader = DedocFileLoader(\n",
+ " \"./example_data/fake-email-attachment.eml\",\n",
+ " with_attachments=True,\n",
+ ")\n",
+ "\n",
+ "docs = loader.load()\n",
+ "\n",
+ "docs[1].metadata[\"type\"], docs[1].page_content"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d435c3f6-703a-4064-8307-ace140de967a",
+ "metadata": {},
+ "source": [
+ "## Loading PDF file\n",
+ "\n",
+ "If you want to handle only `PDF` documents, you can use `DedocPDFLoader` with only `PDF` support.\n",
+ "The loader supports the same parameters for document split, tables and attachments extraction.\n",
+ "\n",
+ "`Dedoc` can extract `PDF` with or without a textual layer, \n",
+ "as well as automatically detect its presence and correctness.\n",
+ "Several `PDF` handlers are available, you can use `pdf_with_text_layer` \n",
+ "parameter to choose one of them.\n",
+ "Please see [parameters description](https://dedoc.readthedocs.io/en/latest/parameters/pdf_handling.html) \n",
+ "to get more details.\n",
+ "\n",
+ "For `PDF` without a textual layer, `Tesseract OCR` and its language packages should be installed.\n",
+ "In this case, [the instruction](https://dedoc.readthedocs.io/en/latest/tutorials/add_new_language.html) can be useful."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "id": "0103a7f3-6b5e-4444-8f4d-83dd3724a9af",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "'\\n2\\n\\nZ. Shen et al.\\n\\n37], layout detection [38, 22], table detection [26], and scene text detection [4].\\n\\nA generalized learning-based framework dramatically reduces the need for the\\n\\nmanual specification of complicated rules, which is the status quo with traditional\\n\\nmethods. DL has the potential to transform DIA pipelines and benefit a broad\\n\\nspectrum of large-scale document digitization projects.\\n'"
+ ]
+ },
+ "execution_count": 9,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "from langchain_community.document_loaders import DedocPDFLoader\n",
+ "\n",
+ "loader = DedocPDFLoader(\n",
+ " \"./example_data/layout-parser-paper.pdf\", pdf_with_text_layer=\"true\", pages=\"2:2\"\n",
+ ")\n",
+ "\n",
+ "docs = loader.load()\n",
+ "\n",
+ "docs[0].page_content[:400]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "13061995-1805-40c2-a77a-a6cd80999e20",
+ "metadata": {},
+ "source": [
+ "## Dedoc API\n",
+ "\n",
+ "If you want to get up and running with less set up, you can use `Dedoc` as a service.\n",
+ "**`DedocAPIFileLoader` can be used without installation of `dedoc` library.**\n",
+ "The loader supports the same parameters as `DedocFileLoader` and\n",
+ "also automatically detects input file types.\n",
+ "\n",
+ "To use `DedocAPIFileLoader`, you should run the `Dedoc` service, e.g. `Docker` container (please see [the documentation](https://dedoc.readthedocs.io/en/latest/getting_started/installation.html#install-and-run-dedoc-using-docker) \n",
+ "for more details):\n",
+ "\n",
+ "```bash\n",
+ "docker pull dedocproject/dedoc\n",
+ "docker run -p 1231:1231\n",
+ "```\n",
+ "\n",
+ "Please do not use our demo URL `https://dedoc-readme.hf.space` in your code."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "id": "211fc0b5-6080-4974-a6c1-f982bafd87d6",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "'\\nMadam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans. \\n\\n\\n\\nLast year COVID-19 kept us apart. This year we are finally together again. \\n\\n\\n\\nTonight, we meet as Democrats Republicans and Independents. But most importantly as Americans. \\n\\n\\n\\nWith a duty to one another to the American people to '"
+ ]
+ },
+ "execution_count": 10,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "from langchain_community.document_loaders import DedocAPIFileLoader\n",
+ "\n",
+ "loader = DedocAPIFileLoader(\n",
+ " \"./example_data/state_of_the_union.txt\",\n",
+ " url=\"https://dedoc-readme.hf.space\",\n",
+ ")\n",
+ "\n",
+ "docs = loader.load()\n",
+ "\n",
+ "docs[0].page_content[:400]"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "faaff475-5209-436f-bcde-97d58daed05c",
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.9.19"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/docs/docs/integrations/providers/dedoc.mdx b/docs/docs/integrations/providers/dedoc.mdx
new file mode 100644
index 0000000000..3f2aaa206e
--- /dev/null
+++ b/docs/docs/integrations/providers/dedoc.mdx
@@ -0,0 +1,56 @@
+# Dedoc
+
+>[Dedoc](https://dedoc.readthedocs.io) is an [open-source](https://github.com/ispras/dedoc)
+library/service that extracts texts, tables, attached files and document structure
+(e.g., titles, list items, etc.) from files of various formats.
+
+`Dedoc` supports `DOCX`, `XLSX`, `PPTX`, `EML`, `HTML`, `PDF`, images and more.
+Full list of supported formats can be found [here](https://dedoc.readthedocs.io/en/latest/#id1).
+
+## Installation and Setup
+
+### Dedoc library
+
+You can install `Dedoc` using `pip`.
+In this case, you will need to install dependencies,
+please go [here](https://dedoc.readthedocs.io/en/latest/getting_started/installation.html)
+to get more information.
+
+```bash
+pip install dedoc
+```
+
+### Dedoc API
+
+If you are going to use `Dedoc` API, you don't need to install `dedoc` library.
+In this case, you should run the `Dedoc` service, e.g. `Docker` container (please see
+[the documentation](https://dedoc.readthedocs.io/en/latest/getting_started/installation.html#install-and-run-dedoc-using-docker)
+for more details):
+
+```bash
+docker pull dedocproject/dedoc
+docker run -p 1231:1231
+```
+
+## Document Loader
+
+* For handling files of any formats (supported by `Dedoc`), you can use `DedocFileLoader`:
+
+ ```python
+ from langchain_community.document_loaders import DedocFileLoader
+ ```
+
+* For handling PDF files (with or without a textual layer), you can use `DedocPDFLoader`:
+
+ ```python
+ from langchain_community.document_loaders import DedocPDFLoader
+ ```
+
+* For handling files of any formats without library installation,
+you can use `Dedoc API` with `DedocAPIFileLoader`:
+
+ ```python
+ from langchain_community.document_loaders import DedocAPIFileLoader
+ ```
+
+Please see a [usage example](/docs/integrations/document_loaders/dedoc) for more details.
diff --git a/libs/community/extended_testing_deps.txt b/libs/community/extended_testing_deps.txt
index 198e8954f8..55335cad97 100644
--- a/libs/community/extended_testing_deps.txt
+++ b/libs/community/extended_testing_deps.txt
@@ -16,6 +16,7 @@ cloudpickle>=2.0.0
cohere>=4,<6
databricks-vectorsearch>=0.21,<0.22
datasets>=2.15.0,<3
+dedoc>=2.2.6,<3
dgml-utils>=0.3.0,<0.4
elasticsearch>=8.12.0,<9
esprima>=4.0.1,<5
diff --git a/libs/community/langchain_community/document_loaders/__init__.py b/libs/community/langchain_community/document_loaders/__init__.py
index 7f111b1aae..e03f769312 100644
--- a/libs/community/langchain_community/document_loaders/__init__.py
+++ b/libs/community/langchain_community/document_loaders/__init__.py
@@ -142,6 +142,10 @@ if TYPE_CHECKING:
from langchain_community.document_loaders.dataframe import (
DataFrameLoader,
)
+ from langchain_community.document_loaders.dedoc import (
+ DedocAPIFileLoader,
+ DedocFileLoader,
+ )
from langchain_community.document_loaders.diffbot import (
DiffbotLoader,
)
@@ -340,6 +344,7 @@ if TYPE_CHECKING:
)
from langchain_community.document_loaders.pdf import (
AmazonTextractPDFLoader,
+ DedocPDFLoader,
MathpixPDFLoader,
OnlinePDFLoader,
PagedPDFSplitter,
@@ -570,6 +575,9 @@ _module_lookup = {
"CubeSemanticLoader": "langchain_community.document_loaders.cube_semantic",
"DataFrameLoader": "langchain_community.document_loaders.dataframe",
"DatadogLogsLoader": "langchain_community.document_loaders.datadog_logs",
+ "DedocAPIFileLoader": "langchain_community.document_loaders.dedoc",
+ "DedocFileLoader": "langchain_community.document_loaders.dedoc",
+ "DedocPDFLoader": "langchain_community.document_loaders.pdf",
"DiffbotLoader": "langchain_community.document_loaders.diffbot",
"DirectoryLoader": "langchain_community.document_loaders.directory",
"DiscordChatLoader": "langchain_community.document_loaders.discord",
@@ -771,6 +779,9 @@ __all__ = [
"CubeSemanticLoader",
"DataFrameLoader",
"DatadogLogsLoader",
+ "DedocAPIFileLoader",
+ "DedocFileLoader",
+ "DedocPDFLoader",
"DiffbotLoader",
"DirectoryLoader",
"DiscordChatLoader",
diff --git a/libs/community/langchain_community/document_loaders/dedoc.py b/libs/community/langchain_community/document_loaders/dedoc.py
new file mode 100644
index 0000000000..ed8ebc11d7
--- /dev/null
+++ b/libs/community/langchain_community/document_loaders/dedoc.py
@@ -0,0 +1,546 @@
+import html
+import json
+import os
+from abc import ABC, abstractmethod
+from typing import (
+ Dict,
+ Iterator,
+ Optional,
+ Tuple,
+ Union,
+)
+
+from langchain_core.documents import Document
+
+from langchain_community.document_loaders.base import BaseLoader
+
+
+class DedocBaseLoader(BaseLoader, ABC):
+ """
+ Base Loader that uses `dedoc` (https://dedoc.readthedocs.io).
+
+ Loader enables extracting text, tables and attached files from the given file:
+ * `Text` can be split by pages, `dedoc` tree nodes, textual lines
+ (according to the `split` parameter).
+ * `Attached files` (when with_attachments=True)
+ are split according to the `split` parameter.
+ For attachments, langchain Document object has an additional metadata field
+ `type`="attachment".
+ * `Tables` (when with_tables=True) are not split - each table corresponds to one
+ langchain Document object.
+ For tables, Document object has additional metadata fields `type`="table"
+ and `text_as_html` with table HTML representation.
+ """
+
+ def __init__(
+ self,
+ file_path: str,
+ *,
+ split: str = "document",
+ with_tables: bool = True,
+ with_attachments: Union[str, bool] = False,
+ recursion_deep_attachments: int = 10,
+ pdf_with_text_layer: str = "auto_tabby",
+ language: str = "rus+eng",
+ pages: str = ":",
+ is_one_column_document: str = "auto",
+ document_orientation: str = "auto",
+ need_header_footer_analysis: Union[str, bool] = False,
+ need_binarization: Union[str, bool] = False,
+ need_pdf_table_analysis: Union[str, bool] = True,
+ delimiter: Optional[str] = None,
+ encoding: Optional[str] = None,
+ ) -> None:
+ """
+ Initialize with file path and parsing parameters.
+
+ Args:
+ file_path: path to the file for processing
+ split: type of document splitting into parts (each part is returned
+ separately), default value "document"
+ "document": document text is returned as a single langchain Document
+ object (don't split)
+ "page": split document text into pages (works for PDF, DJVU, PPTX, PPT,
+ ODP)
+ "node": split document text into tree nodes (title nodes, list item
+ nodes, raw text nodes)
+ "line": split document text into lines
+ with_tables: add tables to the result - each table is returned as a single
+ langchain Document object
+
+ Parameters used for document parsing via `dedoc`
+ (https://dedoc.readthedocs.io/en/latest/parameters/parameters.html):
+
+ with_attachments: enable attached files extraction
+ recursion_deep_attachments: recursion level for attached files
+ extraction, works only when with_attachments==True
+ pdf_with_text_layer: type of handler for parsing PDF documents,
+ available options
+ ["true", "false", "tabby", "auto", "auto_tabby" (default)]
+ language: language of the document for PDF without a textual layer and
+ images, available options ["eng", "rus", "rus+eng" (default)],
+ the list of languages can be extended, please see
+ https://dedoc.readthedocs.io/en/latest/tutorials/add_new_language.html
+ pages: page slice to define the reading range for parsing PDF documents
+ is_one_column_document: detect number of columns for PDF without
+ a textual layer and images, available options
+ ["true", "false", "auto" (default)]
+ document_orientation: fix document orientation (90, 180, 270 degrees)
+ for PDF without a textual layer and images, available options
+ ["auto" (default), "no_change"]
+ need_header_footer_analysis: remove headers and footers from the output
+ result for parsing PDF and images
+ need_binarization: clean pages background (binarize) for PDF without a
+ textual layer and images
+ need_pdf_table_analysis: parse tables for PDF without a textual layer
+ and images
+ delimiter: column separator for CSV, TSV files
+ encoding: encoding of TXT, CSV, TSV
+ """
+ self.parsing_parameters = {
+ key: value
+ for key, value in locals().items()
+ if key not in {"self", "file_path", "split", "with_tables"}
+ }
+ self.valid_split_values = {"document", "page", "node", "line"}
+ if split not in self.valid_split_values:
+ raise ValueError(
+ f"Got {split} for `split`, but should be one of "
+ f"`{self.valid_split_values}`"
+ )
+ self.split = split
+ self.with_tables = with_tables
+ self.file_path = file_path
+
+ structure_type = "tree" if self.split == "node" else "linear"
+ self.parsing_parameters["structure_type"] = structure_type
+ self.parsing_parameters["need_content_analysis"] = with_attachments
+
+ def lazy_load(self) -> Iterator[Document]:
+ """Lazily load documents."""
+ import tempfile
+
+ try:
+ from dedoc import DedocManager
+ except ImportError:
+ raise ImportError(
+ "`dedoc` package not found, please install it with `pip install dedoc`"
+ )
+ dedoc_manager = DedocManager(manager_config=self._make_config())
+ dedoc_manager.config["logger"].disabled = True
+
+ with tempfile.TemporaryDirectory() as tmpdir:
+ document_tree = dedoc_manager.parse(
+ file_path=self.file_path,
+ parameters={**self.parsing_parameters, "attachments_dir": tmpdir},
+ )
+ yield from self._split_document(
+ document_tree=document_tree.to_api_schema().dict(), split=self.split
+ )
+
+ @abstractmethod
+ def _make_config(self) -> dict:
+ """
+ Make configuration for DedocManager according to the file extension and
+ parsing parameters.
+ """
+ pass
+
+ def _json2txt(self, paragraph: dict) -> str:
+ """Get text (recursively) of the document tree node."""
+ subparagraphs_text = "\n".join(
+ [
+ self._json2txt(subparagraph)
+ for subparagraph in paragraph["subparagraphs"]
+ ]
+ )
+ text = (
+ f"{paragraph['text']}\n{subparagraphs_text}"
+ if subparagraphs_text
+ else paragraph["text"]
+ )
+ return text
+
+ def _parse_subparagraphs(
+ self, document_tree: dict, document_metadata: dict
+ ) -> Iterator[Document]:
+ """Parse recursively document tree obtained by `dedoc`."""
+ if len(document_tree["subparagraphs"]) > 0:
+ for subparagraph in document_tree["subparagraphs"]:
+ yield from self._parse_subparagraphs(
+ document_tree=subparagraph, document_metadata=document_metadata
+ )
+ else:
+ yield Document(
+ page_content=document_tree["text"],
+ metadata={**document_metadata, **document_tree["metadata"]},
+ )
+
+ def _split_document(
+ self,
+ document_tree: dict,
+ split: str,
+ additional_metadata: Optional[dict] = None,
+ ) -> Iterator[Document]:
+ """Split document into parts according to the `split` parameter."""
+ document_metadata = document_tree["metadata"]
+ if additional_metadata:
+ document_metadata = {**document_metadata, **additional_metadata}
+
+ if split == "document":
+ text = self._json2txt(paragraph=document_tree["content"]["structure"])
+ yield Document(page_content=text, metadata=document_metadata)
+
+ elif split == "page":
+ nodes = document_tree["content"]["structure"]["subparagraphs"]
+ page_id = nodes[0]["metadata"]["page_id"]
+ page_text = ""
+
+ for node in nodes:
+ if node["metadata"]["page_id"] == page_id:
+ page_text += self._json2txt(node)
+ else:
+ yield Document(
+ page_content=page_text,
+ metadata={**document_metadata, "page_id": page_id},
+ )
+ page_id = node["metadata"]["page_id"]
+ page_text = self._json2txt(node)
+
+ yield Document(
+ page_content=page_text,
+ metadata={**document_metadata, "page_id": page_id},
+ )
+
+ elif split == "line":
+ for node in document_tree["content"]["structure"]["subparagraphs"]:
+ line_metadata = node["metadata"]
+ yield Document(
+ page_content=self._json2txt(node),
+ metadata={**document_metadata, **line_metadata},
+ )
+
+ elif split == "node":
+ yield from self._parse_subparagraphs(
+ document_tree=document_tree["content"]["structure"],
+ document_metadata=document_metadata,
+ )
+
+ else:
+ raise ValueError(
+ f"Got {split} for `split`, but should be one of "
+ f"`{self.valid_split_values}`"
+ )
+
+ if self.with_tables:
+ for table in document_tree["content"]["tables"]:
+ table_text, table_html = self._get_table(table)
+ yield Document(
+ page_content=table_text,
+ metadata={
+ **table["metadata"],
+ "type": "table",
+ "text_as_html": table_html,
+ },
+ )
+
+ for attachment in document_tree["attachments"]:
+ yield from self._split_document(
+ document_tree=attachment,
+ split=self.split,
+ additional_metadata={"type": "attachment"},
+ )
+
+ def _get_table(self, table: dict) -> Tuple[str, str]:
+ """Get text and HTML representation of the table."""
+ table_text = ""
+ for row in table["cells"]:
+ for cell in row:
+ table_text += " ".join(line["text"] for line in cell["lines"])
+ table_text += "\t"
+ table_text += "\n"
+
+ table_html = (
+ '<table border="1" style="border-collapse: collapse; width: 100%;'
+ '">\n<tbody>\n'
+ )
+ for row in table["cells"]:
+ table_html += "<tr>\n"
+ for cell in row:
+ cell_text = "\n".join(line["text"] for line in cell["lines"])
+ cell_text = html.escape(cell_text)
+ table_html += "<td"
+ if cell["invisible"]:
+ table_html += ' style="display: none" '
+ table_html += (
+ f' colspan="{cell["colspan"]}" rowspan='
+ f'"{cell["rowspan"]}">{cell_text}</td>\n'
+ )
+ table_html += "</tr>\n"
+ table_html += "</tbody>\n</table>"
+
+ return table_text, table_html
+
+
+class DedocFileLoader(DedocBaseLoader):
+ """
+ DedocFileLoader document loader integration to load files using `dedoc`.
+
+ The file loader automatically detects the file type (with the correct extension).
+ The list of supported file types is gives at
+ https://dedoc.readthedocs.io/en/latest/index.html#id1.
+ Please see the documentation of DedocBaseLoader to get more details.
+
+ Setup:
+ Install ``dedoc`` package.
+
+ .. code-block:: bash
+
+ pip install -U dedoc
+
+ Instantiate:
+ .. code-block:: python
+
+ from langchain_community.document_loaders import DedocFileLoader
+
+ loader = DedocFileLoader(
+ file_path="example.pdf",
+ # split=...,
+ # with_tables=...,
+ # pdf_with_text_layer=...,
+ # pages=...,
+ # ...
+ )
+
+ Load:
+ .. code-block:: python
+
+ docs = loader.load()
+ print(docs[0].page_content[:100])
+ print(docs[0].metadata)
+
+ .. code-block:: python
+
+ Some text
+ {
+ 'file_name': 'example.pdf',
+ 'file_type': 'application/pdf',
+ # ...
+ }
+
+ Lazy load:
+ .. code-block:: python
+
+ docs = []
+ docs_lazy = loader.lazy_load()
+
+ for doc in docs_lazy:
+ docs.append(doc)
+ print(docs[0].page_content[:100])
+ print(docs[0].metadata)
+
+ .. code-block:: python
+
+ Some text
+ {
+ 'file_name': 'example.pdf',
+ 'file_type': 'application/pdf',
+ # ...
+ }
+ """
+
+ def _make_config(self) -> dict:
+ from dedoc.utils.langchain import make_manager_config
+
+ return make_manager_config(
+ file_path=self.file_path,
+ parsing_params=self.parsing_parameters,
+ split=self.split,
+ )
+
+
+class DedocAPIFileLoader(DedocBaseLoader):
+ """
+ Load files using `dedoc` API.
+ The file loader automatically detects the file type (even with the wrong extension).
+ By default, the loader makes a call to the locally hosted `dedoc` API.
+ More information about `dedoc` API can be found in `dedoc` documentation:
+ https://dedoc.readthedocs.io/en/latest/dedoc_api_usage/api.html
+
+ Please see the documentation of DedocBaseLoader to get more details.
+
+ Setup:
+ You don't need to install `dedoc` library for using this loader.
+ Instead, the `dedoc` API needs to be run.
+ You may use Docker container for this purpose.
+ Please see `dedoc` documentation for more details:
+ https://dedoc.readthedocs.io/en/latest/getting_started/installation.html#install-and-run-dedoc-using-docker
+
+ .. code-block:: bash
+
+ docker pull dedocproject/dedoc
+ docker run -p 1231:1231
+
+ Instantiate:
+ .. code-block:: python
+
+ from langchain_community.document_loaders import DedocAPIFileLoader
+
+ loader = DedocAPIFileLoader(
+ file_path="example.pdf",
+ # url=...,
+ # split=...,
+ # with_tables=...,
+ # pdf_with_text_layer=...,
+ # pages=...,
+ # ...
+ )
+
+ Load:
+ .. code-block:: python
+
+ docs = loader.load()
+ print(docs[0].page_content[:100])
+ print(docs[0].metadata)
+
+ .. code-block:: python
+
+ Some text
+ {
+ 'file_name': 'example.pdf',
+ 'file_type': 'application/pdf',
+ # ...
+ }
+
+ Lazy load:
+ .. code-block:: python
+
+ docs = []
+ docs_lazy = loader.lazy_load()
+
+ for doc in docs_lazy:
+ docs.append(doc)
+ print(docs[0].page_content[:100])
+ print(docs[0].metadata)
+
+ .. code-block:: python
+
+ Some text
+ {
+ 'file_name': 'example.pdf',
+ 'file_type': 'application/pdf',
+ # ...
+ }
+ """
+
+ def __init__(
+ self,
+ file_path: str,
+ *,
+ url: str = "http://0.0.0.0:1231",
+ split: str = "document",
+ with_tables: bool = True,
+ with_attachments: Union[str, bool] = False,
+ recursion_deep_attachments: int = 10,
+ pdf_with_text_layer: str = "auto_tabby",
+ language: str = "rus+eng",
+ pages: str = ":",
+ is_one_column_document: str = "auto",
+ document_orientation: str = "auto",
+ need_header_footer_analysis: Union[str, bool] = False,
+ need_binarization: Union[str, bool] = False,
+ need_pdf_table_analysis: Union[str, bool] = True,
+ delimiter: Optional[str] = None,
+ encoding: Optional[str] = None,
+ ) -> None:
+ """Initialize with file path, API url and parsing parameters.
+
+ Args:
+ file_path: path to the file for processing
+ url: URL to call `dedoc` API
+ split: type of document splitting into parts (each part is returned
+ separately), default value "document"
+ "document": document is returned as a single langchain Document object
+ (don't split)
+ "page": split document into pages (works for PDF, DJVU, PPTX, PPT, ODP)
+ "node": split document into tree nodes (title nodes, list item nodes,
+ raw text nodes)
+ "line": split document into lines
+ with_tables: add tables to the result - each table is returned as a single
+ langchain Document object
+
+ Parameters used for document parsing via `dedoc`
+ (https://dedoc.readthedocs.io/en/latest/parameters/parameters.html):
+
+ with_attachments: enable attached files extraction
+ recursion_deep_attachments: recursion level for attached files
+ extraction, works only when with_attachments==True
+ pdf_with_text_layer: type of handler for parsing PDF documents,
+ available options
+ ["true", "false", "tabby", "auto", "auto_tabby" (default)]
+ language: language of the document for PDF without a textual layer and
+ images, available options ["eng", "rus", "rus+eng" (default)],
+ the list of languages can be extended, please see
+ https://dedoc.readthedocs.io/en/latest/tutorials/add_new_language.html
+ pages: page slice to define the reading range for parsing PDF documents
+ is_one_column_document: detect number of columns for PDF without
+ a textual layer and images, available options
+ ["true", "false", "auto" (default)]
+ document_orientation: fix document orientation (90, 180, 270 degrees)
+ for PDF without a textual layer and images, available options
+ ["auto" (default), "no_change"]
+ need_header_footer_analysis: remove headers and footers from the output
+ result for parsing PDF and images
+ need_binarization: clean pages background (binarize) for PDF without a
+ textual layer and images
+ need_pdf_table_analysis: parse tables for PDF without a textual layer
+ and images
+ delimiter: column separator for CSV, TSV files
+ encoding: encoding of TXT, CSV, TSV
+ """
+ super().__init__(
+ file_path=file_path,
+ split=split,
+ with_tables=with_tables,
+ with_attachments=with_attachments,
+ recursion_deep_attachments=recursion_deep_attachments,
+ pdf_with_text_layer=pdf_with_text_layer,
+ language=language,
+ pages=pages,
+ is_one_column_document=is_one_column_document,
+ document_orientation=document_orientation,
+ need_header_footer_analysis=need_header_footer_analysis,
+ need_binarization=need_binarization,
+ need_pdf_table_analysis=need_pdf_table_analysis,
+ delimiter=delimiter,
+ encoding=encoding,
+ )
+ self.url = url
+ self.parsing_parameters["return_format"] = "json"
+
+ def lazy_load(self) -> Iterator[Document]:
+ """Lazily load documents."""
+ doc_tree = self._send_file(
+ url=self.url, file_path=self.file_path, parameters=self.parsing_parameters
+ )
+ yield from self._split_document(document_tree=doc_tree, split=self.split)
+
+ def _make_config(self) -> dict:
+ return {}
+
+ def _send_file(
+ self, url: str, file_path: str, parameters: dict
+ ) -> Dict[str, Union[list, dict, str]]:
+ """Send POST-request to `dedoc` API and return the results"""
+ import requests
+
+ file_name = os.path.basename(file_path)
+ with open(file_path, "rb") as file:
+ files = {"file": (file_name, file)}
+ r = requests.post(f"{url}/upload", files=files, data=parameters)
+
+ if r.status_code != 200:
+ raise ValueError(f"Error during file handling: {r.content.decode()}")
+
+ result = json.loads(r.content.decode())
+ return result
diff --git a/libs/community/langchain_community/document_loaders/pdf.py b/libs/community/langchain_community/document_loaders/pdf.py
index 02c416a135..ea4485e589 100644
--- a/libs/community/langchain_community/document_loaders/pdf.py
+++ b/libs/community/langchain_community/document_loaders/pdf.py
@@ -26,6 +26,7 @@ from langchain_core.utils import get_from_dict_or_env
from langchain_community.document_loaders.base import BaseLoader
from langchain_community.document_loaders.blob_loaders import Blob
+from langchain_community.document_loaders.dedoc import DedocBaseLoader
from langchain_community.document_loaders.parsers.pdf import (
AmazonTextractPDFParser,
DocumentIntelligenceParser,
@@ -738,6 +739,104 @@ class AmazonTextractPDFLoader(BasePDFLoader):
raise ValueError(f"unsupported mime type: {blob.mimetype}") # type: ignore[attr-defined]
+class DedocPDFLoader(DedocBaseLoader):
+ """
+ DedocPDFLoader document loader integration to load PDF files using `dedoc`.
+ The file loader can automatically detect the correctness of a textual layer in the
+ PDF document.
+ Note that `__init__` method supports parameters that differ from ones of
+ DedocBaseLoader.
+
+ Setup:
+ Install ``dedoc`` package.
+
+ .. code-block:: bash
+
+ pip install -U dedoc
+
+ Instantiate:
+ .. code-block:: python
+
+ from langchain_community.document_loaders import DedocPDFLoader
+
+ loader = DedocPDFLoader(
+ file_path="example.pdf",
+ # split=...,
+ # with_tables=...,
+ # pdf_with_text_layer=...,
+ # pages=...,
+ # ...
+ )
+
+ Load:
+ .. code-block:: python
+
+ docs = loader.load()
+ print(docs[0].page_content[:100])
+ print(docs[0].metadata)
+
+ .. code-block:: python
+
+ Some text
+ {
+ 'file_name': 'example.pdf',
+ 'file_type': 'application/pdf',
+ # ...
+ }
+
+ Lazy load:
+ .. code-block:: python
+
+ docs = []
+ docs_lazy = loader.lazy_load()
+
+ for doc in docs_lazy:
+ docs.append(doc)
+ print(docs[0].page_content[:100])
+ print(docs[0].metadata)
+
+ .. code-block:: python
+
+ Some text
+ {
+ 'file_name': 'example.pdf',
+ 'file_type': 'application/pdf',
+ # ...
+ }
+
+ Parameters used for document parsing via `dedoc`
+ (https://dedoc.readthedocs.io/en/latest/parameters/pdf_handling.html):
+
+ with_attachments: enable attached files extraction
+ recursion_deep_attachments: recursion level for attached files extraction,
+ works only when with_attachments==True
+ pdf_with_text_layer: type of handler for parsing, available options
+ ["true", "false", "tabby", "auto", "auto_tabby" (default)]
+ language: language of the document for PDF without a textual layer,
+ available options ["eng", "rus", "rus+eng" (default)], the list of
+ languages can be extended, please see
+ https://dedoc.readthedocs.io/en/latest/tutorials/add_new_language.html
+ pages: page slice to define the reading range for parsing
+ is_one_column_document: detect number of columns for PDF without a textual
+ layer, available options ["true", "false", "auto" (default)]
+ document_orientation: fix document orientation (90, 180, 270 degrees) for PDF
+ without a textual layer, available options ["auto" (default), "no_change"]
+ need_header_footer_analysis: remove headers and footers from the output result
+ need_binarization: clean pages background (binarize) for PDF without a textual
+ layer
+ need_pdf_table_analysis: parse tables for PDF without a textual layer
+ """
+
+ def _make_config(self) -> dict:
+ from dedoc.utils.langchain import make_manager_pdf_config
+
+ return make_manager_pdf_config(
+ file_path=self.file_path,
+ parsing_params=self.parsing_parameters,
+ split=self.split,
+ )
+
+
class DocumentIntelligenceLoader(BasePDFLoader):
"""Load a PDF with Azure Document Intelligence"""
diff --git a/libs/community/tests/integration_tests/document_loaders/test_dedoc.py b/libs/community/tests/integration_tests/document_loaders/test_dedoc.py
new file mode 100644
index 0000000000..3499167a9f
--- /dev/null
+++ b/libs/community/tests/integration_tests/document_loaders/test_dedoc.py
@@ -0,0 +1,146 @@
+import os
+from pathlib import Path
+
+from langchain_community.document_loaders import (
+ DedocAPIFileLoader,
+ DedocFileLoader,
+ DedocPDFLoader,
+)
+
+EXAMPLE_DOCS_DIRECTORY = str(Path(__file__).parent.parent / "examples/")
+
+FILE_NAMES = [
+ "example.html",
+ "example.json",
+ "fake-email-attachment.eml",
+ "layout-parser-paper.pdf",
+ "slack_export.zip",
+ "stanley-cups.csv",
+ "stanley-cups.xlsx",
+ "whatsapp_chat.txt",
+]
+
+
+def test_dedoc_file_loader() -> None:
+ for file_name in FILE_NAMES:
+ file_path = os.path.join(EXAMPLE_DOCS_DIRECTORY, file_name)
+ loader = DedocFileLoader(
+ file_path,
+ split="document",
+ with_tables=False,
+ pdf_with_text_layer="tabby",
+ pages=":1",
+ )
+ docs = loader.load()
+
+ assert len(docs) == 1
+
+
+def test_dedoc_pdf_loader() -> None:
+ file_name = "layout-parser-paper.pdf"
+ for mode in ("true", "tabby"):
+ file_path = os.path.join(EXAMPLE_DOCS_DIRECTORY, file_name)
+ loader = DedocPDFLoader(
+ file_path,
+ split="document",
+ with_tables=False,
+ pdf_with_text_layer=mode,
+ pages=":1",
+ )
+ docs = loader.load()
+
+ assert len(docs) == 1
+
+
+def test_dedoc_content_html() -> None:
+ file_name = "example.html"
+ file_path = os.path.join(EXAMPLE_DOCS_DIRECTORY, file_name)
+ loader = DedocFileLoader(
+ file_path,
+ split="line",
+ with_tables=False,
+ )
+ docs = loader.load()
+
+ assert docs[0].metadata["file_name"] == "example.html"
+ assert docs[0].metadata["file_type"] == "text/html"
+ assert "Instead of drinking water from the cat bowl" in docs[0].page_content
+ assert "Chase the red dot" not in docs[0].page_content
+
+
+def test_dedoc_content_pdf() -> None:
+ file_name = "layout-parser-paper.pdf"
+ file_path = os.path.join(EXAMPLE_DOCS_DIRECTORY, file_name)
+ loader = DedocFileLoader(
+ file_path, split="page", pdf_with_text_layer="tabby", pages=":5"
+ )
+ docs = loader.load()
+ table_list = [item for item in docs if item.metadata.get("type", "") == "table"]
+
+ assert len(docs) == 6
+ assert docs[0].metadata["file_name"] == "layout-parser-paper.pdf"
+ assert docs[0].metadata["file_type"] == "application/pdf"
+ assert "This paper introduces LayoutParser, an open-source" in docs[0].page_content
+ assert "layout detection [38, 22], table detection [26]" in docs[1].page_content
+ assert "LayoutParser: A Unified Toolkit for DL-Based DIA" in docs[2].page_content
+ assert len(table_list) > 0
+ assert (
+ '\n<tbody>\n<tr>\n<td colspan="1" rowspan="1">'
+ in table_list[0].metadata["text_as_html"]
+ )
+
+
+def test_dedoc_content_json() -> None:
+ file_name = "example.json"
+ file_path = os.path.join(EXAMPLE_DOCS_DIRECTORY, file_name)
+ loader = DedocFileLoader(file_path, split="node")
+ docs = loader.load()
+
+ assert len(docs) == 11
+ assert docs[0].metadata["file_name"] == "example.json"
+ assert docs[0].metadata["file_type"] == "application/json"
+ assert "Bye!" in docs[0].page_content
+
+
+def test_dedoc_content_txt() -> None:
+ file_name = "whatsapp_chat.txt"
+ file_path = os.path.join(EXAMPLE_DOCS_DIRECTORY, file_name)
+ loader = DedocFileLoader(file_path, split="line")
+ docs = loader.load()
+
+ assert len(docs) == 10
+ assert docs[0].metadata["file_name"] == "whatsapp_chat.txt"
+ assert docs[0].metadata["file_type"] == "text/plain"
+ assert "[05.05.23, 15:48:11] James: Hi here" in docs[0].page_content
+ assert "[11/8/21, 9:41:32 AM] User name: Message 123" in docs[1].page_content
+ assert "1/23/23, 3:19 AM - User 2: Bye!" in docs[2].page_content
+
+
+def test_dedoc_table_handling() -> None:
+ file_name = "stanley-cups.csv"
+ file_path = os.path.join(EXAMPLE_DOCS_DIRECTORY, file_name)
+ loader = DedocFileLoader(file_path, split="document")
+ docs = loader.load()
+
+ assert len(docs) == 2
+ assert docs[0].metadata["file_name"] == "stanley-cups.csv"
+ assert docs[0].metadata["file_type"] == "text/csv"
+ assert docs[1].metadata["type"] == "table"
+ assert '<td colspan="1" rowspan="1">1</td>' in docs[1].metadata["text_as_html"]
+ assert "Maple Leafs\tTOR\t13" in docs[1].page_content
+
+
+def test_dedoc_api_file_loader() -> None:
+ file_name = "whatsapp_chat.txt"
+ file_path = os.path.join(EXAMPLE_DOCS_DIRECTORY, file_name)
+ loader = DedocAPIFileLoader(
+ file_path, split="line", url="https://dedoc-readme.hf.space"
+ )
+ docs = loader.load()
+
+ assert len(docs) == 10
+ assert docs[0].metadata["file_name"] == "whatsapp_chat.txt"
+ assert docs[0].metadata["file_type"] == "text/plain"
+ assert "[05.05.23, 15:48:11] James: Hi here" in docs[0].page_content
+ assert "[11/8/21, 9:41:32 AM] User name: Message 123" in docs[1].page_content
+ assert "1/23/23, 3:19 AM - User 2: Bye!" in docs[2].page_content
diff --git a/libs/community/tests/unit_tests/document_loaders/test_imports.py b/libs/community/tests/unit_tests/document_loaders/test_imports.py
index 0f8628d20e..5cd9ce3d40 100644
--- a/libs/community/tests/unit_tests/document_loaders/test_imports.py
+++ b/libs/community/tests/unit_tests/document_loaders/test_imports.py
@@ -51,6 +51,9 @@ EXPECTED_ALL = [
"CubeSemanticLoader",
"DataFrameLoader",
"DatadogLogsLoader",
+ "DedocAPIFileLoader",
+ "DedocFileLoader",
+ "DedocPDFLoader",
"PebbloSafeLoader",
"DiffbotLoader",
"DirectoryLoader",