The agent engineering platform.
Python agentsaiai-agentsanthropicchatgptdeepagentsenterpriseframeworkgeminigenerative-ailangchainlanggraphllmmultiagentopen-sourceopenaipydanticpythonragtypescript
core[patch]: Expand documentation in the indexing namespace (#23134)
Eugene Yurtsev4fe8403bfbb8
diff --git a/libs/core/langchain_core/indexing/api.py b/libs/core/langchain_core/indexing/api.pyindex 01bc9bffb2..86741ae3a2 100644--- a/libs/core/langchain_core/indexing/api.py+++ b/libs/core/langchain_core/indexing/api.py@@ -214,10 +214,18 @@ def index(are not able to specify the uid of the document.IMPORTANT:- if auto_cleanup is set to True, the loader should be returning- the entire dataset, and not just a subset of the dataset.- Otherwise, the auto_cleanup will remove documents that it is not- supposed to.+ * if auto_cleanup is set to True, the loader should be returning+ the entire dataset, and not just a subset of the dataset.+ Otherwise, the auto_cleanup will remove documents that it is not+ supposed to.+ * In incremental mode, if documents associated with a particular+ source id appear across different batches, the indexing API+ will do some redundant work. This will still result in the+ correct end state of the index, but will unfortunately not be+ 100% efficient. For example, if a given document is split into 15+ chunks, and we index them using a batch size of 5, we'll have 3 batches+ all with the same source id. In general, to avoid doing too much+ redundant work select as big a batch size as possible.Args:docs_source: Data loader or iterable of documents to index.diff --git a/libs/core/langchain_core/indexing/base.py b/libs/core/langchain_core/indexing/base.pyindex 776f1f1089..ac73191bbe 100644--- a/libs/core/langchain_core/indexing/base.py+++ b/libs/core/langchain_core/indexing/base.py@@ -5,7 +5,39 @@ from typing import List, Optional, Sequenceclass RecordManager(ABC):- """Abstract base class representing the interface for a record manager."""+ """Abstract base class representing the interface for a record manager.++ The record manager abstraction is used by the langchain indexing API.++ The record manager keeps track of which documents have been+ written into a vectorstore and when they were written.++ The indexing API computes hashes for each document and stores the hash+ together with the write time and the source id in the record manager.++ On subsequent indexing runs, the indexing API can check the record manager+ to determine which documents have already been indexed and which have not.++ This allows the indexing API to avoid re-indexing documents that have+ already been indexed, and to only index new documents.++ The main benefit of this abstraction is that it works across many vectorstores.+ To be supported, a vectorstore needs to only support the ability to add and+ delete documents by ID. Using the record manager, the indexing API will+ be able to delete outdated documents and avoid redundant indexing of documents+ that have already been indexed.++ The main constraints of this abstraction are:++ 1. It relies on the time-stamps to determine which documents have been+ indexed and which have not. This means that the time-stamps must be+ monotonically increasing. The timestamp should be the timestamp+ as measured by the server to minimize issues.+ 2. The record manager is currently implemented separately from the+ vectorstore, which means that the overall system becomes distributed+ and may create issues with consistency. For example, writing to+ record manager succeeds but corresponding writing to vectorstore fails.+ """def __init__(self,