Skip to content

The agent engineering platform.

docs(core): add clarity to base token counting methods (#33958)

Mason Daugherty275dcbf69fbf
diff --git a/libs/core/langchain_core/language_models/base.py b/libs/core/langchain_core/language_models/base.py
index 58c6d4abf7..ff1daed4b3 100644
--- a/libs/core/langchain_core/language_models/base.py
+++ b/libs/core/langchain_core/language_models/base.py
@@ -299,6 +299,9 @@ class BaseLanguageModel(
Useful for checking if an input fits in a model's context window.
+ This should be overridden by model-specific implementations to provide accurate
+ token counts via model-specific tokenizers.
+
Args:
text: The string input to tokenize.
@@ -317,9 +320,17 @@ class BaseLanguageModel(
Useful for checking if an input fits in a model's context window.
+ This should be overridden by model-specific implementations to provide accurate
+ token counts via model-specific tokenizers.
+
!!! note
- The base implementation of `get_num_tokens_from_messages` ignores tool
- schemas.
+
+ * The base implementation of `get_num_tokens_from_messages` ignores tool
+ schemas.
+ * The base implementation of `get_num_tokens_from_messages` adds additional
+ prefixes to messages in represent user roles, which will add to the
+ overall token count. Model-specific implementations may choose to
+ handle this differently.
Args:
messages: The message inputs to tokenize.