The Memory Layer for AI Agents - Drop-in memory infrastructure for AI agents and apps. Context that persists. Built for production.
Python agentic-memoryagentic-memory-systemagentsaiai-agentschatgptgenaillmlong-term-memorymemorymemory-managementpythonragstate-management
[Refactor] Change evaluation script path (#1165)
Deshraj Yadavaffe319460e9
diff --git a/docs/api-reference/pipeline/add.mdx b/docs/api-reference/app/add.mdxsimilarity index 100%rename from docs/api-reference/pipeline/add.mdxrename to docs/api-reference/app/add.mdxdiff --git a/docs/api-reference/pipeline/chat.mdx b/docs/api-reference/app/chat.mdxsimilarity index 100%rename from docs/api-reference/pipeline/chat.mdxrename to docs/api-reference/app/chat.mdxdiff --git a/docs/api-reference/pipeline/delete.mdx b/docs/api-reference/app/delete.mdxsimilarity index 100%rename from docs/api-reference/pipeline/delete.mdxrename to docs/api-reference/app/delete.mdxdiff --git a/docs/api-reference/pipeline/deploy.mdx b/docs/api-reference/app/deploy.mdxsimilarity index 100%rename from docs/api-reference/pipeline/deploy.mdxrename to docs/api-reference/app/deploy.mdxdiff --git a/docs/api-reference/pipeline/evaluate.mdx b/docs/api-reference/app/evaluate.mdxsimilarity index 100%rename from docs/api-reference/pipeline/evaluate.mdxrename to docs/api-reference/app/evaluate.mdxdiff --git a/docs/api-reference/pipeline/overview.mdx b/docs/api-reference/app/overview.mdxsimilarity index 76%rename from docs/api-reference/pipeline/overview.mdxrename to docs/api-reference/app/overview.mdxindex 2f6fc9f6c..8c369cbf8 100644--- a/docs/api-reference/pipeline/overview.mdx+++ b/docs/api-reference/app/overview.mdx@@ -1,34 +1,34 @@----title: "Pipeline"+title: "App"----Create a RAG pipeline object on Embedchain. This is the main entrypoint for a developer to interact with Embedchain APIs. A pipeline configures the llm, vector database, embedding model, and retrieval strategy of your choice.+Create a RAG app object on Embedchain. This is the main entrypoint for a developer to interact with Embedchain APIs. An app configures the llm, vector database, embedding model, and retrieval strategy of your choice.### Attributes<ParamField path="local_id" type="str">- Pipeline ID+ App ID</ParamField><ParamField path="name" type="str" optional>- Name of the pipeline+ Name of the app</ParamField><ParamField path="config" type="BaseConfig">- Configuration of the pipeline+ Configuration of the app</ParamField><ParamField path="llm" type="BaseLlm">- Configured LLM for the RAG pipeline+ Configured LLM for the RAG app</ParamField><ParamField path="db" type="BaseVectorDB">- Configured vector database for the RAG pipeline+ Configured vector database for the RAG app</ParamField><ParamField path="embedding_model" type="BaseEmbedder">- Configured embedding model for the RAG pipeline+ Configured embedding model for the RAG app</ParamField><ParamField path="chunker" type="ChunkerConfig">Chunker configuration</ParamField><ParamField path="client" type="Client" optional>- Client object (used to deploy a pipeline to Embedchain platform)+ Client object (used to deploy an app to Embedchain platform)</ParamField><ParamField path="logger" type="logging.Logger">Logger object@@ -36,7 +36,7 @@ Create a RAG pipeline object on Embedchain. This is the main entrypoint for a de## Usage-You can create an embedchain pipeline instance using the following methods:+You can create an app instance using the following methods:### Default setting@@ -127,4 +127,4 @@ app = App.from_config(config_path="config.json")}```-</CodeGroup>\ No newline at end of file+</CodeGroup>diff --git a/docs/api-reference/pipeline/query.mdx b/docs/api-reference/app/query.mdxsimilarity index 100%rename from docs/api-reference/pipeline/query.mdxrename to docs/api-reference/app/query.mdxdiff --git a/docs/api-reference/pipeline/reset.mdx b/docs/api-reference/app/reset.mdxsimilarity index 100%rename from docs/api-reference/pipeline/reset.mdxrename to docs/api-reference/app/reset.mdxdiff --git a/docs/api-reference/pipeline/search.mdx b/docs/api-reference/app/search.mdxsimilarity index 100%rename from docs/api-reference/pipeline/search.mdxrename to docs/api-reference/app/search.mdxdiff --git a/docs/components/evaluation.mdx b/docs/components/evaluation.mdxindex 6eed725e9..c1143d2ec 100644--- a/docs/components/evaluation.mdx+++ b/docs/components/evaluation.mdx@@ -84,7 +84,7 @@ Once you have created your dataset, you can run evaluation on the dataset by picFor example, you can run evaluation on context relevancy metric using the following code:```python-from embedchain.eval.metrics import ContextRelevance+from embedchain.evaluation.metrics import ContextRelevancemetric = ContextRelevance()score = metric.evaluate(dataset)print(score)@@ -112,20 +112,21 @@ context_relevance_score = num_relevant_sentences_in_context / num_of_sentences_iYou can run the context relevancy evaluation with the following simple code:```python-from embedchain.eval.metrics import ContextRelevance+from embedchain.evaluation.metrics import ContextRelevancemetric = ContextRelevance()score = metric.evaluate(dataset) # 'dataset' is definted in the create dataset sectionprint(score)# 0.27975528364849833```+In the above example, we used sensible defaults for the evaluation. However, you can also configure the evaluation metric as per your needs using the `ContextRelevanceConfig` class.Here is a more advanced example of how to pass a custom evaluation config for evaluating on context relevance metric:```python-from embedchain.config.eval.base import ContextRelevanceConfig-from embedchain.eval.metrics import ContextRelevance+from embedchain.config.evaluation.base import ContextRelevanceConfig+from embedchain.evaluation.metrics import ContextRelevanceeval_config = ContextRelevanceConfig(model="gpt-4", api_key="sk-xxx", language="en")metric = ContextRelevance(config=eval_config)@@ -144,7 +145,7 @@ metric.evaluate(dataset)The language of the dataset being evaluated. We need this to determine the understand the context provided in the dataset. Defaults to `en`.</ParamField><ParamField path="prompt" type="str" optional>- The prompt to extract the relevant sentences from the context. Defaults to `CONTEXT_RELEVANCY_PROMPT`, which can be found at `embedchain.config.eval.base` path.+ The prompt to extract the relevant sentences from the context. Defaults to `CONTEXT_RELEVANCY_PROMPT`, which can be found at `embedchain.config.evaluation.base` path.</ParamField>@@ -161,7 +162,7 @@ answer_relevancy_score = mean(cosine_similarity(generated_questions, original_quYou can run the answer relevancy evaluation with the following simple code:```python-from embedchain.eval.metrics import AnswerRelevance+from embedchain.evaluation.metrics import AnswerRelevancemetric = AnswerRelevance()score = metric.evaluate(dataset)@@ -172,8 +173,8 @@ print(score)In the above example, we used sensible defaults for the evaluation. However, you can also configure the evaluation metric as per your needs using the `AnswerRelevanceConfig` class. Here is a more advanced example where you can provide your own evaluation config:```python-from embedchain.config.eval.base import AnswerRelevanceConfig-from embedchain.eval.metrics import AnswerRelevance+from embedchain.config.evaluation.base import AnswerRelevanceConfig+from embedchain.evaluation.metrics import AnswerRelevanceeval_config = AnswerRelevanceConfig(model='gpt-4',@@ -200,7 +201,7 @@ score = metric.evaluate(dataset)The number of questions to generate for each answer. We use the generated questions to compare the similarity with the original question to determine the score. Defaults to `1`.</ParamField><ParamField path="prompt" type="str" optional>- The prompt to extract the `num_gen_questions` number of questions from the provided answer. Defaults to `ANSWER_RELEVANCY_PROMPT`, which can be found at `embedchain.config.eval.base` path.+ The prompt to extract the `num_gen_questions` number of questions from the provided answer. Defaults to `ANSWER_RELEVANCY_PROMPT`, which can be found at `embedchain.config.evaluation.base` path.</ParamField>## Groundedness <a id="groundedness"></a>@@ -214,7 +215,7 @@ groundedness_score = (sum of all verdicts) / (total # of claims)You can run the groundedness evaluation with the following simple code:```python-from embedchain.eval.metrics import Groundedness+from embedchain.evaluation.metrics import Groundednessmetric = Groundedness()score = metric.evaluate(dataset) # dataset from aboveprint(score)@@ -224,8 +225,8 @@ print(score)In the above example, we used sensible defaults for the evaluation. However, you can also configure the evaluation metric as per your needs using the `GroundednessConfig` class. Here is a more advanced example where you can configure the evaluation config:```python-from embedchain.config.eval.base import GroundednessConfig-from embedchain.eval.metrics import Groundedness+from embedchain.config.evaluation.base import GroundednessConfig+from embedchain.evaluation.metrics import Groundednesseval_config = GroundednessConfig(model='gpt-4', api_key="sk-xxx")metric = Groundedness(config=eval_config)@@ -242,15 +243,15 @@ score = metric.evaluate(dataset)The openai api key to use for the evaluation. Defaults to `None`. If not provided, we will use the `OPENAI_API_KEY` environment variable.</ParamField><ParamField path="answer_claims_prompt" type="str" optional>- The prompt to extract the claims from the provided answer. Defaults to `GROUNDEDNESS_ANSWER_CLAIMS_PROMPT`, which can be found at `embedchain.config.eval.base` path.+ The prompt to extract the claims from the provided answer. Defaults to `GROUNDEDNESS_ANSWER_CLAIMS_PROMPT`, which can be found at `embedchain.config.evaluation.base` path.</ParamField><ParamField path="claims_inference_prompt" type="str" optional>- The prompt to get verdicts on the claims from the answer from the given context. Defaults to `GROUNDEDNESS_CLAIMS_INFERENCE_PROMPT`, which can be found at `embedchain.config.eval.base` path.+ The prompt to get verdicts on the claims from the answer from the given context. Defaults to `GROUNDEDNESS_CLAIMS_INFERENCE_PROMPT`, which can be found at `embedchain.config.evaluation.base` path.</ParamField>## Custom <a id="custom_metric"></a>-You can also create your own evaluation metric by extending the `BaseMetric` class. You can find the source code for the existing metrics at `embedchain.eval.metrics` path.+You can also create your own evaluation metric by extending the `BaseMetric` class. You can find the source code for the existing metrics at `embedchain.evaluation.metrics` path.<Note>You must provide the `name` of your custom metric in the `__init__` method of your class. This name will be used to identify your metric in the evaluation report.@@ -260,7 +261,7 @@ You must provide the `name` of your custom metric in the `__init__` method of yofrom typing import Optionalfrom embedchain.config.base_config import BaseConfig-from embedchain.eval.metrics import BaseMetric+from embedchain.evaluation.metrics import BaseMetricfrom embedchain.utils.eval import EvalDataclass MyCustomMetric(BaseMetric):diff --git a/embedchain/app.py b/embedchain/app.pyindex a9e222fec..966f5b6e8 100644--- a/embedchain/app.py+++ b/embedchain/app.py@@ -11,24 +11,28 @@ import requestsimport yamlfrom tqdm import tqdm-from embedchain.cache import (Config, ExactMatchEvaluation,- SearchDistanceEvaluation, cache,- gptcache_data_manager, gptcache_pre_function)+from embedchain.cache import (+ Config,+ ExactMatchEvaluation,+ SearchDistanceEvaluation,+ cache,+ gptcache_data_manager,+ gptcache_pre_function,+)from embedchain.client import Clientfrom embedchain.config import AppConfig, CacheConfig, ChunkerConfigfrom embedchain.constants import SQLITE_PATHfrom embedchain.embedchain import EmbedChainfrom embedchain.embedder.base import BaseEmbedderfrom embedchain.embedder.openai import OpenAIEmbedder-from embedchain.eval.base import BaseMetric-from embedchain.eval.metrics import (AnswerRelevance, ContextRelevance,- Groundedness)+from embedchain.evaluation.base import BaseMetric+from embedchain.evaluation.metrics import AnswerRelevance, ContextRelevance, Groundednessfrom embedchain.factory import EmbedderFactory, LlmFactory, VectorDBFactoryfrom embedchain.helpers.json_serializable import register_deserializablefrom embedchain.llm.base import BaseLlmfrom embedchain.llm.openai import OpenAILlmfrom embedchain.telemetry.posthog import AnonymousTelemetry-from embedchain.utils.eval import EvalData, EvalMetric+from embedchain.utils.evaluation import EvalData, EvalMetricfrom embedchain.utils.misc import validate_configfrom embedchain.vectordb.base import BaseVectorDBfrom embedchain.vectordb.chroma import ChromaDBdiff --git a/embedchain/config/eval/__init__.py b/embedchain/config/evaluation/__init__.pysimilarity index 100%rename from embedchain/config/eval/__init__.pyrename to embedchain/config/evaluation/__init__.pydiff --git a/embedchain/config/eval/base.py b/embedchain/config/evaluation/base.pysimilarity index 100%rename from embedchain/config/eval/base.pyrename to embedchain/config/evaluation/base.pydiff --git a/embedchain/eval/__init__.py b/embedchain/evaluation/__init__.pysimilarity index 100%rename from embedchain/eval/__init__.pyrename to embedchain/evaluation/__init__.pydiff --git a/embedchain/eval/base.py b/embedchain/evaluation/base.pysimilarity index 93%rename from embedchain/eval/base.pyrename to embedchain/evaluation/base.pyindex d86a8be18..4528e7689 100644--- a/embedchain/eval/base.py+++ b/embedchain/evaluation/base.py@@ -1,6 +1,6 @@from abc import ABC, abstractmethod-from embedchain.utils.eval import EvalData+from embedchain.utils.evaluation import EvalDataclass BaseMetric(ABC):diff --git a/embedchain/eval/metrics/__init__.py b/embedchain/evaluation/metrics/__init__.pysimilarity index 100%rename from embedchain/eval/metrics/__init__.pyrename to embedchain/evaluation/metrics/__init__.pydiff --git a/embedchain/eval/metrics/answer_relevancy.py b/embedchain/evaluation/metrics/answer_relevancy.pysimilarity index 94%rename from embedchain/eval/metrics/answer_relevancy.pyrename to embedchain/evaluation/metrics/answer_relevancy.pyindex 5335449c7..588fc0fd1 100644--- a/embedchain/eval/metrics/answer_relevancy.py+++ b/embedchain/evaluation/metrics/answer_relevancy.py@@ -8,9 +8,9 @@ import numpy as npfrom openai import OpenAIfrom tqdm import tqdm-from embedchain.config.eval.base import AnswerRelevanceConfig-from embedchain.eval.base import BaseMetric-from embedchain.utils.eval import EvalData, EvalMetric+from embedchain.config.evaluation.base import AnswerRelevanceConfig+from embedchain.evaluation.base import BaseMetric+from embedchain.utils.evaluation import EvalData, EvalMetricclass AnswerRelevance(BaseMetric):diff --git a/embedchain/eval/metrics/context_relevancy.py b/embedchain/evaluation/metrics/context_relevancy.pysimilarity index 93%rename from embedchain/eval/metrics/context_relevancy.pyrename to embedchain/evaluation/metrics/context_relevancy.pyindex 44bfe75d7..f821713fa 100644--- a/embedchain/eval/metrics/context_relevancy.py+++ b/embedchain/evaluation/metrics/context_relevancy.py@@ -8,9 +8,9 @@ import pysbdfrom openai import OpenAIfrom tqdm import tqdm-from embedchain.config.eval.base import ContextRelevanceConfig-from embedchain.eval.base import BaseMetric-from embedchain.utils.eval import EvalData, EvalMetric+from embedchain.config.evaluation.base import ContextRelevanceConfig+from embedchain.evaluation.base import BaseMetric+from embedchain.utils.evaluation import EvalData, EvalMetricclass ContextRelevance(BaseMetric):diff --git a/embedchain/eval/metrics/groundedness.py b/embedchain/evaluation/metrics/groundedness.pysimilarity index 95%rename from embedchain/eval/metrics/groundedness.pyrename to embedchain/evaluation/metrics/groundedness.pyindex eac76d4ee..1feaab42e 100644--- a/embedchain/eval/metrics/groundedness.py+++ b/embedchain/evaluation/metrics/groundedness.py@@ -8,9 +8,9 @@ import numpy as npfrom openai import OpenAIfrom tqdm import tqdm-from embedchain.config.eval.base import GroundednessConfig-from embedchain.eval.base import BaseMetric-from embedchain.utils.eval import EvalData, EvalMetric+from embedchain.config.evaluation.base import GroundednessConfig+from embedchain.evaluation.base import BaseMetric+from embedchain.utils.evaluation import EvalData, EvalMetricclass Groundedness(BaseMetric):diff --git a/embedchain/utils/eval.py b/embedchain/utils/evaluation.pysimilarity index 100%rename from embedchain/utils/eval.pyrename to embedchain/utils/evaluation.pydiff --git a/pyproject.toml b/pyproject.tomlindex f70003c74..431f83241 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -1,6 +1,6 @@[tool.poetry]name = "embedchain"-version = "0.1.63"+version = "0.1.64"description = "Data platform for LLMs - Load, index, retrieve and sync any unstructured data"authors = ["Taranjeet Singh <[email protected]>",