Skip to content

Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.

fix(semantic): reduce unsupported entity hallucinations in summaries (#3908)

Jiajie - He/him/hiscfd74413cb43
diff --git a/openviking/prompts/templates/semantic/document_summary.yaml b/openviking/prompts/templates/semantic/document_summary.yaml
index 36e61d5a1..10b45dd2f 100644
--- a/openviking/prompts/templates/semantic/document_summary.yaml
+++ b/openviking/prompts/templates/semantic/document_summary.yaml
@@ -39,15 +39,18 @@ template: |
- Highlight key sections, headings, and their relationships
- Mention any important concepts, definitions, or explanations
- Note the document type (tutorial, reference, guide, API docs, etc.)
- - Describe the target audience and prerequisite knowledge if apparent
- - Include relevant keywords for semantic search
+ - Describe the target audience and prerequisite knowledge if apparent from the file content
+ - Include relevant keywords from the file content for semantic search
- Output plain text directly, no markdown format
+ Faithfulness rules:
+ - Describe only what the file content states; do not invent entities, facts, or relationships not present in it.
+ - Do not introduce entity names not present in the file content (such as names of people, organizations, companies, products, projects, brands, or places); omit unstated entity relationships rather than guessing.
Structure your summary with:
1. One sentence describing the document's primary purpose
2. Main sections or topics covered
3. Key information or takeaways
- 4. Intended audience or use case context
+ 4. Intended audience or use case context, if apparent from the file content
Special considerations:
- For Markdown files: pay attention to heading hierarchy
diff --git a/openviking/prompts/templates/semantic/file_summary.yaml b/openviking/prompts/templates/semantic/file_summary.yaml
index c13559db3..0472b2a49 100644
--- a/openviking/prompts/templates/semantic/file_summary.yaml
+++ b/openviking/prompts/templates/semantic/file_summary.yaml
@@ -35,9 +35,12 @@ template: |
Output requirements:
- Length: 50-150 words
- - Explain what this file is, what it covers, and what it's used for
- - Include core keywords for understanding
+ - Explain what this file is, what it covers, and what it's used for if apparent from the file content
+ - Include core keywords from the file content for understanding
- Output plain text directly, no markdown format
+ Faithfulness rules:
+ - Describe only what the file content states; do not invent entities, facts, or relationships not present in it.
+ - Do not introduce entity names not present in the file content (such as names of people, organizations, companies, products, projects, brands, or places); omit unstated entity relationships rather than guessing.
llm_config:
temperature: 0.0
diff --git a/openviking/prompts/templates/semantic/overview_generation.yaml b/openviking/prompts/templates/semantic/overview_generation.yaml
index dbf3a1868..d717c525f 100644
--- a/openviking/prompts/templates/semantic/overview_generation.yaml
+++ b/openviking/prompts/templates/semantic/overview_generation.yaml
@@ -50,7 +50,11 @@ template: |
Relationship rules:
- Treat child directories as parts of the same repository unless the summaries clearly show they are independent projects.
- Do not describe every child directory as an independent project by default.
- - When the summaries suggest a code repository, explain how subdirectories relate to the whole repo, such as services, libraries, apps, modules, or support folders.
+ - When the summaries indicate a code repository, explain how subdirectories relate to the whole repo, such as services, libraries, apps, modules, or support folders.
+
+ Faithfulness rules:
+ - Describe only what the provided summaries state; do not invent or generalize entities, facts, or relationships not present in them.
+ - Before output, remove any named entity absent from the provided summaries—including organization, company, product, and brand names—and any unstated ownership, affiliation, or other relationship; never fill gaps with outside knowledge.
Output in Markdown format, strictly following this structure:
@@ -59,8 +63,8 @@ template: |
2. **{{ "简要描述" if output_language == "zh-CN" else "Brief Description" }}** (plain text paragraph, 50-150 words):
- Immediately following the title, without any H2 heading
- Explain what this is, what it's about, what it covers
- - Include core keywords for easy searching
- - Who it's suitable for
+ - Include core keywords from the provided summaries for easy searching
+ - Who it's suitable for, if stated in the provided summaries
3. **{{ "快速导航" if output_language == "zh-CN" else "Quick Navigation" }}** (H2): Decision Tree style
- Guide with "What do you want to learn?" or "What do you want to do?"
diff --git a/tests/storage/test_semantic_processor_language.py b/tests/storage/test_semantic_processor_language.py
index f87ed5adc..43d1c0f9f 100644
--- a/tests/storage/test_semantic_processor_language.py
+++ b/tests/storage/test_semantic_processor_language.py
@@ -233,9 +233,16 @@ class TestOverviewGenerationFlow:
in prompt
)
assert (
- "- When the summaries suggest a code repository, explain how subdirectories relate to the whole repo, such as services, libraries, apps, modules, or support folders."
+ "- When the summaries indicate a code repository, explain how subdirectories relate to the whole repo, such as services, libraries, apps, modules, or support folders."
in prompt
)
+ assert (
+ "- Describe only what the provided summaries state; do not invent or generalize entities, facts, or relationships not present in them."
+ in prompt
+ )
+ assert "Before output, remove any named entity absent from the provided summaries" in prompt
+ assert "never fill gaps with outside knowledge" in prompt
+ assert "Who it's suitable for, if stated in the provided summaries" in prompt
def test_chinese_overview_uses_localized_headings(self):
prompt = render_prompt(