RAGFlow is an agent-native database: a Retrieval-Augmented Generation (RAG) engine that grew into context infrastructure.
Go agent-harnessagent-native-databaseagentic-aiagentic-retrievalagentic-searchaiai-agentscontext-enginecontext-engineeringcontext-managementharness-engineeringknowledge-compilationragretrieval-augmented-generationretrieval-harnesssearch-harness
Fix claim/evidence usage in agentic RAG (#21170)
Yingfeng7eb673bfc3ab
diff --git a/conf/agentic_rag.yaml b/conf/agentic_rag.yamlindex 66db98fd4d..81b1df831e 100644--- a/conf/agentic_rag.yaml+++ b/conf/agentic_rag.yaml@@ -27,8 +27,10 @@ templates:- **grep_chunks:** locate by pattern - ONE POSIX regex over chunk content, case-insensitive (like grep -E -i). Pack the key people, objects and verbs into ONE broad alternation, and do NOT anchor it to a subject/verb chain: the same event often appears with the actor as the grammatical object, and a broad alternation matches regardless of role.- **search_bm25_chunks** (when the deployment supplies it): lexical recall - 1-5 keyword queries ranked by BM25 over tokenized chunk fields.- **search_semantic_chunks** (when supplied): the semantic bridge - 1-5 queries ranked purely by word meaning (pure vector, no keyword leg). Use it when the corpus likely words things differently from the question.- - **search_metadata** (when the deployment supplies it): Document-metadata SELECTOR - given metadata conditions, return the document ids (doc_ids) whose metadata matches; it runs NO content retrieval of its own. Use it to narrow the corpus to the documents that satisfy a metadata predicate (e.g. author = "X", year contains "2024", doc_type in ["report","memo"]) before reading them: pass the returned doc_ids as search_chunks' "doc_scope" to search ONLY inside those documents, or as list_chunks' "doc_id" for a deep read.- - **list_chunks:** the deep reader, and the authoritative source for corpus-backed claims. Pass ONE document's doc_id plus 1-20 chunk_ids from locate output (anchor_chunk_ids); set number_neighbors to N to widen each anchor with the N chunks before AND after it, and cover a long document by chaining anchored calls (each reply's doc_chunks_total tells you what is still uncovered). Corpus search snippets locate; only the deep read proves.+ - **search_metadata** (when the deployment supplies it): Document-metadata SELECTOR - given metadata conditions, return the document ids (doc_ids) whose metadata matches; it runs NO content retrieval of its own. Use it to narrow the corpus to the documents that satisfy a metadata predicate (e.g. author = "X", year contains "2024", doc_type in ["report","memo"]) before reading them: pass the returned doc_ids as the "doc_scope" of any search tool (grep_chunks, search_bm25_chunks, search_semantic_chunks, search_chunks) to search ONLY inside those documents, or as list_chunks' "doc_id" for a deep read.+ - **navigate_tree** (only when the tool list contains it): the compiled-navigation ROUTER. Ranks the bound datasets' compiled navigation trees against the query and returns the doc_ids whose structure matches, with a short outline of each. Call it as soon as the question names a section, entity or topic that keywords would only guess at: one routing call replaces several blind searches, and it is the intended entry point to navigate_structure.+ - **navigate_structure** (only when available): the compiled-structure DRILLDOWN for ONE document. Pass a doc_id returned by navigate_tree plus the query. It returns that document's outline together with its compiled CLAIM lines that match the query - a claim is one atomic statement carrying its verbatim evidence quote, the chunk it came from and the handle to cite it by, e.g. "- [claim] <statement> | Evidence: \"<quote>\" [chunks: c1] ref=3". Claim lines ARE answer material: when a document returns them, cite them by their ref handle and skip the deep read. Fall back to list_chunks only when no claim hits the question or a claim is missing the exact figure.+ - **list_chunks:** the deep reader, and the authoritative source for corpus-backed claims. Pass ONE document's doc_id plus 1-20 chunk_ids from locate output (anchor_chunk_ids); set number_neighbors to N to widen each anchor with the N chunks before AND after it, and cover a long document by chaining anchored calls (each reply's doc_chunks_total tells you what is still uncovered). Corpus search snippets locate; only the deep read proves. It is the fallback path, not the default: a claim line already carries its verbatim evidence.- **todo_write** (optional): track multi-step research (worth it from 3+ retrieval steps).- **think** (optional): plan and reflect on what you have retrieved.- **run_javascript** (optional): a self-contained ECMAScript 5.1 sandbox for arithmetic or JSON post-processing on data you already retrieved. It cannot reach the corpus or the network; results come back via console.log.@@ -36,13 +38,14 @@ templates:### How to work1. Always search configured datasets before answering. Start from the most distinctive anchor the question offers (a proper noun, a date, a title) and locate with grep_chunks or a search tool; widen with the other legs when the first returns nothing. If web_search is available, also run at least one web keyword query before answering, even if corpus retrieval found relevant chunks. Use the evidence from both sources; web results supplement dataset evidence.- 2. **Mandatory deep read for corpus evidence:** once a corpus locate step points at a document, read it with list_chunks before answering. Corpus search snippets locate evidence; only the full text proves it. Web snippets are evaluated directly and do not require list_chunks.- 3. Consume what you search: a query whose hits never reach your reading did not happen. When a query returns nothing, rephrase - shorter, rarer, more literal - instead of repeating it. Retrieve fresh for every new question: the corpus may have changed.- 4. Stop when the required retrieval steps are complete and the evidence answers the question; new reads that stop changing the answer are the signal. For a purely conversational message (a greeting, a thank-you), keep the answer brief; when no retrieval sources are configured, answer directly.+ 2. **Navigate before you scan.** When navigate_tree is offered, route first: navigate_tree with the question's own wording, then navigate_structure with a returned doc_id. Compiled claim lines that come back are finished evidence - each one is a statement plus its verbatim quote - so cite them and move on; do not spend calls re-reading the document to confirm what a claim already states. Reserve list_chunks for documents that return no claim hits, or when a claim is missing the exact figure the answer needs.+ 3. **Mandatory deep read for corpus evidence without claims:** when a locate step points at a document and no claim line answers the question, or the claim that comes closest is missing the exact figure the answer needs, read it with list_chunks before answering. Corpus search snippets locate evidence; only the full text proves it. Web snippets are evaluated directly and do not require list_chunks.+ 4. Consume what you search: a query whose hits never reach your reading did not happen. When a query returns nothing, rephrase - shorter, rarer, more literal - instead of repeating it. Retrieve fresh for every new question: the corpus may have changed.+ 5. Stop when the required retrieval steps are complete and the evidence answers the question; new reads that stop changing the answer are the signal. For a purely conversational message (a greeting, a thank-you), keep the answer brief; when no retrieval sources are configured, answer directly.### The answer- ONE message of plain prose in the user's language: the answer itself, with the smallest structure that helps (short paragraphs; a list only when the question asks for several things). No meta-narration ("Let me check...", "Here is what I found"), no tool calls in the final message, no restating the question.- - **Provenance is mechanical:** end every line that states a corpus- or web-backed fact with the handle the tool printed for that passage - `[ID:n]`, where n is that passage's `ref` attribute in the tool output. The pipeline converts these handles into the numbered citation markers the interface renders; a factual claim without one reads as unsupported.+ - **Provenance is mechanical:** end every line that states a corpus- or web-backed fact with the handle the tool printed for that passage - `[ID:n]`, where n is the `ref` value printed beside it (`ref="n"` on a passage element, `ref=n` at the end of a claim line). The pipeline converts these handles into the numbered citation markers the interface renders; a factual claim without one reads as unsupported.- If the retrieved corpus and web evidence does not contain the answer, say so plainly, name what you searched, and give the closest fact the retrieved evidence does support - never a specific entity, date or number recalled from world knowledge alone. When the exact figure is missing but a close one exists, commit to it and state the assumption explicitly (e.g. "based on the 2017 figure; no 2019 figure is present").- **Exact numbers:** use the corpus's full precision; never round intermediates. Dates and ages: count completed units from the date the question anchors (1787-12-12 -> 1873-07-01 = 85), not birth years alone. When comparing two people's ages, anchor each to the event the question names, then subtract. Convert units exactly (ft<->m, lb<->kg).diff --git a/internal/agent/tool/dataset_navigation_test.go b/internal/agent/tool/dataset_navigation_test.goindex f4c0ec9224..b8ad8c9970 100644--- a/internal/agent/tool/dataset_navigation_test.go+++ b/internal/agent/tool/dataset_navigation_test.go@@ -33,6 +33,9 @@ func (f *navRoutingFake) Search(_ context.Context, _, _ string, query string, _f.mu.Unlock()return f.hits, nil}+func (f *navRoutingFake) EncodeQueryVector(context.Context, string, string) []float64 {+ return nil+}func (f *navRoutingFake) ListClusters(context.Context, string, string, string, int, int) ([]nav.NavNode, int64, error) {return f.clusters, int64(len(f.clusters)), nil}diff --git a/internal/agent/tool/http_helper_test.go b/internal/agent/tool/http_helper_test.goindex 990db3810c..81882916f9 100644--- a/internal/agent/tool/http_helper_test.go+++ b/internal/agent/tool/http_helper_test.go@@ -277,8 +277,11 @@ func TestHTTPHelper_Timeout(t *testing.T) {if !errors.Is(err, context.DeadlineExceeded) {t.Errorf("Do err = %v, want context.DeadlineExceeded (callers branch on errors.Is for this)", err)}- if got := atomic.LoadInt32(&hits); got != 1 {- t.Errorf("server hits = %d, want 1 (context errors are not retryable — a regression to retry on ctx-deadline would show 2+)", got)+ // 0 is allowed: under a loaded machine the 50ms deadline can expire during the+ // dial, before the request reaches the handler. What this test really pins is+ // that a ctx deadline is NOT retried (a regression would show 2+ attempts).+ if got := atomic.LoadInt32(&hits); got > 1 {+ t.Errorf("server hits = %d, want at most 1 (context errors are not retryable — a regression to retry on ctx-deadline would show 2+)", got)}}diff --git a/internal/agentic_rag/agent.go b/internal/agentic_rag/agent.goindex 1ada439985..dbd3c82500 100644--- a/internal/agentic_rag/agent.go+++ b/internal/agentic_rag/agent.go@@ -356,6 +356,7 @@ func Run(ctx context.Context, in Input) (string, error) {// "the tool was never offered" would be indistinguishable from "the tool was// offered and the model ignored it" in the server log - which is exactly the// question a tool-usage review asks.+ compiledInjected := falseif len(in.Tools) != 0 {common.InfoCtx(ctx, "agentic_rag: compiled-knowledge tools skipped",zap.Strings("datasets", in.DatasetIDs),@@ -363,6 +364,7 @@ func Run(ctx context.Context, in Input) (string, error) {} else if datasetsHaveCompiledKnowledge(ctx, in.TenantID, in.DatasetIDs) {compiled := compiledKnowledgeTools(in.TenantID, in.DatasetIDs)tools = append(tools, compiled...)+ compiledInjected = truecommon.InfoCtx(ctx, "agentic_rag: compiled-knowledge tools injected",zap.Strings("datasets", in.DatasetIDs),zap.String("reason", "compile_kwd rows present in the bound datasets"),@@ -382,6 +384,48 @@ func Run(ctx context.Context, in Input) (string, error) {maxIter = defaultMaxIterations}+ // ONE claim recall per turn, BEFORE the loop, consumed as a control signal+ // rather than as prompt text.+ //+ // A recall that covers the question needs no document and no chunk: the+ // claims carry their own verbatim evidence, so the turn terminates on a+ // single model call over them. Descending the structure again, or reading+ // the raw chunks they were compiled from, could only re-derive what the+ // claims already state - and every additional step is another chance to+ // prune away the branch that holds the answer.+ //+ // A recall that does not cover the question is discarded here. The loop then+ // runs exactly as it would have without the recall, and navigate_structure+ // stays available for the model to call on demand.+ if compiledInjected {+ pf := recallCompiledClaims(ctx, in.TenantID, in.DatasetIDs, in.Messages)+ if pf.sufficient {+ answer, aerr := answerFromClaims(ctx, synthModelFor(in), tmpl.Content, in.Messages, pf)+ switch {+ case aerr != nil:+ // Falling through keeps the turn answerable: the claims are+ // dropped from the registry's numbering (nothing was shown) and+ // the loop retrieves what it needs itself.+ common.WarnCtx(ctx, "agentic_rag: claim answer failed; falling back to the loop", zap.Error(aerr))+ case strings.TrimSpace(answer) != "":+ // The loop is skipped entirely, so this line is the only record+ // that the turn never read a chunk: a review of tool usage must+ // be able to tell it from a turn the model gave up on.+ common.InfoCtx(ctx, "agentic_rag: answered from compiled claims",+ zap.Int("claims", len(pf.claims)),+ zap.Int("docs", len(pf.docs)),+ zap.Int("answer_bytes", len(answer)))+ emit(ctx, in.OnDelta, answer, "")+ return answer, nil+ default:+ // The claims were judged (or proved) not to answer the question:+ // the loop retrieves what the claims could not supply.+ common.InfoCtx(ctx, "agentic_rag: falling back to the loop after an unusable claim answer",+ zap.Int("claims", len(pf.claims)))+ }+ }+ }+// Wrap every invokable tool so each call is timed and emitted as a debug// log, accumulated into in.ToolCallDurations. Tools that aren't// InvokableTool (e.g. streamable-only) are passed through unwrapped — all@@ -516,11 +560,7 @@ func Run(ctx context.Context, in Input) (string, error) {synthCtx, synthCancel := context.WithTimeout(context.WithoutCancel(ctx), finalizeTimeout)defer synthCancel()synthCtx = tokenizer.WithUsagePhase(synthCtx, "synth")- synthModel := in.Model- if in.SynthModel != nil {- synthModel = in.SynthModel- }- synth, synthErr := finalizeAnswer(synthCtx, synthModel, in.Messages, final, evidence)+ synth, synthErr := finalizeAnswer(synthCtx, synthModelFor(in), in.Messages, final, evidence)if synthErr != nil {// Without this an empty final is completely silent: the run looks// healthy in every other log line and the user simply gets nothing.@@ -773,6 +813,133 @@ const finalizeTimeout = 90 * time.Second// for minutes and blow the client timeout (observed: exactly 300s on q55).const finalizeEvidenceMaxRunes = 16000+// claimAnswerTimeout bounds the sufficient-claims answer. It is a single+// non-streaming call over evidence that was already retrieved, so it needs a+// fraction of the loop's budget - and a hung provider must not eat the turn.+const claimAnswerTimeout = 90 * time.Second++// synthModelFor picks the model that answers without tools: the caller's+// dedicated synthesis instance when it supplied one, else the agent's own model.+func synthModelFor(in Input) *models.EinoChatModel {+ if in.SynthModel != nil {+ return in.SynthModel+ }+ return in.Model+}++// claimAnswerPrompt renders the user message of the sufficient-claims answer.+//+// It states the rules the loop would otherwise carry in its system prompt, in+// the one place they apply: the answer must be built from these claims, each+// fact line must carry the [ID:n] handle of the passage it came from (the+// pipeline turns those handles into the reference payload), and nothing may be+// extrapolated. It deliberately does NOT advertise tools: this call has none, so+// a model that looks for one would burn its only turn.+//+// The reply is a JSON verdict, not prose, because the caller has to act on it:+// "these claims do not answer the question" is a routing decision (fall back to+// the loop), and reading it reliably out of free text is not possible. A model+// that cannot answer from the claims says so here instead of shipping a+// non-answer, which is exactly what a zero-score benchmark run looked like.+func claimAnswerPrompt(evidence string) string {+ var b strings.Builder+ b.WriteString("### Compiled evidence retrieved for this question\n")+ b.WriteString("Each line below is one compiled claim: a statement, the verbatim source quote it was compiled from, ")+ b.WriteString("the chunk that quote came from, and the passage handle `ref=n` it was shown as.\n\n")+ b.WriteString(evidence)+ b.WriteString("\n\n### Task\n")+ b.WriteString("Decide whether this evidence answers the question above, then answer from it.\n")+ b.WriteString("- If it does: combine claims where the question needs it, and end every line that states a fact with ")+ b.WriteString("the handle of the claim it came from: `[ID:n]`, where n is that claim's `ref=n` value. Never ")+ b.WriteString("introduce an entity, date or number that no claim states. Write the answer in the user's language.\n")+ b.WriteString("- If it does not (wrong topic, or the specific fact the question asks for is absent): do NOT ")+ b.WriteString("guess and do NOT answer from world knowledge. Report it as not answerable instead.\n")+ b.WriteString("- Do not call tools and do not narrate what you are about to do: this call is the whole ")+ b.WriteString("turn, so a plan or a promise to search further is not an answer.\n\n")+ b.WriteString("Reply with ONE JSON object and nothing else:\n")+ b.WriteString(`{"answerable": true, "answer": "<the final answer, with [ID:n] handles>"}`)+ b.WriteString(" or ")+ b.WriteString(`{"answerable": false, "missing": "<the fact that is absent>"}`)+ return b.String()+}++// claimAnswer is the model's verdict on a sufficient-claims recall. answerable+// false is a real outcome, not an error: it means the gate's relevance floor let+// through claims that do not actually answer the question, so the caller must+// fall back to the ordinary loop instead of shipping a non-answer.+type claimAnswer struct {+ answerable bool+ answer string+ missing string+}++// parseClaimAnswer reads the JSON verdict. A reply that is not the requested+// object is reported as not answerable: shipping unparsed prose is how a turn+// with no retrieval ends up scored zero, and one extra loop is far cheaper.+func parseClaimAnswer(reply string) claimAnswer {+ obj, ok := decodeJSONObject(reply)+ if !ok {+ return claimAnswer{missing: "model did not return the requested JSON verdict"}+ }+ out := claimAnswer{missing: stringFromAny(obj["missing"])}+ if v, ok := obj["answerable"].(bool); ok {+ out.answerable = v+ }+ if s, ok := obj["answer"].(string); ok {+ out.answer = strings.TrimSpace(s)+ }+ if out.answerable && out.answer == "" {+ out.answerable = false+ }+ return out+}++// answerFromClaims is the termination of a turn whose pre-loop claim recall+// already covers the question: ONE model call over the claims, with no tools and+// therefore no raw-chunk read. It returns ("", nil) when the model judged the+// claims insufficient, which the caller treats as a reason to run the ordinary+// loop instead.+func answerFromClaims(+ ctx context.Context,+ m *models.EinoChatModel,+ instruction string,+ messages []*schema.Message,+ pf claimPrefetch,+) (string, error) {+ if m == nil || strings.TrimSpace(pf.evidence) == "" {+ return "", nil+ }+ // Its own usage phase: the share of turns that never entered the loop is+ // exactly what a cost comparison against the loop path needs.+ ctx, cancel := context.WithTimeout(tokenizer.WithUsagePhase(ctx, "claims"), claimAnswerTimeout)+ defer cancel()++ // The template's instruction goes in as the system message because this path+ // bypasses the agent that would otherwise carry it: the answer must keep the+ // same shape and language rules as every other answer this template ships.+ req := make([]*schema.Message, 0, len(messages)+2)+ req = append(req, schema.SystemMessage(instruction))+ req = append(req, messages...)+ req = append(req, schema.UserMessage(claimAnswerPrompt(pf.evidence)))++ resp, err := m.Generate(ctx, req)+ if err != nil {+ return "", err+ }+ if resp == nil {+ return "", nil+ }+ verdict := parseClaimAnswer(resp.Content)+ if !verdict.answerable {+ common.InfoCtx(ctx, "agentic_rag: claims judged insufficient by the model",+ zap.Int("claims", len(pf.claims)),+ zap.Strings("docs", pf.docs),+ zap.String("missing", truncateForLog(verdict.missing, 200)))+ return "", nil+ }+ return verdict.answer, nil+}+// finalizeAnswer issues a single deterministic synthesis call so a turn that// ended abnormally (empty final, or an error aborted the loop mid-investigation)// still returns a grounded answer instead of a blank. It feeds the model thediff --git a/internal/agentic_rag/compiled_knowledge.go b/internal/agentic_rag/compiled_knowledge.goindex 904a79cce1..b81b4fdac9 100644--- a/internal/agentic_rag/compiled_knowledge.go+++ b/internal/agentic_rag/compiled_knowledge.go@@ -91,3 +91,41 @@ func datasetsHaveCompiledKnowledge(ctx context.Context, tenantID string, dataset}return len(res.Chunks) > 0}++// datasetsHaveClaims reports whether any bound dataset carries knowledge-compiled+// CLAIM rows (entity_type_kwd="claim").+//+// It is the gate for the claim prefetch and is deliberately STRICTER than+// datasetsHaveCompiledKnowledge: a dataset compiled into a navigation tree or a+// timeline has compile_kwd rows but no evidence rows, so the looser probe would+// let the pipeline spend a routing call plus two retrieval legs per document to+// learn that there is nothing to prefetch. claimRowsPresent wraps it so tests can+// drive the gate without an engine.+func datasetsHaveClaims(ctx context.Context, tenantID string, datasetIDs []string) bool {+ if len(datasetIDs) == 0 {+ return false+ }+ de := engine.Get()+ if de == nil {+ return false+ }+ res, err := de.Search(ctx, &enginetypes.SearchRequest{+ IndexNames: []string{indexNameFor(tenantID, "")},+ KbIDs: datasetIDs,+ SelectFields: []string{"id"},+ Filter: map[string]interface{}{"entity_type_kwd": []string{navClaimRowType}},+ Limit: 1,+ IncludeUnavailable: true,+ })+ if err != nil || res == nil {+ common.DebugCtx(ctx, "agentic_rag: claim probe failed",+ zap.String("tenant_id", tenantID),+ zap.Strings("datasets", datasetIDs),+ zap.Error(err))+ return false+ }+ return len(res.Chunks) > 0+}++// claimRowsPresent is the claim-prefetch gate; tests replace it.+var claimRowsPresent = datasetsHaveClaimsdiff --git a/internal/agentic_rag/tool_navigate_structure.go b/internal/agentic_rag/tool_navigate_structure.goindex ad06bfbbcd..cedb0fc622 100644--- a/internal/agentic_rag/tool_navigate_structure.go+++ b/internal/agentic_rag/tool_navigate_structure.go@@ -20,11 +20,17 @@ package agentic_rag// <structure_navigation> outline with chunk pointers the model can feed to// list_chunks.//-// This tool reads the compiled-structure rows directly from the doc-store and-// drills them with a keyword-relevance beam, the documented no-vector-// fallback. The dense-vector seed and claim-recall legs are intentionally-// omitted; they are additive and can be layered on without changing the-// tool's contract.+// Two selection strategies, in priority order:+//+// 1. CLAIM recall (flat): the document's claim rows (entity_type_kwd="claim")+// are atomic propositions carrying their own verbatim evidence, recalled+// with a hybrid lexical + dense leg fused by reciprocal rank. A hit is+// already the answer material, so it REPLACES the drill instead of+// layering on top of it: descending the tree can only re-derive what the+// claim states, and every descent step is another chance to prune the+// branch holding the answer.+// 2. TREE drill-down (fallback): the compiled structure is drilled with a+// keyword-relevance beam for documents without claim hits.//// Typical use: call navigate_tree first to route to a document, take its// doc_id, then call navigate_structure with that doc_id to see where the answer@@ -34,8 +40,11 @@ import ("context""encoding/json""fmt"+ "slices""sort"+ "strconv""strings"+ "unicode"einotool "github.com/cloudwego/eino/components/tool""github.com/cloudwego/eino/schema"@@ -45,14 +54,16 @@ import ("ragflow/internal/common""ragflow/internal/engine"enginetypes "ragflow/internal/engine/types"+ "ragflow/internal/service/nav"+ "ragflow/internal/tokenizer")const navigateStructureToolName = "navigate_structure"const navigateStructureToolDescription = `Drill into ONE document's COMPILED STRUCTURE (the entity/relation outline built at compile time) and return a query-focused outline of where the answer sits, with chunk pointers you can pass to list_chunks to deep-read.-CALL IT AFTER navigate_tree has routed you to a document: take a doc_id from the <tree_navigation> result and pass it as doc_id. It shows the document's internal structure (entities and how they relate) instead of scanning chunks blindly.-Returns an XML <structure_navigation> document: one <doc> element per readable structure, each with a <structure> outline of "name (type): description [chunks: c1,c2]" lines and optional <claims_sufficient/>. Feed the chunk pointers to list_chunks (anchor_chunk_ids).-An empty result (<structure_navigation count="0" error="...">) means this document has NO compiled structure of the requested kind - the error attribute says which: "no compiled navigation tree" (route first with navigate_tree), "no document located" (no doc_id), or "no structure" (structure absent for this kind in THAT document - fall back to grep_chunks / search_*_chunks; the verdict is per document, so the tool stays available for other documents).`+Call it with just a query: it routes the query over the datasets' compiled navigation trees itself and returns the compiled CLAIM evidence of the best-matching documents, so no prior navigate_tree call is needed. To drill ONE document the model already chose, pass the doc_id from a <tree_navigation> result instead; that path returns the document's internal structure (entities and how they relate).+Returns an XML <structure_navigation> document: one <doc> element per readable structure, each with a <structure> outline. Outline lines are either claim lines "- [claim] <statement> | Evidence: \"<verbatim quote>\" [chunks: c1] ref=3" (the document's compiled atomic propositions recalled for your query, each carrying its own evidence and the handle to cite it by) or structure lines "- name (type): description [chunks: c1,c2]". When claims hit they replace the structure descent outright. Feed the chunk pointers to list_chunks (anchor_chunk_ids).+An empty result (<structure_navigation count="0" error="...">) means this document has NO compiled structure or claims of the requested kind - the error attribute says which: "route first with navigate_tree" (no doc_id), or "no compiled structure for this kind" (absent in THAT document - fall back to grep_chunks / search_*_chunks; the verdict is per document, so the tool stays available for other documents).`// navigateStructureArgs is the JSON the model sends into InvokableRun.type navigateStructureArgs struct {@@ -83,13 +94,6 @@ var (shapeKwds = []string{"graph", "entity", "relation"})-// Empty-reason constants, mirroring the harness navigation contract.-const (- navStructReasonBadArgs = "bad_args"- navStructReasonNoStructure = "no_structure"- navStructReasonNoDoc = "no_doc"-)-// Drill-down bounds (mirroring the harness _STRUCT_* limits).const (structMaxDepth = 3 // max TOC levels drilled@@ -100,6 +104,101 @@ const (structMaxChunks = 4 // cap on chunk snippets returned in the outline)+// Claim/evidence leg (mirroring the harness _STRUCT_CLAIM_* limits).+const (+ // navClaimRowType is the entity_type_kwd every compiler gives its evidence+ // rows. They deliberately carry no knowledge_graph_kwd and no type_kwd, so+ // they never show up as artifacts-tree nodes — this filter is the only way+ // to find them.+ navClaimRowType = "claim"+ // navClaimDocScope pins recall to doc-level rows: the KB-wide merged rows+ // written by the Build button carry a doc_id too, and without this filter+ // they would answer for a single document.+ navClaimDocScope = "doc"+ // navClaimTopN is the claim count rendered per document.+ navClaimTopN = 8+ // navClaimEvidenceChars caps the rendered verbatim quote. Generous on+ // purpose: a thin quote makes the answer stage judge the context+ // insufficient, and one extra re-search turn costs a full prompt — far more+ // than the extra characters.+ navClaimEvidenceChars = 1200+ // navClaimSufficientHits is the count at which the model may cite the claims+ // without deep-reading chunks. It is a count of RELEVANT hits: both legs+ // apply a relevance floor (below), because a leg with no floor returns a full+ // page for any document and a count over it states nothing.+ navClaimSufficientHits = 3+ // navClaimMinCoverage is the relevance floor applied to every recalled claim:+ // the share of the question's content tokens the claim text must contain.+ // The store's own defaults make a floor mandatory — full-text recall runs at+ // minimum_should_match 0% (one shared token is a hit) and KNN returns the+ // nearest neighbours at any distance — so without it every document yielded a+ // full page for every question. Measured failure: a Mike Tyson question+ // answered from a TV-series document.+ navClaimMinCoverage = 0.3+ // navClaimDenseSimilarity is the dense leg's cosine floor: the paraphrase leg+ // must not hand the fusion rows that are merely nearest, not near.+ navClaimDenseSimilarity = 0.4+ // navClaimMinDocs is the document diversity a sufficient recall needs. One+ // document's claims answer one hop; a multi-document question needs evidence+ // from more than one place, and a single-document recall is exactly what a+ // degenerate router returns for every question.+ navClaimMinDocs = 2+ // navClaimRRFK is the reciprocal-rank-fusion constant. RRF consumes ranks+ // only, so the two legs' scores never have to be comparable — their FLOORS+ // are what decides membership.+ navClaimRRFK = 60+ // navClaimPool is the per-leg candidate pool (mirroring limit = max(top_n, 32)).+ navClaimPool = 32+ // navQueryRouteDocs bounds the documents a query-only call investigates. The+ // two-step contract (navigate_tree -> navigate_structure) is a real adoption+ // barrier: a query-only call removes it, at the cost of claiming over a few+ // routed documents instead of one chosen by the model.+ navQueryRouteDocs = 3+ // navClaimPrefetchTopN bounds the claims the PIPELINE recalls before the+ // loop. It is both a cost bound and an accuracy bound: the block is the whole+ // prompt of the sufficient-claims answer and also what decides that verdict,+ // so a cap below navClaimSufficientHits x navClaimMinDocs could never+ // terminate early.+ navClaimPrefetchTopN = 3 * navClaimTopN+)++// navStructureRoute resolves candidate doc_ids for a query when the caller+// supplied no doc_id. It is the same navigation-tree router navigate_tree uses,+// so a query-only call and the explicit two-step path rank documents identically.+var navStructureRoute = func(ctx context.Context, tenantID, kbID, query string, limit int) ([][2]string, error) {+ return nav.NewNavServiceRouter().Route(ctx, tenantID, kbID, query, nil, limit)+}++// routeDocIDsForQuery routes the query across the bound datasets and returns the+// top doc_ids, deduped across datasets and capped at limit.+func routeDocIDsForQuery(ctx context.Context, tenantID, query string, datasetIDs []string, limit int) []string {+ if limit <= 0 {+ limit = navQueryRouteDocs+ }+ seen := map[string]bool{}+ var out []string+ for _, kbID := range datasetIDs {+ routed, err := navStructureRoute(ctx, tenantID, kbID, query, limit)+ if err != nil {+ common.DebugCtx(ctx, "navigate_structure: query route failed",+ zap.String("kb_id", kbID), zap.Error(err))+ continue+ }+ for _, pair := range routed {+ did := strings.TrimSpace(pair[0])+ if did == "" || seen[did] {+ continue+ }+ seen[did] = true+ out = append(out, did)+ if len(out) >= limit {+ return out+ }+ }+ }+ return out+}+// structureRel is a parent->child relation of a compiled structure hierarchy.type structureRel struct {from string@@ -113,7 +212,6 @@ type structureNode struct {nodeType stringdesc stringsourceChunkIDs []string- vec []float64}// chunkWithText pairs a chunk id with its text for structure snippet ranking.@@ -122,24 +220,64 @@ type chunkWithText struct {text string}-// structureDrillout is the per-document result of the TOC drill-down.-type structureDrillout struct {- outline string- nodes int- chunkPtrs int- topScore float64- chunkPaths map[string]string- selector string+// claimHit is one recalled claim row: an atomic proposition plus its verbatim+// evidence. chunkID is the FIRST source chunk id — the pointer the model feeds+// to list_chunks when it needs the surrounding passage. docID names the document+// the claim was compiled from, which is what makes document diversity checkable.+type claimHit struct {+ docID string+ chunkID string+ name string+ description string+ quote string+ score float64+}++// claimReader is the OPTIONAL claim-recall capability of a structureReader. A+// reader that does not implement it simply has no claim leg, and the drill-down+// stays the only strategy (the pre-claim behavior).+type claimReader interface {+ // ReadClaims recalls claim rows of the given compile kinds, hybrid (lexical ++ // dense when qvec is non-empty) fused by reciprocal rank, and returns at most+ // limit hits that clear the relevance floor. A nil/empty docIDs spans the+ // whole dataset, which is what the pre-loop recall uses: it needs no document+ // choice.+ ReadClaims(ctx context.Context, tenantID, kbID string, docIDs, kinds []string, text string, qvec []float64, limit int) ([]claimHit, error)+}++// claimQueryVector encodes the drill query for the claim dense leg, using the+// embedder the nav service already resolved for this tenant — the same model+// that wrote the claim rows' q_<dim>_vec columns.+//+// It reads that embedder through the nav service rather than through an injected+// function: this package already depends on nav for routing, and the service is+// where tenant-model resolution lives, so a second wiring point in the server+// bootstrap would only duplicate it. A missing service, an unresolvable model or+// a panicking encoder all yield nil, which leaves the leg out and recall running+// on the lexical leg alone — a misconfigured embedder must never take a tool+// call down.+func claimQueryVector(ctx context.Context, tenantID, text string) (vec []float64) {+ svc := nav.GetNavService()+ if svc == nil || tenantID == "" || strings.TrimSpace(text) == "" {+ return nil+ }+ defer func() {+ if recover() != nil {+ vec = nil+ }+ }()+ if v := svc.EncodeQueryVector(ctx, tenantID, text); len(v) > 0 {+ return v+ }+ return nil}-// StructureRow is one compiled-structure row: the doc-store fields the structure+// structureRow is one compiled-structure row: the doc-store fields the structure// reader reads.-type StructureRow struct {- CompileKwd string+type structureRow struct {TemplateKind stringKnowledgeGraphKwd stringContent string- Vec []float64}// structureReader reads a document's compiled structure rows. The production@@ -147,13 +285,13 @@ type StructureRow struct {// is installed (or the engine is available) navigate_structure reports// EMPTY/no_structure, falling through to the text tools.type structureReader interface {- ReadStructure(ctx context.Context, tenantID, kbID, docID string) ([]StructureRow, error)+ ReadStructure(ctx context.Context, tenantID, kbID, docID string) ([]structureRow, error)}var structReaderInst structureReader-// SetStructureReader installs the (production or test) structure reader.-func SetStructureReader(r structureReader) {+// setStructureReader installs the (production or test) structure reader.+func setStructureReader(r structureReader) {structReaderInst = r}@@ -169,7 +307,7 @@ func getStructureReader() structureReader {// scoped to ONE document (the tool already knows which document to drill).type engineStructureReader struct{}-func (engineStructureReader) ReadStructure(ctx context.Context, tenantID, kbID, docID string) ([]StructureRow, error) {+func (engineStructureReader) ReadStructure(ctx context.Context, tenantID, kbID, docID string) ([]structureRow, error) {de := engine.Get()if de == nil {return nil, nil@@ -177,7 +315,7 @@ func (engineStructureReader) ReadStructure(ctx context.Context, tenantID, kbID,req := &enginetypes.SearchRequest{IndexNames: []string{indexNameFor(tenantID, "")},KbIDs: []string{kbID},- SelectFields: []string{"content_with_weight", "compile_kwd", "compilation_template_kind_kwd", "type_kwd", "knowledge_graph_kwd"},+ SelectFields: []string{"content_with_weight", "compilation_template_kind_kwd", "type_kwd", "knowledge_graph_kwd"},Filter: map[string]interface{}{"doc_id": []string{docID}, "knowledge_graph_kwd": shapeKwds},Limit: 3000,IncludeUnavailable: true,@@ -186,14 +324,13 @@ func (engineStructureReader) ReadStructure(ctx context.Context, tenantID, kbID,if err != nil {return nil, err}- rows := make([]StructureRow, 0, len(res.Chunks))+ rows := make([]structureRow, 0, len(res.Chunks))for _, row := range res.Chunks {kg, _ := row["type_kwd"].(string)if kg == "" {kg, _ = row["knowledge_graph_kwd"].(string)}- sr := StructureRow{- CompileKwd: fmt.Sprint(row["compile_kwd"]),+ sr := structureRow{TemplateKind: fmt.Sprint(row["compilation_template_kind_kwd"]),KnowledgeGraphKwd: kg,Content: fmt.Sprint(row["content_with_weight"]),@@ -203,6 +340,337 @@ func (engineStructureReader) ReadStructure(ctx context.Context, tenantID, kbID,return rows, nil}+// ReadClaims is the claim leg: two independent reads over the claim rows of the+// requested documents — a full-text leg on the tokenized content (exact /+// proper-noun recall) and a KNN leg on the claim's OWN vector (paraphrase+// recall) — fused by reciprocal rank. Each leg is one store round-trip.+//+// A leg that fails is dropped rather than propagated: without an embedder the+// lexical leg alone still recalls claims, and a scope with no compiled claims+// legitimately returns nothing.+//+// The RELEVANCE FLOOR is applied here, in Go, to the fused result. The store+// cannot enforce it: `minimum_should_match` on a multi-field query_string is+// silently ignored (measured: "100%" still returned 1950 rows for a two-term+// query), and a KNN similarity cut would only cover the dense leg. Without a+// floor every document yielded a full page for every question, which is how a+// Mike Tyson question was answered from a TV-series document.+func (engineStructureReader) ReadClaims(ctx context.Context, tenantID, kbID string, docIDs, kinds []string, text string, qvec []float64, limit int) ([]claimHit, error) {+ de := engine.Get()+ if de == nil || strings.TrimSpace(text) == "" {+ return nil, nil+ }+ // The caller owns the cap: the per-document tool path asks for navClaimTopN,+ // while the pre-loop recall asks for the corpus-wide budget+ // (navClaimPrefetchTopN). Clamping every caller to the per-document count+ // silently cut that budget to one page, which is too few claims to satisfy+ // the sufficiency gate's document-diversity rule.+ if limit <= 0 {+ limit = navClaimTopN+ }+ pool := limit+ if pool < navClaimPool {+ pool = navClaimPool+ }+ filter := map[string]interface{}{+ "entity_type_kwd": []string{navClaimRowType},+ "scope_kwd": []string{navClaimDocScope},+ }+ if len(docIDs) > 0 {+ filter["doc_id"] = docIDs+ }+ if len(kinds) > 0 {+ filter["compile_kwd"] = kinds+ }++ leg := func(exprs []interface{}) []claimHit {+ res, err := de.Search(ctx, &enginetypes.SearchRequest{+ IndexNames: []string{indexNameFor(tenantID, "")},+ KbIDs: []string{kbID},+ SelectFields: []string{+ "content_with_weight", "source_chunk_ids", "entity_type_kwd", "compile_kwd", "doc_id",+ },+ Filter: filter,+ MatchExprs: exprs,+ Limit: pool,+ IncludeUnavailable: true,+ })+ if err != nil {+ common.DebugCtx(ctx, "claim recall: leg failed",+ zap.Strings("doc_ids", docIDs), zap.Error(err))+ return nil+ }+ hits := make([]claimHit, 0, len(res.Chunks))+ for _, row := range res.Chunks {+ if h, ok := parseClaimHit(row); ok {+ hits = append(hits, h)+ }+ }+ return hits+ }++ var denseLeg, textLeg []claimHit+ if len(qvec) > 0 {+ denseLeg = leg([]interface{}{&enginetypes.MatchDenseExpr{+ VectorColumnName: fmt.Sprintf("q_%d_vec", len(qvec)),+ EmbeddingData: qvec,+ EmbeddingDataType: "float",+ DistanceType: "cosine",+ TopN: pool,+ ExtraOptions: map[string]interface{}{"similarity": navClaimDenseSimilarity},+ }})+ }+ textLeg = leg([]interface{}{&enginetypes.MatchTextExpr{+ Fields: []string{"content_ltks", "content_sm_ltks"},+ MatchingText: text,+ TopN: pool,+ }})+ if len(denseLeg) == 0 && len(textLeg) == 0 {+ return nil, nil+ }++ fused := rrfFuse(navClaimRRFK, pool, denseLeg, textLeg)+ question := claimTokens(text)+ out := make([]claimHit, 0, len(fused))+ for _, h := range fused {+ if claimCoverage(question, h) < navClaimMinCoverage {+ continue+ }+ out = append(out, h)+ if len(out) >= limit {+ break+ }+ }+ return out, nil+}++// parseClaimHit turns one claim row into a hit. A row that is not a usable+// claim — bad payload, unnamed, or carrying no chunk pointer — is dropped: a+// claim the model cannot open is not an answer anchor.+func parseClaimHit(row map[string]interface{}) (claimHit, bool) {+ payload, ok := decodeJSONObject(contentWithWeightFromRaw(row))+ if !ok {+ return claimHit{}, false+ }+ name := strings.TrimSpace(stringFromAny(payload["name"]))+ if name == "" {+ return claimHit{}, false+ }+ ids := navStringList(row["source_chunk_ids"])+ if len(ids) == 0 {+ ids = navStringList(payload["source_chunk_ids"])+ }+ if len(ids) == 0 {+ return claimHit{}, false+ }+ docID := strings.TrimSpace(fmt.Sprint(row["doc_id"]))+ if docID == "<nil>" {+ docID = ""+ }+ return claimHit{+ docID: docID,+ chunkID: ids[0],+ name: name,+ description: strings.TrimSpace(stringFromAny(payload["description"])),+ quote: firstEvidenceQuote(payload["evidence"]),+ score: floatFromMap(row, "_score"),+ }, true+}++// claimTokens returns the set of content tokens of a question, used as the+// denominator of the claim relevance floor. It uses the corpus tokenizer when+// the pool is up (same analyzer the claim text was indexed with) and falls back+// to a rune-level splitter, so a deployment without the analyzer still gets a+// relevance floor instead of none. Single-rune tokens are dropped: they are+// stopwords in alphabetic languages.+func claimTokens(text string) map[string]struct{} {+ tokens := map[string]struct{}{}+ if tok, err := tokenizer.Tokenize(text); err == nil {+ for _, t := range strings.Fields(strings.ToLower(tok)) {+ if len([]rune(t)) >= 2 {+ tokens[t] = struct{}{}+ }+ }+ }+ if len(tokens) > 0 {+ return tokens+ }+ for _, t := range strings.FieldsFunc(strings.ToLower(text), func(r rune) bool {+ return !unicode.IsLetter(r) && !unicode.IsDigit(r)+ }) {+ if len([]rune(t)) >= 2 {+ tokens[t] = struct{}{}+ }+ }+ return tokens+}++// claimCoverage is the share of the question's content tokens this claim states+// or quotes. It is the floor's only input: a claim that does not repeat any of+// the question's terms cannot be evidence for it, however highly the store+// ranked it (ranks are relative, so the top hit of an irrelevant document is+// still the top hit).+func claimCoverage(question map[string]struct{}, c claimHit) float64 {+ if len(question) == 0 {+ return 1+ }+ text := c.name + " " + c.description + " " + c.quote+ claim := claimTokens(text)+ if len(claim) == 0 {+ return 0+ }+ matched := 0+ for t := range question {+ if _, ok := claim[t]; ok {+ matched+++ }+ }+ return float64(matched) / float64(len(question))+}++// firstEvidenceQuote returns the first non-empty verbatim quote of a claim's+// evidence list.+func firstEvidenceQuote(v any) string {+ list, _ := v.([]any)+ for _, item := range list {+ ev, ok := item.(map[string]any)+ if !ok {+ continue+ }+ if q := strings.TrimSpace(stringFromAny(ev["quote"])); q != "" {+ return q+ }+ }+ return ""+}++// rrfFuse merges retrieval legs by RECIPROCAL RANK (k=60 by default). Ranks+// only, so no similarity threshold has to be calibrated across legs with+// incomparable score scales; a hit keeps its best raw score for diagnostics.+func rrfFuse(k, limit int, legs ...[]claimHit) []claimHit {+ if k <= 0 {+ k = navClaimRRFK+ }+ type fused struct {+ hit claimHit+ score float64+ }+ byName := map[string]*fused{}+ var order []string+ for _, leg := range legs {+ for rank, h := range leg {+ key := strings.ToLower(strings.TrimSpace(h.name))+ if key == "" {+ continue+ }+ f, ok := byName[key]+ if !ok {+ byName[key] = &fused{hit: h, score: 1.0 / float64(k+rank+1)}+ order = append(order, key)+ continue+ }+ f.score += 1.0 / float64(k+rank+1)+ if h.score > f.hit.score {+ f.hit = h+ }+ }+ }+ if len(order) == 0 {+ return nil+ }+ sort.SliceStable(order, func(i, j int) bool {+ a, b := byName[order[i]], byName[order[j]]+ if a.score != b.score {+ return a.score > b.score+ }+ return order[i] < order[j]+ })+ if limit > 0 && len(order) > limit {+ order = order[:limit]+ }+ out := make([]claimHit, 0, len(order))+ for _, key := range order {+ out = append(out, byName[key].hit)+ }+ return out+}++// navStringList coerces a doc-store column into a string slice: engines return+// []string, []interface{} or a bare string for the same field.+func navStringList(v any) []string {+ switch x := v.(type) {+ case []string:+ var out []string+ for _, s := range x {+ if s = strings.TrimSpace(s); s != "" {+ out = append(out, s)+ }+ }+ return out+ case []any:+ var out []string+ for _, item := range x {+ if s := strings.TrimSpace(stringFromAny(item)); s != "" {+ out = append(out, s)+ }+ }+ return out+ case string:+ if s := strings.TrimSpace(x); s != "" {+ return []string{s}+ }+ }+ return nil+}++// stringFromAny reads a scalar as a trimmed string, tolerating the numeric+// types engines decode JSON numbers into.+func stringFromAny(v any) string {+ switch x := v.(type) {+ case string:+ return x+ case fmt.Stringer:+ return x.String()+ case nil:+ return ""+ default:+ return fmt.Sprint(x)+ }+}++// claimOutlineLine renders one claim hit: the statement, its verbatim evidence,+// and the chunk pointer that opens the surrounding passage.+//+// The line is ANSWER MATERIAL (a claim carries its own evidence, so the model is+// expected to cite it instead of re-reading the document), which makes the+// passage handle mandatory: an uncited claim line ships an answer the reference+// payload cannot resolve. The handle is appended unquoted (`ref=3`) on purpose -+// claim lines are rendered inside an escaped <structure> text node, where a+// `ref="3"` attribute would arrive as `ref="3"`.+func claimOutlineLine(ctx context.Context, c claimHit) string {+ line := "- [claim] " + c.name+ if q := strings.TrimSpace(c.quote); q != "" {+ line += ` | Evidence: "` + snippetRunes(q, navClaimEvidenceChars) + `"`+ }+ if c.chunkID != "" {+ line += " [chunks: " + c.chunkID + "]"+ }+ if n := evidenceRegistryFrom(ctx).Stamp(c.chunkID); n >= 0 {+ line += " ref=" + strconv.Itoa(n)+ }+ return line+}++// claimOutline is the claim-first selection: the claims ARE the outline, so+// nothing is drilled and no chunk snippet is loaded.+func claimOutline(ctx context.Context, claims []claimHit) string {+ lines := make([]string, 0, len(claims))+ for _, c := range claims {+ lines = append(lines, claimOutlineLine(ctx, c))+ }+ return strings.Join(lines, "\n")+}+// indexNameFor resolves the document/structure search index: the tenant's// default "ragflow_<tenant_id>" unless a tenant overrides index_name.func indexNameFor(tenantID, configured string) string {@@ -224,22 +692,11 @@ func normalizeKindKey(kind string) string {return strings.ToLower(kind)}-func normalizeKind(row map[string]interface{}) string {- return normalizeKindKey(compilationKind(row))-}--func compilationKind(row map[string]interface{}) string {- if v, ok := row["compilation_template_kind_kwd"].(string); ok && v != "" {- return v- }- return fmt.Sprint(row["knowledge_graph_kwd"])-}-// rowKind derives the normalized structure kind from a compiled-structure row,// using its knowledge_graph_kwd (the row SHAPE: graph / entity / relation) or,// failing that, its compilation template kind. This is what keeps a "graph"// blob from being filtered out as the wrong kind.-func rowKind(row StructureRow) string {+func rowKind(row structureRow) string {kind := row.KnowledgeGraphKwdif kind == "" {kind = row.TemplateKind@@ -247,9 +704,9 @@ func rowKind(row StructureRow) string {return normalizeKindKey(kind)}-// ParseCompiledStructure merges the two row shapes (graph blob / per-entity /+// parseCompiledStructure merges the two row shapes (graph blob / per-entity /// per-relation) into entities / relations, mirroring _query in navigation.py.-func ParseCompiledStructure(rows []StructureRow, kinds []string) ([]map[string]any, []map[string]any) {+func parseCompiledStructure(rows []structureRow, kinds []string) ([]map[string]any, []map[string]any) {want := map[string]bool{}for _, k := range kinds {if k != "" {@@ -269,34 +726,20 @@ func ParseCompiledStructure(rows []StructureRow, kinds []string) ([]map[string]aif !ok {continue}- attachVec := func(list []map[string]any) {- if len(row.Vec) == 0 {- return- }- for _, m := range list {- m["_vec"] = row.Vec- }- }switch row.KnowledgeGraphKwd {case "graph":- es := objectList(graph["entities"])- rs := objectList(graph["relations"])- attachVec(es)- attachVec(rs)- entities = append(entities, es...)- relations = append(relations, rs...)+ entities = append(entities, objectList(graph["entities"])...)+ relations = append(relations, objectList(graph["relations"])...)case "entity":- attachVec([]map[string]any{graph})entities = append(entities, graph)case "relation":- attachVec([]map[string]any{graph})relations = append(relations, graph)}}return entities, relations}-// structureGraphFromRaw maps what ParseCompiledStructure returns onto the typed+// structureGraphFromRaw maps what parseCompiledStructure returns onto the typed// nodes/relations the drill-down consumes (entity -> node conversion included so// the drill-down only ever sees structureNode).func structureGraphFromRaw(rawEntities, rawRels []map[string]any) ([]structureNode, []structureRel) {@@ -309,9 +752,6 @@ func structureGraphFromRaw(rawEntities, rawRels []map[string]any) ([]structureNotyp, _ := e["type"].(string)desc, _ := e["description"].(string)n := structureNode{name: name, nodeType: typ, desc: desc}- if v, ok := e["_vec"].([]float64); ok {- n.vec = v- }if ids, ok := e["source_chunk_ids"].([]interface{}); ok {for _, id := range ids {if s, ok := id.(string); ok && s != "" {@@ -378,55 +818,6 @@ func nodeRelevance(terms []string, name, desc string) int {return n}-// nodeScore mirrors _node_score: cosine when a query vector and the node-// embedding both exist, else keyword relevance. This port leaves nodes-// vector-less, so it always uses keyword relevance.-func nodeScore(qvec []float64, terms []string, n structureNode) float64 {- if len(qvec) > 0 && len(n.vec) > 0 {- return cosine(qvec, n.vec)- }- return float64(nodeRelevance(terms, n.name, n.desc))-}--func cosine(a, b []float64) float64 {- if len(a) == 0 || len(b) == 0 || len(a) != len(b) {- return 0- }- var dot, na, nb float64- for i := range a {- dot += a[i] * b[i]- na += a[i] * a[i]- nb += b[i] * b[i]- }- if na == 0 || nb == 0 {- return 0- }- return dot / (sqrt(na) * sqrt(nb))-}--func sqrt(x float64) float64 {- if x < 0 {- return 0- }- r := x- for i := 0; i < 16; i++ {- if r == 0 {- break- }- r = 0.5 * (r + x/r)- }- return r-}--func sliceContains(s []string, v string) bool {- for _, x := range s {- if x == v {- return true- }- }- return false-}-// buildTocTree mirrors _build_toc_tree: a name->node map, the parent->children// map, the child->parent map (single parent, last-wins) and the root names.func buildTocTree(nodes []structureNode, rels []structureRel) (map[string]structureNode, map[string][]string, map[string]string, []string) {@@ -487,14 +878,12 @@ func chunkPtrs(ids []string) string {}var seen []stringfor _, c := range ids {- if c == "" {+ if c == "" || slices.Contains(seen, c) {continue}- if !sliceContains(seen, c) {- seen = append(seen, c)- if len(seen) >= 8 {- break- }+ seen = append(seen, c)+ if len(seen) >= 8 {+ break}}return strings.Join(seen, ",")@@ -509,10 +898,7 @@ func collectChunkIDs(nodes []structureNode, capN int) []string {var out []stringfor _, n := range nodes {for _, c := range n.sourceChunkIDs {- if c == "" {- continue- }- if sliceContains(out, c) {+ if c == "" || slices.Contains(out, c) {continue}out = append(out, c)@@ -535,34 +921,17 @@ func sortedKeptNames(names map[string]bool) []string {return out}-// drillWithAncestors pulls the ancestors of names into the kept set so a path-// renders as root -> ... -> node.-func drillWithAncestors(names map[string]bool, parents map[string]string) map[string]bool {- for name := range names {- cur := parents[name]- guard := 0- for cur != "" && !names[cur] && guard < structMaxDepth {- names[cur] = true- cur = parents[cur]- guard++- }- }- return names-}--// drillKeptNodes mirrors _drill_kept_nodes: vector-beam BFS descent from the TOC-// roots, keeping top-K children per level by keyword relevance (this port has no-// node vectors, so it always uses the keyword path), returning the kept nodes,-// the single-parent map, the kept-name set and the overall best score.-func drillKeptNodes(qvec []float64, terms []string, nodes []structureNode, rels []structureRel) ([]structureNode, map[string]string, map[string]bool, float64) {+// drillKeptNodes descends from the TOC roots, keeping top-K children per level by+// keyword relevance, and returns the kept nodes, the single-parent map and the+// kept-name set.+func drillKeptNodes(terms []string, nodes []structureNode, rels []structureRel) ([]structureNode, map[string]string, map[string]bool) {byName, children, parents, roots := buildTocTree(nodes, rels)if len(roots) == 0 {- return nil, nil, nil, 0.0+ return nil, nil, nil}frontier := rootskeptNames := map[string]bool{}depth := 0- bestOverall := 0.0for len(frontier) > 0 && depth <= structMaxDepth {type scored struct {score float64@@ -571,20 +940,13 @@ func drillKeptNodes(qvec []float64, terms []string, nodes []structureNode, relsvar scoredList []scoredfor _, n := range frontier {if e, ok := byName[n]; ok {- scoredList = append(scoredList, scored{nodeScore(qvec, terms, e), n})+ scoredList = append(scoredList, scored{float64(nodeRelevance(terms, e.name, e.desc)), n})}}if len(scoredList) == 0 {break}sort.SliceStable(scoredList, func(i, j int) bool { return scoredList[i].score > scoredList[j].score })- best := scoredList[0].score- if best > bestOverall {- bestOverall = best- }- if len(qvec) == 0 && best < float64(structRelevanceMin) {- break- }var top []scoredfor _, s := range scoredList {if s.score >= float64(structRelevanceMin) {@@ -599,14 +961,13 @@ func drillKeptNodes(qvec []float64, terms []string, nodes []structureNode, rels}var next []stringfor _, s := range top {- if !keptNames[s.name] {- keptNames[s.name] = true- }+ keptNames[s.name] = truenext = append(next, children[s.name]...)}frontier = nextdepth++}+ // Ancestors of a kept node are pulled in so a path renders root -> node.for name := range keptNames {cur := parents[name]guard := 0@@ -622,21 +983,7 @@ func drillKeptNodes(qvec []float64, terms []string, nodes []structureNode, relskept = append(kept, e)}}- return kept, parents, keptNames, bestOverall-}--// outlineStats mirrors _outline_stats: the entity / chunk-pointer counts for the-// flat fallback outline (no drill happened => top_score 0).-func outlineStats(nodes []structureNode) (nodeCount, ptrCount int) {- count := len(nodes)- if count > structMaxNodes {- count = structMaxNodes- }- ptrs := 0- for _, n := range nodes[:count] {- ptrs += len(collectChunkIDs([]structureNode{n}, 32))- }- return count, ptrs+ return kept, parents, keptNames}// navTypeOr returns a node type, defaulting to "other" (a missing key reads as@@ -652,7 +999,7 @@ func navTypeOr(t string) string {func navOutlineLine(indent, name, nodeType, desc string, chunks []string) string {line := indent + "- " + name + " (" + navTypeOr(nodeType) + ")"if desc != "" {- line += ": " + Snippet(desc, structDescSnippet)+ line += ": " + snippetRunes(desc, structDescSnippet)}if c := chunkPtrs(chunks); c != "" {line += " [chunks: " + c + "]"@@ -684,26 +1031,18 @@ func renderOutline(nodes []structureNode, rels []structureRel) string {return strings.Join(lines, "\n")}-// renderTocDrilldown mirrors _render_toc_drilldown's flat + beam paths: when no-// terms/vector it falls back to the flat outline; otherwise it vector-beam-// descends toward the query by keyword relevance. loader, when non-nil, fetches-// chunk texts so short snippets can be appended; a nil loader skips the snippet-// pass.-func renderTocDrilldown(query string, qvec []float64, nodes []structureNode, rels []structureRel, loader func(ids []string) []chunkWithText) structureDrillout {- flat := func() structureDrillout {- out := renderOutline(nodes, rels)- n, p := outlineStats(nodes)- return structureDrillout{outline: out, nodes: n, chunkPtrs: p, topScore: 0.0, chunkPaths: map[string]string{}, selector: "flat"}- }+// renderTocDrilldown drills a document's compiled structure toward the query: no+// usable query terms fall back to the flat outline, otherwise the tree is+// descended by keyword relevance. loader, when non-nil, fetches chunk texts so+// short snippets can be appended.+func renderTocDrilldown(query string, nodes []structureNode, rels []structureRel, loader func(ids []string) []chunkWithText) string {terms := queryTerms(query)- if len(terms) == 0 && len(qvec) == 0 {- return flat()+ if len(terms) == 0 {+ return renderOutline(nodes, rels)}- kept, parents, keptNames, best := drillKeptNodes(qvec, terms, nodes, rels)+ kept, parents, keptNames := drillKeptNodes(terms, nodes, rels)if len(kept) == 0 {- out := flat()- out.selector = "beam"- return out+ return renderOutline(nodes, rels)}// depth_of: node -> number of kept ancestors (rendering indentation).depthOf := map[string]int{}@@ -716,33 +1055,6 @@ func renderTocDrilldown(query string, qvec []float64, nodes []structureNode, rel}depthOf[n] = d}- chunkPaths := map[string]string{}- for _, e := range kept {- name := strings.TrimSpace(e.name)- if name == "" {- continue- }- var segs []string- segs = append(segs, name)- cur := parents[name]- guard := 0- for cur != "" && guard < structMaxDepth {- segs = append(segs, cur)- cur = parents[cur]- guard++- }- for i, j := 0, len(segs)-1; i < j; i, j = i+1, j-1 {- segs[i], segs[j] = segs[j], segs[i]- }- path := strings.Join(segs, " -> ")- for _, cid := range e.sourceChunkIDs {- if cid != "" {- if _, seen := chunkPaths[cid]; !seen {- chunkPaths[cid] = path- }- }- }- }var lines []stringcount := len(kept)if count > structMaxNodes {@@ -756,7 +1068,7 @@ func renderTocDrilldown(query string, qvec []float64, nodes []structureNode, relindent := strings.Repeat(" ", depthOf[name])lines = append(lines, navOutlineLine(indent, name, e.nodeType, e.desc, e.sourceChunkIDs))}- if len(kept) > 0 && loader != nil {+ if loader != nil {wanted := collectChunkIDs(kept, 32)if len(wanted) > 0 {chunks := loader(wanted)@@ -769,89 +1081,166 @@ func renderTocDrilldown(query string, qvec []float64, nodes []structureNode, relif text == "" {continue}- lines = append(lines, "- [chunk "+c.id+"]: "+Snippet(text, 300))+ lines = append(lines, "- [chunk "+c.id+"]: "+snippetRunes(text, 300))}}}- return structureDrillout{- outline: strings.Join(lines, "\n"),- nodes: len(kept),- chunkPtrs: len(kept),- topScore: best,- chunkPaths: chunkPaths,- selector: "beam",- }+ return strings.Join(lines, "\n")}// structureDocSegment renders one <doc> element (doc_id / entities / relations// plus the <structure> outline).-func structureDocSegment(docID, query, kind string, rank int, nodes []structureNode, rels []structureRel, drill structureDrillout) string {- esc := func(s string) string { return xmlEscape(s) }+func structureDocSegment(docID string, rank, entities, relations int, outline string) string {var b strings.Builder- fmt.Fprintf(&b, ` <doc rank="%d" doc_id="%s" entities="%d" relations="%d">`, rank, esc(docID), len(nodes), len(rels))- if drill.outline != "" {- b.WriteString("\n <structure>" + esc(drill.outline) + "</structure>")+ fmt.Fprintf(&b, ` <doc rank="%d" doc_id="%s" entities="%d" relations="%d">`, rank, xmlEscape(docID), entities, relations)+ if outline != "" {+ b.WriteString("\n <structure>" + xmlEscape(outline) + "</structure>")}b.WriteString("\n </doc>")return b.String()}-// emptyReasonLabel maps an empty-reason constant to the XML error label.-func emptyReasonLabel(reason string) string {- switch reason {- case navStructReasonBadArgs:- return "query is required"- case navStructReasonNoDoc:- return "no document located"- default:- return "no structure"- }-}-// navigateStructures renders the compiled structures of the given documents into// a single <structure_navigation> with one <doc> per readable structure. A// document without a compiled structure of the requested kind is skipped; an// empty result (no doc carried a structure) reports no_structure.-func navigateStructures(ctx context.Context, tenantID string, query string, docIDs, datasetIDs []string, kind string, reader structureReader) (string, string) {+func navigateStructures(ctx context.Context, tenantID string, query string, docIDs, datasetIDs []string, kind string, reader structureReader) string {if len(docIDs) == 0 {- return navStructEmpty(navStructReasonNoDoc, "route first with navigate_tree"), navStructReasonNoDoc+ return navStructEmpty("route first with navigate_tree")}kinds := structureKindsFor(kind)var segs []string- var docIDsSeen []stringfor _, did := range docIDs {- drill, nodes, rels, empty := readStructureDocCore(ctx, tenantID, query, did, kind, kinds, datasetIDs, reader)- if empty != "" {+ outline, entities, relations, ok := readStructureDocCore(ctx, tenantID, query, did, kinds, datasetIDs, reader)+ if !ok {+ continue+ }+ segs = append(segs, structureDocSegment(did, len(segs)+1, entities, relations, outline))+ }+ if len(segs) == 0 {+ return navStructEmpty("no compiled structure for this kind")+ }+ parts := append([]string{+ fmt.Sprintf(`<structure_navigation count="%d" query="%s" kind="%s">`, len(segs), xmlEscape(query), xmlEscape(kind)),+ }, segs...)+ parts = append(parts, "</structure_navigation>")+ return strings.Join(parts, "\n")+}++// navigateClaimsForQuery is the query-only path: it claims over the ROUTED+// documents and nothing else. No tree read, no drill — a document without claim+// hits contributes nothing, because this path exists to surface compiled+// evidence, not to duplicate what the explicit doc_id path already does.+// limit caps the TOTAL claims rendered across documents (0 = uncapped, the+// model-invoked behavior). Round-robin across documents rather than first-doc+// order, so a tight cap still covers every routed document.+func navigateClaimsForQuery(ctx context.Context, tenantID, query string, docIDs, datasetIDs []string, kind string, limit int, reader structureReader) (string, int) {+ docs, perDoc := recallClaimsForQuery(ctx, tenantID, query, docIDs, datasetIDs, kind, limit, reader)+ var segs []string+ for i, claims := range perDoc {+ if len(claims) == 0 {continue}- segs = append(segs, structureDocSegment(did, query, kind, len(segs)+1, nodes, rels, drill))- docIDsSeen = append(docIDsSeen, did)+ segs = append(segs, structureDocSegment(docs[i], len(segs)+1, 0, 0, claimOutline(ctx, claims)))}if len(segs) == 0 {- return navStructEmpty(navStructReasonNoStructure, "no compiled structure for this kind"), navStructReasonNoStructure+ return "", 0}- esc := func(s string) string { return xmlEscape(s) }parts := append([]string{- fmt.Sprintf(`<structure_navigation count="%d" query="%s" kind="%s">`, len(segs), esc(query), esc(kind)),+ fmt.Sprintf(`<structure_navigation count="%d" query="%s" kind="%s">`, len(segs), xmlEscape(query), xmlEscape(kind)),}, segs...)parts = append(parts, "</structure_navigation>")- return strings.Join(parts, "\n"), ""+ return strings.Join(parts, "\n"), len(segs)+}++// claimKindList flattens the requested compile kinds into the sorted filter the+// claim reader takes. Claim rows are selected by compile_kwd, so every claim read+// needs it — the per-document tool path and the corpus-wide pre-loop recall alike.+func claimKindList(kinds map[string]bool) []string {+ out := make([]string, 0, len(kinds))+ for k := range kinds {+ out = append(out, k)+ }+ sort.Strings(out)+ return out+}++// recallClaimsForQuery runs the claim leg over the given documents and returns+// the documents that hit, aligned with their claims. limit caps the TOTAL claims+// (0 = uncapped); the trim is round-robin, so a tight budget still touches every+// document instead of being spent on the first one.+//+// It is the retrieval half of the per-document tool path; the pre-loop recall+// reads corpus-wide instead (see recallClaimsAcrossDatasets).+func recallClaimsForQuery(ctx context.Context, tenantID, query string, docIDs, datasetIDs []string, kind string, limit int, reader structureReader) ([]string, [][]claimHit) {+ kindList := claimKindList(structureKindsFor(kind))++ var perDoc [][]claimHit+ var docs []string+ for _, did := range docIDs {+ claims := recallClaimsForDoc(ctx, tenantID, query, did, kindList, datasetIDs, reader)+ if len(claims) == 0 {+ continue+ }+ perDoc = append(perDoc, claims)+ docs = append(docs, did)+ }+ if len(perDoc) == 0 {+ return nil, nil+ }+ if limit > 0 {+ perDoc = trimClaimsRoundRobin(perDoc, limit)+ }+ return docs, perDoc+}++// trimClaimsRoundRobin keeps at most limit claims in total, taking one per+// document per round so no document monopolizes a small budget.+func trimClaimsRoundRobin(perDoc [][]claimHit, limit int) [][]claimHit {+ if limit <= 0 {+ return perDoc+ }+ out := make([][]claimHit, len(perDoc))+ total := 0+ for round := 0; total < limit; round++ {+ progressed := false+ for i := range perDoc {+ if round >= len(perDoc[i]) {+ continue+ }+ out[i] = append(out[i], perDoc[i][round])+ total+++ progressed = true+ if total >= limit {+ break+ }+ }+ if !progressed {+ break+ }+ }+ return out}// readStructureDocCore reads ONE document's compiled structure of the requested-// kind and drills it toward the query. It returns the per-document drill data,-// the nodes/rels, and an empty-reason ("" means read succeeded).-func readStructureDocCore(ctx context.Context, tenantID, query, docID, kind string, kinds map[string]bool, datasetIDs []string, reader structureReader) (structureDrillout, []structureNode, []structureRel, string) {- none := structureDrillout{chunkPaths: map[string]string{}}- switch {- case query == "":- return none, nil, nil, navStructReasonBadArgs- case docID == "":- return none, nil, nil, navStructReasonNoDoc+// kind and drills it toward the query. It returns the rendered outline, the+// entity/relation counts, and whether the document carried a structure at all.+//+// Claim recall runs FIRST and, when it hits, replaces the drill entirely: the+// claims carry their own evidence, so descending the tree could only re-derive+// what they already state.+func readStructureDocCore(ctx context.Context, tenantID, query, docID string, kinds map[string]bool, datasetIDs []string, reader structureReader) (string, int, int, bool) {+ if query == "" || docID == "" {+ return "", 0, 0, false+ }+ kindList := claimKindList(kinds)+ if claims := recallClaimsForDoc(ctx, tenantID, query, docID, kindList, datasetIDs, reader); len(claims) > 0 {+ return claimOutline(ctx, claims), 0, 0, true}+var nodes []structureNodevar rels []structureRel- anyStruct := false+ read := falsefor _, kbID := range datasetIDs {rows, err := reader.ReadStructure(ctx, tenantID, kbID, docID)if err != nil {@@ -862,23 +1251,52 @@ func readStructureDocCore(ctx context.Context, tenantID, query, docID, kind striif len(rows) == 0 {continue}- anyStruct = true- var kindList []string- for k := range kinds {- kindList = append(kindList, k)- }- sort.Strings(kindList)- entities, relations := ParseCompiledStructure(rows, kindList)+ read = true+ entities, relations := parseCompiledStructure(rows, kindList)es, rs := structureGraphFromRaw(entities, relations)nodes = append(nodes, es...)rels = append(rels, rs...)}- if !anyStruct || len(nodes) == 0 {- return none, nil, nil, navStructReasonNoStructure+ if !read || len(nodes) == 0 {+ return "", 0, 0, false}loader := func(ids []string) []chunkWithText { return structureNodeLoader(ctx, tenantID, ids) }- drill := renderTocDrilldown(query, nil, nodes, rels, loader)- return drill, nodes, rels, ""+ return renderTocDrilldown(query, nodes, rels, loader), len(nodes), len(rels), true+}++// recallClaimsForDoc runs the claim leg across the document's datasets, keeping+// the fused rank order. Hits are deduped by claim NAME (two datasets cannot both+// hold the same document, but a reader may return a claim twice) — deliberately+// NOT by chunk id: several distinct claims commonly come from one chunk, and+// dropping all but the first would throw evidence away.+func recallClaimsForDoc(ctx context.Context, tenantID, query, docID string, kindList, datasetIDs []string, reader structureReader) []claimHit {+ cr, ok := reader.(claimReader)+ if !ok {+ return nil+ }+ qvec := claimQueryVector(ctx, tenantID, query)+ var out []claimHit+ seen := map[string]bool{}+ for _, kbID := range datasetIDs {+ hits, err := cr.ReadClaims(ctx, tenantID, kbID, []string{docID}, kindList, query, qvec, navClaimTopN)+ if err != nil {+ common.DebugCtx(ctx, "navigate_structure: claim recall failed",+ zap.String("kb_id", kbID), zap.String("doc_id", docID), zap.Error(err))+ continue+ }+ for _, h := range hits {+ key := strings.ToLower(strings.TrimSpace(h.name))+ if h.chunkID == "" || key == "" || seen[key] {+ continue+ }+ seen[key] = true+ out = append(out, h)+ if len(out) >= navClaimTopN {+ return out+ }+ }+ }+ return out}// structureNodeLoader fetches chunk texts by id for the drill-down snippet pass.@@ -934,14 +1352,14 @@ func structureKindsFor(kind string) map[string]bool {}func decodeJSONObject(s string) (map[string]any, bool) {- v := ExtractJSON(s)+ v := firstJSONValue(s)m, ok := v.(map[string]any)return m, ok}-// ExtractJSON finds the first JSON object in s (tolerating surrounding prose or+// firstJSONValue finds the first JSON object in s (tolerating surrounding prose or// code fences) and returns it decoded.-func ExtractJSON(s string) any {+func firstJSONValue(s string) any {start := strings.Index(s, "{")if start < 0 {return nil@@ -997,8 +1415,8 @@ func objectList(v any) []map[string]any {return out}-// Snippet truncates s to at most n runes, appending an ellipsis when cut.-func Snippet(s string, n int) string {+// snippetRunes truncates s to at most n runes, appending an ellipsis when cut.+func snippetRunes(s string, n int) string {r := []rune(s)if len(r) <= n {return s@@ -1008,7 +1426,7 @@ func Snippet(s string, n int) string {// navStructEmpty returns the empty <structure_navigation count="0"> XML carrying// an optional error="..." attribute, mirroring the harness navigation contract.-func navStructEmpty(reason, label string) string {+func navStructEmpty(label string) string {attr := ""if label != "" {attr = ` error="` + xmlEscape(label) + `"`@@ -1040,11 +1458,11 @@ func (g *NavigateStructureTool) Info(_ context.Context) (*schema.ToolInfo, error},"doc_id": {"type": "string",- "description": "The document to drill. Take it from a navigate_tree <doc doc_id> result. Required unless doc_ids is given."+ "description": "OPTIONAL: the document to drill, taken from a navigate_tree <doc doc_id> result. Omit it to route the query yourself and get compiled claim evidence from the best-matching documents directly (no navigate_tree call needed)."},"doc_ids": {"type": "array",- "description": "Optional explicit document ids to drill (at most 10). When omitted, doc_id is used.",+ "description": "Optional explicit document ids to drill (at most 10). When omitted, doc_id is used; when both are omitted the query is routed.","items": { "type": "string" },"maxItems": 10},@@ -1082,6 +1500,209 @@ func (g *NavigateStructureTool) InvokableRun(ctx context.Context, argumentsInJSOreturn guardedToolRun(ctx, navigateStructureToolName, g.invokableRun, argumentsInJSON)}+// claimPrefetch is the ONE claim recall a turn performs before its loop, with+// the verdict the run flow acts on.+//+// The recall exists because the compiled tools are injected rather than+// declared, and a model that prefers one-step keyword search never calls them:+// in the FRAMES runs every turn was offered navigate_structure and the tool was+// called zero times. Evidence a model cannot be relied upon to request has to be+// retrieved by the pipeline, and the claim leg is exactly that - a flat read that+// needs no document choice.+//+// The result is a CONTROL SIGNAL, not prompt text. Claims that answer the+// question terminate the turn without reading a single raw chunk; a recall that+// does not cover the question is dropped and the ordinary tool-driven+// exploration runs. Nothing is appended to the system instruction: a block there+// is re-billed on every model call of the turn, which is the cost this replaces.+type claimPrefetch struct {+ docs []string+ claims []claimHit+ // evidence is the rendered claim block, carrying the [ID:n] handles its+ // source chunks were stamped with. It is rendered ONLY for a sufficient+ // recall, because rendering is what publishes handles: a discarded recall+ // must not leave handles behind for passages the model never saw.+ evidence string+ // sufficient is true when claimsAnswerQuestion accepted the recall.+ sufficient bool+}++// recallCompiledClaims recalls the bound datasets' claims for this turn's+// question and decides whether they answer it themselves.+func recallCompiledClaims(ctx context.Context, tenantID string, datasetIDs []string, messages []*schema.Message) claimPrefetch {+ question := strings.TrimSpace(lastUserQuestion(messages))+ if question == "" || len(datasetIDs) == 0 {+ return claimPrefetch{}+ }+ // A compiled dataset does not imply claims: trees, timelines and mindmaps all+ // carry compile_kwd but no evidence rows. Probing first keeps both retrieval+ // legs from running over a corpus that can only come back empty.+ if !claimRowsPresent(ctx, tenantID, datasetIDs) {+ common.DebugCtx(ctx, "agentic_rag: claim recall skipped",+ zap.Strings("datasets", datasetIDs),+ zap.String("reason", "no claim rows in the bound datasets"))+ return claimPrefetch{}+ }+ reader := getStructureReader()+ // NO document routing: the claim leg is a flat, corpus-wide read, which is+ // the whole reason it can run before the loop. Routing first (the nav-tree+ // KNN) reintroduced the document choice this path exists to avoid, and it+ // fails silently where it matters: a dataset whose nav tree holds a handful+ // of rows routes every question to the same one or two documents, and the+ // recall then dutifully returns their claims for every question.+ //+ // The turn's document scope, when set, is still honoured — as a filter on the+ // claim rows themselves.+ docScope, _ := resolveDocumentScope(ctx, nil)+ // A non-nil EMPTY scope means the turn allows no documents. The reader drops+ // its doc_id filter when the scope is empty, so passing one through would read+ // as "unrestricted" and recall the whole dataset — the opposite of the turn's+ // scope. Stop before the read instead.+ if docScope != nil && len(docScope) == 0 {+ common.DebugCtx(ctx, "agentic_rag: claim recall skipped",+ zap.Strings("datasets", datasetIDs),+ zap.String("reason", "no documents in the turn's scope"))+ return claimPrefetch{}+ }+ claims := recallClaimsAcrossDatasets(ctx, tenantID, datasetIDs, docScope, question, navClaimPrefetchTopN, reader)++ pf := claimPrefetch{claims: claims, docs: claimDocsInOrder(claims)}+ pf.sufficient = claimsAnswerQuestion(claims, len(pf.docs))+ if !pf.sufficient {+ common.InfoCtx(ctx, "agentic_rag: claim recall insufficient",+ zap.String("query", question),+ zap.Int("claims", len(claims)),+ zap.Int("docs", len(pf.docs)),+ zap.Int("required_claims", navClaimSufficientHits),+ zap.Int("required_docs", navClaimMinDocs))+ return pf+ }+ pf.evidence = claimEvidenceBlock(ctx, claims)+ common.InfoCtx(ctx, "agentic_rag: claim recall sufficient",+ zap.String("query", question),+ zap.Int("claims", len(claims)),+ zap.Strings("docs", pf.docs),+ zap.Int("evidence_bytes", len(pf.evidence)))+ return pf+}++// recallClaimsAcrossDatasets runs the flat claim recall over every bound+// dataset and merges the results by claim name, keeping the first (best-ranked)+// occurrence. Each dataset is one pair of legs; the corpus-wide read is what+// makes the pre-loop recall independent of any document ranking.+func recallClaimsAcrossDatasets(ctx context.Context, tenantID string, datasetIDs, docScope []string, query string, limit int, reader structureReader) []claimHit {+ cr, ok := reader.(claimReader)+ if !ok {+ return nil+ }+ kindList := claimKindList(structureKindsFor(""))+ qvec := claimQueryVector(ctx, tenantID, query)++ // Group the hits by source document, in ranked order, then take them+ // round-robin. A ranking's top-N is routinely dominated by ONE document, and+ // evidence from a single document cannot answer a multi-document question;+ // taking one claim per document per round is what makes the sufficiency+ // diversity check meaningful instead of accidental.+ var docOrder []string+ byDoc := map[string][]claimHit{}+ seen := map[string]struct{}{}+ for _, kbID := range datasetIDs {+ hits, err := cr.ReadClaims(ctx, tenantID, kbID, docScope, kindList, query, qvec, limit)+ if err != nil {+ common.DebugCtx(ctx, "claim recall: dataset read failed",+ zap.String("kb_id", kbID), zap.Error(err))+ continue+ }+ for _, h := range hits {+ key := strings.ToLower(strings.TrimSpace(h.name))+ if key == "" {+ continue+ }+ if _, dup := seen[key]; dup {+ continue+ }+ seen[key] = struct{}{}+ if _, ok := byDoc[h.docID]; !ok {+ docOrder = append(docOrder, h.docID)+ }+ byDoc[h.docID] = append(byDoc[h.docID], h)+ }+ }+ if len(docOrder) == 0 {+ return nil+ }+ groups := make([][]claimHit, 0, len(docOrder))+ for _, d := range docOrder {+ groups = append(groups, byDoc[d])+ }+ if limit > 0 {+ groups = trimClaimsRoundRobin(groups, limit)+ }+ out := make([]claimHit, 0, limit)+ for _, g := range groups {+ out = append(out, g...)+ }+ return out+}++// claimDocsInOrder returns the distinct source documents of a claim set, in+// first-appearance order, for logging and evidence grouping.+func claimDocsInOrder(claims []claimHit) []string {+ seen := map[string]struct{}{}+ out := make([]string, 0, len(claims))+ for _, c := range claims {+ if c.docID == "" {+ continue+ }+ if _, dup := seen[c.docID]; dup {+ continue+ }+ seen[c.docID] = struct{}{}+ out = append(out, c.docID)+ }+ return out+}++// claimsAnswerQuestion reports whether a recall covers the question well enough+// to answer without reading any raw chunk.+//+// Two conditions, both necessary:+// - COUNT: at least navClaimSufficientHits claims. Only meaningful because+// recall already applied the relevance floor (navClaimMinCoverage), so a hit+// states something about the question rather than sharing a stopword with it.+// - DIVERSITY: claims from at least navClaimMinDocs documents. A page of+// statements from ONE document answers at most one hop, and a single-document+// recall is also exactly what a broken document ranking produces for every+// question — the shape of the zero-score run this guard was added for.+func claimsAnswerQuestion(claims []claimHit, docs int) bool {+ return len(claims) >= navClaimSufficientHits && docs >= navClaimMinDocs+}++// claimEvidenceBlock renders the recalled claims as answer material, one <doc>+// element per source document so a multi-document question keeps its provenance+// and the answer can name which document each fact came from.+func claimEvidenceBlock(ctx context.Context, claims []claimHit) string {+ var b strings.Builder+ current := ""+ for _, c := range claims {+ if c.docID != current {+ if current != "" {+ b.WriteString("</doc>\n")+ }+ if c.docID != "" {+ fmt.Fprintf(&b, "<doc doc_id=\"%s\">\n", xmlEscape(c.docID))+ }+ current = c.docID+ }+ b.WriteString(claimOutlineLine(ctx, c))+ b.WriteString("\n")+ }+ if current != "" {+ b.WriteString("</doc>\n")+ }+ return strings.TrimRight(b.String(), "\n")+}+func (g *NavigateStructureTool) invokableRun(ctx context.Context, argumentsInJSON string) (string, error) {var args navigateStructureArgsif err := json.Unmarshal([]byte(argumentsInJSON), &args); err != nil {@@ -1101,35 +1722,60 @@ func (g *NavigateStructureTool) invokableRun(ctx context.Context, argumentsInJSO// compiled structure): skip the backend and return the same verdict the// caller falls back from.common.DebugCtx(ctx, "navigate_structure: marked unavailable; skipping backend (no structure)")- return navStructEmpty(navStructReasonNoStructure, "no structure"), nil+ return navStructEmpty("no structure"), nil}datasetIDs, err := resolveDatasetScope(g.datasetIDs, args.DatasetIDs)if err != nil {return "", fmt.Errorf("navigate_structure: %w", err)}if len(datasetIDs) == 0 {- return navStructEmpty(navStructReasonNoStructure, "no bound datasets"), nil+ return navStructEmpty("no bound datasets"), nil}+ reader := getStructureReader()++ text := strings.TrimSpace(query + " " + strings.TrimSpace(args.Keywords))+docIDs := args.DocIDsif len(docIDs) == 0 && args.DocID != "" {docIDs = []string{args.DocID}}+ // A doc_id is optional: with none, route the query ourselves so the tool is+ // usable in one call. Only CLAIMS are read on this path - the drill needs a+ // document the model actually chose.+ routedByQuery := falseif len(docIDs) == 0 {- return navStructEmpty(navStructReasonNoDoc, "route first with navigate_tree"), nil+ // The router ranks the whole bound corpus, so a turn restricted to a set+ // of documents must intersect here - the same rule the pre-loop recall+ // applies to its claim rows.+ docIDs = intersectDocumentScope(ctx, routeDocIDsForQuery(ctx, g.tenantID, text, datasetIDs, navQueryRouteDocs))+ if len(docIDs) == 0 {+ return navStructEmpty("route first with navigate_tree"), nil+ }+ routedByQuery = true}if len(docIDs) > 10 {docIDs = docIDs[:10]}- reader := getStructureReader()- if reader == nil {- return navStructEmpty(navStructReasonNoStructure, "no structure reader installed"), nil+ if routedByQuery {+ // 0 = uncapped: the model asked for this call, so its result lands in the+ // history once instead of being re-sent on every turn of the run. The+ // pipeline's own pre-loop recall applies its cap instead.+ out, docs := navigateClaimsForQuery(ctx, g.tenantID, text, docIDs, datasetIDs, args.Kind, 0, reader)+ common.InfoCtx(ctx, "agentic_rag: navigate_structure (query route)",+ zap.String("query", query),+ zap.Int("datasets", len(datasetIDs)),+ zap.Strings("routed", docIDs),+ zap.Int("docs_with_claims", docs),+ zap.Int("out_bytes", len(out)))+ if docs == 0 {+ return navStructEmpty("no claim hits in the routed documents; fall back to search_chunks or list_chunks"), nil+ }+ return out, nil}- text := strings.TrimSpace(query + " " + strings.TrimSpace(args.Keywords))- out, reason := navigateStructures(ctx, g.tenantID, text, docIDs, datasetIDs, args.Kind, reader)- _ = reason+ out := navigateStructures(ctx, g.tenantID, text, docIDs, datasetIDs, args.Kind, reader)// Per-call INFO: the tool-usage gate (kg_tool_calls) is read off the server// log, so a graph tool that rendered a structure without leaving a line here// cannot be told from one that was never called.diff --git a/internal/agentic_rag/tool_navigate_structure_test.go b/internal/agentic_rag/tool_navigate_structure_test.goindex 8b09fe78f9..9fd0e10fa9 100644--- a/internal/agentic_rag/tool_navigate_structure_test.go+++ b/internal/agentic_rag/tool_navigate_structure_test.go@@ -18,24 +18,107 @@ package agentic_ragimport ("context""encoding/json"+ "sort""strings""testing"++ "github.com/cloudwego/eino/schema")// fakeStructureReader returns canned compiled-structure rows per document.type fakeStructureReader struct {- byDoc map[string][]StructureRow+ byDoc map[string][]structureRow}-func (f fakeStructureReader) ReadStructure(_ context.Context, _ string, _ string, docID string) ([]StructureRow, error) {+func (f fakeStructureReader) ReadStructure(_ context.Context, _ string, _ string, docID string) ([]structureRow, error) {return f.byDoc[docID], nil}func installFakeStructureReader(t *testing.T, r structureReader) {t.Helper()prev := getStructureReader()- SetStructureReader(r)- t.Cleanup(func() { SetStructureReader(prev) })+ setStructureReader(r)+ t.Cleanup(func() { setStructureReader(prev) })+}++// fakeClaimStructureReader is a structureReader that also carries a claim leg:+// structure rows and claim hits are canned separately, so a test can prove+// which strategy actually rendered the outline.+type fakeClaimStructureReader struct {+ fakeStructureReader+ claims map[string][]claimHit+ // gotQVec captures the vector the tool handed to the claim leg.+ gotQVec *[]float64+}++func (f fakeClaimStructureReader) ReadClaims(_ context.Context, _ string, _ string, docIDs, _ []string, _ string, qvec []float64, limit int) ([]claimHit, error) {+ if f.gotQVec != nil {+ *f.gotQVec = qvec+ }+ // nil/empty docIDs is the corpus-wide read: every canned document, in stable+ // order so a test asserts on a deterministic page.+ requested := docIDs+ if len(requested) == 0 {+ for docID := range f.claims {+ requested = append(requested, docID)+ }+ sort.Strings(requested)+ }+ var hits []claimHit+ for _, docID := range requested {+ for _, h := range f.claims[docID] {+ if h.docID == "" {+ h.docID = docID+ }+ hits = append(hits, h)+ }+ }+ // Spread a cap across the requested documents rather than truncating in+ // document order: the ranked reader interleaves documents, so truncating by+ // document would model a reader this codebase does not have.+ if limit > 0 && len(hits) > limit {+ var byDoc [][]claimHit+ for _, docID := range requested {+ var group []claimHit+ for _, h := range hits {+ if h.docID == docID {+ group = append(group, h)+ }+ }+ if len(group) > 0 {+ byDoc = append(byDoc, group)+ }+ }+ var spread []claimHit+ for round := 0; len(spread) < limit; round++ {+ progressed := false+ for _, group := range byDoc {+ if round < len(group) {+ spread = append(spread, group[round])+ progressed = true+ if len(spread) == limit {+ break+ }+ }+ }+ if !progressed {+ break+ }+ }+ hits = spread+ }+ return hits, nil+}++func claimRowJSON(name, desc, quote, chunkID string) structureRow {+ content, _ := json.Marshal(map[string]any{+ "type": "claim",+ "name": name,+ "description": desc,+ "source_chunk_ids": []string{chunkID},+ "evidence": []map[string]any{{"quote": quote, "chunk_id": chunkID}},+ })+ return structureRow{Content: string(content)}}// TestNavigateStructureToolIsRegisteredAndBuildableByTemplate pins the wiring:@@ -62,7 +145,7 @@ func TestNavigateStructureToolIsRegisteredAndBuildableByTemplate(t *testing.T) {}// graphRow is a canned compiled-structure "graph" blob for the drill tests.-func graphRow() StructureRow {+func graphRow() structureRow {content, _ := json.Marshal(map[string]any{"entities": []map[string]any{{"name": "Alpha", "type": "section", "description": "first topic", "source_chunk_ids": []string{"c1", "c2"}},@@ -72,13 +155,13 @@ func graphRow() StructureRow {{"from": "Alpha", "to": "Beta", "type": "contains"},},})- return StructureRow{KnowledgeGraphKwd: "graph", Content: string(content)}+ return structureRow{KnowledgeGraphKwd: "graph", Content: string(content)}}// TestNavigateStructureToolDrillsStructure pins the payload: a routed document's// compiled structure renders as a query-focused outline with chunk pointers.func TestNavigateStructureToolDrillsStructure(t *testing.T) {- installFakeStructureReader(t, fakeStructureReader{byDoc: map[string][]StructureRow{+ installFakeStructureReader(t, fakeStructureReader{byDoc: map[string][]structureRow{"d1": {graphRow()},}})@@ -105,7 +188,7 @@ func TestNavigateStructureToolDrillsStructure(t *testing.T) {// TestNavigateStructureToolFlatFallback renders the whole outline when the query// carries no usable terms (the documented no-keyword fallback).func TestNavigateStructureToolFlatFallback(t *testing.T) {- installFakeStructureReader(t, fakeStructureReader{byDoc: map[string][]StructureRow{+ installFakeStructureReader(t, fakeStructureReader{byDoc: map[string][]structureRow{"d1": {graphRow()},}})@@ -122,7 +205,7 @@ func TestNavigateStructureToolFlatFallback(t *testing.T) {// TestNavigateStructureToolNoDocIsAHintNotAnError: a call without a doc_id is a// usage-level dead end the model reads and routes first with navigate_tree.func TestNavigateStructureToolNoDocIsAHintNotAnError(t *testing.T) {- installFakeStructureReader(t, fakeStructureReader{byDoc: map[string][]StructureRow{}})+ installFakeStructureReader(t, fakeStructureReader{byDoc: map[string][]structureRow{}})tool := NewNavigateStructureTool("tenant-1", []string{"kb1"})out, err := tool.InvokableRun(context.Background(), `{"query":"q"}`)@@ -140,7 +223,7 @@ func TestNavigateStructureToolNoDocIsAHintNotAnError(t *testing.T) {// TestNavigateStructureToolNoStructureIsAHintNotAnError: a document without a// compiled structure of the requested kind is a dead end, not an abort.func TestNavigateStructureToolNoStructureIsAHintNotAnError(t *testing.T) {- installFakeStructureReader(t, fakeStructureReader{byDoc: map[string][]StructureRow{+ installFakeStructureReader(t, fakeStructureReader{byDoc: map[string][]structureRow{"d1": nil, // no structure rows}})@@ -159,7 +242,7 @@ func TestNavigateStructureToolNoStructureIsAHintNotAnError(t *testing.T) {// TestNavigateStructureToolRequiresQuery pins the argument contract.func TestNavigateStructureToolRequiresQuery(t *testing.T) {- installFakeStructureReader(t, fakeStructureReader{byDoc: map[string][]StructureRow{}})+ installFakeStructureReader(t, fakeStructureReader{byDoc: map[string][]structureRow{}})tool := NewNavigateStructureTool("tenant-1", []string{"kb1"})out, err := tool.InvokableRun(context.Background(), `{"query":" "}`)if err != nil {@@ -173,7 +256,7 @@ func TestNavigateStructureToolRequiresQuery(t *testing.T) {// TestNavigateStructureToolRespectsDatasetScope: an out-of-scope dataset_id is// rejected before any read.func TestNavigateStructureToolRespectsDatasetScope(t *testing.T) {- installFakeStructureReader(t, fakeStructureReader{byDoc: map[string][]StructureRow{}})+ installFakeStructureReader(t, fakeStructureReader{byDoc: map[string][]structureRow{}})tool := NewNavigateStructureTool("tenant-1", []string{"kb1"})out, err := tool.InvokableRun(context.Background(), `{"query":"q","doc_id":"d1","dataset_ids":["rogue"]}`)if err != nil {@@ -198,20 +281,20 @@ func TestQueryTermsTokenizes(t *testing.T) {}func TestExtractJSONObjectToleratesProse(t *testing.T) {- v := ExtractJSON("here is the blob: {\"a\": 1, \"b\": [2,3]} trailing text")+ v := firstJSONValue("here is the blob: {\"a\": 1, \"b\": [2,3]} trailing text")m, ok := v.(map[string]any)if !ok || m["a"] != float64(1) {- t.Errorf("ExtractJSON = %#v ok=%v", v, ok)+ t.Errorf("firstJSONValue = %#v ok=%v", v, ok)}- if ExtractJSON("no json here") != nil {- t.Errorf("ExtractJSON must return nil when no object is present")+ if firstJSONValue("no json here") != nil {+ t.Errorf("firstJSONValue must return nil when no object is present")}}// TestExtractJSONHandlesEscapedQuotes: a description carrying an escaped quote// (and braces) must not corrupt the brace depth and silently drop the row.func TestExtractJSONHandlesEscapedQuotes(t *testing.T) {- v := ExtractJSON(`{"a": "say \"hi\" {ok}", "b": 1}`)+ v := firstJSONValue(`{"a": "say \"hi\" {ok}", "b": 1}`)m, ok := v.(map[string]any)if !ok {t.Fatalf("escaped quotes must still parse, got %#v", v)@@ -223,7 +306,7 @@ func TestExtractJSONHandlesEscapedQuotes(t *testing.T) {func TestParseCompiledStructureMergesGraphBlob(t *testing.T) {row := graphRow()- entities, relations := ParseCompiledStructure([]StructureRow{row}, nil)+ entities, relations := parseCompiledStructure([]structureRow{row}, nil)if len(entities) != 2 || len(relations) != 1 {t.Fatalf("parsed entities=%d relations=%d, want 2/1", len(entities), len(relations))}@@ -237,17 +320,17 @@ func TestParseCompiledStructureMergesGraphBlob(t *testing.T) {// rowKind normalizes that row shape to "timeline" (and a plain string compare// would drop every row before parsing).func TestParseCompiledStructureNormalizesRequestedKind(t *testing.T) {- entities, relations := ParseCompiledStructure([]StructureRow{graphRow()}, []string{"graph"})+ entities, relations := parseCompiledStructure([]structureRow{graphRow()}, []string{"graph"})if len(entities) != 2 || len(relations) != 1 {t.Fatalf("kind=graph must not filter out the graph row: entities=%d relations=%d", len(entities), len(relations))}- if entities, relations := ParseCompiledStructure([]StructureRow{graphRow()}, []string{"mind_map"}); len(entities) != 0 || len(relations) != 0 {+ if entities, relations := parseCompiledStructure([]structureRow{graphRow()}, []string{"mind_map"}); len(entities) != 0 || len(relations) != 0 {t.Fatalf("an unrelated kind must still filter the row out: %d/%d", len(entities), len(relations))}}func TestStructureGraphFromRawBuildsNodesAndRels(t *testing.T) {- entities, relations := ParseCompiledStructure([]StructureRow{graphRow()}, nil)+ entities, relations := parseCompiledStructure([]structureRow{graphRow()}, nil)nodes, rels := structureGraphFromRaw(entities, relations)if len(nodes) != 2 || len(rels) != 1 {t.Fatalf("nodes=%d rels=%d, want 2/1", len(nodes), len(rels))@@ -261,7 +344,7 @@ func TestStructureGraphFromRawBuildsNodesAndRels(t *testing.T) {}func TestBuildTocTreeAndOutline(t *testing.T) {- entities, relations := ParseCompiledStructure([]StructureRow{graphRow()}, nil)+ entities, relations := parseCompiledStructure([]structureRow{graphRow()}, nil)nodes, rels := structureGraphFromRaw(entities, relations)byName, children, parents, roots := buildTocTree(nodes, rels)if len(roots) != 1 || roots[0] != "Alpha" {@@ -282,3 +365,584 @@ func TestBuildTocTreeAndOutline(t *testing.T) {}_ = byName}++// --- Claim/evidence leg ---++// TestNavigateStructureClaimLegReplacesDrill pins claim-first: a claim hit takes+// over the selection, so the tree is NOT descended even when structure rows+// exist for the same document.+func TestNavigateStructureClaimLegReplacesDrill(t *testing.T) {+ installFakeStructureReader(t, fakeClaimStructureReader{+ fakeStructureReader: fakeStructureReader{byDoc: map[string][]structureRow{+ "d1": {graphRow()},+ }},+ claims: map[string][]claimHit{+ "d1": {{+ chunkID: "c9",+ name: "Paris is the capital of France",+ description: "capital city",+ quote: "Paris is the capital of France.",+ score: 0.8,+ }},+ },+ })++ out, err := NewNavigateStructureTool("tenant-1", []string{"kb1"}).+ InvokableRun(context.Background(), `{"query":"capital of France","doc_id":"d1"}`)+ if err != nil {+ t.Fatalf("InvokableRun: %v", err)+ }+ for _, want := range []string{+ `<structure_navigation count="1" query="capital of France" kind="">`,+ `<doc rank="1" doc_id="d1"`,+ // The outline is escaped as a whole, so the quote's own quotes ship as+ // " (the same escaping the harness applies).+ `- [claim] Paris is the capital of France | Evidence: "Paris is the capital of France." [chunks: c9]`,+ `</structure>`,+ } {+ if !strings.Contains(out, want) {+ t.Errorf("output must contain %q, got:\n%s", want, out)+ }+ }+ if strings.Contains(out, "Alpha (section)") {+ t.Errorf("a claim hit must replace the drill, so no structure node may render, got:\n%s", out)+ }+}++// TestNavigateStructureRendersEveryClaim: the tool renders the claims it recalled,+// with no self-declared sufficiency verdict - whether a claim set is enough to+// answer from is decided by the pipeline's pre-loop recall, not by a marker the+// model has to interpret (and which could never agree with that gate's+// document-diversity rule).+func TestNavigateStructureRendersEveryClaim(t *testing.T) {+ hits := []claimHit{+ {chunkID: "c1", name: "claim one", quote: "q1"},+ {chunkID: "c2", name: "claim two", quote: "q2"},+ {chunkID: "c3", name: "claim three", quote: "q3"},+ }+ installFakeStructureReader(t, fakeClaimStructureReader{+ fakeStructureReader: fakeStructureReader{byDoc: map[string][]structureRow{}},+ claims: map[string][]claimHit{"d1": hits},+ })++ out, err := NewNavigateStructureTool("tenant-1", []string{"kb1"}).+ InvokableRun(context.Background(), `{"query":"claims","doc_id":"d1"}`)+ if err != nil {+ t.Fatalf("InvokableRun: %v", err)+ }+ if got := strings.Count(out, "- [claim] "); got != len(hits) {+ t.Errorf("rendered %d claim lines, want %d:\n%s", got, len(hits), out)+ }+}++// TestNavigateStructureClaimOnlyDocumentIsNotNoStructure: a document whose+// compiler wrote claims but no tree nodes still renders — counting only entities+// would report "no compiled structure" and throw the evidence away.+func TestNavigateStructureClaimOnlyDocumentIsNotNoStructure(t *testing.T) {+ installFakeStructureReader(t, fakeClaimStructureReader{+ fakeStructureReader: fakeStructureReader{byDoc: map[string][]structureRow{}},+ claims: map[string][]claimHit{"d1": {+ {chunkID: "c1", name: "only claim", quote: "only quote"},+ }},+ })++ out, err := NewNavigateStructureTool("tenant-1", []string{"kb1"}).+ InvokableRun(context.Background(), `{"query":"only","doc_id":"d1"}`)+ if err != nil {+ t.Fatalf("InvokableRun: %v", err)+ }+ if !strings.Contains(out, `count="1"`) {+ t.Fatalf("a claim-only document must still render, got:\n%s", out)+ }+ if strings.Contains(out, "no compiled structure") {+ t.Errorf("claim-only document must not report no_structure, got:\n%s", out)+ }+}++// TestNavigateStructureClaimLegReceivesQueryVector: the tool must hand the nav+// service's query vector to the claim leg (the dense half of hybrid recall), and+// must pass nothing when no service is installed.+func TestNavigateStructureClaimLegReceivesQueryVector(t *testing.T) {+ var got []float64+ installFakeStructureReader(t, fakeClaimStructureReader{+ fakeStructureReader: fakeStructureReader{byDoc: map[string][]structureRow{}},+ claims: map[string][]claimHit{"d1": {{chunkID: "c1", name: "n", quote: "q"}}},+ gotQVec: &got,+ })+ installFakeNavService(t, fakeNavService{+ encode: func(context.Context, string, string) []float64 { return []float64{0.25, 0.5} },+ })++ if _, err := NewNavigateStructureTool("tenant-1", []string{"kb1"}).+ InvokableRun(context.Background(), `{"query":"q","doc_id":"d1"}`); err != nil {+ t.Fatalf("InvokableRun: %v", err)+ }+ if len(got) != 2 || got[0] != 0.25 {+ t.Errorf("claim leg received qvec %v, want the service's vector", got)+ }++ installFakeNavService(t, nil)+ if _, err := NewNavigateStructureTool("tenant-1", []string{"kb1"}).+ InvokableRun(context.Background(), `{"query":"q","doc_id":"d1"}`); err != nil {+ t.Fatalf("InvokableRun: %v", err)+ }+ if len(got) != 0 {+ t.Errorf("without a nav service the claim leg must run lexical-only, got qvec %v", got)+ }+}++// TestClaimQueryVectorSwallowsPanic: a misconfigured embedder degrades recall to+// the lexical leg; it must never take the tool call down.+func TestClaimQueryVectorSwallowsPanic(t *testing.T) {+ installFakeNavService(t, fakeNavService{+ encode: func(context.Context, string, string) []float64 { panic("boom") },+ })+ if vec := claimQueryVector(context.Background(), "tenant-1", "q"); vec != nil {+ t.Errorf("a panicking encoder must yield no vector, got %v", vec)+ }+}++func TestParseClaimHitReadsEvidenceAndPointer(t *testing.T) {+ row := map[string]interface{}{+ "content_with_weight": claimRowJSON("Paris is the capital", "capital city", "Paris is the capital of France.", "c9").Content,+ "_score": float64(0.7),+ }+ hit, ok := parseClaimHit(row)+ if !ok {+ t.Fatalf("a well-formed claim row must parse")+ }+ if hit.chunkID != "c9" || hit.name != "Paris is the capital" || hit.quote != "Paris is the capital of France." {+ t.Errorf("hit = %+v", hit)+ }+ if hit.score != 0.7 {+ t.Errorf("score = %v, want 0.7", hit.score)+ }++ // No chunk pointer: the model could not open it, so it is not a hit.+ orphan, _ := json.Marshal(map[string]any{"name": "orphan claim", "description": "d"})+ if _, ok := parseClaimHit(map[string]interface{}{"content_with_weight": string(orphan)}); ok {+ t.Errorf("a claim without source_chunk_ids must be dropped")+ }+ // Column form (engines return the id list as a column).+ col := map[string]interface{}{+ "content_with_weight": `{"name":"col claim"}`,+ "source_chunk_ids": []interface{}{"c7"},+ }+ if hit, ok := parseClaimHit(col); !ok || hit.chunkID != "c7" {+ t.Errorf("column source_chunk_ids must be honored, got %+v ok=%v", hit, ok)+ }+}++// TestRRFFuseMergesLegsByRank: RRF consumes ranks only, so a hit found by both+// legs outranks one found by a single leg, and the fused list is capped.+func TestRRFFuseMergesLegsByRank(t *testing.T) {+ dense := []claimHit{{name: "both", chunkID: "c1", score: 0.9}, {name: "only dense", chunkID: "c2", score: 0.8}}+ text := []claimHit{{name: "only text", chunkID: "c3", score: 11}, {name: "both", chunkID: "c1", score: 3}}++ fused := rrfFuse(navClaimRRFK, 0, dense, text)+ if len(fused) != 3 {+ t.Fatalf("fused %d hits, want 3: %+v", len(fused), fused)+ }+ if fused[0].name != "both" {+ t.Errorf("the hit found by BOTH legs must rank first, got %+v", fused)+ }+ // A fused hit keeps the best raw score across the legs that found it; the+ // legs' scales are incomparable, which is exactly why fusion uses ranks.+ if fused[0].score != 3 {+ t.Errorf("a fused hit keeps its best raw score, got %v", fused[0].score)+ }+ if len(fused) != 3 || fused[1].name != "only text" || fused[2].name != "only dense" {+ t.Errorf("single-leg hits must follow the double-leg hit, got %+v", fused)+ }+ if capped := rrfFuse(navClaimRRFK, 2, dense, text); len(capped) != 2 {+ t.Errorf("limit must cap the fused list, got %d", len(capped))+ }+}++// TestClaimRecallKeepsDistinctClaimsFromOneChunk: several facts commonly come+// from the same passage, so dedupe must be by claim, not by chunk id.+func TestClaimRecallKeepsDistinctClaimsFromOneChunk(t *testing.T) {+ installFakeStructureReader(t, fakeClaimStructureReader{+ fakeStructureReader: fakeStructureReader{byDoc: map[string][]structureRow{}},+ claims: map[string][]claimHit{"d1": {+ {chunkID: "c1", name: "Acme was founded in 1994", quote: "Acme, founded in 1994"},+ {chunkID: "c1", name: "Acme is based in Zurich", quote: "the Zurich-based firm"},+ }},+ })+ out, err := NewNavigateStructureTool("tenant-1", []string{"kb1"}).+ InvokableRun(context.Background(), `{"query":"Acme","doc_id":"d1"}`)+ if err != nil {+ t.Fatalf("InvokableRun: %v", err)+ }+ if got := strings.Count(out, "- [claim] "); got != 2 {+ t.Errorf("both claims from the shared chunk must render, got %d:\n%s", got, out)+ }+}++// TestClaimOutlineLineShape pins the claim line the model reads.+func TestClaimOutlineLineShape(t *testing.T) {+ got := claimOutlineLine(context.Background(), claimHit{name: "Acme was founded in 1994", quote: "Acme, founded in 1994, ...", chunkID: "c1"})+ want := `- [claim] Acme was founded in 1994 | Evidence: "Acme, founded in 1994, ..." [chunks: c1]`+ if got != want {+ t.Errorf("claim line =\n%s\nwant\n%s", got, want)+ }+ // No evidence: the statement and pointer still render.+ got = claimOutlineLine(context.Background(), claimHit{name: "bare claim", chunkID: "c2"})+ if got != "- [claim] bare claim [chunks: c2]" {+ t.Errorf("bare claim line = %q", got)+ }+}++// TestClaimOutlineLineCarriesHandle: a claim is answer material, so its line+// must publish the passage handle the answer is expected to cite. Without it the+// reference payload cannot resolve a claim-backed answer.+func TestClaimOutlineLineCarriesHandle(t *testing.T) {+ ctx := WithEvidenceRegistry(context.Background(), NewEvidenceRegistry())+ got := claimOutlineLine(ctx, claimHit{name: "Acme was founded in 1994", chunkID: "c1"})+ if !strings.HasSuffix(got, " [chunks: c1] ref=0") {+ t.Fatalf("claim line must carry the passage handle, got %q", got)+ }+ // The same chunk keeps its first handle.+ if again := claimOutlineLine(ctx, claimHit{name: "Acme is in Zurich", chunkID: "c1"}); !strings.HasSuffix(again, "ref=0") {+ t.Errorf("a repeated chunk must keep its handle, got %q", again)+ }+ // A different chunk takes the next one.+ if next := claimOutlineLine(ctx, claimHit{name: "Acme employs 900", chunkID: "c2"}); !strings.HasSuffix(next, "ref=1") {+ t.Errorf("a new chunk must take the next handle, got %q", next)+ }+}++// --- Query-only route (the tool must be usable in one call) ---++func installFakeStructureRoute(t *testing.T, docs [][2]string) {+ t.Helper()+ prev := navStructureRoute+ navStructureRoute = func(context.Context, string, string, string, int) ([][2]string, error) {+ return docs, nil+ }+ t.Cleanup(func() { navStructureRoute = prev })+}++// TestNavigateStructureQueryOnlyRoutesAndClaims: without a doc_id the tool routes+// the query itself and returns compiled claim evidence - this is what makes the+// tool callable in one step, without a prior navigate_tree.+func TestNavigateStructureQueryOnlyRoutesAndClaims(t *testing.T) {+ installFakeStructureRoute(t, [][2]string{{"d1", "Alpha"}})+ installFakeStructureReader(t, fakeClaimStructureReader{+ fakeStructureReader: fakeStructureReader{byDoc: map[string][]structureRow{}},+ claims: map[string][]claimHit{"d1": {{+ chunkID: "c1",+ name: "Acme was founded in 1994",+ quote: "Acme, founded in 1994",+ }}},+ })++ out, err := NewNavigateStructureTool("tenant-1", []string{"kb1"}).+ InvokableRun(context.Background(), `{"query":"when was Acme founded"}`)+ if err != nil {+ t.Fatalf("InvokableRun: %v", err)+ }+ for _, want := range []string{+ `count="1"`,+ `doc_id="d1"`,+ `- [claim] Acme was founded in 1994 | Evidence: "Acme, founded in 1994" [chunks: c1]`,+ } {+ if !strings.Contains(out, want) {+ t.Errorf("output must contain %q, got:\n%s", want, out)+ }+ }+ if strings.Contains(out, "route first with navigate_tree") {+ t.Errorf("a routed query must not ask for navigate_tree, got:\n%s", out)+ }+}++// TestNavigateStructureQueryOnlyNoClaimsIsAHint: routing that reaches documents+// with no claims says so instead of pretending the corpus has no structure.+func TestNavigateStructureQueryOnlyNoClaimsIsAHint(t *testing.T) {+ installFakeStructureRoute(t, [][2]string{{"d1", "Alpha"}})+ installFakeStructureReader(t, fakeClaimStructureReader{+ fakeStructureReader: fakeStructureReader{byDoc: map[string][]structureRow{}},+ claims: map[string][]claimHit{},+ })++ out, err := NewNavigateStructureTool("tenant-1", []string{"kb1"}).+ InvokableRun(context.Background(), `{"query":"nothing here"}`)+ if err != nil {+ t.Fatalf("InvokableRun: %v", err)+ }+ if !strings.Contains(out, `count="0"`) || !strings.Contains(out, "no claim hits") {+ t.Errorf("expected a claim-miss hint, got:\n%s", out)+ }+}++// TestNavigateStructureQueryOnlyNoRouteKeepsTheOldHint: when routing finds nothing+// at all, the tool still tells the model to route with navigate_tree.+func TestNavigateStructureQueryOnlyNoRouteKeepsTheOldHint(t *testing.T) {+ installFakeStructureRoute(t, nil)+ installFakeStructureReader(t, fakeStructureReader{byDoc: map[string][]structureRow{}})++ out, err := NewNavigateStructureTool("tenant-1", []string{"kb1"}).+ InvokableRun(context.Background(), `{"query":"nothing"}`)+ if err != nil {+ t.Fatalf("InvokableRun: %v", err)+ }+ if !strings.Contains(out, "route first with navigate_tree") {+ t.Errorf("expected the routing hint, got:\n%s", out)+ }+}++// TestRecallCompiledClaimsSufficient: enough distinct claims, from more than one+// document, make the turn answerable without reading a single raw chunk - and+// the evidence they carry is rendered WITH the handles the answer must cite.+func TestRecallCompiledClaimsSufficient(t *testing.T) {+ stubClaimProbe(t, true)+ installFakeStructureReader(t, fakeClaimStructureReader{+ fakeStructureReader: fakeStructureReader{byDoc: map[string][]structureRow{}},+ claims: map[string][]claimHit{+ "d1": {+ {chunkID: "c1", name: "Acme was founded in 1994", quote: "Acme, founded in 1994"},+ {chunkID: "c1", name: "Acme is based in Zurich", quote: "the Zurich-based firm"},+ },+ "d2": {+ {chunkID: "c2", name: "Acme employs 900 people", quote: "a workforce of 900"},+ },+ },+ })+ msgs := []*schema.Message{{Role: schema.User, Content: "when was Acme founded"}}+ ctx := WithEvidenceRegistry(context.Background(), NewEvidenceRegistry())+ pf := recallCompiledClaims(ctx, "tenant-1", []string{"kb1"}, msgs)++ if !pf.sufficient {+ t.Fatalf("3 claims over 2 documents must be sufficient, got %d claims over %v", len(pf.claims), pf.docs)+ }+ if len(pf.claims) != 3 || len(pf.docs) != 2 {+ t.Fatalf("recall = %d claims over %v, want 3 over 2 documents", len(pf.claims), pf.docs)+ }+ if !strings.Contains(pf.evidence, `doc_id="d1"`) || !strings.Contains(pf.evidence, `doc_id="d2"`) ||+ !strings.Contains(pf.evidence, "- [claim] Acme was founded in 1994") {+ t.Errorf("evidence must carry both docs and their claims, got:\n%s", pf.evidence)+ }+ for _, want := range []string{"ref=0", "ref=1"} {+ if !strings.Contains(pf.evidence, want) {+ t.Errorf("evidence must publish %q so the answer can cite it, got:\n%s", want, pf.evidence)+ }+ }+}++// TestRecallCompiledClaimsSingleDocumentIsNotSufficient: one document's claims+// answer one hop at most. This is also the exact shape a broken document ranking+// produces for every question, so the gate must refuse it.+func TestRecallCompiledClaimsSingleDocumentIsNotSufficient(t *testing.T) {+ stubClaimProbe(t, true)+ installFakeStructureReader(t, fakeClaimStructureReader{+ fakeStructureReader: fakeStructureReader{byDoc: map[string][]structureRow{}},+ claims: map[string][]claimHit{"d1": {+ {chunkID: "c1", name: "Acme was founded in 1994", quote: "Acme, founded in 1994"},+ {chunkID: "c1", name: "Acme is based in Zurich", quote: "the Zurich-based firm"},+ {chunkID: "c2", name: "Acme employs 900 people", quote: "a workforce of 900"},+ }},+ })+ msgs := []*schema.Message{{Role: schema.User, Content: "when was Acme founded"}}+ pf := recallCompiledClaims(context.Background(), "tenant-1", []string{"kb1"}, msgs)+ if pf.sufficient {+ t.Fatalf("claims from one document must not terminate the turn, got %d claims over %v", len(pf.claims), pf.docs)+ }+ if pf.evidence != "" {+ t.Errorf("an insufficient recall must not render evidence, got:\n%s", pf.evidence)+ }+}++// TestRecallCompiledClaimsEmptyDocumentScopeSkipsRecall: a non-nil EMPTY scope+// means the turn allows no documents. The reader treats an empty docIDs slice as+// "the whole dataset", so the recall must stop before the read instead of+// widening the turn's scope to everything.+func TestRecallCompiledClaimsEmptyDocumentScopeSkipsRecall(t *testing.T) {+ stubClaimProbe(t, true)+ installFakeStructureReader(t, fakeClaimStructureReader{+ fakeStructureReader: fakeStructureReader{byDoc: map[string][]structureRow{}},+ claims: map[string][]claimHit{+ "d1": {{chunkID: "c1", name: "Acme was founded in 1994", quote: "Acme, founded in 1994"}},+ "d2": {{chunkID: "c2", name: "Acme employs 900 people", quote: "a workforce of 900"}},+ },+ })+ msgs := []*schema.Message{{Role: schema.User, Content: "when was Acme founded"}}+ ctx := withDocumentScope(context.Background(), []string{})+ pf := recallCompiledClaims(ctx, "tenant-1", []string{"kb1"}, msgs)+ if len(pf.claims) != 0 || pf.sufficient || pf.evidence != "" {+ t.Fatalf("an empty document scope must skip the recall, got %d claims (sufficient=%v, evidence=%q)",+ len(pf.claims), pf.sufficient, pf.evidence)+ }+}++// TestRecallCompiledClaimsInsufficientIsDropped: a recall below the sufficiency+// count must not reach the model - neither as evidence nor as published handles+// for passages it never saw.+func TestRecallCompiledClaimsInsufficientIsDropped(t *testing.T) {+ stubClaimProbe(t, true)+ installFakeStructureRoute(t, [][2]string{{"d1", "Alpha"}})+ installFakeStructureReader(t, fakeClaimStructureReader{+ fakeStructureReader: fakeStructureReader{byDoc: map[string][]structureRow{}},+ claims: map[string][]claimHit{"d1": {{+ chunkID: "c1",+ name: "Acme was founded in 1994",+ quote: "Acme, founded in 1994",+ }}},+ })+ msgs := []*schema.Message{{Role: schema.User, Content: "when was Acme founded"}}+ registry := NewEvidenceRegistry()+ pf := recallCompiledClaims(WithEvidenceRegistry(context.Background(), registry), "tenant-1", []string{"kb1"}, msgs)+ if pf.sufficient {+ t.Fatalf("1 claim must not be sufficient (%d required)", navClaimSufficientHits)+ }+ if pf.evidence != "" {+ t.Errorf("an insufficient recall must not render evidence, got:\n%s", pf.evidence)+ }+ if got := len(registry.IDs()); got != 0 {+ t.Errorf("a dropped recall must not publish handles, got %d", got)+ }+}++// TestRecallCompiledClaimsSkipsEmptyHits: nothing matched, so nothing is+// offered - the loop runs exactly as it would without the recall.+func TestRecallCompiledClaimsSkipsEmptyHits(t *testing.T) {+ stubClaimProbe(t, true)+ installFakeStructureRoute(t, [][2]string{{"d1", "Alpha"}})+ installFakeStructureReader(t, fakeClaimStructureReader{+ fakeStructureReader: fakeStructureReader{byDoc: map[string][]structureRow{}},+ claims: map[string][]claimHit{},+ })+ msgs := []*schema.Message{{Role: schema.User, Content: "nothing here"}}+ pf := recallCompiledClaims(context.Background(), "tenant-1", []string{"kb1"}, msgs)+ if pf.sufficient || pf.evidence != "" || len(pf.claims) != 0 {+ t.Errorf("an empty recall must yield an empty verdict, got %+v", pf)+ }+}++// stubClaimProbe drives the claim-prefetch gate without an engine.+func stubClaimProbe(t *testing.T, present bool) {+ t.Helper()+ prev := claimRowsPresent+ claimRowsPresent = func(context.Context, string, []string) bool { return present }+ t.Cleanup(func() { claimRowsPresent = prev })+}++// claimsPerDoc builds n distinct claims for one document.+func claimsPerDoc(docID string, n int) []claimHit {+ out := make([]claimHit, 0, n)+ for i := 0; i < n; i++ {+ out = append(out, claimHit{+ chunkID: docID + "-c" + string(rune('a'+i)),+ name: docID + " claim " + string(rune('a'+i)),+ quote: "quote",+ })+ }+ return out+}++// TestRecallCapsClaims: the recall block is the whole prompt of the answer call,+// so it honours its cap - and the cap is spread across documents, not spent on+// the first one.+func TestRecallCapsClaims(t *testing.T) {+ stubClaimProbe(t, true)+ docs := []string{"d1", "d2", "d3", "d4"}+ claims := map[string][]claimHit{}+ for _, d := range docs {+ claims[d] = claimsPerDoc(d, 8)+ }+ installFakeStructureReader(t, fakeClaimStructureReader{+ fakeStructureReader: fakeStructureReader{byDoc: map[string][]structureRow{}},+ claims: claims,+ })+ msgs := []*schema.Message{{Role: schema.User, Content: "acme"}}+ pf := recallCompiledClaims(context.Background(), "tenant-1", []string{"kb1"}, msgs)+ if got := strings.Count(pf.evidence, "- [claim] "); got != navClaimPrefetchTopN {+ t.Errorf("recall rendered %d claims, want the cap %d:\n%s", got, navClaimPrefetchTopN, pf.evidence)+ }+ for _, docID := range docs {+ if !strings.Contains(pf.evidence, `doc_id="`+docID+`"`) {+ t.Errorf("the cap must be spread across documents, %s missing:\n%s", docID, pf.evidence)+ }+ }+}++// TestClaimAnswerPromptCarriesEvidenceAndCitationRule pins what the+// sufficient-claims answer depends on: the evidence itself, the rule that turns+// each fact line into a resolvable reference, and the refusal path - a model that+// narrates instead of answering must have an explicit way to say so, because the+// caller falls back to the loop on it.+func TestClaimAnswerPromptCarriesEvidenceAndCitationRule(t *testing.T) {+ evidence := "- [claim] Acme was founded in 1994 [chunks: c1] ref=0"+ prompt := claimAnswerPrompt(evidence)+ for _, want := range []string{evidence, "[ID:n]", "ref=n", "Do not call tools", `"answerable"`} {+ if !strings.Contains(prompt, want) {+ t.Errorf("claim answer prompt must contain %q, got:\n%s", want, prompt)+ }+ }+}++// TestParseClaimAnswer: the verdict decides control flow, so both the refusal and+// the malformed-reply cases must fall back rather than ship prose.+func TestParseClaimAnswer(t *testing.T) {+ ok := parseClaimAnswer(`{"answerable": true, "answer": "1994 [ID:0]"}`)+ if !ok.answerable || ok.answer != "1994 [ID:0]" {+ t.Errorf("answerable verdict parsed as %+v", ok)+ }+ no := parseClaimAnswer(`{"answerable": false, "missing": "the founding year"}`)+ if no.answerable || no.missing != "the founding year" {+ t.Errorf("refusal parsed as %+v", no)+ }+ // A model that narrates instead of answering must NOT be shipped as the+ // answer: that is the zero-score failure mode.+ narrated := parseClaimAnswer("I need to research who Brian Bergstein is. Let me search for this information.")+ if narrated.answerable || narrated.answer != "" {+ t.Errorf("narrated reply must be treated as not answerable, got %+v", narrated)+ }+ // "answerable" without an answer is not an answer.+ empty := parseClaimAnswer(`{"answerable": true, "answer": " "}`)+ if empty.answerable {+ t.Errorf("empty answer must be not answerable, got %+v", empty)+ }+}++// TestQueryOnlyToolCallIsUncapped: a model-invoked call is not capped - the model+// asked for it and its result lands in the history once.+func TestQueryOnlyToolCallIsUncapped(t *testing.T) {+ installFakeStructureRoute(t, [][2]string{{"d1", "A"}, {"d2", "B"}})+ installFakeStructureReader(t, fakeClaimStructureReader{+ fakeStructureReader: fakeStructureReader{byDoc: map[string][]structureRow{}},+ claims: map[string][]claimHit{+ "d1": claimsPerDoc("d1", 5),+ "d2": claimsPerDoc("d2", 5),+ },+ })+ out, err := NewNavigateStructureTool("tenant-1", []string{"kb1"}).+ InvokableRun(context.Background(), `{"query":"acme"}`)+ if err != nil {+ t.Fatalf("InvokableRun: %v", err)+ }+ if got := strings.Count(out, "- [claim] "); got != 10 {+ t.Errorf("a model-invoked call must not be capped, got %d claims:\n%s", got, out)+ }+}++// TestRecallSkippedWithoutClaimRows: a compiled but claim-less dataset (a tree+// or timeline) must not pay for a routing call and two retrieval legs.+func TestRecallSkippedWithoutClaimRows(t *testing.T) {+ stubClaimProbe(t, false)+ installFakeStructureRoute(t, [][2]string{{"d1", "A"}})+ installFakeStructureReader(t, fakeClaimStructureReader{+ fakeStructureReader: fakeStructureReader{byDoc: map[string][]structureRow{}},+ // Claims exist in the fake store on purpose: the gate must win, otherwise+ // this test would pass for the wrong reason.+ claims: map[string][]claimHit{"d1": {{+ chunkID: "c1",+ name: "Acme was founded in 1994",+ quote: "Acme, founded in 1994",+ }}},+ })+ msgs := []*schema.Message{{Role: schema.User, Content: "when was Acme founded"}}+ pf := recallCompiledClaims(context.Background(), "tenant-1", []string{"kb1"}, msgs)+ if pf.sufficient || pf.evidence != "" || len(pf.claims) != 0 {+ t.Errorf("no claim rows must disable the recall, got %+v", pf)+ }+}diff --git a/internal/agentic_rag/tool_navigate_tree_test.go b/internal/agentic_rag/tool_navigate_tree_test.goindex 7cb9f592df..0e50bcf01a 100644--- a/internal/agentic_rag/tool_navigate_tree_test.go+++ b/internal/agentic_rag/tool_navigate_tree_test.go@@ -28,6 +28,7 @@ import (// Search is exercised by navigate_tree; the rest satisfy the interface.type fakeNavService struct {search func(ctx context.Context, tenantID, kbID, query string, embd []float32, docScope []string, topK int) ([]nav.NavHit, error)+ encode func(ctx context.Context, tenantID, text string) []float64}func (f fakeNavService) UpsertDoc(context.Context, nav.UpsertDocInput) error { return nil }@@ -40,6 +41,12 @@ func (f fakeNavService) Search(ctx context.Context, tenantID, kbID, query string}return nil, nil}+func (f fakeNavService) EncodeQueryVector(ctx context.Context, tenantID, text string) []float64 {+ if f.encode != nil {+ return f.encode(ctx, tenantID, text)+ }+ return nil+}func (f fakeNavService) ListClusters(context.Context, string, string, string, int, int) ([]nav.NavNode, int64, error) {return nil, 0, nil}diff --git a/internal/ingestion/knowledge_compile/nav_lifecycle_test.go b/internal/ingestion/knowledge_compile/nav_lifecycle_test.goindex 4109665b35..a0d64186b1 100644--- a/internal/ingestion/knowledge_compile/nav_lifecycle_test.go+++ b/internal/ingestion/knowledge_compile/nav_lifecycle_test.go@@ -46,6 +46,10 @@ func (f *recordingNavService) Search(context.Context, string, string, string, []return nil, nil}+func (f *recordingNavService) EncodeQueryVector(context.Context, string, string) []float64 {+ return nil+}+func (f *recordingNavService) ListClusters(context.Context, string, string, string, int, int) ([]nav.NavNode, int64, error) {return nil, 0, nil}diff --git a/internal/service/nav/nav.go b/internal/service/nav/nav.goindex ad411533bf..c7af04f9f4 100644--- a/internal/service/nav/nav.go+++ b/internal/service/nav/nav.go@@ -108,6 +108,16 @@ type NavService interface {// returned coverage is trimmed to the scope so no out-of-scope document// surfaces under a cluster that merely overlaps it.Search(ctx context.Context, tenantID, kbID, query string, embd []float32, docScope []string, topK int) ([]NavHit, error)+ // EncodeQueryVector embeds one search query with the tenant's default+ // embedding model and returns it in the engine's float64 form.+ //+ // Every KNN read over compiled rows needs the query-side half of the model+ // that wrote those rows: nav routing (Search) and navigate_structure's claim+ // recall both address q_<dim>_vec columns the service's embedder produced.+ // Exposing it here means a caller that already reaches this service needs no+ // second wiring point for the embedder. nil/empty means the model did not+ // resolve, and the caller must degrade to lexical recall.+ EncodeQueryVector(ctx context.Context, tenantID, text string) []float64// ListClusters returns the depth-0 clusters (parent_kwd=root), or — when// keywords is non-empty — the pruned search forest: the matched leaves plus// the cluster path above each of them. The total is a cluster count in bothdiff --git a/internal/service/nav/nav_service_router_test.go b/internal/service/nav/nav_service_router_test.goindex 6d98d404ad..32e0a3d7f7 100644--- a/internal/service/nav/nav_service_router_test.go+++ b/internal/service/nav/nav_service_router_test.go@@ -37,6 +37,9 @@ func (f *scopeRecordingNav) Search(_ context.Context, _, _ string, _ string, _ [f.scope = append([]string(nil), docScope...)return f.hits, nil}+func (f *scopeRecordingNav) EncodeQueryVector(context.Context, string, string) []float64 {+ return nil+}func (f *scopeRecordingNav) ListClusters(context.Context, string, string, string, int, int) ([]NavNode, int64, error) {return nil, 0, nil}diff --git a/internal/service/nlp/datasetnav.go b/internal/service/nlp/datasetnav.goindex a94349b4d4..55b65890da 100644--- a/internal/service/nlp/datasetnav.go+++ b/internal/service/nlp/datasetnav.go@@ -97,6 +97,26 @@ type NavService struct {llm nav.NavMergeLLM // optional; nil disables LLM merge/summary}+// EncodeQueryVector embeds one SEARCH QUERY for the tenant and returns it in the+// engine's float64 form.+//+// It is the query-side seam the agentic claim recall uses to run a KNN leg over+// claim rows (which store their own q_<dim>_vec). The vector is produced by the+// tenant's DEFAULT embedding model, so on a dataset whose embedding model+// differs in dimension the KNN leg addresses a column the rows do not carry and+// recall falls back to its lexical leg — the documented degradation, and the+// reason an error here returns nil instead of failing the caller.+func (s *NavService) EncodeQueryVector(ctx context.Context, tenantID, text string) []float64 {+ if s == nil || s.embed == nil || tenantID == "" || strings.TrimSpace(text) == "" {+ return nil+ }+ vectors, err := encodeNavQuery(ctx, s.embed, tenantID, []string{text})+ if err != nil || len(vectors) == 0 {+ return nil+ }+ return f32ToF64Slice(vectors[0])+}+// NewNavService builds the ES-backed NavService. embed may be nil when the// caller guarantees every UpsertDocInput carries a precomputed Embedd.func NewNavService(embed NavEmbedder) *NavService {