Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors
cognee - Research Report
Follow-up: the September 2026 landscape refresh (how this project evolved, what the field looks like now, and what we concluded) is in
research-2026-landscape.md.
Source project:
topoteretes/cognee(Python, knowledge-graph + vector + relational, MCP server).
1. Purpose & Scope
Cognee bills itself as "the brain behind your agents", a memory control plane (README.md:40,67). It ingests heterogeneous data (text, PDF, CSV, code, web pages), then continuously builds a hybrid knowledge graph + vector index + relational catalog so that agents can retrieve context by both meaning (embeddings) and structure (graph relationships). The current public SDK is intentionally minimal, with four verbs: remember, recall, forget, improve (README L127). Internally, those map to the older add/cognify/search/prune primitives. Cognee positions itself between a RAG retriever and a "company brain," with multi-tenant isolation, ontology grounding, and a Claude Code plugin for capturing agent traces.
2. The Cognify Pipeline (heart of the system)
The pipeline is a list of Task objects executed by run_pipeline. Canonical definition at cognee/api/v1/cognify/cognify.py:316-344:
default_tasks = [
Task(classify_documents), # EXTRACT
Task(extract_chunks_from_documents, ...), # EXTRACT
Task(extract_graph_and_summarize, graph_model=..., ...), # COGNIFY (LLM)
Task(add_data_points, embed_triplets=embed_triplets, ...), # LOAD
Task(extract_dlt_fk_edges), # LOAD (relational FK edges)
]
Step-by-step:
Classify documents (
cognee/tasks/documents/classify_documents.py). Wraps each rawDatarow in a typedDocumentsubclass (PdfDocument,TextDocument,DltRowDocument, etc.) so downstream chunkers know how to read content.Chunk documents (
cognee/tasks/documents/extract_chunks_from_documents.py+cognee/modules/chunking/TextChunker.py:1-60). The defaultTextChunkeruseschunk_by_paragraphto pack paragraphs up tomax_chunk_sizetokens (calculated asmin(embedding_max_completion_tokens, llm_max_completion_tokens // 2)). Each chunk becomes aDocumentChunkDataPoint withmetadata={"index_fields": ["text"]}; the storage layer later reads thoseindex_fieldsto know what to embed.Extract graph + summarize (
cognee/tasks/graph/extract_graph_and_summarize.py:21-37). Two LLM tasks concurrently on every chunk viaasyncio.gather:
extract_graph_from_data(cognee/tasks/graph/extract_graph_from_data.py:128-222) callsextract_content_graph(chunk.text, graph_model, custom_prompt)for each chunk, aninstructor/litellm-backed structured call that returns a PydanticKnowledgeGraphofNode/Edge. Edges with missing source/target are filtered (L181-188). Entity nodes are then validated against an ontology viaexpand_with_nodes_and_edges(..., ontology_resolver, ...)(L110-112). Provenance is stamped onto every DataPoint (_stamp_provenance_deep, L30-53). Existing edges are deduplicated viaretrieve_existing_edges.summarize_textproduces aTextSummaryfor each chunk used by later "SUMMARIES" search.
- Persist nodes, edges & embeddings (
cognee/tasks/storage/add_data_points.py:30-149).get_graph_from_modelrecursively walks the Pydantic graph into(nodes, edges)tuples, thendeduplicate_nodes_and_edgesremoves duplicates. The pipeline branches onEngineCapability.HYBRID_WRITE:
- Hybrid backend (e.g. Postgres+pgvector):
graph_engine.add_nodes_with_vectors(nodes)in one transaction. - Otherwise, parallel writes:
graph_engine.add_nodes(nodes)+index_data_points(...)to the vector engine. - If
embed_triplets=True, builds(source -› relation -› target)text strings and embeds them as additionalTripletDataPoints (L184-265). This makes graph-walk retrieval findable by similarity.
- DLT FK edges: adds deterministic foreign-key edges for ingested SQL/DLT-backed datasets without LLM cost.
There's also a parallel temporal pipeline (cognify.py:347-392) that swaps the graph extractor for extract_events_and_timestamps → extract_knowledge_graph_from_events to build a time-aware graph.
3. Storage Backends - Triple-Stack, Pluggable
Cognee always uses three stores in parallel, abstracted by a UnifiedStoreEngine (cognee/infrastructure/databases/unified/unified_store_engine.py:11-66):
| Layer | Default | Other supported |
|---|---|---|
| Graph | ladybug (Kuzu fork, file-based) |
neo4j, postgres (AGE-style), kuzu, ladybug-remote, neptune, neptune_analytics (hybrid) |
| Vector | lancedb (file-based, subprocess-isolated) |
pgvector, chromadb, neptune_analytics |
| Relational | sqlite (aiosqlite) |
postgres via SQLAlchemy async |
Selection is environment-driven: GRAPH_DATABASE_PROVIDER, VECTOR_DB_PROVIDER, DB_PROVIDER, with normalized credentials. There is also a USE_UNIFIED_PROVIDER=pghybrid short-circuit that makes a single Postgres instance back graph+vector+relational simultaneously.
Two EngineCapability flags (HYBRID_WRITE, HYBRID_SEARCH) let the pipeline branch on whether nodes+vectors can land in one transaction (saves a round trip) or need two writes.
The default local-only stack is therefore SQLite + LanceDB + Ladybug/Kuzu, all file-based, with no servers required. For homelab parity, that's the lean setup.
4. Search / Retrieval
Recall is multi-strategy and auto-routed. SearchType enum (cognee/modules/search/types/SearchType.py) lists 16 strategies: GRAPH_COMPLETION (default), GRAPH_COMPLETION_COT, GRAPH_COMPLETION_CONTEXT_EXTENSION, GRAPH_COMPLETION_DECOMPOSITION, GRAPH_SUMMARY_COMPLETION, RAG_COMPLETION, TRIPLET_COMPLETION, CHUNKS, CHUNKS_LEXICAL, SUMMARIES, CYPHER, NATURAL_LANGUAGE, TEMPORAL, FEELING_LUCKY, CODING_RULES, AGENTIC_COMPLETION.
The recall() API (cognee/api/v1/recall/recall.py:314-513) picks one via route_query(query_text), a rule-based classifier in query_router.py (regex patterns for "when/before/after" → TEMPORAL, keyword fragments → CHUNKS_LEXICAL, multi-hop wording → GRAPH_COMPLETION_COT, etc.). Default fallback is GRAPH_COMPLETION.
The main retriever is GraphCompletionRetriever (cognee/modules/retrieval/graph_completion_retriever.py). It uses brute_force_triplet_search (cognee/modules/retrieval/utils/brute_force_triplet_search.py:216-355), which:
- Embeds the query, runs vector search across multiple collections in parallel:
Entity_name,TextSummary_text,EntityType_name,DocumentChunk_text,EdgeType_relationship_name(L281-290). - Projects results into a
CogneeGraphmemory fragment, scoring triplets by combined node+edge similarity withtriplet_distance_penaltyandfeedback_influence(L318-333). - Optionally expands a
neighborhood_depthhop-out from top seed nodes. - Resolves the top-K edges to natural-language sentences (
resolve_edges_to_text). - Feeds them into an LLM completion with
graph_context_for_question.txtas the user prompt.
Recall also supports a session-cache first pass (keyword match against recent QA entries in a relational session table) and falls through to the graph when the session has no hit (recall.py:382-397, 447-457). That's the "hybrid working-memory + long-term memory" pattern.
5. Memory Lifecycle - Cognee Does Reorganize
Cognee has explicit memory enrichment beyond one-shot ingest. The improve() API (cognee/api/v1/improve/improve.py:36-232) runs up to five stages:
- Feedback weights: session entries with thumbs-up/down ratings adjust
feedback_weighton the specific graph nodes/edges that were used to answer (apply_feedback_weights_pipeline, L284-299). Tracked viaused_graph_element_idsrecorded at retrieval time. Higher-rated answers boost their source nodes; lower-rated ones decrease them. - Persist session Q&A: cognifies session transcripts into permanent graph tagged
node_set="user_sessions_from_cache". - Triplet enrichment / memify:
cognee/memify_pipelines/create_triplet_embeddings.pybuilds and embeds new triplet datapoints. - Global context index:
global_context_index_pipelinebuilds bucket+root summaries over all text summaries. - Sync graph→session: incrementally copies recently-added graph edges into session caches as JSON-lines so live agents pick up new knowledge without a re-query.
There's also a consolidate_entity_descriptions.py pipeline (cognee/memify_pipelines/consolidate_entity_descriptions.py) that walks Entity nodes, fetches neighbors+edges, and rewrites their description field via an LLM (NodeDescription Pydantic). And apply_frequency_weights.py ages knowledge by usage frequency. Deduplication happens at write time in add_data_points.py and at extraction time in retrieve_existing_edges.
So ingest is not one-shot. There is a feedback loop, summary consolidation, and an edge-aging concept. There is no automatic decay/TTL, but feedback_weight and frequency_weight give you the levers.
6. MCP Integration
Yes, in cognee-mcp/ as a sibling project with its own pyproject. The server (cognee-mcp/src/server.py) uses the official mcp SDK over stdio/SSE/HTTP. The publicly exposed tools are deliberately minimal (L1076-1222):
remember(data, dataset_name?, session_id?, custom_prompt?): withsession_id, a fast session-cache write; without, runs the fulladd + cognifypipeline.recall(query, search_type?, datasets?, session_id?, top_k=10): auto-routes the search type.forget(dataset?, everything=False): deletes across all three stores.
There are internal tools registered but not exposed in API mode: cognify, search, list_data, delete, prune, improve, plus a UI bundle (visualize_graph_ui, upload_file_ui, open_cognee_workspace, cognify_file) that opens an embedded HTML workspace from src/app_bundles/visualize-graph.html.
One design choice worth noting: per-MCP-client auto-named datasets (cursor_vscode_memory, claude_code_memory) so different agents don't pollute each other's memory unless they opt in by passing dataset_name="main_dataset". Toggle with COGNEE_MCP_AGENT_SCOPED=false.
MCP integration is manual in the sense that the agent invokes remember/recall explicitly. The Claude Code plugin (separate repo cognee-integrations) automates it via Claude Code lifecycle hooks (SessionStart, PostToolUse, UserPromptSubmit, PreCompact, SessionEnd); see README L204.
7. Operational Concerns
Deployment: A top-level Dockerfile builds the FastAPI server; cognee-mcp/Dockerfile builds the MCP server. docker-compose.yml ships the API on port 8000 with resource limits of 4 CPUs and 8 GB RAM, which is a lot. Deployment targets include Cognee Cloud, Modal, Railway, Fly.io, Render, Daytona (distributed/deploy/).
LLM cost per ingest: heavy. For every chunk, cognify runs:
- 1 structured-extraction call (entity/relationship → Pydantic KnowledgeGraph)
- 1 summarization call
- N embedding calls (one per Entity/EntityType/DocumentChunk/TextSummary/EdgeType row, plus optional Triplet rows)
A 100-chunk document easily generates 200+ LLM calls plus hundreds of embeddings. chunks_per_batch defaults to 100; LLM_RATE_LIMIT_* env vars exist but are off by default.
What needs API keys: LLM_API_KEY is mandatory unless you point at a local Ollama/HuggingFace model via the ollama/huggingface extras. Default provider is OpenAI (openai>=1.80.1 is a core dep). Defaults to its own embedding model via litellm.
Local-only stack: SQLite + LanceDB + Ladybug means no external services for the storage layer, but the LLM is the cost driver. Replacing OpenAI with Ollama makes it CPU/GPU-heavy but free.
8. Strongest Ideas to Adopt
Task-list pipeline composition.
cognify.py:316-344reads like a data-flow recipe. EachTaskis a thin wrapper with abatch_sizeconfig and a context object. This is the cleanest pattern to port to Rust: represent the pipeline as a typedVec<Box<dyn Task>>and let tasks declare their batching behavior.Triplet embeddings as first-class citizens. Embedding
(source -› relation -› target)text directly (add_data_points.py:184-265) lets a graph become searchable by semantic similarity, by walking. It bridges "vector RAG" and "graph RAG."Capability-flag-driven storage facade.
UnifiedStoreEngine.has_capability(HYBRID_WRITE)lets the same pipeline target separate(Kuzu, LanceDB)or fused(Postgres+pgvector)without conditionals scattered everywhere.improve()lifecycle with feedback weights. Thefeedback_weighton graph elements +used_graph_element_idsfrom the retrieval trace gives you graded knowledge without retraining anything.Multi-collection vector search at recall time. Querying
Entity_name,TextSummary_text,DocumentChunk_text, and edge collections in parallel and merging by triplet score, rather than picking one index, is the core of retrieval.Provenance stamping (
_stamp_provenance_deep): every DataPoint carriessource_pipeline+source_task, making the graph debuggable.
9. Realistic Downsides for a Lean Homelab
- Dependency weight: 40+ core dependencies (
openai,litellm,instructor,sqlalchemy,aiosqlite,lancedb,pylance,ladybug,networkx,pypdf,fastapi,fastapi-users,rdflib,langdetect,datamodel-code-generator,tiktoken,tenacity,aiolimiter,diskcache,fakeredis, plus optional extras (neo4j, chromadb, postgres, anthropic, ollama, huggingface, scraping, dlt, monitoring)). Cold install is huge:poetry.lockis 1.4 MB. - Python async everywhere + LRU-cached engine handles (
closing_lru_cache): debugging stale-adapter issues required adding_GraphEngineHandle(see the 50-line docstring atget_graph_engine.py:56-105). That's complexity you inherit. - Per-cognify LLM cost is high. There's no obvious caching of "I've already extracted entities from this chunk hash", so re-ingestion repeats work.
incremental_loading=Truehelps but only at the dataset level. - Three databases to back up, three to migrate. The "wipe
.cognee_system/when you flipENABLE_BACKEND_ACCESS_CONTROL" caveat (cognee-mcp/README L518-525) shows the multi-store coordination is fragile. - 8 GB RAM limit in docker-compose is the recommended floor. LanceDB + Kuzu + embedding model loaded simultaneously is heavy on a low-end NAS.
- Bus factor on
ladybug: a 0.16.0 pinned Kuzu fork by the same org. If they stop maintaining it, you're on a single-vendor graph DB. - Two-tier MCP API: public 3 tools + 10+ "internal" tools means the API surface is in motion; not stable.
For a Rust homelab MCP server, the lean equivalents would be SQLite (relational) + a single embedded vector store (e.g. lancedb-rs or embedded sqlite-vec) + an embedded graph DB (CozoDB, SurrealDB, or just SQL tables with proper indices). Then port the task pipeline + triplet embedding + capability flag pattern and skip the FastAPI/users/migrations/UI bundle stack entirely.
Key file references
- Pipeline definition:
cognee/api/v1/cognify/cognify.py:316-344 - Graph extraction (LLM call site):
cognee/tasks/graph/extract_graph_from_data.py:128-222 - Triple-store write path:
cognee/tasks/storage/add_data_points.py:62-149 - Triplet embeddings:
cognee/tasks/storage/add_data_points.py:184-265 - Vector backend factory:
cognee/infrastructure/databases/vector/create_vector_engine.py:150-318 - Graph backend factory:
cognee/infrastructure/databases/graph/get_graph_engine.py:241-457 - Unified facade:
cognee/infrastructure/databases/unified/unified_store_engine.py:11-66 - Retrieval (graph+vector hybrid):
cognee/modules/retrieval/utils/brute_force_triplet_search.py:216-355 - Graph completion retriever:
cognee/modules/retrieval/graph_completion_retriever.py - Recall + auto-router:
cognee/api/v1/recall/recall.py:314-513,cognee/api/v1/recall/query_router.py - Improve / lifecycle:
cognee/api/v1/improve/improve.py:36-411 - Memify pipelines:
cognee/memify_pipelines/ - MCP server:
cognee-mcp/src/server.py:1076-1222 - Deps and extras:
pyproject.toml:22-160