Hypergraph is more powerful. Transform unstructured text into structured knowledge with LLMs. Graphs, hypergraphs, and spatio-temporal extractions — with one command.
feat: pickle-free JSON index storage for AutoModel/AutoList (#116)
yifanfeng97a408083b59fc
diff --git a/README.md b/README.mdindex bcf7cb5..504400d 100644--- a/README.md+++ b/README.md@@ -342,7 +342,7 @@ identifiers:## 📰 What's New-**v0.10.0** — 📤 New exports: JSON-LD & Cypher (Neo4j) alongside GraphML/CSV/Obsidian — all four also available via MCP · 📄 `he feed` now accepts directories · 🧠 Native Gemini support (`pip install "hyperextract[google]"`) · 🛡️ Windows config hardening & scoped-search fixes across every KA type.+**v0.10.1** — 🔒 Pickle-free index storage: `AutoModel`/`AutoList` indexes are JSON now — legacy pickle indexes still load with a warning (`he build-index --force` migrates). *(#116)*📰 **[Full release notes](https://yifanfeng97.github.io/Hyper-Extract/latest/news/)** · [All releases](https://github.com/yifanfeng97/hyper-extract/releases)diff --git a/README_ZH.md b/README_ZH.mdindex b8bd877..2eb25de 100644--- a/README_ZH.md+++ b/README_ZH.md@@ -342,7 +342,7 @@ identifiers:## 📰 最新动态-**v0.10.0** — 📤 新增导出:JSON-LD 与 Cypher(Neo4j),连同 GraphML/CSV/Obsidian——四种格式均可经 MCP 调用 · 📄 `he feed` 现在支持目录 · 🧠 原生 Gemini 支持(`pip install "hyperextract[google]"`)· 🛡️ Windows 配置加固与全部知识库类型的范围检索修复。+**v0.10.1** — 🔒 Pickle-free index storage: `AutoModel`/`AutoList` indexes are JSON now — legacy pickle indexes still load with a warning (`he build-index --force` migrates). *(#116)*📰 **[完整版本说明](https://yifanfeng97.github.io/Hyper-Extract/latest/zh/news/)** · [全部 Releases](https://github.com/yifanfeng97/hyper-extract/releases)diff --git a/docs/en/cli/commands/build-index.md b/docs/en/cli/commands/build-index.mdindex a2a798c..39953e1 100644--- a/docs/en/cli/commands/build-index.md+++ b/docs/en/cli/commands/build-index.md@@ -116,7 +116,7 @@ The index is stored in the knowledge abstract directory:└── docstore.json # Document store mapping```-Load an existing `index/` only from Knowledge Abstract directories you created or fully trust. Local FAISS indexes require deserialization on load.+Load an existing `index/` only from Knowledge Abstract directories you created or fully trust. Since v0.10.1 indexes are JSON (safe to load); legacy pickle indexes (<= v0.10.0) are deserialized on load — rebuild with `--force` to migrate.---diff --git a/docs/en/news.md b/docs/en/news.mdindex b4ba50f..15c72e6 100644--- a/docs/en/news.md+++ b/docs/en/news.md@@ -4,6 +4,12 @@ Release notes and highlights. For a complete changelog, see the [GitHub releases---+## v0.10.1 — Pickle-Free Index Storage++- **🔒 Safe index format** — `AutoModel`/`AutoList` indexes are now stored as JSON (`index.json`: vectors + documents) and rebuilt in memory on load — **no pickle, no code execution**. Legacy pickle indexes (<= v0.10.0) still load with a deserialization warning; `he build-index --force` migrates them. *(#116)*++---+## v0.10.0 — More Export Formats, Gemini, and Quality Hardening- **📤 JSON-LD & Cypher exports** — `he export jsonld` (pairwise edges + GraphML-style hyperedges) and `he export cypher` (Neo4j-compatible MERGE script; N-ary edges become Hyperedge nodes, not lossy cliques). *(#139, #141)*diff --git a/docs/en/python/guides/saving-loading.md b/docs/en/python/guides/saving-loading.mdindex 1e1cf8d..19f5923 100644--- a/docs/en/python/guides/saving-loading.md+++ b/docs/en/python/guides/saving-loading.md@@ -54,7 +54,7 @@ result.dump("./my_ka/")## Loading Knowledge Abstracts!!! warning "Trusted directories only"- Local FAISS indexes under `index/` are deserialized when you load a Knowledge Abstract. Only call `load()` (or CLI/MCP commands that load a KA) on directories you generated yourself or fully trust.+ Since v0.10.1, indexes are stored as JSON (`index.json`) and load without code execution. Legacy KAs (<= v0.10.0) keep pickle-based indexes, which ARE deserialized on load — only call `load()` (or CLI/MCP commands that load a KA) on directories you generated yourself or fully trust. Rebuild with `he build-index --force` to migrate a legacy KA.### Basic Loaddiff --git a/docs/zh/cli/commands/build-index.md b/docs/zh/cli/commands/build-index.mdindex 1d51c1b..53373b7 100644--- a/docs/zh/cli/commands/build-index.md+++ b/docs/zh/cli/commands/build-index.md@@ -116,7 +116,7 @@ he build-index ./ka/ -f└── docstore.json # 文档存储映射```-请只加载自己生成或完全信任的知识库目录中的 `index/`。本地 FAISS 索引在加载时需要反序列化。+请只加载自己生成或完全信任的知识库目录中的 `index/`。自 v0.10.1 起索引为 JSON(加载安全);旧版 pickle 索引(<= v0.10.0)加载时会反序列化——用 `--force` 重建即可迁移。---diff --git a/docs/zh/news.md b/docs/zh/news.mdindex 96df3ef..bff2878 100644--- a/docs/zh/news.md+++ b/docs/zh/news.md@@ -4,6 +4,12 @@---+## v0.10.1 — 索引存储脱离 pickle++- **🔒 安全的索引格式** — `AutoModel`/`AutoList` 的索引改为 JSON 存储(`index.json`:向量 + 文档),加载时在内存中重建——**没有 pickle,不执行任何代码**。旧 pickle 索引(<= v0.10.0)仍可加载但会给出警告;`he build-index --force` 即可迁移。*(#116)*++---+## v0.10.0 — 更多导出格式、Gemini 与质量加固- **📤 JSON-LD 与 Cypher 导出** — `he export jsonld`(二元边 + GraphML 风格超边)与 `he export cypher`(Neo4j 兼容 MERGE 脚本;N 元边编码为 Hyperedge 节点,不做有损团 expansion)。*(#139, #141)*diff --git a/docs/zh/python/guides/saving-loading.md b/docs/zh/python/guides/saving-loading.mdindex 191bc50..8ae3f71 100644--- a/docs/zh/python/guides/saving-loading.md+++ b/docs/zh/python/guides/saving-loading.md@@ -53,7 +53,7 @@ result.dump("./my_ka/")## 加载知识库!!! warning "只加载受信任的目录"- 加载知识库时,`index/` 下的本地 FAISS 索引需要反序列化。请只对自己生成或完全信任的知识库目录调用 `load()`(以及会加载 KA 的 CLI/MCP 命令)。+ 自 v0.10.1 起,索引以 JSON(`index.json`)存储,加载时不执行任何代码。旧版本(<= v0.10.0)的 KA 仍是 pickle 索引,加载时会反序列化——请只对自己生成或完全信任的知识库目录调用 `load()`(以及会加载 KA 的 CLI/MCP 命令)。用 `he build-index --force` 重建即可迁移旧 KA。### 基本加载diff --git a/hyperextract/types/list.py b/hyperextract/types/list.pyindex c6a5bee..aa8c1c3 100644--- a/hyperextract/types/list.py+++ b/hyperextract/types/list.py@@ -19,6 +19,7 @@ from langchain_core.language_models.chat_models import BaseChatModelfrom ontosight import view_nodesfrom pydantic import BaseModel, Field, create_model+from hyperextract.utils.json_index import dump_index_json, load_index_jsonfrom hyperextract.utils.logging import get_loggerfrom ._faiss import _warn_untrusted_faiss_load@@ -311,16 +312,26 @@ class AutoList(BaseAutoType[AutoListSchema[ItemSchema]], Generic[ItemSchema]):# ==================== Index Storage ====================def dump_index(self, folder_path: str | Path) -> None:- """Saves FAISS vector index to disk."""+ """Saves the vector index as JSON (no pickle)."""if self._index is None:return- self._index.save_local(folder_path)+ dump_index_json(self._index, folder_path)def load_index(self, folder_path: str | Path) -> None:- """Loads FAISS vector index from disk."""+ """Loads the vector index from disk.++ JSON indexes (written by >=0.10.1) are loaded without any code+ execution. Legacy pickle-based indexes still load with a+ deserialization warning — rebuild with `he build-index --force`+ to migrate them to JSON.+ """folder = Path(folder_path)if not folder.is_dir():raise ValueError(f"Folder does not exist: {folder_path}")+ index = load_index_json(folder, self.embedder)+ if index is not None:+ self._index = index+ return_warn_untrusted_faiss_load(folder)self._index = FAISS.load_local(str(folder), self.embedder, allow_dangerous_deserialization=Truediff --git a/hyperextract/types/model.py b/hyperextract/types/model.pyindex 308280e..86f2177 100644--- a/hyperextract/types/model.py+++ b/hyperextract/types/model.py@@ -14,6 +14,7 @@ from langchain_core.language_models.chat_models import BaseChatModelfrom ontomem.merger import BaseMerger, MergeStrategy, create_mergerfrom ontosight import view_nodes+from hyperextract.utils.json_index import dump_index_json, load_index_jsonfrom hyperextract.utils.logging import get_loggerfrom ._faiss import _warn_untrusted_faiss_load@@ -302,16 +303,26 @@ class AutoModel(BaseAutoType[T]):# ==================== Index Storage ====================def dump_index(self, folder_path: str | Path) -> None:- """Saves FAISS vector index to disk."""+ """Saves the vector index as JSON (no pickle)."""if self._index is None:return- self._index.save_local(folder_path)+ dump_index_json(self._index, folder_path)def load_index(self, folder_path: str | Path) -> None:- """Loads FAISS vector index from disk."""+ """Loads the vector index from disk.++ JSON indexes (written by >=0.10.1) are loaded without any code+ execution. Legacy pickle-based indexes still load with a+ deserialization warning — rebuild with `he build-index --force`+ to migrate them to JSON.+ """folder = Path(folder_path)if not folder.is_dir():raise ValueError(f"Folder does not exist: {folder_path}")+ index = load_index_json(folder, self.embedder)+ if index is not None:+ self._index = index+ return_warn_untrusted_faiss_load(folder)self._index = FAISS.load_local(str(folder), self.embedder, allow_dangerous_deserialization=Truediff --git a/hyperextract/utils/json_index.py b/hyperextract/utils/json_index.pynew file mode 100644index 0000000..87796fe--- /dev/null+++ b/hyperextract/utils/json_index.py@@ -0,0 +1,135 @@+"""JSON-backed FAISS index persistence (no pickle deserialization).++Legacy ``FAISS.save_local`` writes ``index.faiss`` + ``index.pkl``; loading+the pickle executes code inside it, so a KA directory is only as safe as+its source. This module stores the raw vectors and documents as plain JSON+and rebuilds the flat index in memory on load — same search behavior, zero+code execution.++Layout (``index.json``)::++ {+ "version": 1,+ "metric": 1, # faiss metric_type of the saved index+ "dimension": 768,+ "vectors": [[...], ...], # one vector per item, index order+ "texts": [...], # page_content per item+ "metadatas": [...], # metadata per item+ "ids": [...] # docstore ids per item+ }++Only flat indexes (as built by ``FAISS.from_documents`` with default+settings) are supported — that is what AutoModel/AutoList build.+"""++import json+from pathlib import Path+from typing import Any++import numpy as np+from langchain_community.vectorstores import FAISS++from hyperextract.utils.logging import get_logger++logger = get_logger(__name__)++INDEX_JSON_NAME = "index.json"+LEGACY_INDEX_NAME = "index.faiss"+LEGACY_PICKLE_NAME = "index.pkl"+++def has_json_index(folder: str | Path) -> bool:+ """True when the folder contains a JSON index."""+ return (Path(folder) / INDEX_JSON_NAME).is_file()+++def has_legacy_index(folder: str | Path) -> bool:+ """True when the folder contains a legacy pickle-based FAISS index."""+ folder = Path(folder)+ return (folder / LEGACY_INDEX_NAME).is_file() and (+ folder / LEGACY_PICKLE_NAME+ ).is_file()+++def dump_index_json(index: FAISS, folder: str | Path) -> Path:+ """Persist a FAISS vector index as JSON (vectors + documents, no pickle).++ Returns the written ``index.json`` path. Legacy ``index.faiss`` /+ ``index.pkl`` files in the folder are removed after a successful write+ so a stale pickle index can never shadow the JSON one.+ """+ folder = Path(folder)+ folder.mkdir(parents=True, exist_ok=True)++ ntotal = int(index.index.ntotal)+ vectors = (+ index.index.reconstruct_n(0, ntotal)+ if ntotal+ else np.zeros((0, int(index.index.d)), dtype="float32")+ )++ items = []+ for i in range(ntotal):+ doc_id = index.index_to_docstore_id[i]+ doc = index.docstore.search(doc_id)+ items.append(+ {+ "id": doc_id,+ "text": doc.page_content,+ "metadata": doc.metadata,+ }+ )++ payload = {+ "version": 1,+ "metric": int(index.index.metric_type),+ "dimension": int(index.index.d),+ "vectors": np.asarray(vectors, dtype="float32").tolist(),+ "texts": [item["text"] for item in items],+ "metadatas": [item["metadata"] for item in items],+ "ids": [item["id"] for item in items],+ }+ path = folder / INDEX_JSON_NAME+ path.write_text(json.dumps(payload, ensure_ascii=False), encoding="utf-8")+ logger.info("json_index_dumped", path=str(path), items=ntotal)++ for legacy in (LEGACY_INDEX_NAME, LEGACY_PICKLE_NAME):+ stale = folder / legacy+ if stale.exists():+ stale.unlink()+ logger.info("legacy_index_removed", path=str(stale))+ return path+++def load_index_json(folder: str | Path, embedder: Any) -> FAISS | None:+ """Load a JSON index and rebuild the FAISS flat index in memory.++ Returns ``None`` when the folder has no ``index.json``. Vectors are+ restored exactly as saved — no re-embedding happens.+ """+ path = Path(folder) / INDEX_JSON_NAME+ if not path.is_file():+ return None++ payload = json.loads(path.read_text(encoding="utf-8"))+ vectors = payload.get("vectors", [])+ texts = payload.get("texts", [])+ metadatas = payload.get("metadatas", [])+ ids = payload.get("ids", [])+ if not (len(vectors) == len(texts) == len(metadatas) == len(ids)):+ raise ValueError(f"Corrupt index.json: field lengths differ in {path}")+ if not vectors:+ raise ValueError(f"Corrupt index.json: no entries in {path}")++ metric = int(payload.get("metric", 0))+ if metric: # only flat L2 indexes are written today+ saved = metric+ restored = 1 # faiss.METRIC_L2+ if saved != restored:+ logger.warning("json_index_metric_mismatch saved=%s rebuilt=L2", saved)++ index = FAISS.from_embeddings(+ list(zip(texts, vectors)), embedder, metadatas=metadatas, ids=ids+ )+ logger.info("json_index_loaded", path=str(path), items=len(ids))+ return indexdiff --git a/pyproject.toml b/pyproject.tomlindex 20390fd..0d2d5fe 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -1,6 +1,6 @@[project]name = "hyperextract"-version = "0.10.0"+version = "0.10.1"description = "An intelligent, LLM-powered knowledge extraction and evolution framework with semantic search capabilities"readme = "README.md"requires-python = ">=3.11"diff --git a/tests/utils/test_json_index.py b/tests/utils/test_json_index.pynew file mode 100644index 0000000..a419d29--- /dev/null+++ b/tests/utils/test_json_index.py@@ -0,0 +1,172 @@+"""JSON-backed FAISS index persistence for AutoModel/AutoList (#116).++New dumps are plain JSON (no pickle, safe to load); legacy pickle indexes+from older releases still load with a deserialization warning.+"""++import json+import logging++import pytest+from langchain_community.vectorstores import FAISS+from langchain_core.documents import Document+from pydantic import BaseModel++from hyperextract.types import AutoList, AutoModel+from hyperextract.utils.json_index import (+ dump_index_json,+ has_json_index,+ load_index_json,+)+++class _City(BaseModel):+ name: str+ country: str = ""+++def _model(llm_client, embedder, tmp_path):+ ka = AutoModel(+ data_schema=_City,+ llm_client=llm_client,+ embedder=embedder,+ )+ # Seed the index directly: page_content is the field name, metadata+ # carries the raw fields — same shape build_index() produces.+ ka._index = FAISS.from_documents(+ [+ Document(page_content="name", metadata={"raw": {"name": "Paris"}}),+ Document(+ page_content="country",+ metadata={"raw": {"name": "Paris", "country": "FR"}},+ ),+ ],+ ka.embedder,+ )+ return ka+++def _list(llm_client, embedder):+ return AutoList(+ item_schema=_City,+ llm_client=llm_client,+ embedder=embedder,+ )+++def _seed_list(ka):+ ka._index = FAISS.from_documents(+ [+ Document(page_content="name", metadata={"raw": {"name": "Paris"}}),+ Document(page_content="name", metadata={"raw": {"name": "Tokyo"}}),+ ],+ ka.embedder,+ )+++class TestJsonRoundTrip:+ def test_dump_writes_json_and_drops_legacy(self, llm_client, embedder, tmp_path):+ ka = _model(llm_client, embedder, tmp_path)+ # Simulate a legacy folder first+ ka._index.save_local(tmp_path)+ assert (tmp_path / "index.faiss").exists()++ ka.dump_index(tmp_path)++ assert has_json_index(tmp_path)+ assert not (tmp_path / "index.faiss").exists()+ assert not (tmp_path / "index.pkl").exists()++ def test_model_round_trip_preserves_search(self, llm_client, embedder, tmp_path):+ ka = _model(llm_client, embedder, tmp_path)+ ka.build_index()+ ka.dump_index(tmp_path)++ restored = _model(llm_client, embedder, tmp_path)+ restored.load_index(tmp_path)+ # Compare at the index level: AutoModel.search() additionally needs+ # extracted data, which this round-trip does not carry.+ original_docs = ka._index.similarity_search("name", k=2)+ restored_docs = restored._index.similarity_search("name", k=2)+ assert [d.metadata for d in restored_docs] == [+ d.metadata for d in original_docs+ ]++ def test_list_round_trip(self, llm_client, embedder, tmp_path):+ ka = _list(llm_client, embedder)+ _seed_list(ka)+ ka.build_index()+ ka.dump_index(tmp_path)++ restored = _list(llm_client, embedder)+ restored.load_index(tmp_path)+ hits = restored.search("Paris", top_k=2)+ assert {h.name for h in hits} <= {"Paris", "Tokyo"}++ def test_json_load_is_picle_free_payload(self, llm_client, embedder, tmp_path):+ ka = _model(llm_client, embedder, tmp_path)+ ka.build_index()+ ka.dump_index(tmp_path)+ payload = json.loads((tmp_path / "index.json").read_text(encoding="utf-8"))+ assert payload["version"] == 1+ assert payload["dimension"] > 0+ assert len(payload["vectors"]) == len(payload["texts"])+++class TestLegacyCompat:+ def test_legacy_pickle_folder_still_loads(+ self, llm_client, embedder, tmp_path, caplog+ ):+ ka = _model(llm_client, embedder, tmp_path)+ ka.build_index()+ # Simulate an old KA: raw save_local, no index.json+ ka._index.save_local(tmp_path)+ assert not has_json_index(tmp_path)++ restored = _model(llm_client, embedder, tmp_path)+ with caplog.at_level(logging.WARNING):+ restored.load_index(tmp_path)++ assert restored._index is not None+ assert any(+ "deserialization" in r.getMessage().lower() for r in caplog.records+ ), "legacy load must warn about deserialization"++ def test_json_preferred_over_legacy(self, llm_client, embedder, tmp_path, caplog):+ ka = _model(llm_client, embedder, tmp_path)+ ka.build_index()+ ka._index.save_local(tmp_path) # legacy files present...+ ka.dump_index(tmp_path) # ...but dump removes them and writes JSON++ caplog.clear()+ restored = _model(llm_client, embedder, tmp_path)+ restored.load_index(tmp_path)+ assert not any(+ "deserialization" in r.getMessage().lower() for r in caplog.records+ ), "JSON load must not trigger the legacy deserialization warning"++ def test_missing_folder_raises(self, llm_client, embedder, tmp_path):+ ka = _model(llm_client, embedder, tmp_path)+ with pytest.raises(ValueError, match="does not exist"):+ ka.load_index(tmp_path / "nope")+++class TestHelpers:+ def test_load_index_json_returns_none_without_json(self, tmp_path, embedder):+ tmp_path.mkdir(exist_ok=True)+ assert load_index_json(tmp_path, embedder) is None++ def test_corrupt_json_raises(self, tmp_path, embedder):+ (tmp_path / "index.json").write_text('{"version": 1}', encoding="utf-8")+ with pytest.raises(ValueError, match="Corrupt"):+ load_index_json(tmp_path, embedder)++ def test_dump_index_json_round_trip_helper(self, tmp_path, embedder):+ idx = FAISS.from_documents(+ [Document(page_content="f", metadata={"raw": {"k": "v"}})], embedder+ )+ path = dump_index_json(idx, tmp_path)+ assert path.name == "index.json"+ restored = load_index_json(tmp_path, embedder)+ assert restored is not None+ assert restored.index.ntotal == idx.index.ntotal