Skip to content

Hypergraph is more powerful. Transform unstructured text into structured knowledge with LLMs. Graphs, hypergraphs, and spatio-temporal extractions — with one command.

feat: pickle-free JSON index storage for AutoModel/AutoList (#116)

yifanfeng97a408083b59fc
diff --git a/README.md b/README.md
index bcf7cb5..504400d 100644
--- a/README.md
+++ b/README.md
@@ -342,7 +342,7 @@ identifiers:
## 📰 What's New
-**v0.10.0** — 📤 New exports: JSON-LD & Cypher (Neo4j) alongside GraphML/CSV/Obsidian — all four also available via MCP · 📄 `he feed` now accepts directories · 🧠 Native Gemini support (`pip install "hyperextract[google]"`) · 🛡️ Windows config hardening & scoped-search fixes across every KA type.
+**v0.10.1** — 🔒 Pickle-free index storage: `AutoModel`/`AutoList` indexes are JSON now — legacy pickle indexes still load with a warning (`he build-index --force` migrates). *(#116)*
📰 **[Full release notes](https://yifanfeng97.github.io/Hyper-Extract/latest/news/)** · [All releases](https://github.com/yifanfeng97/hyper-extract/releases)
diff --git a/README_ZH.md b/README_ZH.md
index b8bd877..2eb25de 100644
--- a/README_ZH.md
+++ b/README_ZH.md
@@ -342,7 +342,7 @@ identifiers:
## 📰 最新动态
-**v0.10.0** — 📤 新增导出:JSON-LD 与 Cypher(Neo4j),连同 GraphML/CSV/Obsidian——四种格式均可经 MCP 调用 · 📄 `he feed` 现在支持目录 · 🧠 原生 Gemini 支持(`pip install "hyperextract[google]"`)· 🛡️ Windows 配置加固与全部知识库类型的范围检索修复。
+**v0.10.1** — 🔒 Pickle-free index storage: `AutoModel`/`AutoList` indexes are JSON now — legacy pickle indexes still load with a warning (`he build-index --force` migrates). *(#116)*
📰 **[完整版本说明](https://yifanfeng97.github.io/Hyper-Extract/latest/zh/news/)** · [全部 Releases](https://github.com/yifanfeng97/hyper-extract/releases)
diff --git a/docs/en/cli/commands/build-index.md b/docs/en/cli/commands/build-index.md
index a2a798c..39953e1 100644
--- a/docs/en/cli/commands/build-index.md
+++ b/docs/en/cli/commands/build-index.md
@@ -116,7 +116,7 @@ The index is stored in the knowledge abstract directory:
└── docstore.json # Document store mapping
```
-Load an existing `index/` only from Knowledge Abstract directories you created or fully trust. Local FAISS indexes require deserialization on load.
+Load an existing `index/` only from Knowledge Abstract directories you created or fully trust. Since v0.10.1 indexes are JSON (safe to load); legacy pickle indexes (<= v0.10.0) are deserialized on load — rebuild with `--force` to migrate.
---
diff --git a/docs/en/news.md b/docs/en/news.md
index b4ba50f..15c72e6 100644
--- a/docs/en/news.md
+++ b/docs/en/news.md
@@ -4,6 +4,12 @@ Release notes and highlights. For a complete changelog, see the [GitHub releases
---
+## v0.10.1 — Pickle-Free Index Storage
+
+- **🔒 Safe index format** — `AutoModel`/`AutoList` indexes are now stored as JSON (`index.json`: vectors + documents) and rebuilt in memory on load — **no pickle, no code execution**. Legacy pickle indexes (<= v0.10.0) still load with a deserialization warning; `he build-index --force` migrates them. *(#116)*
+
+---
+
## v0.10.0 — More Export Formats, Gemini, and Quality Hardening
- **📤 JSON-LD & Cypher exports** — `he export jsonld` (pairwise edges + GraphML-style hyperedges) and `he export cypher` (Neo4j-compatible MERGE script; N-ary edges become Hyperedge nodes, not lossy cliques). *(#139, #141)*
diff --git a/docs/en/python/guides/saving-loading.md b/docs/en/python/guides/saving-loading.md
index 1e1cf8d..19f5923 100644
--- a/docs/en/python/guides/saving-loading.md
+++ b/docs/en/python/guides/saving-loading.md
@@ -54,7 +54,7 @@ result.dump("./my_ka/")
## Loading Knowledge Abstracts
!!! warning "Trusted directories only"
- Local FAISS indexes under `index/` are deserialized when you load a Knowledge Abstract. Only call `load()` (or CLI/MCP commands that load a KA) on directories you generated yourself or fully trust.
+ Since v0.10.1, indexes are stored as JSON (`index.json`) and load without code execution. Legacy KAs (<= v0.10.0) keep pickle-based indexes, which ARE deserialized on load — only call `load()` (or CLI/MCP commands that load a KA) on directories you generated yourself or fully trust. Rebuild with `he build-index --force` to migrate a legacy KA.
### Basic Load
diff --git a/docs/zh/cli/commands/build-index.md b/docs/zh/cli/commands/build-index.md
index 1d51c1b..53373b7 100644
--- a/docs/zh/cli/commands/build-index.md
+++ b/docs/zh/cli/commands/build-index.md
@@ -116,7 +116,7 @@ he build-index ./ka/ -f
└── docstore.json # 文档存储映射
```
-请只加载自己生成或完全信任的知识库目录中的 `index/`。本地 FAISS 索引在加载时需要反序列化。
+请只加载自己生成或完全信任的知识库目录中的 `index/`。自 v0.10.1 起索引为 JSON(加载安全);旧版 pickle 索引(<= v0.10.0)加载时会反序列化——用 `--force` 重建即可迁移。
---
diff --git a/docs/zh/news.md b/docs/zh/news.md
index 96df3ef..bff2878 100644
--- a/docs/zh/news.md
+++ b/docs/zh/news.md
@@ -4,6 +4,12 @@
---
+## v0.10.1 — 索引存储脱离 pickle
+
+- **🔒 安全的索引格式** — `AutoModel`/`AutoList` 的索引改为 JSON 存储(`index.json`:向量 + 文档),加载时在内存中重建——**没有 pickle,不执行任何代码**。旧 pickle 索引(<= v0.10.0)仍可加载但会给出警告;`he build-index --force` 即可迁移。*(#116)*
+
+---
+
## v0.10.0 — 更多导出格式、Gemini 与质量加固
- **📤 JSON-LD 与 Cypher 导出** — `he export jsonld`(二元边 + GraphML 风格超边)与 `he export cypher`(Neo4j 兼容 MERGE 脚本;N 元边编码为 Hyperedge 节点,不做有损团 expansion)。*(#139, #141)*
diff --git a/docs/zh/python/guides/saving-loading.md b/docs/zh/python/guides/saving-loading.md
index 191bc50..8ae3f71 100644
--- a/docs/zh/python/guides/saving-loading.md
+++ b/docs/zh/python/guides/saving-loading.md
@@ -53,7 +53,7 @@ result.dump("./my_ka/")
## 加载知识库
!!! warning "只加载受信任的目录"
- 加载知识库时,`index/` 下的本地 FAISS 索引需要反序列化。请只对自己生成或完全信任的知识库目录调用 `load()`(以及会加载 KA 的 CLI/MCP 命令)。
+ 自 v0.10.1 起,索引以 JSON(`index.json`)存储,加载时不执行任何代码。旧版本(<= v0.10.0)的 KA 仍是 pickle 索引,加载时会反序列化——请只对自己生成或完全信任的知识库目录调用 `load()`(以及会加载 KA 的 CLI/MCP 命令)。用 `he build-index --force` 重建即可迁移旧 KA。
### 基本加载
diff --git a/hyperextract/types/list.py b/hyperextract/types/list.py
index c6a5bee..aa8c1c3 100644
--- a/hyperextract/types/list.py
+++ b/hyperextract/types/list.py
@@ -19,6 +19,7 @@ from langchain_core.language_models.chat_models import BaseChatModel
from ontosight import view_nodes
from pydantic import BaseModel, Field, create_model
+from hyperextract.utils.json_index import dump_index_json, load_index_json
from hyperextract.utils.logging import get_logger
from ._faiss import _warn_untrusted_faiss_load
@@ -311,16 +312,26 @@ class AutoList(BaseAutoType[AutoListSchema[ItemSchema]], Generic[ItemSchema]):
# ==================== Index Storage ====================
def dump_index(self, folder_path: str | Path) -> None:
- """Saves FAISS vector index to disk."""
+ """Saves the vector index as JSON (no pickle)."""
if self._index is None:
return
- self._index.save_local(folder_path)
+ dump_index_json(self._index, folder_path)
def load_index(self, folder_path: str | Path) -> None:
- """Loads FAISS vector index from disk."""
+ """Loads the vector index from disk.
+
+ JSON indexes (written by >=0.10.1) are loaded without any code
+ execution. Legacy pickle-based indexes still load with a
+ deserialization warning — rebuild with `he build-index --force`
+ to migrate them to JSON.
+ """
folder = Path(folder_path)
if not folder.is_dir():
raise ValueError(f"Folder does not exist: {folder_path}")
+ index = load_index_json(folder, self.embedder)
+ if index is not None:
+ self._index = index
+ return
_warn_untrusted_faiss_load(folder)
self._index = FAISS.load_local(
str(folder), self.embedder, allow_dangerous_deserialization=True
diff --git a/hyperextract/types/model.py b/hyperextract/types/model.py
index 308280e..86f2177 100644
--- a/hyperextract/types/model.py
+++ b/hyperextract/types/model.py
@@ -14,6 +14,7 @@ from langchain_core.language_models.chat_models import BaseChatModel
from ontomem.merger import BaseMerger, MergeStrategy, create_merger
from ontosight import view_nodes
+from hyperextract.utils.json_index import dump_index_json, load_index_json
from hyperextract.utils.logging import get_logger
from ._faiss import _warn_untrusted_faiss_load
@@ -302,16 +303,26 @@ class AutoModel(BaseAutoType[T]):
# ==================== Index Storage ====================
def dump_index(self, folder_path: str | Path) -> None:
- """Saves FAISS vector index to disk."""
+ """Saves the vector index as JSON (no pickle)."""
if self._index is None:
return
- self._index.save_local(folder_path)
+ dump_index_json(self._index, folder_path)
def load_index(self, folder_path: str | Path) -> None:
- """Loads FAISS vector index from disk."""
+ """Loads the vector index from disk.
+
+ JSON indexes (written by >=0.10.1) are loaded without any code
+ execution. Legacy pickle-based indexes still load with a
+ deserialization warning — rebuild with `he build-index --force`
+ to migrate them to JSON.
+ """
folder = Path(folder_path)
if not folder.is_dir():
raise ValueError(f"Folder does not exist: {folder_path}")
+ index = load_index_json(folder, self.embedder)
+ if index is not None:
+ self._index = index
+ return
_warn_untrusted_faiss_load(folder)
self._index = FAISS.load_local(
str(folder), self.embedder, allow_dangerous_deserialization=True
diff --git a/hyperextract/utils/json_index.py b/hyperextract/utils/json_index.py
new file mode 100644
index 0000000..87796fe
--- /dev/null
+++ b/hyperextract/utils/json_index.py
@@ -0,0 +1,135 @@
+"""JSON-backed FAISS index persistence (no pickle deserialization).
+
+Legacy ``FAISS.save_local`` writes ``index.faiss`` + ``index.pkl``; loading
+the pickle executes code inside it, so a KA directory is only as safe as
+its source. This module stores the raw vectors and documents as plain JSON
+and rebuilds the flat index in memory on load — same search behavior, zero
+code execution.
+
+Layout (``index.json``)::
+
+ {
+ "version": 1,
+ "metric": 1, # faiss metric_type of the saved index
+ "dimension": 768,
+ "vectors": [[...], ...], # one vector per item, index order
+ "texts": [...], # page_content per item
+ "metadatas": [...], # metadata per item
+ "ids": [...] # docstore ids per item
+ }
+
+Only flat indexes (as built by ``FAISS.from_documents`` with default
+settings) are supported — that is what AutoModel/AutoList build.
+"""
+
+import json
+from pathlib import Path
+from typing import Any
+
+import numpy as np
+from langchain_community.vectorstores import FAISS
+
+from hyperextract.utils.logging import get_logger
+
+logger = get_logger(__name__)
+
+INDEX_JSON_NAME = "index.json"
+LEGACY_INDEX_NAME = "index.faiss"
+LEGACY_PICKLE_NAME = "index.pkl"
+
+
+def has_json_index(folder: str | Path) -> bool:
+ """True when the folder contains a JSON index."""
+ return (Path(folder) / INDEX_JSON_NAME).is_file()
+
+
+def has_legacy_index(folder: str | Path) -> bool:
+ """True when the folder contains a legacy pickle-based FAISS index."""
+ folder = Path(folder)
+ return (folder / LEGACY_INDEX_NAME).is_file() and (
+ folder / LEGACY_PICKLE_NAME
+ ).is_file()
+
+
+def dump_index_json(index: FAISS, folder: str | Path) -> Path:
+ """Persist a FAISS vector index as JSON (vectors + documents, no pickle).
+
+ Returns the written ``index.json`` path. Legacy ``index.faiss`` /
+ ``index.pkl`` files in the folder are removed after a successful write
+ so a stale pickle index can never shadow the JSON one.
+ """
+ folder = Path(folder)
+ folder.mkdir(parents=True, exist_ok=True)
+
+ ntotal = int(index.index.ntotal)
+ vectors = (
+ index.index.reconstruct_n(0, ntotal)
+ if ntotal
+ else np.zeros((0, int(index.index.d)), dtype="float32")
+ )
+
+ items = []
+ for i in range(ntotal):
+ doc_id = index.index_to_docstore_id[i]
+ doc = index.docstore.search(doc_id)
+ items.append(
+ {
+ "id": doc_id,
+ "text": doc.page_content,
+ "metadata": doc.metadata,
+ }
+ )
+
+ payload = {
+ "version": 1,
+ "metric": int(index.index.metric_type),
+ "dimension": int(index.index.d),
+ "vectors": np.asarray(vectors, dtype="float32").tolist(),
+ "texts": [item["text"] for item in items],
+ "metadatas": [item["metadata"] for item in items],
+ "ids": [item["id"] for item in items],
+ }
+ path = folder / INDEX_JSON_NAME
+ path.write_text(json.dumps(payload, ensure_ascii=False), encoding="utf-8")
+ logger.info("json_index_dumped", path=str(path), items=ntotal)
+
+ for legacy in (LEGACY_INDEX_NAME, LEGACY_PICKLE_NAME):
+ stale = folder / legacy
+ if stale.exists():
+ stale.unlink()
+ logger.info("legacy_index_removed", path=str(stale))
+ return path
+
+
+def load_index_json(folder: str | Path, embedder: Any) -> FAISS | None:
+ """Load a JSON index and rebuild the FAISS flat index in memory.
+
+ Returns ``None`` when the folder has no ``index.json``. Vectors are
+ restored exactly as saved — no re-embedding happens.
+ """
+ path = Path(folder) / INDEX_JSON_NAME
+ if not path.is_file():
+ return None
+
+ payload = json.loads(path.read_text(encoding="utf-8"))
+ vectors = payload.get("vectors", [])
+ texts = payload.get("texts", [])
+ metadatas = payload.get("metadatas", [])
+ ids = payload.get("ids", [])
+ if not (len(vectors) == len(texts) == len(metadatas) == len(ids)):
+ raise ValueError(f"Corrupt index.json: field lengths differ in {path}")
+ if not vectors:
+ raise ValueError(f"Corrupt index.json: no entries in {path}")
+
+ metric = int(payload.get("metric", 0))
+ if metric: # only flat L2 indexes are written today
+ saved = metric
+ restored = 1 # faiss.METRIC_L2
+ if saved != restored:
+ logger.warning("json_index_metric_mismatch saved=%s rebuilt=L2", saved)
+
+ index = FAISS.from_embeddings(
+ list(zip(texts, vectors)), embedder, metadatas=metadatas, ids=ids
+ )
+ logger.info("json_index_loaded", path=str(path), items=len(ids))
+ return index
diff --git a/pyproject.toml b/pyproject.toml
index 20390fd..0d2d5fe 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -1,6 +1,6 @@
[project]
name = "hyperextract"
-version = "0.10.0"
+version = "0.10.1"
description = "An intelligent, LLM-powered knowledge extraction and evolution framework with semantic search capabilities"
readme = "README.md"
requires-python = ">=3.11"
diff --git a/tests/utils/test_json_index.py b/tests/utils/test_json_index.py
new file mode 100644
index 0000000..a419d29
--- /dev/null
+++ b/tests/utils/test_json_index.py
@@ -0,0 +1,172 @@
+"""JSON-backed FAISS index persistence for AutoModel/AutoList (#116).
+
+New dumps are plain JSON (no pickle, safe to load); legacy pickle indexes
+from older releases still load with a deserialization warning.
+"""
+
+import json
+import logging
+
+import pytest
+from langchain_community.vectorstores import FAISS
+from langchain_core.documents import Document
+from pydantic import BaseModel
+
+from hyperextract.types import AutoList, AutoModel
+from hyperextract.utils.json_index import (
+ dump_index_json,
+ has_json_index,
+ load_index_json,
+)
+
+
+class _City(BaseModel):
+ name: str
+ country: str = ""
+
+
+def _model(llm_client, embedder, tmp_path):
+ ka = AutoModel(
+ data_schema=_City,
+ llm_client=llm_client,
+ embedder=embedder,
+ )
+ # Seed the index directly: page_content is the field name, metadata
+ # carries the raw fields — same shape build_index() produces.
+ ka._index = FAISS.from_documents(
+ [
+ Document(page_content="name", metadata={"raw": {"name": "Paris"}}),
+ Document(
+ page_content="country",
+ metadata={"raw": {"name": "Paris", "country": "FR"}},
+ ),
+ ],
+ ka.embedder,
+ )
+ return ka
+
+
+def _list(llm_client, embedder):
+ return AutoList(
+ item_schema=_City,
+ llm_client=llm_client,
+ embedder=embedder,
+ )
+
+
+def _seed_list(ka):
+ ka._index = FAISS.from_documents(
+ [
+ Document(page_content="name", metadata={"raw": {"name": "Paris"}}),
+ Document(page_content="name", metadata={"raw": {"name": "Tokyo"}}),
+ ],
+ ka.embedder,
+ )
+
+
+class TestJsonRoundTrip:
+ def test_dump_writes_json_and_drops_legacy(self, llm_client, embedder, tmp_path):
+ ka = _model(llm_client, embedder, tmp_path)
+ # Simulate a legacy folder first
+ ka._index.save_local(tmp_path)
+ assert (tmp_path / "index.faiss").exists()
+
+ ka.dump_index(tmp_path)
+
+ assert has_json_index(tmp_path)
+ assert not (tmp_path / "index.faiss").exists()
+ assert not (tmp_path / "index.pkl").exists()
+
+ def test_model_round_trip_preserves_search(self, llm_client, embedder, tmp_path):
+ ka = _model(llm_client, embedder, tmp_path)
+ ka.build_index()
+ ka.dump_index(tmp_path)
+
+ restored = _model(llm_client, embedder, tmp_path)
+ restored.load_index(tmp_path)
+ # Compare at the index level: AutoModel.search() additionally needs
+ # extracted data, which this round-trip does not carry.
+ original_docs = ka._index.similarity_search("name", k=2)
+ restored_docs = restored._index.similarity_search("name", k=2)
+ assert [d.metadata for d in restored_docs] == [
+ d.metadata for d in original_docs
+ ]
+
+ def test_list_round_trip(self, llm_client, embedder, tmp_path):
+ ka = _list(llm_client, embedder)
+ _seed_list(ka)
+ ka.build_index()
+ ka.dump_index(tmp_path)
+
+ restored = _list(llm_client, embedder)
+ restored.load_index(tmp_path)
+ hits = restored.search("Paris", top_k=2)
+ assert {h.name for h in hits} <= {"Paris", "Tokyo"}
+
+ def test_json_load_is_picle_free_payload(self, llm_client, embedder, tmp_path):
+ ka = _model(llm_client, embedder, tmp_path)
+ ka.build_index()
+ ka.dump_index(tmp_path)
+ payload = json.loads((tmp_path / "index.json").read_text(encoding="utf-8"))
+ assert payload["version"] == 1
+ assert payload["dimension"] > 0
+ assert len(payload["vectors"]) == len(payload["texts"])
+
+
+class TestLegacyCompat:
+ def test_legacy_pickle_folder_still_loads(
+ self, llm_client, embedder, tmp_path, caplog
+ ):
+ ka = _model(llm_client, embedder, tmp_path)
+ ka.build_index()
+ # Simulate an old KA: raw save_local, no index.json
+ ka._index.save_local(tmp_path)
+ assert not has_json_index(tmp_path)
+
+ restored = _model(llm_client, embedder, tmp_path)
+ with caplog.at_level(logging.WARNING):
+ restored.load_index(tmp_path)
+
+ assert restored._index is not None
+ assert any(
+ "deserialization" in r.getMessage().lower() for r in caplog.records
+ ), "legacy load must warn about deserialization"
+
+ def test_json_preferred_over_legacy(self, llm_client, embedder, tmp_path, caplog):
+ ka = _model(llm_client, embedder, tmp_path)
+ ka.build_index()
+ ka._index.save_local(tmp_path) # legacy files present...
+ ka.dump_index(tmp_path) # ...but dump removes them and writes JSON
+
+ caplog.clear()
+ restored = _model(llm_client, embedder, tmp_path)
+ restored.load_index(tmp_path)
+ assert not any(
+ "deserialization" in r.getMessage().lower() for r in caplog.records
+ ), "JSON load must not trigger the legacy deserialization warning"
+
+ def test_missing_folder_raises(self, llm_client, embedder, tmp_path):
+ ka = _model(llm_client, embedder, tmp_path)
+ with pytest.raises(ValueError, match="does not exist"):
+ ka.load_index(tmp_path / "nope")
+
+
+class TestHelpers:
+ def test_load_index_json_returns_none_without_json(self, tmp_path, embedder):
+ tmp_path.mkdir(exist_ok=True)
+ assert load_index_json(tmp_path, embedder) is None
+
+ def test_corrupt_json_raises(self, tmp_path, embedder):
+ (tmp_path / "index.json").write_text('{"version": 1}', encoding="utf-8")
+ with pytest.raises(ValueError, match="Corrupt"):
+ load_index_json(tmp_path, embedder)
+
+ def test_dump_index_json_round_trip_helper(self, tmp_path, embedder):
+ idx = FAISS.from_documents(
+ [Document(page_content="f", metadata={"raw": {"k": "v"}})], embedder
+ )
+ path = dump_index_json(idx, tmp_path)
+ assert path.name == "index.json"
+ restored = load_index_json(tmp_path, embedder)
+ assert restored is not None
+ assert restored.index.ntotal == idx.index.ntotal