Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.
Agent Memory Leaderboard adapter
This directory contains three deliberately separated stages:
server.pyexposes the competition-facing AML Add/Search API;eval/run_public.pyreads the already prepared datasets underresult/aml_dataand produces retrieval results;eval/evaluate.pyoptionally runs each dataset's public Answer/Judge pipeline after retrieval, including deterministic PersonaMem MCQ grading.
The deployed adapter never imports the local evaluation code. In a formal AML run, the platform supplies the datasets and controls Answer, Judge, and score aggregation.
Add/Search adapter
For a local OpenViking server using server.auth_mode: dev:
export OPENVIKING_URL=http://127.0.0.1:1933
export AML_API_KEY='<memory-system key>'
python -m benchmark.aml.server --host 127.0.0.1 --port 8088
Do not set an OpenViking key or account for this local dev setup. The SDK
passes each AML user_id directly as the OpenViking user; there is no hash or
second identity mapping. Like other OpenViking SDK clients, it also reads an
existing ~/.openviking/ovcli.conf, so that file must match the server being
used (or be disabled with OPENVIKING_CLI_CONFIG_FILE pointing to a missing
file).
For a remote multi-user deployment, configure the OpenViking server with
server.auth_mode: trusted, then set the trusted-mode root key and account:
export OPENVIKING_URL='https://openviking.example.com'
export OPENVIKING_API_KEY='<trusted-mode root key>'
export OPENVIKING_ACCOUNT=default
export AML_API_KEY='<memory-system key>'
python -m benchmark.aml.server --host 0.0.0.0 --port 8088
OpenViking's standard api_key mode cannot use one root key to switch among
dynamic AML users: in that mode a request must use the key belonging to the
target user. AML_API_KEY is unrelated; it only protects this adapter's
incoming /add and /search endpoints.
The adapter provides:
GET /healthPOST /addPOST /search
/add calls the OpenViking SDK's batch_add_messages, commit_session, and
get_task in that order. /search calls OpenViking find, restricted to
viking://~/memories and level 2 under the same AML user. Each dataset case is
isolated by its user_id; session IDs only separate histories inside that
user's space. Transient SDK calls are retried up to three times by
default; use AML_RETRY_ATTEMPTS and AML_RETRY_DELAY_SECONDS to adjust that
behavior. Token, Bearer, and X-Api-Key authentication are accepted when
AML_API_KEY is set.
/add accepts either a content string (Textual/Coding) or an ordered array of
text and image_url parts in messages[].content (Multimodal). Strings are
forwarded as native content; arrays are forwarded as native Session parts,
preserving part order, message roles, and timestamps. Images must be inline
Base64 JPEG, PNG, or WebP Data URIs, with at most 10 MiB decoded per image and
30 MiB decoded across one Add request. For example:
{
"request_id": "example:add:1",
"user_id": "example:user",
"session_id": "example:session",
"messages": [{
"role": "user",
"content": [
{"type": "text", "text": "Remember this receipt."},
{"type": "image_url", "image_url": {"url": "data:image/png;base64,<image-bytes>"}}
]
}]
}
The example's image payload is a placeholder; supply actual Base64 image bytes. Image understanding and memory extraction use OpenViking's existing Session commit pipeline and require a configured vision-capable VLM. The adapter does not perform separate image captioning.
/search accepts a query string or an array of the same text and image_url
parts, including image-only and multiple-image queries. Text parts are joined
with newlines and image URLs retain their relative order. The adapter forwards
them to native find using query and image_url, without an extra VLM call.
String queries keep their original request and response behavior. For example:
{
"user_id": "example:user",
"query": [
{"type": "text", "text": "Which workshop was this receipt for?"},
{"type": "image_url", "image_url": {"url": "data:image/png;base64,<image-bytes>"}}
],
"top_k": 5
}
The same 10 MiB per-image limit applies, with a 30 MiB decoded total per Search request. Image queries require a multimodal dense embedding model compatible with the stored vectors. In hybrid mode, OV embeds the complete image/text input through dense and only the text through sparse; image-only queries skip sparse. Volcengine text-only queries retain their combined dense/sparse request. Keep the configured dense model and dimension consistent with the existing index. Search responses remain text memory evidence rather than raw images.
The native find call uses quick retrieval without reranking, even when the
OpenViking server has a reranker configured. The adapter returns each hit's
content when present, otherwise its abstract, without an additional
full-text read.
Run these modules from a source checkout with the project's Python dependencies
installed. The standard OpenViking image and installed package do not include
benchmark/aml.
Local Add/Search
The input files already exist locally and are not generated by this code:
python -m benchmark.aml.eval.run_public \
--prepared result/aml_data/prepared/lme_locomo_script.json \
--prepared result/aml_data/prepared/persona_cl.json \
--prepared result/aml_data/prepared/beam.json \
--namespace local-check-v1 \
--base-url http://127.0.0.1:8088 \
--output-dir result/aml-public/retrieval \
--add-concurrency 32 --search-concurrency 16
The main flow stops after Add/Search and writes one retrieval.jsonl under
each runnable dataset directory, plus a small top-level summary.json. Empty
private-data declarations are ignored because they contain nothing that can be
sent to Add or Search.
The prepared files contain history_key, not an AML user_id. The local
runner therefore creates one stable AML user for each unique
namespace + unit + history_key + history contents: questions sharing a
history share that user, while different histories stay isolated. This is ID
creation in the local runner; server.py forwards the resulting ID unchanged.
Use repeated --unit arguments to select individual datasets. Add runs up to
16 histories (AML users) concurrently by default, while Search runs up to 16 questions
concurrently. Override them independently with --add-concurrency and
--search-concurrency; the existing --concurrency option remains a shorthand
baseline for both. Phase-specific options take precedence. Within each history,
sessions run in prepared order, and chunks within each session run in order.
Every Add waits for its commit to finish before the next request for that user;
Search starts after all histories finish importing. For BEAM, this preserves
batch order within a conversation while different conversations run concurrently.
For smoke/full presets, run_ecs accepts a provider URL through --base-url
or AML_BASE_URL; it defaults to the local adapter. It reads authentication
from AML_API_KEY or --api-key-file and creates a fresh namespace per run:
python -m benchmark.aml.eval.run_ecs list
python -m benchmark.aml.eval.run_ecs full beam_100k \
--base-url http://127.0.0.1:8088 --add-concurrency 32 --dry-run
Remove --dry-run to import. The generic callers do not resume an interrupted
import. Reconcile the original server tasks and archives before retrying an
ambiguous Add; a new namespace starts a separate experiment.
PersonaMem-v2 text preparation
eval/prepare_personamem.py follows the exact chat_history_32k_link and
chat_history_128k_link paths in benchmark/text/benchmark.csv. Each size is a
separate condition; their histories are not concatenated. Shared histories
are imported once per run and reused by their questions.
The preparer removes all system persona messages and blank messages, preserves the remaining user/assistant text and order, and omits timestamps absent from the source. It does not assemble histories from raw persona records or gold snippets. Reference answers, preference annotations and distractors stay in case metadata and never enter Add messages.
python -m benchmark.aml.eval.prepare_personamem \
--source-dir result/aml_data/datasets/personamem_v2 \
--verification result/aml_data/personamem_v2_download/verification.json \
--output-dir result/aml_data/prepared/full
The downloaded source and its verification manifest must already exist. The
manifest contains success: true, the source revision, and a files array
with relative path and sha256 for the CSV and linked chat files. Preparation
checks those hashes, writes both size-specific prepared files and backs up any
replaced outputs. Optional --smoke-file refreshes existing 32K smoke cases.
Preparation makes no Add/Search or model calls and covers text inputs only.
Optional Answer/Eval
The optional evaluator consumes one completed retrieval artifact and never
repeats Add or Search. It reads the artifact's unit and dispatches to the
matching public AML pipeline for LongMemEval-S, LoCoMo-Refined, BEAM,
CLBench, or PersonaMem-v2:
export ANSWER_API_BASE='https://answer-endpoint/v1'
export ANSWER_API_KEY='...'
export ANSWER_MODEL='...'
export JUDGE_API_BASE='https://judge-endpoint/v1'
export JUDGE_API_KEY='...'
export JUDGE_MODEL='...'
python -m benchmark.aml.eval.evaluate \
--retrieval result/aml-public/retrieval/longmemeval_s/retrieval.jsonl \
--aml-repo result/aml-public-eval/agent-memory-leaderboard \
--answer-concurrency 16 --eval-concurrency 32
This produces answers.jsonl, judgments.jsonl, and summary.json beside the
retrieval file. Answer prompts and scoring stay inside the pinned public
dataset pipeline; evaluate.py only selects it and adapts the generic
retrieved_context field where CLBench or PersonaMem expects a different input
shape. Answer and Judge each run up to 32 pipeline shards concurrently by
default; use --answer-concurrency and --eval-concurrency to tune them
independently. Outputs are merged back into retrieval order and published only
after every shard succeeds. These remain local proxy results rather than
official leaderboard scores. Model options and retries follow the selected
public pipeline; this wrapper does not provide the experiment-specific pacing
or resume logic. The pinned pipelines currently combine
an asynchronous HTTP client and a regular output file in one async with
statement; the wrapper corrects only that context-manager syntax in a temporary
runtime copy, leaving the verified checkout, prompts, and scoring code unchanged.
PersonaMem defaults to --persona-mode mcq, even for older retrieval files
marked as generative. Answer receives only retrieved memory, the question and
four shuffled choices from correct_answer plus incorrect_answers; preference
labels and the correct option marker are not added to the prompt. Search does
not receive these choices. The public pipeline maps the predicted letter to
its option text and grades it locally; invalid answers count as incorrect.
Only the three ANSWER_* variables are required for MCQ. Use
--persona-mode generative explicitly for the public narrow LLM judge instead.
Subprocesses default to PYTHONHASHSEED=0 to keep option mappings stable across
shard counts. These are OV retrieval-based evaluations, not reproductions of
the authors' trained MemAgent implementation.
Static checks
python -m ruff check benchmark/aml
python -m ruff format --check benchmark/aml