MemoryAgentBench: LLM Agent Memory Evaluation with Under 1,000 Samples

MemoryAgentBench is a benchmark for evaluating LLM agent memory and retrieval capabilities, released by ai-hyz in July 2025. It contains fewer than 1,000 samples across specialized tasks including DetectiveQA, Longmemeval, and LRU-based keypoint tracking. The dataset was iteratively refined through October 2025 to remove high-cost and inefficient evaluation samples.

Use Cases

• Measuring retrieval accuracy in memory agents using qa_pair_ids
• Evaluating long-context reasoning through the Longmemeval task
• Testing memory prioritization logic using keypoints in the LRU task

Strengths

• MIT licensed for open research use
• Includes specialized DetectiveQA sub-samples for complex reasoning
• Refined to remove high-cost and inefficient samples as of September 2025

Limitations

• Small sample size of under 1,000 records
• Task-specific identifiers like question_ids are restricted to the Longmemeval subset
• Geographic bias toward the US region as indicated by metadata tags

Provenance

Source

Associated with Arxiv paper 2507.05257

Collection Method

Expert-annotated and curated benchmark tasks

Time Range

2025

Freshness

Last updated in October 2025 following a series of refinements to the DetectiveQA and Longmemeval subsets.

Geography

United States