MemCalib: Benchmarking and Optimizing Memory Use in LLM Agents

A support or data assistant repeats work across related customer requests.

What you would change

Try the paper's memory change in a replay of past tasks, with separate memory per customer.

Try the paper's memory change in a replay of past tasks, with separate memory per customer.

Compare against

Your current memory pipeline plus a no-memory control using the same agent.

1. Locate the paper's implementation and check its license, model support, data needs, and hardware requirements. Stop here if you cannot obtain the required artifacts.
2. Save a small set of representative product tasks and expected outcomes. Keep evaluation tasks out of any training or tuning, and choose quality and cost limits before running.
3. Run the baseline and the proposed change on identical inputs. Keep other settings fixed and repeat paired runs to expose variation.
4. Save per-task outputs, failures, timings, and all usage costs. Review regressions before expanding the test.

What to measure

Task success, stale memories, retrieval misses, response latency, and all memory maintenance costs.

When to keep it

Keep the change only if it meets your preselected quality and latency limits and improves the metric you care about after setup and operating costs. Otherwise retain the baseline. This is an exploration plan, not a validated recommendation.

recommendation

watch

research area

memory and context

technical change

Bidirectional counterfactual RL for calibrated agent memory use

published

2026-09-21

The core contribution requires RL post-training of mid-scale models that is infeasible on the reader's constrained infrastructure, so the practical value lies in tracking the MemCalib benchmark as an evaluation tool rather than adopting the training method now.

1. claim 01Paperraft inference
   Read the brief interpretation

   MemCalib-RL replaces generic post-training (GRPO, on-policy self-distillation) for memory-use behavior with an ordered bidirectional counterfactual credit-assignment algorithm that separates over-use and under-use signals and localizes credit via exact atom ablation. It costs a full RL training run on 8B to 35B-parameter models, which exceeds a single 24 GB GPU and requires counterfactual ablation infrastructure and training data; API-only users cannot apply it. After adoption it can fail if the calibrated behavior does not transfer to the deployed memory system or if the residual directional skew between over-use and under-use persists on out-of-distribution tasks.

   This is a Paperraft inference. It is not pinned to a verified PDF passage.

   Inspect the source
2. claim 02Paperraft inference

   MemCalib-RL achieves the best overall performance on MemCalib across Qwen3-8B, Ministral-3-8B, and Qwen3.5-35B-A3B while better balancing over-use and under-use of memory, with gains generalizing to external benchmarks; no multiplicative factor is reported in the abstract.

   This is a Paperraft inference. It is not pinned to a verified PDF passage.

   Inspect the source