TRACE: A Self-Evolving Skill Bank for Consistent, Limit-Aware LLM Agents

A support or data assistant repeats work across related customer requests.

What you would change

Try the paper's memory change in a replay of past tasks, with separate memory per customer.

Try the paper's memory change in a replay of past tasks, with separate memory per customer.

Compare against

Your current memory pipeline plus a no-memory control using the same agent.

1. Locate the paper's implementation and check its license, model support, data needs, and hardware requirements. Stop here if you cannot obtain the required artifacts.
2. Save a small set of representative product tasks and expected outcomes. Keep evaluation tasks out of any training or tuning, and choose quality and cost limits before running.
3. Run the baseline and the proposed change on identical inputs. Keep other settings fixed and repeat paired runs to expose variation.
4. Save per-task outputs, failures, timings, and all usage costs. Review regressions before expanding the test.

What to measure

Task success, stale memories, retrieval misses, response latency, and all memory maintenance costs.

When to keep it

Keep the change only if it meets your preselected quality and latency limits and improves the metric you care about after setup and operating costs. Otherwise retain the baseline. This is an exploration plan, not a validated recommendation.

recommendation

test

research area

memory and context

technical change

Self-evolving retrievable skill bank for agents

published

2026-08-24

The weight-free, API-compatible design fits constrained infrastructure and the consistency problem is real in production agents, but the evidence is single-domain and benchmark-specific, so a small-scale pilot on the reader's own tool-use workload is warranted before adoption.

equations from arXiv HTML · fetched 2026-09-19

1. claim 01Paperraft inference
   Read the brief interpretation

   TRACE replaces static hand-written system prompts and agent policies with a Skill Bank of modular, retrievable tool-use rules that is iteratively refined by contrasting successful and failed trajectories, without modifying model weights. It costs an offline evolution loop of repeated evaluation rounds and trajectory analysis (API tokens and engineering effort), plus per-turn retrieval and state-conditioned skill orchestration that add latency and prompt complexity at deployment. It can fail when the contrastive refinement overfits to evaluation trajectories, when skills conflict or are mis-retrieved for out-of-distribution requests, and its gains are demonstrated only on a single in-car assistant benchmark with specific frontier models, so transfer to other domains is unverified.

   This is a Paperraft inference. It is not pinned to a verified PDF passage.

   Inspect the source
2. claim 02Paperraft inference

   On CAR-bench with GPT-5.5, TRACE raises Pass^3 consistency from 59.9% to 94.5% (+34.6 points); on the hidden set it reaches 70% Pass^3 with GPT-5.6-Sol, a 40% relative gain over baseline.

   This is a Paperraft inference. It is not pinned to a verified PDF passage.

   Inspect the source