When Agents Slow Down: Understanding LLM Agents' Test-Time Strategies via Elo-per-token Analysis
The method replaces uninstrumented agent budget allocation and single-score leaderboards with a per-token best-solution curve aggregated across tasks via a Bradley-Terry Elo model, using independent sampling as a reference to locate the point where marginal gains diminish. It costs additional evaluation infrastructure: tasks must expose continuous intermediate scores, trajectories must be logged per token, and Elo estimation plus reference sampling add compute and engineering effort on top of the agent runs themselves. It can fail when benchmarks lack intermediate scoring, when the independent-sampling reference is unrepresentative of the task, or when the empirically estimated inflection point does not transfer across models, tasks, or budget regimes.
