Agent Error Dataset: Scaling 50,000 Error--Diagnosis Pairs for Failure Analysis and Error-Aware Post-Training

Tool errors or unavailable services cause your agent to abandon recoverable tasks.

What you would change

Inject repeatable tool failures and change only the recovery policy.

Inject repeatable tool failures and change only the recovery policy.

Compare against

Your current retry and fallback behavior under the same injected failures.

1. Locate the paper's implementation and check its license, model support, data needs, and hardware requirements. Stop here if you cannot obtain the required artifacts.
2. Save a small set of representative product tasks and expected outcomes. Keep evaluation tasks out of any training or tuning, and choose quality and cost limits before running.
3. Run the baseline and the proposed change on identical inputs. Keep other settings fixed and repeat paired runs to expose variation.
4. Save per-task outputs, failures, timings, and all usage costs. Review regressions before expanding the test.

What to measure

Recovered tasks, duplicate side effects, retries, time to recovery, and total cost.

When to keep it

Keep the change only if it meets your preselected quality and latency limits and improves the metric you care about after setup and operating costs. Otherwise retain the baseline. This is an exploration plan, not a validated recommendation.

recommendation

test

research area

failure recovery

technical change

Error-diagnosis pairs for agent failure recovery post-training

published

2026-09-30

Fine-tuning an 8B model on a public error dataset is feasible on constrained infrastructure, but the single-seed actor comparison and dependence on replay support require small-scale validation before production adoption.

1. claim 01Paperraft inference
   Read the brief interpretation

   The method replaces discarding failed agent rollouts or training only on successful trajectories with a pipeline that converts failures into error-diagnosis and correction pairs for fine-tuning. Cost is pipeline complexity: it requires multi-model trace collection, LLM-generated diagnoses verified against recorded evidence, and replay infrastructure, plus separate fine-tuning runs that fit a 24 GB GPU only at the 8B scale demonstrated. It can fail when replay is unsupported in a given environment, when LLM-generated diagnoses are mislabeled against the recorded evidence, and when gains do not transfer beyond the specific harness families and benchmarks evaluated.

   This is a Paperraft inference. It is not pinned to a verified PDF passage.

   Inspect the source
2. claim 02Paperraft inference

   First-proposal corrections raise verifier pass rates from 18.4% to 51.1% (+32.7 points) over 3,062 matched replay pairs; diagnosis fine-tuning raises Qwen3-8B teacher-label agreement from 47.2% to 63.6%; action-only repair training beats success-only training by 6.67 points on WebShop-lite (single seed).

   This is a Paperraft inference. It is not pinned to a verified PDF passage.

   Inspect the source