OnTrack: Real-Time Monitoring and Intervention in LLM Agent Trajectories via Streaming Structure-Aware Optimal Transport

Tool errors or unavailable services cause your agent to abandon recoverable tasks.

What you would change

Inject repeatable tool failures and change only the recovery policy.

Inject repeatable tool failures and change only the recovery policy.

Compare against

Your current retry and fallback behavior under the same injected failures.

1. Locate the paper's implementation and check its license, model support, data needs, and hardware requirements. Stop here if you cannot obtain the required artifacts.
2. Save a small set of representative product tasks and expected outcomes. Keep evaluation tasks out of any training or tuning, and choose quality and cost limits before running.
3. Run the baseline and the proposed change on identical inputs. Keep other settings fixed and repeat paired runs to expose variation.
4. Save per-task outputs, failures, timings, and all usage costs. Review regressions before expanding the test.

What to measure

Recovered tasks, duplicate side effects, retries, time to recovery, and total cost.

When to keep it

Keep the change only if it meets your preselected quality and latency limits and improves the metric you care about after setup and operating costs. Otherwise retain the baseline. This is an exploration plan, not a validated recommendation.

recommendation

test

research area

failure recovery

technical change

Streaming OT-based agent trajectory monitoring

published

2026-10-08

The method runs in milliseconds on CPU alongside a single-GPU or API-based agent and directly addresses wasted compute on failing runs, but its 18% saving and AUROC gain come from one benchmark and must be validated on the reader's own trajectory distribution before production adoption.

1. claim 01Paperraft inference
   Read the brief interpretation

   OnTrack replaces per-step safeguard-agent calls (which add cost and latency to every action) and post-hoc log evaluation (which only reports after tokens are spent) with a streaming structure-aware optimal-transport comparison of live steps against reference trajectories or tool schemas, at roughly one millisecond per step. The cost is maintaining reference runs or tool schemas, integrating a blocking/alerting hook into the agent loop, and accepting degraded capability when access drops to raw logs only (loop, stall, and repeat detection rather than plan-violation detection). It can fail through false aborts on valid but atypical trajectories, missed failures when the reference set does not cover the current task distribution, and the approximately 17% incorrect-intervention rate observed in the reported abort policy.

   This is a Paperraft inference. It is not pinned to a verified PDF passage.

   Inspect the source
2. claim 02Paperraft inference

   Aborting runs flagged as failing saves about 18% of compute on SWE-bench trajectories, with 5 of 6 aborts (83%) being correct; ranking failing below succeeding trajectories improves by +0.057 AUROC over content-similarity baselines using the first 8 steps.

   This is a Paperraft inference. It is not pinned to a verified PDF passage.

   Inspect the source