The Tasteful Agent: Measuring and Improving Taste in Long-Horizon Tasks

An agent looks good in demos, but you cannot tell which product tasks it fails.

What you would change

Use the study's evaluation idea to inspect a small set of tasks with human-checked answers.

Use the study's evaluation idea to inspect a small set of tasks with human-checked answers.

Compare against

Your current evaluator and a manually reviewed reference set.

1. Locate the paper's implementation and check its license, model support, data needs, and hardware requirements. Stop here if you cannot obtain the required artifacts.
2. Save a small set of representative product tasks and expected outcomes. Keep evaluation tasks out of any training or tuning, and choose quality and cost limits before running.
3. Run the baseline and the proposed change on identical inputs. Keep other settings fixed and repeat paired runs to expose variation.
4. Save per-task outputs, failures, timings, and all usage costs. Review regressions before expanding the test.

What to measure

Agreement with human review, missed failures, false alarms, and evaluation cost per task.

When to keep it

Keep the change only if it meets your preselected quality and latency limits and improves the metric you care about after setup and operating costs. Otherwise retain the baseline. This is an exploration plan, not a validated recommendation.

recommendation

test

research area

agent evaluation

technical change

Taste-Bench benchmark and outcome-conditioned taste distillation

published

2026-09-22

The fork-mining evaluation idea is cheap to replicate and directly useful for the reader's agent work, but the demonstrated gains require outcome-labeled trajectory generation and distillation whose benefit on the reader's specific workloads is unverified, warranting a small-scale pilot rather than adoption.

1. claim 01Paperraft inference
   Read the brief interpretation

   The paper replaces end-to-end agent success metrics with a benchmark of automatically mined decision forks from parallel attempts and detours, plus a distillation method that replaces ad hoc prompting of long-horizon decisions with a student trained on a teacher that has seen outcomes. The benchmark itself costs only trajectory collection and comparison (no human annotation), but the training component requires generating parallel rollouts, teacher labeling with outcome access, and fine-tuning, which is feasible on a 24 GB GPU only for smaller students and is more realistically done via API-based teachers plus local fine-tuning. Findings that later-evidence forks are hard for all models and that larger reasoning budgets do not help indicate the capability gap will not close with inference-time scaling, and distillation can fail if teacher outcomes are noisy proxies for decision quality, if mined forks correlate with spurious cues, or if gains do not transfer beyond SWE-bench-like coding tasks.

   This is a Paperraft inference. It is not pinned to a verified PDF passage.

   Inspect the source
2. claim 02Paperraft inference

   Best frontier model answers 59.7% of Taste-Bench decision-fork questions correctly; distilling an outcome-aware teacher's judgment into a student improves the student's decisions on unseen tasks and its end-to-end success on held-out SWE-bench Pro tasks, with no multiplicative factor reported.

   This is a Paperraft inference. It is not pinned to a verified PDF passage.

   Inspect the source