Stronger Judges Do Not See More: Deployment-Integrity Blindness in LLM Evaluation

An agent looks good in demos, but you cannot tell which product tasks it fails.

What you would change

Use the study's evaluation idea to inspect a small set of tasks with human-checked answers.

Use the study's evaluation idea to inspect a small set of tasks with human-checked answers.

Compare against

Your current evaluator and a manually reviewed reference set.

1. Locate the paper's implementation and check its license, model support, data needs, and hardware requirements. Stop here if you cannot obtain the required artifacts.
2. Save a small set of representative product tasks and expected outcomes. Keep evaluation tasks out of any training or tuning, and choose quality and cost limits before running.
3. Run the baseline and the proposed change on identical inputs. Keep other settings fixed and repeat paired runs to expose variation.
4. Save per-task outputs, failures, timings, and all usage costs. Review regressions before expanding the test.

What to measure

Agreement with human review, missed failures, false alarms, and evaluation cost per task.

When to keep it

Keep the change only if it meets your preselected quality and latency limits and improves the metric you care about after setup and operating costs. Otherwise retain the baseline. This is an exploration plan, not a validated recommendation.

recommendation

adopt

research area

agent evaluation

technical change

Deployment-identity manifest hashing for LLM evaluation

published

2026-10-15

Manifest hashing and deployment-identity descriptors are cheap, infrastructure-agnostic process changes that close a demonstrated blind spot in output-only evaluation, so a small team using APIs or a single GPU can adopt them immediately without any compute cost.

1. claim 01Paperraft inference
   Read the brief interpretation

   The paper's proposed practice replaces output-only evaluation reporting with a graded deployment-identity descriptor plus a manifest hash of the runtime stack (model, weights, tokenizer, prompts, decoding config, tools) attached to evaluation reports. Cost is near-zero in compute and latency; the burden is added process complexity: generating, versioning, and verifying stack manifests in the CI/eval pipeline. It can fail because hashing proves identity but not correctness of the manifest itself, fingerprinting covers only the model component (one of six), trigger-conditional tampers can still evade any fixed evaluation suite, and an adversary controlling the reporting channel can forge descriptors unless hashes are anchored to a trusted source.

   This is a Paperraft inference. It is not pinned to a verified PDF passage.

   Inspect the source
2. claim 02Paperraft inference

   Stronger output-only judges accept more tampered deployments (88.6% rule-based vs 94.1% GPT-5-nano acceptance); zero-output-shift tampers are provably undetectable from outputs at any sample size, while a manifest-hash identity channel detects them with zero samples; behavioral fingerprinting identifies the model component at 76.7% but only one of six stack components.

   This is a Paperraft inference. It is not pinned to a verified PDF passage.

   Inspect the source