AfterDeleteBench: Auditing Post-Deletion Information Persistence in LLM Agents

An agent reads external documents or tool responses that may contain malicious instructions.

What you would change

Replay attacks and legitimate lookalikes in an isolated copy of your agent workflow.

Replay attacks and legitimate lookalikes in an isolated copy of your agent workflow.

Compare against

Your existing checks with the same tasks, permissions, and tool responses.

1. Locate the paper's implementation and check its license, model support, data needs, and hardware requirements. Stop here if you cannot obtain the required artifacts.
2. Save a small set of representative product tasks and expected outcomes. Keep evaluation tasks out of any training or tuning, and choose quality and cost limits before running.
3. Run the baseline and the proposed change on identical inputs. Keep other settings fixed and repeat paired runs to expose variation.
4. Save per-task outputs, failures, timings, and all usage costs. Review regressions before expanding the test.

What to measure

Successful attacks missed, legitimate actions blocked, detection delay, and review cost.

When to keep it

Keep the change only if it meets your preselected quality and latency limits and improves the metric you care about after setup and operating costs. Otherwise retain the baseline. This is an exploration plan, not a validated recommendation.

recommendation

adopt

research area

safety and guardrails

technical change

Behavioral audit of post-deletion memory persistence in agents

published

2026-10-22

Any production agent with persistent memory faces deletion-compliance obligations, and the paper shows on small open models (7B-class) that source deletion alone demonstrably fails while provenance-aware closure is a concrete, low-infrastructure mitigation the reader can implement and audit directly.

1. claim 01Paperraft inference
   Read the brief interpretation

   The method replaces treating deletion as a storage-level operation with a behavioral audit that compares exposed-then-deleted agents against matched never-exposed controls, and replaces naive exact-literal write filters with provenance-aware closure that also removes derived records such as summaries and decomposed components. The cost is engineering complexity: memory systems must track provenance from source records to all derived state, and audits require building controlled exposure/deletion test harnesses rather than relying on storage APIs. It can fail when provenance tracking is incomplete across heterogeneous memory stores, when response parsers misclassify target-bearing replies as clean, or when information persists in forms outside the tracked lineage.

   This is a Paperraft inference. It is not pinned to a verified PDF passage.

   Inspect the source
2. claim 02Paperraft inference

   Deleted secrets are recovered in roughly two thirds of single queries (Qwen2.5-7B-Instruct, zero in never-exposed controls); reconstruction from separately stored components succeeds in 62-99% of cases across Qwen, Mistral, and Phi-3; provenance-aware closure eliminates observed recovery in the tested positive-baseline settings.

   This is a Paperraft inference. It is not pinned to a verified PDF passage.

   Inspect the source