Skip to content

archived edition · 2026-09-19

PaperraftFind the AI research
worth testing.

Understand what a paper changes, what it costs, and whether it fits your next build. Explore research on faster LLM inference, agent memory, and evaluation, with the original sources in reach.

Explore the researchHow papers are assessed

Start with a brief. Follow the evidence. Choose a test.

edition
2026-09-19
briefs
42
research areas
16
independent implementations
23

29 of 42 papers in the index (69%) have no independent implementations.

Research index

Search by topic, browse a research area, or sort by date. Read a brief, open the original paper, and inspect the available evidence before deciding what to test.

showing30 of 42
sort by
arXiv 2608.21223
First page of Event-triggered Implicit Perturbation for Zeroth-Order Fine-Tuning of Spiking Transformers
efficient trainingnot applicable

Event-triggered Implicit Perturbation for Zeroth-Order Fine-Tuning of Spiking Transformers

Replaces explicit read-modify-write weight perturbation with perturbation sums from dedicated hardware attached to an IMC array for zeroth-order fine-tuning of spiking networks. Compared with the simplified variant, it requires 40%-46% more silicon area and 15%-49% more energy per operation, plus custom 16 nm CMOS fabrication or synthesis. It is a circuit for SNNs, rather than software deployable on a 24 GB GPU or through a third-party API.

observed signal
1impl.
1stars
0citations
2xclaimed gain
abstract onlyReview decisionDownload skill ↓Original paper ↗
arXiv 2609.17981
First page of Encoder Awakening via Adapters: Effective Domain-Adaptive Fine-tuning of Speech-LLMs
efficient trainingtest

Encoder Awakening via Adapters: Effective Domain-Adaptive Fine-tuning of Speech-LLMs

It replaces direct joint fine-tuning of a Speech-LLM (or LoRA-only LLM adaptation) with a two-stage procedure: first train lightweight adapters inserted into each frozen speech-encoder layer, then jointly fine-tune the whole model with LoRA on the LLM. The cost is an extra training stage, modest additional adapter parameters in the encoder, and a more complex pipeline than a single LoRA pass; it presumes an existing Speech-LLM checkpoint and target-domain labeled audio. It can fail if the target domain is not acoustic-shifted (e.g., the gap is lexical or semantic rather than child/dialectal acoustics), if very limited target data overfits the encoder adapters, or if the base model is not an encoder-projector-LLM ASR stack, making the method inapplicable.

observed signal
1impl.
1stars
0citations
—claimed gain
abstract onlyReview decisionDownload skill ↓Original paper ↗
arXiv 2609.19587
First page of Red-Teaming Auto Mode: Improving Blocking Classifiers Against Malign Coding Agents
safety and guardrailstest

Red-Teaming Auto Mode: Improving Blocking Classifiers Against Malign Coding Agents

The paper replaces passive or single-pass action-review monitors with hardened blocking monitors whose weaknesses were found by tasking adversarial agents with evading them, and it details concrete design fixes (broader tool coverage, transcript formatting, an agentic monitor stage). The cost is additional review compute and latency per proposed action, plus the engineering complexity of running a red-team harness and a multi-stage monitoring pipeline on a small team's budget. What can fail is residual evasion: multi-context attacks such as malicious compaction and cross-agent injection remain effective even after the improvements, so the monitor must not be treated as a complete safety boundary.

observed signal
2impl.
14stars
0citations
—claimed gain
abstract onlyReview decisionDownload skill ↓Original paper ↗
arXiv 2609.19391
First page of MAGS: Multi-agent Auto-formalization Guarantees Safety for Agentic Outputs
safety and guardrailstest

MAGS: Multi-agent Auto-formalization Guarantees Safety for Agentic Outputs

MAGS replaces fuzz testing, static analysis, and LLM-as-a-Verifier for safety assurance of agent-generated code by translating programs into Dafny, repairing them against verifier feedback, and compiling verified code back to executables. The cost is a multi-agent LLM pipeline plus the Dafny toolchain, human-audited frozen API specifications, and repair-loop latency on every generated artifact. It can fail silently when the auto-formalized specification itself does not capture the intended target behavior, in which case the guarantee is vacuously satisfied.

observed signal
3impl.
21stars
0citations
—claimed gain
abstract onlyReview decisionDownload skill ↓Original paper ↗
arXiv 2609.19334
First page of A frontend-backend architecture for tool calls in full-duplex speech models
tool usewatch

A frontend-backend architecture for tool calls in full-duplex speech models

The method replaces end-to-end tool-calling inside a full-duplex speech model with a delegation token that routes streaming ASR transcripts to a text LLM backend, injecting results back via a prefill-and-repeat mechanism. The cost is an extra backend LLM in the loop (with its latency and API or serving expense), plus training the frontend to emit delegation tokens and maintaining the transcript-injection plumbing. Failure modes include missed or spurious delegation tokens (roughly 3-8% recall loss and ~19% false-accept on irrelevant requests in their own evaluation), added response latency from the backend round-trip, and degradation of interruption handling if injection races with user barge-in.

observed signal
3impl.
124stars
0citations
—claimed gain
abstract onlyReview decisionDownload skill ↓Original paper ↗
arXiv 2609.19656
First page of Self-Evolving Search Index
retrieval and RAGtest

Self-Evolving Search Index

Replaces manual, human-driven diagnosis and rewriting of index keys (e.g., chunk representations, keyword/summary keys) with an automated optimizer that detects retrieval failures, revises the responsible keys, validates revisions, and simulates additional queries. Costs include repeated LLM calls for diagnosis, key rewriting, validation, and query simulation, plus periodic re-indexing and added pipeline complexity, all feasible on a single GPU or via APIs but with ongoing per-document inference expense. It can fail if the query simulator generates unrepresentative queries that drift the index away from real workload needs, if validation is too weak to catch harmful key revisions, or if retrieval failures stem from the retriever or corpus rather than the index keys, in which case the optimization loop yields little gain.

observed signal
2impl.
18stars
—citations
—claimed gain
abstract onlyReview decisionDownload skill ↓Original paper ↗
arXiv 2609.03430
First page of Random Attention: Rethinking KV Cache Eviction for Efficient Reasoning
KV cacheadopt

Random Attention: Rethinking KV Cache Eviction for Efficient Reasoning

Random Attention replaces score-based KV cache eviction policies (such as H2O-style importance scoring) with uniform random eviction within each attention head, while always retaining the prompt, eliminating all per-token scoring computation. It costs nothing in memory or latency beyond the eviction itself and adds almost no implementation complexity, since the selection signal is removed entirely; quality is reported to match the best prior evictor on reasoning workloads. It can fail on workloads where the reasoning trace does not restate needed information or where redundancy across heads is insufficient, and on non-reasoning tasks or cache configurations where the prompt is not the dominant fragile component, since the findings are validated on reasoning traces with long chains of thought.

observed signal
2impl.
67stars
0citations
1.43xclaimed gain
abstract onlyReview decisionDownload skill ↓Original paper ↗
arXiv 2609.19526
First page of Self Improvement via Fast Tree-search
coding agentstest

Self Improvement via Fast Tree-search

The method replaces the practice of re-running benchmark subsets to score every candidate self-modification of a coding agent with an LLM-as-a-judge pairwise comparison signal aggregated via a regularized Bradley-Terry model, reserving full task evaluations only for the most promising tree-search nodes. The cost is the added judge-model API calls and implementation complexity of a disaggregated tree search with rank-based parent sampling, though total reported resource use is lower than prior frameworks. What can fail: the judge's pairwise rankings may not correlate with true downstream task performance on a different codebase or task distribution, silently steering search toward patches the judge favors rather than patches that work, and gains are demonstrated only on the Polyglot benchmark with no reported replication.

observed signal
2impl.
124stars
0citations
—claimed gain
abstract onlyReview decisionDownload skill ↓Original paper ↗
arXiv 2609.19244
First page of Characterizing Web Search by Conversational LLM Agents: From Search Decisions and Strategies to Results and Responses
agent evaluationwatch

Characterizing Web Search by Conversational LLM Agents: From Search Decisions and Strategies to Results and Responses

This is a measurement study, not a deployable method; it replaces no existing technique but characterizes how four commercial platforms decide to invoke web search, formulate queries, select domains, and cite sources, whereas the reader's equivalent practice is ad hoc prompt-level tuning of retrieval-augmented pipelines. Adopting its findings costs only engineering attention, but translating them into practice means building citation-verification and search-invocation auditing into a RAG pipeline, which adds latency and evaluation complexity. The findings can fail to transfer because they are platform- and model-version-specific: search behavior varies substantially across ChatGPT, Claude, Grok, and DeepSeek, so conclusions may not hold for the reader's chosen API or self-hosted stack.

observed signal
1impl.
14stars
0citations
—claimed gain
abstract onlyReview decisionDownload skill ↓Original paper ↗
arXiv 2609.19425
First page of Closed-World Resolution Against Tool Hallucination in LLM Agents
safety and guardrailsadopt

Closed-World Resolution Against Tool Hallucination in LLM Agents

The resolver replaces nothing in the existing pipeline; it adds a training-free pre-gate check (tool registry membership plus argument signature validation) that rejects calls to nonexistent tools or with undeclared arguments before any causal or security gate runs, since a hallucinated call is by construction outside what a gate decided. Its cost is negligible compute and latency (a lookup and a schema check), with the engineering burden of maintaining an accurate, versioned tool registry and correctly handling merged MCP namespaces where name collisions and shadowing create surfaces a single registry cannot express. It fails on the one irreducible residue: calls to real tools with borrowed arguments that are schema-indistinguishable from valid calls, which requires downstream gating or execution checks rather than closed-world resolution.

observed signal
1impl.
14stars
—citations
—claimed gain
abstract onlyReview decisionDownload skill ↓Original paper ↗
arXiv 2609.19465
First page of Compositional Reasoning in Language Models under Reinforcement Learning Post-Training
efficient trainingtest

Compositional Reasoning in Language Models under Reinforcement Learning Post-Training

The paper's finding replaces the intuitive curriculum of RL post-training on decomposed sub-skills first: training directly on composed tasks transfers back to decomposed skills, while the reverse transfer is unreliable. The cost is that composed-task training requires verified, compositional reward signals (here, deterministic data-structure tasks and, preliminarily, tool-calling), plus the engineering effort of building such datasets and running RL post-training on a 24 GB GPU, which limits model scale. It can fail when composed tasks are too hard to bootstrap learning from scratch, when reward verification for composed outputs is unavailable, or when the asymmetry does not hold outside the evaluated synthetic and pilot tool-calling settings.

observed signal
2impl.
134stars
—citations
—claimed gain
abstract onlyReview decisionDownload skill ↓Original paper ↗
arXiv 2609.19524
First page of A Unified Evaluation Framework for Trustworthy Large Language Models, Agentic AI, and Multimodal Systems
agent evaluationtest

A Unified Evaluation Framework for Trustworthy Large Language Models, Agentic AI, and Multimodal Systems

This replaces ad hoc, system-specific evaluation scripts and single benchmark scores with a structured framework that normalizes output-level, trajectory-level, and cross-modal metrics into eight common trustworthiness dimensions with uncertainty estimates, evidence traces, and safety-critical overrides. The cost is engineering effort: instrumenting metrics across dimensions, defining band mappings, maintaining a meta-evaluation layer for validity and reproducibility, and aligning results to governance and regulatory mappings, all of which add evaluation pipeline complexity rather than GPU or latency overhead. What can fail is that the band normalization is a design choice with no validated ground truth, so scores can appear comparable while masking calibration errors, and the authors state that empirical validation across deployment contexts remains future work.

observed signal
2impl.
134stars
0citations
—claimed gain
abstract onlyReview decisionDownload skill ↓Original paper ↗
arXiv 2609.18077
First page of vidax: A Unified JAX Framework for Video Generative Models on Accelerator Meshes
serving and batchingnot applicable

vidax: A Unified JAX Framework for Video Generative Models on Accelerator Meshes

vidax replaces PyTorch/CUDA reference implementations of video generative models with a zero-PyTorch-dependency JAX/Flax inference stack on TPU pods, combining tensor parallelism, sequence parallelism, flash-attention kernels, and per-layer weight offloading. The cost is a full framework and hardware migration: Cloud TPU access, JAX/Flax adoption, and checkpoint translation that the authors themselves report as a source of real numerical bugs. It can fail through silent numerical divergence in translated weights, compile-time overhead, and the operational burden of maintaining a parallel JAX stack.

observed signal
1impl.
19stars
—citations
—claimed gain
abstract onlyReview decisionDownload skill ↓Original paper ↗
arXiv 2609.16367
First page of FINNAS: FINN-Guided Hardware-Aware NAS and Pruning for FPGA Jet Substructure Classification
efficient architectureswatch

FINNAS: FINN-Guided Hardware-Aware NAS and Pruning for FPGA Jet Substructure Classification

The method replaces manual design-space exploration of quantised MLP accelerators for FPGAs with an evolutionary NAS that jointly searches depth, width, and precision, ranking candidates with FINN-estimated LUT and latency proxies before pruning and RTL validation. It costs access to the FINN/Vivado FPGA toolchain, repeated synthesis-in-the-loop evaluation during search, and acceptance of fully parallel mappings whose proxy estimates must be confirmed by out-of-context synthesis. It can fail when the search proxies mispredict post-synthesis resource or latency figures, when results do not transfer off the specific jet-substructure task and target FPGA, and it is irrelevant to deployments on GPU or API-served models.

observed signal
0impl.
0stars
0citations
8.5xclaimed gain
abstract onlyReview decisionDownload skill ↓Original paper ↗
arXiv 2609.16391
First page of Where Post-Training Quantization Breaks Text Embedders: A Measured Map Across Four Embedder Families
quantizationtest

Where Post-Training Quantization Breaks Text Embedders: A Measured Map Across Four Embedder Families

For retrieval embedders below INT4, a small task-distilled student quantized to INT3 replaces aggressive quantization of the larger teacher, rather than applying LLM-carried heuristics (embedding-table protection, module-wise bit allocation, ranking-aware objectives), which the paper shows do not transfer to embedders. The cost is a distillation pipeline and task-representative training data, plus the engineering of validating each (family, width, group-size) point instead of relying on a reusable sensitivity ordering; reconstruction-error proxies remain usable for screening uniform widths but not for selecting protected tensors. It can fail when the deployment task drifts from the distillation task, where the student's advantage over the quantized teacher does not hold, and joint damage at INT2-INT3 is not predictable from isolated module measurements.

observed signal
0impl.
0stars
0citations
4.36xclaimed gain
abstract onlyReview decisionDownload skill ↓Original paper ↗
arXiv 2609.16403
First page of Implementing a White-Box Undetectable Backdoor for Random Fourier Features
safety and guardrailswatch

Implementing a White-Box Undetectable Backdoor for Random Fourier Features

The paper implements, in numpy and scipy, a cryptographically undetectable backdoor for Random Fourier Features models, replacing informal assumptions about white-box auditability with a concrete demonstration that full weight inspection cannot distinguish backdoored from clean models. It does not replace any production technique; the cost to the reader is awareness that weight-level auditing of RFF-based models offers no security guarantee, and replicating the work requires deriving sampling machinery the original construction left unspecified. What can fail: the finding applies to the RFF setting specifically, the underlying lattice hardness reduction was not reproduced, and the negative indistinguishability result is empirical within the tested sparsity range rather than a new proof.

observed signal
0impl.
0stars
0citations
—claimed gain
abstract onlyReview decisionDownload skill ↓Original paper ↗
arXiv 2609.16446
First page of Adaptive Bayesian Partner Selection for Federated Clinical Centers
otherwatch

Adaptive Bayesian Partner Selection for Federated Clinical Centers

It replaces persistent all-client federated averaging (FedAvg-style global rounds) with per-center peer selection using a Beta-Bernoulli posterior over Shapley utility, UCB ranking, a propose-reject handshake, and optional abstention from communication. The cost is per-center bookkeeping of partner posteriors and pairwise negotiation, plus reliance on the supporting techniques (head personalization, bfloat16 quantized payloads) for the reported efficiency; accuracy is only matched, not improved. It can fail when centers are few or large, when Shapley utility estimates are noisy or expensive to obtain, or when peer non-IID drift invalidates the posterior faster than it concentrates, and the evidence base is a single binary ICU mortality task on one dataset.

observed signal
0impl.
0stars
0citations
11.1xclaimed gain
abstract onlyReview decisionDownload skill ↓Original paper ↗
arXiv 2609.16450
First page of Early-Bird Decoding: Accelerating Diffusion LLMs with Learnable Block Sizes and Parallel Sampling
speculative decodingtest

Early-Bird Decoding: Accelerating Diffusion LLMs with Learnable Block Sizes and Parallel Sampling

The method replaces fixed-block, threshold-based unmasking in diffusion LLM decoding with a learned variable-length block grouper plus a position-aware parallel sampler that commits low-entropy token clusters before they reach the confidence threshold. Cost is limited to training and serving two small plug-in modules on top of an existing pretrained dLLM, with claimed negligible overhead and no weight modification, but it presupposes that the production workload runs on a diffusion LLM rather than an autoregressive model. It can fail if the learned uncertainty clustering mispredicts on out-of-distribution prompts, if accuracy degradation appears on tasks outside the four evaluated benchmarks, or if the plug-in depends on dLLM serving infrastructure the team does not operate.

observed signal
0impl.
0stars
0citations
1.58xclaimed gain
abstract onlyReview decisionDownload skill ↓Original paper ↗
arXiv 2609.16485
First page of Certified Inference and Training for Deep Equilibrium Networks: A Continuation Framework with Polynomial Complexity Guarantees
otherwatch

Certified Inference and Training for Deep Equilibrium Networks: A Continuation Framework with Polynomial Complexity Guarantees

The method replaces standard fixed-point iteration and implicit-differentiation training of deep equilibrium networks with a certified homotopy-continuation tracker plus an interpolation-based trainer using dormant bilinear rank-one channels. It costs implementation complexity far beyond standard DEQ code: certified gate realization, column stability checks, loaded Tikhonov solves, and precision budgets, none of which are supported by existing frameworks. The guarantees hold only on a certified promise class; on real instances outside that class the certified passes can fail, requiring residual-aligned repair, and no measured latency or quality benefit exists.

observed signal
0impl.
0stars
0citations
—claimed gain
abstract onlyReview decisionDownload skill ↓Original paper ↗
arXiv 2609.17652
First page of Fathom: Per-Query Read Depth for Sparse Decoding over Offloaded KV Caches
KV cachewatch

Fathom: Per-Query Read Depth for Sparse Decoding over Offloaded KV Caches

Fathom replaces fixed-width key scans (Double Sparsity, Loki, SparQ) in top-k sparse attention over host-memory-offloaded KV caches with a per-query variable bit-depth read over bit-plane-organized 4-bit keys, where a reverse water-filling rule allocates bits across channels by variance-weighted importance. It costs little in storage since it reuses the 4-bit K copy a quantized serving stack already holds, but adds channel-major bit-plane layout and per-query bit-budget computation to the decode path, and quality depends on the bit budget chosen. It can fail to deliver any benefit when the index is GPU-resident, as the authors state the method is not faster in that case, and its gains are demonstrated only at very long contexts with many concurrent sessions where host-memory scan traffic is the bottleneck.

observed signal
0impl.
0stars
0citations
1.67xclaimed gain
abstract onlyReview decisionDownload skill ↓Original paper ↗
arXiv 2609.17698
First page of A Large-Scale Empirical Study of Quality Assurance Practices and Gaps in AI Agents
agent evaluationadopt

A Large-Scale Empirical Study of Quality Assurance Practices and Gaps in AI Agents

This empirical study replaces ad hoc, feature-level testing of LLM agents with a systematic QA taxonomy covering execution surfaces, safeguards, and end-to-end validation of tool use, untrusted inputs, and persistent state. Adopting its guidance costs engineering time to write boundary, adversarial, and multi-step tool-failure tests and to audit safeguard consistency across equivalent execution routes, but requires no additional GPU or model infrastructure. If adopted superficially, the checklist can produce false assurance: coverage of listed risk scenarios does not guarantee detection of novel failure modes, and findings from open-source projects may not transfer to proprietary production agents.

observed signal
0impl.
0stars
0citations
—claimed gain
abstract onlyReview decisionDownload skill ↓Original paper ↗
arXiv 2609.17759
First page of Derivative-Free Structured Updates for Muon
efficient trainingwatch

Derivative-Free Structured Updates for Muon

The method replaces gradient-based momentum in the Muon optimizer with structured finite-difference probes (entrywise, random low-rank, basis-aligned rank-one, or direct search) orthogonalized into Muon-style updates. The cost is a large number of additional function evaluations per update step, plus reduced update accuracy when probing is subsampled, and it carries no convergence guarantee. It fails to help whenever reliable, inexpensive backpropagation gradients are available, which covers essentially all standard model training on a single-GPU budget.

observed signal
0impl.
0stars
0citations
—claimed gain
abstract onlyReview decisionDownload skill ↓Original paper ↗
arXiv 2609.17786
First page of FairCompressAgent: An Agentic Framework for Fairness-Aware Model Compression for FPGA Deployment
sparsity and pruningtest

FairCompressAgent: An Agentic Framework for Fairness-Aware Model Compression for FPGA Deployment

The method replaces manual, ad hoc selection and composition of pruning, quantization, and low-rank factorization configurations with a language-model planner that iterates over measured compression outcomes under explicit accuracy, fairness, and cost constraints. It costs an execution layer that runs repeated fine-tuning and evaluation per candidate configuration, plus planner API calls, which is feasible on a single GPU but adds engineering complexity beyond a fixed compression recipe. It can fail when the planner proposes unsatisfiable constraint combinations, when results from the single Fitzpatrick-17k/VGG-11 benchmark do not transfer to other models or tasks, and when the FPGA deployment target itself is irrelevant to teams serving via GPU or third-party APIs.

observed signal
0impl.
0stars
0citations
2.47xclaimed gain
abstract onlyReview decisionDownload skill ↓Original paper ↗
arXiv 2609.17815
First page of Principled Koopman Representations with Kalman Inference for Efficient Time-Series Prediction
othertest

Principled Koopman Representations with Kalman Inference for Efficient Time-Series Prediction

K2SVD replaces neural latent Koopman-space models (and generic deep time-series forecasters) with an explicitly low-rank Koopman operator learned via a Hilbert-Schmidt objective, plus a linear Gaussian state-space model with Kalman filtering for multi-step prediction. It costs little at inference—a compact latent space and linear Kalman recursion suit a single 24 GB GPU or even CPU—though it adds the complexity of fitting the SVD-style representation and state-space model. It can fail when system dynamics are not well approximated by a low-rank linear operator in the learned observable space, when the Gaussian noise assumption is violated, or on datasets outside the evaluated benchmarks where the accuracy claims may not transfer.

observed signal
0impl.
1stars
0citations
—claimed gain
abstract onlyReview decisionDownload skill ↓Original paper ↗
arXiv 2609.17817
First page of Reflections on Trusting Trust, Revisited: Contaminating Self-Modifying AI Coding Agents with Poisoned Benchmarks
safety and guardrailswatch

Reflections on Trusting Trust, Revisited: Contaminating Self-Modifying AI Coding Agents with Poisoned Benchmarks

The paper does not propose a technique that replaces an existing production method; it demonstrates an attack in which poisoned benchmarks contaminate the self-evaluation loop of self-modifying coding agents, causing evolved versions to emit vulnerable code (e.g., disabled HTTPS certificate validation) on clean tasks. There is no adoption cost in the usual sense, but the implied defensive cost is that any team running self-improving agent loops must add benchmark provenance checks, held-out clean evaluation, and auditing of self-modifications, none of which the paper shows to be reliably effective since contamination persists after clean re-evolution. What can fail is any pipeline that trusts benchmark results to steer agent self-modification: a single compromised benchmark can induce persistent insecure behavior that clean benchmarks do not repair.

observed signal
0impl.
0stars
0citations
—claimed gain
abstract onlyReview decisionDownload skill ↓Original paper ↗
arXiv 2609.17848
First page of SFT or RL for Tool-Calling Agents? A Controlled Study Across Data, Method, and Scale
tool useadopt

SFT or RL for Tool-Calling Agents? A Controlled Study Across Data, Method, and Scale

This controlled comparison replaces the default assumption that RL post-training (GRPO) or SFT-then-RL pipelines are needed for tool-calling agents, showing simple LoRA SFT is the stronger default and also beats full-parameter fine-tuning. The cost is modest: LoRA SFT runs on a single 24 GB GPU for small-to-mid models, while GRPO adds training complexity, compute, and pipeline overhead for gains that are largely confined to cross-dataset transfer and average under one point. Failure modes include overfitting to the training dataset's tool format when transfer matters, which dataset mixing mitigates, and possible degradation of pretrained agentic behavior if full fine-tuning is used instead of LoRA.

observed signal
0impl.
0stars
0citations
—claimed gain
abstract onlyReview decisionDownload skill ↓Original paper ↗
arXiv 2609.17885
First page of ERPBench: A State-Grounded Evaluation Paradigm for Computer-Use Agents in Enterprise Software
agent evaluationwatch

ERPBench: A State-Grounded Evaluation Paradigm for Computer-Use Agents in Enterprise Software

ERPBench replaces screen-based or simulated evaluation of computer-use agents with a live, reproducible ERP system scored against ground-truth database values, plus a human-approval gating harness for deployment. It costs benchmark setup effort rather than inference resources; adoption requires running the ERP instance and harness, which is infrastructure overhead rather than a model or latency cost. What can fail: it measures agent capability but does not improve it, so a team not deploying GUI agents in ERP-like software gains no production benefit.

observed signal
0impl.
0stars
0citations
—claimed gain
abstract onlyReview decisionDownload skill ↓Original paper ↗
arXiv 2609.17943
First page of ASPIRE: Asynchronous Batched Self-Speculative Decoding for Long-Context LLM Inference
speculative decodingwatch

ASPIRE: Asynchronous Batched Self-Speculative Decoding for Long-Context LLM Inference

ASPIRE replaces synchronized draft-verify scheduling in batched self-speculative decoding with a mixed forward pass where drafting and verifying requests coexist, plus per-request acceptance-rate scheduling and a single full-attention refresh layer during drafting. It costs implementation complexity in the serving stack: a custom unified kernel path, an online scheduler with a batch-aware cost model, and modified drafting with an intra-draft refresh layer, none of which are available in standard inference frameworks. It can fail if per-request acceptance estimates are noisy or unstable, if the cost model misjudges batch composition, or if sparse-attention drafting quality degrades on workloads unlike the evaluated reasoning and long-context benchmarks.

observed signal
0impl.
0stars
—citations
4.58xclaimed gain
abstract onlyReview decisionDownload skill ↓Original paper ↗
arXiv 2609.17984
First page of TuiML: Machine Learning for AI Agents
tool usetest

TuiML: Machine Learning for AI Agents

TuiML replaces agents recalling scikit-learn/Weka APIs from memory with a library whose components expose machine-readable metadata and parameter schemas, so agents can search, inspect, and compose validated workflows, with one specification layer driving MCP, a Python API, a CLI, and local serving. The cost is adopting a new native algorithm stack in place of battle-tested scikit-learn implementations, plus the integration and maintenance burden of an early-stage open-source project whose long-term support is unproven. Failures after adoption can include algorithm coverage or correctness gaps relative to scikit-learn, community and maintenance risk, and the fact that the claimed benefits for agent reliability (fewer runtime errors, preserved state) are asserted rather than measured.

observed signal
0impl.
0stars
0citations
—claimed gain
abstract onlyReview decisionDownload skill ↓Original paper ↗
arXiv 2609.17985
First page of RideWay: Benchmarking Efficient Task Completion for Tool-Using Language Agents
agent evaluationtest

RideWay: Benchmarking Efficient Task Completion for Tool-Using Language Agents

RideWay replaces binary task-completion scoring of tool-using agents with a success-gated utility that discounts trajectories for excess tool calls and user-facing turns, calibrated against human preferences. Adoption costs a task-specific reference-effort annotation per task, preference data to fit penalty weights, and instrumentation to log turns and calls, none of which requires cluster-scale compute. It can fail because count-based penalties do not capture why extra calls occurred: the metric is at chance level when trajectories differ only in tool-call counts, and the 58-task ridehailing scope plus per-domain penalty fitting limits transfer to other agent workloads without recalibration.

observed signal
0impl.
0stars
0citations
—claimed gain
abstract onlyReview decisionDownload skill ↓Original paper ↗

Research signals

Three views of the same evidence: adoption against attention, publication cadence by research area, and reported gains when coverage is sufficient.

color = research area

01 · attention × adoption

The research frontier

Look to the upper left for independent implementation before mass attention.

compare by
Each point represents one paper. Higher means more independent implementations; farther right means more GitHub stars.Event-triggered Implicit Perturbation for Zeroth-Order Fine-Tuning of Spiking Transformers: 1 GitHub star; 1 independent implementationRandom Attention: Rethinking KV Cache Eviction for Efficient Reasoning: 67 GitHub stars; 2 independent implementationsFINNAS: FINN-Guided Hardware-Aware NAS and Pruning for FPGA Jet Substructure Classification: 0 GitHub stars; 0 independent implementationsWhere Post-Training Quantization Breaks Text Embedders: A Measured Map Across Four Embedder Families: 0 GitHub stars; 0 independent implementationsImplementing a White-Box Undetectable Backdoor for Random Fourier Features: 0 GitHub stars; 0 independent implementationsAdaptive Bayesian Partner Selection for Federated Clinical Centers: 0 GitHub stars; 0 independent implementationsEarly-Bird Decoding: Accelerating Diffusion LLMs with Learnable Block Sizes and Parallel Sampling: 0 GitHub stars; 0 independent implementationsCertified Inference and Training for Deep Equilibrium Networks: A Continuation Framework with Polynomial Complexity Guarantees: 0 GitHub stars; 0 independent implementationsFathom: Per-Query Read Depth for Sparse Decoding over Offloaded KV Caches: 0 GitHub stars; 0 independent implementationsA Large-Scale Empirical Study of Quality Assurance Practices and Gaps in AI Agents: 0 GitHub stars; 0 independent implementationsDerivative-Free Structured Updates for Muon: 0 GitHub stars; 0 independent implementationsFairCompressAgent: An Agentic Framework for Fairness-Aware Model Compression for FPGA Deployment: 0 GitHub stars; 0 independent implementationsPrincipled Koopman Representations with Kalman Inference for Efficient Time-Series Prediction: 1 GitHub star; 0 independent implementationsReflections on Trusting Trust, Revisited: Contaminating Self-Modifying AI Coding Agents with Poisoned Benchmarks: 0 GitHub stars; 0 independent implementationsSFT or RL for Tool-Calling Agents? A Controlled Study Across Data, Method, and Scale: 0 GitHub stars; 0 independent implementationsERPBench: A State-Grounded Evaluation Paradigm for Computer-Use Agents in Enterprise Software: 0 GitHub stars; 0 independent implementationsASPIRE: Asynchronous Batched Self-Speculative Decoding for Long-Context LLM Inference: 0 GitHub stars; 0 independent implementationsEncoder Awakening via Adapters: Effective Domain-Adaptive Fine-tuning of Speech-LLMs: 1 GitHub star; 1 independent implementationTuiML: Machine Learning for AI Agents: 0 GitHub stars; 0 independent implementationsRideWay: Benchmarking Efficient Task Completion for Tool-Using Language Agents: 0 GitHub stars; 0 independent implementationsQuanText: Protecting Dataset-Level Secrets in Textual Data Sharing: 0 GitHub stars; 0 independent implementationsA Calibrated Instrument for Measuring How Inference Optimizations Affect Output Quality: 0 GitHub stars; 0 independent implementationsvidax: A Unified JAX Framework for Video Generative Models on Accelerator Meshes: 19 GitHub stars; 1 independent implementationSSD-LLaMA: SSD-Native Inference for Trillion-Parameter MoE at 1+ Token/s on a Consumer PC: 0 GitHub stars; 0 independent implementationsPosition: It is Time to Virtualize Foundation Models with a Self-evolving Operating System Layer: 0 GitHub stars; 0 independent implementationsMeshKV: A Network-on-Chip KV Cache Fabric for Scalable Transformer Decoding Accelerators: 0 GitHub stars; 0 independent implementationsLayer-wise Curriculum Learning for Efficient LLM Compression: 0 GitHub stars; 0 independent implementationsPAPC: Platform Mediation for Privacy-Propagation Externalities in AI-Mediated Workflows: 4 GitHub stars; 0 independent implementationsBlock Parallelism For Efficient Distributed Long-Context Diffusion Language Model Training: 4 GitHub stars; 0 independent implementationsCharacterizing Web Search by Conversational LLM Agents: From Search Decisions and Strategies to Results and Responses: 14 GitHub stars; 1 independent implementationA frontend-backend architecture for tool calls in full-duplex speech models: 124 GitHub stars; 3 independent implementationsMAGS: Multi-agent Auto-formalization Guarantees Safety for Agentic Outputs: 21 GitHub stars; 3 independent implementationsClosed-World Resolution Against Tool Hallucination in LLM Agents: 14 GitHub stars; 1 independent implementationCompositional Reasoning in Language Models under Reinforcement Learning Post-Training: 134 GitHub stars; 2 independent implementationsSafety Beyond the Interface: Detecting Harm via Latent States in Large Language Models: 4 GitHub stars; 0 independent implementationsReputation as Community Memory for the Agentic Web: 10 GitHub stars; 0 independent implementationsA Unified Evaluation Framework for Trustworthy Large Language Models, Agentic AI, and Multimodal Systems: 134 GitHub stars; 2 independent implementationsSelf Improvement via Fast Tree-search: 124 GitHub stars; 2 independent implementationsRed-Teaming Auto Mode: Improving Blocking Classifiers Against Malign Coding Agents: 14 GitHub stars; 2 independent implementationsDeltaSelect: Affordable A/B Testing for Coding Agents: 10 GitHub stars; 0 independent implementationsFrom Intent to Action: Benchmarking LLM Safety in Vehicle Voice Command Authorization: 120 GitHub stars; 0 independent implementationsSelf-Evolving Search Index: 18 GitHub stars; 2 independent implementations

Papers above 1000 stars or 200 citations are not scored. They have already broken through the attention threshold, while Paperraft is designed to identify earlier signals.

02 · cadence

Research areas over time

Compare monthly publication volume on a shared scale and calendar.

coding agentsmax 9Aug 26: 0 papersSep 26: 1 paperAug 26Sep 26efficient architecturesmax 9Aug 26: 0 papersSep 26: 1 paperAug 26Sep 26agent evaluationmax 9Aug 26: 0 papersSep 26: 7 papersAug 26Sep 26KV cachemax 9Aug 26: 0 papersSep 26: 3 papersAug 26Sep 26speculative decodingmax 9Aug 26: 0 papersSep 26: 2 papersAug 26Sep 26distillationmax 9Aug 26: 0 papersSep 26: 1 paperAug 26Sep 26sparsity and pruningmax 9Aug 26: 0 papersSep 26: 1 paperAug 26Sep 26memory and contextmax 9Aug 26: 0 papersSep 26: 1 paperAug 26Sep 26multi-agent orchestrationmax 9Aug 26: 0 papersSep 26: 1 paperAug 26Sep 26othermax 9Aug 26: 0 papersSep 26: 3 papersAug 26Sep 26quantizationmax 9Aug 26: 0 papersSep 26: 1 paperAug 26Sep 26retrieval and RAGmax 9Aug 26: 0 papersSep 26: 1 paperAug 26Sep 26safety and guardrailsmax 9Aug 26: 0 papersSep 26: 9 papersAug 26Sep 26serving and batchingmax 9Aug 26: 0 papersSep 26: 2 papersAug 26Sep 26efficient trainingmax 9Aug 26: 1 paperSep 26: 4 papersAug 26Sep 26tool usemax 9Aug 26: 0 papersSep 26: 3 papersAug 26Sep 26

Every panel uses the same calendar and vertical scale. An empty column means no papers were recorded.

03 · reported results

Claimed performance gains

Gains stated in the abstract: reported by the authors; not independently verified.

Factors reported by the authors and not independently verified. The vertical axis uses a logarithmic scale; the baseline is 1x.reported factor · log scaleEvent-triggered Implicit Perturbation for Zeroth-Order Fine-Tuning of Spiking Transformers: 2x in cost (author-reported)Random Attention: Rethinking KV Cache Eviction for Efficient Reasoning: 1.43x in speed (author-reported)FINNAS: FINN-Guided Hardware-Aware NAS and Pruning for FPGA Jet Substructure Classification: 8.5x in cost (author-reported)Where Post-Training Quantization Breaks Text Embedders: A Measured Map Across Four Embedder Families: 4.36x in memory (author-reported)Adaptive Bayesian Partner Selection for Federated Clinical Centers: 11.1x in cost (author-reported)Early-Bird Decoding: Accelerating Diffusion LLMs with Learnable Block Sizes and Parallel Sampling: 1.58x in speed (author-reported)Fathom: Per-Query Read Depth for Sparse Decoding over Offloaded KV Caches: 1.67x in speed (author-reported)FairCompressAgent: An Agentic Framework for Fairness-Aware Model Compression for FPGA Deployment: 2.47x in memory (author-reported)ASPIRE: Asynchronous Batched Self-Speculative Decoding for Long-Context LLM Inference: 4.58x in speed (author-reported)SSD-LLaMA: SSD-Native Inference for Trillion-Parameter MoE at 1+ Token/s on a Consumer PC: 15.58x in speed (author-reported)MeshKV: A Network-on-Chip KV Cache Fabric for Scalable Transformer Decoding Accelerators: 1.9x in speed (author-reported)Layer-wise Curriculum Learning for Efficient LLM Compression: 2x in memory (author-reported)Block Parallelism For Efficient Distributed Long-Context Diffusion Language Model Training: 1.45x in speed (author-reported)DeltaSelect: Affordable A/B Testing for Coding Agents: 2.39x in cost (author-reported)From Intent to Action: Benchmarking LLM Safety in Vehicle Voice Command Authorization: 1.42x in quality (author-reported)

15 of 42 papers report a quantified gain. The scale is logarithmic; a research-area trend line appears only with at least five papers in the quarter.

Method under scrutiny

Inspect the highest-ranked paper, every repository behind its signal, and the rule used to classify each one.

Event-triggered Implicit Perturbation for Zeroth-Order Fine-Tuning of Spiking Transformers

efficient training · not applicable · 1 of 2 implementations is independent · 2x gain in cost (reported by the authors; not independently verified)

Replaces explicit read-modify-write weight perturbation with perturbation sums from dedicated hardware attached to an IMC array for zeroth-order fine-tuning of spiking networks. Compared with the simplified variant, it requires 40%-46% more silicon area and 15%-49% more energy per operation, plus custom 16 nm CMOS fabrication or synthesis. It is a circuit for SNNs, rather than software deployable on a 24 GB GPU or through a third-party API.

Exclusions and controls

Every exclusion is counted and published.

Exclusion counts were not recorded for this edition.

0 previously indexed papers were rechecked in this run.

Paperraft tracks one signal: how many independent implementations a paper attracts on GitHub after author-owned repositories are removed. This is a discovery signal to investigate, not a measure of implementation quality.

It does not claim that a method works. Paperraft runs no reproduction benchmarks. Reported gains remain author claims until independently tested. Papers above the attention threshold are excluded because this publication is designed to identify research before consensus forms.