Transformers as Generic Deep Neural Networks: Exact Compilation, the Cost of Removing Activation-Activation Products, and Multi-Agent Attention Clustering

Profiling shows attention computation is a bottleneck in your self-hosted service.

What you would change

Check the supported GPU, tensor shapes, and precision, then replace only the relevant kernel.

Check the supported GPU, tensor shapes, and precision, then replace only the relevant kernel.

Compare against

The existing kernel inside the same complete serving pipeline.

1. Locate the paper's implementation and check its license, model support, data needs, and hardware requirements. Stop here if you cannot obtain the required artifacts.
2. Save a small set of representative product tasks and expected outcomes. Keep evaluation tasks out of any training or tuning, and choose quality and cost limits before running.
3. Run the baseline and the proposed change on identical inputs. Keep other settings fixed and repeat paired runs to expose variation.
4. Save per-task outputs, failures, timings, and all usage costs. Review regressions before expanding the test.

What to measure

Numerical differences, end-to-end latency, throughput, and memory at realistic shapes.

When to keep it

Keep the change only if it meets your preselected quality and latency limits and improves the metric you care about after setup and operating costs. Otherwise retain the baseline. This is an exploration plan, not a validated recommendation.

recommendation

test

research area

attention and kernels

technical change

Exact compilation of transformers to generic DNNs

published

2026-10-20

The compilation results are validated only on small models (DeiT-Tiny, MambaIRv2-Light, Pythia-70M), so the approach is testable on a single GPU but its production value for the reader's workloads requires hardware-operator and clustering-quality validation.

1. claim 01Paperraft inference
   Read the brief interpretation

   The method replaces Transformer-specific NPU operators and Transformer-specific quantization pipelines with an exact compilation of attention, LayerNorm, and GELU MLP into generic affine, routing, square, sigmoid, and inverse-square-root primitives, plus an optional top-k attention scheme where centroid clustering reduces computed scores from N^2 to Nk per head. The cost is implementation complexity: exact compilation requires an elementwise activation-activation product to avoid an N^2 blowup in scalar nonlinear evaluations, and the top-k gains depend on centroid-based key preselection and tail-mass estimation rather than standard dense attention. It can fail when the target hardware lacks the required elementwise product or sigmoid and inverse-square-root operators, when query or agent clustering is weak so centroid key selection misses relevant keys, or when tail scores are not exchangeable, which breaks the unbiasedness guarantee and reintroduces the conditional bias of tail dropping.

   This is a Paperraft inference. It is not pinned to a verified PDF passage.

   Inspect the source
2. claim 02Paperraft inference

   The exact tail-mass estimate matches or beats tail dropping with half as many keys (2x fewer keys) on DeiT-Tiny and, for k<=8, on Pythia-70M; dense-score compilation without activation-activation products adds at least N^2 scalar nonlinear evaluations per head.

   This is a Paperraft inference. It is not pinned to a verified PDF passage.

   Inspect the source