
The COUNT: Counting N-grams Unlocks Scaling on Host Memory and Training-Free Adaptation
COUNT replaces learned n-gram embedding tables (trained from scratch on GPU) with precomputed corpus n-gram counts stored in host memory, which weight the LM's own unembedding rows to form embeddings with no additional learned parameters. It costs host RAM for the count table, preprocessing to build it, and an added embedding lookup per token; the benefit only materializes if the backbone is pretrained with COUNT as an input. It can fail for the reader because the gains are demonstrated only for models pretrained from scratch with the method, and no compatible pretrained checkpoint may exist; count tables can also be stale or mismatched to the target domain, degrading the training-free adaptation.







































