
MetaKV: Adaptive KV Cache Compression for Constrained LLM Inference
MetaKV replaces a single fixed KV cache compression configuration with per-prompt selection among existing methods (KVQuant, H2O, RocketKV, FP16) guided by lightweight predictors of latency, peak memory, and correctness. The cost is training and maintaining three prediction models plus a selection step per prompt, adding engineering complexity and a small inference-time overhead on top of the underlying compressors. It can fail if the accuracy or latency predictors mispredict on out-of-distribution prompts or on hardware and models different from the evaluation setup, causing constraint violations or suboptimal choices.








































