ResidualQuant cuts KV cache memory in looped Transformers via 2-bit residuals
Looped Transformers face a critical memory wall: KV cache size grows linearly with recurrence depth, throttling batch sizes and throughput despite parameter efficiency gains. ResidualQuant exploits structural similarity across loops by storing only the final KV state in full precision and encoding earlier loops as low-bit residuals, dramatically reducing cache footprint without accuracy collapse. This technique directly addresses a scaling bottleneck that will matter as recurrent architectures compete with standard Transformers for production inference workloads, particularly in latency-sensitive or memory-constrained settings.62
