Geometric Factual Recall in Transformers
Researchers have identified a fundamentally different mechanism by which transformers store factual knowledge, challenging the prevailing assumption that weight matrices function as direct associative lookups. Rather than scaling parameter counts linearly with facts, the model encodes relational structure through geometric superposition of embeddings, with MLPs acting as selective routers. This finding reshapes how we understand transformer memory efficiency and has implications for scaling language models to handle vastly larger fact sets without proportional parameter growth.62
