Fast & Faithful Function Vectors
Researchers have refined function vectors, a technique for steering LLM behavior during in-context learning, by optimizing two critical design choices. Using gradient-based attribution methods to select attention heads improves both computational efficiency and accuracy, while distributing steering signals across multiple layers outperforms naive aggregation. The work addresses a gap in how these task representations are actually constructed, offering practitioners concrete improvements for controllable LLM inference. Public code release amplifies adoption potential among teams building interpretable or steerable systems.
Modelwire context
ExplainerThe paper's core contribution isn't that function vectors work, but rather that the way you construct them matters more than previously acknowledged. Specifically, using gradient-based attribution to select which attention heads receive steering signals beats both random selection and simpler aggregation schemes, suggesting that steering effectiveness depends heavily on targeting the right computational pathways rather than just signal strength.
This connects directly to the activation-based active learning study from June 3rd, which found that MLP activations fail to predict practical performance improvements. Function vectors face a similar challenge: they rely on identifying which model components actually drive task behavior. Where that prior work showed activation patterns alone aren't reliable signals, this paper demonstrates that gradient attribution (which traces actual influence rather than just measuring activation magnitude) provides a more trustworthy foundation for steering. Both papers converge on the same insight: naive activation-based selection underperforms methods that explicitly measure causal impact.
If teams adopting the public code report that gradient-selected heads remain stable across model scales (7B to 70B), that validates the approach's generality. If instead head selection becomes brittle at larger scales or requires retuning per model size, it suggests the method is solving a narrow efficiency problem rather than uncovering a fundamental principle about how steering should work.
Coverage we drew on
This analysis is generated by Modelwire’s editorial layer from our archive and the summary above. It is not a substitute for the original reporting. How we write it.
MentionsLarge Language Models · Function Vectors · Layer-wise Relevance Propagation
Modelwire Editorial
This synthesis and analysis was prepared by the Modelwire editorial team. We use advanced language models to read, ground, and connect the day’s most significant AI developments, providing original strategic context that helps practitioners and leaders stay ahead of the frontier.
Modelwire summarizes, we don’t republish. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.