From Heads to Neurons: Causal Attribution and Steering in Multi-Task Vision-Language Models

The deeper issue HONES addresses is that individual neurons in large multi-task models rarely do one thing cleanly: they respond to signals from multiple tasks simultaneously, which makes naive attribution methods misleading rather than merely imprecise. HONES sidesteps this by tracing causal paths through feed-forward layers rather than reading off attention head weights directly.

Mechanistic interpretability has been gaining traction as a practical concern rather than a purely academic one. The ORCA framework for SVMs covered here in mid-April approached the same underlying problem from a different angle: how do you explain what a model is actually doing without retraining it or relying on surrogates? HONES applies analogous post-hoc reasoning to a far messier substrate, the feed-forward networks inside vision-language models, where task interference is structural. The humor-understanding IRS paper from April 16 is also relevant context: both IRS and HONES decompose a complex model behavior into traceable sub-processes, reflecting a broader methodological trend toward modular causal analysis in multimodal research.

The real test is whether HONES-identified neurons can be selectively suppressed to degrade one task without measurably harming others on a held-out benchmark. If that surgical steering result holds across at least two distinct model families, the gradient-free attribution claim becomes credible at scale.