Output tokenization shapes model learning more than input tokenization

Researchers challenge the conventional view of tokenization as a mere input preprocessing step, demonstrating that output token granularity functions as a hidden supervision mechanism in autoregressive models. By decoupling input and output tokenization in controlled numeric reasoning tasks, the team shows that model performance, learning dynamics, and internal representations are primarily shaped by what the model must predict at each step, not how inputs are segmented. This reframes a foundational design choice in LLM architecture as a critical lever for task difficulty and representation learning, with implications for how practitioners should think about tokenizer selection across different model families and applications.

MentionsarXiv

Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “When Tokenization is Secretly Output Supervision”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.