Qwen 3.8 27B struggles with arithmetic expressed as words

Willison's experiment isolates a specific failure mode: Qwen 3.8 27B doesn't merely struggle with large arithmetic, it fails differently when forced to express answers as English words rather than numerals. This suggests the weakness isn't purely computational but involves how the model tokenizes and sequences quantitative information across different representational formats.

This builds directly on Colin Frasier's GPT-4o work from early October, but it also connects to the broader diagnostic framework from 'The Missing Primitive' (arXiv, Oct 1). That paper introduced a four-dimensional benchmark to separate surface correctness from genuine mathematical understanding. Willison's word-based arithmetic test functions as a new probe within that same diagnostic space, revealing that models can fail at the Execution layer even when the underlying calculation should be identical. The pattern mirrors findings from 'Correct Answers, Invalid Traces' (arXiv, Sept 29), which showed models produce right answers through invalid intermediate steps. Here, the format shift exposes that the model's reasoning pathway depends on token representation, not abstract number sense.

If Qwen 3.8 27B's accuracy on word-expressed arithmetic matches its numeral-based performance on the same test set, the weakness is representational rather than computational. If the gap persists or widens, watch whether fine-tuning on word-based math problems closes it faster than training on symbolic computation layers would, which would indicate the fix is within the language modeling pathway rather than requiring external symbolic reasoning.