Chartographer: Counterfactual Chart Generation for Evaluating Vision-Language Models

The deeper issue Chartographer surfaces is not just that models cheat on chart benchmarks, but that the field has lacked a principled way to distinguish visual reasoning from distributional shortcuts at the chart-structure level. Reverse-engineering charts into executable code is the enabling mechanism here, and it's what makes the counterfactuals controlled rather than arbitrary.

This connects directly to the same-day finding covered in 'Real Images, Worse Judgments,' where adding visual inputs to VLM tasks degraded rather than improved performance. Both papers are probing the same underlying question from different angles: do multimodal models actually use visual information in the way we assume, or do they route around it? Chartographer adds a generation-based diagnostic to what has mostly been a passive observation problem. Together, these two pieces suggest a broader reckoning with how the field has been measuring visual understanding, and that current benchmarks may be systematically flattering model capability.

Watch whether the major chart QA benchmark maintainers, particularly ChartQA and FigureQA, incorporate counterfactual variants into their next evaluation releases. If they do not within the next two benchmark cycles, Chartographer risks becoming a cited methodology that doesn't actually change how frontier models are compared.