Researcher tests whether AI labs optimized for niche image generation tasks

The more pointed question buried in Castillo's methodology isn't whether pelicans score higher than penguins, but whether any systematic cross-animal, cross-vehicle pattern emerges that would be hard to explain by training data frequency alone. If certain animal-vehicle pairs consistently outperform others in ways that don't track obvious internet prevalence, that would be the actual signal worth isolating.

This is largely disconnected from recent activity in our archive, as we have no prior coverage to anchor it to. It belongs, though, to a persistent thread in AI evaluation discourse: the concern that frontier models are increasingly shaped by what gets measured publicly, even when the measurements are informal or whimsical. Willison's original observation functioning as an accidental benchmark is a useful case study in how evaluation pressure can emerge from unexpected directions. The rigor Castillo applies is appropriate precisely because informal benchmarks have historically been dismissed until someone bothers to formalize them.

If Castillo or another researcher publishes cross-model results showing that models updated after Willison's original post score measurably higher on pelican-bicycle prompts than on comparable animal-vehicle pairs with no public benchmark history, that would be credible evidence of targeted optimization rather than coincidence.