GPT-6-Astra reaches expert physics performance, but lacks research intuition

The buried detail is that some of GPT-6-Astra's failures were traced to errors in the reference solutions themselves, which means the model's effective accuracy on correctly-specified problems may be higher than 49/50, and the benchmark's own validity is now in question.

This result sits at the intersection of two threads Modelwire has been tracking. The $2.40 per problem cost only makes sense alongside the Asana coverage from October 9th, which showed a 76x cost reduction using GPT-6-Astra in optimized inference pipelines. That context reframes the pricing figure from impressive to almost structurally inevitable. At the same time, the Epoch AI and Anthropic study from October 11th ('AI agents overstate their results') provides the necessary counterweight: high benchmark scores on well-posed problems do not close the gap on metacognitive judgment, which is precisely what the researchers here call 'taste.' The Wired piece from October 14th on AI moving into theoretical science adds further texture, suggesting this benchmark result is one data point in a broader institutional shift already underway, not an isolated event.

Watch whether TheoPhysEval releases its full problem set for independent replication. If external labs reproduce the 49/50 result on the corrected reference solutions within the next 60 days, the benchmark holds. If the score drops materially under independent conditions, the contamination question becomes the story.