RespondeoQA: a Benchmark for Bilingual Latin-English Question Answering

The benchmark's sourcing from historical pedagogical materials is the detail worth pausing on: these are texts designed to teach Latin, meaning the QA pairs test grammatical reasoning and translation judgment rather than factual recall, which is a fundamentally different failure mode than what most benchmarks expose.

Modelwire has been tracking a wave of domain-specific benchmarks across April 2026, including QuantCode-Bench for algorithmic trading and MADE for medical adverse event classification. The pattern across all of them is the same: general-purpose models underperform when a task requires combining domain knowledge with structured reasoning rather than pattern-matching against training data. RespondeoQA fits squarely in that trend, and the Latin case is arguably the clearest illustration yet because the language's morphological complexity makes surface-level retrieval nearly useless. DiscoTrace, covered around the same period, adds a related angle: LLMs systematically lack rhetorical variety, which would compound the difficulty of producing well-formed Latin constructions that depend on precise syntactic choices.

Watch whether any of the tested models, particularly o3-mini given OpenAI's current reasoning-focused development track, show measurable improvement on skill-oriented subsets if retested with chain-of-thought prompting. That would clarify whether the gap is a reasoning deficit or a training data gap, and the distinction matters for how the field responds.