Three Models of RLHF Annotation: Extension, Evidence, and Authority

The paper's sharpest contribution isn't the taxonomy itself but the diagnosis: most RLHF teams are already committed to one of these models without knowing it, which means their validation logic and their collection logic are often pulling in opposite directions by design.

This connects most directly to the 'paradox of AI fluency' coverage from late April, which showed that user sophistication shapes model interaction in ways that standard evaluation frameworks don't capture. That finding is partly a symptom of the problem this paper names: if annotators are treated as proxies for designer intent (the extension model) but the actual user population behaves like skilled collaborators iterating on hard problems, the reward signal is calibrated to the wrong population entirely. The fluency paradox piece documented the gap at the product layer; this paper offers a vocabulary for tracing that gap back to annotation design choices made much earlier in the pipeline.

Watch whether any major RLHF tooling providers (Scale AI, Surge, or internal teams at frontier labs) publish annotation guidelines that explicitly name which model they're operating under within the next 12 months. Adoption of the vocabulary in practitioner documentation would be a concrete signal that the framework is doing real work rather than staying inside academic discourse.