Selection bias reverses NLI monotonicity findings in unselected populations

The critical finding isn't just that selection bias exists, but that it inverts the relationship between linguistic properties and human agreement. Prior work on a curated subset found non-monotonicity correlated with disagreement; the full population shows the opposite. This suggests the curated subset wasn't representative of the underlying phenomenon.

This connects directly to the evaluation-quality work from last month, particularly the GAMUT paper on factual completeness and the CircuitKIT toolkit piece. Both emphasized that benchmarks and evaluation frameworks can systematically misrepresent model behavior when they're built on incomplete or biased samples. Here we see the same problem at the annotation stage: if you validate your hypothesis only on a filtered dataset, you may be measuring the filter's properties, not the phenomenon itself. The preregistration aspect also echoes the reproducibility concerns surfaced in the prompt-design study, where controlled methodology became necessary to isolate real effects from confounds.

If subsequent work on other NLI datasets (e.g., XNLI or newly collected unselected corpora) replicates the higher-agreement finding for non-upward operators, that confirms the original ChaosNLI result was an artifact of curation. If the effect reverses again or disappears, it signals the relationship is genuinely dataset-dependent and researchers need explicit criteria for when curated subsets are valid for generalization.