Education's measurement problem: AI forces curriculum change but results take years

Willison's framing surfaces something the education reform conversation usually buries: the problem isn't that educators lack ideas, it's that the validation cycle for new pedagogy takes years, meaning any curriculum redesigned today won't have measurable outcomes until well after the current cohort has graduated into whatever the job market looks like then.

Stanford's revised CS146S course, covered here via Latent Space in early November, is the clearest live experiment in the space: Mihail Eric's approach of embedding open-source maintainers as reviewers is a direct attempt to solve exactly the assessment problem Willison describes. It's notable because it sidesteps the feedback-lag issue by grounding evaluation in real production work rather than institutional rubrics. Meanwhile, the Google-in-schools coverage (Ars Technica, TechCrunch, WIRED, all from early November) shows that the infrastructure layer underneath education is being restructured by vendors faster than institutions can set policy, which compounds the paralysis Willison identifies. Australian universities moving toward ChatGPT integration, per the OpenAI piece from November 5, represents the opposite institutional bet: adopt the tools and redefine competency around them rather than trying to wall them off.

Watch whether Stanford's CS146S model gets adopted by at least one other top-ten CS program within the next two academic cycles. If it does, that's evidence the field is converging on a replicable answer; if it stays isolated, Willison's paralysis thesis holds.