Google checks websites for llms.txt in new agentic browsing audit
Google is expanding Lighthouse, its web performance audit tool, to measure how well websites accommodate AI agents through a new 'Agentic Browsing' category that checks for llms.txt compliance. This signals a structural shift in how the web is being optimized: rather than just human visitors, sites must now account for machine agents crawling and interacting with their content. The move reflects growing pressure on publishers and platforms to establish machine-readable protocols for AI access, effectively standardizing agent behavior expectations across the internet. For developers and site owners, this represents a new compliance surface alongside SEO and accessibility.
Modelwire context
Analyst takeThe more consequential detail is that Google is the one defining what 'agentic readiness' looks like, not a neutral standards body. By baking llms.txt compliance into Lighthouse, Google effectively sets the default benchmark that developers optimize against, which concentrates significant norm-setting power in the same company whose search dominance is already under pressure.
This connects directly to our coverage of 'Six search engines worth trying now that Google isn't really Google anymore' from TechCrunch this week. That piece framed Google's AI Overview expansion as a displacement of traditional link-based discovery. The Lighthouse move is the supply-side complement to that demand-side shift: if Google's agents are increasingly how content gets surfaced, then llms.txt compliance becomes the new SEO, and Google controls both the crawler and the audit tool that tells you whether you pass. Publishers who were already navigating AI Overview visibility now have a second optimization layer to manage, defined entirely on Google's terms.
Watch whether Bing's developer tools or any independent accessibility auditor adopts llms.txt as a formal check within the next two quarters. If they do, the standard gains legitimacy independent of Google; if they don't, this remains a Google-specific compliance surface with all the lock-in that implies.
Coverage we drew on
This analysis is generated by Modelwire’s editorial layer from our archive and the summary above. It is not a substitute for the original reporting. How we write it.
MentionsGoogle · Lighthouse · llms.txt · Agentic Browsing
Modelwire Editorial
This synthesis and analysis was prepared by the Modelwire editorial team. We use advanced language models to read, ground, and connect the day’s most significant AI developments, providing original strategic context that helps practitioners and leaders stay ahead of the frontier.
Modelwire summarizes, we don’t republish. The full content lives on the-decoder.com. If you’re a publisher and want a different summarization policy for your work, see our takedown page.