All Green, Still Broken: Real-Flow Verification Lessons from an LLM-Integrated, Multi-Market Web Application

The buried lede is not that testing was insufficient, but that the team had extensive testing and still shipped bugs at scale, meaning the failure mode is architectural: LLM outputs create integration surfaces that standard unit-test philosophy was never designed to cover.

The related sign language recognition paper from arXiv on June 21 is largely disconnected from this story in terms of subject matter, though both involve LLM-adjacent pipelines where outputs feed downstream components. The more relevant context for this finding sits in the broader pattern of production AI deployments discovering that model-level correctness does not guarantee system-level correctness. The rental-search case is a concrete, quantified instance of a problem the field has discussed abstractly: when a language model sits between user intent and application logic, the failure surface shifts to the joins between components rather than the components themselves. That shift is not well-served by inherited software engineering practices built around deterministic functions.

Watch whether testing framework vendors (Playwright, Cypress, or similar) publish LLM-specific integration test primitives within the next six months. If they do, it signals the industry has accepted this failure mode as structural rather than a one-off implementation mistake.