ChatGPT shows strong product bias in commercial advice audit

The audit doesn't just measure inconsistency; it quantifies how dramatically different vendors' disclosure norms are (79% vs 7% vs 2% preference statements). The buried finding is that consistency itself varies wildly on repeat queries, meaning even the 7% and 2% figures may mask unreliable behavior within those vendors' own products.

This connects directly to the EviGen work from earlier this month, which tackled a parallel problem in clinical AI: systems that generate plausible-sounding output without grounding in verifiable evidence. Both papers expose a core tension in LLM deployment: end-to-end generation feels fluent but obscures where the model is actually reasoning versus confabulating. The product recommendation audit is essentially asking whether ChatGPT's higher preference-expression rate reflects genuine reasoning or just more confident hallucination. The methodological rigor here (auditing consistency across repeated queries) mirrors how the tokeniser decomposition paper from the same week separated confounded variables to isolate what actually drives behavior.

If OpenAI or Google release transparency reports in the next 60 days that explain the preference-expression gap by reference to their training data or RLHF objectives, that's a signal they're taking the audit seriously. If neither vendor responds with specifics and instead pivots to generic 'responsible AI' statements, the finding will likely harden into a competitive liability that regulators cite in future guidance on AI advisory services.