The core argument cuts through months of unfounded AI evangelism: most organisations deploying AI in customer experience lack any rigorous measurement framework to validate whether their implementations actually work. The piece exposes a fundamental gap between confidence and evidence—teams are making decisions based on anecdotal preference (does Claude perform better with politeness?) rather than controlled testing, creating an environment where vendor claims and internal assumptions go unchallenged. For CX professionals managing Zendesk, Freshdesk, or Salesforce implementations, this represents a critical blind spot. If your team has deployed AI-assisted routing, response generation, or ticket classification without establishing baseline metrics and A/B testing protocols, you're operating on assumption rather than data. The question becomes urgent: how many of your current AI initiatives would survive scrutiny against a control group, and what are you actually measuring—deflection rates, CSAT, resolution time, or simply adoption?
The implications extend beyond individual implementations to how vendors position their AI capabilities and how procurement decisions get made. When organisations cannot articulate what success looks like before deployment, they become vulnerable to feature bloat and misaligned tool selection. A Zendesk admin implementing Agentforce without pre-defined KPIs and post-deployment measurement cadence is essentially running an uncontrolled experiment on live customer interactions. The broader ecosystem problem is that this measurement vacuum creates perverse incentives: vendors optimise for impressive demos rather than measurable outcomes, and internal stakeholders defend investments based on narrative rather than performance data. For support leaders and CX consultants, the immediate action is establishing what you're actually testing for—whether that's cost per resolution, first-contact resolution, or customer effort score—before any AI tool touches your ticketing system.
If you can’t measure it, you’re guessing. I have watched grown adults argue about whether Claude does better work if you say please and thank you. One of them had a whole theory. None of them had tested it. Ivan Pavlov never had subjects this willing. That’s roughly the state of AI advice. Someone t