AI-powered customer service tools are generating numbers that sound plausible but fail basic accuracy tests, creating material liability for businesses that treat LLM outputs as settled facts. The core problem is architectural: large language models predict text token-by-token rather than performing deterministic calculations, which means they cannot reliably produce figures customers will act on—price quotes, shipping estimates, tax calculations, payment amounts. Research from Omni Calculator's ORCA Benchmark found accuracy rates ranging from 48.4% to 70.4% across leading models, with Claude and ChatGPT actively reversing correct answers into incorrect ones 60–65% of the time when questioned. The BMW dealership incident exemplifies the real-world consequence: an AI chatbot passed a loan balance to a customer as a buyback offer, eroding trust only after a salesperson intervened and media pressure forced reinstatement. With 77% of SMBs now using AI regularly and customer service ranking among the top three use cases, this failure mode is no longer theoretical.
The solution is architectural separation: models should interpret customer requests and select appropriate tools, but deterministic engines—calculation APIs, scripts, or services like Wolfram Alpha—must produce the actual numbers. This distinction matters acutely for CX teams already embedding AI into ticketing systems and customer portals. When a support agent uses an AI assistant to draft responses, hallucinated figures embedded in those responses become your liability the moment a customer acts on them. The reproducibility test is straightforward: if the same input must always return the same output, the model cannot be the source of truth. For teams running Agentforce, Zendesk's AI layer, or similar platforms, this means auditing which numerical outputs flow directly from the model versus which are routed through deterministic backends—and whether your current architecture even permits that routing.
The implications extend beyond individual transactions to systemic trust. Customers cannot reliably distinguish between a plausible-sounding error and a correct calculation, meaning the reputational damage compounds across multiple interactions before detection. CX leaders must treat this as a governance issue, not a feature limitation: any AI-generated number that influences customer decisions requires explicit validation against a deterministic source before delivery. This is particularly urgent for smaller vendors building AI-native support tools, where the cost of a single high-profile miscalculation could outweigh the efficiency gains of automation.
The Test Every Business Should Run Before Letting AI Calculate Prices, Payments and Other Numbers Customers Act On entrepreneur.com