Visa benchmark: 54% of failed AI shopping runs still ended in a purchase
Visa’s benchmark, released on 5 November 2026, tested 22 AI models from 12 labs on 15 shopping tasks; the models failed 42.8% of runs.

Visa on 5 November 2026 released the Visa Agentic Commerce Benchmark, which measures how reliably AI models make decisions in realistic shopping scenarios. In the first round, 22 models from 12 labs were tested on 15 shopping tasks and failed 42.8% of runs. The more striking finding: 54% of failed runs still ended in a purchase.
How the test works
Visa says shopping is hard for AI because consumers often do not know exactly what they want at the outset; they may know they want a TV but not the specifications they need. In the benchmark, models must uncover the customer’s needs through conversation. If nothing meets the requirements, the AI should not buy. To rule out lucky guesses, the model must also gather enough information to justify its decision.
Tasks include buying a TV, a laptop or a personal care appliance. In Visa’s example, the customer mentions a TV brand recommended by a relative and a membership with a third-party supplier but leaves other details unsaid. The model has to discover the budget, format compatibility, HDR10+ support for family videos and gaming capabilities through conversation.
Results: best 82.2%, worst 28.9%
- The overall pass rate was 57.2%. The best models passed 82.2% of attempts and the weakest 28.9%.
- The five models that uncovered the most hidden buying factors averaged a 70.2% pass rate; the five that uncovered the fewest passed 44.4%.
- The five models with the highest average cost per run passed 67.1%, versus 50.2% for the five cheapest.
- Each model attempted every scenario three times. Across all models and scenarios, all three attempts passed only 34.2% of the time.
- About half (54%) of failed runs still ended with the AI making a purchase. In those runs the model either recommended a product that did not meet the identified requirements or failed to uncover all of them.
Visa notes that models repeating a scenario they had already passed often failed, showing how difficult it is to deliver a consistent consumer experience.
The trust problem
“AI agents have the potential to power the next era of commerce, but trust is key to making it work,” said Rubail Birwadker, Global Head of Growth at Visa. Visa’s Trust Index, published in September 2026, found that while 72% of US consumers had used an AI assistant, only 23% trusted AI to handle payment transactions on their behalf.
Visa stresses that the results do not mean AI agents are not ready for commerce; the top models performed promisingly, but consistency and reliability remain challenges.
Checkout and payment tests next
The first round measures only needs discovery and the decision to buy or not. Visa says it will add shopping scenarios and product categories and extend testing to checkout, payment and order placement. The benchmark has been shared with leading model providers for feedback. Consumers, researchers, developers and merchants can try the test themselves on Visa Payments Frontier, watch replays of AI attempts and read the whitepaper.
Visa says the benchmark is not directly part of Visa Intelligent Commerce, its portfolio of agentic commerce products, but complementary research for the wider industry. The results reflect model behaviour in controlled test scenarios, not real-world sales.
AI-assisted translation from the Turkish Ecomtrex source.
Frequently asked questions
How often do AI agents buy the wrong product for a customer?
In Visa’s test 42.8% of runs failed, and 54% of those failed runs still ended in a purchase: the model either recommended a product that did not meet requirements or failed to uncover them all.
Are the models consistent?
No. Each model tried every scenario three times; all three attempts passed only 34.2% of the time.
Does the test cover checkout?
The first round measures only needs discovery and the purchase decision. Visa says it will extend testing to checkout, payment and order placement.
Can merchants try the benchmark?
Yes. It is publicly available on Visa Payments Frontier, with replays of AI attempts and a whitepaper.
Sources
- Visa Releases First-of-Its-Kind Industry Benchmark for AI Shopping; Almost Half of Test Runs Fail. Visa. Accessed: November 8, 2026.









