On 5 November, Visa published the first benchmark that tests AI models as shoppers. Across 22 models from 12 labs, the agents failed 42.8% of their test runs. One line further down the report sits the result I care about more: the agents that asked the shopper better questions passed far more often.
I build selling agents for webshops, so I read the report with one question in mind. What should a merchant do differently on Monday?
What Visa tested
Visa gave each model 15 shopping tasks, such as buying a TV, a laptop, or a personal care appliance. The shopper never gave every detail up front, so the agent had to find out what the person needed by talking to them. If nothing on offer fit, the right answer was to buy nothing. To rule out lucky guesses, the agent also had to collect enough information to justify its choice.
That is how real shopping works. People rarely know every requirement when they start. They know they want a moisturizer. They do not yet know they need one without fragrance, safe to use with retinol, and under thirty euros.
The results
- The models passed 57.2% of runs overall. The best passed 82.2%, the worst 28.9%.
- Each model got three tries per task, and all three passed only 34.2% of the time. A model often failed a task it had passed before.
- In 54% of failed runs, the agent still bought something. It either picked a product that missed the requirements or never found out what the requirements were.
- The five models that uncovered the most hidden needs through conversation passed 70.2% of runs. The five that uncovered the fewest passed 44.4%.
That last result is the one merchants should remember.
The best agents asked first
The strong agents won because they asked the right questions before they decided. Every good shop assistant already works this way. You ask how the customer's skin feels at the end of the day, what they use now, and what went wrong last time, and only then do you walk them to a shelf.
At CAIDEL we see the same thing in every conversation on a webshop: the questions decide the sale. An agent that skips them buys the wrong thing, and Visa's numbers show it often buys it with confidence.
The shop pays for the wrong purchase
When an agent buys the wrong product, the shop pays for it. The refund, the shipping both ways, and the disappointed customer all land on the merchant, even when the product page was perfect.
When it comes to doing the purchase step, trust is low as well, as I've also described in my previous post on agentic commerce. Visa's Trust Index from September found that 72% of US consumers had used an AI assistant, but only 23% trusted AI to handle payments for them. Agents that still buy something in more than half of their failed runs will not close that gap soon.
What to do with your webshop now
Autonomous checkout will come, on its own timeline. Three things pay off today:
- Answer the questions before they are asked. If a shopper or an agent needs to know whether a serum is fragrance-free or safe during pregnancy, the product page should say so plainly. An agent cannot ask about something your page never mentions.
- Say who the product is wrong for. The benchmark rewards agents that decide not to buy when nothing fits. Give them the facts to make that call, and you prevent purchases that would have been returned and refunded.
- Put the questions on your own store. A guided selling assistant in your shop can ask shoppers the right questions today, in your words and with your catalog, and it records what people need. A guided selling assistant helps both shoppers reading your website, and AI agents doing the reading for them.
That third point is why we built CAIDEL. The best agents in Visa's test behaved like a good advisor, and we think every WooCommerce shop should have one that works for the merchant.
Want to see how AI agents read your store today? We built a free snapshot for WooCommerce shops.
Frequently asked questions
What is the Visa Agentic Commerce Benchmark?
Visa launched it on 5 November 2026 to put AI models through realistic shopping tasks. To pass, a model must uncover every requirement of the shopper through conversation and then make the right call to buy or not buy.
Are AI shopping agents ready to buy on their own?
Not reliably. The best models passed about 82% of runs, but all three attempts at the same task passed only about a third of the time.
What should a webshop do about it?
Make product pages answer the questions shoppers ask, including who a product is wrong for, and put a guided selling assistant on your own store so the right questions get asked before the sale.
Sources
Visa Agentic Commerce Benchmark. Visa press release, 5 November 2026. https://usa.visa.com/about-visa/newsroom/press-releases.releaseId.22936.html
Coverage and model rankings. The Next Web, "Visa tested 22 AI models as shoppers, and they failed 42.8% of the time". https://thenextweb.com/news/visa-ai-shopping-benchmark-agentic-commerce-ai-models-fail
