Visa benchmark finds AI models failed 42.8% of shopping test runs
- 42.8% of runs failed
Visa benchmark finds AI models failed 42.8% of shopping test runs
Visa released the Visa Agentic Commerce Benchmark, which tested 22 AI models from 12 labs on 15 shopping challenges. Models failed 42.8% of test runs, and 54% of failed runs still ended with the AI making a purchase.
Related essays and timeline events
Hello shoppers, welcome to the world of harness
Everyday shoppers will need a harness for their AI shopping agents. Why confirming every purchase won't work, and why payment, marketplace and agent each own part of it.
Google rolls out UCP-powered direct checkout to shoppers in Canada
Google began rolling out direct checkout powered by the Universal Commerce Protocol to Canadian shoppers in AI Mode.
Shopify brings Shop Pay checkout into Muse, Claude Duet and Instinct
Shopify's Fall '26 Edition added Shop Pay checkout inside third-party personal AI agents, starting with Meta's Muse, Anthropic's Claude Duet and Instinct, without sharing the card with the agent.
Meta launches Muse in Singapore with Shopee, Grab and six other partners
Meta made its Muse personal agent available in Singapore with eight early local connector partners: Shopee, Grab, Singapore Airlines, Agoda, ShopBack, Klook, Trip.com and Lazada.
Anthropic previews Claude Duet, an agent that can buy through Link and Shop Pay
Anthropic opened a waitlisted research preview of Claude Duet, an agent that can buy for a business using Link, Stripe's digital wallet, or check out at online stores with Shop Pay.
Get the next one by email
Original essays on Hard-to-Keyword demand and agentic commerce, plus key protocol milestones. No spam.