Coinbase Finds Newer AI Models Detected Less Fraud in Payment Screening Test
Upgraded AI Models Caught Less Fraud, Coinbase Data Shows
Coinbase reported on Oct. 7 that newer versions of three major AI model families caught fewer fraudulent payments and a smaller share of fraud value during a historical replay of its Onramp payment screening service. The results challenge the assumption that upgrading a model automatically improves an existing fraud screener. The evaluation replayed 16,140 transactions from 7,293 users, including 813 confirmed fraudulent transactions, across a nine-week period before Coinbase's risk agent was rolled out. Every newer model version tested showed lower recall, which measures the share of fraud cases a model catches, as well as a lower combined precision-and-recall score and lower dollar-weighted recall, which measures how much of the total fraud value is caught.
Key Results From the Replay
- Sonnet's recall fell 22.2 percentage points, and its dollar-weighted recall dropped 22.9 points.
- Opus's recall declined 0.8 percentage points.
- GPT-5.6's precision rose 11.5 percentage points, but its recall fell 20.7 points and dollar-weighted recall fell 21.8 points, meaning fewer fraud cases and less fraud value were detected despite more accurate fraud flags.
- Precision measures how many transactions flagged as fraud are actually fraudulent. Both newer Opus and Sonnet models also had lower precision.
The replay did not measure actual customer losses from deploying the newer versions. Coinbase said it could identify the regressions but could not establish their cause.
A Custom Model Performed Better
In an Oct. 8 disclosure, Coinbase reported that a post-trained custom model based on Qwen3.5-9B exceeded Opus 4.5 across four fraud-detection metrics. F1 improved by 9.6 percentage points and dollar-weighted recall rose by 35.4 points. The model was specialized using Coinbase's historical fraud outcomes and deterministic rewards that balanced fraudulent and legitimate examples.
Production measurements also showed median end-to-end LLM request latency at 0.683 seconds for the custom model, compared to 1.515 seconds for Opus 4.5 — a 55% reduction. Faster inference and stronger benchmark detection came from separate evaluations, Coinbase noted.
What Researchers and the Company Caution
Coinbase's researchers said the proprietary dataset cannot be released, which restricts independent replication and generalization. The related payment-fraud study first appeared on Sept. 23 and was revised on Sept. 30, before the October blog posts.
Coinbase previously ran an online experiment comparing a selective LLM review added to its existing models and rules. That agent-enabled flow recorded 30% fewer fraudulent transactions and 22% less fraud value, but it did not compare newer model versions against each other.
What This Means for Payment Screening
For payment providers, the relevant question is whether a candidate model improves fraud coverage under their actual decision setup. Coinbase recommends testing the full configuration first before evaluating changed prompts or thresholds separately, and assessing latency, reliability and cost alongside detection quality.
What Is Confirmed and What Is Not
Confirmed: In a fixed-policy historical replay, newer versions of Opus, Sonnet, and GPT detected fewer fraud cases and less fraud value than their predecessors. A custom post-trained Qwen3.5-9B model outperformed Opus 4.5 on multiple metrics. The replay did not measure actual customer losses.
Not established: The cause of the performance regressions in the newer models, and the impact on real customer losses from deploying them.
Why This Matters
Many crypto companies use AI to screen payments for fraud, assuming that newer or larger models perform better. Coinbase's findings show that simply upgrading a model does not guarantee improved fraud detection and may reduce it. Companies relying on AI payment screening should test full configurations under their own decision policies before deploying upgrades.
Newisty Editorial Team covers technology, cryptocurrency, digital products, online platforms, developer tools and the wider digital economy. Our content is researched from official sources, company announcements, public documentation, market data and other primary or reputable sources. Articles are reviewed and edited before publication for clarity, accuracy and useful context.