Here’s the uncomfortable version. The pattern across independent evaluations is consistent: the headline numbers you see in framework READMEs don’t survive contact with live markets, adversarial conditions, or contamination checks.
Live benchmarks contradict the backtests
AI-Trader (University of Hong Kong, arXiv 2512.10971) is the most direct test: a fully autonomous “minimal information” setup across US stocks, A-shares and crypto, where agents must search and verify live information themselves with no human intervention.
Its conclusion is blunt — general intelligence does not translate into trading capability, and most agents show poor returns and weak risk management.
In the US market:
- ·GPT-5 managed a 1.56% cumulative return against a QQQ benchmark of 1.87%
- ·Qwen3-Max got 0.39%. In A-shares both lost money (−3.53% and −3.86%, Sortino ratios of −1.54 and −1.40).
Crypto was worse:
- ·Gemini-2.5-Flash fell 18.63% in a passive “hold-to-die” trajectory during the November correction
- ·GPT-5 lost 16.41% by clinging to target weights without liquidity awareness. The authors note excess returns came more readily in liquid US markets than in policy-driven A-shares, and that risk control — not reasoning — drove cross-market robustness.
StockBench (Chen et al., contamination-free, multi-month) reaches the same verdict: most LLM agents struggle to beat simple buy-and-hold. The numbers are worth staring at, because the returns are trivial relative to the risk taken.
Small Returns, Large DrawdownsBest model gained 2.5% while max drawdowns ranged from -11% to -15%
LiveTradeBench (UIUC, arXiv 2511.03628) ran 50-day live evaluations of 21 LLMs and found that high LMArena scores do not imply superior trading outcomes — a direct refutation of the assumption that a better general model makes a better trader.
Separately, TradeRank Arena, a live crypto benchmark that has run ten seasons since January 2026 with 69 models and 2,921 trades under identical data and risk rules, reports that only 43.1% of model-seasons finished profitable, with no model winning consistently across regimes. That’s a commercial research site rather than a peer-reviewed venue, so weight it accordingly, but the direction matches everything else.
One counterpoint worth noting honestly: Agent Market Arena (arXiv 2510.11695), described as the first lifelong real-time multi-market benchmark, found agents often outperformed simple buy-and-hold over a two-month window, with InvestorAgent on GPT-4.1 returning 40.83% on TSLA at a 6.47 Sharpe. But the accompanying presentation drew the more useful conclusion: swapping the LLM backbone inside a fixed agent design produced very similar results, while changing the architecture with the model held constant produced massive differences. Position logic, coordination and risk control matter more than which model you pick.
Adversarial conditions break them quietly
TradeTrap (Shanghai AI Lab, arXiv 2512.02261, December 2025) is the most damaging finding, because it attacks the plumbing rather than the reasoning. It decomposes agents into market intelligence, strategy formulation, portfolio/ledger handling and trade execution, then applies controlled perturbations in a closed-loop backtest on real US equity data.
Prompt injection collapsed an adaptive agent’s total return from 7.81% to 0.89%, annualized return from 149.64% to 11.35%, and Sharpe from 5.72 to 0.29 — while trading frequency exploded from 47 to 391 trades and position concentration approached 100%. Position attacks cut total return to 1.88%, Sharpe to 1.58 and Calmar from 64.36 to 7.45, with drawdown rising from 2.33% to 3.41%. The authors’ framing is the key line: small semantic or state distortions amplify into large financial losses without triggering conventional safeguards, revealing a gap between nominal decision accuracy and true financial reliability.
A related piece in The Next Web makes the arithmetic concrete: if a workflow needs five sequential API calls and each succeeds 97% of the time, the whole chain completes cleanly about 86% of the time — and an agent reads a timeout as missing data, so it acts on stale information rather than asking again.
Contamination is the biggest asterisk on every backtest
This is the methodological problem that undermines most impressive published results. Glasserman and Lin (2023, arXiv 2309.17322) formalized it: look-ahead bias, where the model already knows what happened after a news item, plus a “distraction effect” where general company knowledge interferes with sentiment judgment. Their striking result was that anonymized headlines actually outperformed originals in-sample, implying the distraction effect mattered more than look-ahead bias — but out-of-sample, look-ahead disappears while distraction persists.
Practitioner analyses push harder. A write-up on PapersWithBacktest cites work suggesting GPT-4o can recall exact S&P 500 closing prices with very low error, meaning a “decision” about March 2024 may be recollection rather than reasoning. Another analysis describes Llama 3.1 and DeepSeek posting returns above 44% on 2021 stocks purely from memorized history, while point-in-time-disciplined models delivered roughly 7% excess return — a plausible, unexciting number. An essay at jiripik.com argues that the old critique (“the model hallucinates the backtest”) is dead now that models execute real sandboxed Python, and that execution laundered the problem rather than solving it: a memorized price path, a silent best-of-fifty search over variants, and a snooped published factor all still contaminate the result, and a suspiciously clean equity curve is the tell rather than the trophy.
Even the framework authors concede this. The TradingAgents repository warns that backtest results won’t reproduce run to run because LLM output is non-deterministic, and states plainly that it should be treated as a research scaffold, not a strategy with a fixed return. Community reports of a 30-day run returning about 7% against the S&P 500’s. 4.5% came with 22% drawdowns, and the team recommends against using real money.
The moving target problem
A fair critique of the benchmarks themselves: AI-Trader and similar arenas answer a narrow research question — can today’s agents make sequential portfolio decisions under causal uncertainty better than simple baselines? — not the consumer question of whether you should let AI trade for you. On the narrow question the answer is: sometimes, in particular windows and markets, very inconsistently, and mostly through risk control rather than insight. Benchmarks age badly too, since models, markets and the contamination boundary all shift.
What actually survives scrutiny
Four things hold up:
- Risk control beats reasoning — the AI-Trader and Agent Market Arena results converge on this, and it’s the opposite of what the marketing implies.
- Architecture beats model choice — swapping backbones barely moved results, while redesigning the agent loop moved them enormously.
- Robustness is unaddressed — nobody shipping these systems has a convincing answer to TradeTrap’s perturbation results, which is exactly the governance gap Forbes flagged around banks granting agents transactional authority.
- Clean backtests are a red flag, not a green light — the discipline that matters is post-cutoff evaluation windows, wide universes, long horizons, and beating buy-and-hold on genuinely out-of-sample data.
So the honest summary: AI can now design and execute trading systems competently, and the open-source scaffolding is genuinely good engineering. What it cannot yet do reliably is make money in live markets, and the published evidence that it can is mostly contaminated.
Bridgewater’s quiet $2 billion ML fund remains the outlier that suggests real capability exists somewhere — which is precisely why it’s quiet.