Home › Leaderboard › Benchmarks compared
LLM trading agent benchmarks compared
Several benchmarks now ask whether LLM agents can trade. They differ in what the agent may know when it decides, whether a result can be run again, and whether the model could simply remember the test period. This page sets rta-bench-1, the benchmark behind our LLM trading leaderboard, next to eight others on those points.
Every row except ours is taken from the paper's arXiv abstract page, read on 13 November 2026. "Not described" means the abstract page does not say; the full paper may. Corrections are welcome as an issue or a pull request.
The comparison
| Benchmark | News at the minute it was published | Frozen periods | Blind and named | Scored apart at the training cutoff | One-file submission that replays |
|---|---|---|---|---|---|
| rta-bench-1 (this project) Replay of 56 US large caps | Yes: licensed headlines, plus SEC 8-K filings in 2026, arrive at the minute they were published; decisions after the close, fills at the next open | 3: 2026 to date, the 2022 bear market, the 2020 crash | Both, for every model, plus a memory probe | Yes: scores after each model's cutoff are ranked, earlier ones listed apart | Yes: one JSON entry holds every reply of the model, and CI replays it to the same numbers |
| StockBench Multi-month stock trading environments | News is among the daily signals; timing not described | Not described | Not described | Not described; the benchmark is called contamination-free | Not described; released as open source |
| CLQT Year-long backtest plus four weeks of live paper trading | Not described | One year-long backtest, plus the live track | Not described | The live track runs on post-cutoff data; point-in-time access through a "TimeGate" | Not described |
| LiveTradeBench Live: US stocks and Polymarket, 21 LLMs over 50 days | Streamed live | None: live | Not described | Live data only | Not described |
| AI-Trader Live: US stocks, China A-shares and crypto | Agents search for information themselves | None: live | Not described | Live data only; called data-uncontaminated | Not described; code and evaluation data on GitHub |
| InvestorBench Stocks, crypto and ETFs; 13 LLM backbones | Not described | Not described | Not described | Not described | Not described; open-source multi-modal datasets |
| FinSABER Backtests over two decades and more than 100 symbols | Not described | Historical: two decades | Not described | Not described; targets survivorship and data-snooping bias | Not described |
| BlindTrade Historical backtest with anonymized names | Not described | Historical: 2025 up to 1 August, extended over 2024–2025 | Blind: tickers and company names anonymized, with negative controls | Not described | Not described |
| Agent Market Arena Live and continuous: crypto and stocks | Expert-checked news, live | None: live | Not described | Live data only | Not described |
Where rta-bench-1 differs
- News by the minute, in a replay. Live benchmarks avoid look-ahead by trading the present, but a model released next year cannot trade those days without having read about them. rta-bench-1 replays history instead: each headline arrives at the minute it was published, and every run passes a look-ahead check of what the model was shown. Many news backtests join headlines to prices by calendar date; in our example that turns a signal that loses 6.7 bp a trade into one that makes 32 bp.
- Three different markets. A rising 2026, the 2022 rate-hike bear market and the 2020 crash, so a model that only does well when everything rises shows as such.
- Memory is measured, not assumed away. The same model trades each period blind and with names. Scores after its training cutoff are ranked, and earlier ones are listed apart, next to a memory probe. Hiding the names shows what a model does without them; running it both ways shows how much the names were worth.
- A result is a file. An entry holds every reply of the model, so continuous integration replays it through the backtest and the numbers must match. Submitting a model is a pull request that adds one JSON file.
- Free to run.
pip install risk-aware-trading-agentsbrings the prices of the three periods and the backtester, and the prices are also on Hugging Face. The results are based on news replayed by publication time; the headline data has license restrictions and is not distributed with the package.
Live arenas answer a question frozen periods cannot: how a model trades today, on news nobody has seen. Frozen periods answer the others: whether a result can be repeated, compared and checked. The two kinds of benchmark complement each other.