Home › Guides

Test LLM trading agents on a frozen market

Asking an LLM what it would have bought last spring is a poor test: it may know what happened, and a careless setup hands it tomorrow's prices or headlines. Risk-Aware Trading Agents lets any model trade the 2026 US stock market one close at a time, on data that holds only what was known that day, and saves every order with the result.

Why LLM trading results are hard to trust

• Memory. A model trained on text from the test period may recall how a stock or the market did.
• Look-ahead in the prompt. Prices, indicators or headlines from after the decision slip into the context.
• Fills that could not happen. Trading at the same close the model just read.
• No record. Without the exact data, prompts and orders, nobody can check or repeat the run.
• Untrusted text. Headlines are written by others; the model should treat them as quotes, never as instructions.

How a session works

On each decision day the model gets an observation: the market regime and VIX, fear and greed, sector breadth and the macro events close by; its own portfolio; and for every stock the main indicators and up to three headlines published before the close, each with its verified status. It answers with orders, which fill at the next open:

{"orders": [
  {"symbol": "AAPL", "action": "buy",    "weight": 0.10, "reason": "pullback to EMA50 in an uptrend"},
  {"symbol": "TSLA", "action": "sell",   "fraction": 1.0, "reason": "red flag in the news"},
  {"symbol": "MSFT", "action": "target", "weight": 0.05, "reason": "trim"}
], "note": "hold the rest"}

Positions are long only, at most a set number at a time. The model can decide every day or, to save tokens, once a week. The run is saved like any experiment: data hashes, settings, every step's orders, the equity curve against SPY and the equal-weight universe, and statistics with confidence intervals.

Blind mode

With blind mode on, tickers become S01, S02 and so on, dates become day numbers, prices are rebased to 100 on the first day, and company names and tickers in headlines become the same aliases, so a headline reads "S01 Stock Moves Lower As Investors Assess Its AI Edge Against S03". It lowers the chance that the model recalls what happened; it cannot rule it out, since an unusual headline, a sector and a price path together may still give a stock away.

Run any model

riskaware research trade --llm ollama:qwen2.5:3b --universe core --decide-every 5 --blind
riskaware research trade --llm openai:gpt-5.6-luna --universe megacap --decide-every 1
riskaware research trade --llm claude-cli --universe core --decide-every 5

Local models run through Ollama; OpenAI-compatible servers, Anthropic, and the Claude Code, Codex and Gemini command-line tools work too. From Python, any function from an observation to orders is a policy:

from riskaware.research import Experiment
from riskaware.research.agent import llm_policy, run_policy

result = run_policy(llm_policy("ollama:qwen2.5:3b"),
                    Experiment(universe="core", max_positions=5), decide_every=5, blind=True)
print(result.summary()["metrics"], result.folder)

Let Claude trade it, or test your strategy, over MCP

Connect the package's MCP server once (riskaware mcp-setup claude-desktop --apply, or claude mcp add riskaware -- riskaware mcp). Claude can then trade the year itself with trading_session_start and trading_session_step, look at any day with market_day, or turn a strategy you describe into code and test it with run_experiment:

The code Claude writes is checked for dates, tickers and side effects, runs in a separate process, and comes back with the metrics, confidence intervals and a look-ahead verdict: every experiment is re-run with a random future after three cut days, and the decisions before each cut must not change.

Tips for papers

• Say which model, how often it decided, whether it ran blind, and give the run folder.
• Fix the period and the settings before looking at results; tune on the first half and test once on the second (split="2026-07-01").
• Report the intervals: ten months is a short sample.
• Compare with the five swing baselines and the equal-weight universe, not only with SPY.

More guides